Jump to content


Photo

How To Synchronize Files To A Shared Host


  • Please log in to reply
3 replies to this topic

#1 drewkeller

drewkeller
  • Members
  • 3 posts

Posted 05 February 2012 - 05:33 PM

The Problem

The ONLY protocol supported by TCH shared hosts is FTP. (Not counting the cPanel file manager and such.)

(Just so you know... FTP is insecure. Nothing is encrypted. Usernames, passwords, commands, and most data are sent in plain text so any packet sniffer can figure them out.)

Ok, so we know that FTP is insecure and inefficient, but we want to use it any way (what choice do we have?).

We've installed a web server on our local computer and designed and tested our site, at least to the point where we can put it on the internet so some other people can check it out. The normal way to get our files on the server is FileZilla (free for Windows, Mac, and Linux). Just copy the local files to the server by drag-n-drop.

But let's say some people have checked out the site and tweaked files on the server. If that's all that's changed, we could just copy all the files from the server (it might take a while). What if we've continued working on the site locally, so we have changes both locally and on the remote site. Do we want to go through each folder and find all the files that are different and figure out which one is really the newest? Not really. Even worse, what if some files contain changes both locally and on the server? FileZilla is not the tool for this.

We need a way to synchronize files both ways and resolve conflicts when both sides have been changed.

Windows Tools

On Windows, it looks like WinSCP *could* work for this (I haven't actually tried it this way). You can also use a method in Windows similar to the one I'll be discussing for Linux below... Unison works on Windows. And you can mount an FTP server as a windows drive using FTPUse (ferrobackup.com/ftpuse/)

For Development Within A Linux VirtualBox (On Windows)

I use Windows as my main computer, because Windows still has a lot more programs and such available than Linux. I use Linux (I'm using a Debian-based distro) in a VirtualBox for web development, because a lot of web server stuff is just simpler to install in Linux (or not even available for Windows... just to contradict myself).

I have not figured out a very good way of reading VirtualBox disk images in Windows, so I do the synchronization from within Linux. I had lots of trouble getting FTP to work reliably until I changed from the NAT mode (the default) to Bridged mode. So the first thing to do is change Adapter 1. For some details....
  • Run VirtualBox
  • Select the Linux virtual machine that you use (I'm assuming you have one installed... if not, I'm leaving it as an exercise for the student to install one)
  • Click the toolbar icon to open the virtual machine's settings
  • Select Network
  • Enable the adapter if it's not enabled. Change the "Attached to" setting from NAT to Bridged Adapter. For "Name", select the network adapter your computer uses. I set my promiscuous mode to "Allow all", but it might not be necessary.
  • Start the virtual machine.
Linux Tools

On Linux, it looks like csync could be used to sync to an FTP site, but I couldn't get it to work, and it's not really being developed any more. A lot of the Linux methods require the remote side to be running the same software as the local side. But shared hosts don't have that option, so we're stuck.

Or are we? Enter curlftpfs. This program mounts a remote folder onto the local file system, so we can use it just like local files. This opens a lot of doors. To get curlftpfs working (I'll show the commandline stuff because it's more universal)...
  • Install using Synaptic or some other package manager, or enter
    > sudo apt-get install curlftpfs
  • Create the folder where you want to mount the remote folder. The normal place for mounting things in Linux is /mnt/ (or perhaps /media/). Your folder name doesn't have to be the same as the site, but it makes I think it makes it more clear what it is.
    > sudo mkdir /mnt/mysite.com
  • Now we need to do the actual mounting. TCH ftp usernames are something like 'myftpusername@mysite.com'. The '@' makes this tricky and we need to encode it using %40 as below.
    > sudo curlftpfs -o allow_other myftpusername%40mysite.com:mypassword@ftp.mysite.com /mnt/mysite.com
  • So if you change to that folder and list the files, you should see them
    > cd /mnt/mysite.com
                        > ls -l
  • If you want to mount this more easily (step 3 is kind of long), add a line to /etc/fstab, like this
    > echo "curlftpfs#myftpuser%40mysite.com:maypassword@ftp.mysite.com /mnt/mysite.com fuse allow_other,rw,user,noauto 0 0" | sudo tee -a /etc/fstab
    and then to mount, you just need to
    > sudo mount /mnt/mysite.com
So now, we have the ftp site psuedo-local, it's just a matter of syncing from one folder to another. There are a whole bunch of ways to do this. One might think 'rsync', but actually, it's really only for one-way syncing. I am currently using Unison for the two-way sync.
  • Install Unison. Meld here is optional; it's for comparing the local and the remote version of the same file against each other. You can use some other diff/merge program if you want.
    > sudo apt-get-install unity meld
  • Add a profile to the settings (we'll name the profile mysite.com)
    > sudo nano ~/.unity/mysite.com.prf
    Enter the following. Change as needed.
    # roots indicate which folders are being synced. The first one appears on the left# pane of the unison GUI and the second one appears on the right pane.
                        root = /path/to/my/local/folder
                        root = /mnt/mysite.com
                        # what program to use (and how to start it) for comparisons when the same file
                        # has been changed in both the local and remote folder
                        diff = /usr/bin/meld CURRENT1 CURRENT2
                        merge = Name * -> meld CURRENT1 CURRENT2
                        confirmmerge = true
There is an excellent write-up about Unison usage here (granneman.com/techinfo/security/backup/unisonbackup/), so that's all I'm going to say about it.

#2 TCH-Bala

TCH-Bala

    Technical Support Manager

  • Staff
  • PipPipPipPip
  • 1,066 posts

Posted 08 February 2012 - 08:00 PM

The ONLY protocol supported by TCH shared hosts is FTP. (Not counting the cPanel file manager and such.)


We have SFTP running on all our servers for a few years now which is a secure way to transfer files.
Balakrishnan
Manager - Technical Support
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image

#3 drewkeller

drewkeller
  • Members
  • 3 posts

Posted 10 February 2012 - 12:24 AM

Well, that's interesting. Here's what a TCH tech told me in a recent support ticket: "there is no other protocol to transfer files from local to the server than FTP - for shared server accounts."

Semantics aside (whether SFTP aka FTP over SSH is the same protocol as FTP), either one is still pretty inefficient.

Edited by drewkeller, 10 February 2012 - 12:24 AM.


#4 TCH-Bala

TCH-Bala

    Technical Support Manager

  • Staff
  • PipPipPipPip
  • 1,066 posts

Posted 11 February 2012 - 01:56 AM

Shared hosting accounts have their limitations but does not mean they are insecure or inefficient.
Balakrishnan
Manager - Technical Support
TotalChoice Hosting, Inc.
http://www.totalchoicehosting.com

TCH Help Desk .. || .. TCH Blog

Posted Image




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users