lftp: backup and mirroring

This is a wiki page. Be bold and improve it!

If you have any questions about the content on this page, don't hesitate to open a new ticket and we'll do our best to assist you.

You can use the lftp mirror command to make a backup of a web site.

Since the whole command can get very long very quickly, it would be burdensome to have to type it in its entirety each time, so we are going to write the command once into a file, and tell lftp to execute the content of that file.

Let's call that file lftp_command.txt. It can then be executed this way:

lftp -f /home/user/lftp_command.txt

Now, let's start creating the content of lftp_command.txt. The first thing we need to do is to connect to the remote server. You can do so easily by explicitly setting the site, username and password:

lftp -u username,password ftpsite

However, in our example, we'll assume that you have a bookmark called remote_site with the proper credentials, so that you can connect this way:
lftp remote_site

Remember: put this within the lftp_command.txt file and execute it as expressed above.

Next, we are going to actually mirror the content of the remote server onto our local machine. See the man page for all the arguments and flags that the command mirror can take. The most basic usage is:

mirror path/to/source_directory path/to/target_directory

Here are some of the most useful switches that you can use:

--delete                  delete files not present at remote site
--only-existing           download only files already existing at target
--only-newer              download only newer files
--no-recursion            don't go to subdirectories
--reverse                 reverse mirror (put files)
--include RX              include matching files
--exclude RX              exclude matching files
--verbose                 verbose operation

Check the lftp man page for all the advanced flags that mirror can take.

Now, our lftp_command.txt has the following:

lftp remote_site -e "mirror --verbose --only-newer path/to/source_directory path/to/target_directory"

The -e switch tells lftp to execute the command that follows.

If you want the command to be executed at a specified time, use the at switch and add an ampersand at the end of the command, so that the process is detached from your console window (which you can then close). Here, the command will be executed at midnight:

lftp at 00:00 remote_site -e "mirror --verbose --only-newer path/to/source_directory path/to/target_directory" &

And to upload a site from a local source, simply user the --reverse switch:

lftp remote_site -e "mirror --verbose --only-newer --reverse path/to/source_directory path/to/target_directory" &

Alternatives

Note that it would be best to ensure that your ftp connection is secure. Check the section using lftp on a secure network within this site's wiki.

Also, note that other tools offer more or same the same feature, secured by default, e.g. rsync:

rsync -a -e ssh user@from.server.com /backup/to/location/locally

Check also: unison, bacula, amanda.