Alex Biddle     About     Archive     Feed

Backup to S3 with duplicity

Duplicity is written in python, but is run through the shell. The shell configuration options can get quite detailed, so we’re going to use the duplicity-backup script that is available on github. So first of all, we clone this into a directory.

git clone https://github.com/zertrin/duplicity-backup.git
cd duplicity-backup
mv duplicity-backup.conf.example duplicity-backup.conf

Open duplicity-backup.conf with your favourite text editor. You generate your security credentials by signing into your Amazon account and clicking on your username.

Now we need a GPG key. I am creating a new one specifically for backup purposes. If you want to run the backup as root, then you need to use sudo su to generate the keys for the root user’s keyring. I am running backups under my regular user, so I run it with regular privileges. Before generating our GPG key, we will install haveged, using the command below so that you don’t have to waste your time tapping on the keyboard to generate entropy (this can be a particular problem with headless cloud servers). APG is optional, but it is a nice phonetic password generator that you can use to generate your secret key password. If you already have a GPG key, it is good to generate another one, as you will need to store your secret key password in plaintext. The other dependencies may be required for Python. It’s no harm running the whole command as it is.

sudo apt-get install haveged apg python-boto python-paramiko s3cmd
gpg --gen-keys
[... and follow the instructions]

If successful you should find your keys with:

gpg --list-keys
gpg --list-secret-keys

In duplicity-backup.conf, enter your secret key passphrase and copy the key signature from the list keys command above. Leave the sign key as it is.

Create an Amazon S3 bucket, with a folder like “backups” and enter it into the DEST variable in duplicity-backup.conf. The format depends on what region you are in and is detailed in the Amazon documentation. As my bucket is in Ireland, I use the Western European region:

DEST="s3://s3-eu-west-1.amazonaws.com/[snip]/[snip]" 

The other configuration options in duplicity-backup.conf should be self-explanatory. Now let’s try and run it:

 ./duplicity-backup.sh --backup
WARNING: s3cmd is not configured, run 's3cmd --configure' in order to retrieve remote file size information. Remote file size information unavailable.

As you can see, there is still a little configuration to go, so complete s3cmd –configure and the duplicity-backup script should run with no errors. The configuration file will be stored in your home directory, which may be an issue if you want to run the script as root.

In order to restore your files, have a look at options by running ./duplicity-backup.sh with no arguments. You could automate the backup with CRON, but I am just using it to backup my home directory, so run the command periodically using my current user. If you’re worried about having your GPG key saved in plaintext and you’re on a shared server just make sure that it is only readable by your user. The .s3cfg file should already only be available for your user.

chmod a-rwx duplicity-backup.conf
chmod u+rwx duplicity-backup.conf

There is no way to use asymmetric encryption using duplicity. If you’re interested in why not, then you can have a look at this thread.