Backing up and restoring S3 from a versioned bucket

Here is a set of scripts used to manage backups. You can back up a bucket to another and then restore the source bucket to a certain snap shot in time as well as a few other things.

Example:

//to run back up (see script at the bottom)

prompt> bkupToBucket.sh

Later on when you need to restore a back up...

//determine the timestamps range you want to restore from the back up 

prompt> php awsUtil.php list-timestamps example-of-a-backup-bucket-name

//output:

//restore backup from files that were versioned between these two timestamps

//this will restore version 0EHqEYAubEXtO4KqU7A316S65tWRN.xT of backup.log

prompt> export startSnapShotTimeStamp=1;

prompt> export endSnapShotTimeStamp=1427577799;

//do the restore

prompt> php awsUtil.php sync example-of-a-backup-bucket-name example-of-a-bucket-name $startSnapShotTimeStamp-$endSnapShotTimeStamp

Setting up the script

You will need to edit the aws.properties file and add credentials that you created using AWS IAM (See the script at the bottom)

Make sure you set up proper credentials for the user by giving it a user policy which only allows you to back up to the back up bucket and to read from it. Here is an example of such a policy:

Other uses for the awsUtil.php script:

1.) Copy most recent file versions in a time range from one bucket to another

prompt> php awsUtil.php copy-versions example-of-a-backup-bucket-name example-of-a-bucket-name 1427577790-1427577799

2.) Copy most recent delete markers in a time range from one bucket to another

prompt> php awsUtil.php copy-delete-markers example-of-a-backup-bucket-name example-of-a-bucket-name 1427577790-1427577799

3.) Delete all versions and delete markers between a time range

prompt> php awsUtil.php delete-versions example-of-a-backup-bucket-name example-of-a-bucket-name 1427577790-1427577799

4.) List timestamp and version info of a pattern matching a file

prompt> php awsUtil.php list-timestamps example-of-a-backup-bucket-name "*.log"

Here is the scripts needed for awsUtil.php: