Remove all versions from S3 bucket using AWS tools
Published by Nicholas Dunbar on March 26th, 2015
This will help you answer any of the following questions:
Delete all deletion markers for every object in a bucket.
Delete all objects and versions for every object in a bucket.
Delete a bucket that has had versioning turned on.
This article will walk you through all the steps needed to remove all your S3 object's pervious versions including delete markers using the AWS provided command line tool "aws" and then remove the bucket.
Create A User
First you need to create a user that has access to your S3 Buckets. If you already have this then script to step 9.
1. ) Go to https://console.aws.amazon.com/iam/home
2.) Click users
3.) Create user
4.) Enter user name
5.) Copy your security credentials to a file (This will be the last time you will be able to access them)
6.) Back at the main user list screen, edit the user by clicking it in the list
7.) Attach a policy that allows the user to access the S3 service (click attach policy)
8.) Search for "S3" in the policy filter and then select "AmazonS3FullAccess"
9.) Click "Attach Policy"
Delete all versions
10.) Disable versioning in your bucket
11.) Open up a terminal (I'm using max OS X, but these interactions will work also on a Linux Machine
12.) Use SSH to login to your EC2 server or you can also run it locally. To run it locally you will have to have installed was cli (command line interface) tools (http://docs.aws.amazon.com/cli/latest/userguide/installing.html). Elastic beanstalk instances come preloaded with was cli tools. It will work faster if it is on you EC2 sever since the server is in the cloud.
ssh -i myawspemfile.pem ec2-user@124.93.286.021;
13.) Define variables for the was tools command line utility by typing the following into your terminal (remember to not use my credentials and use the ones that you downloaded with the temp2 user):
prompt> export AWS_ACCESS_KEY_ID=AKIAJ3RHXVS6TJJFBGFA;
prompt> export AWS_SECRET_ACCESS_KEY=o8oeqYLMu+F9nL97+992U+8Q9wj35BEHHo2Eqs;
14.) Remove all old versions of files (cut and paste the green into your terminal, replace the red words)
prompt> echo '#!/bin/bash' > deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket yourbucket-name-backup | grep -E "^VERSIONS" | awk '{print "aws s3api delete-object --bucket yourbucket-name-backup --key "$4" --version-id "$8";"}' >> deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh;
WARNING: this will delete the entire object including all its previous versions and only leave behind delete markers.
This operation could take hours depending on how much you have in your bucket.
Here is the documentation for these commands:
15.) Remove all delete markers (deleted versions) from the s3 bucket.
prompt> echo '#!/bin/bash' > deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket yourbucket-name-backup | grep -E "^DELETEMARKERS" | awk '{print "aws s3api delete-object --bucket yourbucket-name-backup --key "$3" --version-id "$5";"}' >> deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh;
You should get a bring of JSON output like the following:
{
"VersionId": "9PcePHd64KCFxZH_ihqG5F1zg9BiMsb0",
"DeleteMarker": true
}
Now all the objects in the bucket should be definitively removed.
16.) Check the bucket and remove anything that may have been missed at
17.) Delete the user so that your apps permission return to what they should be.
18.) In your terminal you should run
prompt> export AWS_ACCESS_KEY_ID=""; AWS_SECRET_ACCESS_KEY=""; history -c;
This makes sure that your credentials can't be accessed if you didn't delete them in step 17.
Also don't forget to re-enable versioning on your bucket if you need it.
you can use
aws s3 rb s3://yourbucket-name-backup --force
if you want to delete the bucket once you have removed the versions.
You might have found this article because you were looking to answer the following questions:
How to delete a bucket using the command line
Life cycle rules does not empty a bucket with versioning enabled
aws s3 rb --force does not remove bucket
Posted on March 26th, 2015