The AWS S3 tools

The AWS S3 tools are modeled on the unix directory and file abstraction. An S3 bucket is like a unix directory, and an S3 key is like a unix file. The basic operations are: create and remove a bucket, put files into a bucket, get files from a bucket, and list the contents of a bucket.

These tools store an MD5 checksum of the data that comprises the S3 object in the objects metadata (user-md5).

The s3rsync command can be used to sync a local directory/s3 bucket with another local directory/s3 bucket, including the MD5 checksum of the data objects.

The s3snap tool creates a snapshot of a source bucket in a destination bucket. Since the AWS S3 copy tool does not always copy the source metadata, including the user MD5 checksum of the data, the s3snap-fixup tool is needed to update the destination objects with the user MD5 checksum.

AWS CLI

Ubuntu 24.01 $ snap install aws-cli --classic

Boto

The python boto3 library is the interface to the Amazon Web Services. The library can be found at https://github.com/boto.

Installing boto

See the boto3 README.

Ubuntu 20.10: $ apt install python3-boto3

Installing the AWS tools

The makefile installs the tools in the user's bin directory ($HOME/bin).

$ make install

Setting up AWS access keys

export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=YOUR_DEFAULT_REGION

API

List the S3 buckets

$ s3ls
OR
$ aws s3 ls

Make an S3 bucket

$ s3mb s3://BUCKET
OR
$ aws s3 mb s3://BUCKET --region=YOUR_DEFAULT_REGION

List the keys in an S3 bucket

$ s3ls -l s3://BUCKET
OR
$ aws s3 ls --recursive s3://BUCKET

Includes the creation date and time of the object, the object size, the MD5 checksum on the data, and the key name.

List the keys with a given PREFIX

$ s3ls s3://BUCKET/KEY-PREFIX

List the newest keys

$ s3ls -l s3://BUCKET | sort | tail

Compute the sum of the key sizes in a bucket

$ s3ls -l s3://BUCKET | gawk -M '{s+=$3}END{print s};
$ s3ls -l s3://BUCKET/KEY-PREFIX | gawk -M '{s+=$3}END{print s}'

Put a file into an S3 bucket

$ s3put BUCKET KEY FILE

Recursively put a directory into an S3 bucket

$ s3put -r BUCKET directory

Get an object from an S3 bucket

$ s3get BUCKET KEY OUTFILE

Recursively get a directory from an S3 Bucket

$ s3get -r BUCKET directory

Delete a key from an S3 bucket

$ s3rm s3://BUCKET/KEY
OR
$ aws s3 rm s3://BUCKET/KEY

Sync the contents of a destination S3 bucket (DEST) with a local directory (SRC)

$ s3rsync LOCAL-SRC-PATH s3://BUCKET/DEST-PATH

s3rsync uses the object size and the user MD5 checksum to determine if the objects differ.

Sync the contents of a local directory (DEST) with a source S3 bucket (SRC)

$ s3rsync s3://BUCKET/SRC-PATH LOCAL-DEST-PATH

Create a snapshot of an S3 bucket

$ aws s3 mb s3://NEW-BUCKET
$ s3snap s3://OLD-BUCKET s3://NEW-BUCKET

Snapshots may be useful as a backup.