Uploading Backups to AWS S3

This tutorial shows how to use the AWS Command Line Interface (CLI) to access Amazon S3, so that you can build your own scripts to back up your database backup files to the cloud and retrieve them as needed.

Use it together with Scheduled Database Backup: configure the tbdbbackup script first, then add a cron job that pushes the generated files to your S3 bucket.

Create an AWS IAM User

In this step, you create a user account that the AWS CLI uses to access AWS services.

Security note: the steps below attach the AdministratorAccess policy, as in the original procedure. For a backup uploader, attach a least-privilege policy scoped to your bucket instead (s3:PutObject and s3:ListBucket, plus s3:GetObject if you also restore from the CLI).

1. Log in to the AWS Management Console, type IAM in the search bar and select IAM to open the Identity and Access Management dashboard.

1. From the AWS Identity and Access Management dashboard, click Users on the left side.

1. Click the Add user button.

1. Enter a user name in the textbox next to User name (this example uses AWS_Admin) and select Programmatic access in the Select AWS Access Type section. Click Next: Permissions.

1. Click Attach existing policies directly, select the policy to grant, then click Next: Review.

1. Click Create user.

1. Click the Download Credentials button and save the credentials.csv file in a safe location - you need it in the next section - then click Close.

Install and Configure the AWS CLI

Now that the IAM user exists, install the AWS Command Line Interface (CLI).

1. Install the latest version of the AWS CLI:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

1. Confirm the installation:

aws --version
aws-cli/2.1.12 Python/3.7.3 Linux/3.10.0-1127.18.2.el7.tb.x86_64 exe/x86_64.centos.7 prompt/off

1. Type aws configure and press enter. Enter the following when prompted:

AWS Access Key ID [None]: the Access Key Id from the credentials.csv file downloaded above
                          (looks like AKIAPWINCOKAO3U4FWTN)
AWS Secret Access Key [None]: the Secret Access Key from the same credentials.csv file
                          (looks like 5dqQFBaGuPNf5z7NhFrgou4V5JJNaWPy1XFzBfX3)
Default region name [None]: us-east-1
Default output format [None]: json

Using the AWS CLI with Amazon S3

In this step, you use the AWS CLI to create a bucket in S3 and copy a file to the bucket.

1. Creating a bucket is optional if you already have one you want to use. To create a new bucket named my-first-backup-bucket:

aws s3 mb s3://my-first-backup-bucket

Note: bucket names must be globally unique across all AWS accounts, so the command above returns a BucketAlreadyExists error as-is. Choose your own bucket name.

1. To upload a backup file to the bucket, use the following command. Quote the path if the file name contains spaces:

aws s3 cp /lib/tb/toolpack/pkg/database_backups/my-first-backup.sql.gz s3://my-first-backup-bucket/

1. To download a file from S3 to the local directory, reverse the order of the arguments:

aws s3 cp s3://my-first-backup-bucket/my-first-backup.sql.gz ./

1. To delete a file from the bucket:

aws s3 rm s3://my-first-backup-bucket/my-first-backup.sql.gz

Automating the Upload

Add a cron job that runs after tbdbbackup and copies the newly generated files to your bucket. For a script example of automatic and scheduled uploads, see Uploading to S3 in 18 lines of Shell.