S3 stands for Simple Storage Service, which is designed for text, data lake, web applications without costing too much. In Amazon AWS, GUI is ideal to manage this S3 bucket.

A S3 bucket contains viewing, uploading, and downloading content from it. Yet, this AWS GUI doesn’t support users to download the whole S3 bucket. You can only download individual items each time, which is time-consuming.

This tutorial will teach you how to download an entire bucket from S3.

How To Download An Entire Bucket From S3?

Method 1: Use AWS S3 Sync Command Line Interface

Using the AWS S3 command line interface is one of the easiest and most effective approaches to download an entire S3 bucket. After installing and setting up the AWS CLI, don’t forget to list all the bucket content with the aws s3 ls command.

Now run the command for S3 sync to download all files:

aws s3 sync s3://jhooq-test-bucket-1 /home/vagrant/my-destination-directo

In this example, there are two arguments that need to be passed. The first one is the S3 bucket’s name. It should be the one that you want to copy. The other parameter is the destination directory path.

Method 2: Use MinIO CLI

The open-source MinIO utility is well distributed under AGPLv3. To use this function, you need to install the suitable version for your operating system.

After successfully downloading the utility, it would be better to name it mc onto your local drive. This way, you can easily call it for use. To change the mode, let’s run the command as follows:

chmod +x mc 

-rwxrwxr-x 1 vagrant vagrant 21786624 Nov  5 10:06 mc

Now you have changed the file permission and made it executable. It’s time to copy the file to the /user/local/bin/ location:

sudo cp mc /usr/local/bin/

In the next step, an alias for S3 bucket is needed, so use the AWS Access Key and Secret Key in the Security Credentials:

 mc alias set s3 https://s3.amazonaws.com <ACCESS_KEY> <SECRET_KEY>

Added `s3` successfully.

Verify by running the mc alias ls command. Here is the code to download the entire S3 bucket:

mc cp --recursive s3/jhooq-test-bucket-1 .

...cket-1/test.txt:  146.56 KiB / 146.56 KiB

Method 3: Use s3cmd CLI

Before installing the s3cmd, you need to install Python and unzip the master.zip by running the unzip master.zip command. Now it’s time to activate the s3cmd CLI tool:

cd /home/vagrant/s3cmd/s3cmd-master/s3cmd

sudo pip install s3cmd

After installing this utility, let’s set up Access Key and Secret Key in the configure command. Here, you can download an entire S3 bucket recursively with two things: S3 bucket name and destination directory:

s3cmd get --recursive s3://jhooq-test-bucket-1 /home/vagrant/s3-bucket/

Conclusion

There are various methods to download an entire S3 bucket. Such command line tools like MinIO, AWS CLI, s3cmd, and cyberduck will surely help you accomplish the task.