How To List All AWS S3 Objects In A Bucket Using Java

Amazon Web Services provides its Simple Storage service for uploading data to the cloud. Data in S3 is stored as objects inside buckets. This data can then be used as per the bucket policy. Accessing data through code is one means of processing this stored data in the cloud. In the following sections of this article, we will look at how we can see what objects we have inside our AWS bucket using the AWS Java SDK.

Introduction

Conveniently for us, the Spring framework in Java provides an easy way to help us interact with our cloud resources. To begin with, let’s understand the prerequisites for this guide. To begin using AWS with Java, the following steps are needed:

1. Create a bucket and store some sample objects in it.
2. Attach the read-all policy to the bucket so that its objects can be listed. (This is for demonstration purposes only. Please consider the policy based on the data specific to your use case to enhance application security.)
3. Obtain access keys from the AWS Management console.
4. Generate a new spring project from the spring starters.
5. Set up our development environment to allow our IDE to communicate with our cloud account by adding the access keys to the application.properties file.
6. Add the AWS-s3 SDK to the maven file.
7. Develop code to list all bucket objects.

Recommendations

1. Keep your bucket name relevant to improve the code’s understandability and readability.
2. It is recommended to create a separate bucket that holds the objects that you want to access through code only. This ensures that the principle of least privilege is upheld.
3. Keep your AWS access keys and secret keys private.
4. The minimum JDK version should be at least 1.8 (Java 8).
5. The correct way of using the S3 SDK is through Maven central dependency only.
6. Add dependencies for only those services that you need in your project to improve memory performance. Unless you are working on a complex application and have to use many services in code.

Now that we have seen an overview of the tutorial and the best practices, let us go through the steps one by one. By the end of the tutorial, you will successfully be able to list all S3 objects inside your AWS bucket using Java.

1. Create a bucket and add some Objects to it

To learn how to create a bucket and objects to it, please visit this link. After you have created a bucket please follow through the remaining steps. For the purpose of this tutorial, the name of my bucket is “spring-aws-sdk-demo”.

2. Attach Bucket Policy

To learn more on policies and how to create them, please visit this link. The policy I will use for my bucket in this tutorial is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::{12-digit-unique-user-id}:user/{username}"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::spring-aws-sdk-demo"
        }
    ]
}

3. Obtain Access Keys from AWS Management Console

Open the AWS management console and in the top right corner, open the drop down next to the user-name. Click on the `security credentials` option.

This will take you to the IAM Credential section. When there, scroll to the part where you can see ‘Access-Keys’. Click on the access-key option and follow the prompts to create an access key for SDK access. Please refer the following screenshot.

4. Generate new project from spring starter

The version used is Java 17 and the only dependency we need to add right now is spring web. Click on generate and open the project in your IDE. For the purpose of this tutorial, we will use IntelliJ Idea.

5. Connecting IDE with AWS-Bucket

Now, using the access keys we obtained earlier, we will connect our IDE with the cloud. To do this, head over to the application.properties file in the resources folder of the spring project that you just created and imported in your IDE. Next, add the following four properties:

• cloud.aws.credentials.access-key={paste-your-access-key-here}
• cloud.aws.credentials.secret-key={paste-your-secret-key-here}
• cloud.aws.region.static={the-region-of-cloud eg: ap-south-1}
• aws.s3.bucket.name={the-name-of-the-bucket-whose-contents-you-want-to-list}

This will allow your ide to interact and fetch resources from your cloud account. Remember to remove your secret keys and access keys, incase you want to share your code.

6. Adding S3 SDK to Your Project

Add the following dependency to your project. This dependency will pull the s3 code from maven central and make it available for you to use in your local environment.

<dependency>
            <groupId>com.amazonaws</groupoid>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.12.565</version>
</dependency>

This completes the actual setup that we need before using the AWS Java SDK for programmatic access to our cloud resources. Next, we will look at the code that will allow us to list all our s3 object names.

7. Develop Code to List Resources

We will have three main modules as follows:-

1. S3Config – To setup the S3 Client to use in service.
2. S3Service – To create a service to request list of all objects in bucket.
3. S3Controller – To receive web requests to list all objects.

Step 1: S3 Config

@Configuration
public class S3Config {
    @Value("${cloud.aws.region.static}")
    private String region; 
    @Value("${cloud.aws.credentials.access-key}")
    private String accessKey;
    @Value("${cloud.aws.credentials.secret-key}")
    private String secretKey;
    @Bean
    public AmazonS3 s3Client() {
        BasicAWSCredentials credentials=new BasicAWSCredentials(accessKey,secretKey);
        return AmazonS3ClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(credentials))
                .withRegion(region)
                .build();
      //the above statement provides your credentials, each time you try to access the resources.
    }
}

Step 2: S3 Service

@Service
@Slf4j
public class S3Service {
    @Value("${aws.s3.bucket.name}")
    private String bucketName;
    private final AmazonS3 s3Client;
    @Autowired
    public S3Service(AmazonS3 s3Client) {
        this.s3Client=s3Client;
    }
    public List<String> listObjects() {
        ObjectListing objectListing=s3Client.listObjects(bucketName);
        return objectListing.getObjectSummaries()
                .stream()
                .map(S3ObjectSummary::getKey)
                .collect(Collectors.toList());
    }
}

Step 3: S3 Controller

@RestController
@RequestMapping("/file")
public class S3Controller {
    @Autowired
    private S3Service s3Service;
    @GetMapping("/listAllObjects")
    public ResponseEntity<List<String>> viewObjects() {
        return new ResponseEntity<>(s3Service.listObjects(),HttpStatus.OK);
    }
}

After completing this much, start your application and head over to `http://localhost:8080/file/listAllObjects` in your browser. You should be able to see a list of all your AWS Bucket objects as follows:

The above example shows that I only have an image of type png called “word-cloud” in my bucket. You will likely see a different output here based on the resources you have inside your bucket.

Conclusion

We have seen how to prepare our IDE to connect with the cloud by obtaining access keys and adding them to our project. Next, we took a look at how we can list all our s3 resources using Java and AWS. Please feel free to leave comments incase you face any issues.

How to list all AWS S3 objects in a bucket using Java – FAQ’s

How to list all objects in S3 bucket Java?

You may use the AWS SDK for Java to list every object in an S3 bucket. A list of the items in the designated bucket can be obtained by using the listObjects method from the AmazonS3 client and the bucket name. To obtain information about each object, iterate through the result.

How do I get the list of all files in S3 bucket?

Use the AWS SDK in Java to get a list of every file in an S3 bucket. To retrieve information about each file, use the listObjects method from the AmazonS3 client, pass in the name of the bucket, and loop through the output.