Create a REST API as an Amazon S3 Proxy in API Gateway Automation

Amazon Web Services is a leading cloud provider which provides us with plenty of Paas, and Iaas, and services that we can use to build and deploy our applications. we gonna build and Deploy a REST API with API Gateway which acts as a proxy to S3 and can be used to perform Read/Write on S3 without any intermediate service, here we gonna use an open-source framework Serverless for deploying our API.

API Gateway

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.

S3

Amazon Simple Storage Service (Amazon S3) is an object storage service offering scalability, data availability, security, and performance.

Serverless

Serverless is a framework that can be used to build applications on AWS, this will encapsulate things and provide us with a simple structure to create something in AWS. We can Code less and Build more with Serverless.

Install Serverless

Install serverless by running the below command

npm i -g serverless

Check the version of serverless

serverless -v

Check Serverless Version

Setup IAM Role for API Gateway to invoke AWS S3

To invoke S3 from API Gateway, it must have the required IAM permissions, for that we need to create an IAM Role and attach it to API Gateway.

Best Practice: Write IAM policies as strictly as possible to avoid any security issues.

IAM Policy Statement:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::my-bucket/*",
            "Effect": "Allow"
        }
    ]
}

The above policy has one statement `s3:PutObject`, for resource `<S3 ARN>` which allows Put Object requests on S3 Bucket.

Trusted Policy: It is a policy in which we define the principals that we allow to assume the role, so here only the API Gateway service can assume this role based on the below policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "apigateway.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Note: We’ll be creating the above role using the serverless script given below.

Create REST API and Its Resources

To create a REST API, we need to do the following:

• Create REST API in API Gateway.
• Create Resources in created REST API.
• Configure S3 Integration.

We can automate all the above things using the below script.

Serverless config

Write the below content serverless.yml

service: api-gateway-to-s3-rest-api

useDotenv: true

provider:
  name: aws
  stage: ${opt:stage, 'dev'}
  region: ap-south-1
  lambdaHashingVersion: 20201221
  logs:
    restApi: true
    level: INFO
  deploymentBucket:
    blockPublicAccess: true
    name: ${self:custom.config.CODE_DEPLOYMENT_BUCKET}
    maxPreviousDeploymentArtifacts: 3

custom:
  config:
    CODE_DEPLOYMENT_BUCKET: ${env:CODE_DEPLOYMENT_BUCKET}
    S3_BUCKET: ${env:S3_BUCKET}


resources:
  Resources:
    IAMRoleForAPIGateway:
      Type: AWS::IAM::Role
      Properties: 
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service: apigateway.amazonaws.com
              Action: sts:AssumeRole
        Description: Role for ${self:service}
        Path: /
        Policies:
          - PolicyName: ${self:service}-role-policy
            PolicyDocument: 
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action: s3:PutObject
                    Resource: arn:aws:s3:::${self:custom.config.S3_BUCKET}/*
        RoleName: ${self:service}-${self:provider.stage}-Role
    RestAPI:
      Type: AWS::ApiGateway::RestApi
      Properties: 
        Description: API to write request_body to S3
        EndpointConfiguration: 
          Types:
            - REGIONAL
        Name: ${self:service}-${self:provider.stage}
    RestAPIResource:
      Type: AWS::ApiGateway::Resource
      Properties: 
        ParentId: !GetAtt RestAPI.RootResourceId
        PathPart: api
        RestApiId: !Ref RestAPI
    RestAPIObjectPathResource:
      Type: AWS::ApiGateway::Resource
      Properties: 
        ParentId: !Ref RestAPIResource
        PathPart: '{object_path+}'
        RestApiId: !Ref RestAPI
    RestAPIPutMethod:
      Type: AWS::ApiGateway::Method
      Properties: 
        HttpMethod: PUT
        AuthorizationType: NONE
        Integration:
            Type: AWS
            IntegrationHttpMethod: PUT
            Uri: arn:aws:apigateway:${self:provider.region}
            :s3:path/${stageVariables.bucket_name}/{object_path}
            Credentials: !GetAtt IAMRoleForAPIGateway.Arn
            RequestParameters:
              "integration.request.path.object_path" : "method.request.path.object_path"
            IntegrationResponses:
              - StatusCode: 200
              - StatusCode: 400
        RequestParameters:
          "method.request.path.object_path": true
        OperationName: write-data-to-s3
        ResourceId: !Ref RestAPIObjectPathResource
        RestApiId: !Ref RestAPI
        MethodResponses:
          - StatusCode: 200
            ResponseModels:
              application/json: "Empty"
          - StatusCode: 400
            ResponseModels:
              application/json: "Error"
             
    RestAPIDeployment:
      Type: AWS::ApiGateway::Deployment
      Properties:
        RestApiId: !Ref RestAPI
      DependsOn:
        - RestAPIPutMethod

    RestAPIStage:
      Type: AWS::ApiGateway::Stage
      Properties: 
        DeploymentId: !Ref RestAPIDeployment
        Description: ${self:provider.stage} stage
        RestApiId: !Ref RestAPI
        StageName: ${self:provider.stage}
        Variables: 
          bucket_name : ${self:custom.config.S3_BUCKET}

Provider

In the provider, we configure cloud provider details like cloud provider name, region, logs config, etc.

Custom

In this section (not a default option, we can replace custom with any name) we gonna maintain any configuration variables required for this script.

 Resources

resources to create, here in this case we’re creating the REST API along with associated resources, methods, and its integration.

Here we’ll not be creating a new bucket through this script, create an s3 bucket if not exists already.

Environment variables

Here in this script, we’ve below env variables:

1. CODE_DEPLOYMENT_BUCKET: Cloudformation script generated based on our serverless script and application code (if any) will be stored in this bucket and it will then be used for deployment.
2. S3_BUCKET: S3 Bucket on which we gonna perform an API request.

Deploy Our REST API

To deploy our REST API, you can run the below command.

serverless deploy --stage alpha

API Gateway Resources

After running the above command successfully you can check API Gateway, you’ll find REST API with GET and PUT methods configured with S3 as a proxy.

Here object_path+ (suffix +) indicates that we can pass as many folders in the place of object_path /dir_1/dir_2/…./dir_n/myfile.txt, without + we can only pass one string /myfile.txt.

Test our API

We can invoke the API through any browser client to test out the API which will read/write a file in the S3.

PUT Method

In the below image I’ve invoked an API  with the PUT method, containing a request body Hello Geeks the new file will be created in our bucket with the object path dillip/file.txt in the below case.

Note: As we’ve set NONE for method authorization, no need to pass any authorization headers or data.

GET Method

In the below image, I’ve invoked an API  with the GET method, retrieving the file in our bucket with the object path dillip/file.txt we’ve created in the above PUT request.

Other Ways

There is an alternate way to do the above thing, i.e API Gateway REST API with Lambda integration, but we’ve to choose the approach based on our required outcome.

API Gateway -> AWS Lambda -> AWS S3 (Lambda as Intermediate), this approach is suitable if you wanna perform some complex logic upon objects before making a request to S3.

Conclusion

In this article, we’ve learned a simple way to build REST API which exposes GET and PUT methods to retrieve or manipulate the objects in the AWS S3, here AWS API Gateway acts as a  proxy to AWS S3 Bucket.