Open In App

AWS Lambda – Copy Object Among S3 Based on Events

Last Updated : 24 Jan, 2021
Improve
Improve
Like Article
Like
Save
Share
Report

In this article, we will make AWS Lambda function to copy files from one s3 bucket to another s3 bucket. The lambda function will get triggered upon receiving the file in the source bucket. We will make use of Amazon S3 Events. Every file when uploaded to the source bucket will be an event, this needs to trigger a Lambda function which can then process this file and copy it to the destination bucket.

Steps to configure Lambda function have been given below:

  • Select Author from scratch template. In this, we need to write the code from scratch.

  • Provide the function name.
  • Select Runtime. There are several runtimes provided by AWS such as Java, Python, NodeJS, Ruby, etc.

  • Select the execution role. Execution Roles are permissions provided to Lambda Function.

Note: Lambda must have access to the S3 source and destination buckets. Therefore, make an IAM Role that has AmazonS3FullAccess policy attached. In this case, s3tos3 has full access to s3 buckets.

Once the function is created we need to add a trigger that will invoke the lambda function. The steps to add trigger is given below.

  1. In Select Trigger, select S3. There are numerous AWS services that can act as a trigger. Since this article is focused on moving objects from one bucket to another we choose S3.
  2. In Bucket, select source bucket. This bucket will act as a trigger. We will specify the event type associated with this bucket which will further invoke our lambda function.
  3. Select Event type as All object create events. All object create event includes put, copy, post, and multi-part upload. Any one of the actions will invoke our lambda function. In our case, when we upload a file into the source bucket, the event type is PUT.
  4. Prefix and Suffix are optional. Prefix and suffix are used to match the filenames with predefined prefixes and suffixes.

The most remarkable thing about setting the Lambda S3 trigger is that whenever a file is uploaded, it will trigger our function. We make use of the event object to gather all the required information.

The sample event object is shown below. This object is passed to our Lambda function.

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "ap-south-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "GeeksforGeeks"
      },
      "requestParameters": {
        "sourceIPAddress": "XXX.X.X.X"
      },
      "responseElements": {
        "x-amz-request-id": "EXAMPLE123456789",
        "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "gfg-source-bucket",
          "ownerIdentity": {
            "principalId": "GeeksforGeeks"
          },
          "arn": "arn:aws:s3:::gfg-source-bucket"
        },
        "object": {
          "key": "geeksforgeeks.txt",
          "size": 1024,
          "eTag": "0123456789abcdef0123456789abcdef",
          "sequencer": "0A1B2C3D4E5F678901"
        }
      }
    }
  ]
}

Your Lambda function makes use of this event dictionary to identify the location where the file is uploaded. The lambda code is given below:

import json
import boto3
s3_client=boto3.client('s3')

# lambda function to copy file from 1 s3 to another s3
def lambda_handler(event, context):
    #specify source bucket
    source_bucket_name=event['Records'][0]['s3']['bucket']['name']
    #get object that has been uploaded
    file_name=event['Records'][0]['s3']['object']['key']
    #specify destination bucket
    destination_bucket_name='gfg-destination-bucket'
    #specify from where file needs to be copied
    copy_object={'Bucket':source_bucket_name,'Key':file_name}
    #write copy statement 
    s3_client.copy_object(CopySource=copy_object,Bucket=destination_bucket_name,Key=file_name)

    return {
        'statusCode': 3000,
        'body': json.dumps('File has been Successfully Copied')
    }

Note: After writing the code, don’t forget to click Deploy. 

Now when we upload a file in source bucket ‘gfg-source-bucket’, this will trigger the ‘s3Tos3-demo’ lambda function which will copy the uploaded file into destination bucket ‘gfg-destination-bucket‘. The images have been shown below:

File uploaded in Source Bucket

File Copied to Destination Bucket

Destination Bucket

The result can also be verified by clicking on Monitoring Tab in lambda function and then clicking on View logs in Cloudwatch.

CloudWatch Logs


Similar Reads

AWS Lambda - Create a Lambda Function in Python, Integrated with API Gateway
Pre-requisite: AWS We will begin with creating a lambda function in the AWS Lambda console and write our Python code for the same. We will test our function by creating the new event and verifying the response. Next, we will configure the lambda function with the JSON placeholder which is a sample API. We will be using the concept of layers in the
3 min read
AWS Lambda With Amazon S3 Events
AWS Lambda and AWS S3 are two important AWS services used by organizations. AWS Lambda is a serverless compute service that executes the code without any management of servers by the user, while AWS S3 is an object-storing service. In this guide, I have first discussed what is AWS Lambda and then I have discussed what the AWS S3 service is. After t
5 min read
AWS DynamoDB - Insert Data Using AWS Lambda
In this article, we will look into the process of inserting data into a DynamoDB table using AWS Lambda. Amazon DynamoDB is a completely owned NoSQL proprietary provider that helps key-value and textual statistics systems and is supplied via way of means of Amazon.com as a part of Amazon Web Services. AWS Lambda is an event-driven, serverless compu
3 min read
How To Deploy GraphQL API Using AWS Lambda And AWS API Gateway ?
GraphQL is known for its flexibility and efficiency. AWS Lambda, on the other hand, provides a serverless computing service that allows developers to run code without managing servers, and we pay for what we use. We can combine the power of GraphQL with the scalability and ease of AWS Lambda to serve our GraphQL application easily with minimal step
6 min read
AWS Lambda Functions With AWS CLI
Amazon Web Services (AWS) is a comprehensive cloud computing platform offering many services, including storage, computing, databases, and more. AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS). we can create functions and self-contained applications. With AWS Lambda functions, we can perform any kind of computing
6 min read
AWS Lambda Deployments with AWS CloudFormation
AWS CloudFormation allows to create AWS resources with help of templates. It provisions and deploy all the resources required for the entire application described as a Stack. Templates can be updated and replicated as required to reduce the overhead of implementing and provisioning infrastructure. let's learn how we can provision lambda deployments
6 min read
AWS OpsWorks vs AWS Beanstalk vs AWS CloudFormation
Overview :AWS lists out a wide range of services when it comes to supporting DevOps. Be it in terms of Integration, Delivery, Strategies, Infrastructure as a Code. OpsWorks falls under Config management service, CloudFormation helps to maintain Infrastructure as a code and AWS Beanstalk offers easy web deployment. Let's understand each one of them.
3 min read
copy in Python (Deep Copy and Shallow Copy)
In Python, Assignment statements do not copy objects, they create bindings between a target and an object. When we use the = operator, It only creates a new variable that shares the reference of the original object. In order to create "real copies" or "clones" of these objects, we can use the copy module in Python. Syntax of Python DeepcopySyntax:
5 min read
Blue/Green Deployment in AWS Lambda
Blue Green Deployment is just like we deploy two versions of our application, one is the stable version, and another is a new feature or bug fix let's say, forwarding a certain percentage of traffic to the second version as well in production to ensure that everything is working fine. The Blue environment represents the currently active version of
7 min read
How to Pass the Query String Parameters to AWS Lambda Function or HTTP Endpoint?
In this article, we will explore the process of passing query string parameters from a client to the Backend Lambda or HTTP endpoint API gateway. As we know there are two types of integrations for Lambda and HTTP endpoints. The first kind is Proxy integration and second kind is Non-proxy integration. While proxy implementation is the recommended ap
4 min read
Practice Tags :