AWS DynamoDB – Working with Streams

DynamoDB Streams is a DynamoDB feature that allows users to keep track of any changes made to the data in DynamoDB. It is an “ordered flow of data” that contains information about changes made to the data in the DynamoDB table. Let us talk of a use case. Consider a “users” table in DynamoDB and your application demands to do some pre-processing/verification of user information whenever a new user is added or user data is changed. You can enable dynamo DB streams for this data, which will allow you to access the log about all the changes made to any data in this table, in near-real-time.

Features of DyanamoDB streams

• Data is stored for up to 24 hours. Any application processing data in DynamoDB Streams must consume the data within 24 hours to avoid any data loss.
• Streams for an encrypted DynamoDB table are encrypted.

Enable streams on a DynamoDB table

Follow the below steps to enable streams on the DynamoDB table:

Step 1: Open the AWS console. Navigate to DynamoDB Dashboard -> Choose tables -> Select the existing table you want to enable DynamoDB streams for. Click on Manage DynamoDB Streams Button as shown below.

Step 2: Chose the appropriate option for your use case and click on Enable.

1. Keys only – Only the key attributes of the modified item are received in the stream.
2. New image – The entire updated item is received, as it appears after the update happened.
3. Old image – The entire updated item is received, as it appears before the update happened.
4. New and old images – Both records of the updated item are received, as it appears before and after the update happened.

This will enable DynamoDB streams which will receive logs of all the changes made to data in DynamoDB. Please note that if you perform PUT or UPDATE operations in the table and no item is modified the operation logs/data are not recorded in DynamoDB streams. 

Using Data in DynamoDB streams

DynamoDB Stream Data provides you with near real-time data consisting of all changes made to the DynamoDB Table. To read and process a stream, you need to create a custom application, which must connect to the DynamoDB Streams endpoint through API requests. Another way of connecting to DynamoDB streams is to create a DynamoDB trigger in AWS Lambda. Every time data is pushed into DynamoDB streams it invokes a Lambda Function for processing.

Adding a lambda trigger: