How-To Transfer Messages From DynamoDB to SNS [& Back]

SNS vs DynamoDB - Understand The Difference

Amazon Simple Notification Service (SNS) and DynamoDB are very different technologies for very different use cases. Although completely different technologies, they can be seamlessly integrated to connect your notifications system into DynamoDB. One of the major benefits of Amazon Web Services is the ability to seamlessly connect different services to build a system that meets your specific needs. Connecting DynamoDB & SNS is a great example of this power.

What is Amazon SNS?

Amazon Simple Notification Service (SNS) is an extremely scalable notification service. You can post notifications to your topic, which can then fan out those notifications to subscribers, triggering event-driven use cases.

Some common use cases for SNS include:

• Standard Notifications: SNS supports sending notifications to email, SMS text messages, mobile push notifications, HTTP endpoints, and more.
• Fanout: You can use SNS to fan out messages to multiple consumers. For example, you might want to send the user an email when a payment is successful, as well as upload the customer's receipt to S3, and maybe also send a Slack message to your company's internal chat room to let everyone know a payment was successful. SNS works great for this use case.
• Scalability: Due to the scalable nature of SNS, you can easily add more messages to a topic. As your topic grows in size, AWS handles scaling notification delivery to your subscribers automatically for you without the need to provision additional resources.
• And Many More: The possibilities of what you can do with SNS are endless. This scalable notification service can be used for a wide variety of use cases.

Now let's talk about the two types of topics (Standard & FIFO) that SNS offers:

Standard Topics

Standard topics include the following unique features:

• Unlimited throughput: Standard topics support nearly unlimited throughput, allowing you to process as many messages as you want.
• At-Least-Once Delivery: Each message will be guaranteed to be delivered at least once. However, it is possible that a message will be delivered more than once. In practice, this is extremely rare, even at scale, but it is important to account for in your use case.
• Best-Effort Ordering: Sometimes messages might be delivered in a different order than they were received in. Once again, this is rare, but it is important to understand and account for.

FIFO Topics

FIFO topics include the following unique features:

• High Throughput: FIFO topics support up to 300 messages per second.
• Exactly-Once Processing: Each message is delivered once and remains available until you process and delete it. Duplicates are not possible with FIFO topics.
• First-In-First-Out Delivery: Messages are guaranteed to be delivered in the same order in which they were added to the topic.

What is Amazon DynamoDB?

Unlike SNS, DynamoDB is a fully managed NoSQL database service. Items stored in DynamoDB can be queried, directly retrieved, updated, and persisted. DynamoDB also has larger storage limits and can sometimes be more expensive than SNS.

It is important to understand that DynamoDB and SNS are not considered alternatives to one another. They are two completely different technologies for different use cases.

DynamoDB is designed for high availability and scalability, making it suitable for applications that require consistent, single-digit millisecond latency at any scale. It supports both document and key-value store models, which makes it versatile for various types of applications.

It is highly suggested that you read the guide on DynamoDB for more information.

What are Amazon DynamoDB Streams?

DynamoDB Streams are a technology built into DynamoDB that lets you subscribe to changes made to your DynamoDB table. This technology is a great way to add messages to an SNS topic when a change is made to your DynamoDB table. You can then have a subscriber on the SNS topic to further handle the DynamoDB change (e.g., send an email, SMS, push notification, etc).

DynamoDB Streams are the primary way we will work with SNS in this guide. However, we will also be using the AWS SDK directly to add/modify/delete data from our DynamoDB table based on SNS messages.

The Dynobase guide on DynamoDB Streams is a great resource for learning more about DynamoDB Streams and how to get started with them.

DynamoDB Streams & SNS - What You Need To Know

There are two directions to work with DynamoDB and SNS. Either consuming messages from SNS and processing them into DynamoDB, or sending notifications to SNS when a change is made to your DynamoDB table.

We will be covering both of these directions in this guide. Both strategies will utilize Lambda functions to add messages to the SNS topic and process items from the SNS topic.

DynamoDB Streams to SNS - Step-by-Step Guide

To start, we will use DynamoDB Streams to send messages to SNS based on changes made to your DynamoDB table.

You can then set up a subscriber on the SNS topic to run an action based on the DynamoDB change. We won't be covering this part in much detail as it is very application and use case dependent.

Step 1: Enable DynamoDB Streams on Your Table

To start, you will need to enable DynamoDB Streams on your DynamoDB table. This will allow DynamoDB to send messages to SNS when a change is made to your table.

Once you have located your DynamoDB Table in the AWS Console, navigate to the Exports and streams tab.

At the bottom of this page, there is a section called DynamoDB stream details with an Enable button. Click this to set up the DynamoDB Stream on your table.

This will bring up a page asking which view type you would like to use for the Stream.

Select one of the View types, then click Enable stream:

• Key attributes only: Only the key attributes of the changed item.
• New image: The entire item as it appears after it was changed.
• Old image: The entire item as it appears before it was changed.
• New and old images: Both the new and old images of the changed item.

For now, I will choose New image. However, this option truly depends on your use case and how you want to process changes from your DynamoDB table.

That is it for enabling DynamoDB Streams on your table. In a future step, we will connect the Lambda function to the DynamoDB Stream.

Step 2: Create SNS Topic

Next, you will need to create an SNS topic to send notifications to from our DynamoDB Stream.

Within the AWS Console, there is a wizard for creating a new SNS topic.

We won't be going over all of the options in the wizard. But the important thing to note is that you can select the type of SNS topic you want to create at the top of the page. You can read more about the types of topics in the What is Amazon SNS? section above.

Step 3: Create Lambda Connector Function (With Sample Code)

Finally, we need to create a Lambda function that will be triggered by our DynamoDB Stream and add messages to our SNS topic.

Within the Lambda function I created, I added the following JavaScript/Node.js code:

const AWS = require("aws-sdk");
const sns = new AWS.SNS();

exports.handler = async (event) => {
    await sns.publish({
        "Message": JSON.stringify(event.Records[0].dynamodb.NewImage),
        "TopicArn": "arn:aws:sns:us-west-2:123456789012:DynamoDBToSNSExample1"
    }).promise();
};
const AWS = require("aws-sdk");
const sns = new AWS.SNS();

exports.handler = async (event) => {
    await sns.publish({
        "Message": JSON.stringify(event.Records[0].dynamodb.NewImage),
        "TopicArn": "arn:aws:sns:us-west-2:123456789012:DynamoDBToSNSExample1"
    }).promise();
};

This will add a message to the SNS topic with the NewImage of the DynamoDB change. This is the same as the New image View type we selected in Step 1.

Remember, you can write additional JavaScript code to further modify the message before it is added to the SNS topic. This can be useful if you want to send an SMS notification to a user based on a change in your DynamoDB table but don't want to send the RAW object itself.

Step 4: Connect Lambda Function to DynamoDB Stream

Now that we have our Lambda function and SNS topic setup, we need to connect the Lambda function to the DynamoDB Stream.

You can do this by adding the Lambda function as a trigger to the DynamoDB Stream.

Click on the Create trigger button to set up the trigger.

Then simply click Create trigger to finish the setup.

Testing

Now let's try it out in action and see how it works.

For this simple example, I added a subscriber to email me whenever a notification is posted.

I'm going to create an item in my DynamoDB table.

Then I will check my email to see if the SNS notification was delivered.

As you can see, the item successfully made it from DynamoDB to our Lambda Connector Function to SNS, which then had a subscriber to email me whenever a notification was posted.

SNS to DynamoDB - Step-by-Step Guide

Now let's talk about how to add items to DynamoDB from your SNS topic.

Step 1: Create resources

Just like above, we need to create a DynamoDB table, SNS topic, and Lambda function.

Step 2: Write Lambda Code (With Sample Code)

If we put the following code into our Lambda function, this will take the event and put an item into our DynamoDB Table for that item.

const AWS = require("aws-sdk");
const ddb = new AWS.DynamoDB();

exports.handler = async (event) => {
    await ddb.putItem({
        "TableName": "SNSToDynamoDBExample1",
        "Item": JSON.parse(event.Records[0].Sns.Message)
    }).promise();
};
const AWS = require("aws-sdk");
const ddb = new AWS.DynamoDB();

exports.handler = async (event) => {
    await ddb.putItem({
        "TableName": "SNSToDynamoDBExample1",
        "Item": JSON.parse(event.Records[0].Sns.Message)
    }).promise();
};

Step 3: Add Lambda SNS Trigger

In our Lambda function, there is an Add trigger button that you can click to add a trigger.

Then you can fill out the add trigger page with the SNS topic settings.

Testing

Now we can test this integration.

First, I'm going to add an item to the SNS topic.

Then let's look in our DynamoDB Table to see if the item was added.

And it was!

Frequently Asked Questions

Can DynamoDB subscribe to SNS?

No, DynamoDB cannot directly subscribe to SNS topics. However, you can use a Lambda function to process SNS messages and then write them to DynamoDB.

Can DynamoDB stream to SNS?

Yes. DynamoDB Streams can be used to stream changes to an SNS topic with a Lambda connector function.

Can a DynamoDB table have multiple streams?

No, a DynamoDB table can have only one stream. However, you can add multiple triggers to a single DynamoDB Stream.

Can I transform an SNS message before storing it in DynamoDB?

Yes. Since we use a Lambda function to write from SNS to DynamoDB, you can easily write code in your Lambda function to transform the SNS message before writing it to DynamoDB.

What are the costs associated with using DynamoDB Streams and SNS?

Both DynamoDB Streams and SNS have associated costs. DynamoDB Streams charges are based on the number of read request units consumed, while SNS charges are based on the number of messages published and delivered. It's important to monitor your usage to avoid unexpected costs.