DynamoDB Stream vs Kinesis Stream - The Ultimate Comparison
Written by Nishani Dissanayake
Published on August 14th, 2022
Time to 10x your DynamoDB productivity with Dynobase [learn more]
DynamoDB Stream and Amazon Kinesis Stream are two powerful technologies with similar characteristics. This article discusses these two technologies with their differences, pros, and cons to help you identify the best for your requirements.
What is DynamoDB Stream?
DynamoDB Stream is a Change Data Capture (CDC) mechanism that contains an ordered flow of information about changes made to a table in DynamoDB. It stores the information log for up to 24 hours, allowing you to view the data before and after each change.
DynamoDB Stream creates a stream record with the modified items' primary key attributes whenever an application makes a change (create, update, delete) to a table. Each stream record contains information about the data modifications done to a single item in a DynamoDB table. So, DynamoDB Stream is capable of tracking granular level changes done to a DynamoDB table item.
What is Kinesis Stream?
Amazon Kinesis is a scalable collection of services that efficiently collect, process, and analyze real-time data. Kinesis is the ideal solution when processing and streaming AWS real-time events. It is highly cost-effective and supports better decision-making in business processes by utilizing the processed kinesis data.
When considering the Kinesis services, you may come across the below four service types with different functionalities.
- Kinesis Data Streams - Kinesis Data Stream (KDS) is a real-time data streaming service capable of capturing gigabytes of data per second from thousands of sources (Example: Location tracking events, Financial transactions). This data is utilizable for real-time analytics in milliseconds. (Example: Real-time dashboards)
- Kinesis Video Streams - Kinesis Video Streams is a service that allows secure real-time upload of videos for video processing, video playback, analytics, and machine learning. (Example: Livestreaming)
- Kinesis Data Firehose - Kinesis Data Firehose is a straightforward mechanism for capturing, transforming, and loading streaming data to analytics services and data stores. Firehose can batch, compress, transform, and encrypt the data streams to enhance security and reduce the storage requirements.
- Kinesis Data Analytics - With Apache Flink ( a framework to process data streams), Kinesis Data Analytics is a simple way to transform and analyze streaming data. That also simplifies the build, management, and integration of Apache Flink with other AWS services.
DynamoDB Stream vs Kinesis Stream: Key Differences
There are several key differences between Amazon DynamoDB Stream and Kinesis Stream that will help you to select the best choice for your requirements.
- DynamoDB Stream is local to DynamoDB. It only captures granular level changes made to DynamoDB tables. But, Kinesis Stream allows consuming and producing large volumes of data from any source.
- DynamoDB and Kinesis Stream default data retention is only up to 24 hours. However, we can extend the Kinesis Stream retention period up to 7 days by paying extra.
- Unlike in Kinesis Stream, DynamoDB Stream guarantees no duplicates. You can read more about handling duplicate records in Kinesis Stream here.
- DyanmoDB Streams process stream records using AWS Lambda, whereas Kinesis Stream uses Kinesis Analytics, Kinesis Firehose, and Lambda.
- The pricing depends on different criteria in DynamoDB and Kinesis Stream. You can find more details in the next section.
One important feature that distinguishes Amazon DynamoDB is that it creates a log of the changes made, which you can use to activate other services. In addition, the near real-time records allow you to build applications to take actions consuming these streams.
In both modes for DynamoDB Stream, the first 2,500,000 read request units each month are free. After exceeding that, the charge is $0.02 for 100,000 read request units. This value can slightly change depending on the region.
Kinesis Data Stream
- Provisioned capacity mode: You have to specify the number of shards needed for your application, and the prices vary slightly based on the region. Usually, the Shard Hour (cost for one shard per hour) price is $0.015, and 1,000,000 PUT payload units will cost $0.014.
- On-demand mode: Here, the pricing is calculated based on each GB of data written and read for your data stream. The price per stream per hour is usually $0.04 (depending on the region).
Pros and Cons: DynamoDB Stream
DynamoDB has a multitude of use cases. Some examples are content moderation and data replication; you can read more about these here.
Advantages of DynamoDB Streams
- No duplicates. In a stream, each stream record appears only one time.
- Each stream record appears in the same order of modifications made to a table item.
- It comes with auto-scaling functionality to scale shards.
- The read requests are free (no payment required) when processing DynamoDB Stream events with Lambda.
Disadvantages and Limitations of DynamoDB Streams
- Only two processes can simultaneously read from the same stream shard.
- DynamoDB Streams Lambda handlers have the possibility of infinite recursion. (For instance, if you perform another modification on a table after triggering a modification on the same table inside the stream handler.)
Pros and Cons: Kinesis Stream
Advantages of Kinesis Stream
- Even though the default data retention is up to 24 hours, you can extend it up to 7 days.
- Highly scalable since Kinesis Stream has no upper limit on how many shards you can have in a stream. Beneficial in large-scale applications as you can have thousands of shards if needed.
- Allows real-time processing and real-time insights.
Disadvantages and Limitations of Kinesis Stream
- Record duplication is possible.
- One shard supports only 1000 PUT records per second.
In addition, if you want to capture the item-level changes made to your DynamoDB tables as a Kinesis Data Stream, you can use Amazon Kinesis Data Stream for Amazon DynamoDB feature.
This article explains what DynamoDB Stream and Kinesis Stream are and how they differ from each other. The key differences, advantages, disadvantages, and limitations are listed to help you determine which solution is the ideal one for your application.
I hope you find this article helpful. Thank you for reading!