Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. KDS can continuously capture gigabytes of data per second from various sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. The data collected is available in milliseconds to enable real-time analytics for real-time dashboards, real-time anomaly detection, dynamic pricing, etc.

 
For Kinesis cost control, like all other cost control measures, we shall follow the Awareness, Control, and Monitor Paradigm.

Amazon Kinesis Data Streams are made up of shards. A shard represents a sequence of records in a stream and is the base throughput unit. Kinesis Data Streams (or shards) has no upfront cost, and you only pay for the resources you use.
 
Each shard:

Has an ingest rate of 1 MiB/second or 1,000 records/second

Supports up to a maximum total data read rate of 2 MiB/second

Costs $0.015/shard/hour and $0.014/1 million PUT payload units

With Kinesis, you pay for the number of shards you use and for how long you use them. So, there is a lot of math to be done before you know how many shards you need and for how long.

The first step is to optimize the cost is to become cost-aware by monitoring the existing shards with the available metrics.
AWS Cost Explorer
 
With Cost Explorer, you can view the Kinesis cost breakdowns to the level of storage vs input vs output, based on API call filters. This helps you understand the usage type of your application.

 
Basic Monitoring
 
Stream-level data is sent automatically every minute at no charge. Here is a screenshot of the basic monitoring section. These graphs show – Get/Put record(s) – sum (Bytes and Count), Get records iterator age – maximum (Milli sec), Get/Put record(s) latency – average (milli sec), Get/Put records success – average (Percent), Incoming data – sum (Bytes and Count) and Write throughput exceeded – average.


 
If volumes of your Kinesis streams are fairly predictable and steady, then these graphs are enough to estimate the correct number of shards and capacity required.
 

Enhanced (shard-level) Monitoring
 
If your application is not predictable and has extreme peaks in terms of data flow, then enable this option for a detailed individual shard level monitoring. Shard-level data is sent every minute for an additional cost. To get this level of data, you must specifically enable it for the stream using the EnableEnhancedMonitoring operation.

The following options are available for enhanced monitoring:

Here is an example of the monitoring metrics:

Once you are aware of your usage in Kinesis, then go through the following factors controlling Kinesis cost.

1. Adjust the number of shards – Remove all the obsolete shards present in any region, as the number of shards, without input and output data flow, also incurs a cost. One shard can process up to 1000 records per second – you can calculate the number of ingests per second via monitoring (incoming) and launch the exact required shard, or can make modifications accordingly.
2. Pre-process the data – We already know that input stored and output data is counted as a cost in 3 different points, so it makes sense to reduce that data. Try to pre-process the data and only ingest a required subset of data. You may use AWS Lambda to do this.
3. Do not retain data for 7 days – This directly increases your storage requirements. It is absolutely not recommended unless there is a compliance requirement. By default, there is a retention period of 24 hours.
4. Implement Kinesis with an autoscale setup – This is to avoid over-committing the resources and scale only when required. Auto-scaling Kinesis is possible, but it brings new challenges. It can’t scale instantly, and you won’t get data from new shards until you finish handling the data from old shards.
5. Avoid enhanced fan-out – This feature means that there is one important consumer that requires a dedicated 2MB/sec/shard output. A consumer that uses enhanced fan-out doesn’t have to compete with other consumers that are receiving data from the stream. This is a massive bump up in the Kinesis cost. Kinesis is not for a dedicated throughput requirement, at least not for egress requirements of 2MB/sec/shard. Enhanced fan-out shard hours cost $36.00 (USD) per day!
6. Aggregate your PUT record size – The Kinesis cost is also calculated around PUT record size and count. Irrespective of whether your PUT record size is 4KB or more, it always gets round up to 25KB. Therefore, it makes sense for the application team to adjust record size to approximately 25KB.
7. Use Terraform/Cloudformation if the use case permits – If the application use case demands limited streaming or on-demand streaming, then use Terraform/Cloudformation to create a stream on-demand.
   1. If you are aware of the time your app uses Kinesis – if it is once a day or once in a week use, you can automate the creation and pass the details with the provisioning tool.
   2. Terminate once the process is done.
8. Reduce the size of data that is being read by shard – Max is 1 MB (ingest) and output data is a max of 2MB.

Cost optimization is not a one time task, you have to keep a constant eye on the requirements and the cost explorer, to understand the exact need. You have to keep an eye on the monitoring charts, because if the data reduces, then Kinesis usage will also reduce, which can help in reducing the number of shards.

You may want to check out https://comcastsamples.github.io/KinesisShardCalculator/ which will help you get a round figure on the Kinesis shard requirements.