Lesson 2: Kinesis Data Streams from First Principles

What a streaming source is

A batch source hands you a finished file. A streaming source emits small records continuously, and you read them while more keep arriving. Clickstreams, IoT telemetry and application logs are typical examples.

Amazon Kinesis Data Streams is the AWS service that captures such records durably so that other applications can read them in real time. Two roles matter:

• Producers put records into the stream (for example, a web server sending log data).
• Consumers get records from the stream and process them. Level 3 covers consumers.

You know SQL, so think of a stream as an append-only table that you can only INSERT into and read in order. After you write a record, you cannot modify it or change its position.

Learn the anatomy of a record and a shard

A Kinesis data stream is a set of shards. A shard is a uniquely identified sequence of data records and provides a fixed unit of capacity.

Each data record has three parts:

• Data blob: an immutable sequence of bytes. Kinesis Data Streams does not inspect or change it. The default maximum record size is 1 MiB. You can raise it per stream up to 10 MiB for intermittent large records.
• Partition key: a Unicode string of up to 256 characters that the producer must supply.
• Sequence number: assigned by Kinesis Data Streams after the write. It is unique per partition key within its shard and generally increases over time.

Memorise the per-shard limits. Most sizing questions reduce to them:

A request that exceeds a limit fails with ProvisionedThroughputExceededException.

Retention period: records stay readable for 24 hours by default. You can increase this up to 8,760 hours (365 days) with IncreaseStreamRetentionPeriod. The minimum is 24 hours, and retention above 24 hours costs extra.

Trace how a partition key picks a shard

1. The producer sends a record with a partition key, for example device-4411.
2. Kinesis Data Streams applies an MD5 hash function, which maps the key to a 128-bit integer.
3. Each shard owns a range of hash key values. The record goes to the shard whose range contains that integer.
4. Kinesis Data Streams appends the record and returns the ShardId and SequenceNumber.

Two consequences follow:

• Ordering: all records with the same partition key map to the same shard, so they stay in order relative to each other. Ordering is per shard, not across the stream.
• Hot shards: a skewed key sends too much traffic to one shard. If 70% of records carry country=US, one shard is throttled while the others sit idle. Adding shards does not help, because a single key value cannot be spread across shards.

What to do: choose a key with many distinct, evenly used values, such as a device ID, user ID or order ID. Use a low-cardinality key only when you need strict ordering for that group and its volume fits within one shard.

Choose a capacity mode

A capacity mode determines how capacity is managed and how you are charged.

• Provisioned mode: you specify the number of shards and are charged per shard at an hourly rate. It suits predictable traffic that is easy to forecast. You scale with UpdateShardCount, or split and merge individual shards for fine-grained control.
• On-demand mode (On-demand Standard): Kinesis Data Streams manages the shards for you, and you pay per GB of data written and read. It suits unpredictable, highly variable traffic.
  • A new on-demand stream starts with 4 MB/s of write and 8 MB/s of read throughput.
  • The stream accommodates up to double the peak write throughput seen in the previous 30 days.
  • Write throttling can occur if traffic grows to more than double the previous peak within 15 minutes.
• On-demand Advantage: an account-level setting for on-demand streams. It changes the pricing structure and lets you configure warm throughput to prepare capacity before a known spike.

You can switch a stream between on-demand and provisioned modes twice within 24 hours, without disrupting producers or consumers.

On-demand mode does not remove the hot-shard problem. You still supply partition keys, and a single key that exceeds 1 MB/s or 1,000 records per second is still throttled.

Size a provisioned stream

1. Calculate write bandwidth: average record size (rounded up to the nearest 1 KB) × records per second.
2. Calculate read bandwidth: write bandwidth × number of consumers.
3. Apply the formula:

number_of_shards = ceiling(max(write_KB / 1024, read_KB / 2048))

4. Check the record count as well: records per second ÷ 1,000, rounded up. Use the larger result.

Example: 2 KB records at 3,000 records per second with 2 consumers.

• Write: 6,000 KB/s ÷ 1,024 = 5.86
• Read: 12,000 KB/s ÷ 2,048 = 5.86
• Count: 3,000 ÷ 1,000 = 3
• Result: 6 shards

Create the stream with the AWS CLI:

aws kinesis create-stream --stream-name orders --shard-count 6

Choose a producer

• AWS SDK or AWS CLI with PutRecord: writes one record per call. Use it for low volume, for the lowest latency, or when you must preserve the write order of records in a shard.
• AWS SDK or AWS CLI with PutRecords: writes up to 500 records per request. Throughput is higher, but a single failed record does not stop the others, so ordering is not guaranteed. Check FailedRecordCount and retry only the failed entries.
• Kinesis Producer Library (KPL): a library that batches with PutRecords, aggregates many small user records into larger stream records, retries automatically and sends metrics to Amazon CloudWatch. Use it for high-throughput producers. It can add a delay of up to RecordMaxBufferedTime, and consumers must de-aggregate the records.
• Kinesis Agent: a stand-alone Java application for Linux servers. It monitors a set of files and sends new data to the stream, handling file rotation, checkpointing and retries. Use it to ship log files without writing code.

aws kinesis put-record --stream-name orders \
  --partition-key customer-1042 \
  --data '{"order_id":981}' \
  --cli-binary-format raw-in-base64-out

What the exam asks about this

• Symptom to cause: one shard is throttled while the others are idle. The cause is a skewed partition key, not too few shards.
• Mode selection: spiky or unknown traffic with minimal operations points to on-demand. Steady, forecastable traffic points to provisioned.
• Shard arithmetic: apply 1 MB/s or 1,000 records per second for writes and 2 MB/s for reads.
• Producer selection: log files on a server point to Kinesis Agent, many small records at high volume point to the KPL, and strict ordering or the lowest latency points to PutRecord.

Key takeaways

• A stream is a set of shards. A record is a data blob, a partition key and a sequence number.
• The MD5 hash of the partition key selects the shard. The same key always goes to the same shard, in order.
• A shard accepts 1 MB/s or 1,000 records per second and serves 2 MB/s of reads.
• Retention is 24 hours by default and at most 8,760 hours (365 days).
• On-demand mode manages shards for you. Provisioned mode needs you to size and scale them.
• Match the producer to the scenario: PutRecord, PutRecords, KPL or Kinesis Agent.