second first commit
This commit is contained in:
Zakaria
@@ -0,0 +1,156 @@
|
||||
# Kinesis
|
||||
|
||||
**Protocol:** JSON 1.1 (`X-Amz-Target: Kinesis_20131202.*`)
|
||||
**Endpoint:** `POST http://localhost:4566/`
|
||||
|
||||
## Supported Actions
|
||||
|
||||
<!-- floci:actions:start -->
|
||||
| Action | Description |
|
||||
| --- | --- |
|
||||
| `CreateStream` | Create a stream |
|
||||
| `DeleteStream` | Delete a stream |
|
||||
| `ListStreams` | List all streams |
|
||||
| `DescribeStream` | Get stream details and shard info |
|
||||
| `DescribeStreamSummary` | Lightweight stream description |
|
||||
| `RegisterStreamConsumer` | Register an enhanced fan-out consumer |
|
||||
| `DeregisterStreamConsumer` | Remove a consumer |
|
||||
| `DescribeStreamConsumer` | Get consumer details |
|
||||
| `ListStreamConsumers` | List consumers for a stream |
|
||||
| `SubscribeToShard` | Subscribe to a shard for enhanced fan-out |
|
||||
| `AddTagsToStream` | Tag a stream |
|
||||
| `RemoveTagsFromStream` | Remove tags |
|
||||
| `ListTagsForStream` | List tags |
|
||||
| `StartStreamEncryption` | Enable KMS encryption |
|
||||
| `StopStreamEncryption` | Disable encryption |
|
||||
| `SplitShard` | Split a shard into two |
|
||||
| `MergeShards` | Merge two adjacent shards |
|
||||
| `PutRecord` | Write a single record |
|
||||
| `PutRecords` | Write up to 500 records |
|
||||
| `GetShardIterator` | Get an iterator for reading |
|
||||
| `GetRecords` | Read records from a shard |
|
||||
| `ListShards` | - |
|
||||
| `IncreaseStreamRetentionPeriod` | Increase retention up to 8760 hours (365 days) |
|
||||
| `DecreaseStreamRetentionPeriod` | Decrease retention down to 24 hours |
|
||||
| `EnableEnhancedMonitoring` | - |
|
||||
| `DisableEnhancedMonitoring` | - |
|
||||
| `UpdateStreamMode` | - |
|
||||
| `UpdateMaxRecordSize` | - |
|
||||
| `UpdateShardCount` | - |
|
||||
<!-- floci:actions:end -->
|
||||
|
||||
## Local Inspection Endpoints
|
||||
|
||||
These endpoints are Floci-local read-only helpers for the UI and tests. AWS SDK
|
||||
traffic should continue to use the JSON 1.1 Kinesis API on `/`.
|
||||
|
||||
| Endpoint | Description |
|
||||
|---|---|
|
||||
| `GET /_aws/kinesis/streams` | List streams in the resolved region with shard, mode, tag, and record counts |
|
||||
| `GET /_aws/kinesis/records?StreamName=<name>` | Peek up to 100 stream records with shard attribution without consuming them |
|
||||
| `GET /_aws/kinesis/records?StreamName=<name>&Limit=<n>` | Peek up to `n` records, capped at 1000 |
|
||||
| `GET /_aws/kinesis/records?StreamName=<name>&ShardId=<id>` | Peek records for one shard |
|
||||
|
||||
## Stream Addressing
|
||||
|
||||
Most actions accept either `StreamName` or `StreamARN` to identify a stream. When both are provided, `StreamName` takes precedence. `CreateStream` only accepts `StreamName`.
|
||||
|
||||
```bash
|
||||
# By name
|
||||
aws kinesis describe-stream --stream-name events --endpoint-url $AWS_ENDPOINT_URL
|
||||
|
||||
# By ARN
|
||||
aws kinesis describe-stream --stream-arn arn:aws:kinesis:us-east-1:000000000000:stream/events --endpoint-url $AWS_ENDPOINT_URL
|
||||
```
|
||||
|
||||
## Shard Iterators
|
||||
|
||||
`GetShardIterator` supports all five iterator types: `TRIM_HORIZON`, `LATEST`, `AT_SEQUENCE_NUMBER`, `AFTER_SEQUENCE_NUMBER`, `AT_TIMESTAMP`.
|
||||
|
||||
A `LATEST` iterator is positioned at the shard tip at the moment the iterator is created, matching AWS: records written after the iterator was obtained are returned, records written before are not. This supports the standard tailing pattern: obtain a `LATEST` iterator, trigger the action that produces the record, then poll `GetRecords` following `NextShardIterator`.
|
||||
|
||||
## Record Routing
|
||||
|
||||
`PutRecord` and `PutRecords` honor `ExplicitHashKey` when it is provided. The value must be a decimal integer in the Kinesis hash-key space, and records are written to the open shard whose `HashKeyRange` contains that value. Without `ExplicitHashKey`, Floci keeps using the partition key to choose a shard.
|
||||
|
||||
## Shard Scaling (UpdateShardCount)
|
||||
|
||||
`UpdateShardCount` enforces the documented default limits: at most double the current
|
||||
open shard count per call, at most ten calls per rolling 24-hour period per stream, a
|
||||
10000-shard ceiling, and the asymmetric rule that a stream already above 10000 shards can
|
||||
only move to a target strictly below it.
|
||||
|
||||
The minimum bound (`TargetShardCount` cannot go below half the current open shard count)
|
||||
rounds up for an odd current count, so `current=5` accepts `target=3` but rejects
|
||||
`target=2`. AWS's documentation says only "below half" with no stated rounding, so which
|
||||
direction an odd count rounds is Floci's own call rather than observed AWS behavior; this
|
||||
is the more conservative reading (fewer shards allowed, not more).
|
||||
|
||||
AWS also documents a separate 10 TPS call-rate limit on this action ("Make over 10 TPS.
|
||||
TPS over 10 will trigger the LimitExceededException"), independent of the ten-calls-per-24h
|
||||
limit above. Floci does not simulate real-time request throttling for this or any other
|
||||
action, so that limit is intentionally unenforced here.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `FLOCI_SERVICES_KINESIS_ENABLED` | `true` | Enable or disable the service |
|
||||
|
||||
## Enhanced Fan-Out (EFO)
|
||||
|
||||
`SubscribeToShard` uses a snapshot-and-close model: the server returns one batch of records as a binary EventStream response and closes the connection. The SDK resubscribes automatically using the `ContinuationSequenceNumber` from the last delivered record. All five `StartingPosition` types are supported: `TRIM_HORIZON`, `LATEST`, `AT_SEQUENCE_NUMBER`, `AFTER_SEQUENCE_NUMBER`, `AT_TIMESTAMP`.
|
||||
|
||||
```bash
|
||||
export AWS_ENDPOINT_URL=http://localhost:4566
|
||||
STREAM=my-stream
|
||||
|
||||
# Register a consumer
|
||||
aws kinesis register-stream-consumer \
|
||||
--stream-arn $(aws kinesis describe-stream --stream-name $STREAM \
|
||||
--query StreamDescription.StreamARN --output text) \
|
||||
--consumer-name my-consumer
|
||||
|
||||
# Subscribe (AWS CLI streams events to stdout)
|
||||
aws kinesis subscribe-to-shard \
|
||||
--consumer-arn <consumer-arn> \
|
||||
--shard-id shardId-000000000000 \
|
||||
--starting-position Type=TRIM_HORIZON
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
export AWS_ENDPOINT_URL=http://localhost:4566
|
||||
|
||||
# Create a stream
|
||||
aws kinesis create-stream \
|
||||
--stream-name events \
|
||||
--shard-count 2 \
|
||||
--endpoint-url $AWS_ENDPOINT_URL
|
||||
|
||||
# Put a record
|
||||
aws kinesis put-record \
|
||||
--stream-name events \
|
||||
--partition-key "user-123" \
|
||||
--data '{"event":"page_view","page":"/home"}' \
|
||||
--endpoint-url $AWS_ENDPOINT_URL
|
||||
|
||||
# Get a shard iterator
|
||||
SHARD_ID=$(aws kinesis describe-stream \
|
||||
--stream-name events \
|
||||
--query 'StreamDescription.Shards[0].ShardId' --output text \
|
||||
--endpoint-url $AWS_ENDPOINT_URL)
|
||||
|
||||
ITERATOR=$(aws kinesis get-shard-iterator \
|
||||
--stream-name events \
|
||||
--shard-id $SHARD_ID \
|
||||
--shard-iterator-type TRIM_HORIZON \
|
||||
--query ShardIterator --output text \
|
||||
--endpoint-url $AWS_ENDPOINT_URL)
|
||||
|
||||
# Read records
|
||||
aws kinesis get-records \
|
||||
--shard-iterator $ITERATOR \
|
||||
--endpoint-url $AWS_ENDPOINT_URL
|
||||
```
|
||||
Reference in New Issue
Block a user