Zakaria
133 lines
6.8 KiB
Markdown
133 lines
6.8 KiB
Markdown
# MSK (Managed Streaming for Kafka)
|
||
|
||
**Protocol:** REST-JSON
|
||
**Endpoint:** `http://localhost:4566/`
|
||
|
||
Floci emulates Amazon MSK by orchestrating **Redpanda** containers. This provides high compatibility with the Kafka API while maintaining a low footprint.
|
||
|
||
## Supported Actions
|
||
|
||
| Action | Description |
|
||
|---|---|
|
||
| `CreateCluster` | Spawns a new Redpanda container for the cluster |
|
||
| `CreateClusterV2` | Create a provisioned or serverless cluster |
|
||
| `ListClusters` | List all emulated clusters |
|
||
| `ListClustersV2` | List all emulated clusters using V2 API |
|
||
| `DescribeCluster` | Get cluster metadata and state |
|
||
| `DescribeClusterV2` | Get cluster metadata and state using V2 API |
|
||
| `DeleteCluster` | Stops and removes the Redpanda container |
|
||
| `GetBootstrapBrokers` | Get the connection strings for the cluster |
|
||
| `CreateConfiguration` | Create a broker configuration (`server.properties`) |
|
||
| `ListConfigurations` | List all configurations |
|
||
| `DescribeConfiguration` | Get configuration metadata and latest revision |
|
||
| `DeleteConfiguration` | Delete a configuration |
|
||
| `UpdateConfiguration` | Create a new revision of a configuration |
|
||
| `ListConfigurationRevisions` | List all revisions of a configuration |
|
||
| `DescribeConfigurationRevision` | Get a specific revision, including its `server.properties` |
|
||
| `ListTagsForResource` | List the tags on a cluster or configuration |
|
||
| `TagResource` | Add tags to a cluster or configuration |
|
||
| `UntagResource` | Remove tags from a cluster or configuration |
|
||
|
||
## Configuration
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `FLOCI_SERVICES_MSK_ENABLED` | `true` | Enable or disable the service |
|
||
| `FLOCI_SERVICES_MSK_MOCK` | `false` | `true` = metadata-only CRUD, no Docker containers |
|
||
| `FLOCI_SERVICES_MSK_DEFAULT_IMAGE` | `redpandadata/redpanda:latest` | Docker image for Redpanda (Kafka) containers |
|
||
|
||
## How it works
|
||
|
||
When `mock` is set to `false` (default), Floci uses the Docker API to start a Redpanda container for each created cluster. For Docker socket setup, private registry authentication, and other Docker settings see [Docker Configuration](../configuration/docker.md).
|
||
|
||
- **Port Mapping**: The Kafka API (9092) is mapped to a dynamic host port.
|
||
- **Persistence**: Each cluster gets a named Docker volume (`floci-msk-{volumeId}`). In memory mode the volume is removed on cluster delete; in persistent modes it is retained unless `FLOCI_STORAGE_PRUNE_VOLUMES_ON_DELETE=true`.
|
||
- **Readiness**: The cluster state transitions to `ACTIVE` once the Redpanda `/ready` endpoint is reachable.
|
||
|
||
## Cluster metadata
|
||
|
||
`CreateCluster` and `CreateClusterV2` persist the metadata you pass — broker node group
|
||
(instance type, subnets, security groups, storage, connectivity), number of broker nodes,
|
||
encryption, client authentication, enhanced monitoring, open monitoring, logging,
|
||
configuration, storage mode, rebalancing and tags —
|
||
and echo it back from `DescribeCluster`/`DescribeClusterV2` and the matching `List` calls.
|
||
|
||
The two API versions return different shapes, matching AWS:
|
||
|
||
- **v1** (`DescribeCluster`) returns a flat `ClusterInfo`: `brokerNodeGroupInfo`,
|
||
`encryptionInfo`, `clientAuthentication`, `enhancedMonitoring`, `loggingInfo`,
|
||
`openMonitoring`, `storageMode`, `rebalancing`, `numberOfBrokerNodes` and
|
||
`zookeeperConnectString` all sit directly on the cluster.
|
||
- **v2** (`DescribeClusterV2`) nests all of those under `ClusterInfo.Provisioned`, alongside a
|
||
top-level `ClusterType` of `PROVISIONED`. `ClusterArn`, `ClusterName`, `State`,
|
||
`CreationTime`, `CurrentVersion` and `Tags` stay top-level.
|
||
|
||
Two details worth knowing:
|
||
|
||
- `configurationInfo` is a *request-only* member. The configuration a cluster was created
|
||
with is reported back on `currentBrokerSoftwareInfo` as `configurationArn` and
|
||
`configurationRevision`, which is where the Terraform AWS provider reads it from.
|
||
- Members AWS defaults server-side are defaulted here too, so a `terraform plan` converges:
|
||
`enhancedMonitoring` becomes `DEFAULT`, and `encryptionInfo.encryptionInTransit` becomes
|
||
`clientBroker: TLS_PLAINTEXT` / `inCluster: true` when the request omits them.
|
||
|
||
### Serverless clusters
|
||
|
||
`CreateClusterV2` accepts a `serverless` member as well as `provisioned` — exactly one of the
|
||
two, as on AWS. A serverless cluster stores its `vpcConfigs` and `clientAuthentication`, reports
|
||
`clusterType: SERVERLESS`, and comes back from `DescribeClusterV2` under a `serverless` envelope
|
||
rather than a `provisioned` one. It is backed by the same emulated Kafka endpoint, so
|
||
`GetBootstrapBrokers` works normally.
|
||
|
||
The v1 API predates serverless and its `ClusterInfo` cannot represent one, so — as on AWS —
|
||
`DescribeCluster` on a serverless cluster returns `BadRequestException` pointing you at
|
||
`DescribeClusterV2`, and `ListClusters` omits serverless clusters entirely.
|
||
|
||
### Validation
|
||
|
||
`CreateCluster`/`CreateClusterV2` reject the values AWS rejects: a missing or over-64-character
|
||
`clusterName`, a `numberOfBrokerNodes` below 1, an EBS `volumeSize` outside 1–16384, a
|
||
`configurationInfo.revision` below 1, and any unrecognised `enhancedMonitoring`, `storageMode`,
|
||
`rebalancing.status` or `encryptionInTransit.clientBroker` value. Each 400 names the offending
|
||
member in `invalidParameter`, as MSK's `Error` schema does.
|
||
|
||
There is deliberately **no** upper bound on `numberOfBrokerNodes`. The SDK model caps it at 15,
|
||
but the REST API reference documents no maximum for it — on a page that gives ranges for its
|
||
neighbours — and the quota page allows 30 brokers per ZooKeeper cluster and 60 per KRaft
|
||
cluster, both adjustable. A 30-broker cluster is an ordinary thing to have.
|
||
|
||
Unlike AWS, the *presence* of `kafkaVersion`, `numberOfBrokerNodes` and `brokerNodeGroupInfo` is
|
||
not required — the emulator defaults them so that a minimal create request still works.
|
||
|
||
### Tags
|
||
|
||
Cluster and configuration tags are managed through `/v1/tags/{resourceArn}`
|
||
(`ListTagsForResource`, `TagResource`, `UntagResource`). Cluster tags also round-trip through
|
||
`DescribeCluster`/`DescribeClusterV2`, so a tag change is visible to a refresh either way.
|
||
Configuration tags are reachable only through the tag endpoints, because AWS's
|
||
`DescribeConfiguration` response has no tags member.
|
||
|
||
## Examples
|
||
|
||
```bash
|
||
export AWS_ENDPOINT_URL=http://localhost:4566
|
||
|
||
# Create a cluster
|
||
aws kafka create-cluster \
|
||
--cluster-name my-cluster \
|
||
--kafka-version "3.6.1" \
|
||
--number-of-broker-nodes 1 \
|
||
--broker-node-group-info '{"InstanceType":"kafka.m5.large","ClientSubnets":["subnet-1"]}' \
|
||
--endpoint-url $AWS_ENDPOINT_URL
|
||
|
||
# List clusters
|
||
aws kafka list-clusters --endpoint-url $AWS_ENDPOINT_URL
|
||
|
||
# Get bootstrap brokers
|
||
CLUSTER_ARN=$(aws kafka list-clusters --query 'ClusterInfoList[0].ClusterArn' --output text --endpoint-url $AWS_ENDPOINT_URL)
|
||
aws kafka get-bootstrap-brokers --cluster-arn $CLUSTER_ARN --endpoint-url $AWS_ENDPOINT_URL
|
||
|
||
# Delete a cluster
|
||
aws kafka delete-cluster --cluster-arn $CLUSTER_ARN --endpoint-url $AWS_ENDPOINT_URL
|
||
```
|