Zakaria
46 lines
2.1 KiB
Markdown
46 lines
2.1 KiB
Markdown
# MSK Connect
|
|
|
|
**Protocol:** REST JSON
|
|
**Signing name:** `kafkaconnect`
|
|
|
|
Floci implements the MSK Connect control plane separately from MSK. Custom plugins are ZIP
|
|
archives stored in Floci S3. A connector starts a pinned stock Apache Kafka Connect worker, extracts
|
|
the referenced ZIP, and mounts only that extracted plugin directory into the worker.
|
|
|
|
## Supported Actions
|
|
|
|
| Action | Description |
|
|
|---|---|
|
|
| `CreateCustomPlugin` | Register a ZIP custom plugin from an S3 bucket ARN and key |
|
|
| `ListCustomPlugins` | List custom plugins |
|
|
| `DescribeCustomPlugin` | Describe a custom plugin |
|
|
| `DeleteCustomPlugin` | Delete a custom plugin |
|
|
| `CreateWorkerConfiguration` | Create a worker properties configuration |
|
|
| `ListWorkerConfigurations` | List worker configurations |
|
|
| `DescribeWorkerConfiguration` | Describe a worker configuration |
|
|
| `DeleteWorkerConfiguration` | Delete a worker configuration |
|
|
| `CreateConnector` | Start a connector using a custom plugin |
|
|
| `ListConnectors` | List connectors |
|
|
| `DescribeConnector` | Describe a connector |
|
|
| `UpdateConnector` | Update connector configuration |
|
|
| `DeleteConnector` | Stop and delete a connector |
|
|
|
|
## Worker behavior
|
|
|
|
Workers use the internal Redpanda address of a Floci-managed MSK cluster. The public bootstrap
|
|
address supplied to `CreateConnector` is resolved to that cluster but is never passed into the
|
|
container. This keeps the worker on Floci's Docker network while host clients continue using the
|
|
public MSK bootstrap endpoint.
|
|
|
|
No connector plugin is bundled. To run an S3 sink, upload a compatible S3 sink connector ZIP to
|
|
Floci S3, create a custom plugin for it, and use that plugin in `CreateConnector`. The worker's
|
|
container logs are streamed to CloudWatch Logs under `/aws/msk-connect/{connectorName}`.
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `FLOCI_SERVICES_KAFKACONNECT_ENABLED` | `true` | Enable or disable MSK Connect |
|
|
| `FLOCI_SERVICES_KAFKACONNECT_MOCK` | `false` | Persist control-plane metadata without starting workers |
|
|
| `FLOCI_SERVICES_KAFKACONNECT_WORKER_IMAGE` | `apache/kafka:3.9.0` | Pinned stock Kafka Connect worker image |
|