second first commit
This commit is contained in:
Zakaria
@@ -0,0 +1,172 @@
|
||||
# AWS Batch
|
||||
|
||||
**Protocol:** REST JSON
|
||||
**Endpoint:** `http://localhost:4566/v1/...`
|
||||
|
||||
Floci Batch implements the AWS Batch control plane for local integration tests. It supports queue and job-definition metadata, immediate job completion for fast contract tests, Docker-backed execution when enabled, CloudFormation resource provisioning, and EventBridge rule targets with `BatchParameters`.
|
||||
|
||||
## Supported Operations
|
||||
|
||||
| Operation | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| `CreateComputeEnvironment` | `POST /v1/createcomputeenvironment` | Store a local compute environment and return its ARN |
|
||||
| `DescribeComputeEnvironments` | `POST /v1/describecomputeenvironments` | Describe all or selected compute environments |
|
||||
| `UpdateComputeEnvironment` | `POST /v1/updatecomputeenvironment` | Update a compute environment's state, service role, and compute resources |
|
||||
| `DeleteComputeEnvironment` | `POST /v1/deletecomputeenvironment` | Delete a `DISABLED` compute environment not referenced by any job queue; deleting a missing one is a no-op |
|
||||
| `CreateJobQueue` | `POST /v1/createjobqueue` | Store a local job queue attached to compute environments |
|
||||
| `UpdateJobQueue` | `POST /v1/updatejobqueue` | Update a job queue's state, priority, and compute environment order |
|
||||
| `DeleteJobQueue` | `POST /v1/deletejobqueue` | Delete a `DISABLED` job queue; deleting a missing queue is a no-op |
|
||||
| `DescribeJobQueues` | `POST /v1/describejobqueues` | Describe all or selected job queues |
|
||||
| `RegisterJobDefinition` | `POST /v1/registerjobdefinition` | Register a revisioned `container` or `multinode` job definition |
|
||||
| `DeregisterJobDefinition` | `POST /v1/deregisterjobdefinition` | Mark a job definition revision inactive |
|
||||
| `DescribeJobDefinitions` | `POST /v1/describejobdefinitions` | List job definitions by name, ARN, revision, and status |
|
||||
| `SubmitJob` | `POST /v1/submitjob` | Submit a local Batch job |
|
||||
| `CancelJob` | `POST /v1/canceljob` | Cancel a job that has not started running |
|
||||
| `TerminateJob` | `POST /v1/terminatejob` | Terminate a nonterminal job and stop its running container or nodes |
|
||||
| `DescribeJobs` | `POST /v1/describejobs` | Describe jobs by job ID |
|
||||
| `ListJobs` | `POST /v1/listjobs` | List jobs by queue, status, AWS `filters`, and pagination |
|
||||
|
||||
## Runner Modes
|
||||
|
||||
Batch uses `floci.services.batch.runner-mode`.
|
||||
|
||||
| Value | Behavior |
|
||||
|---|---|
|
||||
| `immediate` | Default. `SubmitJob` persists the job, records lifecycle timestamps, creates one successful attempt, and returns after the job is `SUCCEEDED`. |
|
||||
| `docker` | Starts one Docker container per attempt from the job-definition image, passes resolved command and environment values, applies `MEMORY` resource requirements as Docker memory limits, captures a CloudWatch Logs stream name, and sets `SUCCEEDED` or `FAILED` from the container exit code. Timed-out jobs fail without retry, matching AWS Batch timeout behavior. |
|
||||
|
||||
`process` mode is not implemented.
|
||||
|
||||
## Submit Behavior
|
||||
|
||||
Supported `SubmitJob` fields:
|
||||
|
||||
- `jobName`
|
||||
- `jobDefinition`
|
||||
- `jobQueue`
|
||||
- `parameters`
|
||||
- `containerOverrides.command`
|
||||
- `containerOverrides.environment`
|
||||
- `arrayProperties.size` (array jobs)
|
||||
- `nodeOverrides.numNodes`, `nodeOverrides.nodePropertyOverrides` (multi-node parallel jobs)
|
||||
- `timeout.attemptDurationSeconds`
|
||||
- `retryStrategy.attempts`
|
||||
- `tags`
|
||||
|
||||
`containerOverrides.command` replaces the job-definition command. Command entries like `Ref::inputKey` are resolved from the merged parameter map before execution. Submit-time environment overrides merge over job-definition environment variables. `dependsOn` (job dependencies) is not supported.
|
||||
|
||||
Jobs move through these local statuses:
|
||||
|
||||
```text
|
||||
SUBMITTED -> PENDING -> RUNNABLE -> STARTING -> RUNNING -> SUCCEEDED|FAILED
|
||||
```
|
||||
|
||||
Driving every immediate-mode job through `PENDING` is a local simplification for tests. AWS may skip that state when there are no dependency or capacity waits.
|
||||
|
||||
## Job Control
|
||||
|
||||
`CancelJob` accepts jobs in `SUBMITTED`, `PENDING`, or `RUNNABLE` and moves them to `FAILED` with
|
||||
the requested reason. Jobs that already reached `STARTING` or `RUNNING` are left unchanged, matching
|
||||
AWS behavior. `TerminateJob` moves any nonterminal job to `FAILED`; in Docker mode it also stops and
|
||||
removes the running container. Both operations are idempotent for terminal jobs and require `jobId`
|
||||
and `reason`.
|
||||
|
||||
For an array parent, job control cascades to its children. Cancellation affects only children that
|
||||
have not started, while termination also stops running children. Terminating an MNP job stops every
|
||||
running node container. The stored terminal reason wins over a concurrent container exit, so the
|
||||
background runner cannot overwrite a canceled or terminated job with a later result.
|
||||
|
||||
### Array jobs
|
||||
|
||||
Submitting with `arrayProperties.size` (2-10,000) fans out one child job per index, sharing the
|
||||
parent's job name and running through the same execution pipeline as a normal job (each child gets
|
||||
its own Docker container in `docker` mode). The parent's `status` and `arrayProperties.statusSummary`
|
||||
are computed live from its children on every `DescribeJobs`/`ListJobs` call rather than stored and
|
||||
updated incrementally, so there is nothing to race when many children finish concurrently. A
|
||||
queue-level `ListJobs` shows one entry per array job (the parent); pass `arrayJobId` to list its
|
||||
children. `CancelJob` and `TerminateJob` on the parent cascade to the children using the job-control
|
||||
rules above.
|
||||
|
||||
### Multi-node parallel (MNP) jobs
|
||||
|
||||
A job definition registered with `"type": "multinode"` and `nodeProperties` (`numNodes`, `mainNode`,
|
||||
`nodeRangeProperties`) becomes an MNP job definition; `SubmitJob` against it (with optional
|
||||
`nodeOverrides`) runs one container per node concurrently in `docker` mode, each with
|
||||
`AWS_BATCH_JOB_NODE_INDEX`, `AWS_BATCH_JOB_MAIN_NODE_INDEX`, and `AWS_BATCH_JOB_NUM_NODES` set. The
|
||||
job's status, retry decision, and `startedAt`/`stoppedAt`/`statusReason` are all determined solely by
|
||||
the **main node** (matching AWS: a child node failing does not fail or retry the job). Every node
|
||||
normally runs to completion, while `TerminateJob` stops all running nodes. A natural main-node exit
|
||||
does not yet preempt the remaining nodes (see Limitations). `DescribeJobs`/queue-level `ListJobs`
|
||||
return one entry for the whole
|
||||
job (with `nodeProperties`, no per-node `container`); pass `multiNodeJobId` to `ListJobs` for a
|
||||
one-entry-per-node breakdown with each node's own `container` and `nodeProperties.nodeIndex`.
|
||||
`containerOverrides` and array jobs are not supported on MNP job definitions; use `nodeOverrides`
|
||||
instead.
|
||||
|
||||
## EventBridge
|
||||
|
||||
EventBridge targets whose ARN points at a Batch job queue can include:
|
||||
|
||||
```json
|
||||
{
|
||||
"BatchParameters": {
|
||||
"JobDefinition": "my-job:1",
|
||||
"JobName": "nightly-job",
|
||||
"ArrayProperties": {"Size": 2},
|
||||
"RetryStrategy": {"Attempts": 2}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When the rule fires, Floci submits an equivalent Batch job to the target queue. If the target payload contains a root `Parameters` object, those key/value pairs are stringified and passed as Batch submit parameters. Flat payload fields are not converted into Batch parameters.
|
||||
|
||||
`ArrayProperties` is accepted and returned as target metadata for local deployment compatibility, but Batch still submits one local job and does not fan out array children.
|
||||
|
||||
## CloudFormation
|
||||
|
||||
Floci provisions these resource types:
|
||||
|
||||
- `AWS::Batch::ComputeEnvironment`
|
||||
- `AWS::Batch::JobQueue`
|
||||
- `AWS::Batch::JobDefinition`
|
||||
|
||||
IAM roles, VPC fields, Fargate declarations, log configuration, storage, and resource requirements are accepted as metadata. Docker mode applies `MEMORY` requirements as container memory limits; local scheduling does not simulate AWS capacity, VCPU allocation, or VPC networking.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `FLOCI_SERVICES_BATCH_ENABLED` | `true` | Enable or disable Batch |
|
||||
| `FLOCI_SERVICES_BATCH_RUNNER_MODE` | `immediate` | `immediate` or `docker` |
|
||||
| `FLOCI_SERVICES_BATCH_DOCKER_NETWORK` | *(unset)* | Docker network for Batch containers |
|
||||
| `FLOCI_STORAGE_SERVICES_BATCH_MODE` | *(inherits global)* | Optional storage mode override |
|
||||
| `FLOCI_STORAGE_SERVICES_BATCH_FLUSH_INTERVAL_MS` | `5000` | Persistent storage flush interval |
|
||||
|
||||
## Limitations
|
||||
|
||||
- No IAM enforcement.
|
||||
- No VPC/subnet/security-group simulation.
|
||||
- No AWS-faithful capacity scheduling.
|
||||
- No job dependencies (`dependsOn`).
|
||||
- EventBridge input transformers work through the existing EventBridge target input path; full Batch-specific input-transformer parity is not implemented.
|
||||
- An MNP job does not finish the instant its main node exits. AWS stops the job (and its remaining
|
||||
child nodes) as soon as the main node exits; Floci instead waits for every node's container to
|
||||
exit before scoring the attempt, so a child that outlives the main node keeps the job `RUNNING`
|
||||
longer than AWS would. `TerminateJob` can stop the whole job, but main-node completion does not
|
||||
yet trigger that same preemption path automatically.
|
||||
|
||||
**Known follow-ups on the array/MNP surface** (not required for `SubmitJob`/`DescribeJobs`/`ListJobs`
|
||||
correctness, deliberately deferred):
|
||||
|
||||
- `submitFromEventBridge` does not forward `arrayProperties`/`nodeOverrides` from a rule target's
|
||||
`BatchParameters`, even though `ArrayProperties` is accepted and echoed back as target metadata
|
||||
(see EventBridge section above). An EventBridge-triggered Batch job cannot fan out as an array or
|
||||
run as MNP today; only a direct `SubmitJob` call can.
|
||||
- CloudFormation's `AWS::Batch::JobDefinition` provisioner only registers `container`-type job
|
||||
definitions; there is no way to provision a `multinode` job definition via CloudFormation, only
|
||||
via a direct `RegisterJobDefinition` call.
|
||||
- `ListJobs` with `arrayJobId` ignores the `filters` parameter (only `jobStatus` is honored for
|
||||
children). This matches AWS's own note that filters don't apply to child jobs, but it means array
|
||||
children can't be filtered by `JOB_NAME`/`JOB_DEFINITION`/etc.
|
||||
- `ListJobs` with `multiNodeJobId` has no pagination (`maxResults`/`nextToken`); it always returns
|
||||
every node in one response.
|
||||
Reference in New Issue
Block a user