Compare commits
23 Commits
607ac0803b
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c9dbdbe807 | |||
| bc3d16bfb6 | |||
| 7617e1f87c | |||
| 299a7704c2 | |||
| a52602186d | |||
| 57aa866144 | |||
| ac96c6451e | |||
| 39664957fe | |||
| 1b00488b7c | |||
| dfbfa4d2a7 | |||
| 5dd9a1e346 | |||
| 00b3af3813 | |||
| d8d7dd021b | |||
| eaf5ca6ade | |||
| cc64729599 | |||
| 9f17f72cc3 | |||
| d8d94a773d | |||
| 957579c8b7 | |||
| d9a1a36dc8 | |||
| 5d4b010710 | |||
| 2486869b6b | |||
| 327173c8f6 | |||
| dcb007f8b8 |
@@ -1,21 +1,17 @@
|
||||
<p align="center">
|
||||
<img src="docs/assets/floci-black.svg#gh-light-mode-only" alt="Floci" width="500" />
|
||||
<img src="docs/assets/floci-white.svg#gh-dark-mode-only" alt="Floci" width="500" />
|
||||
<img src="docs/assets/thecloud-logo.png" alt="TheCloud" width="160" />
|
||||
</p>
|
||||
|
||||
<h1 align="center">TheCloud AWS runtime</h1>
|
||||
|
||||
<p align="center">
|
||||
<strong>Any Cloud. Locally.</strong><br />
|
||||
Light, fluffy, and always free<br />
|
||||
The local AWS emulator behind TheCloud.<br />
|
||||
No account. No auth token. No feature gates. Just <code>docker compose up</code>.
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/floci-io/floci/releases/latest"><img src="https://img.shields.io/github/v/release/floci-io/floci?label=latest%20release&color=blue" alt="Latest Release"></a>
|
||||
<a href="https://github.com/floci-io/floci/actions/workflows/release.yml"><img src="https://img.shields.io/github/actions/workflow/status/floci-io/floci/release.yml?label=build" alt="Build Status"></a>
|
||||
<a href="https://hub.docker.com/r/floci/floci"><img src="https://img.shields.io/docker/pulls/floci/floci?label=docker%20pulls" alt="Docker Pulls"></a>
|
||||
<a href="https://hub.docker.com/r/floci/floci"><img src="https://img.shields.io/docker/image-size/floci/floci/latest?label=image%20size" alt="Docker Image Size"></a>
|
||||
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT"></a>
|
||||
<a href="https://github.com/floci-io/floci/stargazers"><img src="https://img.shields.io/github/stars/floci-io/floci?style=flat" alt="GitHub Stars"></a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
@@ -24,41 +20,43 @@
|
||||
<a href="#supported-services">Services</a> ·
|
||||
<a href="#sdk-integration">SDKs</a> ·
|
||||
<a href="#testcontainers">Testcontainers</a> ·
|
||||
<a href="#migrating-from-localstack">Migration</a> ·
|
||||
<a href="https://floci.io/floci/">Docs</a>
|
||||
<a href="#migrating-from-localstack">Migration</a>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
## What is Floci?
|
||||
## What is this?
|
||||
|
||||
Floci is a free, open-source local AWS emulator for development, testing, and CI.
|
||||
TheCloud's AWS runtime is a local AWS emulator for development, testing, and CI. It is the runtime behind [TheCloud](https://gitea.zacksolutions.dev/Zakaria/floci-ui), a self-hosted cloud console.
|
||||
|
||||
It gives you AWS-shaped services on your machine without requiring a cloud account, an auth token, or paid feature gates. Point your AWS SDK, CLI, Terraform, CDK, OpenTofu, or test suite at `http://localhost:4566` and keep your existing workflows.
|
||||
It gives you AWS-shaped services on your machine without a cloud account, an auth token, or paid feature gates. Point your AWS SDK, CLI, Terraform, CDK, OpenTofu, or test suite at `http://localhost:4566` and keep your existing workflows.
|
||||
|
||||
Already using LocalStack? Floci is a drop-in replacement: swap the image and keep going. See [Migrating from LocalStack](#migrating-from-localstack).
|
||||
It is a fork of the open-source [Floci](https://github.com/floci-io/floci) emulator. Upstream fixes are brought in by cherry-picking from Floci's `main`, alongside TheCloud's own changes, for example:
|
||||
|
||||
Floci is the AWS member of the [Floci](https://github.com/floci-io) emulator family, named after [floccus](https://en.wikipedia.org/wiki/Cirrocumulus_floccus), the cloud formation that looks like popcorn.
|
||||
- Cognito hosted login with identity-provider buttons, and `PreventUserExistenceErrors` honoured on app clients.
|
||||
- CloudFront distributions served at `<id>.cloudfront.localhost`.
|
||||
- SES delivering real email through an SMTP relay that signs in with a password (Gmail app passwords work).
|
||||
- HTTP API JWT authorizers that verify tokens from this runtime's own Cognito pools.
|
||||
|
||||
| Emulator | Cloud | Port |
|
||||
|---|---|:---:|
|
||||
| **[floci](https://github.com/floci-io/floci)** | **AWS** | **4566** |
|
||||
| [floci-az](https://github.com/floci-io/floci-az) | Azure | 4577 |
|
||||
| [floci-gcp](https://github.com/floci-io/floci-gcp) | GCP | 4588 |
|
||||
| [floci-oci](https://github.com/floci-io/floci-oci) | OCI | 4599 |
|
||||
Companion runtimes for other clouds come from upstream: [floci-az](https://github.com/floci-io/floci-az) for Azure on port 4577 and [floci-gcp](https://github.com/floci-io/floci-gcp) for GCP on port 4588.
|
||||
|
||||
## Quick Start
|
||||
|
||||
The fastest way to run Floci is with the official [CLI](https://github.com/floci-io/floci-cli)
|
||||
Build and start the runtime from this repository:
|
||||
|
||||
```bash
|
||||
floci start
|
||||
git clone https://gitea.zacksolutions.dev/Zakaria/FlociClone.git
|
||||
cd FlociClone
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Export the AWS environment variables:
|
||||
|
||||
```bash
|
||||
eval $(floci env)
|
||||
export AWS_ENDPOINT_URL=http://localhost:4566
|
||||
export AWS_DEFAULT_REGION=us-east-1
|
||||
export AWS_ACCESS_KEY_ID=test
|
||||
export AWS_SECRET_ACCESS_KEY=test
|
||||
```
|
||||
|
||||
Use your existing AWS tools normally:
|
||||
@@ -75,121 +73,41 @@ aws dynamodb create-table \
|
||||
aws dynamodb list-tables
|
||||
```
|
||||
|
||||
### Watch it run
|
||||
|
||||
This short demo shows the CLI flow: start Floci, export the local AWS environment, run standard AWS CLI commands, and stop the emulator.
|
||||
|
||||
https://github.com/user-attachments/assets/b55714dc-ef36-40ae-a734-cd2cadc288a8
|
||||
|
||||
All AWS services are available at `http://localhost:4566`. Any region works. Credentials can be any non-empty values unless you explicitly enable stricter service-specific auth checks.
|
||||
|
||||
<details>
|
||||
<summary>Prefer Docker Compose?</summary>
|
||||
<summary>Prefer <code>docker run</code>, or dev mode?</summary>
|
||||
|
||||
Create a `compose.yaml` file:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
floci:
|
||||
image: floci/floci:latest
|
||||
ports:
|
||||
- "4566:4566"
|
||||
```
|
||||
|
||||
Start Floci:
|
||||
Build the image, then run it with the Docker socket so container-backed services work:
|
||||
|
||||
```bash
|
||||
docker compose up
|
||||
docker build -f docker/Dockerfile -t floci-floci:latest .
|
||||
docker run -d --name floci -p 4566:4566 \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
floci-floci:latest
|
||||
```
|
||||
|
||||
Then configure your AWS environment manually:
|
||||
Or run it in Quarkus dev mode, which needs JDK 25:
|
||||
|
||||
```bash
|
||||
export AWS_ENDPOINT_URL=http://localhost:4566
|
||||
export AWS_DEFAULT_REGION=us-east-1
|
||||
export AWS_ACCESS_KEY_ID=test
|
||||
export AWS_SECRET_ACCESS_KEY=test
|
||||
./mvnw quarkus:dev
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Using the old <code>hectorvent/floci</code> image?</summary>
|
||||
|
||||
Update your image name:
|
||||
|
||||
```yaml
|
||||
# Before
|
||||
image: hectorvent/floci:latest
|
||||
|
||||
# After
|
||||
image: floci/floci:latest
|
||||
```
|
||||
|
||||
The old `hectorvent/floci` repository no longer receives updates.
|
||||
|
||||
</details>
|
||||
|
||||
## Web Console
|
||||
|
||||
Floci ships a browser console for inspecting the resources in your local emulator.
|
||||
[TheCloud](https://gitea.zacksolutions.dev/Zakaria/floci-ui) is the console for this runtime. It runs as its own app on port 4500, talks to the runtime at `FLOCI_ENDPOINT`, and covers the services listed in its README.
|
||||
|
||||
**Open it at:** `http://localhost:4566/_floci/ui`
|
||||
|
||||
Nothing runs at boot. The first request pulls the console image, starts it as a sidecar container on Floci's Docker network, hands it Floci's own reachable address plus the standard AWS environment, polls its health endpoint, and redirects the browser once it reports it can reach Floci. The sidecar's port is bound by Docker, so it needs no `ports:` entry of your own, and its logs are streamed into CloudWatch Logs under `/floci/ui`.
|
||||
|
||||
Starting a container needs the Docker socket:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
floci:
|
||||
image: floci/floci:latest
|
||||
ports:
|
||||
- "4566:4566"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
```
|
||||
The runtime can also start a console for you as a sidecar container on first visit to `http://localhost:4566/_floci/ui`. That needs the Docker socket, and the image is set with `FLOCI_SERVICES_UI_IMAGE`; point it at your TheCloud build to use it.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `FLOCI_SERVICES_UI_ENABLED` | `true` | Enable the console sidecar |
|
||||
| `FLOCI_SERVICES_UI_IMAGE` | `floci/floci-ui:latest` | Console image to run |
|
||||
| `FLOCI_SERVICES_UI_IMAGE` | `floci/floci-ui:latest` | Console image to run, for example `floci/floci-ui:local` for a local TheCloud build |
|
||||
| `FLOCI_SERVICES_UI_CONTAINER_NAME` | `floci-ui` | Name of the sidecar container |
|
||||
| `FLOCI_SERVICES_UI_PORT` | `4500` | Host port the console is published on |
|
||||
| `FLOCI_SERVICES_UI_KEEP_RUNNING_ON_SHUTDOWN` | `false` | Leave the sidecar running when Floci stops |
|
||||
|
||||
### Running a different console
|
||||
|
||||
The console is not fixed to the one Floci ships. Any console implementing the [Floci console contract](https://floci.io/floci/ui/console-contract/) runs with nothing but an image name: listen on the port in `PORT`, serve `GET /api/health`, talk to Floci at `AWS_ENDPOINT_URL`.
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
FLOCI_SERVICES_UI_IMAGE: acme/my-console:1.0
|
||||
```
|
||||
|
||||
A console that differs from the contract's defaults says so in its own `io.floci.console.*` image labels, so its operators do not have to. [StackPort](https://github.com/DaviReisVieira/stackport), for example, listens on 8080 rather than 4500; on an image that predates the labels, say it by hand:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
FLOCI_SERVICES_UI_IMAGE: davireis/stackport:latest
|
||||
FLOCI_SERVICES_UI_CONTAINER_NAME: floci-stackport
|
||||
FLOCI_SERVICES_UI_PORT: "8080"
|
||||
FLOCI_SERVICES_UI_INTERNAL_PORT: "8080"
|
||||
```
|
||||
|
||||
[Floci Dash](https://github.com/ofsazib/floci-dash) is another. It honours `PORT`, so only the two things it does differ on need naming:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
FLOCI_SERVICES_UI_IMAGE: ghcr.io/ofsazib/floci-dash:latest
|
||||
FLOCI_SERVICES_UI_CONTAINER_NAME: floci-dash
|
||||
FLOCI_SERVICES_UI_ENDPOINT_ENV: FLOCI_URL
|
||||
FLOCI_SERVICES_UI_STATUS_PATH: /api/healthz
|
||||
```
|
||||
|
||||
The endpoint itself is never configured by hand: Floci resolves its own reachable address at start time and injects it as `AWS_ENDPOINT_URL`.
|
||||
|
||||
Full reference: [Web Console](https://floci.io/floci/ui/) and [Console Contract v1](https://floci.io/floci/ui/console-contract/).
|
||||
| `FLOCI_SERVICES_UI_KEEP_RUNNING_ON_SHUTDOWN` | `false` | Leave the sidecar running when the runtime stops |
|
||||
|
||||
## Features
|
||||
|
||||
@@ -212,14 +130,7 @@ Lambda, RDS, Neptune, ElastiCache, MSK, MSK Connect, ECS, EC2, EKS, OpenSearch,
|
||||
|
||||
Point standard AWS clients at `http://localhost:4566`. Existing credentials, regions, SDKs, CLI commands, and IaC workflows stay familiar.
|
||||
|
||||
Terraform can provision AWS-shaped resources locally through Floci using the standard HashiCorp AWS provider. See the [Terraform with Floci guide](docs/getting-started/terraform.md) for provider configuration, resource examples, and optional emulated S3 state.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Fast enough for CI</strong></summary>
|
||||
|
||||
The native image starts in milliseconds and keeps idle memory low, making it practical for local development and test pipelines.
|
||||
Terraform can provision AWS-shaped resources locally through this runtime using the standard HashiCorp AWS provider. See the [Terraform guide](docs/getting-started/terraform.md) for provider configuration, resource examples, and optional emulated S3 state.
|
||||
|
||||
</details>
|
||||
|
||||
@@ -230,36 +141,13 @@ Choose from in-memory, persistent, hybrid, and write-ahead log storage depending
|
||||
|
||||
</details>
|
||||
|
||||
## Why Floci?
|
||||
|
||||
LocalStack's community edition [sunset in March 2026](https://blog.localstack.cloud/the-road-ahead-for-localstack/), requiring auth tokens and freezing security updates. Floci is the no-strings-attached alternative.
|
||||
|
||||
| Capability | Floci | LocalStack Community |
|
||||
|---|:---:|:---:|
|
||||
| Auth token required | No | Yes |
|
||||
| Security updates | Yes | Frozen |
|
||||
| Startup time | ~24 ms | ~3.3 s |
|
||||
| Idle memory | ~13 MiB | ~143 MiB |
|
||||
| Docker image size | ~90 MB | ~1.0 GB |
|
||||
| License | MIT | Restricted |
|
||||
| API Gateway v2 / HTTP API | Yes | No |
|
||||
| Cognito | Yes | No |
|
||||
| RDS, ElastiCache, MSK, MSK Connect | Real Docker | No |
|
||||
| Neptune (graph DB + Gremlin WebSocket) | Real Docker | No |
|
||||
| DocumentDB (MongoDB-compatible) | Real Docker | No |
|
||||
| ECS, EC2, EKS | Real Docker | No |
|
||||
| CodeBuild | Real Docker execution | No |
|
||||
| Native binary | ~40 MB | No |
|
||||
|
||||
**Broad AWS coverage. Free forever.** See the [Services Overview](https://floci.io/floci/services/) for the full list of emulated services.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Client["AWS SDK / CLI"]
|
||||
|
||||
subgraph Floci ["Floci, port 4566"]
|
||||
subgraph Runtime ["TheCloud AWS runtime, port 4566"]
|
||||
Router["HTTP Router\nJAX-RS / Vert.x"]
|
||||
|
||||
subgraph Stateless ["Stateless Services"]
|
||||
@@ -288,7 +176,7 @@ flowchart LR
|
||||
|
||||
## Supported Services
|
||||
|
||||
Floci supports local emulation for application services, data services, eventing, identity, infrastructure, billing, and container-backed workloads.
|
||||
The runtime emulates application services, data services, eventing, identity, infrastructure, billing, and container-backed workloads.
|
||||
|
||||
| Category | Services |
|
||||
|---|---|
|
||||
@@ -303,7 +191,7 @@ Floci supports local emulation for application services, data services, eventing
|
||||
| Cost and billing | AWS Budgets, Pricing, Cost Explorer, Cost and Usage Reports, BCM Pricing Calculator, BCM Data Exports |
|
||||
| Resilience, backup, and config | AWS FIS, AWS Backup, AWS Config, AppConfig, AppConfigData, CloudFormation, Cloud Control API |
|
||||
|
||||
For operation-level compatibility, see the [Services Overview](https://floci.io/floci/services/).
|
||||
For operation-level detail, see upstream Floci's [Services Overview](https://floci.io/floci/services/); it can list upstream work this fork has not picked up yet.
|
||||
|
||||
<details>
|
||||
<summary>Detailed service notes</summary>
|
||||
@@ -378,7 +266,7 @@ For operation-level compatibility, see the [Services Overview](https://floci.io/
|
||||
| AWS Backup | In-process | Vaults, backup plans, selections, simulated job lifecycle, recovery points |
|
||||
| AWS FIS | In-process | All 26 management APIs for templates, experiments, target accounts, action and target discovery, safety lever, tagging, and pagination; experiment execution is a safe control-plane simulation and does not inject faults into other services |
|
||||
| AWS Config | In-process | Config rules, evaluation-driven compliance (PutEvaluations, compliance details and summaries), configuration recorders, delivery channels, retention configuration, conformance packs, tagging |
|
||||
| Organizations | In-process | Organizations, roots, nested OUs, member accounts, all policy types with `FullAWSAccess` and effective-policy inheritance, trusted service access, delegated administrators, resource policy, and the full invitation/handshake flow; created accounts are usable Floci accounts and member accounts can read the organization they belong to |
|
||||
| Organizations | In-process | Organizations, roots, nested OUs, member accounts, all policy types with `FullAWSAccess` and effective-policy inheritance, trusted service access, delegated administrators, resource policy, and the full invitation/handshake flow; created accounts are usable accounts in the runtime and member accounts can read the organization they belong to |
|
||||
| CloudTrail | In-process | Trails, event selectors (S3 data events with bucket/prefix matching), `StartLogging`/`StopLogging`, scheduled gzipped log file emission to the destination bucket at AWS-shaped key paths, IAM-deny path emits `AccessDenied` records |
|
||||
| CloudFront | In-process | Distributions, origins, cache behaviors, invalidations, tagging |
|
||||
| WAF v2 | In-process | Web ACLs, IP sets, regex pattern sets, rule groups, logging configs, resource associations, tagging (REGIONAL and CLOUDFRONT scopes) |
|
||||
@@ -389,7 +277,7 @@ For operation-level compatibility, see the [Services Overview](https://floci.io/
|
||||
| Textract | In-process stub | API-compatible stubs, dummy block data, async job simulation |
|
||||
| Transcribe | In-process stub | Transcription jobs and custom vocabularies; jobs complete immediately, no real audio processing |
|
||||
| Pricing | In-process with static snapshot | Product discovery, attributes, price list files, pagination |
|
||||
| Cost Explorer | In-process | Cost synthesized from Floci resource state and pricing snapshots |
|
||||
| Cost Explorer | In-process | Cost synthesized from the runtime's resource state and pricing snapshots |
|
||||
| Cost and Usage Reports | In-process with floci-duck sidecar | CUR 2.0 and FOCUS 1.2 columns, account-scoped storage, Parquet emission |
|
||||
| BCM Pricing Calculator | In-process | Workload estimate create, usage pricing, read, and delete lifecycle |
|
||||
| BCM Data Exports | In-process | Export lifecycle, executions, update and delete operations |
|
||||
@@ -398,7 +286,7 @@ For operation-level compatibility, see the [Services Overview](https://floci.io/
|
||||
|
||||
## Real Docker Integration
|
||||
|
||||
Floci uses real Docker containers when in-process emulation would reduce fidelity. This applies to stateful databases, connection-heavy protocols, runtimes, and build systems.
|
||||
The runtime uses real Docker containers when in-process emulation would reduce fidelity. This applies to stateful databases, connection-heavy protocols, runtimes, and build systems.
|
||||
|
||||
| Service | Default image | What is real |
|
||||
|---|---|---|
|
||||
@@ -416,12 +304,12 @@ Floci uses real Docker containers when in-process emulation would reduce fidelit
|
||||
| EC2 | AMI-mapped Linux images | Linux containers, SSH key injection, UserData, IMDS, IAM credentials |
|
||||
| ECS | User-specified task image | Container lifecycle, start, stop, health checks |
|
||||
| EKS | `rancher/k3s:latest` | Kubernetes API server via k3s |
|
||||
| MWAA | `apache/airflow:<version>-python3.12` + `postgres:16-alpine` | Real Apache Airflow (LocalExecutor: scheduler + webserver) with its own Postgres metadata DB; webserver/CLI reachable via Floci's proxy |
|
||||
| MWAA | `apache/airflow:<version>-python3.12` + `postgres:16-alpine` | Real Apache Airflow (LocalExecutor: scheduler + webserver) with its own Postgres metadata DB; webserver/CLI reachable via the runtime's proxy |
|
||||
| CodeBuild | User-specified environment image | Buildspec execution, log streaming, S3 artifact upload |
|
||||
| OpenSearch | `opensearchproject/opensearch:2` | Full OpenSearch engine with REST API |
|
||||
| ECR | `registry:2` | OCI-compatible registry for docker push and docker pull |
|
||||
| Verified Permissions | `floci/floci-sidecar-cedar:1.1.0` | Cedar 4 policy parsing, schema validation and authorization decisions, via a [Floci sidecar](https://github.com/floci-io/floci-sidecars) |
|
||||
| AppSync | `floci/floci-sidecar-graphql:0.2.0` | graphql-java schema validation, query planning and execution, via a [Floci sidecar](https://github.com/floci-io/floci-sidecars) |
|
||||
| Verified Permissions | `floci/floci-sidecar-cedar:1.1.0` | Cedar 4 policy parsing, schema validation and authorization decisions, via an upstream [Floci sidecar](https://github.com/floci-io/floci-sidecars) |
|
||||
| AppSync | `floci/floci-sidecar-graphql:0.2.0` | graphql-java schema validation, query planning and execution, via an upstream [Floci sidecar](https://github.com/floci-io/floci-sidecars) |
|
||||
|
||||
Docker-backed services require the Docker socket:
|
||||
|
||||
@@ -430,7 +318,7 @@ docker run -d --name floci \
|
||||
-p 4566:4566 \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-u root \
|
||||
floci/floci:latest
|
||||
floci-floci:latest
|
||||
```
|
||||
|
||||
### Overriding default images
|
||||
@@ -454,7 +342,7 @@ docker run -d --name floci \
|
||||
|
||||
## Persistence and Storage Modes
|
||||
|
||||
Floci can trade speed for durability depending on the workflow. Configure the default mode with `FLOCI_STORAGE_MODE`, or override storage per service.
|
||||
The runtime can trade speed for durability depending on the workflow. Configure the default mode with `FLOCI_STORAGE_MODE`, or override storage per service.
|
||||
|
||||
| Mode | Behavior | Best for | Durability |
|
||||
|---|---|---|:---:|
|
||||
@@ -465,22 +353,22 @@ Floci can trade speed for durability depending on the workflow. Configure the de
|
||||
|
||||
Use `memory` for fast test runs. Use `hybrid` when you want state preserved across container restarts without much overhead.
|
||||
|
||||
For more detail, see the [Storage Configuration documentation](https://floci.io/floci/configuration/storage/).
|
||||
For more detail, see upstream Floci's [Storage Configuration documentation](https://floci.io/floci/configuration/storage/).
|
||||
|
||||
## Multi-Account Isolation
|
||||
|
||||
Floci supports per-account resource isolation with no extra setup. If `AWS_ACCESS_KEY_ID` is exactly 12 digits, Floci uses it as the account ID. Resources created by one account are invisible to another.
|
||||
The runtime supports per-account resource isolation with no extra setup. If `AWS_ACCESS_KEY_ID` is exactly 12 digits, it is used as the account ID. Resources created by one account are invisible to another.
|
||||
|
||||
```bash
|
||||
AWS_ACCESS_KEY_ID=111111111111 aws sqs create-queue --queue-name orders
|
||||
AWS_ACCESS_KEY_ID=222222222222 aws sqs create-queue --queue-name orders
|
||||
```
|
||||
|
||||
Any other key format, such as `test` or `AKIA...`, causes Floci to fall back to `FLOCI_DEFAULT_ACCOUNT_ID`, which defaults to `000000000000`.
|
||||
Any other key format, such as `test` or `AKIA...`, falls back to `FLOCI_DEFAULT_ACCOUNT_ID`, which defaults to `000000000000`.
|
||||
|
||||
STS temporary credentials are routed too: credentials from `AssumeRole` resolve to the assumed role's account, so the cross-account assume-role-then-provision pattern works locally. Resolution precedence is 12-digit AKID → temporary-session lookup → `FLOCI_DEFAULT_ACCOUNT_ID`.
|
||||
|
||||
See the [Multi-Account Isolation docs](https://floci.io/floci/configuration/multi-account/).
|
||||
See upstream Floci's [Multi-Account Isolation docs](https://floci.io/floci/configuration/multi-account/).
|
||||
|
||||
## SDK Integration
|
||||
|
||||
@@ -526,7 +414,7 @@ client = boto3.client(
|
||||
|
||||
client.put_parameter(
|
||||
Name="/demo/app/message",
|
||||
Value="hello from floci",
|
||||
Value="hello from TheCloud",
|
||||
Type="String",
|
||||
Overwrite=True,
|
||||
)
|
||||
@@ -552,7 +440,7 @@ const client = new SQSClient({
|
||||
await client.send(
|
||||
new SendMessageCommand({
|
||||
QueueUrl: "http://localhost:4566/000000000000/demo-queue",
|
||||
MessageBody: "hello from floci",
|
||||
MessageBody: "hello from TheCloud",
|
||||
}),
|
||||
);
|
||||
```
|
||||
@@ -613,7 +501,7 @@ use aws_sdk_secretsmanager::Client;
|
||||
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
|
||||
.region(Region::new("us-east-1"))
|
||||
.credentials_provider(Credentials::new("test", "test", None, None, "floci"))
|
||||
.credentials_provider(Credentials::new("test", "test", None, None, "thecloud"))
|
||||
.endpoint_url("http://localhost:4566")
|
||||
.load()
|
||||
.await;
|
||||
@@ -623,7 +511,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
client
|
||||
.create_secret()
|
||||
.name("demo/secret")
|
||||
.secret_string("hello from floci")
|
||||
.secret_string("hello from TheCloud")
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
@@ -649,7 +537,7 @@ aws --endpoint-url http://localhost:4566 s3 ls
|
||||
|
||||
## Testcontainers
|
||||
|
||||
Floci has Testcontainers modules for starting isolated Floci instances directly from tests. This avoids shared state, manual daemon setup, and port conflicts.
|
||||
Upstream Floci publishes Testcontainers modules for starting isolated instances directly from tests. This avoids shared state, manual daemon setup, and port conflicts. They start upstream's `floci/floci` image by default, so they test against upstream Floci rather than this fork unless you give them your own image.
|
||||
|
||||
For Testcontainers 1.x, use the versions as indicated in the table below.
|
||||
|
||||
@@ -767,7 +655,7 @@ def test_s3_create_bucket():
|
||||
|
||||
## Compatibility Testing
|
||||
|
||||
The [`compatibility-tests`](./compatibility-tests/) directory validates Floci across SDKs and tooling workflows.
|
||||
The [`compatibility-tests`](./compatibility-tests/) directory validates the runtime across SDKs and tooling workflows.
|
||||
|
||||
| Module | Language / Tool | SDK / Client | Tests |
|
||||
|---|---|---|---:|
|
||||
@@ -784,22 +672,19 @@ The [`compatibility-tests`](./compatibility-tests/) directory validates Floci ac
|
||||
|
||||
## Migrating from LocalStack
|
||||
|
||||
Floci is a drop-in replacement for LocalStack Community. The port, credentials, SDK configuration, and CLI endpoint pattern work the same way. Swap the image and keep going.
|
||||
The runtime is a drop-in replacement for LocalStack Community. The port, credentials, SDK configuration, and CLI endpoint pattern work the same way. Swap the image and keep going.
|
||||
|
||||
```yaml
|
||||
# Before
|
||||
image: localstack/localstack
|
||||
|
||||
# After, standard image
|
||||
image: floci/floci:latest
|
||||
|
||||
# After, if init scripts need AWS CLI or boto3
|
||||
image: floci/floci:latest-compat
|
||||
# After, your local build of this repository
|
||||
image: floci-floci:latest
|
||||
```
|
||||
|
||||
LocalStack environment variables are translated automatically:
|
||||
|
||||
| LocalStack | Floci equivalent |
|
||||
| LocalStack | Equivalent |
|
||||
|---|---|
|
||||
| `LOCALSTACK_HOST` | `FLOCI_HOSTNAME` |
|
||||
| `PERSISTENCE=1` | `FLOCI_STORAGE_MODE=persistent` |
|
||||
@@ -809,43 +694,23 @@ LocalStack environment variables are translated automatically:
|
||||
|
||||
Init scripts mounted under `/etc/localstack/init/` run unchanged. The `/_localstack/init` and `/_localstack/health` endpoints are still served. Once the emulator is up, the log also ends with a LocalStack-style `Ready.` line, so tooling that watches the log for it, such as the default wait strategy of Testcontainers' `LocalStackContainer`, works unchanged. Set `LOCALSTACK_PARITY=false` to opt out of automatic translation.
|
||||
|
||||
See the [full migration guide](https://floci.io/floci/getting-started/migrate-from-localstack/).
|
||||
See upstream Floci's [full migration guide](https://floci.io/floci/getting-started/migrate-from-localstack/).
|
||||
|
||||
## Image Tags
|
||||
## Building and Updating
|
||||
|
||||
Every tag combines a variant and a channel.
|
||||
The image is built from this repository; nothing is published to Docker Hub:
|
||||
|
||||
| Channel | Standard | Baseline (ARM64 only) | Compat with AWS CLI and boto3 |
|
||||
|---|---|---|---|
|
||||
| Release, floating | `latest` | `latest-baseline` | `latest-compat` |
|
||||
| Release, pinned | `x.y.z` | `x.y.z-baseline` | `x.y.z-compat` |
|
||||
| Nightly, floating | `nightly` | — | `nightly-compat` |
|
||||
| Nightly, dated | `nightly-mmddyyyy` | — | `nightly-mmddyyyy-compat` |
|
||||
|
||||
Use `latest` for stable releases, a pinned version for reproducible builds, and `nightly` to track `main`.
|
||||
|
||||
```yaml
|
||||
# Recommended
|
||||
image: floci/floci:latest
|
||||
|
||||
# Includes AWS CLI and boto3
|
||||
image: floci/floci:latest-compat
|
||||
|
||||
# ARM64 baseline for Raspberry Pi 4 / pre-LSE cores
|
||||
image: floci/floci:latest-baseline
|
||||
|
||||
# Pinned release
|
||||
image: floci/floci:x.y.z
|
||||
|
||||
# Track main
|
||||
image: floci/floci:nightly
|
||||
```bash
|
||||
docker build -f docker/Dockerfile -t floci-floci:latest .
|
||||
```
|
||||
|
||||
### Release train
|
||||
Tests run with Maven and JDK 25:
|
||||
|
||||
Stable releases ship on the **1st and 3rd Tuesday of each month**. Between trains, `floci/floci:nightly` tracks `main`. Every merged fix is available the next day, and dated `nightly-mmddyyyy` tags let you pin a specific night's build.
|
||||
```bash
|
||||
./mvnw test
|
||||
```
|
||||
|
||||
Versions are derived from Conventional Commits by [semantic-release](https://github.com/semantic-release/semantic-release); `CHANGELOG.md` is generated, never hand-edited. Releases are cut from `main` only: there are no maintenance branches.
|
||||
Upstream fixes come in by cherry-picking commits from Floci's `main` onto a branch, which then goes through review like any other change.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -853,26 +718,26 @@ All settings are overridable through environment variables with the `FLOCI_` pre
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `FLOCI_PORT` | `4566` | Port exposed by the Floci API |
|
||||
| `FLOCI_PORT` | `4566` | Port the runtime's API listens on |
|
||||
| `FLOCI_DEFAULT_REGION` | `us-east-1` | Default AWS region |
|
||||
| `FLOCI_DEFAULT_ACCOUNT_ID` | `000000000000` | Default AWS account ID |
|
||||
| `FLOCI_BASE_URL` | `http://localhost:4566` | Base URL used when Floci returns service URLs |
|
||||
| `FLOCI_HOSTNAME` | Unset | Hostname used in returned URLs when Floci runs inside Docker Compose |
|
||||
| `FLOCI_BASE_URL` | `http://localhost:4566` | Base URL used in service URLs the runtime returns |
|
||||
| `FLOCI_HOSTNAME` | Unset | Hostname used in returned URLs when the runtime runs inside Docker Compose |
|
||||
| `FLOCI_STORAGE_MODE` | `memory` | Storage mode: `memory`, `persistent`, `hybrid`, or `wal` |
|
||||
| `FLOCI_STORAGE_PERSISTENT_PATH` | `./data` | Directory used for persisted state |
|
||||
| `FLOCI_SERVICES_LAMBDA_ECR_BASE_URI` | `public.ecr.aws` | ECR base URI used when pulling Lambda runtime images (legacy name `FLOCI_ECR_BASE_URI` still works) |
|
||||
| `FLOCI_SERVICES_S3_ENFORCE_AUTH` | `false` | Enforce S3 public/private read access and reject unknown signed S3 access keys |
|
||||
|
||||
Full reference: [configuration docs](https://floci.io/floci/configuration/advanced/application-yml)
|
||||
Full reference: [`src/main/resources/application.yml`](src/main/resources/application.yml), and upstream Floci's [configuration docs](https://floci.io/floci/configuration/advanced/application-yml).
|
||||
|
||||
### Multi-container Docker Compose
|
||||
|
||||
When your application runs in a different container, set `FLOCI_HOSTNAME` to the Floci service name so returned URLs, such as SQS `QueueUrl` values, resolve correctly.
|
||||
When your application runs in a different container, set `FLOCI_HOSTNAME` to the runtime's service name so returned URLs, such as SQS `QueueUrl` values, resolve correctly.
|
||||
|
||||
```yaml
|
||||
services:
|
||||
floci:
|
||||
image: floci/floci:latest
|
||||
image: floci-floci:latest
|
||||
ports:
|
||||
- "4566:4566"
|
||||
environment:
|
||||
@@ -887,59 +752,6 @@ services:
|
||||
|
||||
Without this, services may return URLs using `localhost`, which points to the wrong container from the application container.
|
||||
|
||||
## Community
|
||||
|
||||
Join the Floci community on [Slack](https://join.slack.com/t/floci/shared_invite/zt-3tjn02s3q-A00kEjJ1cZxsg_imTfy6Cw) or [GitHub Discussions](https://github.com/orgs/floci-io/discussions). Feature ideas, compatibility questions, design tradeoffs, and rough proposals are welcome.
|
||||
|
||||
## Sponsors
|
||||
|
||||
Floci is independent open source, funded by the people and companies who use it.
|
||||
Sponsorship buys gratitude and nothing else: every emulated service is free for
|
||||
everyone, forever, and no sponsor gets features, priority, or roadmap influence
|
||||
that the rest of the Flock does not.
|
||||
|
||||
### 🥇 Gold
|
||||
|
||||
Large logo with top placement in the emulator READMEs and on floci.io, plus a
|
||||
mention in release notes.
|
||||
|
||||
[IceGuard](https://github.com/iceguard) · [Softmax](https://softmax.com/)
|
||||
|
||||
### 🥈 Silver
|
||||
|
||||
Logo in the emulator READMEs and on floci.io, plus a mention in release notes.
|
||||
|
||||
*Your logo here. [Become a sponsor](https://github.com/sponsors/floci-io).*
|
||||
|
||||
### 🥉 Community
|
||||
|
||||
Name in the emulator READMEs, a sponsor badge on GitHub, and our sincere thanks.
|
||||
|
||||
[AutoScout24](https://www.autoscout24.com) · [Nexxion AI](https://nexxion.ai/)
|
||||
|
||||
Every sponsor, including the Friends of the Flock who support Floci outside these
|
||||
tiers, is listed in [THANKS.md](https://github.com/floci-io/.github/blob/main/THANKS.md).
|
||||
|
||||
**[Sponsor Floci](https://github.com/sponsors/floci-io)**
|
||||
|
||||
## Star History
|
||||
|
||||
<p align="center">
|
||||
<a href="https://www.star-history.com/?repos=floci-io%2Ffloci&type=date&legend=top-left">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=floci-io/floci&type=date&theme=dark&legend=top-left" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=floci-io/floci&type=date&legend=top-left" />
|
||||
<img width="600" alt="Star History Chart" src="https://api.star-history.com/chart?repos=floci-io/floci&type=date&legend=top-left" />
|
||||
</picture>
|
||||
</a>
|
||||
</p>
|
||||
|
||||
## Contributors
|
||||
|
||||
<a href="https://github.com/floci-io/floci/graphs/contributors">
|
||||
<img src="https://contrib.rocks/image?repo=floci-io/floci&max=100&columns=20" />
|
||||
</a>
|
||||
|
||||
## License
|
||||
|
||||
MIT. Use it however you want.
|
||||
MIT. TheCloud's AWS runtime is a fork of [Floci](https://github.com/floci-io/floci); thanks to the Floci contributors and sponsors whose work it builds on.
|
||||
|
||||
@@ -12,6 +12,11 @@ services:
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ./data:/app/data
|
||||
# Optional SES SMTP relay settings (FLOCI_SERVICES_SES_SMTP_*), kept outside the repo
|
||||
# because they hold a password. Without the file, sent emails stay in the SES mailbox.
|
||||
env_file:
|
||||
- path: ${HOME}/.config/thecloud/ses-smtp.env
|
||||
required: false
|
||||
environment:
|
||||
FLOCI_SERVICES_DOCKER_NETWORK: floci_default
|
||||
FLOCI_SERVICES_RDS_PROXY_BASE_PORT: "7001"
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 91 KiB |
@@ -1040,6 +1040,18 @@ public interface EmulatorConfig {
|
||||
interface EfsServiceConfig {
|
||||
@WithDefault("true")
|
||||
boolean enabled();
|
||||
|
||||
/**
|
||||
* Serve every mount target over NFSv4.1 at its own address on the VPC's Docker network,
|
||||
* from a privileged helper container running the kernel NFS server, so an EC2 instance
|
||||
* can mount the file system as on AWS. Needs VPC networks ({@code floci.services.ec2.vpc-networks}).
|
||||
*/
|
||||
@WithDefault("true")
|
||||
boolean nfsServers();
|
||||
|
||||
/** Image of the NFS server; the default is built locally from Floci's own recipe on first use. */
|
||||
@WithDefault("floci/efs-nfs-server:local")
|
||||
String nfsServerImage();
|
||||
}
|
||||
|
||||
interface LightsailServiceConfig {
|
||||
|
||||
@@ -68,6 +68,14 @@ public class EmbeddedDnsServer {
|
||||
// localhost.floci.io → localhost.floci.io, *.localhost.floci.io
|
||||
public static final List<String> BUILTIN_SUFFIXES = List.of(DEFAULT_SUFFIX, LOCALSTACK_SUFFIX);
|
||||
|
||||
/** Answers A queries for names a service owns, such as an EFS file system's DNS name. */
|
||||
@FunctionalInterface
|
||||
public interface NameResolver {
|
||||
Optional<String> resolve(String name);
|
||||
}
|
||||
|
||||
private final List<NameResolver> resolvers = new java.util.concurrent.CopyOnWriteArrayList<>();
|
||||
|
||||
private volatile String serverIp;
|
||||
private final SequencedSet<String> suffixes = new LinkedHashSet<>();
|
||||
private volatile List<String> upstreamDnsServers = List.of();
|
||||
@@ -163,11 +171,30 @@ public class EmbeddedDnsServer {
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Adds names a service answers for; a name no resolver knows is forwarded upstream as before. */
|
||||
public void addResolver(NameResolver resolver) {
|
||||
resolvers.add(resolver);
|
||||
}
|
||||
|
||||
Optional<String> resolveARecord(String name, String myIp) {
|
||||
if (matchesSuffix(name)) {
|
||||
return Optional.of(myIp);
|
||||
}
|
||||
return resolveEc2PrivateDnsName(name);
|
||||
Optional<String> ec2Name = resolveEc2PrivateDnsName(name);
|
||||
if (ec2Name.isPresent()) {
|
||||
return ec2Name;
|
||||
}
|
||||
for (NameResolver resolver : resolvers) {
|
||||
try {
|
||||
Optional<String> address = resolver.resolve(name);
|
||||
if (address.isPresent()) {
|
||||
return address;
|
||||
}
|
||||
} catch (RuntimeException e) {
|
||||
LOG.debugv("DNS resolver failed for {0}: {1}", name, e.getMessage());
|
||||
}
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
Optional<String> resolveEc2PrivateDnsName(String name) {
|
||||
|
||||
@@ -362,6 +362,20 @@ public class ContainerBuilder {
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds a named volume mount of one directory inside the volume, given relative to its root.
|
||||
* The directory must already exist; Docker refuses to start the container otherwise.
|
||||
*/
|
||||
public Builder withNamedVolumeSubpath(String volumeName, String subpath, String containerPath, boolean readOnly) {
|
||||
this.mounts.add(new Mount()
|
||||
.withType(MountType.VOLUME)
|
||||
.withSource(volumeName)
|
||||
.withTarget(containerPath)
|
||||
.withReadOnly(readOnly)
|
||||
.withVolumeOptions(new SubpathVolumeOptions(subpath)));
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Inherits every volume declared by another container.
|
||||
*/
|
||||
|
||||
+74
-9
@@ -34,6 +34,7 @@ import java.io.ByteArrayOutputStream;
|
||||
import java.io.Closeable;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
@@ -588,6 +589,77 @@ public class ContainerLifecycleManager {
|
||||
rootPermissions.ifPresent(p -> script.append("chmod ").append(p).append(' ').append(mount).append(" && "));
|
||||
script.append("true");
|
||||
|
||||
Integer status = runVolumeHelper(volumeName, mount, initImage, List.of("sh", "-c", script.toString()));
|
||||
if (status == null || status != 0) {
|
||||
// Throw so the caller leaves the volume unmemoised and retries on the next launch,
|
||||
// rather than leaving it root:root 0755 with no further attempt.
|
||||
throw new IllegalStateException("shared-volume init for " + volumeName
|
||||
+ " exited with status " + status + " (cmd: " + script + ")");
|
||||
}
|
||||
LOG.infov("Initialised shared volume {0} root (cmd: {1})", volumeName, script);
|
||||
}
|
||||
|
||||
/** Owner and permissions for a directory created inside a shared volume (EFS {@code CreationInfo}). */
|
||||
public record DirectoryCreation(long ownerUid, long ownerGid, String permissions) {
|
||||
public DirectoryCreation {
|
||||
if (permissions == null || !permissions.matches("^[0-7]{3,4}$")) {
|
||||
throw new IllegalArgumentException("permissions must be 3-4 octal digits: " + permissions);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Walks the path one directory at a time so that every directory it creates, not only the
|
||||
// last, gets the owner and permissions, as EFS does for an access point's root directory.
|
||||
private static final String ENSURE_DIRECTORY_SCRIPT = """
|
||||
set -f
|
||||
cd /floci-shared-volume || exit 2
|
||||
[ -d "$1" ] && exit 0
|
||||
[ "$#" -eq 4 ] || exit 3
|
||||
p=
|
||||
IFS=/
|
||||
for part in $1; do
|
||||
[ -n "$part" ] || continue
|
||||
p="${p:+$p/}$part"
|
||||
if [ ! -d "$p" ]; then
|
||||
mkdir "$p" && chown "$2:$3" "$p" && chmod "$4" "$p" || exit 1
|
||||
fi
|
||||
done
|
||||
""";
|
||||
|
||||
/**
|
||||
* Ensures {@code relativePath} exists inside the named volume, emulating how Amazon EFS creates
|
||||
* an access point's root directory on first use: each missing directory on the path is created
|
||||
* with {@code creation}'s owner and permissions. When the directory is missing and
|
||||
* {@code creation} is empty nothing is created and this returns false, since EFS then refuses
|
||||
* the mount. The volume itself must already exist.
|
||||
*/
|
||||
public boolean ensureSharedVolumeDirectory(String volumeName, String relativePath,
|
||||
Optional<DirectoryCreation> creation, String initImage) {
|
||||
if (relativePath == null || relativePath.isBlank() || relativePath.startsWith("/")
|
||||
|| List.of(relativePath.split("/")).contains("..")) {
|
||||
throw new IllegalArgumentException("directory must be relative to the volume root: " + relativePath);
|
||||
}
|
||||
String memoKey = volumeName + "/" + relativePath;
|
||||
if (initializedSharedVolumes.containsKey(memoKey)) {
|
||||
return true;
|
||||
}
|
||||
List<String> cmd = new ArrayList<>(List.of("sh", "-c", ENSURE_DIRECTORY_SCRIPT, "sh", relativePath));
|
||||
creation.ifPresent(c -> cmd.addAll(List.of(
|
||||
Long.toString(c.ownerUid()), Long.toString(c.ownerGid()), c.permissions())));
|
||||
Integer status = runVolumeHelper(volumeName, "/floci-shared-volume", initImage, cmd);
|
||||
if (status != null && status == 3) {
|
||||
return false;
|
||||
}
|
||||
if (status == null || status != 0) {
|
||||
throw new IllegalStateException("creating " + relativePath + " in volume " + volumeName
|
||||
+ " exited with status " + status);
|
||||
}
|
||||
initializedSharedVolumes.put(memoKey, Boolean.TRUE);
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Runs a one-off helper container with the volume mounted at {@code mount}; returns its exit status. */
|
||||
private Integer runVolumeHelper(String volumeName, String mount, String initImage, List<String> cmd) {
|
||||
String image = (initImage != null && !initImage.isBlank()) ? initImage : "busybox:stable";
|
||||
imageCacheService.ensureImageExists(image);
|
||||
|
||||
@@ -595,22 +667,15 @@ public class ContainerLifecycleManager {
|
||||
new Mount().withType(MountType.VOLUME).withSource(volumeName).withTarget(mount)));
|
||||
CreateContainerResponse created = dockerClient.createContainerCmd(image)
|
||||
.withHostConfig(hostConfig)
|
||||
.withCmd("sh", "-c", script.toString())
|
||||
.withCmd(cmd.toArray(String[]::new))
|
||||
.withLabels(ContainerStorageHelper.defaultLabels(config))
|
||||
.exec();
|
||||
String helperId = created.getId();
|
||||
try {
|
||||
startContainer(helperId);
|
||||
Integer status = dockerClient.waitContainerCmd(helperId)
|
||||
return dockerClient.waitContainerCmd(helperId)
|
||||
.exec(new WaitContainerResultCallback())
|
||||
.awaitStatusCode(60, TimeUnit.SECONDS);
|
||||
if (status == null || status != 0) {
|
||||
// Throw so the caller leaves the volume unmemoised and retries on the next launch,
|
||||
// rather than leaving it root:root 0755 with no further attempt.
|
||||
throw new IllegalStateException("shared-volume init for " + volumeName
|
||||
+ " exited with status " + status + " (cmd: " + script + ")");
|
||||
}
|
||||
LOG.infov("Initialised shared volume {0} root (cmd: {1})", volumeName, script);
|
||||
} finally {
|
||||
try {
|
||||
dockerClient.removeContainerCmd(helperId).withForce(true).exec();
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
package io.github.hectorvent.floci.core.common.docker;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.github.dockerjava.api.model.VolumeOptions;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
|
||||
/**
|
||||
* {@link VolumeOptions} with the {@code Subpath} field (Docker Engine API 1.45+), which mounts one
|
||||
* directory of a named volume instead of its root. docker-java 3.7 does not model the field yet.
|
||||
*/
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class SubpathVolumeOptions extends VolumeOptions {
|
||||
|
||||
@JsonProperty("Subpath")
|
||||
private final String subpath;
|
||||
|
||||
public SubpathVolumeOptions(String subpath) {
|
||||
this.subpath = subpath;
|
||||
}
|
||||
|
||||
public String getSubpath() {
|
||||
return subpath;
|
||||
}
|
||||
}
|
||||
@@ -563,8 +563,10 @@ public class AcmJsonHandler {
|
||||
|
||||
private CertificateOptions parseOptions(JsonNode optionsNode) {
|
||||
if (optionsNode.isMissingNode()) return null;
|
||||
String ctPref = optionsNode.path("CertificateTransparencyLoggingPreference").asText("ENABLED");
|
||||
return new CertificateOptions(ctPref, "DISABLED");
|
||||
String ctPref = parseOptionValue(optionsNode, "CertificateTransparencyLoggingPreference");
|
||||
String export = parseOptionValue(optionsNode, "Export");
|
||||
// An exportable public certificate is requested with Export ENABLED, and keeps it for life.
|
||||
return new CertificateOptions(ctPref != null ? ctPref : "ENABLED", export != null ? export : "DISABLED");
|
||||
}
|
||||
|
||||
private List<CertificateStatus> parseCertificateStatuses(JsonNode node) {
|
||||
|
||||
+26
@@ -4,8 +4,10 @@ import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import io.github.hectorvent.floci.config.EmulatorConfig;
|
||||
import io.github.hectorvent.floci.core.common.SsrfProtection;
|
||||
import io.github.hectorvent.floci.services.cognito.CognitoService;
|
||||
import jakarta.annotation.PreDestroy;
|
||||
import jakarta.enterprise.context.ApplicationScoped;
|
||||
import jakarta.enterprise.inject.Instance;
|
||||
import jakarta.inject.Inject;
|
||||
import org.apache.hc.client5.http.DnsResolver;
|
||||
import org.apache.hc.client5.http.SystemDefaultDnsResolver;
|
||||
@@ -37,6 +39,7 @@ import java.util.Base64;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.function.Function;
|
||||
|
||||
/**
|
||||
* Verifies the RS256 signature of a token presented to an HTTP API v2 JWT authorizer, the same
|
||||
@@ -66,8 +69,27 @@ public class JwtSignatureVerifier implements AutoCloseable {
|
||||
private final CloseableHttpClient httpClient;
|
||||
private final boolean allowPrivateNetworkTargets;
|
||||
private final Map<String, CachedJwks> jwksCache = new ConcurrentHashMap<>();
|
||||
private Function<String, Optional<Map<String, RSAPublicKey>>> localIssuerKeys = issuer -> Optional.empty();
|
||||
|
||||
/**
|
||||
* An issuer that is one of this emulator's own Cognito pools is served at Floci's base URL, which
|
||||
* is plain HTTP on a hostname such as {@code localhost} or a Compose service name, so the HTTPS
|
||||
* and address rules below would refuse to fetch it. Its keys are read from the pool instead:
|
||||
* they are the same keys the pool publishes at its JWKS URL.
|
||||
*/
|
||||
@Inject
|
||||
public JwtSignatureVerifier(ObjectMapper objectMapper, EmulatorConfig config, Instance<CognitoService> cognito) {
|
||||
this(objectMapper, config);
|
||||
withLocalIssuerKeys(issuer -> cognito.isResolvable()
|
||||
? cognito.get().localIssuerKeys(issuer) : Optional.empty());
|
||||
}
|
||||
|
||||
/** Where the keys of in-process issuers come from; package-private so tests can stand one in. */
|
||||
JwtSignatureVerifier withLocalIssuerKeys(Function<String, Optional<Map<String, RSAPublicKey>>> localIssuerKeys) {
|
||||
this.localIssuerKeys = localIssuerKeys;
|
||||
return this;
|
||||
}
|
||||
|
||||
public JwtSignatureVerifier(ObjectMapper objectMapper, EmulatorConfig config) {
|
||||
this(objectMapper, SystemDefaultDnsResolver.INSTANCE,
|
||||
config.security().allowPrivateJwtTargets());
|
||||
@@ -162,6 +184,10 @@ public class JwtSignatureVerifier implements AutoCloseable {
|
||||
}
|
||||
|
||||
private RSAPublicKey resolveKey(String issuer, String kid) throws JwtVerificationException {
|
||||
Optional<Map<String, RSAPublicKey>> local = localIssuerKeys.apply(issuer);
|
||||
if (local.isPresent()) {
|
||||
return local.get().get(kid);
|
||||
}
|
||||
CachedJwks cached = jwksCache.get(issuer);
|
||||
if (cached != null && !cached.isExpired()) {
|
||||
RSAPublicKey key = cached.keysByKid().get(kid);
|
||||
|
||||
@@ -47,7 +47,7 @@ public class AthenaJsonHandler {
|
||||
resultConfiguration = mapper.treeToValue(request.get("ResultConfiguration"), ResultConfiguration.class);
|
||||
}
|
||||
|
||||
String id = athenaService.startQueryExecution(query, workGroup, context, resultConfiguration);
|
||||
String id = athenaService.startQueryExecution(query, workGroup, context, resultConfiguration, region);
|
||||
yield Response.ok(Map.of("QueryExecutionId", id)).build();
|
||||
}
|
||||
case "GetQueryExecution" -> {
|
||||
@@ -62,7 +62,7 @@ public class AthenaJsonHandler {
|
||||
}
|
||||
case "ListQueryExecutions" -> {
|
||||
yield Response.ok(Map.of("QueryExecutionIds",
|
||||
athenaService.listQueryExecutions().stream()
|
||||
athenaService.listQueryExecutions(request.has("WorkGroup") ? request.get("WorkGroup").asText() : null).stream()
|
||||
.map(QueryExecution::getQueryExecutionId).toList())).build();
|
||||
}
|
||||
case "StopQueryExecution" -> {
|
||||
|
||||
@@ -59,6 +59,9 @@ public class AthenaService {
|
||||
public static final String DEFAULT_CATALOG = "AwsDataCatalog";
|
||||
private static final String DEFAULT_OUTPUT_BUCKET = "floci-athena-results";
|
||||
private static final String DEFAULT_WORKGROUP = "primary";
|
||||
private static final Pattern ENGINE_FAILURE = Pattern.compile("^floci-duck \\w+ returned HTTP \\d+: (\\{.*})$", Pattern.DOTALL);
|
||||
private static final com.fasterxml.jackson.databind.ObjectMapper FAILURE_READER = new com.fasterxml.jackson.databind.ObjectMapper();
|
||||
private static final Pattern TRAILING_SEMICOLONS = Pattern.compile("[\\s;]+$");
|
||||
private static final String DEFAULT_ENGINE_VERSION = "Athena engine version 3";
|
||||
private static final long MIN_BYTES_SCANNED_CUTOFF_PER_QUERY = 10_000_000L;
|
||||
private static final String WORKGROUP_RESOURCE = "workgroup/";
|
||||
@@ -122,6 +125,20 @@ public class AthenaService {
|
||||
String workGroup,
|
||||
QueryExecutionContext context,
|
||||
ResultConfiguration resultConfiguration) {
|
||||
return startQueryExecution(query, workGroup, context, resultConfiguration, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a query in a workgroup. As in AWS, the workgroup's result location applies when the
|
||||
* request names none, and always when the workgroup enforces its configuration; an unknown
|
||||
* workgroup is refused. A null region skips the workgroup lookup.
|
||||
*/
|
||||
public String startQueryExecution(String query,
|
||||
String workGroup,
|
||||
QueryExecutionContext context,
|
||||
ResultConfiguration resultConfiguration,
|
||||
String region) {
|
||||
WorkGroupConfiguration workGroupConfiguration = region == null ? null : workGroupFor(region, workGroup).getConfiguration();
|
||||
String id = UUID.randomUUID().toString();
|
||||
String database = context != null && context.getDatabase() != null ? context.getDatabase() : "default";
|
||||
QueryExecutionContext resolvedContext = context != null ? context : new QueryExecutionContext();
|
||||
@@ -135,7 +152,7 @@ public class AthenaService {
|
||||
// directory prefix — the same key is written and returned to the client.
|
||||
// Statements that do not return rows must not be wrapped in a result COPY.
|
||||
String outputLocation = producesResultRows(query)
|
||||
? resolveOutputLocation(resultConfiguration, id)
|
||||
? resolveOutputLocation(resultConfiguration, workGroupConfiguration, id)
|
||||
: null;
|
||||
ResultConfiguration resolvedResult = outputLocation != null
|
||||
? new ResultConfiguration(outputLocation)
|
||||
@@ -177,7 +194,7 @@ public class AthenaService {
|
||||
if (outputLocation != null) {
|
||||
ensureOutputBucket(outputLocation);
|
||||
}
|
||||
duckClient.execute(query, setupDdl, outputLocation);
|
||||
duckClient.execute(withoutTrailingSemicolons(query), setupDdl, outputLocation);
|
||||
return null;
|
||||
}).onSuccess(v -> {
|
||||
markSucceeded(id, execution);
|
||||
@@ -214,6 +231,16 @@ public class AthenaService {
|
||||
return queryStore.scan(k -> true);
|
||||
}
|
||||
|
||||
/** A workgroup's queries, most recent first, as ListQueryExecutions returns them. */
|
||||
public List<QueryExecution> listQueryExecutions(String workGroup) {
|
||||
String name = workGroup == null || workGroup.isBlank() ? DEFAULT_WORKGROUP : workGroup;
|
||||
return queryStore.scan(k -> true).stream()
|
||||
.filter(execution -> name.equals(execution.getWorkGroup() == null ? DEFAULT_WORKGROUP : execution.getWorkGroup()))
|
||||
.sorted(Comparator.comparing((QueryExecution execution) -> execution.getStatus().getSubmissionDateTime(),
|
||||
Comparator.nullsLast(Comparator.naturalOrder())).reversed())
|
||||
.toList();
|
||||
}
|
||||
|
||||
public void stopQueryExecution(String id) {
|
||||
QueryExecution execution = getQueryExecution(id);
|
||||
execution.getStatus().setState(QueryExecutionState.CANCELLED);
|
||||
@@ -531,6 +558,15 @@ public class AthenaService {
|
||||
return properties;
|
||||
}
|
||||
|
||||
/**
|
||||
* Athena accepts one statement with or without a closing semicolon. The engine wraps the
|
||||
* statement in {@code COPY (...) TO ...}, where a semicolon is a syntax error, so it is
|
||||
* dropped before the query is handed over.
|
||||
*/
|
||||
static String withoutTrailingSemicolons(String query) {
|
||||
return query == null ? null : TRAILING_SEMICOLONS.matcher(query).replaceFirst("");
|
||||
}
|
||||
|
||||
private static boolean producesResultRows(String query) {
|
||||
return RESULT_STATEMENT_PATTERN.matcher(statementText(query)).find();
|
||||
}
|
||||
@@ -576,17 +612,53 @@ public class AthenaService {
|
||||
|
||||
private void markFailed(String id, QueryExecution execution, Throwable failure) {
|
||||
execution.getStatus().setState(QueryExecutionState.FAILED);
|
||||
execution.getStatus().setStateChangeReason(failure.getMessage());
|
||||
execution.getStatus().setStateChangeReason(failureReason(failure.getMessage()));
|
||||
queryStore.put(id, execution);
|
||||
}
|
||||
|
||||
/**
|
||||
* The engine's own explanation of a failed query, as Athena reports it: without the HTTP
|
||||
* envelope of the engine call, and without the echoed statement, which shows the
|
||||
* internal {@code COPY ... TO} wrapper rather than the query that was sent.
|
||||
*/
|
||||
static String failureReason(String message) {
|
||||
if (message == null) {
|
||||
return null;
|
||||
}
|
||||
Matcher envelope = ENGINE_FAILURE.matcher(message);
|
||||
if (!envelope.matches()) {
|
||||
return message;
|
||||
}
|
||||
try {
|
||||
String engineMessage = FAILURE_READER.readTree(envelope.group(1)).path("message").asText("");
|
||||
if (engineMessage.isBlank()) {
|
||||
return message;
|
||||
}
|
||||
int echo = engineMessage.indexOf("\n\nLINE ");
|
||||
return (echo >= 0 ? engineMessage.substring(0, echo) : engineMessage).strip();
|
||||
} catch (Exception e) {
|
||||
return message;
|
||||
}
|
||||
}
|
||||
|
||||
private record CreateDatabaseDdl(String name, boolean ifNotExists, String comment,
|
||||
String location, Map<String, String> properties) {
|
||||
}
|
||||
|
||||
private String resolveOutputLocation(ResultConfiguration rc, String queryId) {
|
||||
String base = (rc != null && rc.getOutputLocation() != null && !rc.getOutputLocation().isBlank())
|
||||
private String resolveOutputLocation(ResultConfiguration rc, WorkGroupConfiguration workGroupConfiguration, String queryId) {
|
||||
String requested = rc != null && rc.getOutputLocation() != null && !rc.getOutputLocation().isBlank()
|
||||
? rc.getOutputLocation()
|
||||
: null;
|
||||
String workGroupLocation = workGroupConfiguration != null
|
||||
&& workGroupConfiguration.getResultConfiguration() != null
|
||||
&& workGroupConfiguration.getResultConfiguration().getOutputLocation() != null
|
||||
&& !workGroupConfiguration.getResultConfiguration().getOutputLocation().isBlank()
|
||||
? workGroupConfiguration.getResultConfiguration().getOutputLocation()
|
||||
: null;
|
||||
boolean enforced = workGroupConfiguration != null && Boolean.TRUE.equals(workGroupConfiguration.getEnforceWorkGroupConfiguration());
|
||||
String base = enforced && workGroupLocation != null ? workGroupLocation
|
||||
: requested != null ? requested
|
||||
: workGroupLocation != null ? workGroupLocation
|
||||
: "s3://" + DEFAULT_OUTPUT_BUCKET + "/results/";
|
||||
return base.endsWith("/") ? base + queryId + ".csv" : base + "/" + queryId + ".csv";
|
||||
}
|
||||
@@ -692,6 +764,12 @@ public class AthenaService {
|
||||
return engineVersion;
|
||||
}
|
||||
|
||||
private WorkGroup workGroupFor(String region, String name) {
|
||||
return name == null || name.isBlank() || DEFAULT_WORKGROUP.equals(name)
|
||||
? primaryWorkGroup(region)
|
||||
: requireWorkGroup(region, name);
|
||||
}
|
||||
|
||||
private WorkGroup primaryWorkGroup(String region) {
|
||||
return workGroupStore.get(workGroupKey(region, DEFAULT_WORKGROUP)).orElseGet(() -> {
|
||||
WorkGroup workGroup = new WorkGroup();
|
||||
|
||||
+4
-1
@@ -291,7 +291,10 @@ final class CognitoAuthFlowHandler {
|
||||
if (!"UserNotFoundException".equals(ae.getErrorCode())) throw ae;
|
||||
user = tryUserMigration(pool, client, username, password, null, clientMetadata,
|
||||
"UserMigration_Authentication");
|
||||
if (user == null) throw ae;
|
||||
if (user == null) {
|
||||
if (!CognitoService.hidesUserExistence(client)) throw ae;
|
||||
throw new AwsException("NotAuthorizedException", "Incorrect username or password", 400);
|
||||
}
|
||||
}
|
||||
|
||||
firePreAuthentication(pool, client, user, null, clientMetadata, false);
|
||||
|
||||
@@ -2592,7 +2592,8 @@ public class CognitoService implements ResourceProvider {
|
||||
String confirmationCode) {
|
||||
UserPool pool = poolStore.get(userPoolId)
|
||||
.orElseThrow(() -> userPoolNotFound(userPoolId));
|
||||
CognitoUser user = adminGetUser(client.getUserPoolId(), username);
|
||||
CognitoUser user = userForClientOrNull(client, username);
|
||||
if (user == null) throw codeMismatch();
|
||||
if (verificationCodeService != null && isSignUpConfirmationEnabled(pool)) {
|
||||
try {
|
||||
verificationCodeService.consume(client.getUserPoolId(), user.getUsername(),
|
||||
@@ -2636,7 +2637,8 @@ public class CognitoService implements ResourceProvider {
|
||||
.orElseThrow(() -> new AwsException("ResourceNotFoundException", "Client not found",
|
||||
400));
|
||||
UserPool pool = describeUserPool(client.getUserPoolId());
|
||||
CognitoUser user = adminGetUser(client.getUserPoolId(), username);
|
||||
CognitoUser user = userForClientOrNull(client, username);
|
||||
if (user == null) return simulatedCodeDelivery(username);
|
||||
if (!"UNCONFIRMED".equals(user.getUserStatus())) {
|
||||
throw new AwsException("NotAuthorizedException",
|
||||
"User cannot be confirmed. Current status is " + user.getUserStatus(), 400);
|
||||
@@ -2774,7 +2776,8 @@ public class CognitoService implements ResourceProvider {
|
||||
public Map<String, Object> forgotPassword(String clientId, String username) {
|
||||
UserPoolClient client = clientStore.get(clientId)
|
||||
.orElseThrow(() -> new AwsException("ResourceNotFoundException", "Client not found", 400));
|
||||
CognitoUser user = adminGetUser(client.getUserPoolId(), username);
|
||||
CognitoUser user = userForClientOrNull(client, username);
|
||||
if (user == null) return new LinkedHashMap<>(simulatedCodeDelivery(username));
|
||||
UserPool pool = describeUserPool(client.getUserPoolId());
|
||||
ensureVerificationWiring();
|
||||
DeliveryTarget deliveryTarget = resolveForgotPasswordDeliveryTarget(pool, user);
|
||||
@@ -2800,7 +2803,8 @@ public class CognitoService implements ResourceProvider {
|
||||
public void confirmForgotPassword(String clientId, String username, String confirmationCode, String newPassword) {
|
||||
UserPoolClient client = clientStore.get(clientId)
|
||||
.orElseThrow(() -> new AwsException("ResourceNotFoundException", "Client not found", 400));
|
||||
CognitoUser user = adminGetUser(client.getUserPoolId(), username);
|
||||
CognitoUser user = userForClientOrNull(client, username);
|
||||
if (user == null) throw codeMismatch();
|
||||
ensureVerificationWiring();
|
||||
try {
|
||||
verificationCodeService.consume(client.getUserPoolId(), user.getUsername(),
|
||||
@@ -3075,6 +3079,18 @@ public class CognitoService implements ResourceProvider {
|
||||
return overrideId;
|
||||
}
|
||||
|
||||
/**
|
||||
* The signing keys, by kid, of this emulator's own pool that {@code issuer} names exactly, so a
|
||||
* JWT authorizer can verify the pool's tokens in-process; empty for every other issuer.
|
||||
*/
|
||||
public Optional<Map<String, RSAPublicKey>> localIssuerKeys(String issuer) {
|
||||
if (issuer == null) return Optional.empty();
|
||||
String trimmed = issuer.replaceAll("/+$", "");
|
||||
String poolId = trimmed.substring(trimmed.lastIndexOf('/') + 1);
|
||||
if (poolId.isEmpty() || !getIssuer(poolId).equals(trimmed)) return Optional.empty();
|
||||
return poolStore.get(poolId).map(pool -> Map.of(getSigningKeyId(pool), getSigningPublicKey(pool)));
|
||||
}
|
||||
|
||||
public String getJwksUri(String poolId) {
|
||||
return getIssuer(poolId) + "/.well-known/jwks.json";
|
||||
}
|
||||
@@ -4446,10 +4462,38 @@ public class CognitoService implements ResourceProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the app client hides which usernames exist (PreventUserExistenceErrors=ENABLED). Real Cognito then
|
||||
* answers for an unknown user exactly as for a real one: a failed sign-in is "Incorrect username or password",
|
||||
* a code request reports a simulated delivery and a code check reports a wrong code. Unset means LEGACY.
|
||||
*/
|
||||
static boolean hidesUserExistence(UserPoolClient client) {
|
||||
return "ENABLED".equals(client.getPreventUserExistenceErrors());
|
||||
}
|
||||
|
||||
/** The user, or null when it does not exist and the client hides that; otherwise UserNotFoundException. */
|
||||
private CognitoUser userForClientOrNull(UserPoolClient client, String username) {
|
||||
try {
|
||||
return adminGetUser(client.getUserPoolId(), username);
|
||||
} catch (AwsException e) {
|
||||
if ("UserNotFoundException".equals(e.getErrorCode()) && hidesUserExistence(client)) return null;
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
private Map<String, String> simulatedCodeDelivery(String username) {
|
||||
return username != null && username.contains("@")
|
||||
? Map.of("AttributeName", "email", "DeliveryMedium", "EMAIL", "Destination", maskEmail(username))
|
||||
: Map.of("AttributeName", "phone_number", "DeliveryMedium", "SMS", "Destination", "+*******0000");
|
||||
}
|
||||
|
||||
private static AwsException codeMismatch() {
|
||||
return new AwsException("CodeMismatchException", "Invalid verification code provided, please try again.", 400);
|
||||
}
|
||||
|
||||
private AwsException mapVerificationCodeException(VerificationCodeException e) {
|
||||
return switch (e.getKind()) {
|
||||
case MISMATCH, NOT_FOUND -> new AwsException("CodeMismatchException",
|
||||
"Invalid verification code provided, please try again.", 400);
|
||||
case MISMATCH, NOT_FOUND -> codeMismatch();
|
||||
case EXPIRED -> new AwsException("ExpiredCodeException",
|
||||
"Invalid code provided, please request a code again.", 400);
|
||||
case RATE_LIMIT -> new AwsException("LimitExceededException",
|
||||
|
||||
@@ -8554,6 +8554,12 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
|
||||
"Network interface '" + networkInterfaceId + "' is currently in use", 400);
|
||||
}
|
||||
networkInterfaces.delete(key(region, networkInterfaceId));
|
||||
// Its addresses go back to the subnet, or every task or mount target that ever held one
|
||||
// would leave it leased until the emulator restarts.
|
||||
if (vpcNetworkManager != null && ni.getPrivateIpAddresses() != null) {
|
||||
ni.getPrivateIpAddresses().forEach(address ->
|
||||
vpcNetworkManager.releasePrivateIp(region, ni.getSubnetId(), address.getPrivateIpAddress()));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -285,7 +285,10 @@ public class VpcNetworkManager {
|
||||
}
|
||||
|
||||
int desiredPrefix = declared.map(Cidr4::prefix).orElse(24);
|
||||
Cidr4 slice = allocateSubBlock(vpc.effective, desiredPrefix, siblings);
|
||||
Cidr4 slice = translatedIntoSubstitute(vpc, declared, siblings);
|
||||
if (slice == null) {
|
||||
slice = allocateSubBlock(vpc.effective, desiredPrefix, siblings);
|
||||
}
|
||||
if (slice == null) {
|
||||
LOG.warnv("Subnet {0} in VPC {1}: no free /{2} slice remains inside {3}; instances in this "
|
||||
+ "subnet fall back to synthesised private addresses.",
|
||||
@@ -301,6 +304,26 @@ public class VpcNetworkManager {
|
||||
return bindSubnet(vpc, subnetId, declared.orElse(null), slice, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* The declared subnet moved to the same place inside its VPC's substitute range, or null when
|
||||
* that does not apply. Keeping each subnet's offset, rather than handing out the next free
|
||||
* slice, is what makes the plan come out the same in every run whatever order the subnets are
|
||||
* declared in, so the addresses persisted instances and mount targets hold stay valid.
|
||||
*/
|
||||
private static Cidr4 translatedIntoSubstitute(VpcBinding vpc, Optional<Cidr4> declared, List<Cidr4> siblings) {
|
||||
if (!vpc.substituted || vpc.declared == null || declared.isEmpty()
|
||||
|| vpc.declared.prefix() != vpc.effective.prefix() || !vpc.declared.contains(declared.get())) {
|
||||
return null;
|
||||
}
|
||||
long offset = declared.get().network() - vpc.declared.network();
|
||||
Cidr4 translated = Cidr4.parse(Cidr4.format(vpc.effective.network() + offset) + "/" + declared.get().prefix())
|
||||
.orElse(null);
|
||||
if (translated == null || siblings.stream().anyMatch(sibling -> sibling.overlaps(translated))) {
|
||||
return null;
|
||||
}
|
||||
return translated;
|
||||
}
|
||||
|
||||
/**
|
||||
* Binds a subnet to a range it can actually allocate out of.
|
||||
*
|
||||
|
||||
+15
-63
@@ -14,6 +14,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
|
||||
import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
|
||||
import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;
|
||||
import io.github.hectorvent.floci.services.ec2.Ec2Service;
|
||||
import io.github.hectorvent.floci.services.efs.EfsMounts;
|
||||
import io.github.hectorvent.floci.services.ec2.SecurityGroupFirewallManager;
|
||||
import io.github.hectorvent.floci.services.ec2.SecurityGroupNftCompiler;
|
||||
import io.github.hectorvent.floci.services.ec2.model.NetworkInterface;
|
||||
@@ -63,14 +64,11 @@ import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.format.DateTimeParseException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashSet;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
@@ -124,6 +122,7 @@ public class EcsContainerManager {
|
||||
private final HostVolumePolicy hostVolumePolicy;
|
||||
private Ec2Service ec2Service;
|
||||
private SecurityGroupFirewallManager firewallManager;
|
||||
private EfsMounts efsMounts;
|
||||
|
||||
@Inject
|
||||
public EcsContainerManager(ContainerBuilder containerBuilder,
|
||||
@@ -139,11 +138,13 @@ public class EcsContainerManager {
|
||||
EcrRegistryManager ecrRegistryManager,
|
||||
HostVolumePolicy hostVolumePolicy,
|
||||
Ec2Service ec2Service,
|
||||
SecurityGroupFirewallManager firewallManager) {
|
||||
SecurityGroupFirewallManager firewallManager,
|
||||
EfsMounts efsMounts) {
|
||||
this(containerBuilder, lifecycleManager, logStreamer, containerDetector, config, regionResolver,
|
||||
awsEnv, ssmService, secretsManagerService, s3Service, ecrRegistryManager, hostVolumePolicy);
|
||||
this.ec2Service = ec2Service;
|
||||
this.firewallManager = firewallManager;
|
||||
this.efsMounts = efsMounts;
|
||||
}
|
||||
|
||||
public EcsContainerManager(ContainerBuilder containerBuilder,
|
||||
@@ -170,6 +171,8 @@ public class EcsContainerManager {
|
||||
this.s3Service = s3Service;
|
||||
this.hostVolumePolicy = hostVolumePolicy;
|
||||
this.ecrRegistryManager = ecrRegistryManager;
|
||||
// Without the EFS service only mounts that name no access point can be resolved.
|
||||
this.efsMounts = new EfsMounts(null, lifecycleManager, config);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -391,7 +394,7 @@ public class EcsContainerManager {
|
||||
specBuilder.withBind(sourcePath, mp.containerPath());
|
||||
}
|
||||
} else if (efs != null) {
|
||||
mountEfsVolume(specBuilder, efs, mp);
|
||||
mountEfsVolume(specBuilder, efs, mp, region);
|
||||
} else {
|
||||
LOG.warnv("Skipping mountPoint with unresolved volume {0} on container {1}",
|
||||
mp.sourceVolume(), def.getName());
|
||||
@@ -1769,20 +1772,15 @@ public class EcsContainerManager {
|
||||
}
|
||||
|
||||
/**
|
||||
* Materialises an EFS-configured task volume as a shared local Docker named volume scoped to
|
||||
* both the file system and the mount's effective root (see {@link #efsVolumeName}), then
|
||||
* initialises the volume root's POSIX ownership to emulate the EFS access point's
|
||||
* RootDirectory.CreationInfo (no-op unless {@code floci.storage.efs} configures owner/permissions)
|
||||
* and applies the configured PosixUser emulation (uid[:gid] and/or supplementary group) so a
|
||||
* non-root task image can read/write the shared volume.
|
||||
* Mounts an EFS-configured task volume: the file system's shared Docker volume, or the
|
||||
* access point's or {@code rootDirectory}'s directory inside it (see {@link EfsMounts}). The
|
||||
* container keeps its own user, as on AWS, unless {@code floci.storage.efs.mount-user} sets one.
|
||||
*/
|
||||
private void mountEfsVolume(ContainerBuilder.Builder specBuilder, EfsVolumeConfiguration efs, MountPoint mp) {
|
||||
String efsVolumeName = efsVolumeName(efs.fileSystemId(), efs.accessPointId(), efs.rootDirectory());
|
||||
private void mountEfsVolume(ContainerBuilder.Builder specBuilder, EfsVolumeConfiguration efs, MountPoint mp,
|
||||
String region) {
|
||||
efsMounts.prepare(region, efs.fileSystemId(), efs.accessPointId(), efs.rootDirectory())
|
||||
.applyTo(specBuilder, mp.containerPath(), mp.readOnly());
|
||||
var efsCfg = config.storage().efs();
|
||||
lifecycleManager.ensureSharedVolume(efsVolumeName,
|
||||
efsCfg.ownerUid(), efsCfg.ownerGid(), efsCfg.rootPermissions(),
|
||||
efsCfg.initImage());
|
||||
specBuilder.withNamedVolume(efsVolumeName, mp.containerPath(), mp.readOnly());
|
||||
efsCfg.mountUser().ifPresent(u -> {
|
||||
if (!u.matches("^\\d+(:\\d+)?$")) {
|
||||
throw new IllegalArgumentException(
|
||||
@@ -1793,52 +1791,6 @@ public class EcsContainerManager {
|
||||
efsCfg.mountGroupAdd().ifPresent(gid -> specBuilder.withGroupAdd(String.valueOf(gid)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Name of the local Docker named volume backing an EFS-configured task volume, scoped to
|
||||
* both the file system and the mount's effective root so two mounts of the same file system
|
||||
* with a different {@code rootDirectory}/{@code accessPointId} land on isolated volumes
|
||||
* while identical configurations keep sharing one, matching how AWS scopes an EFS mount to
|
||||
* a subpath. When {@code accessPointId} is set, it determines the effective root: on real
|
||||
* AWS an access point's own root directory takes precedence over any {@code rootDirectory}
|
||||
* on the volume.
|
||||
*/
|
||||
static String efsVolumeName(String fileSystemId, String accessPointId, String rootDirectory) {
|
||||
if (accessPointId != null && !accessPointId.isBlank()) {
|
||||
return "floci-efs-" + fileSystemId + "-" + sha256Hex("accessPoint:" + accessPointId);
|
||||
}
|
||||
String normalizedRoot = normalizeRootDirectory(rootDirectory);
|
||||
if ("/".equals(normalizedRoot)) {
|
||||
return "floci-efs-" + fileSystemId;
|
||||
}
|
||||
return "floci-efs-" + fileSystemId + "-" + sha256Hex(normalizedRoot);
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonicalises a {@code rootDirectory} path so syntactically different but equivalent
|
||||
* paths (missing/blank, a missing leading slash, repeated slashes, a trailing slash) resolve
|
||||
* to the same effective root instead of silently splitting shared storage across local
|
||||
* volumes that AWS would treat as identical.
|
||||
*/
|
||||
private static String normalizeRootDirectory(String rootDirectory) {
|
||||
if (rootDirectory == null || rootDirectory.isBlank()) {
|
||||
return "/";
|
||||
}
|
||||
String collapsed = rootDirectory.trim().replaceAll("/+", "/");
|
||||
if (collapsed.length() > 1 && collapsed.endsWith("/")) {
|
||||
collapsed = collapsed.substring(0, collapsed.length() - 1);
|
||||
}
|
||||
return collapsed.startsWith("/") ? collapsed : "/" + collapsed;
|
||||
}
|
||||
|
||||
private static String sha256Hex(String value) {
|
||||
try {
|
||||
byte[] digest = MessageDigest.getInstance("SHA-256").digest(value.getBytes(StandardCharsets.UTF_8));
|
||||
return HexFormat.of().formatHex(digest);
|
||||
} catch (NoSuchAlgorithmException e) {
|
||||
throw new IllegalStateException("SHA-256 is required but not available", e);
|
||||
}
|
||||
}
|
||||
|
||||
// Inner enum to avoid import cycle — mirrors model.TaskStatus for readability
|
||||
private enum TaskStatus {RUNNING}
|
||||
}
|
||||
|
||||
@@ -29,6 +29,14 @@ public class EfsException extends AwsException {
|
||||
return new EfsException(Status.NOT_FOUND, "MountTargetNotFound", "Mount target " + mountTargetId + " does not exist.");
|
||||
}
|
||||
|
||||
public static EfsException subnetNotFound(String subnetId) {
|
||||
return new EfsException(Status.BAD_REQUEST, "SubnetNotFound", "The subnet ID '" + subnetId + "' does not exist");
|
||||
}
|
||||
|
||||
public static EfsException securityGroupNotFound(String message) {
|
||||
return new EfsException(Status.BAD_REQUEST, "SecurityGroupNotFound", message);
|
||||
}
|
||||
|
||||
public static EfsException mountTargetConflict(String message) {
|
||||
return new EfsException(Status.CONFLICT, "MountTargetConflict", message);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
package io.github.hectorvent.floci.services.efs;
|
||||
|
||||
import io.github.hectorvent.floci.config.EmulatorConfig;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.DirectoryCreation;
|
||||
import io.github.hectorvent.floci.services.efs.model.AccessPointDescription;
|
||||
import io.github.hectorvent.floci.services.efs.model.CreationInfo;
|
||||
import io.github.hectorvent.floci.services.efs.model.PosixUser;
|
||||
import jakarta.enterprise.context.ApplicationScoped;
|
||||
import jakarta.inject.Inject;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
/**
|
||||
* Turns an EFS mount (a file system, optionally through an access point) into a Docker volume
|
||||
* mount. Each file system is one named volume, so every mount of it, from Lambda or ECS, through
|
||||
* any access point, shares the same data. An access point mounts its root directory, a subfolder
|
||||
* of that volume, which is created on first use from the access point's {@code CreationInfo}.
|
||||
*/
|
||||
@ApplicationScoped
|
||||
public class EfsMounts {
|
||||
|
||||
/** A file system directory ready to mount, and the identity its access point enforces, if any. */
|
||||
public record Mount(String volumeName, String subpath, Optional<PosixUser> accessPointUser) {
|
||||
|
||||
public void applyTo(ContainerBuilder.Builder spec, String containerPath, boolean readOnly) {
|
||||
if (subpath.isEmpty()) {
|
||||
spec.withNamedVolume(volumeName, containerPath, readOnly);
|
||||
} else {
|
||||
spec.withNamedVolumeSubpath(volumeName, subpath, containerPath, readOnly);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A plain rootDirectory (an ECS volume without an access point) has no CreationInfo; create it
|
||||
// the way a Docker named volume's own root is created.
|
||||
private static final DirectoryCreation PLAIN_ROOT_DIRECTORY = new DirectoryCreation(0, 0, "755");
|
||||
|
||||
private final EfsService efsService;
|
||||
private final ContainerLifecycleManager lifecycleManager;
|
||||
private final EmulatorConfig config;
|
||||
|
||||
@Inject
|
||||
public EfsMounts(EfsService efsService, ContainerLifecycleManager lifecycleManager, EmulatorConfig config) {
|
||||
this.efsService = efsService;
|
||||
this.lifecycleManager = lifecycleManager;
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
public static String volumeName(String fileSystemId) {
|
||||
return "floci-efs-" + fileSystemId;
|
||||
}
|
||||
|
||||
/** Prepares a Lambda {@code FileSystemConfig}, which names an access point by ARN. */
|
||||
public Mount prepareAccessPoint(String accessPointArn) {
|
||||
String[] parts = accessPointArn == null ? new String[0] : accessPointArn.split(":", 6);
|
||||
if (parts.length != 6 || !parts[5].startsWith("access-point/")) {
|
||||
throw new IllegalArgumentException("Not an EFS access point ARN: " + accessPointArn);
|
||||
}
|
||||
return prepare(parts[3], null, parts[5].substring("access-point/".length()), null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepares a mount of {@code fileSystemId}. With an access point its own root directory and
|
||||
* owner apply and {@code rootDirectory} is ignored, as on AWS; {@code fileSystemId} may then
|
||||
* be null, and otherwise must be the access point's file system.
|
||||
*/
|
||||
public Mount prepare(String region, String fileSystemId, String accessPointId, String rootDirectory) {
|
||||
String root;
|
||||
Optional<DirectoryCreation> creation;
|
||||
Optional<PosixUser> user = Optional.empty();
|
||||
if (accessPointId != null && !accessPointId.isBlank()) {
|
||||
if (efsService == null) {
|
||||
throw new IllegalStateException("EFS access points are unavailable: the EFS service is not running");
|
||||
}
|
||||
AccessPointDescription accessPoint = efsService.describeAccessPoints(region, null, accessPointId)
|
||||
.stream().findFirst()
|
||||
.orElseThrow(() -> EfsException.accessPointNotFound(accessPointId));
|
||||
if (fileSystemId != null && !fileSystemId.equals(accessPoint.getFileSystemId())) {
|
||||
throw new IllegalArgumentException("Access point " + accessPointId + " belongs to file system "
|
||||
+ accessPoint.getFileSystemId() + ", not " + fileSystemId);
|
||||
}
|
||||
fileSystemId = accessPoint.getFileSystemId();
|
||||
var directory = accessPoint.getRootDirectory();
|
||||
root = normalizeRootDirectory(directory == null ? null : directory.getPath());
|
||||
creation = Optional.ofNullable(directory == null ? null : directory.getCreationInfo())
|
||||
.flatMap(EfsMounts::directoryCreation);
|
||||
user = Optional.ofNullable(accessPoint.getPosixUser());
|
||||
} else {
|
||||
root = normalizeRootDirectory(rootDirectory);
|
||||
creation = Optional.of(PLAIN_ROOT_DIRECTORY);
|
||||
}
|
||||
|
||||
String volumeName = volumeName(fileSystemId);
|
||||
var efsCfg = config.storage().efs();
|
||||
lifecycleManager.ensureSharedVolume(volumeName,
|
||||
efsCfg.ownerUid(), efsCfg.ownerGid(), efsCfg.rootPermissions(), efsCfg.initImage());
|
||||
String subpath = root.substring(1);
|
||||
if (!subpath.isEmpty()
|
||||
&& !lifecycleManager.ensureSharedVolumeDirectory(volumeName, subpath, creation, efsCfg.initImage())) {
|
||||
throw new IllegalStateException("The root directory " + root + " of access point " + accessPointId
|
||||
+ " does not exist on file system " + fileSystemId
|
||||
+ ", and the access point has no CreationInfo to create it");
|
||||
}
|
||||
return new Mount(volumeName, subpath, user);
|
||||
}
|
||||
|
||||
/** {@code "uid:gid"} for a container that runs as the access point's POSIX user. */
|
||||
public static String containerUser(PosixUser user) {
|
||||
return user.getUid() + ":" + user.getGid();
|
||||
}
|
||||
|
||||
public static List<Long> secondaryGids(PosixUser user) {
|
||||
return user.getSecondaryGids() == null ? List.of() : user.getSecondaryGids();
|
||||
}
|
||||
|
||||
private static Optional<DirectoryCreation> directoryCreation(CreationInfo info) {
|
||||
if (info.getOwnerUid() == null || info.getOwnerGid() == null || info.getPermissions() == null) {
|
||||
return Optional.empty();
|
||||
}
|
||||
return Optional.of(new DirectoryCreation(info.getOwnerUid(), info.getOwnerGid(), info.getPermissions()));
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonicalises a root directory so equivalent spellings (missing or blank, no leading slash,
|
||||
* repeated or trailing slashes) name the same directory. A {@code ..} component is refused.
|
||||
*/
|
||||
static String normalizeRootDirectory(String rootDirectory) {
|
||||
if (rootDirectory == null || rootDirectory.isBlank()) {
|
||||
return "/";
|
||||
}
|
||||
String collapsed = rootDirectory.trim().replaceAll("/+", "/");
|
||||
if (collapsed.length() > 1 && collapsed.endsWith("/")) {
|
||||
collapsed = collapsed.substring(0, collapsed.length() - 1);
|
||||
}
|
||||
String normalized = collapsed.startsWith("/") ? collapsed : "/" + collapsed;
|
||||
if (List.of(normalized.split("/")).contains("..")) {
|
||||
throw new IllegalArgumentException("A root directory cannot contain '..': " + rootDirectory);
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package io.github.hectorvent.floci.services.efs;
|
||||
|
||||
import com.github.dockerjava.api.DockerClient;
|
||||
import com.github.dockerjava.api.async.ResultCallback;
|
||||
import com.github.dockerjava.api.command.BuildImageResultCallback;
|
||||
import com.github.dockerjava.api.exception.NotFoundException;
|
||||
import com.github.dockerjava.api.model.Frame;
|
||||
import io.github.hectorvent.floci.config.EmulatorConfig;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
|
||||
import io.github.hectorvent.floci.services.ec2.net.VpcNetworkManager;
|
||||
import io.github.hectorvent.floci.services.efs.model.MountTarget;
|
||||
import jakarta.annotation.PreDestroy;
|
||||
import jakarta.enterprise.context.ApplicationScoped;
|
||||
import jakarta.inject.Inject;
|
||||
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
|
||||
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
|
||||
import org.jboss.logging.Logger;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Consumer;
|
||||
|
||||
/**
|
||||
* Serves each EFS mount target over NFSv4.1, as on AWS: a privileged helper container running the
|
||||
* kernel NFS server holds the mount target's address on its VPC's Docker network and exports the
|
||||
* file system's volume, the same one Lambda and ECS mount directly. An EC2 instance in the VPC can
|
||||
* then mount the file system with the commands AWS documents, by address or by its DNS name.
|
||||
*/
|
||||
@ApplicationScoped
|
||||
public class EfsNfsServers {
|
||||
|
||||
private static final Logger LOG = Logger.getLogger(EfsNfsServers.class);
|
||||
|
||||
static final String LABEL_COMPONENT = "floci_component";
|
||||
static final String COMPONENT_VALUE = "efs-nfs-server";
|
||||
static final String LABEL_OWNER = "floci_efs_owner";
|
||||
static final String DEFAULT_IMAGE = "floci/efs-nfs-server:local";
|
||||
private static final int READY_ATTEMPTS = 60;
|
||||
|
||||
private final EmulatorConfig config;
|
||||
private final DockerClient dockerClient;
|
||||
private final ContainerBuilder containerBuilder;
|
||||
private final ContainerLifecycleManager lifecycleManager;
|
||||
private final VpcNetworkManager vpcNetworkManager;
|
||||
private final ExecutorService starter = Executors.newSingleThreadExecutor(r -> {
|
||||
Thread thread = new Thread(r, "efs-nfs-servers");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
});
|
||||
|
||||
@Inject
|
||||
public EfsNfsServers(EmulatorConfig config, DockerClient dockerClient, ContainerBuilder containerBuilder,
|
||||
ContainerLifecycleManager lifecycleManager, VpcNetworkManager vpcNetworkManager) {
|
||||
this.config = config;
|
||||
this.dockerClient = dockerClient;
|
||||
this.containerBuilder = containerBuilder;
|
||||
this.lifecycleManager = lifecycleManager;
|
||||
this.vpcNetworkManager = vpcNetworkManager;
|
||||
}
|
||||
|
||||
@PreDestroy
|
||||
void shutdown() {
|
||||
starter.shutdownNow();
|
||||
}
|
||||
|
||||
/** Whether mount targets get an NFS server: configured on, and VPCs have Docker networks to hold their address. */
|
||||
public boolean enabled() {
|
||||
return config.services().efs().nfsServers() && vpcNetworkManager.enabled();
|
||||
}
|
||||
|
||||
static String containerName(String mountTargetId) {
|
||||
return "floci-efs-nfs-" + mountTargetId;
|
||||
}
|
||||
|
||||
/**
|
||||
* Starts the mount target's NFS server in the background and then runs {@code onSettled}
|
||||
* with whether it is serving. A server left from an earlier run is replaced.
|
||||
*/
|
||||
public void serve(String region, String accountId, MountTarget mountTarget, Consumer<Boolean> onSettled) {
|
||||
starter.execute(() -> {
|
||||
boolean serving = false;
|
||||
try {
|
||||
serving = start(region, accountId, mountTarget);
|
||||
} catch (RuntimeException e) {
|
||||
LOG.warnv("Mount target {0} of {1} has no NFS server: {2}",
|
||||
mountTarget.getMountTargetId(), mountTarget.getFileSystemId(), e.getMessage());
|
||||
}
|
||||
onSettled.accept(serving);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-claims a persisted mount target's address after a restart. The VPC address leases live
|
||||
* only in memory, so without this the next instance launched in the subnet could be handed it.
|
||||
*/
|
||||
public void reserveAddress(String region, MountTarget mountTarget) {
|
||||
vpcNetworkManager.reservePrivateIp(region, mountTarget.getSubnetId(), mountTarget.getIpAddress());
|
||||
}
|
||||
|
||||
/** Stops and removes the mount target's NFS server, if it has one. */
|
||||
public void stop(String mountTargetId) {
|
||||
lifecycleManager.removeIfExists(containerName(mountTargetId));
|
||||
}
|
||||
|
||||
/** Removes every NFS server this emulator started, such as those a previous run left behind. */
|
||||
public void removeAll() {
|
||||
if (!enabled()) {
|
||||
return;
|
||||
}
|
||||
dockerClient.listContainersCmd().withShowAll(true)
|
||||
.withLabelFilter(Map.of(LABEL_COMPONENT, COMPONENT_VALUE, LABEL_OWNER, owner()))
|
||||
.exec()
|
||||
.forEach(container -> lifecycleManager.removeIfExists(container.getId()));
|
||||
}
|
||||
|
||||
private boolean start(String region, String accountId, MountTarget mountTarget) {
|
||||
String name = containerName(mountTarget.getMountTargetId());
|
||||
String image = ensureImage();
|
||||
String volumeName = EfsMounts.volumeName(mountTarget.getFileSystemId());
|
||||
var efsCfg = config.storage().efs();
|
||||
lifecycleManager.ensureSharedVolume(volumeName,
|
||||
efsCfg.ownerUid(), efsCfg.ownerGid(), efsCfg.rootPermissions(), efsCfg.initImage());
|
||||
lifecycleManager.removeIfExists(name);
|
||||
|
||||
var spec = containerBuilder.newContainer(image)
|
||||
.withName(name)
|
||||
.withPrivileged(true)
|
||||
.withNamedVolume(volumeName, "/export", false)
|
||||
.withLogRotation()
|
||||
.withLabels(ContainerStorageHelper.resourceIdentityLabels(
|
||||
"efs", mountTarget.getMountTargetId(), accountId, region))
|
||||
.withLabels(Map.of(LABEL_COMPONENT, COMPONENT_VALUE, LABEL_OWNER, owner()))
|
||||
.build();
|
||||
String containerId = lifecycleManager.createAndStart(spec).containerId();
|
||||
if (vpcNetworkManager.attach(region, mountTarget.getVpcId(), mountTarget.getSubnetId(),
|
||||
containerId, mountTarget.getIpAddress()).isEmpty()) {
|
||||
lifecycleManager.removeIfExists(name);
|
||||
LOG.warnv("Mount target {0} could not take its address {1} on the VPC network; it has no NFS server",
|
||||
mountTarget.getMountTargetId(), mountTarget.getIpAddress());
|
||||
return false;
|
||||
}
|
||||
if (!awaitReady(containerId)) {
|
||||
lifecycleManager.removeIfExists(name);
|
||||
LOG.warnv("The NFS server of mount target {0} did not start", mountTarget.getMountTargetId());
|
||||
return false;
|
||||
}
|
||||
LOG.infov("Mount target {0} serves file system {1} over NFS at {2}",
|
||||
mountTarget.getMountTargetId(), mountTarget.getFileSystemId(), mountTarget.getIpAddress());
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Ready once the kernel NFS server has threads and mountd answers export lookups. */
|
||||
private boolean awaitReady(String containerId) {
|
||||
for (int attempt = 0; attempt < READY_ATTEMPTS; attempt++) {
|
||||
try {
|
||||
String execId = dockerClient.execCreateCmd(containerId)
|
||||
.withCmd("sh", "-c", "[ \"$(cat /proc/fs/nfsd/threads)\" -gt 0 ] && pgrep rpc.mountd")
|
||||
.withAttachStdout(true).withAttachStderr(true)
|
||||
.exec().getId();
|
||||
dockerClient.execStartCmd(execId).exec(new ResultCallback.Adapter<Frame>()).awaitCompletion(10, TimeUnit.SECONDS);
|
||||
Long exitCode = dockerClient.inspectExecCmd(execId).exec().getExitCodeLong();
|
||||
if (exitCode != null && exitCode == 0) {
|
||||
return true;
|
||||
}
|
||||
TimeUnit.MILLISECONDS.sleep(500);
|
||||
} catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
return false;
|
||||
} catch (RuntimeException e) {
|
||||
LOG.debugv("NFS server {0} not ready yet: {1}", containerId, e.getMessage());
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
private synchronized String ensureImage() {
|
||||
String configured = config.services().efs().nfsServerImage();
|
||||
String image = containerBuilder.resolveImage(configured);
|
||||
if (!DEFAULT_IMAGE.equals(configured)) {
|
||||
return image;
|
||||
}
|
||||
try {
|
||||
dockerClient.inspectImageCmd(image).exec();
|
||||
return image;
|
||||
} catch (NotFoundException missing) {
|
||||
// Build Floci's own recipe in the daemon that runs the workloads.
|
||||
}
|
||||
try (InputStream dockerfile = getClass().getResourceAsStream("/docker/efs-nfs-server.Dockerfile")) {
|
||||
if (dockerfile == null) {
|
||||
throw new IllegalStateException("The EFS NFS server Dockerfile is missing");
|
||||
}
|
||||
byte[] content = dockerfile.readAllBytes();
|
||||
ByteArrayOutputStream archive = new ByteArrayOutputStream(content.length + 1024);
|
||||
try (TarArchiveOutputStream tar = new TarArchiveOutputStream(archive)) {
|
||||
TarArchiveEntry entry = new TarArchiveEntry("Dockerfile");
|
||||
entry.setSize(content.length);
|
||||
tar.putArchiveEntry(entry);
|
||||
tar.write(content);
|
||||
tar.closeArchiveEntry();
|
||||
}
|
||||
try (BuildImageResultCallback callback = new BuildImageResultCallback()) {
|
||||
String built = dockerClient.buildImageCmd(new ByteArrayInputStream(archive.toByteArray()))
|
||||
.withTags(Set.of(image)).exec(callback).awaitImageId();
|
||||
if (built == null || built.isBlank()) {
|
||||
throw new IllegalStateException("Docker did not build the EFS NFS server image");
|
||||
}
|
||||
}
|
||||
return image;
|
||||
} catch (IOException e) {
|
||||
throw new IllegalStateException("Cannot build the EFS NFS server image", e);
|
||||
}
|
||||
}
|
||||
|
||||
/** The emulator that owns a server, so one emulator never removes another's on a shared daemon. */
|
||||
private String owner() {
|
||||
String namespace = config.docker() == null || config.docker().resourceNamespace() == null
|
||||
? "" : config.docker().resourceNamespace().orElse("");
|
||||
return namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,15 @@ package io.github.hectorvent.floci.services.efs;
|
||||
|
||||
import io.github.hectorvent.floci.core.storage.StorageBackend;
|
||||
import io.github.hectorvent.floci.core.storage.StorageFactory;
|
||||
import io.github.hectorvent.floci.core.common.AwsException;
|
||||
import io.github.hectorvent.floci.core.common.Resettable;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
|
||||
import io.github.hectorvent.floci.core.common.dns.EmbeddedDnsServer;
|
||||
import io.github.hectorvent.floci.services.ec2.Ec2Service;
|
||||
import io.github.hectorvent.floci.services.ec2.model.NetworkInterface;
|
||||
import io.github.hectorvent.floci.services.ec2.model.Subnet;
|
||||
import io.quarkus.runtime.StartupEvent;
|
||||
import jakarta.enterprise.event.Observes;
|
||||
import io.github.hectorvent.floci.services.efs.model.*;
|
||||
import jakarta.enterprise.context.ApplicationScoped;
|
||||
import jakarta.inject.Inject;
|
||||
@@ -14,6 +22,9 @@ import java.util.Comparator;
|
||||
import java.util.Date;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.Optional;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
import java.util.stream.Collectors;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
@@ -32,14 +43,29 @@ public class EfsService implements Resettable {
|
||||
private final StorageBackend<String, List<LifecyclePolicy>> lifecycleConfigurationStore;
|
||||
private final ConcurrentHashMap<String, Object> syncLocks = new ConcurrentHashMap<>();
|
||||
private final RegionResolver regionResolver;
|
||||
private ContainerLifecycleManager lifecycleManager;
|
||||
private EfsNfsServers nfsServers;
|
||||
private EmbeddedDnsServer dnsServer;
|
||||
|
||||
// A file system's DNS name, optionally for one Availability Zone's mount target:
|
||||
// [az.]fs-id.efs.region.amazonaws.com
|
||||
private static final Pattern FILE_SYSTEM_DNS_NAME = Pattern.compile(
|
||||
"^(?:([a-z0-9-]+)\\.)?(fs-[0-9a-f]+)\\.efs\\.([a-z0-9-]+)\\.amazonaws\\.com\\.?$");
|
||||
|
||||
// RootDirectory.Path as the EFS API documents it: "/" or up to 100 "/name" segments, where a
|
||||
// name does not start with "." and holds none of $#<>;`|&?{}^* or a newline.
|
||||
private static final Pattern ROOT_DIRECTORY_PATH = Pattern.compile("^(/|(/(?!\\.)[^$#<>;`|&?{}^*/\\n]+){1,100})$");
|
||||
|
||||
private final Ec2Service ec2Service;
|
||||
|
||||
private Object lockFor(String key) {
|
||||
return syncLocks.computeIfAbsent(key, k -> new Object());
|
||||
}
|
||||
|
||||
@Inject
|
||||
public EfsService(StorageFactory storageFactory, RegionResolver regionResolver) {
|
||||
public EfsService(StorageFactory storageFactory, RegionResolver regionResolver, Ec2Service ec2Service) {
|
||||
this.regionResolver = regionResolver;
|
||||
this.ec2Service = ec2Service;
|
||||
this.fileSystemStore = storageFactory.create("efs", "efs-filesystems.json",
|
||||
new com.fasterxml.jackson.core.type.TypeReference<java.util.Map<String, FileSystem>>() {});
|
||||
this.mountTargetStore = storageFactory.create("efs", "efs-mounttargets.json",
|
||||
@@ -54,6 +80,82 @@ public class EfsService implements Resettable {
|
||||
new com.fasterxml.jackson.core.type.TypeReference<java.util.Map<String, List<LifecyclePolicy>>>() {});
|
||||
}
|
||||
|
||||
/** Lets a deleted file system take its stored data with it; absent in tests without Docker. */
|
||||
@Inject
|
||||
void setLifecycleManager(ContainerLifecycleManager lifecycleManager) {
|
||||
this.lifecycleManager = lifecycleManager;
|
||||
}
|
||||
|
||||
/** Serves mount targets over NFS; absent in tests that build the service by hand. */
|
||||
@Inject
|
||||
void setNfsServers(EfsNfsServers nfsServers, EmbeddedDnsServer dnsServer) {
|
||||
this.nfsServers = nfsServers;
|
||||
this.dnsServer = dnsServer;
|
||||
}
|
||||
|
||||
/**
|
||||
* The address a file system's DNS name resolves to: the mount target in the named Availability
|
||||
* Zone, or, for the plain name, the first mount target that is serving. AWS answers the plain
|
||||
* name with the mount target in the client's own zone; every mount target here is reachable
|
||||
* from the whole VPC, so any serving one will do.
|
||||
*/
|
||||
Optional<String> resolveDnsName(String name) {
|
||||
Matcher matcher = FILE_SYSTEM_DNS_NAME.matcher(name.toLowerCase());
|
||||
if (!matcher.matches()) {
|
||||
return Optional.empty();
|
||||
}
|
||||
String zone = matcher.group(1);
|
||||
String fileSystemId = matcher.group(2);
|
||||
String region = matcher.group(3);
|
||||
return mountTargetStore.scan(k -> k.startsWith(region + "::")).stream()
|
||||
.filter(mt -> fileSystemId.equals(mt.getFileSystemId()))
|
||||
.filter(mt -> mt.getLifeCycleState() == LifeCycleState.available)
|
||||
.filter(mt -> zone == null || zone.equals(mt.getAvailabilityZoneName()))
|
||||
.sorted(Comparator.comparing(MountTarget::getAvailabilityZoneName))
|
||||
.map(MountTarget::getIpAddress)
|
||||
.findFirst();
|
||||
}
|
||||
|
||||
/**
|
||||
* Brings back the NFS servers of persisted mount targets after a restart: each one re-claims
|
||||
* its address on the VPC network, so no instance is handed it, and serves again.
|
||||
*/
|
||||
void onStart(@Observes StartupEvent ignored) {
|
||||
if (nfsServers == null || !nfsServers.enabled()) {
|
||||
return;
|
||||
}
|
||||
dnsServer.addResolver(this::resolveDnsName);
|
||||
nfsServers.removeAll();
|
||||
for (String key : mountTargetStore.keys()) {
|
||||
String region = key.substring(0, key.indexOf("::"));
|
||||
mountTargetStore.get(key).ifPresent(mt -> {
|
||||
// Looking the subnet up first also brings up EC2, which is what rebuilds the VPC
|
||||
// networks the address reservation and the server's attachment both need.
|
||||
try {
|
||||
requireSubnet(region, mt.getSubnetId());
|
||||
} catch (EfsException e) {
|
||||
LOG.warnv("Mount target {0} is in subnet {1}, which no longer exists; it has no NFS server",
|
||||
mt.getMountTargetId(), mt.getSubnetId());
|
||||
return;
|
||||
}
|
||||
nfsServers.reserveAddress(region, mt);
|
||||
serveOverNfs(region, mt);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private void serveOverNfs(String region, MountTarget mt) {
|
||||
nfsServers.serve(region, regionResolver.getAccountId(), mt, serving -> {
|
||||
String key = regionKey(region, mt.getMountTargetId());
|
||||
synchronized (lockFor(key)) {
|
||||
mountTargetStore.get(key).ifPresent(current -> {
|
||||
current.setLifeCycleState(LifeCycleState.available);
|
||||
mountTargetStore.put(key, current);
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Override
|
||||
public void clear() {
|
||||
fileSystemStore.clear();
|
||||
@@ -267,6 +369,10 @@ public class EfsService implements Resettable {
|
||||
|
||||
fileSystemStore.delete(key);
|
||||
}
|
||||
if (lifecycleManager != null && !lifecycleManager.removeVolume(EfsMounts.volumeName(fileSystemId))) {
|
||||
LOG.warnv("The data of deleted file system {0} is still in Docker volume {1}, which is in use",
|
||||
fileSystemId, EfsMounts.volumeName(fileSystemId));
|
||||
}
|
||||
}
|
||||
|
||||
// --- Tags ---
|
||||
@@ -361,37 +467,60 @@ public class EfsService implements Resettable {
|
||||
public MountTarget createMountTarget(CreateMountTargetRequest request, String region) {
|
||||
synchronized (lockFor(regionKey(region, request.getFileSystemId()))) {
|
||||
FileSystem fs = getFileSystem(region, request.getFileSystemId());
|
||||
|
||||
for (MountTarget existing : mountTargetStore.scan(k -> k.startsWith(region + "::"))) {
|
||||
Subnet subnet = requireSubnet(region, request.getSubnetId());
|
||||
|
||||
List<MountTarget> regionMountTargets = mountTargetStore.scan(k -> k.startsWith(region + "::"));
|
||||
for (MountTarget existing : regionMountTargets) {
|
||||
if (existing.getFileSystemId().equals(request.getFileSystemId())) {
|
||||
if (existing.getSubnetId().equals(request.getSubnetId())) {
|
||||
throw EfsException.mountTargetConflict("Mount target already exists in this subnet");
|
||||
}
|
||||
// AWS allows one mount target per Availability Zone, all in one VPC.
|
||||
if (subnet.getAvailabilityZone().equals(existing.getAvailabilityZoneName())) {
|
||||
throw EfsException.mountTargetConflict("Mount target already exists in this AZ");
|
||||
}
|
||||
if (!subnet.getVpcId().equals(existing.getVpcId())) {
|
||||
throw EfsException.mountTargetConflict("Mount targets of a file system must be in the same VPC");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
String mtId = "fsmt-" + UUID.randomUUID().toString().replace("-", "").substring(0, 17);
|
||||
|
||||
// As on AWS, the mount target is a network interface in the subnet, which is what
|
||||
// gives it an address no instance can be handed as well.
|
||||
NetworkInterface eni;
|
||||
try {
|
||||
eni = ec2Service.createNetworkInterface(region, request.getSubnetId(),
|
||||
"EFS mount target for " + fs.getFileSystemId() + " (" + mtId + ")",
|
||||
request.getIpAddress(), null, request.getSecurityGroups(), null);
|
||||
} catch (AwsException e) {
|
||||
if ("InvalidGroup.NotFound".equals(e.getErrorCode())) {
|
||||
throw EfsException.securityGroupNotFound(e.getMessage());
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
ec2Service.holdNetworkInterfaceForService(region, eni.getNetworkInterfaceId());
|
||||
|
||||
MountTarget mt = new MountTarget();
|
||||
mt.setMountTargetId(mtId);
|
||||
mt.setFileSystemId(request.getFileSystemId());
|
||||
mt.setSubnetId(request.getSubnetId());
|
||||
mt.setIpAddress(request.getIpAddress() != null ? request.getIpAddress() : "10.0.0.10");
|
||||
mt.setLifeCycleState(LifeCycleState.available);
|
||||
mt.setVpcId("vpc-12345678");
|
||||
mt.setAvailabilityZoneName(region + "a");
|
||||
mt.setAvailabilityZoneId(generateAzId(region + "a"));
|
||||
mt.setNetworkInterfaceId("eni-" + UUID.randomUUID().toString().replace("-", "").substring(0, 17));
|
||||
if (request.getSecurityGroups() != null) {
|
||||
mt.setSecurityGroups(new ArrayList<>(request.getSecurityGroups()));
|
||||
} else {
|
||||
mt.setSecurityGroups(new ArrayList<>());
|
||||
}
|
||||
mt.setIpAddress(eni.getPrivateIpAddress());
|
||||
mt.setVpcId(subnet.getVpcId());
|
||||
mt.setAvailabilityZoneName(subnet.getAvailabilityZone());
|
||||
mt.setAvailabilityZoneId(generateAzId(subnet.getAvailabilityZone()));
|
||||
mt.setNetworkInterfaceId(eni.getNetworkInterfaceId());
|
||||
mt.setSecurityGroups(eni.getGroups().stream().map(group -> group.getGroupId())
|
||||
.collect(Collectors.toCollection(ArrayList::new)));
|
||||
boolean servedOverNfs = nfsServers != null && nfsServers.enabled();
|
||||
mt.setLifeCycleState(servedOverNfs ? LifeCycleState.creating : LifeCycleState.available);
|
||||
|
||||
fs.setNumberOfMountTargets(fs.getNumberOfMountTargets() + 1);
|
||||
fileSystemStore.put(regionKey(region, fs.getFileSystemId()), fs);
|
||||
|
||||
mountTargetStore.put(regionKey(region, mtId), mt);
|
||||
if (servedOverNfs) {
|
||||
serveOverNfs(region, mt);
|
||||
}
|
||||
return mt;
|
||||
}
|
||||
}
|
||||
@@ -453,7 +582,11 @@ public class EfsService implements Resettable {
|
||||
if (mt == null) {
|
||||
throw EfsException.mountTargetNotFound(mountTargetId);
|
||||
}
|
||||
|
||||
if (nfsServers != null) {
|
||||
nfsServers.stop(mountTargetId);
|
||||
}
|
||||
releaseNetworkInterface(region, mt);
|
||||
|
||||
try {
|
||||
synchronized (lockFor(regionKey(region, mt.getFileSystemId()))) {
|
||||
FileSystem fs = getFileSystem(region, mt.getFileSystemId());
|
||||
@@ -499,6 +632,7 @@ public class EfsService implements Resettable {
|
||||
synchronized (lockFor(regionKey(region, request.getFileSystemId()))) {
|
||||
// Validate file system exists
|
||||
getFileSystem(region, request.getFileSystemId());
|
||||
validateAccessPoint(request);
|
||||
|
||||
for (AccessPointDescription existing : accessPointStore.scan(k -> k.startsWith(region + "::"))) {
|
||||
if (token.equals(existing.getClientToken())) {
|
||||
@@ -530,6 +664,31 @@ public class EfsService implements Resettable {
|
||||
}
|
||||
}
|
||||
|
||||
private static void validateAccessPoint(CreateAccessPointRequest request) {
|
||||
PosixUser user = request.getPosixUser();
|
||||
if (user != null && (user.getUid() == null || user.getGid() == null)) {
|
||||
throw EfsException.badRequest("PosixUser requires both Uid and Gid");
|
||||
}
|
||||
RootDirectory root = request.getRootDirectory();
|
||||
if (root == null) {
|
||||
return;
|
||||
}
|
||||
if (root.getPath() != null && (root.getPath().length() > 100
|
||||
|| !ROOT_DIRECTORY_PATH.matcher(root.getPath()).matches())) {
|
||||
throw EfsException.badRequest("RootDirectory.Path is not a valid directory path: " + root.getPath());
|
||||
}
|
||||
CreationInfo info = root.getCreationInfo();
|
||||
if (info != null) {
|
||||
if (info.getOwnerUid() == null || info.getOwnerGid() == null || info.getPermissions() == null) {
|
||||
throw EfsException.badRequest("CreationInfo requires OwnerUid, OwnerGid and Permissions");
|
||||
}
|
||||
if (!info.getPermissions().matches("^[0-7]{3,4}$")) {
|
||||
throw EfsException.badRequest("CreationInfo.Permissions must be 3 or 4 octal digits: "
|
||||
+ info.getPermissions());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public List<AccessPointDescription> describeAccessPoints(String region, String fileSystemId, String accessPointId) {
|
||||
return accessPointStore.scan(k -> k.startsWith(region + "::")).stream()
|
||||
.filter(ap -> fileSystemId == null || ap.getFileSystemId().equals(fileSystemId))
|
||||
@@ -613,6 +772,30 @@ public class EfsService implements Resettable {
|
||||
return region + "::" + id;
|
||||
}
|
||||
|
||||
private Subnet requireSubnet(String region, String subnetId) {
|
||||
if (subnetId == null || subnetId.isBlank()) {
|
||||
throw EfsException.badRequest("SubnetId is required.");
|
||||
}
|
||||
try {
|
||||
return ec2Service.requireSubnet(region, subnetId);
|
||||
} catch (AwsException e) {
|
||||
throw EfsException.subnetNotFound(subnetId);
|
||||
}
|
||||
}
|
||||
|
||||
private void releaseNetworkInterface(String region, MountTarget mt) {
|
||||
if (mt.getNetworkInterfaceId() == null) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
ec2Service.releaseNetworkInterfaceFromService(region, mt.getNetworkInterfaceId());
|
||||
ec2Service.deleteNetworkInterface(region, mt.getNetworkInterfaceId());
|
||||
} catch (AwsException e) {
|
||||
LOG.debugv("Network interface {0} of mount target {1} is already gone: {2}",
|
||||
mt.getNetworkInterfaceId(), mt.getMountTargetId(), e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private String generateAzId(String azName) {
|
||||
if (azName == null || azName.length() < 2) return azName + "-id";
|
||||
String region = azName.substring(0, azName.length() - 1);
|
||||
|
||||
@@ -243,6 +243,9 @@ public class GlueService {
|
||||
}
|
||||
database.setName(databaseName);
|
||||
database.setCatalogId(regionResolver.getAccountId());
|
||||
if (database.getCreateTime() == null) {
|
||||
database.setCreateTime(Instant.now());
|
||||
}
|
||||
databaseStore.put(databaseName, database);
|
||||
if (tags != null && !tags.isEmpty()) {
|
||||
resourceGroupsTaggingService.tagResources(List.of(databaseArn(region, databaseName)), tags, region);
|
||||
|
||||
@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.glue.model;
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
|
||||
import java.time.Instant;
|
||||
@@ -47,10 +48,12 @@ public class Connection {
|
||||
|
||||
@JsonProperty("CreationTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant creationTime;
|
||||
|
||||
@JsonProperty("LastUpdatedTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant lastUpdatedTime;
|
||||
|
||||
@JsonProperty("LastUpdatedBy")
|
||||
@@ -64,6 +67,7 @@ public class Connection {
|
||||
|
||||
@JsonProperty("LastConnectionValidationTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant lastConnectionValidationTime;
|
||||
|
||||
@JsonProperty("AuthenticationConfiguration")
|
||||
|
||||
@@ -1,12 +1,15 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class Crawler {
|
||||
@JsonProperty("Classifiers")
|
||||
private List<String> classifiers;
|
||||
@@ -22,6 +25,7 @@ public class Crawler {
|
||||
|
||||
@JsonProperty("CreationTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant creationTime;
|
||||
|
||||
@JsonProperty("DatabaseName")
|
||||
@@ -38,6 +42,7 @@ public class Crawler {
|
||||
|
||||
@JsonProperty("LastUpdated")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant lastUpdated;
|
||||
|
||||
@JsonProperty("LineageConfiguration")
|
||||
|
||||
@@ -1,12 +1,15 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class Database {
|
||||
@JsonProperty("Name")
|
||||
private String name;
|
||||
@@ -18,6 +21,7 @@ public class Database {
|
||||
private Map<String, String> parameters;
|
||||
@JsonProperty("CreateTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant createTime;
|
||||
@JsonProperty("CatalogId")
|
||||
private String catalogId;
|
||||
|
||||
@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.glue.model;
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
|
||||
import java.time.Instant;
|
||||
@@ -22,10 +23,12 @@ public class GluePolicy {
|
||||
|
||||
@JsonProperty("CreateTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant createTime;
|
||||
|
||||
@JsonProperty("UpdateTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant updateTime;
|
||||
|
||||
public String getPolicyInJson() { return policyInJson; }
|
||||
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonGenerator;
|
||||
import com.fasterxml.jackson.databind.JsonSerializer;
|
||||
import com.fasterxml.jackson.databind.SerializerProvider;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.math.BigDecimal;
|
||||
import java.time.Instant;
|
||||
|
||||
/**
|
||||
* Writes a Glue timestamp as AWS does: epoch seconds to the millisecond, in the shortest
|
||||
* form ({@code 1792964720.27}, not {@code 1792964720.270000000}). The AWS SDK for
|
||||
* JavaScript v3 keeps any number whose text differs from its shortest form as a
|
||||
* BigDecimal, and then refuses it as a timestamp.
|
||||
*/
|
||||
@RegisterForReflection
|
||||
public class GlueTimestampSerializer extends JsonSerializer<Instant> {
|
||||
|
||||
@Override
|
||||
public void serialize(Instant value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
|
||||
gen.writeNumber(epochSeconds(value));
|
||||
}
|
||||
|
||||
static String epochSeconds(Instant value) {
|
||||
return BigDecimal.valueOf(value.toEpochMilli(), 3).stripTrailingZeros().toPlainString();
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,9 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
|
||||
@@ -9,6 +11,7 @@ import java.time.Instant;
|
||||
import java.util.Map;
|
||||
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class Job {
|
||||
@JsonProperty("AllocatedCapacity")
|
||||
private Integer allocatedCapacity;
|
||||
@@ -24,6 +27,7 @@ public class Job {
|
||||
|
||||
@JsonProperty("CreatedOn")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant createdOn;
|
||||
|
||||
@JsonProperty("DefaultArguments")
|
||||
@@ -49,6 +53,7 @@ public class Job {
|
||||
|
||||
@JsonProperty("LastModifiedOn")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant lastModifiedOn;
|
||||
|
||||
@JsonProperty("LogUri")
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
import java.time.Instant;
|
||||
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class LastCrawlInfo {
|
||||
@JsonProperty("ErrorMessage")
|
||||
private String errorMessage;
|
||||
@@ -21,6 +24,7 @@ public class LastCrawlInfo {
|
||||
|
||||
@JsonProperty("StartTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant startTime;
|
||||
|
||||
@JsonProperty("Status")
|
||||
|
||||
@@ -1,13 +1,16 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class Partition {
|
||||
@JsonProperty("Values")
|
||||
private List<String> values;
|
||||
@@ -17,9 +20,11 @@ public class Partition {
|
||||
private String tableName;
|
||||
@JsonProperty("CreationTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant creationTime;
|
||||
@JsonProperty("LastAccessTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant lastAccessTime;
|
||||
@JsonProperty("StorageDescriptor")
|
||||
private StorageDescriptor storageDescriptor;
|
||||
|
||||
@@ -1,13 +1,16 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class Table {
|
||||
@JsonProperty("Name")
|
||||
private String name;
|
||||
@@ -19,12 +22,15 @@ public class Table {
|
||||
private String owner;
|
||||
@JsonProperty("CreateTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant createTime;
|
||||
@JsonProperty("UpdateTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant updateTime;
|
||||
@JsonProperty("LastAccessTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant lastAccessTime;
|
||||
@JsonProperty("PartitionKeys")
|
||||
private List<Column> partitionKeys;
|
||||
|
||||
@@ -1,13 +1,16 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonFormat;
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
|
||||
import io.quarkus.runtime.annotations.RegisterForReflection;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
@RegisterForReflection
|
||||
@JsonInclude(JsonInclude.Include.NON_NULL)
|
||||
public class UserDefinedFunction {
|
||||
@JsonProperty("FunctionName")
|
||||
private String functionName;
|
||||
@@ -23,6 +26,7 @@ public class UserDefinedFunction {
|
||||
private String ownerType;
|
||||
@JsonProperty("CreateTime")
|
||||
@JsonFormat(shape = JsonFormat.Shape.NUMBER)
|
||||
@JsonSerialize(using = GlueTimestampSerializer.class)
|
||||
private Instant createTime;
|
||||
@JsonProperty("ResourceUris")
|
||||
private List<ResourceUri> resourceUris;
|
||||
|
||||
+28
-39
@@ -10,6 +10,8 @@ import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
|
||||
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
|
||||
import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
|
||||
import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;
|
||||
import io.github.hectorvent.floci.services.efs.EfsMounts;
|
||||
import io.github.hectorvent.floci.services.efs.model.PosixUser;
|
||||
import io.github.hectorvent.floci.services.iam.model.SessionCreds;
|
||||
import io.github.hectorvent.floci.services.lambda.LambdaLayerService;
|
||||
import io.github.hectorvent.floci.services.lambda.model.ContainerState;
|
||||
@@ -36,17 +38,13 @@ import java.io.Closeable;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.AtomicMoveNotSupportedException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardCopyOption;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.LocalDate;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.CancellationException;
|
||||
@@ -110,6 +108,7 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
|
||||
private final LambdaLayerService layerService;
|
||||
private final LaunchedContainerAwsEnv awsEnv;
|
||||
private final LambdaExecutionRoleCredentials executionRoleCredentials;
|
||||
private final EfsMounts efsMounts;
|
||||
|
||||
@Inject
|
||||
public ContainerLauncher(ContainerBuilder containerBuilder,
|
||||
@@ -122,7 +121,8 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
|
||||
EcrRegistryManager ecrRegistryManager,
|
||||
LambdaLayerService layerService,
|
||||
LaunchedContainerAwsEnv awsEnv,
|
||||
LambdaExecutionRoleCredentials executionRoleCredentials) {
|
||||
LambdaExecutionRoleCredentials executionRoleCredentials,
|
||||
EfsMounts efsMounts) {
|
||||
this.containerBuilder = containerBuilder;
|
||||
this.lifecycleManager = lifecycleManager;
|
||||
this.logStreamer = logStreamer;
|
||||
@@ -134,6 +134,7 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
|
||||
this.layerService = layerService;
|
||||
this.awsEnv = awsEnv;
|
||||
this.executionRoleCredentials = executionRoleCredentials;
|
||||
this.efsMounts = efsMounts;
|
||||
this.populateSemaphore = new java.util.concurrent.Semaphore(resolvePopulateConcurrency(config));
|
||||
}
|
||||
|
||||
@@ -310,21 +311,28 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
|
||||
|
||||
if (fn.getFileSystemConfigs() != null && !fn.getFileSystemConfigs().isEmpty()) {
|
||||
var efsCfg = config.storage().efs();
|
||||
fn.getFileSystemConfigs().forEach(fileSystem -> {
|
||||
String volumeName = efsVolumeName(fileSystem.getArn());
|
||||
lifecycleManager.ensureSharedVolume(volumeName,
|
||||
efsCfg.ownerUid(), efsCfg.ownerGid(), efsCfg.rootPermissions(),
|
||||
efsCfg.initImage());
|
||||
specBuilder.withNamedVolume(volumeName, fileSystem.getLocalMountPath(), false);
|
||||
});
|
||||
efsCfg.mountUser().ifPresent(user -> {
|
||||
if (!user.matches("^\\d+(:\\d+)?$")) {
|
||||
throw new IllegalArgumentException(
|
||||
"floci.storage.efs.mount-user must be \"uid\" or \"uid:gid\": " + user);
|
||||
}
|
||||
specBuilder.withUser(user);
|
||||
});
|
||||
efsCfg.mountGroupAdd().ifPresent(gid -> specBuilder.withGroupAdd(String.valueOf(gid)));
|
||||
// Lambda allows one file system per function. Running as the access point's POSIX user
|
||||
// is how its identity reaches every file the function creates, as EFS enforces it.
|
||||
Optional<PosixUser> accessPointUser = Optional.empty();
|
||||
for (var fileSystem : fn.getFileSystemConfigs()) {
|
||||
EfsMounts.Mount mount = efsMounts.prepareAccessPoint(fileSystem.getArn());
|
||||
mount.applyTo(specBuilder, fileSystem.getLocalMountPath(), false);
|
||||
accessPointUser = accessPointUser.or(mount::accessPointUser);
|
||||
}
|
||||
if (accessPointUser.isPresent()) {
|
||||
specBuilder.withUser(EfsMounts.containerUser(accessPointUser.get()));
|
||||
EfsMounts.secondaryGids(accessPointUser.get())
|
||||
.forEach(gid -> specBuilder.withGroupAdd(String.valueOf(gid)));
|
||||
} else {
|
||||
efsCfg.mountUser().ifPresent(user -> {
|
||||
if (!user.matches("^\\d+(:\\d+)?$")) {
|
||||
throw new IllegalArgumentException(
|
||||
"floci.storage.efs.mount-user must be \"uid\" or \"uid:gid\": " + user);
|
||||
}
|
||||
specBuilder.withUser(user);
|
||||
});
|
||||
efsCfg.mountGroupAdd().ifPresent(gid -> specBuilder.withGroupAdd(String.valueOf(gid)));
|
||||
}
|
||||
}
|
||||
|
||||
// For Image package type use ImageConfig.Command/EntryPoint/WorkingDirectory if set, otherwise fall back to Handler (Zip-style)
|
||||
@@ -1133,25 +1141,6 @@ public class ContainerLauncher implements LambdaRuntimeLauncher {
|
||||
return namePrefix + "-code-" + fname + "-" + h;
|
||||
}
|
||||
|
||||
static String efsVolumeName(String accessPointArn) {
|
||||
int separator = Math.max(accessPointArn.lastIndexOf('/'), accessPointArn.lastIndexOf(':'));
|
||||
String resourceId = separator >= 0 ? accessPointArn.substring(separator + 1) : accessPointArn;
|
||||
if (resourceId.isBlank()) {
|
||||
throw new IllegalArgumentException("File system access point ARN must include a resource id");
|
||||
}
|
||||
return "floci-efs-" + resourceId + "-" + sha256Hex(accessPointArn);
|
||||
}
|
||||
|
||||
private static String sha256Hex(String value) {
|
||||
try {
|
||||
byte[] digest = MessageDigest.getInstance("SHA-256")
|
||||
.digest(value.getBytes(StandardCharsets.UTF_8));
|
||||
return HexFormat.of().formatHex(digest);
|
||||
} catch (NoSuchAlgorithmException e) {
|
||||
throw new IllegalStateException("SHA-256 is required but not available", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Buffer for the tar-streaming pipe. The default {@link java.io.PipedInputStream} buffer is
|
||||
* only 1KB, which forces a writer/reader thread hand-off (wait/notify) every 1KB. Streaming a
|
||||
|
||||
@@ -57,6 +57,8 @@ public class SmtpRelay {
|
||||
|
||||
private static final Logger LOG = Logger.getLogger(SmtpRelay.class);
|
||||
private static final long RELAY_TIMEOUT_SECONDS = 30;
|
||||
/** Every SASL method the mail client supports except XOAUTH2, which expects a token. */
|
||||
static final String PASSWORD_AUTH_METHODS = "CRAM-SHA256 CRAM-SHA1 CRAM-MD5 DIGEST-MD5 LOGIN PLAIN";
|
||||
|
||||
/** SES control header naming the configuration set to apply to a raw send. */
|
||||
public static final String HEADER_CONFIGURATION_SET = "X-SES-CONFIGURATION-SET";
|
||||
@@ -118,6 +120,9 @@ public class SmtpRelay {
|
||||
.filter(u -> !u.isBlank())
|
||||
.ifPresent(u -> {
|
||||
mailConfig.setUsername(u);
|
||||
// smtp-pass is a password, never an OAuth token: without this the client prefers
|
||||
// XOAUTH2 whenever the server offers it (Gmail does) and the login is refused.
|
||||
mailConfig.setAuthMethods(PASSWORD_AUTH_METHODS);
|
||||
config.services().ses().smtpPass()
|
||||
.filter(p -> !p.isBlank())
|
||||
.ifPresent(mailConfig::setPassword);
|
||||
|
||||
@@ -747,6 +747,8 @@ floci:
|
||||
enabled: true
|
||||
efs:
|
||||
enabled: true
|
||||
nfs-servers: true # serve each mount target over NFSv4.1 for EC2 instances
|
||||
nfs-server-image: floci/efs-nfs-server:local # built locally on first use
|
||||
organizations:
|
||||
enabled: true
|
||||
# management-account-email: "" # When set, overrides the management-account email reported by DescribeOrganization and account listings.
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
FROM alpine:3.21
|
||||
RUN apk add --no-cache nfs-utils
|
||||
# Serves /export, the EFS file system's volume, over NFSv4.1 from the kernel NFS server, as an EFS
|
||||
# mount target does. fsid=0 makes it the NFSv4 root, so clients mount "<address>:/" like on AWS.
|
||||
CMD ["sh", "-c", "set -e; mount -t nfsd nfsd /proc/fs/nfsd; echo '/export *(rw,fsid=0,no_subtree_check,no_root_squash,insecure,sync)' > /etc/exports; exportfs -ra; rpc.nfsd -N 3 -V 4 -V 4.1 -V 4.2 8; trap 'rpc.nfsd 0; exportfs -ua; exit 0' TERM INT; rpc.mountd -N 2 -N 3 -F & wait"]
|
||||
@@ -103,6 +103,18 @@ class EmbeddedDnsServerTest {
|
||||
dns.resolveARecord("ip-172-16-128-9.ec2.internal", "172.16.128.5").orElseThrow());
|
||||
}
|
||||
|
||||
@Test
|
||||
void resolveARecord_asksAddedResolversForOtherNames() {
|
||||
dns.addResolver(name -> name.equals("fs-1.efs.us-east-1.amazonaws.com")
|
||||
? java.util.Optional.of("172.31.0.10") : java.util.Optional.empty());
|
||||
dns.addResolver(name -> { throw new IllegalStateException("broken resolver"); });
|
||||
|
||||
assertEquals("172.31.0.10",
|
||||
dns.resolveARecord("fs-1.efs.us-east-1.amazonaws.com", "172.16.128.5").orElseThrow());
|
||||
assertTrue(dns.resolveARecord("example.com", "172.16.128.5").isEmpty(),
|
||||
"a name no resolver knows is forwarded upstream, even when a resolver fails");
|
||||
}
|
||||
|
||||
// ── matchesSuffix — built-in emulator domains ─────────────────────────────
|
||||
|
||||
@Test
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
package io.github.hectorvent.floci.services.acm;
|
||||
|
||||
import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
|
||||
import io.quarkus.test.junit.QuarkusTest;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static io.restassured.RestAssured.given;
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.notNullValue;
|
||||
|
||||
@QuarkusTest
|
||||
class AcmExportableRequestTest {
|
||||
|
||||
private static final String ACM_CONTENT_TYPE = "application/x-amz-json-1.1";
|
||||
|
||||
@BeforeAll
|
||||
static void configureRestAssured() {
|
||||
RestAssuredJsonUtils.configureAwsContentTypes();
|
||||
}
|
||||
|
||||
@Test
|
||||
void requestCertificateKeepsTheExportOption() {
|
||||
String exportable = request("{\"DomainName\":\"exportable.example.com\",\"Options\":{\"Export\":\"ENABLED\"}}");
|
||||
|
||||
call("DescribeCertificate", "{\"CertificateArn\":\"" + exportable + "\"}")
|
||||
.statusCode(200)
|
||||
.body("Certificate.Options.Export", equalTo("ENABLED"))
|
||||
.body("Certificate.Options.CertificateTransparencyLoggingPreference", equalTo("ENABLED"));
|
||||
|
||||
call("ExportCertificate", "{\"CertificateArn\":\"" + exportable + "\",\"Passphrase\":\"c2VjcmV0\"}")
|
||||
.statusCode(200)
|
||||
.body("PrivateKey", notNullValue());
|
||||
}
|
||||
|
||||
@Test
|
||||
void requestCertificateWithoutExportIsNotExportable() {
|
||||
String plain = request("{\"DomainName\":\"plain.example.com\",\"Options\":{\"CertificateTransparencyLoggingPreference\":\"DISABLED\"}}");
|
||||
|
||||
call("DescribeCertificate", "{\"CertificateArn\":\"" + plain + "\"}")
|
||||
.statusCode(200)
|
||||
.body("Certificate.Options.Export", equalTo("DISABLED"))
|
||||
.body("Certificate.Options.CertificateTransparencyLoggingPreference", equalTo("DISABLED"));
|
||||
|
||||
call("ExportCertificate", "{\"CertificateArn\":\"" + plain + "\",\"Passphrase\":\"c2VjcmV0\"}")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("ValidationException"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void requestCertificateRejectsAnUnknownExportValue() {
|
||||
call("RequestCertificate", "{\"DomainName\":\"bad.example.com\",\"Options\":{\"Export\":\"MAYBE\"}}")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("ValidationException"));
|
||||
}
|
||||
|
||||
private String request(String body) {
|
||||
return call("RequestCertificate", body).statusCode(200).extract().jsonPath().getString("CertificateArn");
|
||||
}
|
||||
|
||||
private io.restassured.response.ValidatableResponse call(String action, String body) {
|
||||
return given()
|
||||
.header("X-Amz-Target", "CertificateManager." + action)
|
||||
.contentType(ACM_CONTENT_TYPE)
|
||||
.body(body)
|
||||
.when()
|
||||
.post("/")
|
||||
.then();
|
||||
}
|
||||
}
|
||||
+20
@@ -98,6 +98,26 @@ class JwtSignatureVerifierTest {
|
||||
verifier.verify(token, issuer); // does not throw
|
||||
}
|
||||
|
||||
@Test
|
||||
void verifiesAnInProcessIssuerWithoutFetchingIt() throws Exception {
|
||||
// Floci's own Cognito issuer: plain HTTP on a hostname, which the strict fetch rules refuse.
|
||||
String localIssuer = "http://floci:4566/us-east-1_local";
|
||||
strictVerifier.withLocalIssuerKeys(candidate -> candidate.equals(localIssuer)
|
||||
? java.util.Optional.of(Map.of("test-key-1", publicKey)) : java.util.Optional.empty());
|
||||
|
||||
assertDoesNotThrow(() -> strictVerifier.verify(signToken("test-key-1", privateKey), localIssuer));
|
||||
assertThrows(JwtSignatureVerifier.JwtVerificationException.class,
|
||||
() -> strictVerifier.verify(signToken("other-key", privateKey), localIssuer));
|
||||
KeyPairGenerator gen = KeyPairGenerator.getInstance("RSA");
|
||||
gen.initialize(2048);
|
||||
String forged = signToken("test-key-1", (RSAPrivateKey) gen.generateKeyPair().getPrivate());
|
||||
assertThrows(JwtSignatureVerifier.JwtVerificationException.class,
|
||||
() -> strictVerifier.verify(forged, localIssuer));
|
||||
// Any other plain-HTTP issuer still goes through the strict fetch, and is refused.
|
||||
assertThrows(JwtSignatureVerifier.JwtVerificationException.class,
|
||||
() -> strictVerifier.verify(signToken("test-key-1", privateKey), "http://floci:4566/us-east-1_other"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void rejectsATokenSignedWithADifferentKey() throws Exception {
|
||||
KeyPairGenerator gen = KeyPairGenerator.getInstance("RSA");
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
package io.github.hectorvent.floci.services.athena;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
class AthenaQueryTextTest {
|
||||
|
||||
@Test
|
||||
void dropsTheClosingSemicolonsAthenaAccepts() {
|
||||
assertEquals("SELECT * FROM orders LIMIT 10", AthenaService.withoutTrailingSemicolons("SELECT * FROM orders LIMIT 10;"));
|
||||
assertEquals("SELECT 1", AthenaService.withoutTrailingSemicolons("SELECT 1 ; ;\n"));
|
||||
assertEquals("SELECT ';' AS s", AthenaService.withoutTrailingSemicolons("SELECT ';' AS s"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void reportsTheEngineErrorWithoutItsEnvelopeOrTheWrappedStatement() {
|
||||
String engine = "floci-duck execute returned HTTP 500: {\"status\":\"error\",\"message\":\"Catalog Error: Table with name nope does not exist!\\nDid you mean \\\"orders\\\"?\\n\\nLINE 1: COPY (SELECT * FROM nope) TO 's3://results/q.csv'\"}";
|
||||
assertEquals("Catalog Error: Table with name nope does not exist!\nDid you mean \"orders\"?", AthenaService.failureReason(engine));
|
||||
assertEquals("Failed to call floci-duck execute: timeout", AthenaService.failureReason("Failed to call floci-duck execute: timeout"));
|
||||
}
|
||||
}
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
package io.github.hectorvent.floci.services.athena;
|
||||
|
||||
import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
|
||||
import io.quarkus.test.junit.QuarkusTest;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
import static io.restassured.RestAssured.given;
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.hasItem;
|
||||
import static org.hamcrest.Matchers.not;
|
||||
import static org.hamcrest.Matchers.startsWith;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
/** Queries belong to their workgroup: they use its result location and are listed under it. */
|
||||
@QuarkusTest
|
||||
class AthenaWorkGroupQueriesIntegrationTest {
|
||||
|
||||
private static final String CONTENT_TYPE = "application/x-amz-json-1.1";
|
||||
|
||||
@BeforeAll
|
||||
static void configureRestAssured() {
|
||||
RestAssuredJsonUtils.configureAwsContentTypes();
|
||||
}
|
||||
|
||||
private static io.restassured.response.ValidatableResponse athena(String action, String body) {
|
||||
return given()
|
||||
.header("X-Amz-Target", "AmazonAthena." + action)
|
||||
.contentType(CONTENT_TYPE)
|
||||
.body(body)
|
||||
.when().post("/")
|
||||
.then();
|
||||
}
|
||||
|
||||
private static String workGroup(String location, boolean enforce) {
|
||||
String name = "wg-" + UUID.randomUUID().toString().substring(0, 8);
|
||||
athena("CreateWorkGroup", """
|
||||
{"Name": "%s", "Configuration": {"ResultConfiguration": {"OutputLocation": "%s"}, "EnforceWorkGroupConfiguration": %s}}
|
||||
""".formatted(name, location, enforce)).statusCode(200);
|
||||
return name;
|
||||
}
|
||||
|
||||
private static String start(String workGroup, String outputLocation) {
|
||||
String result = outputLocation == null ? "" : ", \"ResultConfiguration\": {\"OutputLocation\": \"" + outputLocation + "\"}";
|
||||
return athena("StartQueryExecution", "{\"QueryString\": \"SELECT 1\", \"WorkGroup\": \"" + workGroup + "\"" + result + "}")
|
||||
.statusCode(200)
|
||||
.extract().jsonPath().getString("QueryExecutionId");
|
||||
}
|
||||
|
||||
private static String outputLocation(String queryId) {
|
||||
return athena("GetQueryExecution", "{\"QueryExecutionId\": \"" + queryId + "\"}")
|
||||
.statusCode(200)
|
||||
.body("QueryExecution.WorkGroup", not(equalTo(null)))
|
||||
.extract().jsonPath().getString("QueryExecution.ResultConfiguration.OutputLocation");
|
||||
}
|
||||
|
||||
@Test
|
||||
void usesTheWorkGroupResultLocationWhenTheQueryNamesNone() {
|
||||
String name = workGroup("s3://team-results/athena/", false);
|
||||
String queryId = start(name, null);
|
||||
|
||||
assertEquals("s3://team-results/athena/" + queryId + ".csv", outputLocation(queryId));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aRequestedLocationWinsUnlessTheWorkGroupEnforcesItsOwn() {
|
||||
String open = workGroup("s3://team-results/open/", false);
|
||||
String openQuery = start(open, "s3://mine/results/");
|
||||
assertEquals("s3://mine/results/" + openQuery + ".csv", outputLocation(openQuery));
|
||||
|
||||
String enforced = workGroup("s3://team-results/enforced/", true);
|
||||
String enforcedQuery = start(enforced, "s3://mine/results/");
|
||||
assertEquals("s3://team-results/enforced/" + enforcedQuery + ".csv", outputLocation(enforcedQuery));
|
||||
}
|
||||
|
||||
@Test
|
||||
void refusesAWorkGroupThatDoesNotExist() {
|
||||
athena("StartQueryExecution", "{\"QueryString\": \"SELECT 1\", \"WorkGroup\": \"no-such-group\"}")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("InvalidRequestException"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void listsOnlyTheWorkGroupQueriesNewestFirst() {
|
||||
String name = workGroup("s3://team-results/listed/", false);
|
||||
String first = start(name, null);
|
||||
String second = start(name, null);
|
||||
String elsewhere = start("primary", null);
|
||||
|
||||
List<String> listed = athena("ListQueryExecutions", "{\"WorkGroup\": \"" + name + "\"}")
|
||||
.statusCode(200)
|
||||
.extract().jsonPath().getList("QueryExecutionIds", String.class);
|
||||
assertEquals(List.of(second, first), listed);
|
||||
|
||||
athena("ListQueryExecutions", "{}")
|
||||
.statusCode(200)
|
||||
.body("QueryExecutionIds", hasItem(elsewhere))
|
||||
.body("QueryExecutionIds", not(hasItem(first)));
|
||||
}
|
||||
|
||||
@Test
|
||||
void thePrimaryWorkGroupKeepsItsDefaultLocation() {
|
||||
String queryId = start("primary", null);
|
||||
org.hamcrest.MatcherAssert.assertThat(outputLocation(queryId), startsWith("s3://floci-athena-results/results/"));
|
||||
}
|
||||
}
|
||||
@@ -373,6 +373,63 @@ class CognitoServiceTest {
|
||||
null, null, List.of(), null, null);
|
||||
}
|
||||
|
||||
@Nested
|
||||
class PreventUserExistenceErrors {
|
||||
|
||||
private UserPoolClient client(UserPool pool, String preventUserExistenceErrors) {
|
||||
return service.createUserPoolClient(pool.getId(), "existence-client", false, false, List.of(), List.of(),
|
||||
null, List.of(), null, List.of("ALLOW_USER_PASSWORD_AUTH"), null, null, List.of(),
|
||||
preventUserExistenceErrors, List.of(), null, null, null, List.of(), null, null);
|
||||
}
|
||||
|
||||
@Test
|
||||
void enabledAnswersAnUnknownUserLikeAWrongPassword() {
|
||||
UserPool pool = createPoolAndUser();
|
||||
UserPoolClient client = client(pool, "ENABLED");
|
||||
|
||||
AwsException unknown = assertThrows(AwsException.class, () -> service.initiateAuth(client.getClientId(),
|
||||
"USER_PASSWORD_AUTH", Map.of("USERNAME", "nobody", "PASSWORD", "Perm1234!")));
|
||||
AwsException wrongPassword = assertThrows(AwsException.class, () -> service.initiateAuth(client.getClientId(),
|
||||
"USER_PASSWORD_AUTH", Map.of("USERNAME", "alice", "PASSWORD", "Wrong1234!")));
|
||||
|
||||
assertEquals("NotAuthorizedException", unknown.getErrorCode());
|
||||
assertEquals(wrongPassword.getErrorCode(), unknown.getErrorCode());
|
||||
assertEquals(wrongPassword.getMessage(), unknown.getMessage());
|
||||
}
|
||||
|
||||
@Test
|
||||
void enabledSimulatesCodeDeliveryAndRejectsCodesForAnUnknownUser() {
|
||||
UserPool pool = createPoolAndUser();
|
||||
UserPoolClient client = client(pool, "ENABLED");
|
||||
|
||||
assertEquals(Map.of("AttributeName", "email", "DeliveryMedium", "EMAIL", "Destination", "n***@e***"),
|
||||
service.forgotPassword(client.getClientId(), "nobody@example.com"));
|
||||
assertEquals("EMAIL", service.resendConfirmationCode(client.getClientId(), "nobody@example.com")
|
||||
.get("DeliveryMedium"));
|
||||
assertEquals("CodeMismatchException", assertThrows(AwsException.class, () ->
|
||||
service.confirmForgotPassword(client.getClientId(), "nobody@example.com", "123456", "Perm1234!"))
|
||||
.getErrorCode());
|
||||
assertEquals("CodeMismatchException", assertThrows(AwsException.class, () ->
|
||||
service.confirmSignUp(client.getClientId(), "nobody@example.com", "123456")).getErrorCode());
|
||||
assertThrows(AwsException.class, () -> service.adminGetUser(pool.getId(), "nobody@example.com"),
|
||||
"no user is created along the way");
|
||||
}
|
||||
|
||||
@Test
|
||||
void legacyAndUnsetStillReportUserNotFound() {
|
||||
UserPool pool = createPoolAndUser();
|
||||
for (String setting : new String[] {"LEGACY", null}) {
|
||||
UserPoolClient client = client(pool, setting);
|
||||
|
||||
assertEquals("UserNotFoundException", assertThrows(AwsException.class, () -> service.initiateAuth(
|
||||
client.getClientId(), "USER_PASSWORD_AUTH", Map.of("USERNAME", "nobody", "PASSWORD", "x")))
|
||||
.getErrorCode());
|
||||
assertEquals("UserNotFoundException", assertThrows(AwsException.class, () ->
|
||||
service.forgotPassword(client.getClientId(), "nobody")).getErrorCode());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@CsvSource({
|
||||
"Short1!a",
|
||||
@@ -1312,6 +1369,22 @@ class CognitoServiceTest {
|
||||
assertEquals("http://localhost:4566/custompool", service.getIssuer(pool.getId()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void localIssuerKeysServeOnlyThisEmulatorsOwnPoolIssuers() throws Exception {
|
||||
UserPool pool = createPoolAndUser();
|
||||
UserPoolClient client = clientWithFlows(pool, "ALLOW_USER_PASSWORD_AUTH");
|
||||
String token = (String) ((Map<?, ?>) service.initiateAuth(client.getClientId(), "USER_PASSWORD_AUTH",
|
||||
Map.of("USERNAME", "alice", "PASSWORD", "Perm1234!")).get("AuthenticationResult")).get("AccessToken");
|
||||
String kid = MAPPER.readTree(Base64.getUrlDecoder().decode(token.split("\\.")[0])).path("kid").asText();
|
||||
|
||||
Map<String, java.security.interfaces.RSAPublicKey> keys = service.localIssuerKeys(service.getIssuer(pool.getId())).orElseThrow();
|
||||
assertTrue(keys.containsKey(kid), "the key that signed the pool's tokens");
|
||||
assertTrue(service.localIssuerKeys(service.getIssuer(pool.getId()) + "/").isPresent());
|
||||
assertTrue(service.localIssuerKeys("https://cognito-idp.us-east-1.amazonaws.com/" + pool.getId()).isEmpty());
|
||||
assertTrue(service.localIssuerKeys(service.getIssuer("us-east-1_missing")).isEmpty());
|
||||
assertTrue(service.localIssuerKeys(null).isEmpty());
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Groups
|
||||
// =========================================================================
|
||||
|
||||
@@ -284,6 +284,23 @@ class VpcNetworkManagerTest {
|
||||
assertTrue(second.containsAddress(Cidr4.parse(b + "/32").orElseThrow().network()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSubstitutedVpcKeepsEachSubnetAtItsOffsetWhateverTheDeclarationOrder() {
|
||||
existingNetwork("someone-else", "172.31.0.0/16", Map.of());
|
||||
|
||||
manager.declareVpc(REGION, "vpc-default", "172.31.0.0/16");
|
||||
manager.declareSubnet(REGION, "vpc-default", "subnet-c", "172.31.32.0/20");
|
||||
manager.declareSubnet(REGION, "vpc-default", "subnet-a", "172.31.0.0/20");
|
||||
manager.declareSubnet(REGION, "vpc-default", "subnet-b", "172.31.16.0/20");
|
||||
|
||||
assertTrue(manager.isSubstituted(REGION, "vpc-default"));
|
||||
assertEquals("10.240.0.0/16", manager.effectiveVpcCidr(REGION, "vpc-default").orElseThrow());
|
||||
assertEquals("10.240.0.0/20", manager.effectiveSubnetCidr(REGION, "subnet-a").orElseThrow());
|
||||
assertEquals("10.240.16.0/20", manager.effectiveSubnetCidr(REGION, "subnet-b").orElseThrow());
|
||||
assertEquals("10.240.32.0/20", manager.effectiveSubnetCidr(REGION, "subnet-c").orElseThrow(),
|
||||
"a restart that declares subnets in another order must reach the same plan");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSubnetOutsideItsVpcRangeIsRemappedInsideIt() {
|
||||
manager.declareVpc(REGION, "vpc-1", "10.0.0.0/16");
|
||||
|
||||
-169
@@ -1,169 +0,0 @@
|
||||
package io.github.hectorvent.floci.services.ecs.container;
|
||||
|
||||
import com.github.dockerjava.api.DockerClient;
|
||||
import com.github.dockerjava.core.command.WaitContainerResultCallback;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
|
||||
import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
|
||||
import io.github.hectorvent.floci.services.ecs.model.EcsTask;
|
||||
import io.github.hectorvent.floci.services.ecs.model.EfsVolumeConfiguration;
|
||||
import io.github.hectorvent.floci.services.ecs.model.MountPoint;
|
||||
import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;
|
||||
import io.github.hectorvent.floci.services.ecs.model.Volume;
|
||||
import io.quarkus.test.junit.QuarkusTest;
|
||||
import jakarta.inject.Inject;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.Assumptions;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
/**
|
||||
* Docker-backed proof that an EFS-configured task volume's local mount is scoped to
|
||||
* {@code (fileSystemId, rootDirectory, accessPointId)}, not just {@code fileSystemId} (#2563).
|
||||
* Each scenario runs a real busybox container against the same {@code fileSystemId}, writing
|
||||
* and checking marker files under {@code /mnt/efs} with a shell one-liner whose exit code
|
||||
* proves whether the previous task's data was visible.
|
||||
*/
|
||||
@QuarkusTest
|
||||
class EcsContainerManagerEfsIsolationDockerIntegrationTest {
|
||||
|
||||
private static final String IMAGE = "public.ecr.aws/docker/library/busybox:latest";
|
||||
|
||||
@Inject
|
||||
EcsContainerManager containerManager;
|
||||
|
||||
@Inject
|
||||
ContainerLifecycleManager lifecycleManager;
|
||||
|
||||
@Inject
|
||||
DockerClient dockerClient;
|
||||
|
||||
private final List<String> volumesToCleanUp = new ArrayList<>();
|
||||
|
||||
@BeforeEach
|
||||
void requireDocker() {
|
||||
Assumptions.assumeTrue(isDockerAvailable(),
|
||||
"Docker daemon must be available for ECS EFS isolation integration tests");
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
void cleanUpVolumes() {
|
||||
volumesToCleanUp.forEach(lifecycleManager::removeVolume);
|
||||
volumesToCleanUp.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* Task A writes a marker under {@code rootDirectory} {@code /team-a}; task B, on the same
|
||||
* file system but a different root, must not see it (if the bug were still present, one
|
||||
* shared volume keyed only by {@code fileSystemId} would leak team-a's marker into team-b's
|
||||
* task before it ever reaches its own write); a third task back on {@code /team-a} must
|
||||
* likewise not see what team-b wrote.
|
||||
*/
|
||||
@Test
|
||||
void differentRootDirectoriesOnSameFileSystemAreIsolated() {
|
||||
String fileSystemId = "fs-" + UUID.randomUUID().toString().substring(0, 8);
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/team-a", null,
|
||||
"echo team-a > /mnt/efs/only-a.txt"),
|
||||
"task A should write its marker cleanly");
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/team-b", null,
|
||||
"test ! -e /mnt/efs/only-a.txt && echo team-b > /mnt/efs/only-b.txt"),
|
||||
"task B must not see task A's rootDirectory-scoped data");
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/team-a", null,
|
||||
"test ! -e /mnt/efs/only-b.txt"),
|
||||
"task on /team-a must not see /team-b's data");
|
||||
}
|
||||
|
||||
/**
|
||||
* A second task using the identical {@code rootDirectory} must see the first task's data:
|
||||
* sharing an identical configuration is existing, desired behaviour.
|
||||
*/
|
||||
@Test
|
||||
void sameRootDirectoryOnSameFileSystemStillShares() {
|
||||
String fileSystemId = "fs-" + UUID.randomUUID().toString().substring(0, 8);
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/shared", null,
|
||||
"echo hello > /mnt/efs/shared.txt"),
|
||||
"first task should write its marker cleanly");
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/shared", null,
|
||||
"grep -qx hello /mnt/efs/shared.txt"),
|
||||
"a second task with the same rootDirectory must share the first task's data");
|
||||
}
|
||||
|
||||
/**
|
||||
* A plain {@code rootDirectory} mount (no {@code accessPointId}) on the same file system
|
||||
* must not see access-point-scoped data and vice versa, while a second task through the
|
||||
* same access point must still share its data.
|
||||
*/
|
||||
@Test
|
||||
void accessPointIdIsolatesFromRootDirectoryOnSameFileSystem() {
|
||||
String fileSystemId = "fs-" + UUID.randomUUID().toString().substring(0, 8);
|
||||
String accessPointId = "fsap-" + UUID.randomUUID().toString().substring(0, 8);
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, null, accessPointId,
|
||||
"echo via-ap > /mnt/efs/via-ap.txt"),
|
||||
"task mounted through the access point should write its marker cleanly");
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/", null,
|
||||
"test ! -e /mnt/efs/via-ap.txt"),
|
||||
"a plain rootDirectory mount must not see access-point-scoped data");
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, null, accessPointId,
|
||||
"grep -qx via-ap /mnt/efs/via-ap.txt"),
|
||||
"a second task through the same access point must share its data");
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a single-container ECS task that mounts {@code fileSystemId} (scoped by
|
||||
* {@code rootDirectory}/{@code accessPointId}) at {@code /mnt/efs} and executes
|
||||
* {@code shellCommand}, then blocks until the container exits and returns its exit code.
|
||||
*/
|
||||
private int runEfsCommand(String fileSystemId, String rootDirectory, String accessPointId, String shellCommand) {
|
||||
String suffix = UUID.randomUUID().toString().substring(0, 8);
|
||||
|
||||
ContainerDefinition app = new ContainerDefinition();
|
||||
app.setName("app");
|
||||
app.setImage(IMAGE);
|
||||
app.setCommand(List.of("sh", "-c", shellCommand));
|
||||
app.setMountPoints(List.of(new MountPoint("efs-data", "/mnt/efs", false)));
|
||||
|
||||
TaskDefinition taskDef = new TaskDefinition();
|
||||
taskDef.setFamily("efs-isolation-" + suffix);
|
||||
taskDef.setContainerDefinitions(List.of(app));
|
||||
taskDef.setVolumes(List.of(new Volume("efs-data", null,
|
||||
new EfsVolumeConfiguration(fileSystemId, rootDirectory, null, null, accessPointId, null))));
|
||||
|
||||
EcsTask task = new EcsTask();
|
||||
task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/efs-isolation/" + suffix);
|
||||
|
||||
volumesToCleanUp.add(EcsContainerManager.efsVolumeName(fileSystemId, accessPointId, rootDirectory));
|
||||
|
||||
EcsTaskHandle handle = containerManager.startTask(task, taskDef, List.of(), "us-east-1");
|
||||
String dockerId = handle.getContainerIds().get("app");
|
||||
try {
|
||||
return dockerClient.waitContainerCmd(dockerId)
|
||||
.exec(new WaitContainerResultCallback())
|
||||
.awaitStatusCode(60, TimeUnit.SECONDS);
|
||||
} finally {
|
||||
containerManager.stopTask(handle);
|
||||
}
|
||||
}
|
||||
|
||||
private boolean isDockerAvailable() {
|
||||
try {
|
||||
dockerClient.pingCmd().exec();
|
||||
return true;
|
||||
} catch (Exception e) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
+194
@@ -0,0 +1,194 @@
|
||||
package io.github.hectorvent.floci.services.ecs.container;
|
||||
|
||||
import com.github.dockerjava.api.DockerClient;
|
||||
import com.github.dockerjava.core.command.WaitContainerResultCallback;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
|
||||
import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
|
||||
import io.github.hectorvent.floci.services.ecs.model.EcsTask;
|
||||
import io.github.hectorvent.floci.services.ecs.model.EfsVolumeConfiguration;
|
||||
import io.github.hectorvent.floci.services.ecs.model.MountPoint;
|
||||
import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;
|
||||
import io.github.hectorvent.floci.services.ecs.model.Volume;
|
||||
import io.github.hectorvent.floci.services.efs.EfsMounts;
|
||||
import io.github.hectorvent.floci.services.efs.EfsService;
|
||||
import io.github.hectorvent.floci.services.efs.model.CreateAccessPointRequest;
|
||||
import io.github.hectorvent.floci.services.efs.model.CreateFileSystemRequest;
|
||||
import io.github.hectorvent.floci.services.efs.model.CreationInfo;
|
||||
import io.github.hectorvent.floci.services.efs.model.PosixUser;
|
||||
import io.github.hectorvent.floci.services.efs.model.RootDirectory;
|
||||
import io.quarkus.test.junit.QuarkusTest;
|
||||
import jakarta.inject.Inject;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.Assumptions;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
|
||||
/**
|
||||
* Docker-backed proof that an EFS file system is one shared store, as on AWS: every task volume
|
||||
* of it mounts the same Docker volume, a {@code rootDirectory} or access point mounts its own
|
||||
* folder of it, and a mount of the file system root sees every folder. Each scenario runs a real
|
||||
* busybox container whose exit code proves what it could see under {@code /mnt/efs}.
|
||||
*/
|
||||
@QuarkusTest
|
||||
class EcsContainerManagerEfsSharingDockerIntegrationTest {
|
||||
|
||||
private static final String IMAGE = "public.ecr.aws/docker/library/busybox:latest";
|
||||
private static final String REGION = "us-east-1";
|
||||
|
||||
@Inject
|
||||
EcsContainerManager containerManager;
|
||||
|
||||
@Inject
|
||||
ContainerLifecycleManager lifecycleManager;
|
||||
|
||||
@Inject
|
||||
EfsService efsService;
|
||||
|
||||
@Inject
|
||||
DockerClient dockerClient;
|
||||
|
||||
private final List<String> volumesToCleanUp = new ArrayList<>();
|
||||
|
||||
@BeforeEach
|
||||
void requireDocker() {
|
||||
Assumptions.assumeTrue(isDockerAvailable(),
|
||||
"Docker daemon must be available for ECS EFS integration tests");
|
||||
}
|
||||
|
||||
@AfterEach
|
||||
void cleanUpVolumes() {
|
||||
volumesToCleanUp.forEach(lifecycleManager::removeVolume);
|
||||
volumesToCleanUp.clear();
|
||||
}
|
||||
|
||||
/** Two root directories are separate folders, and the file system root sees both. */
|
||||
@Test
|
||||
void rootDirectoriesAreFoldersOfOneFileSystem() {
|
||||
String fileSystemId = "fs-" + UUID.randomUUID().toString().substring(0, 8);
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/team-a", null,
|
||||
"echo team-a > /mnt/efs/only-a.txt"));
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/team-b", null,
|
||||
"test ! -e /mnt/efs/only-a.txt && echo team-b > /mnt/efs/only-b.txt"),
|
||||
"a task on /team-b must not see /team-a's files");
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/team-a", null,
|
||||
"grep -qx team-a /mnt/efs/only-a.txt && test ! -e /mnt/efs/only-b.txt"),
|
||||
"a second task on /team-a shares its files and still does not see /team-b's");
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/", null,
|
||||
"grep -qx team-a /mnt/efs/team-a/only-a.txt && grep -qx team-b /mnt/efs/team-b/only-b.txt"),
|
||||
"a task on the file system root sees every root directory's files");
|
||||
}
|
||||
|
||||
/**
|
||||
* An access point mounts its root directory, created on first use with its CreationInfo, and
|
||||
* shares it with every other mount of the file system.
|
||||
*/
|
||||
@Test
|
||||
void anAccessPointMountsItsFolderOfTheFileSystem() {
|
||||
String fileSystemId = createFileSystem();
|
||||
String accessPointId = createAccessPoint(fileSystemId, "/apps/orders", 1001L, 1002L, "750");
|
||||
|
||||
assertEquals(0, runEfsCommand(fileSystemId, null, accessPointId,
|
||||
"echo order-1 > /mnt/efs/orders.txt"));
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/", null,
|
||||
"grep -qx order-1 /mnt/efs/apps/orders/orders.txt"
|
||||
+ " && test \"$(stat -c '%u:%g %a' /mnt/efs/apps/orders)\" = '1001:1002 750'"
|
||||
+ " && test \"$(stat -c '%u:%g %a' /mnt/efs/apps)\" = '1001:1002 750'"),
|
||||
"the root mount sees the access point's files, in folders created from its CreationInfo");
|
||||
assertEquals(0, runEfsCommand(fileSystemId, "/apps", null,
|
||||
"grep -qx order-1 /mnt/efs/orders/orders.txt"),
|
||||
"a rootDirectory above the access point's folder sees it too");
|
||||
}
|
||||
|
||||
/** Without CreationInfo, an access point whose root directory does not exist cannot be mounted. */
|
||||
@Test
|
||||
void anAccessPointWithAMissingRootDirectoryAndNoCreationInfoIsRefused() {
|
||||
String fileSystemId = createFileSystem();
|
||||
String accessPointId = createAccessPoint(fileSystemId, "/absent", null, null, null);
|
||||
|
||||
assertThrows(IllegalStateException.class,
|
||||
() -> runEfsCommand(fileSystemId, null, accessPointId, "true"));
|
||||
}
|
||||
|
||||
private String createFileSystem() {
|
||||
CreateFileSystemRequest request = new CreateFileSystemRequest();
|
||||
request.setCreationToken("efs-sharing-" + UUID.randomUUID());
|
||||
return efsService.createFileSystem(request, REGION).getFileSystemId();
|
||||
}
|
||||
|
||||
private String createAccessPoint(String fileSystemId, String path, Long ownerUid, Long ownerGid,
|
||||
String permissions) {
|
||||
RootDirectory root = new RootDirectory();
|
||||
root.setPath(path);
|
||||
if (permissions != null) {
|
||||
CreationInfo info = new CreationInfo();
|
||||
info.setOwnerUid(ownerUid);
|
||||
info.setOwnerGid(ownerGid);
|
||||
info.setPermissions(permissions);
|
||||
root.setCreationInfo(info);
|
||||
}
|
||||
PosixUser user = new PosixUser();
|
||||
user.setUid(1001L);
|
||||
user.setGid(1002L);
|
||||
CreateAccessPointRequest request = new CreateAccessPointRequest();
|
||||
request.setFileSystemId(fileSystemId);
|
||||
request.setRootDirectory(root);
|
||||
request.setPosixUser(user);
|
||||
return efsService.createAccessPoint(REGION, request).getAccessPointId();
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a single-container ECS task that mounts {@code fileSystemId} (through
|
||||
* {@code rootDirectory} or {@code accessPointId}) at {@code /mnt/efs} and executes
|
||||
* {@code shellCommand}, then blocks until the container exits and returns its exit code.
|
||||
*/
|
||||
private int runEfsCommand(String fileSystemId, String rootDirectory, String accessPointId, String shellCommand) {
|
||||
String suffix = UUID.randomUUID().toString().substring(0, 8);
|
||||
|
||||
ContainerDefinition app = new ContainerDefinition();
|
||||
app.setName("app");
|
||||
app.setImage(IMAGE);
|
||||
app.setCommand(List.of("sh", "-c", shellCommand));
|
||||
app.setMountPoints(List.of(new MountPoint("efs-data", "/mnt/efs", false)));
|
||||
|
||||
TaskDefinition taskDef = new TaskDefinition();
|
||||
taskDef.setFamily("efs-sharing-" + suffix);
|
||||
taskDef.setContainerDefinitions(List.of(app));
|
||||
taskDef.setVolumes(List.of(new Volume("efs-data", null,
|
||||
new EfsVolumeConfiguration(fileSystemId, rootDirectory, null, null, accessPointId, null))));
|
||||
|
||||
EcsTask task = new EcsTask();
|
||||
task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/efs-sharing/" + suffix);
|
||||
|
||||
if (!volumesToCleanUp.contains(EfsMounts.volumeName(fileSystemId))) {
|
||||
volumesToCleanUp.add(EfsMounts.volumeName(fileSystemId));
|
||||
}
|
||||
|
||||
EcsTaskHandle handle = containerManager.startTask(task, taskDef, List.of(), REGION);
|
||||
String dockerId = handle.getContainerIds().get("app");
|
||||
try {
|
||||
return dockerClient.waitContainerCmd(dockerId)
|
||||
.exec(new WaitContainerResultCallback())
|
||||
.awaitStatusCode(60, TimeUnit.SECONDS);
|
||||
} finally {
|
||||
containerManager.stopTask(handle);
|
||||
}
|
||||
}
|
||||
|
||||
private boolean isDockerAvailable() {
|
||||
try {
|
||||
dockerClient.pingCmd().exec();
|
||||
return true;
|
||||
} catch (Exception e) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
-87
@@ -1,87 +0,0 @@
|
||||
package io.github.hectorvent.floci.services.ecs.container;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
/**
|
||||
* Unit tests for {@link EcsContainerManager#efsVolumeName}, the naming logic that scopes an
|
||||
* EFS-backed task volume's shared Docker named volume to both {@code fileSystemId} and its
|
||||
* effective root (#2563). The name must be stable for a given (fileSystemId, rootDirectory,
|
||||
* accessPointId) combination (so identical mounts keep sharing one volume), distinct across
|
||||
* different roots on the same file system (so mounts do not collide on the same tree), and
|
||||
* always a legal Docker volume name.
|
||||
*/
|
||||
class EcsContainerManagerEfsVolumeNamingTest {
|
||||
|
||||
private static final Pattern DOCKER_VOLUME_NAME = Pattern.compile("^[a-zA-Z0-9][a-zA-Z0-9_.-]*$");
|
||||
|
||||
@Test
|
||||
void defaultsToPlainFileSystemNameWhenNoRootOrAccessPointIsSet() {
|
||||
assertEquals("floci-efs-fs-abc", EcsContainerManager.efsVolumeName("fs-abc", null, null));
|
||||
assertEquals("floci-efs-fs-abc", EcsContainerManager.efsVolumeName("fs-abc", null, ""));
|
||||
assertEquals("floci-efs-fs-abc", EcsContainerManager.efsVolumeName("fs-abc", null, "/"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void differentRootDirectoriesOnSameFileSystemYieldDifferentVolumes() {
|
||||
String rootA = EcsContainerManager.efsVolumeName("fs-shared", null, "/team-a");
|
||||
String rootB = EcsContainerManager.efsVolumeName("fs-shared", null, "/team-b");
|
||||
String rootNone = EcsContainerManager.efsVolumeName("fs-shared", null, null);
|
||||
|
||||
assertNotEquals(rootA, rootB, "different rootDirectory values must isolate the volume");
|
||||
assertNotEquals(rootA, rootNone, "a non-root rootDirectory must not collapse to the plain name");
|
||||
assertTrue(DOCKER_VOLUME_NAME.matcher(rootA).matches(), "must be docker-safe, was: " + rootA);
|
||||
assertTrue(DOCKER_VOLUME_NAME.matcher(rootB).matches(), "must be docker-safe, was: " + rootB);
|
||||
}
|
||||
|
||||
@Test
|
||||
void sameRootDirectoryOnSameFileSystemStillShares() {
|
||||
String first = EcsContainerManager.efsVolumeName("fs-shared", null, "/team-a");
|
||||
String second = EcsContainerManager.efsVolumeName("fs-shared", null, "/team-a");
|
||||
|
||||
assertEquals(first, second, "identical (fileSystemId, rootDirectory) must keep sharing one volume");
|
||||
}
|
||||
|
||||
@Test
|
||||
void accessPointIdTakesPrecedenceOverRootDirectory() {
|
||||
String viaAccessPoint = EcsContainerManager.efsVolumeName("fs-shared", "fsap-111", "/ignored");
|
||||
String viaRootOnly = EcsContainerManager.efsVolumeName("fs-shared", null, "/ignored");
|
||||
|
||||
assertNotEquals(viaAccessPoint, viaRootOnly,
|
||||
"an access point's own root must take precedence over rootDirectory, not merge with it");
|
||||
}
|
||||
|
||||
@Test
|
||||
void differentAccessPointsOnSameFileSystemYieldDifferentVolumes() {
|
||||
String apOne = EcsContainerManager.efsVolumeName("fs-shared", "fsap-111", null);
|
||||
String apTwo = EcsContainerManager.efsVolumeName("fs-shared", "fsap-222", null);
|
||||
|
||||
assertNotEquals(apOne, apTwo, "different accessPointId values must isolate the volume");
|
||||
}
|
||||
|
||||
@Test
|
||||
void sameAccessPointStillShares() {
|
||||
String first = EcsContainerManager.efsVolumeName("fs-shared", "fsap-111", null);
|
||||
String second = EcsContainerManager.efsVolumeName("fs-shared", "fsap-111", "/whatever-is-ignored");
|
||||
|
||||
assertEquals(first, second,
|
||||
"the same accessPointId must keep sharing one volume regardless of rootDirectory");
|
||||
}
|
||||
|
||||
@Test
|
||||
void equivalentRootDirectorySpellingsShareTheSameVolume() {
|
||||
String canonical = EcsContainerManager.efsVolumeName("fs-shared", null, "/team-a");
|
||||
|
||||
assertEquals(canonical, EcsContainerManager.efsVolumeName("fs-shared", null, "/team-a/"),
|
||||
"a trailing slash must not create a distinct volume for the same directory");
|
||||
assertEquals(canonical, EcsContainerManager.efsVolumeName("fs-shared", null, "//team-a"),
|
||||
"a repeated leading slash must not create a distinct volume for the same directory");
|
||||
assertEquals(canonical, EcsContainerManager.efsVolumeName("fs-shared", null, "team-a"),
|
||||
"a missing leading slash must not create a distinct volume for the same directory");
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -89,7 +89,7 @@ class EcsContainerManagerSecurityGroupTest {
|
||||
containerDetector, config, regionResolver, awsEnv,
|
||||
mock(SsmService.class), mock(SecretsManagerService.class), mock(S3Service.class),
|
||||
ecrRegistryManager, mock(HostVolumePolicy.class),
|
||||
ec2Service, firewallManager);
|
||||
ec2Service, firewallManager, mock(io.github.hectorvent.floci.services.efs.EfsMounts.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+7
-7
@@ -69,6 +69,7 @@ class EcsContainerManagerVolumesTest {
|
||||
lifecycleManager = mock(ContainerLifecycleManager.class);
|
||||
when(lifecycleManager.createAndStart(any()))
|
||||
.thenReturn(new ContainerInfo("docker-id", Map.of()));
|
||||
when(lifecycleManager.ensureSharedVolumeDirectory(anyString(), anyString(), any(), any())).thenReturn(true);
|
||||
|
||||
ContainerLogStreamer logStreamer = mock(ContainerLogStreamer.class);
|
||||
ContainerDetector containerDetector = mock(ContainerDetector.class);
|
||||
@@ -132,9 +133,8 @@ class EcsContainerManagerVolumesTest {
|
||||
}
|
||||
|
||||
/**
|
||||
* Each EFS volume resolves to a shared local Docker named volume keyed by file system id
|
||||
* AND effective root (here: rootDirectory, since neither volume sets an accessPointId),
|
||||
* read-write or read-only per the mountPoint.
|
||||
* Each EFS volume mounts its file system's one Docker named volume: the whole volume for the
|
||||
* file system root, or the rootDirectory's folder of it, read-write or read-only per the mountPoint.
|
||||
*/
|
||||
@Test
|
||||
void efsVolumeMountPointsResolveToSharedNamedVolumesByAccessMode() {
|
||||
@@ -159,10 +159,10 @@ class EcsContainerManagerVolumesTest {
|
||||
|
||||
manager.startTask(task, taskDef, List.of(), "us-east-1");
|
||||
|
||||
verify(builder, times(1)).withNamedVolumeSubpath(
|
||||
"floci-efs-fs-0123456789abcdef0", "dps", "/mnt/efs", false);
|
||||
verify(builder, times(1)).withNamedVolume(
|
||||
EcsContainerManager.efsVolumeName("fs-0123456789abcdef0", null, "/dps"), "/mnt/efs", false);
|
||||
verify(builder, times(1)).withNamedVolume(
|
||||
EcsContainerManager.efsVolumeName("fs-00000000000000001", null, null), "/mnt/shared", true);
|
||||
"floci-efs-fs-00000000000000001", "/mnt/shared", true);
|
||||
|
||||
// EFS volumes are never bind-mounted as host paths.
|
||||
verify(builder, never()).withBind(any(), any());
|
||||
@@ -202,7 +202,7 @@ class EcsContainerManagerVolumesTest {
|
||||
configured.startTask(task, taskDef, List.of(), "us-east-1");
|
||||
|
||||
verify(lifecycleManager, times(1)).ensureSharedVolume(
|
||||
EcsContainerManager.efsVolumeName("fs-abc", null, "/dps"),
|
||||
"floci-efs-fs-abc",
|
||||
OptionalInt.of(1001), OptionalInt.of(1001), Optional.of("2775"), "busybox:stable");
|
||||
}
|
||||
|
||||
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
package io.github.hectorvent.floci.services.efs;
|
||||
|
||||
import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
|
||||
import io.quarkus.test.junit.QuarkusTest;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static io.restassured.RestAssured.given;
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
|
||||
/** CreateAccessPoint refuses a root directory or identity that EFS itself would refuse. */
|
||||
@QuarkusTest
|
||||
class EfsAccessPointValidationTest {
|
||||
|
||||
@BeforeAll
|
||||
static void configureRestAssured() {
|
||||
RestAssuredJsonUtils.configureAwsContentTypes();
|
||||
}
|
||||
|
||||
private String createFileSystem(String token) {
|
||||
return given()
|
||||
.contentType("application/json")
|
||||
.body("{\"CreationToken\": \"" + token + "\"}")
|
||||
.when()
|
||||
.post("/2015-02-01/file-systems")
|
||||
.then()
|
||||
.statusCode(201)
|
||||
.extract().jsonPath().getString("FileSystemId");
|
||||
}
|
||||
|
||||
private io.restassured.response.ValidatableResponse createAccessPoint(String body) {
|
||||
return given()
|
||||
.contentType("application/json")
|
||||
.body(body)
|
||||
.when()
|
||||
.post("/2015-02-01/access-points")
|
||||
.then();
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsANestedRootDirectoryWithCreationInfo() {
|
||||
String fileSystemId = createFileSystem("ap-valid");
|
||||
|
||||
createAccessPoint("{\"FileSystemId\": \"" + fileSystemId + "\","
|
||||
+ " \"PosixUser\": {\"Uid\": 1001, \"Gid\": 1001},"
|
||||
+ " \"RootDirectory\": {\"Path\": \"/apps/my-function\","
|
||||
+ " \"CreationInfo\": {\"OwnerUid\": 1001, \"OwnerGid\": 1001, \"Permissions\": \"0755\"}}}")
|
||||
.statusCode(200)
|
||||
.body("RootDirectory.Path", equalTo("/apps/my-function"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void refusesARootDirectoryThatClimbsOutOrHidesItself() {
|
||||
String fileSystemId = createFileSystem("ap-bad-path");
|
||||
|
||||
for (String path : new String[] {"/a/../b", "/.hidden", "relative", "/a;rm"}) {
|
||||
createAccessPoint("{\"FileSystemId\": \"" + fileSystemId + "\","
|
||||
+ " \"RootDirectory\": {\"Path\": \"" + path + "\"}}")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("BadRequest"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void refusesPartialCreationInfoAndPosixUser() {
|
||||
String fileSystemId = createFileSystem("ap-partial");
|
||||
|
||||
createAccessPoint("{\"FileSystemId\": \"" + fileSystemId + "\","
|
||||
+ " \"RootDirectory\": {\"Path\": \"/data\", \"CreationInfo\": {\"OwnerUid\": 1001}}}")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("BadRequest"));
|
||||
createAccessPoint("{\"FileSystemId\": \"" + fileSystemId + "\","
|
||||
+ " \"RootDirectory\": {\"Path\": \"/data\","
|
||||
+ " \"CreationInfo\": {\"OwnerUid\": 1001, \"OwnerGid\": 1001, \"Permissions\": \"rwx\"}}}")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("BadRequest"));
|
||||
createAccessPoint("{\"FileSystemId\": \"" + fileSystemId + "\", \"PosixUser\": {\"Uid\": 1001}}")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("BadRequest"));
|
||||
}
|
||||
}
|
||||
@@ -90,8 +90,8 @@ class EfsIntegrationTest {
|
||||
.body("""
|
||||
{
|
||||
"FileSystemId": "%s",
|
||||
"SubnetId": "subnet-12345",
|
||||
"SecurityGroups": ["sg-11111"]
|
||||
"SubnetId": "subnet-default-us-east-1-a",
|
||||
"SecurityGroups": ["sg-default-us-east-1"]
|
||||
}
|
||||
""".formatted(fileSystemId))
|
||||
.when()
|
||||
@@ -100,7 +100,12 @@ class EfsIntegrationTest {
|
||||
.statusCode(200)
|
||||
.body("MountTargetId", startsWith("fsmt-"))
|
||||
.body("FileSystemId", equalTo(fileSystemId))
|
||||
.body("SubnetId", equalTo("subnet-12345"))
|
||||
.body("SubnetId", equalTo("subnet-default-us-east-1-a"))
|
||||
.body("VpcId", equalTo("vpc-default-us-east-1"))
|
||||
.body("AvailabilityZoneName", equalTo("us-east-1a"))
|
||||
.body("IpAddress", startsWith("172.31.0."))
|
||||
.body("NetworkInterfaceId", startsWith("eni-"))
|
||||
.body("LifeCycleState", equalTo("available"))
|
||||
.extract().jsonPath().getString("MountTargetId");
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
package io.github.hectorvent.floci.services.efs;
|
||||
|
||||
import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
|
||||
import io.quarkus.test.junit.QuarkusTest;
|
||||
import jakarta.inject.Inject;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.Optional;
|
||||
|
||||
import static io.restassured.RestAssured.given;
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.startsWith;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
/**
|
||||
* Mount targets take their VPC, zone and address from the EC2 subnet they are created in, hold an
|
||||
* EC2 network interface for that address, and answer the file system's DNS name.
|
||||
*/
|
||||
@QuarkusTest
|
||||
class EfsMountTargetSubnetTest {
|
||||
|
||||
@Inject
|
||||
EfsService efsService;
|
||||
|
||||
@BeforeAll
|
||||
static void configureRestAssured() {
|
||||
RestAssuredJsonUtils.configureAwsContentTypes();
|
||||
}
|
||||
|
||||
private String createFileSystem(String token) {
|
||||
return given()
|
||||
.contentType("application/json")
|
||||
.body("{\"CreationToken\": \"" + token + "\"}")
|
||||
.when()
|
||||
.post("/2015-02-01/file-systems")
|
||||
.then()
|
||||
.statusCode(201)
|
||||
.extract().jsonPath().getString("FileSystemId");
|
||||
}
|
||||
|
||||
private io.restassured.response.ValidatableResponse createMountTarget(String fileSystemId, String subnetId) {
|
||||
return given()
|
||||
.contentType("application/json")
|
||||
.body("{\"FileSystemId\": \"" + fileSystemId + "\", \"SubnetId\": \"" + subnetId + "\"}")
|
||||
.when()
|
||||
.post("/2015-02-01/mount-targets")
|
||||
.then();
|
||||
}
|
||||
|
||||
@Test
|
||||
void usesTheZoneAndVpcOfEachSubnet() {
|
||||
String fileSystemId = createFileSystem("subnet-zones");
|
||||
|
||||
createMountTarget(fileSystemId, "subnet-default-us-east-1-b")
|
||||
.statusCode(200)
|
||||
.body("VpcId", equalTo("vpc-default-us-east-1"))
|
||||
.body("AvailabilityZoneName", equalTo("us-east-1b"))
|
||||
.body("IpAddress", startsWith("172.31.16."));
|
||||
createMountTarget(fileSystemId, "subnet-default-us-east-1-c")
|
||||
.statusCode(200)
|
||||
.body("AvailabilityZoneName", equalTo("us-east-1c"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void refusesASubnetThatDoesNotExist() {
|
||||
String fileSystemId = createFileSystem("subnet-missing");
|
||||
|
||||
createMountTarget(fileSystemId, "subnet-nope")
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("SubnetNotFound"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void refusesASecurityGroupThatDoesNotExist() {
|
||||
String fileSystemId = createFileSystem("subnet-missing-sg");
|
||||
|
||||
given()
|
||||
.contentType("application/json")
|
||||
.body("{\"FileSystemId\": \"" + fileSystemId + "\", \"SubnetId\": \"subnet-default-us-east-1-a\","
|
||||
+ " \"SecurityGroups\": [\"sg-nope\"]}")
|
||||
.when()
|
||||
.post("/2015-02-01/mount-targets")
|
||||
.then()
|
||||
.statusCode(400)
|
||||
.body("__type", equalTo("SecurityGroupNotFound"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void allowsOneMountTargetPerZone() {
|
||||
String fileSystemId = createFileSystem("subnet-one-per-zone");
|
||||
String vpcId = "vpc-default-us-east-1";
|
||||
String secondSubnet = given()
|
||||
.formParam("Action", "CreateSubnet")
|
||||
.formParam("VpcId", vpcId)
|
||||
.formParam("CidrBlock", "172.31.200.0/24")
|
||||
.formParam("AvailabilityZone", "us-east-1a")
|
||||
.header("Authorization", "AWS4-HMAC-SHA256 Credential=test/20260101/us-east-1/ec2/aws4_request")
|
||||
.when()
|
||||
.post("/")
|
||||
.then()
|
||||
.statusCode(200)
|
||||
.extract().path("CreateSubnetResponse.subnet.subnetId");
|
||||
|
||||
createMountTarget(fileSystemId, "subnet-default-us-east-1-a").statusCode(200);
|
||||
createMountTarget(fileSystemId, secondSubnet)
|
||||
.statusCode(409)
|
||||
.body("__type", equalTo("MountTargetConflict"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void holdsAnEc2NetworkInterfaceWithItsAddressUntilDeleted() {
|
||||
String fileSystemId = createFileSystem("subnet-eni");
|
||||
var mountTarget = createMountTarget(fileSystemId, "subnet-default-us-east-1-c")
|
||||
.statusCode(200)
|
||||
.extract().jsonPath();
|
||||
String eniId = mountTarget.getString("NetworkInterfaceId");
|
||||
|
||||
describeNetworkInterface(eniId)
|
||||
.statusCode(200)
|
||||
.body("DescribeNetworkInterfacesResponse.networkInterfaceSet.item.privateIpAddress",
|
||||
equalTo(mountTarget.getString("IpAddress")))
|
||||
.body("DescribeNetworkInterfacesResponse.networkInterfaceSet.item.status", equalTo("in-use"))
|
||||
.body("DescribeNetworkInterfacesResponse.networkInterfaceSet.item.subnetId",
|
||||
equalTo("subnet-default-us-east-1-c"));
|
||||
|
||||
given()
|
||||
.when()
|
||||
.delete("/2015-02-01/mount-targets/" + mountTarget.getString("MountTargetId"))
|
||||
.then()
|
||||
.statusCode(204);
|
||||
describeNetworkInterface(eniId).statusCode(400);
|
||||
}
|
||||
|
||||
@Test
|
||||
void resolvesTheFileSystemDnsNameToAMountTarget() {
|
||||
String fileSystemId = createFileSystem("subnet-dns");
|
||||
String inZoneB = createMountTarget(fileSystemId, "subnet-default-us-east-1-b")
|
||||
.statusCode(200).extract().jsonPath().getString("IpAddress");
|
||||
String inZoneA = createMountTarget(fileSystemId, "subnet-default-us-east-1-a")
|
||||
.statusCode(200).extract().jsonPath().getString("IpAddress");
|
||||
|
||||
assertEquals(Optional.of(inZoneA),
|
||||
efsService.resolveDnsName(fileSystemId + ".efs.us-east-1.amazonaws.com"));
|
||||
assertEquals(Optional.of(inZoneB),
|
||||
efsService.resolveDnsName("us-east-1b." + fileSystemId + ".efs.us-east-1.amazonaws.com."));
|
||||
assertEquals(Optional.empty(),
|
||||
efsService.resolveDnsName("us-east-1c." + fileSystemId + ".efs.us-east-1.amazonaws.com"));
|
||||
assertEquals(Optional.empty(),
|
||||
efsService.resolveDnsName(fileSystemId + ".efs.eu-west-1.amazonaws.com"));
|
||||
assertEquals(Optional.empty(), efsService.resolveDnsName("example.com"));
|
||||
}
|
||||
|
||||
private io.restassured.response.ValidatableResponse describeNetworkInterface(String eniId) {
|
||||
return given()
|
||||
.formParam("Action", "DescribeNetworkInterfaces")
|
||||
.formParam("NetworkInterfaceId.1", eniId)
|
||||
.header("Authorization", "AWS4-HMAC-SHA256 Credential=test/20260101/us-east-1/ec2/aws4_request")
|
||||
.when()
|
||||
.post("/")
|
||||
.then();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
package io.github.hectorvent.floci.services.efs;
|
||||
|
||||
import io.github.hectorvent.floci.config.EmulatorConfig;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
|
||||
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.DirectoryCreation;
|
||||
import io.github.hectorvent.floci.services.efs.model.AccessPointDescription;
|
||||
import io.github.hectorvent.floci.services.efs.model.CreationInfo;
|
||||
import io.github.hectorvent.floci.services.efs.model.PosixUser;
|
||||
import io.github.hectorvent.floci.services.efs.model.RootDirectory;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.OptionalInt;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
|
||||
import static org.mockito.Mockito.RETURNS_SELF;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
/** Every mount of a file system shares its one volume; access points mount their root directory in it. */
|
||||
class EfsMountsTest {
|
||||
|
||||
private EfsService efsService;
|
||||
private ContainerLifecycleManager lifecycleManager;
|
||||
private EfsMounts mounts;
|
||||
|
||||
@BeforeEach
|
||||
void setUp() {
|
||||
efsService = mock(EfsService.class);
|
||||
lifecycleManager = mock(ContainerLifecycleManager.class);
|
||||
when(lifecycleManager.ensureSharedVolumeDirectory(anyString(), anyString(), any(), any())).thenReturn(true);
|
||||
EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
|
||||
when(config.storage().efs().ownerUid()).thenReturn(OptionalInt.empty());
|
||||
when(config.storage().efs().ownerGid()).thenReturn(OptionalInt.empty());
|
||||
when(config.storage().efs().rootPermissions()).thenReturn(Optional.empty());
|
||||
when(config.storage().efs().initImage()).thenReturn("busybox:stable");
|
||||
mounts = new EfsMounts(efsService, lifecycleManager, config);
|
||||
}
|
||||
|
||||
@Test
|
||||
void theFileSystemRootMountsTheWholeVolume() {
|
||||
EfsMounts.Mount mount = mounts.prepare("us-east-1", "fs-abc", null, null);
|
||||
|
||||
assertEquals("floci-efs-fs-abc", mount.volumeName());
|
||||
assertEquals("", mount.subpath());
|
||||
assertTrue(mount.accessPointUser().isEmpty());
|
||||
verify(lifecycleManager).ensureSharedVolume("floci-efs-fs-abc",
|
||||
OptionalInt.empty(), OptionalInt.empty(), Optional.empty(), "busybox:stable");
|
||||
verify(lifecycleManager, never()).ensureSharedVolumeDirectory(anyString(), anyString(), any(), any());
|
||||
}
|
||||
|
||||
@Test
|
||||
void aRootDirectoryMountsItsFolderOfTheSameVolume() {
|
||||
EfsMounts.Mount mount = mounts.prepare("us-east-1", "fs-abc", null, "/team-a/");
|
||||
|
||||
assertEquals("floci-efs-fs-abc", mount.volumeName());
|
||||
assertEquals("team-a", mount.subpath());
|
||||
verify(lifecycleManager).ensureSharedVolumeDirectory("floci-efs-fs-abc", "team-a",
|
||||
Optional.of(new DirectoryCreation(0, 0, "755")), "busybox:stable");
|
||||
}
|
||||
|
||||
@Test
|
||||
void anAccessPointMountsItsRootDirectoryAndCarriesItsUser() {
|
||||
when(efsService.describeAccessPoints("us-east-1", null, "fsap-1"))
|
||||
.thenReturn(List.of(accessPoint("fs-abc", "/my-function", creationInfo(1001L, 1001L, "755"))));
|
||||
|
||||
EfsMounts.Mount mount = mounts.prepare("us-east-1", null, "fsap-1", "/ignored");
|
||||
|
||||
assertEquals("floci-efs-fs-abc", mount.volumeName());
|
||||
assertEquals("my-function", mount.subpath());
|
||||
assertEquals("1001:1001", EfsMounts.containerUser(mount.accessPointUser().orElseThrow()));
|
||||
assertEquals(List.of(2000L), EfsMounts.secondaryGids(mount.accessPointUser().orElseThrow()));
|
||||
verify(lifecycleManager).ensureSharedVolumeDirectory("floci-efs-fs-abc", "my-function",
|
||||
Optional.of(new DirectoryCreation(1001, 1001, "755")), "busybox:stable");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aLambdaAccessPointArnIsLookedUpInItsRegion() {
|
||||
when(efsService.describeAccessPoints("eu-west-1", null, "fsap-1"))
|
||||
.thenReturn(List.of(accessPoint("fs-abc", "/", null)));
|
||||
|
||||
EfsMounts.Mount mount = mounts.prepareAccessPoint(
|
||||
"arn:aws:elasticfilesystem:eu-west-1:000000000000:access-point/fsap-1");
|
||||
|
||||
assertEquals("floci-efs-fs-abc", mount.volumeName());
|
||||
assertEquals("", mount.subpath());
|
||||
}
|
||||
|
||||
@Test
|
||||
void aMissingRootDirectoryWithoutCreationInfoCannotBeMounted() {
|
||||
when(efsService.describeAccessPoints("us-east-1", null, "fsap-1"))
|
||||
.thenReturn(List.of(accessPoint("fs-abc", "/absent", null)));
|
||||
when(lifecycleManager.ensureSharedVolumeDirectory("floci-efs-fs-abc", "absent", Optional.empty(),
|
||||
"busybox:stable")).thenReturn(false);
|
||||
|
||||
IllegalStateException error = assertThrows(IllegalStateException.class,
|
||||
() -> mounts.prepare("us-east-1", "fs-abc", "fsap-1", null));
|
||||
|
||||
assertTrue(error.getMessage().contains("/absent"), error.getMessage());
|
||||
}
|
||||
|
||||
@Test
|
||||
void anAccessPointOfAnotherFileSystemIsRefused() {
|
||||
when(efsService.describeAccessPoints("us-east-1", null, "fsap-1"))
|
||||
.thenReturn(List.of(accessPoint("fs-other", "/", null)));
|
||||
|
||||
assertThrows(IllegalArgumentException.class, () -> mounts.prepare("us-east-1", "fs-abc", "fsap-1", null));
|
||||
}
|
||||
|
||||
@Test
|
||||
void anUnknownAccessPointIsRefused() {
|
||||
when(efsService.describeAccessPoints("us-east-1", null, "fsap-404")).thenReturn(List.of());
|
||||
|
||||
assertThrows(EfsException.class, () -> mounts.prepare("us-east-1", "fs-abc", "fsap-404", null));
|
||||
}
|
||||
|
||||
@Test
|
||||
void equivalentRootDirectorySpellingsNameOneFolder() {
|
||||
assertEquals("/", EfsMounts.normalizeRootDirectory(null));
|
||||
assertEquals("/", EfsMounts.normalizeRootDirectory(" "));
|
||||
assertEquals("/team-a", EfsMounts.normalizeRootDirectory("team-a"));
|
||||
assertEquals("/team-a", EfsMounts.normalizeRootDirectory("//team-a/"));
|
||||
assertThrows(IllegalArgumentException.class, () -> EfsMounts.normalizeRootDirectory("/a/../b"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aFolderMountUsesTheVolumeSubpath() {
|
||||
ContainerBuilder.Builder spec = mock(ContainerBuilder.Builder.class, RETURNS_SELF);
|
||||
|
||||
new EfsMounts.Mount("floci-efs-fs-abc", "team-a", Optional.empty()).applyTo(spec, "/mnt/efs", true);
|
||||
new EfsMounts.Mount("floci-efs-fs-abc", "", Optional.empty()).applyTo(spec, "/mnt/all", false);
|
||||
|
||||
verify(spec).withNamedVolumeSubpath("floci-efs-fs-abc", "team-a", "/mnt/efs", true);
|
||||
verify(spec).withNamedVolume("floci-efs-fs-abc", "/mnt/all", false);
|
||||
}
|
||||
|
||||
private static AccessPointDescription accessPoint(String fileSystemId, String path, CreationInfo creationInfo) {
|
||||
RootDirectory root = new RootDirectory();
|
||||
root.setPath(path);
|
||||
root.setCreationInfo(creationInfo);
|
||||
PosixUser user = new PosixUser();
|
||||
user.setUid(1001L);
|
||||
user.setGid(1001L);
|
||||
user.setSecondaryGids(List.of(2000L));
|
||||
AccessPointDescription accessPoint = new AccessPointDescription();
|
||||
accessPoint.setAccessPointId("fsap-1");
|
||||
accessPoint.setFileSystemId(fileSystemId);
|
||||
accessPoint.setRootDirectory(root);
|
||||
accessPoint.setPosixUser(user);
|
||||
return accessPoint;
|
||||
}
|
||||
|
||||
private static CreationInfo creationInfo(Long uid, Long gid, String permissions) {
|
||||
CreationInfo info = new CreationInfo();
|
||||
info.setOwnerUid(uid);
|
||||
info.setOwnerGid(gid);
|
||||
info.setPermissions(permissions);
|
||||
return info;
|
||||
}
|
||||
}
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
package io.github.hectorvent.floci.services.glue;
|
||||
|
||||
import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
|
||||
import io.quarkus.test.junit.QuarkusTest;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.UUID;
|
||||
|
||||
import static io.restassured.RestAssured.given;
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.hasKey;
|
||||
import static org.hamcrest.Matchers.not;
|
||||
import static org.hamcrest.Matchers.notNullValue;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
/**
|
||||
* AWS leaves unset fields out of Glue responses and reports times in milliseconds.
|
||||
* A null or nanosecond timestamp instead breaks the AWS SDK for JavaScript v3,
|
||||
* which refuses to read either.
|
||||
*/
|
||||
@QuarkusTest
|
||||
class GlueOmitsNullFieldsIntegrationTest {
|
||||
|
||||
private static final String CONTENT_TYPE = "application/x-amz-json-1.1";
|
||||
private static final String DATABASE = "nulls_" + UUID.randomUUID().toString().substring(0, 8);
|
||||
|
||||
@BeforeAll
|
||||
static void configureRestAssured() {
|
||||
RestAssuredJsonUtils.configureAwsContentTypes();
|
||||
}
|
||||
|
||||
private static io.restassured.response.ValidatableResponse glue(String action, String body) {
|
||||
return given().contentType(CONTENT_TYPE)
|
||||
.header("X-Amz-Target", "AWSGlue." + action)
|
||||
.body(body)
|
||||
.when().post("/")
|
||||
.then().statusCode(200);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aNewTableHasOnlyAMillisecondCreateTime() {
|
||||
glue("CreateDatabase", "{\"DatabaseInput\": {\"Name\": \"" + DATABASE + "\"}}");
|
||||
glue("CreateTable", """
|
||||
{
|
||||
"DatabaseName": "%s",
|
||||
"TableInput": {
|
||||
"Name": "orders",
|
||||
"StorageDescriptor": {"Columns": [{"Name": "id", "Type": "int"}], "Location": "s3://data/orders/"}
|
||||
}
|
||||
}
|
||||
""".formatted(DATABASE));
|
||||
|
||||
String tables = glue("GetTables", "{\"DatabaseName\": \"" + DATABASE + "\"}")
|
||||
.body("TableList[0].Name", equalTo("orders"))
|
||||
.body("TableList[0].CreateTime", notNullValue())
|
||||
.body("TableList[0]", not(hasKey("UpdateTime")))
|
||||
.body("TableList[0]", not(hasKey("LastAccessTime")))
|
||||
.extract().asString();
|
||||
// Milliseconds in their shortest form, as AWS sends them: no trailing zeros.
|
||||
assertTrue(tables.matches("(?s).*\"CreateTime\":\\d+(\\.\\d{0,2}[1-9])?[,}].*"), tables);
|
||||
|
||||
glue("GetDatabase", "{\"Name\": \"" + DATABASE + "\"}")
|
||||
.body("Database", not(hasKey("Description")))
|
||||
.body("Database.CreateTime", notNullValue());
|
||||
}
|
||||
}
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
package io.github.hectorvent.floci.services.glue.model;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.Instant;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
class GlueTimestampSerializerTest {
|
||||
|
||||
@Test
|
||||
void writesMillisecondsInTheirShortestForm() {
|
||||
assertEquals("1792964720.273", GlueTimestampSerializer.epochSeconds(Instant.ofEpochSecond(1792964720, 273_456_789)));
|
||||
assertEquals("1792964720.27", GlueTimestampSerializer.epochSeconds(Instant.ofEpochSecond(1792964720, 270_000_000)));
|
||||
assertEquals("1792964720", GlueTimestampSerializer.epochSeconds(Instant.ofEpochSecond(1792964720)));
|
||||
}
|
||||
}
|
||||
@@ -250,7 +250,8 @@ class LambdaImageConfigTest {
|
||||
launcher = new ContainerLauncher(containerBuilder, lifecycleManager, logStreamer, imageResolver,
|
||||
runtimeApiServerFactory, dockerHostResolver, config, ecrRegistryManager,
|
||||
mock(io.github.hectorvent.floci.services.lambda.LambdaLayerService.class), awsEnv,
|
||||
mock(io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials.class));
|
||||
mock(io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials.class),
|
||||
mock(io.github.hectorvent.floci.services.efs.EfsMounts.class));
|
||||
|
||||
when(runtimeApiServerFactory.create()).thenReturn(runtimeApiServer);
|
||||
when(runtimeApiServer.getPort()).thenReturn(9000);
|
||||
|
||||
+1
-1
@@ -59,7 +59,7 @@ class ContainerLauncherCodeVolumeReuseTest {
|
||||
mock(ImageResolver.class), mock(RuntimeApiServerFactory.class),
|
||||
mock(DockerHostResolver.class), config, mock(EcrRegistryManager.class),
|
||||
mock(LambdaLayerService.class), mock(LaunchedContainerAwsEnv.class),
|
||||
mock(LambdaExecutionRoleCredentials.class));
|
||||
mock(LambdaExecutionRoleCredentials.class), mock(io.github.hectorvent.floci.services.efs.EfsMounts.class));
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
+48
-12
@@ -12,6 +12,9 @@ import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
|
||||
import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;
|
||||
import io.github.hectorvent.floci.services.iam.model.SessionCreds;
|
||||
import io.github.hectorvent.floci.services.lambda.model.LambdaFileSystemConfig;
|
||||
import io.github.hectorvent.floci.services.efs.EfsMounts;
|
||||
import io.github.hectorvent.floci.services.efs.model.PosixUser;
|
||||
import io.github.hectorvent.floci.core.common.docker.SubpathVolumeOptions;
|
||||
import io.github.hectorvent.floci.services.lambda.model.LambdaFunction;
|
||||
import io.github.hectorvent.floci.services.lambda.runtime.RuntimeApiServer;
|
||||
import io.github.hectorvent.floci.services.lambda.runtime.RuntimeApiServerFactory;
|
||||
@@ -87,6 +90,7 @@ class ContainerLauncherTest {
|
||||
Path tempDir;
|
||||
|
||||
ContainerLauncher launcher;
|
||||
EfsMounts efsMounts = mock(EfsMounts.class);
|
||||
/** Collects remote paths passed to withRemotePath across all copy mocks. */
|
||||
final java.util.List<String> capturedRemotePaths = new java.util.ArrayList<>();
|
||||
|
||||
@@ -134,7 +138,7 @@ class ContainerLauncherTest {
|
||||
launcher = new ContainerLauncher(containerBuilder, lifecycleManager, logStreamer, imageResolver,
|
||||
runtimeApiServerFactory, dockerHostResolver, config, ecrRegistryManager,
|
||||
mock(io.github.hectorvent.floci.services.lambda.LambdaLayerService.class), awsEnv,
|
||||
executionRoleCredentials);
|
||||
executionRoleCredentials, efsMounts);
|
||||
|
||||
when(runtimeApiServerFactory.create()).thenReturn(runtimeApiServer);
|
||||
when(runtimeApiServer.getPort()).thenReturn(9000);
|
||||
@@ -469,8 +473,15 @@ class ContainerLauncherTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
void launchFunction_mountsConfiguredFileSystemVolume() throws Exception {
|
||||
void launchFunction_mountsTheAccessPointFolderAsItsUser() throws Exception {
|
||||
Path codePath = Files.createDirectory(tempDir.resolve("efs-code"));
|
||||
String accessPointArn = "arn:aws:elasticfilesystem:us-east-1:000000000000:access-point/fsap-0123456789abcdef0";
|
||||
PosixUser user = new PosixUser();
|
||||
user.setUid(1001L);
|
||||
user.setGid(1002L);
|
||||
user.setSecondaryGids(List.of(2000L));
|
||||
when(efsMounts.prepareAccessPoint(accessPointArn))
|
||||
.thenReturn(new EfsMounts.Mount("floci-efs-fs-abc", "my-function", Optional.of(user)));
|
||||
|
||||
LambdaFunction fn = new LambdaFunction();
|
||||
fn.setFunctionName("efs-fn");
|
||||
@@ -478,23 +489,48 @@ class ContainerLauncherTest {
|
||||
fn.setRuntime("nodejs20.x");
|
||||
fn.setHandler("index.handler");
|
||||
fn.setCodeLocalPath(codePath.toString());
|
||||
fn.setFileSystemConfigs(List.of(new LambdaFileSystemConfig(
|
||||
"arn:aws:elasticfilesystem:us-east-1:000000000000:access-point/fsap-0123456789abcdef0",
|
||||
"/mnt/shared")));
|
||||
fn.setFileSystemConfigs(List.of(new LambdaFileSystemConfig(accessPointArn, "/mnt/shared")));
|
||||
|
||||
launcher.launch(fn);
|
||||
|
||||
String expectedVolumeName = "floci-efs-fsap-0123456789abcdef0-"
|
||||
+ "9d6eafd2aec94d4518a004f005725b4b3c673c1506436bb7368cfd5450fc0810";
|
||||
verify(lifecycleManager).ensureSharedVolume(expectedVolumeName,
|
||||
OptionalInt.empty(), OptionalInt.empty(), Optional.empty(), "busybox:stable");
|
||||
Mount mount = captureRealContainerSpec().mounts().stream()
|
||||
ContainerSpec spec = captureRealContainerSpec();
|
||||
Mount mount = spec.mounts().stream()
|
||||
.filter(candidate -> "/mnt/shared".equals(candidate.getTarget()))
|
||||
.findFirst()
|
||||
.orElseThrow();
|
||||
assertEquals(MountType.VOLUME, mount.getType());
|
||||
assertEquals(expectedVolumeName, mount.getSource());
|
||||
assertEquals("floci-efs-fs-abc", mount.getSource());
|
||||
assertEquals("my-function", ((SubpathVolumeOptions) mount.getVolumeOptions()).getSubpath());
|
||||
assertTrue(!Boolean.TRUE.equals(mount.getReadOnly()));
|
||||
assertEquals("1001:1002", spec.user());
|
||||
assertEquals(List.of("2000"), spec.groupAdd());
|
||||
}
|
||||
|
||||
@Test
|
||||
void launchFunction_withoutAnAccessPointUserKeepsTheImageUser() throws Exception {
|
||||
Path codePath = Files.createDirectory(tempDir.resolve("efs-root-code"));
|
||||
String accessPointArn = "arn:aws:elasticfilesystem:us-east-1:000000000000:access-point/fsap-1";
|
||||
when(efsMounts.prepareAccessPoint(accessPointArn))
|
||||
.thenReturn(new EfsMounts.Mount("floci-efs-fs-abc", "", Optional.empty()));
|
||||
|
||||
LambdaFunction fn = new LambdaFunction();
|
||||
fn.setFunctionName("efs-root-fn");
|
||||
fn.setFunctionArn("arn:aws:lambda:us-east-1:000000000000:function:efs-root-fn");
|
||||
fn.setRuntime("nodejs20.x");
|
||||
fn.setHandler("index.handler");
|
||||
fn.setCodeLocalPath(codePath.toString());
|
||||
fn.setFileSystemConfigs(List.of(new LambdaFileSystemConfig(accessPointArn, "/mnt/efs")));
|
||||
|
||||
launcher.launch(fn);
|
||||
|
||||
ContainerSpec spec = captureRealContainerSpec();
|
||||
Mount mount = spec.mounts().stream()
|
||||
.filter(candidate -> "/mnt/efs".equals(candidate.getTarget()))
|
||||
.findFirst()
|
||||
.orElseThrow();
|
||||
assertEquals("floci-efs-fs-abc", mount.getSource());
|
||||
assertEquals(null, mount.getVolumeOptions());
|
||||
assertEquals(null, spec.user());
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -1631,7 +1667,7 @@ class ContainerLauncherTest {
|
||||
ecrRegistryManager,
|
||||
mock(io.github.hectorvent.floci.services.lambda.LambdaLayerService.class),
|
||||
new LaunchedContainerAwsEnv(reachableEndpoint),
|
||||
executionRoleCredentials);
|
||||
executionRoleCredentials, efsMounts);
|
||||
|
||||
stubExtensionDiscovery("otel-collector");
|
||||
// Feed a real stdout frame through whatever callback the launcher hands to execStartCmd.
|
||||
|
||||
-27
@@ -100,33 +100,6 @@ class ContainerLauncherVolumeNamingTest {
|
||||
"a later lastModified must produce a different volume name");
|
||||
}
|
||||
|
||||
@Test
|
||||
void efsVolumeNameIncludesStableHashOfCompleteArn() {
|
||||
String arn = "arn:aws:elasticfilesystem:us-east-1:000000000000:"
|
||||
+ "access-point/fsap-0123456789abcdef0";
|
||||
|
||||
String volumeName = ContainerLauncher.efsVolumeName(arn);
|
||||
|
||||
assertEquals("floci-efs-fsap-0123456789abcdef0-"
|
||||
+ "9d6eafd2aec94d4518a004f005725b4b3c673c1506436bb7368cfd5450fc0810",
|
||||
volumeName);
|
||||
assertEquals(volumeName, ContainerLauncher.efsVolumeName(arn));
|
||||
assertTrue(DOCKER_VOLUME_NAME.matcher(volumeName).matches(),
|
||||
"EFS volume name must be Docker-safe, was: " + volumeName);
|
||||
}
|
||||
|
||||
@Test
|
||||
void efsVolumeNameDistinguishesSameResourceIdAcrossArns() {
|
||||
String eastArn = "arn:aws:elasticfilesystem:us-east-1:000000000000:"
|
||||
+ "access-point/fsap-0123456789abcdef0";
|
||||
String westArn = "arn:aws:elasticfilesystem:us-west-2:111111111111:"
|
||||
+ "access-point/fsap-0123456789abcdef0";
|
||||
|
||||
assertNotEquals(ContainerLauncher.efsVolumeName(eastArn),
|
||||
ContainerLauncher.efsVolumeName(westArn),
|
||||
"the complete ARN must participate in volume identity");
|
||||
}
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
|
||||
@@ -287,7 +287,8 @@ class SmtpRelayServerTest {
|
||||
String command = line.toUpperCase(Locale.ROOT);
|
||||
if (command.startsWith("EHLO")) {
|
||||
reply(out, "250-floci-stub");
|
||||
reply(out, "250 AUTH PLAIN");
|
||||
// Gmail also offers OAuth sign-in, which a password must never be sent as.
|
||||
reply(out, "250 AUTH PLAIN XOAUTH2 OAUTHBEARER");
|
||||
} else if (command.startsWith("HELO")) {
|
||||
reply(out, "250 floci-stub");
|
||||
} else if (command.startsWith("AUTH")) {
|
||||
|
||||
@@ -434,6 +434,7 @@ floci:
|
||||
enabled: true
|
||||
efs:
|
||||
enabled: true
|
||||
nfs-servers: false
|
||||
organizations:
|
||||
enabled: true
|
||||
rum:
|
||||
|
||||
Reference in New Issue
Block a user