second first commit

2026-09-22 08:57:00 -04:00
parent e88d7e2ff3
commit 0aef7fb2d8
4766 changed files with 1236986 additions and 0 deletions
@@ -0,0 +1,377 @@
+# CloudWatch
+
+Floci supports both CloudWatch Logs and CloudWatch Metrics.
+
+---
+
+## CloudWatch Logs
+
+**Protocol:** JSON 1.1 (`X-Amz-Target: Logs.*`)
+**Endpoint:** `POST http://localhost:4566/`
+
+### Supported Actions
+
+| Action | Description |
+|---|---|
+| `CreateLogGroup` | Create a log group |
+| `DeleteLogGroup` | Delete a log group |
+| `PutLogGroupDeletionProtection` | Enable or disable deletion protection for a log group by name or ARN |
+| `DescribeLogGroups` | List log groups |
+| `CreateLogStream` | Create a log stream inside a log group |
+| `DeleteLogStream` | Delete a log stream |
+| `DescribeLogStreams` | List log streams in a group |
+| `PutLogEvents` | Write log events to a stream |
+| `GetLogEvents` | Read log events from a stream |
+| `FilterLogEvents` | Search log events with a filter pattern |
+| `PutRetentionPolicy` | Set log retention (days) |
+| `DeleteRetentionPolicy` | Remove log retention policy |
+| `AssociateKmsKey` | Associate a KMS key with a log group (`logGroupName` or `resourceIdentifier`) |
+| `DisassociateKmsKey` | Remove a log group's KMS key association |
+| `TagLogGroup` | Tag a log group |
+| `UntagLogGroup` | Remove tags |
+| `ListTagsLogGroup` | List tags |
+| `TagResource` | Tag a log group by ARN |
+| `UntagResource` | Remove tags from a log group by ARN |
+| `ListTagsForResource` | List tags for a log group ARN |
+| `PutSubscriptionFilter` | Create or update a subscription filter (stored only, see note below) |
+| `DescribeSubscriptionFilters` | List subscription filters on a log group |
+| `DeleteSubscriptionFilter` | Delete a subscription filter |
+| `PutMetricFilter` | Create or replace a metric filter on a log group (see [Metric filters](#metric-filters)) |
+| `DescribeMetricFilters` | List metric filters by log group and name prefix, or by metric name and namespace |
+| `DeleteMetricFilter` | Delete a metric filter |
+| `TestMetricFilter` | Run a filter pattern over sample messages and return the matches with their extracted values |
+| `PutResourcePolicy` | Create or update an account-level resource policy |
+| `DescribeResourcePolicies` | List account-level resource policies |
+| `PutDestination` | Create or update a cross-account subscription destination |
+| `PutDestinationPolicy` | Create or update the access policy for a destination |
+| `PutAccountPolicy` | Create or update an account-level Logs policy |
+| `DescribeAccountPolicies` | List account-level Logs policies by type and optional name |
+| `GetDataProtectionPolicy` | Return the resolved log group identifier (see note below) |
+| `StartQuery` | Start a Logs Insights query (see [Logs Insights](#logs-insights)) |
+| `GetQueryResults` | Get the status and results of a Logs Insights query |
+| `StopQuery` | Stop a query that has not completed yet |
+
+Log group deletion protection defaults to disabled and is persisted with the log group. When it is
+enabled, `DeleteLogGroup` returns `ValidationException` until protection is explicitly disabled
+with `PutLogGroupDeletionProtection`.
+
+Three actions are currently simplified:
+
+- **`PutSubscriptionFilter`** stores the filter so that `DescribeSubscriptionFilters`
+  returns it, but log events are **not** forwarded to the destination ARN. Lambda,
+  Kinesis, and Firehose subscription destinations are not wired up.
+- **`GetDataProtectionPolicy`** does not model data-protection policies. It returns
+  HTTP 200 with the resolved `logGroupIdentifier` and no `policyDocument` — including for
+  a log group that does not exist, where real AWS returns `ResourceNotFoundException`.
+- **`FilterLogEvents`** matches `filterPattern` as a plain substring of the message. The full
+  filter pattern syntax described below is applied by metric filters.
+
+### Metric filters {#metric-filters}
+
+A metric filter turns newly ingested log events into CloudWatch metric samples. Each matching
+event contributes the literal `metricValue`, or its referenced field (`$.latency` in JSON,
+`$size` in space-delimited logs), at the **event timestamp**, not ingestion or filter-creation time.
+Events ingested before the filter existed are not replayed. Backdated and future event timestamps
+remain backdated and future metric timestamps.
+
+A configured `defaultValue` contributes **once per nonmatching event**, including alongside matches
+in the same batch or minute. A matching event with a missing or JSON-null metric-value field also
+uses the default; a present nonnumeric value is skipped, not replaced by the default. Without a
+default, those nonmatches and missing/null values produce no sample. No events means no samples.
+Healthy publication is immediate and needs no subsequent ingestion, reads, or timer. For example,
+pattern `ERROR`, value 3 and default 7 produce Sum 13 / SampleCount 3 for `ERROR, INFO, ERROR`,
+and three quiet nonmatches produce Sum 21 / SampleCount 3. A late match appends its value without
+retracting earlier defaults. CloudWatch aggregates these contributions by period; Floci has no
+minute-default accumulator or deduplication history.
+
+Ordinary transformation dimensions are all-or-none: a complete configured set is emitted, while
+a missing member produces a dimensionless sample, not a partial set. Requested system dimensions
+(`@aws.account`, `@aws.region`) are populated on matches using the resolved receiving account and
+region, including regular noncentralized Logs ingestion. **Pattern-nonmatch defaults are
+dimensionless**, even when system dimensions are requested. These scenarios are supported by the
+recorded [live AWS observations](cloudwatch-metric-filters-verification.md).
+
+**Inferred emulator policy, not a live AWS claim:** matching missing/null metric-value fallbacks
+retain requested system dimensions. Likewise, an incomplete ordinary dimension set is discarded
+as a unit while requested system dimensions are retained. Null ordinary dimension values are
+treated as missing; empty strings retain the existing scalar-extraction behavior. These combinations
+were not measured by the recorded probes. Floci does not infer centralized source provenance.
+
+**Publication runs off the request thread:** `PutLogEvents` evaluates the group's filters as they
+are at that moment, builds one immutable sample per contributing event (canonical account, region,
+group, filter, transformation snapshot, event timestamp and a stable internal publication ID) and
+hands them to a process-wide queue. One managed worker writes them to the Metrics store, so
+ingestion latency does not depend on the Metrics sink and a `PutMetricFilter` or a reset waits for
+at most one sample write. The queue holds **100,000 samples**; a batch that does not fit is written
+on the request thread under the filter-store monitor instead of being dropped, so accepted samples
+are never lost to the bound and only an overfull queue makes the request wait on the sink.
+A read issued immediately after `PutLogEvents` may run before the worker has written the sample,
+as on AWS; SDK tests should poll. Shutdown drains the queue before storage shuts down.
+
+**Runtime publication failures:** accepted Logs ingestion remains successful if the Metrics sink
+reports a failure. Each failed sample keeps its snapshot and publication ID, and retries replace
+that same metric-store key, making partial or ambiguous writes safe without deduplicating distinct
+identical log events. The existing AWS `PutMetricData` APIs continue appending samples as before.
+
+The same worker retries failed publications at one-second intervals while both Logs and Metrics
+services are enabled; idle ticks never create defaults. The retry map holds at most **10,000
+samples**, not 10,000 batches. This fixed conservative limit bounds outage memory without
+introducing a configuration surface. When full, new failed samples are dropped with an **ERROR**
+containing account, region, group, filter and dropped count; already queued samples and their IDs are
+preserved. Successful retries and cancellations release capacity. Each filter-owned outage logs
+one detailed stack, followed by a stack-free aggregate warning every 60 failed retry ticks.
+Recovery, cancellation and reset release that suppression. This is bounded best effort, not
+lossless delivery through an indefinite outage. Pending work is in memory only; crash-durable
+pending publication is not implemented. Storage failures that a backend only logs rather than
+throws are not observable to this retry path; configured storage durability semantics are unchanged.
+
+A filter update or deletion affects new ingestion only: samples already accepted for a batch still
+publish under the definition that was current at ingestion, as on AWS, and deleting a filter or
+group cancels only its own account/region-scoped failed retries without retracting stored metrics.
+Reset pauses publication, waits for the active sample write, discards queued and failed samples,
+wipes storage and resumes the same live service. The canonical filter-store monitor serializes
+each sample write with update/delete; reset hooks release it before the storage-factory wipe to
+avoid lock inversion. Shutdown writes the queued samples, abandons failed retries and stops the
+worker (up to five seconds to await executor termination). Storage backends expose synchronous
+operations without a cancellation deadline; a stuck backend can delay that drain. Runtime
+cancellation is not a crash-recovery guarantee.
+
+Samples are readable with `GetMetricStatistics` and `GetMetricData`, and alarms evaluate normally.
+
+The filter pattern syntax follows the AWS reference:
+
+- Terms: `ERROR ARGUMENTS` (all present), `?ERROR ?ARGUMENTS` (any present), `ERROR -ARGUMENTS`
+  (exclusion), `"exact phrase"`. Terms are case-sensitive substrings of the message.
+- Regular expressions between percent signs, `%^[hc]at%`, with the operators and escapes AWS
+  allows. Parentheses and characters outside ASCII are rejected, as on AWS.
+- JSON patterns: `{ $.eventType = "UpdateTrail" && $.code >= 400 }`, with `=`, `!=`, `<`, `<=`,
+  `>`, `>=`, `IS NULL`, `IS TRUE`, `IS FALSE`, `NOT EXISTS`, `&&`, `||`, parentheses, array indexes,
+  `[*]` and `.*` wildcards, and `$.['a.b']` for a property with a dot in its name. AWS's wildcard
+  quotas apply: one per property selector and three per pattern.
+- Space-delimited patterns: `[ip, ..., status_code = 4*, bytes]`, with named fields, `...` for any
+  number of fields, conditions on any field, `w1`/`w2` indicators, and `%regex%` values. Text
+  between double quotes or square brackets is one field.
+
+`PutMetricFilter` applies the rules AWS applies: the log group must exist, the pattern must parse,
+at most two regular expressions per pattern, exactly one transformation whose `metricValue` is a
+number or a single-valued field reference, at most three dimensions in total, and 100 metric
+filters per log group. Scalar JSON metric-value and dimension references need not be mentioned
+in the filter pattern. Wildcard metric-value references such as `$.values[*]` are rejected;
+wildcards remain available in the filter pattern itself. Ordinary dimensions are
+available only for JSON or space-delimited patterns and cannot be combined with a `defaultValue`;
+system-only dimensions can. The shared group regex quota counts up to five **regex-bearing filters**
+across metric and subscription filters, independently of the two expressions allowed per pattern.
+This shared pool is documentation-backed, not a new live mixed-writer claim. `TestMetricFilter`
+reports one-based event numbers. JSON matches have empty public `extractedValues`; space-delimited
+matches expose named fields as `$name` and positional fields as `$1`, `$2` and so on.
+
+`fieldSelectionCriteria` selects which batches a filter processes from the system fields
+`@aws.account` and `@aws.region`, with `=`, `!=`, `IN`, `NOT IN` and the `AND` and `OR` the API
+documents, as in `@aws.account IN ["111111111111"]`. Both fields describe the ingested batch rather
+than the individual event, so one evaluation decides the batch. A criterion that does not parse is
+rejected when the filter is stored.
+
+`emitSystemFieldDimensions` adds `@aws.account` and `@aws.region` on matching contributions as
+described above. They count toward the same combined limit of three dimensions.
+
+`applyOnTransformedLogs` is stored and returned unchanged. Request acceptance was verified on AWS,
+including `true` on the tested group without a group-level transformer. Floci does not execute
+group-level or account-level log transformations, including stored `TRANSFORMER_POLICY`
+configuration. Accepting the flag therefore does not establish transformed-log processing or
+full compatibility with an AWS group that has an active transformer.
+
+See [Metric-filter contract verification](cloudwatch-metric-filters-verification.md) for the
+scenarios tagged **Verified on Live AWS**, the recorded fixtures and their evidence boundaries.
+
+### Logs Insights {#logs-insights}
+
+`StartQuery` / `GetQueryResults` / `StopQuery` run a **subset** of the Logs Insights
+query language. Supported commands:
+
+| Command | Notes |
+|---|---|
+| `fields a, b, ...` | Projection. Defaults to `@timestamp, @message`. `display` is an alias |
+| `filter <field> = 'v'` | Equality. `!=` and `==` are also accepted; `where` is an alias |
+| `sort <field> [asc\|desc]` | `order` is an alias |
+| `dedup <field, ...>` | Keeps the first row per unique tuple, applied after sorting |
+| `limit N` | The effective cap is the smallest of this value, the `StartQuery` `limit` parameter, and `FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_EVENTS_PER_QUERY` |
+
+Fields may be `@timestamp`, `@message`, `@ingestionTime`, `@ptr`, or a dotted path into
+a JSON log message (for example `level` or `params.job_id`). A `@ptr` column is always
+included in each result row, appended unless `fields` already names it.
+
+Unsupported syntax never fails the query, so it is worth knowing how each case degrades:
+
+| Input | Result |
+|---|---|
+| An unsupported command (`stats`, `parse`, ...) | Skipped with a warning in the server log. No aggregation happens |
+| A `filter` whose operator is not `=`, `!=` or `==` — for example `<`, `<=`, `>`, `>=`, `like /ERROR/` or `=~ /ERROR/` | The whole stage is dropped with a warning, so **every** row is returned |
+| A `filter` combining conditions with `and` / `or` — for example `filter level = 'ERROR' and status = 200` | Only the leftmost operator is parsed; the rest of the line becomes the compared value, so nothing matches and you get **no** rows. No warning is logged |
+| A projected field that does not exist | Rendered as an empty string. No warning |
+| A `sort` direction other than `asc` / `desc` | Treated as ascending. No warning |
+
+In short, a query can come back either wider or narrower than intended without any error. When a
+result set looks wrong, check the server log for `Ignoring unsupported Logs Insights ...` — and note
+that the compound-filter case above produces no log line at all.
+
+For simple substring matching, `FilterLogEvents` is the more predictable option today. Note that
+Floci matches `--filter-pattern` as a plain substring of the message; the real filter-pattern
+syntax (`?ERROR ?WARN`, `{ $.level = "ERROR" }`, and so on) is not parsed.
+
+### Reading events past the limit
+
+`FilterLogEvents` and `GetLogEvents` both page, and they signal the end of the results differently
+because the AWS APIs do.
+
+`FilterLogEvents` pages forward only. Its `nextToken` is an `f/<index>` offset into the matched set,
+so the offset counts matches, not stored events: a request narrowed by `--filter-pattern`,
+`--start-time` or `--log-stream-names` pages through only what it matched. A missing token starts
+from the oldest match, and **a response with no `nextToken` means pagination is finished**, so the
+final page omits it. An unrecognized, non-numeric or negative token returns
+`InvalidParameterException` (400). `startFromHead` is not supported, so results always run oldest
+first.
+
+`GetLogEvents` pages in both directions with `f/<index>` and `b/<index>`, and always returns
+`nextForwardToken` and `nextBackwardToken`. It signals the end by returning the same token it was
+given rather than by omitting it, which is what its SDK paginators expect.
+
+### Configuration
+
+| Variable | Default | Description |
+|---|---|---|
+| `FLOCI_SERVICES_CLOUDWATCHLOGS_ENABLED` | `true` | Enable or disable the CloudWatch Logs service |
+| `FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_EVENTS_PER_QUERY` | `10000` | Maximum events returned per `FilterLogEvents` / `GetLogEvents` call, and the upper bound for a Logs Insights `limit` |
+| `FLOCI_SERVICES_CLOUDWATCHLOGS_QUERY_COMPLETION_DELAY_MS` | `0` | Artificial Logs Insights query delay. With `0` a query completes immediately. A positive value emulates the real asynchronous lifecycle (`Running` → `Complete` after the delay), which also makes `StopQuery` on a still-running query return `success=true` |
+
+### Examples
+
+```bash
+export AWS_ENDPOINT_URL=http://localhost:4566
+
+# Create a log group and stream
+aws logs create-log-group --log-group-name /app/backend --endpoint-url $AWS_ENDPOINT_URL
+aws logs create-log-stream \
+  --log-group-name /app/backend \
+  --log-stream-name 2025/01/app-1 \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# Write log events
+TIMESTAMP=$(date +%s%3N)   # milliseconds
+aws logs put-log-events \
+  --log-group-name /app/backend \
+  --log-stream-name 2025/01/app-1 \
+  --log-events "[{\"timestamp\":$TIMESTAMP,\"message\":\"Service started\"}]" \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# Read log events
+aws logs get-log-events \
+  --log-group-name /app/backend \
+  --log-stream-name 2025/01/app-1 \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# Search logs
+aws logs filter-log-events \
+  --log-group-name /app/backend \
+  --filter-pattern "ERROR" \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# Run a Logs Insights query
+QUERY_ID=$(aws logs start-query \
+  --log-group-name /app/backend \
+  --start-time $(($(date +%s) - 3600)) \
+  --end-time $(date +%s) \
+  --query-string 'fields @timestamp, @message | sort @timestamp desc | limit 20' \
+  --query queryId --output text \
+  --endpoint-url $AWS_ENDPOINT_URL)
+
+aws logs get-query-results \
+  --query-id "$QUERY_ID" \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# Set retention
+aws logs put-retention-policy \
+  --log-group-name /app/backend \
+  --retention-in-days 30 \
+  --endpoint-url $AWS_ENDPOINT_URL
+```
+
+---
+
+## CloudWatch Metrics {#metrics}
+
+**Protocol:** Query (XML) and JSON 1.1 (both supported)
+**Endpoint:** `POST http://localhost:4566/`
+
+### Supported Actions
+
+| Action | Description |
+|---|---|
+| `PutMetricData` | Publish custom metrics |
+| `ListMetrics` | List available metrics |
+| `GetMetricStatistics` | Get metric statistics (Average, Sum, etc.) |
+| `GetMetricData` | Query metrics with math expressions |
+| `PutMetricAlarm` | Create a metric alarm |
+| `DescribeAlarms` | List alarms |
+| `DeleteAlarms` | Delete alarms |
+| `SetAlarmState` | Manually set alarm state |
+| `PutMetricStream` | Create or update a metric stream definition |
+| `GetMetricStream` | Read a metric stream definition |
+| `ListMetricStreams` | List metric streams |
+| `DeleteMetricStream` | Delete a metric stream |
+| `StartMetricStreams` | Move metric streams to `running` |
+| `StopMetricStreams` | Move metric streams to `stopped` |
+| `PutDashboard` | Create a dashboard, or replace its body when the name is taken (tags apply on create only) |
+| `GetDashboard` | Read a dashboard body and ARN |
+| `ListDashboards` | List dashboards, optionally by name prefix |
+| `DeleteDashboards` | Delete dashboards by name |
+| `TagResource` | Tag an alarm or dashboard by ARN |
+| `UntagResource` | Remove tags from an alarm or dashboard |
+| `ListTagsForResource` | List the tags of an alarm or dashboard |
+
+Metric streams are stored as definitions with their `running` or `stopped` state. Floci never
+delivers metrics to the Firehose delivery stream a metric stream names.
+
+### Examples
+
+```bash
+export AWS_ENDPOINT_URL=http://localhost:4566
+
+# Publish a custom metric
+aws cloudwatch put-metric-data \
+  --namespace MyApp \
+  --metric-data '[{
+    "MetricName": "RequestCount",
+    "Value": 42,
+    "Unit": "Count",
+    "Dimensions": [{"Name":"Service","Value":"api"}]
+  }]' \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# List metrics
+aws cloudwatch list-metrics \
+  --namespace MyApp \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# Get statistics
+aws cloudwatch get-metric-statistics \
+  --namespace MyApp \
+  --metric-name RequestCount \
+  --dimensions Name=Service,Value=api \
+  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ) \
+  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
+  --period 300 \
+  --statistics Sum \
+  --endpoint-url $AWS_ENDPOINT_URL
+
+# Create an alarm
+aws cloudwatch put-metric-alarm \
+  --alarm-name high-error-rate \
+  --metric-name ErrorCount \
+  --namespace MyApp \
+  --statistic Sum \
+  --period 60 \
+  --threshold 10 \
+  --comparison-operator GreaterThanThreshold \
+  --evaluation-periods 1 \
+  --endpoint-url $AWS_ENDPOINT_URL
+```