Zakaria
378 lines
20 KiB
Markdown
378 lines
20 KiB
Markdown
# CloudWatch
|
|
|
|
Floci supports both CloudWatch Logs and CloudWatch Metrics.
|
|
|
|
---
|
|
|
|
## CloudWatch Logs
|
|
|
|
**Protocol:** JSON 1.1 (`X-Amz-Target: Logs.*`)
|
|
**Endpoint:** `POST http://localhost:4566/`
|
|
|
|
### Supported Actions
|
|
|
|
| Action | Description |
|
|
|---|---|
|
|
| `CreateLogGroup` | Create a log group |
|
|
| `DeleteLogGroup` | Delete a log group |
|
|
| `PutLogGroupDeletionProtection` | Enable or disable deletion protection for a log group by name or ARN |
|
|
| `DescribeLogGroups` | List log groups |
|
|
| `CreateLogStream` | Create a log stream inside a log group |
|
|
| `DeleteLogStream` | Delete a log stream |
|
|
| `DescribeLogStreams` | List log streams in a group |
|
|
| `PutLogEvents` | Write log events to a stream |
|
|
| `GetLogEvents` | Read log events from a stream |
|
|
| `FilterLogEvents` | Search log events with a filter pattern |
|
|
| `PutRetentionPolicy` | Set log retention (days) |
|
|
| `DeleteRetentionPolicy` | Remove log retention policy |
|
|
| `AssociateKmsKey` | Associate a KMS key with a log group (`logGroupName` or `resourceIdentifier`) |
|
|
| `DisassociateKmsKey` | Remove a log group's KMS key association |
|
|
| `TagLogGroup` | Tag a log group |
|
|
| `UntagLogGroup` | Remove tags |
|
|
| `ListTagsLogGroup` | List tags |
|
|
| `TagResource` | Tag a log group by ARN |
|
|
| `UntagResource` | Remove tags from a log group by ARN |
|
|
| `ListTagsForResource` | List tags for a log group ARN |
|
|
| `PutSubscriptionFilter` | Create or update a subscription filter (stored only, see note below) |
|
|
| `DescribeSubscriptionFilters` | List subscription filters on a log group |
|
|
| `DeleteSubscriptionFilter` | Delete a subscription filter |
|
|
| `PutMetricFilter` | Create or replace a metric filter on a log group (see [Metric filters](#metric-filters)) |
|
|
| `DescribeMetricFilters` | List metric filters by log group and name prefix, or by metric name and namespace |
|
|
| `DeleteMetricFilter` | Delete a metric filter |
|
|
| `TestMetricFilter` | Run a filter pattern over sample messages and return the matches with their extracted values |
|
|
| `PutResourcePolicy` | Create or update an account-level resource policy |
|
|
| `DescribeResourcePolicies` | List account-level resource policies |
|
|
| `PutDestination` | Create or update a cross-account subscription destination |
|
|
| `PutDestinationPolicy` | Create or update the access policy for a destination |
|
|
| `PutAccountPolicy` | Create or update an account-level Logs policy |
|
|
| `DescribeAccountPolicies` | List account-level Logs policies by type and optional name |
|
|
| `GetDataProtectionPolicy` | Return the resolved log group identifier (see note below) |
|
|
| `StartQuery` | Start a Logs Insights query (see [Logs Insights](#logs-insights)) |
|
|
| `GetQueryResults` | Get the status and results of a Logs Insights query |
|
|
| `StopQuery` | Stop a query that has not completed yet |
|
|
|
|
Log group deletion protection defaults to disabled and is persisted with the log group. When it is
|
|
enabled, `DeleteLogGroup` returns `ValidationException` until protection is explicitly disabled
|
|
with `PutLogGroupDeletionProtection`.
|
|
|
|
Three actions are currently simplified:
|
|
|
|
- **`PutSubscriptionFilter`** stores the filter so that `DescribeSubscriptionFilters`
|
|
returns it, but log events are **not** forwarded to the destination ARN. Lambda,
|
|
Kinesis, and Firehose subscription destinations are not wired up.
|
|
- **`GetDataProtectionPolicy`** does not model data-protection policies. It returns
|
|
HTTP 200 with the resolved `logGroupIdentifier` and no `policyDocument` — including for
|
|
a log group that does not exist, where real AWS returns `ResourceNotFoundException`.
|
|
- **`FilterLogEvents`** matches `filterPattern` as a plain substring of the message. The full
|
|
filter pattern syntax described below is applied by metric filters.
|
|
|
|
### Metric filters {#metric-filters}
|
|
|
|
A metric filter turns newly ingested log events into CloudWatch metric samples. Each matching
|
|
event contributes the literal `metricValue`, or its referenced field (`$.latency` in JSON,
|
|
`$size` in space-delimited logs), at the **event timestamp**, not ingestion or filter-creation time.
|
|
Events ingested before the filter existed are not replayed. Backdated and future event timestamps
|
|
remain backdated and future metric timestamps.
|
|
|
|
A configured `defaultValue` contributes **once per nonmatching event**, including alongside matches
|
|
in the same batch or minute. A matching event with a missing or JSON-null metric-value field also
|
|
uses the default; a present nonnumeric value is skipped, not replaced by the default. Without a
|
|
default, those nonmatches and missing/null values produce no sample. No events means no samples.
|
|
Healthy publication is immediate and needs no subsequent ingestion, reads, or timer. For example,
|
|
pattern `ERROR`, value 3 and default 7 produce Sum 13 / SampleCount 3 for `ERROR, INFO, ERROR`,
|
|
and three quiet nonmatches produce Sum 21 / SampleCount 3. A late match appends its value without
|
|
retracting earlier defaults. CloudWatch aggregates these contributions by period; Floci has no
|
|
minute-default accumulator or deduplication history.
|
|
|
|
Ordinary transformation dimensions are all-or-none: a complete configured set is emitted, while
|
|
a missing member produces a dimensionless sample, not a partial set. Requested system dimensions
|
|
(`@aws.account`, `@aws.region`) are populated on matches using the resolved receiving account and
|
|
region, including regular noncentralized Logs ingestion. **Pattern-nonmatch defaults are
|
|
dimensionless**, even when system dimensions are requested. These scenarios are supported by the
|
|
recorded [live AWS observations](cloudwatch-metric-filters-verification.md).
|
|
|
|
**Inferred emulator policy, not a live AWS claim:** matching missing/null metric-value fallbacks
|
|
retain requested system dimensions. Likewise, an incomplete ordinary dimension set is discarded
|
|
as a unit while requested system dimensions are retained. Null ordinary dimension values are
|
|
treated as missing; empty strings retain the existing scalar-extraction behavior. These combinations
|
|
were not measured by the recorded probes. Floci does not infer centralized source provenance.
|
|
|
|
**Publication runs off the request thread:** `PutLogEvents` evaluates the group's filters as they
|
|
are at that moment, builds one immutable sample per contributing event (canonical account, region,
|
|
group, filter, transformation snapshot, event timestamp and a stable internal publication ID) and
|
|
hands them to a process-wide queue. One managed worker writes them to the Metrics store, so
|
|
ingestion latency does not depend on the Metrics sink and a `PutMetricFilter` or a reset waits for
|
|
at most one sample write. The queue holds **100,000 samples**; a batch that does not fit is written
|
|
on the request thread under the filter-store monitor instead of being dropped, so accepted samples
|
|
are never lost to the bound and only an overfull queue makes the request wait on the sink.
|
|
A read issued immediately after `PutLogEvents` may run before the worker has written the sample,
|
|
as on AWS; SDK tests should poll. Shutdown drains the queue before storage shuts down.
|
|
|
|
**Runtime publication failures:** accepted Logs ingestion remains successful if the Metrics sink
|
|
reports a failure. Each failed sample keeps its snapshot and publication ID, and retries replace
|
|
that same metric-store key, making partial or ambiguous writes safe without deduplicating distinct
|
|
identical log events. The existing AWS `PutMetricData` APIs continue appending samples as before.
|
|
|
|
The same worker retries failed publications at one-second intervals while both Logs and Metrics
|
|
services are enabled; idle ticks never create defaults. The retry map holds at most **10,000
|
|
samples**, not 10,000 batches. This fixed conservative limit bounds outage memory without
|
|
introducing a configuration surface. When full, new failed samples are dropped with an **ERROR**
|
|
containing account, region, group, filter and dropped count; already queued samples and their IDs are
|
|
preserved. Successful retries and cancellations release capacity. Each filter-owned outage logs
|
|
one detailed stack, followed by a stack-free aggregate warning every 60 failed retry ticks.
|
|
Recovery, cancellation and reset release that suppression. This is bounded best effort, not
|
|
lossless delivery through an indefinite outage. Pending work is in memory only; crash-durable
|
|
pending publication is not implemented. Storage failures that a backend only logs rather than
|
|
throws are not observable to this retry path; configured storage durability semantics are unchanged.
|
|
|
|
A filter update or deletion affects new ingestion only: samples already accepted for a batch still
|
|
publish under the definition that was current at ingestion, as on AWS, and deleting a filter or
|
|
group cancels only its own account/region-scoped failed retries without retracting stored metrics.
|
|
Reset pauses publication, waits for the active sample write, discards queued and failed samples,
|
|
wipes storage and resumes the same live service. The canonical filter-store monitor serializes
|
|
each sample write with update/delete; reset hooks release it before the storage-factory wipe to
|
|
avoid lock inversion. Shutdown writes the queued samples, abandons failed retries and stops the
|
|
worker (up to five seconds to await executor termination). Storage backends expose synchronous
|
|
operations without a cancellation deadline; a stuck backend can delay that drain. Runtime
|
|
cancellation is not a crash-recovery guarantee.
|
|
|
|
Samples are readable with `GetMetricStatistics` and `GetMetricData`, and alarms evaluate normally.
|
|
|
|
The filter pattern syntax follows the AWS reference:
|
|
|
|
- Terms: `ERROR ARGUMENTS` (all present), `?ERROR ?ARGUMENTS` (any present), `ERROR -ARGUMENTS`
|
|
(exclusion), `"exact phrase"`. Terms are case-sensitive substrings of the message.
|
|
- Regular expressions between percent signs, `%^[hc]at%`, with the operators and escapes AWS
|
|
allows. Parentheses and characters outside ASCII are rejected, as on AWS.
|
|
- JSON patterns: `{ $.eventType = "UpdateTrail" && $.code >= 400 }`, with `=`, `!=`, `<`, `<=`,
|
|
`>`, `>=`, `IS NULL`, `IS TRUE`, `IS FALSE`, `NOT EXISTS`, `&&`, `||`, parentheses, array indexes,
|
|
`[*]` and `.*` wildcards, and `$.['a.b']` for a property with a dot in its name. AWS's wildcard
|
|
quotas apply: one per property selector and three per pattern.
|
|
- Space-delimited patterns: `[ip, ..., status_code = 4*, bytes]`, with named fields, `...` for any
|
|
number of fields, conditions on any field, `w1`/`w2` indicators, and `%regex%` values. Text
|
|
between double quotes or square brackets is one field.
|
|
|
|
`PutMetricFilter` applies the rules AWS applies: the log group must exist, the pattern must parse,
|
|
at most two regular expressions per pattern, exactly one transformation whose `metricValue` is a
|
|
number or a single-valued field reference, at most three dimensions in total, and 100 metric
|
|
filters per log group. Scalar JSON metric-value and dimension references need not be mentioned
|
|
in the filter pattern. Wildcard metric-value references such as `$.values[*]` are rejected;
|
|
wildcards remain available in the filter pattern itself. Ordinary dimensions are
|
|
available only for JSON or space-delimited patterns and cannot be combined with a `defaultValue`;
|
|
system-only dimensions can. The shared group regex quota counts up to five **regex-bearing filters**
|
|
across metric and subscription filters, independently of the two expressions allowed per pattern.
|
|
This shared pool is documentation-backed, not a new live mixed-writer claim. `TestMetricFilter`
|
|
reports one-based event numbers. JSON matches have empty public `extractedValues`; space-delimited
|
|
matches expose named fields as `$name` and positional fields as `$1`, `$2` and so on.
|
|
|
|
`fieldSelectionCriteria` selects which batches a filter processes from the system fields
|
|
`@aws.account` and `@aws.region`, with `=`, `!=`, `IN`, `NOT IN` and the `AND` and `OR` the API
|
|
documents, as in `@aws.account IN ["111111111111"]`. Both fields describe the ingested batch rather
|
|
than the individual event, so one evaluation decides the batch. A criterion that does not parse is
|
|
rejected when the filter is stored.
|
|
|
|
`emitSystemFieldDimensions` adds `@aws.account` and `@aws.region` on matching contributions as
|
|
described above. They count toward the same combined limit of three dimensions.
|
|
|
|
`applyOnTransformedLogs` is stored and returned unchanged. Request acceptance was verified on AWS,
|
|
including `true` on the tested group without a group-level transformer. Floci does not execute
|
|
group-level or account-level log transformations, including stored `TRANSFORMER_POLICY`
|
|
configuration. Accepting the flag therefore does not establish transformed-log processing or
|
|
full compatibility with an AWS group that has an active transformer.
|
|
|
|
See [Metric-filter contract verification](cloudwatch-metric-filters-verification.md) for the
|
|
scenarios tagged **Verified on Live AWS**, the recorded fixtures and their evidence boundaries.
|
|
|
|
### Logs Insights {#logs-insights}
|
|
|
|
`StartQuery` / `GetQueryResults` / `StopQuery` run a **subset** of the Logs Insights
|
|
query language. Supported commands:
|
|
|
|
| Command | Notes |
|
|
|---|---|
|
|
| `fields a, b, ...` | Projection. Defaults to `@timestamp, @message`. `display` is an alias |
|
|
| `filter <field> = 'v'` | Equality. `!=` and `==` are also accepted; `where` is an alias |
|
|
| `sort <field> [asc\|desc]` | `order` is an alias |
|
|
| `dedup <field, ...>` | Keeps the first row per unique tuple, applied after sorting |
|
|
| `limit N` | The effective cap is the smallest of this value, the `StartQuery` `limit` parameter, and `FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_EVENTS_PER_QUERY` |
|
|
|
|
Fields may be `@timestamp`, `@message`, `@ingestionTime`, `@ptr`, or a dotted path into
|
|
a JSON log message (for example `level` or `params.job_id`). A `@ptr` column is always
|
|
included in each result row, appended unless `fields` already names it.
|
|
|
|
Unsupported syntax never fails the query, so it is worth knowing how each case degrades:
|
|
|
|
| Input | Result |
|
|
|---|---|
|
|
| An unsupported command (`stats`, `parse`, ...) | Skipped with a warning in the server log. No aggregation happens |
|
|
| A `filter` whose operator is not `=`, `!=` or `==` — for example `<`, `<=`, `>`, `>=`, `like /ERROR/` or `=~ /ERROR/` | The whole stage is dropped with a warning, so **every** row is returned |
|
|
| A `filter` combining conditions with `and` / `or` — for example `filter level = 'ERROR' and status = 200` | Only the leftmost operator is parsed; the rest of the line becomes the compared value, so nothing matches and you get **no** rows. No warning is logged |
|
|
| A projected field that does not exist | Rendered as an empty string. No warning |
|
|
| A `sort` direction other than `asc` / `desc` | Treated as ascending. No warning |
|
|
|
|
In short, a query can come back either wider or narrower than intended without any error. When a
|
|
result set looks wrong, check the server log for `Ignoring unsupported Logs Insights ...` — and note
|
|
that the compound-filter case above produces no log line at all.
|
|
|
|
For simple substring matching, `FilterLogEvents` is the more predictable option today. Note that
|
|
Floci matches `--filter-pattern` as a plain substring of the message; the real filter-pattern
|
|
syntax (`?ERROR ?WARN`, `{ $.level = "ERROR" }`, and so on) is not parsed.
|
|
|
|
### Reading events past the limit
|
|
|
|
`FilterLogEvents` and `GetLogEvents` both page, and they signal the end of the results differently
|
|
because the AWS APIs do.
|
|
|
|
`FilterLogEvents` pages forward only. Its `nextToken` is an `f/<index>` offset into the matched set,
|
|
so the offset counts matches, not stored events: a request narrowed by `--filter-pattern`,
|
|
`--start-time` or `--log-stream-names` pages through only what it matched. A missing token starts
|
|
from the oldest match, and **a response with no `nextToken` means pagination is finished**, so the
|
|
final page omits it. An unrecognized, non-numeric or negative token returns
|
|
`InvalidParameterException` (400). `startFromHead` is not supported, so results always run oldest
|
|
first.
|
|
|
|
`GetLogEvents` pages in both directions with `f/<index>` and `b/<index>`, and always returns
|
|
`nextForwardToken` and `nextBackwardToken`. It signals the end by returning the same token it was
|
|
given rather than by omitting it, which is what its SDK paginators expect.
|
|
|
|
### Configuration
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `FLOCI_SERVICES_CLOUDWATCHLOGS_ENABLED` | `true` | Enable or disable the CloudWatch Logs service |
|
|
| `FLOCI_SERVICES_CLOUDWATCHLOGS_MAX_EVENTS_PER_QUERY` | `10000` | Maximum events returned per `FilterLogEvents` / `GetLogEvents` call, and the upper bound for a Logs Insights `limit` |
|
|
| `FLOCI_SERVICES_CLOUDWATCHLOGS_QUERY_COMPLETION_DELAY_MS` | `0` | Artificial Logs Insights query delay. With `0` a query completes immediately. A positive value emulates the real asynchronous lifecycle (`Running` → `Complete` after the delay), which also makes `StopQuery` on a still-running query return `success=true` |
|
|
|
|
### Examples
|
|
|
|
```bash
|
|
export AWS_ENDPOINT_URL=http://localhost:4566
|
|
|
|
# Create a log group and stream
|
|
aws logs create-log-group --log-group-name /app/backend --endpoint-url $AWS_ENDPOINT_URL
|
|
aws logs create-log-stream \
|
|
--log-group-name /app/backend \
|
|
--log-stream-name 2025/01/app-1 \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# Write log events
|
|
TIMESTAMP=$(date +%s%3N) # milliseconds
|
|
aws logs put-log-events \
|
|
--log-group-name /app/backend \
|
|
--log-stream-name 2025/01/app-1 \
|
|
--log-events "[{\"timestamp\":$TIMESTAMP,\"message\":\"Service started\"}]" \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# Read log events
|
|
aws logs get-log-events \
|
|
--log-group-name /app/backend \
|
|
--log-stream-name 2025/01/app-1 \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# Search logs
|
|
aws logs filter-log-events \
|
|
--log-group-name /app/backend \
|
|
--filter-pattern "ERROR" \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# Run a Logs Insights query
|
|
QUERY_ID=$(aws logs start-query \
|
|
--log-group-name /app/backend \
|
|
--start-time $(($(date +%s) - 3600)) \
|
|
--end-time $(date +%s) \
|
|
--query-string 'fields @timestamp, @message | sort @timestamp desc | limit 20' \
|
|
--query queryId --output text \
|
|
--endpoint-url $AWS_ENDPOINT_URL)
|
|
|
|
aws logs get-query-results \
|
|
--query-id "$QUERY_ID" \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# Set retention
|
|
aws logs put-retention-policy \
|
|
--log-group-name /app/backend \
|
|
--retention-in-days 30 \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
```
|
|
|
|
---
|
|
|
|
## CloudWatch Metrics {#metrics}
|
|
|
|
**Protocol:** Query (XML) and JSON 1.1 (both supported)
|
|
**Endpoint:** `POST http://localhost:4566/`
|
|
|
|
### Supported Actions
|
|
|
|
| Action | Description |
|
|
|---|---|
|
|
| `PutMetricData` | Publish custom metrics |
|
|
| `ListMetrics` | List available metrics |
|
|
| `GetMetricStatistics` | Get metric statistics (Average, Sum, etc.) |
|
|
| `GetMetricData` | Query metrics with math expressions |
|
|
| `PutMetricAlarm` | Create a metric alarm |
|
|
| `DescribeAlarms` | List alarms |
|
|
| `DeleteAlarms` | Delete alarms |
|
|
| `SetAlarmState` | Manually set alarm state |
|
|
| `PutMetricStream` | Create or update a metric stream definition |
|
|
| `GetMetricStream` | Read a metric stream definition |
|
|
| `ListMetricStreams` | List metric streams |
|
|
| `DeleteMetricStream` | Delete a metric stream |
|
|
| `StartMetricStreams` | Move metric streams to `running` |
|
|
| `StopMetricStreams` | Move metric streams to `stopped` |
|
|
| `PutDashboard` | Create a dashboard, or replace its body when the name is taken (tags apply on create only) |
|
|
| `GetDashboard` | Read a dashboard body and ARN |
|
|
| `ListDashboards` | List dashboards, optionally by name prefix |
|
|
| `DeleteDashboards` | Delete dashboards by name |
|
|
| `TagResource` | Tag an alarm or dashboard by ARN |
|
|
| `UntagResource` | Remove tags from an alarm or dashboard |
|
|
| `ListTagsForResource` | List the tags of an alarm or dashboard |
|
|
|
|
Metric streams are stored as definitions with their `running` or `stopped` state. Floci never
|
|
delivers metrics to the Firehose delivery stream a metric stream names.
|
|
|
|
### Examples
|
|
|
|
```bash
|
|
export AWS_ENDPOINT_URL=http://localhost:4566
|
|
|
|
# Publish a custom metric
|
|
aws cloudwatch put-metric-data \
|
|
--namespace MyApp \
|
|
--metric-data '[{
|
|
"MetricName": "RequestCount",
|
|
"Value": 42,
|
|
"Unit": "Count",
|
|
"Dimensions": [{"Name":"Service","Value":"api"}]
|
|
}]' \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# List metrics
|
|
aws cloudwatch list-metrics \
|
|
--namespace MyApp \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# Get statistics
|
|
aws cloudwatch get-metric-statistics \
|
|
--namespace MyApp \
|
|
--metric-name RequestCount \
|
|
--dimensions Name=Service,Value=api \
|
|
--start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ) \
|
|
--end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
|
|
--period 300 \
|
|
--statistics Sum \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
|
|
# Create an alarm
|
|
aws cloudwatch put-metric-alarm \
|
|
--alarm-name high-error-rate \
|
|
--metric-name ErrorCount \
|
|
--namespace MyApp \
|
|
--statistic Sum \
|
|
--period 60 \
|
|
--threshold 10 \
|
|
--comparison-operator GreaterThanThreshold \
|
|
--evaluation-periods 1 \
|
|
--endpoint-url $AWS_ENDPOINT_URL
|
|
```
|