Hermes-agent
This commit is contained in:
Zakaria
@@ -0,0 +1,157 @@
|
||||
# Hermes Achievements Performance Implementation Plan
|
||||
|
||||
Status: Ready for execution after hackathon review window
|
||||
Constraint: Plugin remains frozen until judging is complete
|
||||
Decision: `/overview` and top-banner slots are out of scope and will be removed.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Baseline & Safety (no behavior change)
|
||||
|
||||
### Task 0.1: Add perf benchmark script (local)
|
||||
Objective: Repro baseline before/after.
|
||||
|
||||
Acceptance:
|
||||
- Can print endpoint timings for `/achievements` (3 runs each, cold + warm).
|
||||
|
||||
### Task 0.2: Define acceptance thresholds
|
||||
Objective: Lock success criteria now.
|
||||
|
||||
Acceptance:
|
||||
- Documented SLOs:
|
||||
- `/achievements` p95 < 1s (cached)
|
||||
- max active scan jobs = 1
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Remove unused overview/slot surface (highest certainty)
|
||||
|
||||
### Task 1.1: Remove `/overview` backend route
|
||||
Objective: Eliminate duplicate heavy endpoint path.
|
||||
|
||||
Acceptance:
|
||||
- `plugin_api.py` no longer exposes `/overview`.
|
||||
|
||||
### Task 1.2: Remove slot registration and SummarySlot frontend code
|
||||
Objective: Remove cross-tab banner fetch behavior.
|
||||
|
||||
Acceptance:
|
||||
- No `registerSlot(..."sessions:top"...)` or `registerSlot(..."analytics:top"...)`.
|
||||
- No frontend call to `api("/overview")`.
|
||||
|
||||
### Task 1.3: Update plugin manifest
|
||||
Objective: Reflect final UI scope.
|
||||
|
||||
Acceptance:
|
||||
- `manifest.json` removes `slots` declarations.
|
||||
- Tab registration remains intact.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Shared snapshot persistence + single-flight for `/achievements`
|
||||
|
||||
### Task 2.1: Introduce snapshot store abstraction + on-disk persistence
|
||||
Objective: Single source of truth for Achievements data that survives process restarts.
|
||||
|
||||
Acceptance:
|
||||
- One structure contains dataset consumed by `/achievements`.
|
||||
- Repeated requests do not recompute when cache is fresh.
|
||||
- Snapshot persisted at `~/.hermes/plugins/hermes-achievements/scan_snapshot.json`.
|
||||
|
||||
### Task 2.2: Single-flight scan coordinator
|
||||
Objective: Prevent concurrent recomputes.
|
||||
|
||||
Acceptance:
|
||||
- Simultaneous requests result in one compute run.
|
||||
|
||||
### Task 2.3: Refactor `/achievements` to read snapshot
|
||||
Objective: Remove direct repeated compute from request path.
|
||||
|
||||
Acceptance:
|
||||
- `/achievements` does not run independent full recompute per request when cache is valid.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Stale-While-Revalidate
|
||||
|
||||
### Task 3.1: TTL state (`FRESH`/`STALE`)
|
||||
Objective: Serve immediately when stale, refresh in background.
|
||||
|
||||
Acceptance:
|
||||
- Cached response returned quickly even when expired.
|
||||
- Refresh is asynchronous.
|
||||
|
||||
### Task 3.2: Add `scan-status` endpoint (optional)
|
||||
Objective: Let UI/ops inspect scan state.
|
||||
|
||||
Acceptance:
|
||||
- Returns state, last success time, last duration, last error.
|
||||
|
||||
### Task 3.3: Add metadata fields to `/achievements`
|
||||
Objective: Improve transparency.
|
||||
|
||||
Acceptance:
|
||||
- Response includes `generated_at`, `is_stale`, maybe `scan_id`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Incremental Scanning (optional but recommended)
|
||||
|
||||
### Task 4.1: Add per-session checkpoint file
|
||||
Objective: Track session-level changes, not just global scan time.
|
||||
|
||||
Acceptance:
|
||||
- Checkpoint persisted at `~/.hermes/plugins/hermes-achievements/scan_checkpoint.json`.
|
||||
- For each session: `session_id`, fingerprint (`updated_at`/message_count/hash), and cached contribution.
|
||||
|
||||
### Task 4.2: Incremental aggregation
|
||||
Objective: Recompute only changed/new sessions and reuse unchanged contributions.
|
||||
|
||||
Acceptance:
|
||||
- Typical refresh time drops materially below full scan.
|
||||
- Aggregate rebuild uses: subtract old contribution + add new contribution for changed sessions.
|
||||
|
||||
### Task 4.3: Full rebuild fallback
|
||||
Objective: Preserve correctness.
|
||||
|
||||
Acceptance:
|
||||
- Manual full rescan always possible.
|
||||
- Schema/version changes invalidate checkpoint and force full rebuild.
|
||||
|
||||
---
|
||||
|
||||
## Test Plan
|
||||
|
||||
1. Unit tests
|
||||
- Snapshot lifecycle transitions
|
||||
- Dedupe logic under parallel requests
|
||||
- `/achievements` response compatibility
|
||||
|
||||
2. Integration tests
|
||||
- Opening Achievements repeatedly causes <=1 heavy scan while in-flight
|
||||
- `/achievements` warm-cache load is fast
|
||||
- manual rescan updates snapshot and timestamps
|
||||
|
||||
3. Manual benchmarks
|
||||
- Compare pre/post `/achievements` timings with same history dataset
|
||||
|
||||
---
|
||||
|
||||
## Rollout Plan
|
||||
|
||||
1. Release internal branch with Phase 1 (remove overview/slots).
|
||||
2. Validate no UI regression in Achievements tab.
|
||||
3. Add Phase 2 snapshot/dedupe.
|
||||
4. Add Phase 3 stale-while-revalidate + status metadata.
|
||||
5. Optional: incremental scanner.
|
||||
|
||||
Rollback: keep old compute path behind temporary feature flag for one release window.
|
||||
|
||||
---
|
||||
|
||||
## Definition of Done
|
||||
|
||||
- Achievements tab remains fully functional (counts, latest, tiers, cards, filters).
|
||||
- No `/overview` endpoint or slot calls remain.
|
||||
- Repeated Achievements loads feel immediate after warm cache.
|
||||
- Metrics/unlocks remain unchanged versus baseline.
|
||||
@@ -0,0 +1,219 @@
|
||||
# Hermes Achievements Implementation Spec (Detailed)
|
||||
|
||||
This document is implementation-facing detail to execute the performance refactor later.
|
||||
|
||||
Decision scope: keep only Achievements tab flow; remove `/overview` + top-banner slot integration.
|
||||
|
||||
---
|
||||
|
||||
## A) Current Behavior Summary
|
||||
|
||||
- `evaluate_all()` performs:
|
||||
- full `scan_sessions()`
|
||||
- `SessionDB.list_sessions_rich(...)`
|
||||
- `db.get_messages(session_id)` for each session
|
||||
- text/tool regex analysis + aggregation + evaluation
|
||||
- `/overview` and `/achievements` both currently call `evaluate_all()` directly.
|
||||
- slot calls (`sessions:top`, `analytics:top`) currently invoke `/overview`.
|
||||
|
||||
Consequence: repeated full recomputes and contention.
|
||||
|
||||
---
|
||||
|
||||
## B) De-scope/Removal Changes
|
||||
|
||||
1. Remove backend route:
|
||||
- `GET /overview`
|
||||
|
||||
2. Remove frontend slot usage:
|
||||
- `SummarySlot` component
|
||||
- `registerSlot("sessions:top")`
|
||||
- `registerSlot("analytics:top")`
|
||||
|
||||
3. Remove manifest slot declarations:
|
||||
- `"slots": ["sessions:top", "analytics:top"]`
|
||||
|
||||
4. Keep:
|
||||
- tab route/page for Achievements
|
||||
- `/achievements` endpoint and full tab rendering
|
||||
|
||||
---
|
||||
|
||||
## C) Target Internal Interfaces
|
||||
|
||||
### 1) `SnapshotStore`
|
||||
Responsibilities:
|
||||
- hold latest computed snapshot in memory
|
||||
- persist/load snapshot from disk
|
||||
- expose age and staleness checks
|
||||
|
||||
Storage path:
|
||||
- `~/.hermes/plugins/hermes-achievements/scan_snapshot.json`
|
||||
|
||||
Methods (conceptual):
|
||||
- `get()` -> snapshot | null
|
||||
- `set(snapshot)`
|
||||
- `is_stale(ttl_seconds)`
|
||||
|
||||
### 2) `ScanCoordinator`
|
||||
Responsibilities:
|
||||
- single-flight guard for compute jobs
|
||||
- track scan status
|
||||
|
||||
Methods:
|
||||
- `run_if_needed(force: bool = false)`
|
||||
- `get_status()`
|
||||
|
||||
State fields:
|
||||
- `state`: `idle|running|failed`
|
||||
- `started_at`, `finished_at`
|
||||
- `last_error`
|
||||
- `run_count`
|
||||
|
||||
### 3) `build_snapshot()`
|
||||
Responsibilities:
|
||||
- execute current compute logic once
|
||||
- on first run, perform full scan and materialize per-session contributions
|
||||
- on subsequent runs, process only changed/new sessions via checkpoint fingerprints
|
||||
- produce shape consumed by `/achievements`
|
||||
|
||||
Output:
|
||||
- `achievements`
|
||||
- count fields
|
||||
- optional `scan_meta`
|
||||
|
||||
---
|
||||
|
||||
## D) Endpoint Behavior Matrix (No `/overview`)
|
||||
|
||||
| Endpoint | Cache fresh | Cache stale | No cache | Force rescan |
|
||||
|---|---|---|---|---|
|
||||
| `/achievements` | return cached | return stale + trigger bg refresh | blocking bootstrap scan | n/a |
|
||||
| `/rescan` | trigger refresh | trigger refresh | trigger refresh | yes |
|
||||
| `/scan-status` | status only | status only | status only | status only |
|
||||
|
||||
Notes:
|
||||
- At most one scan run active.
|
||||
- Other callers either await same run or receive stale snapshot according to policy.
|
||||
|
||||
---
|
||||
|
||||
## E) Data Shape (Proposed)
|
||||
|
||||
```json
|
||||
{
|
||||
"generated_at": 0,
|
||||
"is_stale": false,
|
||||
"scan_meta": {
|
||||
"duration_ms": 0,
|
||||
"sessions_scanned": 0,
|
||||
"messages_scanned": 0,
|
||||
"mode": "full",
|
||||
"error": null
|
||||
},
|
||||
"achievements": [],
|
||||
"unlocked_count": 0,
|
||||
"discovered_count": 0,
|
||||
"secret_count": 0,
|
||||
"total_count": 0,
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
Compatibility guidance:
|
||||
- Keep existing `/achievements` keys.
|
||||
- Add metadata keys without breaking old callers.
|
||||
|
||||
Checkpoint file (new):
|
||||
- `~/.hermes/plugins/hermes-achievements/scan_checkpoint.json`
|
||||
|
||||
Suggested checkpoint shape:
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"generated_at": 0,
|
||||
"sessions": {
|
||||
"<session_id>": {
|
||||
"fingerprint": {
|
||||
"updated_at": 0,
|
||||
"message_count": 0,
|
||||
"hash": "optional"
|
||||
},
|
||||
"contribution": {
|
||||
"metrics": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Notes:
|
||||
- fingerprint mismatch => recompute that session contribution only.
|
||||
- unchanged fingerprint => reuse stored contribution.
|
||||
|
||||
---
|
||||
|
||||
## F) Concurrency Contract
|
||||
|
||||
- Any request path that needs fresh data must pass through single-flight coordinator.
|
||||
- If a scan is running:
|
||||
- do not start second scan
|
||||
- either await in-flight run (bounded) or serve stale snapshot immediately
|
||||
- lock scope must include scan start/finish state transitions.
|
||||
|
||||
---
|
||||
|
||||
## G) Error Handling Contract
|
||||
|
||||
- If refresh fails and prior snapshot exists:
|
||||
- return prior snapshot with `is_stale=true` and error metadata
|
||||
- If refresh fails and no prior snapshot:
|
||||
- return explicit error response (current behavior equivalent)
|
||||
- `scan-status` should always return last known state/error.
|
||||
|
||||
---
|
||||
|
||||
## H) Frontend Integration Contract
|
||||
|
||||
- Achievements page:
|
||||
- one fetch on mount to `/achievements`
|
||||
- optional background refresh indicator if stale
|
||||
- no top-banner slot integration
|
||||
- avoid duplicate in-flight calls during fast navigation by cancellation/debounce.
|
||||
|
||||
---
|
||||
|
||||
## I) Validation Checklist
|
||||
|
||||
- [ ] `/overview` route removed
|
||||
- [ ] manifest has no `sessions:top`/`analytics:top` slots
|
||||
- [ ] frontend has no `api("/overview")` calls
|
||||
- [ ] repeated Achievements navigation does not create multiple heavy scans
|
||||
- [ ] average warm load times meet SLOs
|
||||
- [ ] unlock totals match pre-refactor baseline for same history
|
||||
- [ ] no schema regression in `/achievements` response
|
||||
|
||||
---
|
||||
|
||||
## J) Suggested File Placement for Future Work
|
||||
|
||||
- backend changes: `dashboard/plugin_api.py`
|
||||
- optional extraction:
|
||||
- `dashboard/perf_snapshot.py`
|
||||
- `dashboard/perf_scan_coordinator.py`
|
||||
- frontend request hygiene: `dashboard/dist/index.js` (or source if available)
|
||||
- plugin metadata: `dashboard/manifest.json`
|
||||
- persisted runtime files:
|
||||
- `~/.hermes/plugins/hermes-achievements/state.json` (existing unlock state)
|
||||
- `~/.hermes/plugins/hermes-achievements/scan_snapshot.json` (new)
|
||||
- `~/.hermes/plugins/hermes-achievements/scan_checkpoint.json` (new)
|
||||
|
||||
---
|
||||
|
||||
## K) Post-Implementation Reporting Template
|
||||
|
||||
Record:
|
||||
- dataset size (sessions/messages/tool calls)
|
||||
- pre/post `/achievements` timings (cold/warm)
|
||||
- whether single-flight dedupe triggered under repeated tab open
|
||||
- any behavioral diffs in unlock counts
|
||||
@@ -0,0 +1,174 @@
|
||||
# Hermes Achievements Performance Spec (Post-Hackathon)
|
||||
|
||||
Status: Draft (no code changes yet)
|
||||
Owner: hermes-achievements plugin
|
||||
Scope: `dashboard/plugin_api.py` + `dashboard/dist/index.js` request behavior
|
||||
Decision: **Drop `/overview` and top-banner slots**; keep only Achievements tab data path.
|
||||
|
||||
---
|
||||
|
||||
## 1) Problem Statement
|
||||
|
||||
Current plugin endpoints `/achievements` and `/overview` both execute a full history recomputation (`evaluate_all()`), which performs a full SessionDB scan each request.
|
||||
|
||||
Observed on this machine/repo:
|
||||
- ~83 sessions
|
||||
- ~7,125 messages
|
||||
- ~3,623 tool calls
|
||||
- `evaluate_all()` ~13–16s per call
|
||||
- `/achievements` ~13–15s per call
|
||||
- `/overview` ~12–15s per call
|
||||
- Overlap between endpoints increases perceived wait.
|
||||
|
||||
Given current product direction, `/overview` and cross-tab top-banner slots are not needed.
|
||||
|
||||
---
|
||||
|
||||
## 2) Goals
|
||||
|
||||
- Keep achievement correctness unchanged.
|
||||
- Keep all Achievements-tab UX/data (unlocked/discovered/secrets/highest/latest/cards).
|
||||
- Remove unused summary path (`/overview`) and slot wiring.
|
||||
- Make Achievements tab faster by avoiding duplicate endpoint pathways.
|
||||
- Ensure at most one heavy scan can run at a time.
|
||||
|
||||
Non-goals (phase 1):
|
||||
- Rewriting achievement rules.
|
||||
- Changing badge semantics/states.
|
||||
|
||||
---
|
||||
|
||||
## 3) Endpoint Semantics (Target)
|
||||
|
||||
### `GET /api/plugins/hermes-achievements/achievements`
|
||||
Single source endpoint for Achievements UI.
|
||||
Returns full payload used by the tab:
|
||||
- `achievements`
|
||||
- `unlocked_count`
|
||||
- `discovered_count`
|
||||
- `secret_count`
|
||||
- `total_count`
|
||||
- `error`
|
||||
|
||||
### `POST /api/plugins/hermes-achievements/rescan` (optional)
|
||||
Manual refresh trigger.
|
||||
Prefer async trigger + immediate status response.
|
||||
|
||||
### `GET /api/plugins/hermes-achievements/scan-status` (optional new)
|
||||
Reports scan state for UX/ops.
|
||||
|
||||
### Removed
|
||||
- `GET /api/plugins/hermes-achievements/overview`
|
||||
|
||||
---
|
||||
|
||||
## 4) UI Scope (Target)
|
||||
|
||||
Keep:
|
||||
- Achievements page/tab (`/achievements` in plugin tab manifest)
|
||||
- All existing Achievements tab stats/cards/filters
|
||||
|
||||
Remove:
|
||||
- Top-banner summary slot components using `sessions:top` and `analytics:top`
|
||||
- Any frontend call path to `/overview`
|
||||
|
||||
---
|
||||
|
||||
## 5) Runtime State Machine (for `/achievements`)
|
||||
|
||||
- `FRESH`: cached snapshot age <= TTL
|
||||
- `STALE`: snapshot exists but expired
|
||||
- `SCANNING`: background recompute running
|
||||
- `FAILED`: last recompute failed, last good snapshot still served
|
||||
|
||||
Rules:
|
||||
1. FRESH -> serve immediately.
|
||||
2. STALE + not scanning -> serve stale snapshot immediately and launch background refresh.
|
||||
3. SCANNING -> do not start another scan; join single-flight in-flight job.
|
||||
4. No snapshot yet -> allow one blocking bootstrap scan.
|
||||
|
||||
---
|
||||
|
||||
## 6) Caching & Invalidation
|
||||
|
||||
### Phase 1
|
||||
- In-memory cache + persisted snapshot file.
|
||||
- TTL: 60–180 seconds (configurable).
|
||||
- Single-flight dedupe for scan requests.
|
||||
- Persist plugin data under:
|
||||
- `~/.hermes/plugins/hermes-achievements/scan_snapshot.json`
|
||||
|
||||
### Phase 2
|
||||
- Incremental scan checkpoints with per-session fingerprints.
|
||||
- Persist checkpoint data under:
|
||||
- `~/.hermes/plugins/hermes-achievements/scan_checkpoint.json`
|
||||
- Checkpoint stores, per session:
|
||||
- `session_id`
|
||||
- fingerprint (`updated_at`, message_count, or hash)
|
||||
- cached per-session contribution used for aggregate recomposition
|
||||
- Scan policy:
|
||||
- First run: full scan and materialize snapshot + checkpoint.
|
||||
- Next runs: process only new/changed sessions, reuse unchanged contributions.
|
||||
- Full rebuild only on:
|
||||
- schema/version change
|
||||
- checkpoint corruption
|
||||
- explicit full rescan
|
||||
|
||||
---
|
||||
|
||||
## 7) Frontend Contract
|
||||
|
||||
- Achievements tab requests `/achievements` once on mount.
|
||||
- No slot-based summary fetches.
|
||||
- If response says `is_stale=true`, UI may display “Updating in background”.
|
||||
- Avoid duplicate mount-triggered calls and cancel stale requests on navigation.
|
||||
|
||||
---
|
||||
|
||||
## 8) SLO Targets
|
||||
|
||||
- `/achievements` p95 < 1s (cached)
|
||||
- Max concurrent heavy scans: 1
|
||||
- Background refresh should not block UI
|
||||
|
||||
---
|
||||
|
||||
## 9) Observability Requirements
|
||||
|
||||
Track:
|
||||
- scan count
|
||||
- scan duration avg/p95
|
||||
- dedupe hit count (joined in-flight scans)
|
||||
- stale-served count
|
||||
- failures + last error
|
||||
|
||||
Expose minimal diagnostics in `/scan-status`.
|
||||
|
||||
---
|
||||
|
||||
## 10) Backward Compatibility
|
||||
|
||||
- Keep `/achievements` response shape backward-compatible.
|
||||
- Removing `/overview` is acceptable because slot UI is intentionally removed.
|
||||
- If temporary compatibility is needed, `/overview` can return static deprecation response for one release.
|
||||
|
||||
---
|
||||
|
||||
## 11) Risks
|
||||
|
||||
- Stale data confusion -> mitigate with `generated_at` and explicit refresh status.
|
||||
- Cache invalidation bugs -> start with conservative TTL + manual rescan.
|
||||
- Concurrency bugs -> protect scan section with lock/single-flight guard.
|
||||
- Session mutation edge cases -> use per-session fingerprint invalidation (not global timestamp only).
|
||||
|
||||
---
|
||||
|
||||
## 12) Persistence Files (Explicit)
|
||||
|
||||
Plugin state directory:
|
||||
- `~/.hermes/plugins/hermes-achievements/`
|
||||
|
||||
Files:
|
||||
- `state.json` (existing): unlock tracking
|
||||
- `scan_snapshot.json` (new): latest materialized achievements payload
|
||||
- `scan_checkpoint.json` (new): per-session fingerprints + contributions for incremental refresh
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.4 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Reference in New Issue
Block a user