Hermes-agent

2026-06-14 14:30:48 -04:00
commit dac4b88b94
5058 changed files with 1884848 additions and 0 deletions
@@ -0,0 +1,104 @@
+# ICIJ Offshore Leaks Database
+
+## 1. Summary
+
+The International Consortium of Investigative Journalists (ICIJ) publishes a
+combined database of offshore entities from the Panama Papers, Paradise Papers,
+Pandora Papers, Bahamas Leaks, and Offshore Leaks. ~800,000+ offshore entities
+with their officers, intermediaries, and addresses.
+
+## 2. Access Methods
+
+- **Bulk download (primary):** `https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip` (~70 MB ZIP, refreshed periodically)
+- **Search UI (human):** `https://offshoreleaks.icij.org/`
+- **Auth:** None
+- **Note:** The previous Open Refine reconciliation endpoint at
+  `/reconcile` now returns 404. ICIJ has removed it. The bulk ZIP is the
+  remaining stable access path. The skill's `fetch_icij_offshore.py` caches
+  the ZIP locally (default `~/.cache/hermes-osint/icij/`, refreshes after
+  30 days) and searches it offline.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_icij_offshore.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `node_id` | int | ICIJ canonical node ID |
+| `name` | str | Entity / officer / intermediary name |
+| `node_type` | str | entity / officer / intermediary / address |
+| `country_codes` | str | Semicolon-separated ISO codes |
+| `countries` | str | Country names |
+| `jurisdiction` | str | Offshore jurisdiction (BVI, Panama, etc.) |
+| `incorporation_date` | str | YYYY-MM-DD |
+| `inactivation_date` | str | YYYY-MM-DD (if struck) |
+| `source` | str | Panama Papers / Paradise Papers / Pandora Papers / etc. |
+| `entity_url` | str | Link to ICIJ page |
+| `connections` | str | Semicolon-separated node IDs of related entities |
+
+## 4. Coverage
+
+- Worldwide offshore entity records
+- Earliest records: 1970s (Bahamas Leaks). Most data 1990–2018.
+- NOT updated in real-time — new leaks added when ICIJ publishes them
+- ~810,000 offshore entities + ~750,000 officers + ~150,000 intermediaries
+
+## 5. Cross-Reference Potential
+
+- **SEC EDGAR** ↔ `name` (public companies with offshore arms)
+- **USAspending** ↔ `name` (federal contractors with offshore structure)
+- **OFAC SDN** ↔ `name` (sanctioned entities using offshore vehicles)
+
+Join key: normalized entity/officer name. `node_id` is canonical for cross-
+referencing within ICIJ. Connections graph traversal is in-script (BFS over
+`connections`).
+
+## 6. Data Quality
+
+- Offshore entity names sometimes appear in multiple leaks with slight variations
+- Officers may be nominees (front persons), not beneficial owners
+- Some entries have minimal info (just a name + jurisdiction)
+- The connections graph is incomplete — some relationships are documented in
+  source materials but not in the structured database
+- Inactive/struck-off entities are still included with `inactivation_date`
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_icij_offshore.py`
+
+```bash
+# Search by entity name (case-insensitive substring across the bulk DB)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
+    --out data/icij.csv
+
+# Search by officer (individual person)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH JOHN" \
+    --out data/icij.csv
+
+# Search by jurisdiction (filter on cached results)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH" \
+    --jurisdiction "BRITISH VIRGIN ISLANDS" --out data/icij_bvi.csv
+
+# Force a fresh download (default refresh window is 30 days)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
+    --force-refresh --out data/icij.csv
+```
+
+First call downloads the ~70 MB ZIP under `~/.cache/hermes-osint/icij/`
+(or `$HERMES_OSINT_CACHE/icij/`). Subsequent calls reuse the cache for 30 days.
+
+## 8. Legal & Licensing
+
+- Public record as published by ICIJ under explicit publication
+- No copyright on the underlying facts (entity names, jurisdictions)
+- ICIJ asks for attribution if used in derivative reporting
+- **Ethical note**: Presence in this database does NOT imply wrongdoing. Many
+  offshore structures are legal. The database is a research tool, not a list of
+  criminals.
+
+## 9. References
+
+- Database: https://offshoreleaks.icij.org/
+- About the data: https://offshoreleaks.icij.org/pages/about
+- Methodology: https://www.icij.org/investigations/panama-papers/
+- API hints: Open Refine reconciliation endpoint at `https://offshoreleaks.icij.org/reconcile`