forked from Zakaria/hermes-agent
Zakaria
105 lines
4.3 KiB
Markdown
105 lines
4.3 KiB
Markdown
# ICIJ Offshore Leaks Database
|
||
|
||
## 1. Summary
|
||
|
||
The International Consortium of Investigative Journalists (ICIJ) publishes a
|
||
combined database of offshore entities from the Panama Papers, Paradise Papers,
|
||
Pandora Papers, Bahamas Leaks, and Offshore Leaks. ~800,000+ offshore entities
|
||
with their officers, intermediaries, and addresses.
|
||
|
||
## 2. Access Methods
|
||
|
||
- **Bulk download (primary):** `https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip` (~70 MB ZIP, refreshed periodically)
|
||
- **Search UI (human):** `https://offshoreleaks.icij.org/`
|
||
- **Auth:** None
|
||
- **Note:** The previous Open Refine reconciliation endpoint at
|
||
`/reconcile` now returns 404. ICIJ has removed it. The bulk ZIP is the
|
||
remaining stable access path. The skill's `fetch_icij_offshore.py` caches
|
||
the ZIP locally (default `~/.cache/hermes-osint/icij/`, refreshes after
|
||
30 days) and searches it offline.
|
||
|
||
## 3. Data Schema
|
||
|
||
Key fields emitted by `fetch_icij_offshore.py`:
|
||
|
||
| Column | Type | Description |
|
||
|--------|------|-------------|
|
||
| `node_id` | int | ICIJ canonical node ID |
|
||
| `name` | str | Entity / officer / intermediary name |
|
||
| `node_type` | str | entity / officer / intermediary / address |
|
||
| `country_codes` | str | Semicolon-separated ISO codes |
|
||
| `countries` | str | Country names |
|
||
| `jurisdiction` | str | Offshore jurisdiction (BVI, Panama, etc.) |
|
||
| `incorporation_date` | str | YYYY-MM-DD |
|
||
| `inactivation_date` | str | YYYY-MM-DD (if struck) |
|
||
| `source` | str | Panama Papers / Paradise Papers / Pandora Papers / etc. |
|
||
| `entity_url` | str | Link to ICIJ page |
|
||
| `connections` | str | Semicolon-separated node IDs of related entities |
|
||
|
||
## 4. Coverage
|
||
|
||
- Worldwide offshore entity records
|
||
- Earliest records: 1970s (Bahamas Leaks). Most data 1990–2018.
|
||
- NOT updated in real-time — new leaks added when ICIJ publishes them
|
||
- ~810,000 offshore entities + ~750,000 officers + ~150,000 intermediaries
|
||
|
||
## 5. Cross-Reference Potential
|
||
|
||
- **SEC EDGAR** ↔ `name` (public companies with offshore arms)
|
||
- **USAspending** ↔ `name` (federal contractors with offshore structure)
|
||
- **OFAC SDN** ↔ `name` (sanctioned entities using offshore vehicles)
|
||
|
||
Join key: normalized entity/officer name. `node_id` is canonical for cross-
|
||
referencing within ICIJ. Connections graph traversal is in-script (BFS over
|
||
`connections`).
|
||
|
||
## 6. Data Quality
|
||
|
||
- Offshore entity names sometimes appear in multiple leaks with slight variations
|
||
- Officers may be nominees (front persons), not beneficial owners
|
||
- Some entries have minimal info (just a name + jurisdiction)
|
||
- The connections graph is incomplete — some relationships are documented in
|
||
source materials but not in the structured database
|
||
- Inactive/struck-off entities are still included with `inactivation_date`
|
||
|
||
## 7. Acquisition Script
|
||
|
||
Path: `scripts/fetch_icij_offshore.py`
|
||
|
||
```bash
|
||
# Search by entity name (case-insensitive substring across the bulk DB)
|
||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
|
||
--out data/icij.csv
|
||
|
||
# Search by officer (individual person)
|
||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH JOHN" \
|
||
--out data/icij.csv
|
||
|
||
# Search by jurisdiction (filter on cached results)
|
||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH" \
|
||
--jurisdiction "BRITISH VIRGIN ISLANDS" --out data/icij_bvi.csv
|
||
|
||
# Force a fresh download (default refresh window is 30 days)
|
||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
|
||
--force-refresh --out data/icij.csv
|
||
```
|
||
|
||
First call downloads the ~70 MB ZIP under `~/.cache/hermes-osint/icij/`
|
||
(or `$HERMES_OSINT_CACHE/icij/`). Subsequent calls reuse the cache for 30 days.
|
||
|
||
## 8. Legal & Licensing
|
||
|
||
- Public record as published by ICIJ under explicit publication
|
||
- No copyright on the underlying facts (entity names, jurisdictions)
|
||
- ICIJ asks for attribution if used in derivative reporting
|
||
- **Ethical note**: Presence in this database does NOT imply wrongdoing. Many
|
||
offshore structures are legal. The database is a research tool, not a list of
|
||
criminals.
|
||
|
||
## 9. References
|
||
|
||
- Database: https://offshoreleaks.icij.org/
|
||
- About the data: https://offshoreleaks.icij.org/pages/about
|
||
- Methodology: https://www.icij.org/investigations/panama-papers/
|
||
- API hints: Open Refine reconciliation endpoint at `https://offshoreleaks.icij.org/reconcile`
|