Sami/hermes-agent
1
0
Fork 0
forked from Zakaria/hermes-agent
Code Pull Requests Activity

Hermes-agent

Browse Source
This commit is contained in:
Zakaria
2026-06-14 14:30:48 -04:00
commit dac4b88b94
5058 changed files with 1884848 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
optional-skills/research/osint-investigation/references/sources/courtlistener.md
+98
View File
@@ -0,0 +1,98 @@
# CourtListener — Free Law Project
## 1. Summary
CourtListener (Free Law Project) aggregates court opinions, dockets, oral
arguments, and judge data. Covers ~10M federal and state court opinions
back to colonial America, plus PACER docket data from RECAP submissions.
## 2. Access Methods
- **REST API v4:** `https://www.courtlistener.com/api/rest/v4/`
- **Auth:** Anonymous reads allowed on most endpoints; token raises rate
limits and unlocks bulk export
- **Rate limit:** ~5,000 req/hour unauthenticated for search; higher with token
Set `COURTLISTENER_TOKEN` env var. Get a free token at
https://www.courtlistener.com/sign-in/ then create an API key.
## 3. Data Schema
Key fields emitted by `fetch_courtlistener.py`:
| Column | Type | Description |
|--------|------|-------------|
| `case_name` | str | Case name |
| `court` | str | Court name |
| `court_id` | str | Court ID (e.g. `nysd`, `scotus`, `ca9`) |
| `date_filed` | str | YYYY-MM-DD |
| `docket_number` | str | Court docket number |
| `judge` | str | Judge name(s) |
| `citation` | str | Reporter citation(s) |
| `result_type` | str | opinions / dockets / oral / people |
| `snippet` | str | Search-match snippet (up to 500 chars) |
| `absolute_url` | str | Direct CourtListener URL |
## 4. Coverage
- Federal: all circuit and district courts, SCOTUS
- State: all 50 state supreme/appellate courts, many trial courts
- Opinions: ~10M back to 1600s (colonial), full coverage 1950 → present
- Dockets via RECAP: ~3M+ from user-submitted PACER PDFs
- Updated continuously
## 5. Cross-Reference Potential
- **OpenCorporates** ↔ `case_name` (corporate litigation)
- **SEC EDGAR** ↔ `case_name` (securities class actions)
- **OFAC SDN** ↔ `case_name` (sanctions-related civil/criminal cases)
Join key: party name from `case_name`. Note: `case_name` often abbreviates
("Smith v. Jones" rather than full party names) — use the full case URL
to get all parties.
## 6. Data Quality
- Older opinions (pre-1990) often lack docket numbers and judges
- State coverage is more uneven than federal
- PACER docket coverage depends on RECAP user submissions — not exhaustive
- Sealed documents are excluded
- Party names in case captions don't always match filing names exactly
## 7. Acquisition Script
Path: `scripts/fetch_courtlistener.py`
```bash
# Search opinions for a party / keyword
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
--out data/cl.csv
# PACER dockets (best for recent litigation)
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
--type dockets --out data/cl_dockets.csv
# Restrict to a court
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Microsoft" \
--court ca9 --out data/cl_9th.csv
# Date range
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
--date-from 2020-01-01 --date-to 2024-12-31 --out data/cl.csv
```
Pass `--token` or set `COURTLISTENER_TOKEN`.
## 8. Legal & Licensing
- Court opinions are public domain
- Free Law Project provides the data under CC0 / public domain dedication
- No commercial use restrictions on opinion text or metadata
- Some PACER PDFs have copyright on layout (not text) — fair use applies
## 9. References
- API docs: https://www.courtlistener.com/help/api/rest/
- Court IDs: https://www.courtlistener.com/api/jurisdictions/
- RECAP archive: https://www.courtlistener.com/recap/
- Bulk data: https://www.courtlistener.com/help/api/bulk-data/
optional-skills/research/osint-investigation/references/sources/gdelt.md
+104
View File
@@ -0,0 +1,104 @@
# GDELT — Global News Monitoring
## 1. Summary
GDELT (Global Database of Events, Language, and Tone) monitors world news
in 100+ languages with full-text indexing. Updated every 15 minutes.
~2015 → present, ~1B+ articles indexed. Free anonymous access.
GDELT is wider than Google News (more international, more long-tail
sources) and indexed by tone/sentiment, themes (CAMEO codes), people, and
organizations.
## 2. Access Methods
- **DOC 2.0 API:** `https://api.gdeltproject.org/api/v2/doc/doc`
- **Events / GKG 2.0:** `https://api.gdeltproject.org/api/v2/events/events`
- **Auth:** None
- **Rate limit:** **1 request per 5 seconds** for the DOC API — strict
The fetch script automatically retries after a 6-second sleep when a
429 is received.
## 3. Data Schema
Key fields emitted by `fetch_gdelt.py`:
| Column | Type | Description |
|--------|------|-------------|
| `title` | str | Article title |
| `url` | str | Article URL |
| `seen_date` | str | When GDELT first saw the article (UTC) |
| `domain` | str | Publisher domain |
| `language` | str | Source language |
| `source_country` | str | 2-letter country code |
| `tone` | str | GDELT-computed tone score (negative = negative coverage) |
| `social_image` | str | Open Graph image URL when available |
## 4. Coverage
- Worldwide news in 100+ languages
- ~2015 → present (Events back to 1979 via a separate stream)
- Update frequency: 15 minutes
- Bias: heavily Anglophone in volume but very wide source list overall
## 5. Cross-Reference Potential
- **All sources** ↔ `title` / `url` (news context for any subject)
- **Wikipedia** ↔ event timeline for notable entities
- **Wayback Machine** ↔ recover articles whose URLs have died
- **OFAC SDN** ↔ news context for sanctions designations
- **SEC EDGAR** ↔ news context for 8-K material events
Join key: entity name appearing in article title or full-text. GDELT also
extracts named entities into a separate stream (GKG) not exposed by this
fetcher — query GDELT directly for entity-level filtering.
## 6. Data Quality
- Title extraction is automated and can be wrong (sometimes captures the
site name + delimiter + article title; sometimes a generic page title)
- Sentiment / tone is computed by GDELT, not source-supplied
- Some domains are oversampled (newswires, aggregators)
- Source country is inferred from domain registration / TLD — can be
wrong for international news sites with country-neutral domains
- Article URLs can rot — pair with Wayback Machine to preserve content
## 7. Acquisition Script
Path: `scripts/fetch_gdelt.py`
```bash
# Recent news mentioning an entity
python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Nous Research" \
--timespan 6m --out data/gdelt.csv
# Phrase-exact (use double quotes inside single quotes for the shell)
python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Dillon Rolnick"' \
--timespan 1y --out data/gdelt.csv
# Filter to a country / language
python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Microsoft" \
--source-country US --source-lang English --out data/gdelt.csv
# Date range
python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Microsoft" \
--start 2024-01-01 --end 2024-12-31 --out data/gdelt.csv
```
GDELT supports its own query operators: phrase quoting, AND/OR/NOT,
`sourcecountry:US`, `theme:ECON_BANKRUPTCY`, `tone<-5`, etc.
See https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/ for syntax.
## 8. Legal & Licensing
- GDELT data is provided free for academic and journalistic use
- Article URLs link out to original publishers — copyright remains with
the publisher
- GDELT is NOT a content archive; it's a metadata index
## 9. References
- DOC 2.0 API: https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/
- Themes & query syntax: https://blog.gdeltproject.org/gkg-2-0-our-global-knowledge-graph-2-0-amazing-data-at-your-fingertips/
- Project home: https://www.gdeltproject.org/
optional-skills/research/osint-investigation/references/sources/icij-offshore.md
+104
View File
@@ -0,0 +1,104 @@
# ICIJ Offshore Leaks Database
## 1. Summary
The International Consortium of Investigative Journalists (ICIJ) publishes a
combined database of offshore entities from the Panama Papers, Paradise Papers,
Pandora Papers, Bahamas Leaks, and Offshore Leaks. ~800,000+ offshore entities
with their officers, intermediaries, and addresses.
## 2. Access Methods
- **Bulk download (primary):** `https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip` (~70 MB ZIP, refreshed periodically)
- **Search UI (human):** `https://offshoreleaks.icij.org/`
- **Auth:** None
- **Note:** The previous Open Refine reconciliation endpoint at
`/reconcile` now returns 404. ICIJ has removed it. The bulk ZIP is the
remaining stable access path. The skill's `fetch_icij_offshore.py` caches
the ZIP locally (default `~/.cache/hermes-osint/icij/`, refreshes after
30 days) and searches it offline.
## 3. Data Schema
Key fields emitted by `fetch_icij_offshore.py`:
| Column | Type | Description |
|--------|------|-------------|
| `node_id` | int | ICIJ canonical node ID |
| `name` | str | Entity / officer / intermediary name |
| `node_type` | str | entity / officer / intermediary / address |
| `country_codes` | str | Semicolon-separated ISO codes |
| `countries` | str | Country names |
| `jurisdiction` | str | Offshore jurisdiction (BVI, Panama, etc.) |
| `incorporation_date` | str | YYYY-MM-DD |
| `inactivation_date` | str | YYYY-MM-DD (if struck) |
| `source` | str | Panama Papers / Paradise Papers / Pandora Papers / etc. |
| `entity_url` | str | Link to ICIJ page |
| `connections` | str | Semicolon-separated node IDs of related entities |
## 4. Coverage
- Worldwide offshore entity records
- Earliest records: 1970s (Bahamas Leaks). Most data 1990–2018.
- NOT updated in real-time — new leaks added when ICIJ publishes them
- ~810,000 offshore entities + ~750,000 officers + ~150,000 intermediaries
## 5. Cross-Reference Potential
- **SEC EDGAR** ↔ `name` (public companies with offshore arms)
- **USAspending** ↔ `name` (federal contractors with offshore structure)
- **OFAC SDN** ↔ `name` (sanctioned entities using offshore vehicles)
Join key: normalized entity/officer name. `node_id` is canonical for cross-
referencing within ICIJ. Connections graph traversal is in-script (BFS over
`connections`).
## 6. Data Quality
- Offshore entity names sometimes appear in multiple leaks with slight variations
- Officers may be nominees (front persons), not beneficial owners
- Some entries have minimal info (just a name + jurisdiction)
- The connections graph is incomplete — some relationships are documented in
source materials but not in the structured database
- Inactive/struck-off entities are still included with `inactivation_date`
## 7. Acquisition Script
Path: `scripts/fetch_icij_offshore.py`
```bash
# Search by entity name (case-insensitive substring across the bulk DB)
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
--out data/icij.csv
# Search by officer (individual person)
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH JOHN" \
--out data/icij.csv
# Search by jurisdiction (filter on cached results)
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH" \
--jurisdiction "BRITISH VIRGIN ISLANDS" --out data/icij_bvi.csv
# Force a fresh download (default refresh window is 30 days)
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
--force-refresh --out data/icij.csv
```
First call downloads the ~70 MB ZIP under `~/.cache/hermes-osint/icij/`
(or `$HERMES_OSINT_CACHE/icij/`). Subsequent calls reuse the cache for 30 days.
## 8. Legal & Licensing
- Public record as published by ICIJ under explicit publication
- No copyright on the underlying facts (entity names, jurisdictions)
- ICIJ asks for attribution if used in derivative reporting
- **Ethical note**: Presence in this database does NOT imply wrongdoing. Many
offshore structures are legal. The database is a research tool, not a list of
criminals.
## 9. References
- Database: https://offshoreleaks.icij.org/
- About the data: https://offshoreleaks.icij.org/pages/about
- Methodology: https://www.icij.org/investigations/panama-papers/
- API hints: Open Refine reconciliation endpoint at `https://offshoreleaks.icij.org/reconcile`
optional-skills/research/osint-investigation/references/sources/nyc-acris.md
+90
View File
@@ -0,0 +1,90 @@
# NYC ACRIS — NYC Real Property Records
## 1. Summary
The Automated City Register Information System (ACRIS) is NYC's index of
recorded property documents: deeds, mortgages, satisfactions, liens, UCC
filings. Covers Manhattan, Bronx, Brooklyn, Queens, Staten Island.
Published as 4 linked Socrata datasets on the NYC Open Data portal.
## 2. Access Methods
- **Socrata API:** `https://data.cityofnewyork.us/resource/636b-3b5g.json` (Parties)
- **Other datasets:** `bnx9-e6tj` (Master), `8h5j-fqxa` (Legal), `uqqa-hym2` (References)
- **Auth:** None for read access (Socrata `$app_token` raises rate limits if needed)
- **Rate limit:** Generous (~1000 req/hour unauthenticated)
## 3. Data Schema
Key fields emitted by `fetch_nyc_acris.py` (Parties joined to Master):
| Column | Type | Description |
|--------|------|-------------|
| `document_id` | str | ACRIS document ID |
| `name` | str | Party name as recorded (often "LAST, FIRST" but varies) |
| `party_type` | str | 1=grantor, 2=grantee, 3=other |
| `party_role` | str | Human-readable role label |
| `address_1` | str | Property or party address line 1 |
| `city`, `state`, `zip`, `country` | str | Address parts |
| `doc_type` | str | DEED, MTGE (mortgage), SAT (satisfaction), AGMT, etc. |
| `doc_date`, `recorded_date` | str | YYYY-MM-DD |
| `borough` | str | Manhattan / Bronx / Brooklyn / Queens / Staten Island |
| `amount` | str | Document amount (USD, when applicable) |
| `filing_url` | str | Direct ACRIS DocumentImageView link |
## 4. Coverage
- NYC 5 boroughs only — other counties have their own recorders
- 1966 → present (older filings exist on microfilm at the County Clerk)
- Updated nightly
- ~70M+ party records cumulative
## 5. Cross-Reference Potential
- **SEC EDGAR** ↔ `name` (insider filers with NYC property)
- **USAspending** ↔ `name` (federal contractors with NYC property)
- **Senate LDA** ↔ `name` (lobbyists / clients with NYC property)
- **ICIJ Offshore** ↔ `name` (NYC properties owned via offshore vehicles)
Join key: normalized party name. NYC property records typically store names
as "LAST, FIRST" or full LLC names — use `entity_resolution.py`.
## 6. Data Quality
- Same person appears with multiple name formats over time
- LLC and trust ownership obscures beneficial owners
- Recording lag can be 2-4 weeks after closing
- Older documents have spottier address data
- Sealed records (e.g. domestic violence shelters) are excluded by law
## 7. Acquisition Script
Path: `scripts/fetch_nyc_acris.py`
```bash
# By party name
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "ROLNICK" --out data/acris.csv
# By address (useful when you know the property but not the names)
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" --out data/acris.csv
# Restrict to grantees (buyers / mortgagees)
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "ROLNICK" --party-type 2 \
--out data/acris_buyers.csv
```
The script joins Parties → Master to populate doc_type, dates, borough, and
amount. Pass `--no-enrich` to skip the join (faster, fewer columns).
## 8. Legal & Licensing
- Public record under NYS Real Property Law and NYC Charter
- No commercial use restrictions on the data
- All ACRIS data is public information by statute
## 9. References
- ACRIS portal: https://a836-acris.nyc.gov/CP/
- NYC Open Data: https://data.cityofnewyork.us/
- Parties dataset: https://data.cityofnewyork.us/City-Government/ACRIS-Real-Property-Parties/636b-3b5g
- Document type codes: https://www1.nyc.gov/site/finance/taxes/acris.page
optional-skills/research/osint-investigation/references/sources/ofac-sdn.md
+92
View File
@@ -0,0 +1,92 @@
# OFAC SDN — Specially Designated Nationals List
## 1. Summary
The Office of Foreign Assets Control (OFAC) publishes the Specially Designated
Nationals and Blocked Persons List (SDN). US persons are generally prohibited
from dealing with individuals and entities on this list. Also published:
non-SDN consolidated lists (BIS Denied Persons, FSE, etc.).
## 2. Access Methods
- **Full XML:** `https://www.treasury.gov/ofac/downloads/sdn.xml`
- **Delimited:** `https://www.treasury.gov/ofac/downloads/sdn.csv`
- **Consolidated:** `https://www.treasury.gov/ofac/downloads/consolidated/consolidated.xml`
- **Auth:** None
- **Rate limit:** None (static file downloads). Updated continuously.
## 3. Data Schema
Key fields emitted by `fetch_ofac_sdn.py`:
| Column | Type | Description |
|--------|------|-------------|
| `entity_id` | int | OFAC unique ID |
| `name` | str | Primary name |
| `entity_type` | str | individual / entity / vessel / aircraft |
| `program_list` | str | Semicolon-separated sanctions programs (e.g. SDGT;IRAN) |
| `title` | str | For individuals: title/role |
| `nationalities` | str | Semicolon-separated country codes |
| `aka_list` | str | Semicolon-separated "also known as" names |
| `addresses` | str | Semicolon-separated known addresses |
| `dob` | str | Date of birth (individuals) |
| `pob` | str | Place of birth (individuals) |
| `remarks` | str | OFAC's free-text remarks |
| `last_updated` | str | YYYY-MM-DD (publication date) |
## 4. Coverage
- Worldwide — all entities sanctioned by US Treasury
- ~10,000 entries on SDN, ~15,000 on consolidated lists
- Updated continuously (sometimes daily during active enforcement)
- Includes AKAs (very common, can be 10+ per entity)
## 5. Cross-Reference Potential
- **SEC EDGAR** ↔ `name` (public companies sanctioned)
- **USAspending** ↔ `name` (sanctioned entity as federal contractor — should
be impossible but verify)
- **ICIJ Offshore** ↔ `name` (offshore entities also sanctioned)
Join key: normalized name. **CRITICAL**: must match against `aka_list` too.
Many sanctioned entities are caught only via aliases.
## 6. Data Quality
- Names are transliterated from many scripts — multiple romanizations possible
- AKAs often differ wildly from primary name
- Some entries have minimal info (no DOB, no address) for individuals
- Free-text `remarks` contain critical context — read them
- "Specially Designated Global Terrorists" (SDGT) and "Cyber-related" (CYBER2)
programs add and remove entries frequently
## 7. Acquisition Script
Path: `scripts/fetch_ofac_sdn.py`
```bash
# Full snapshot
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
# Filter to specific program
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --program SDGT --out data/sdn_sdgt.csv
# Entities only (skip individuals, vessels, aircraft)
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --entity-type entity --out data/sdn_entities.csv
```
## 8. Legal & Licensing
- Public record under Executive Order authority and statutory sanctions programs
- US persons MUST screen against this list — it is enforced
- No restrictions on the data itself; restrictions are on transactions with
the listed entities
- ZERO penalty for "over-matching" — false positives must be cleared but are not
prohibited
## 9. References
- OFAC home: https://ofac.treasury.gov/
- SDN list: https://ofac.treasury.gov/specially-designated-nationals-and-blocked-persons-list-sdn-human-readable-lists
- Data formats: https://ofac.treasury.gov/sdn-list/sanctions-list-search-tool
- Compliance guidance: https://ofac.treasury.gov/recent-actions
optional-skills/research/osint-investigation/references/sources/opencorporates.md
+103
View File
@@ -0,0 +1,103 @@
# OpenCorporates — Global Corporate Registry
## 1. Summary
OpenCorporates aggregates corporate registry data from 130+ jurisdictions
worldwide (~200M companies). Covers US state-level filings (NY DOS, Delaware
DOC, California SOS, etc.), UK Companies House, EU registries, and most
common-law jurisdictions.
## 2. Access Methods
- **REST API:** `https://api.opencorporates.com/v0.4/`
- **HTML fallback:** `https://opencorporates.com/companies?q=...`
- **Auth:** API token required (free tier 500 calls/month, paid plans available)
- **Rate limit:** Token-bound; un-tokened requests return 401
Set `OPENCORPORATES_API_TOKEN` env var. Get a free token at
https://opencorporates.com/api_accounts/new.
## 3. Data Schema
Key fields emitted by `fetch_opencorporates.py`:
| Column | Type | Description |
|--------|------|-------------|
| `name` | str | Company legal name |
| `company_number` | str | Registry-assigned number |
| `jurisdiction_code` | str | e.g. `us_ny`, `us_de`, `gb` |
| `jurisdiction_name` | str | Human-readable jurisdiction |
| `incorporation_date` | str | YYYY-MM-DD |
| `dissolution_date` | str | YYYY-MM-DD (empty if active) |
| `company_type` | str | Domestic LLC / Foreign Corp / etc. |
| `status` | str | Active / Inactive / Dissolved |
| `registered_address` | str | Registered office address |
| `opencorporates_url` | str | Link to OpenCorporates entity page |
| `officers_count` | str | Total officers on record |
| `source` | str | `api`, `html`, or `html-fallback` |
## 4. Coverage
- US: all 50 states + DC at state level (LLCs, corps, LPs)
- International: UK, EU, Canada, Australia, NZ, many APAC + LATAM jurisdictions
- ~200M company records cumulative
- Update frequency varies by jurisdiction (UK CH is near-realtime; some
state registries lag months)
## 5. Cross-Reference Potential
- **NYC ACRIS** ↔ `name` (LLC/corp owners of NYC property)
- **USAspending** ↔ `name` (corporate federal contractors)
- **SEC EDGAR** ↔ `name` (public companies + their subsidiaries)
- **ICIJ Offshore** ↔ `name` (international corporate structures)
Join key: normalized company name. Some entries have `previous_names` arrays
which are not currently exported by the fetch script — query OC directly
for that.
## 6. Data Quality
- Company-name spellings vary across re-incorporations and renames
- Officer records are spottier than company records (many jurisdictions
don't require officer disclosure)
- Beneficial-ownership data is generally NOT here — most jurisdictions
don't require it. UK Companies House has PSC (people with significant
control) but that's not universal.
- Cross-jurisdictional links (parent / subsidiary) are based on registry
filings only; corporate trees are often incomplete
## 7. Acquisition Script
Path: `scripts/fetch_opencorporates.py`
```bash
# Search globally by name
python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
--out data/oc.csv
# Restrict to a jurisdiction
python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
--jurisdiction us_ny --out data/oc_ny.csv
# Set token via env or flag
OPENCORPORATES_API_TOKEN=xxx python3 SKILL_DIR/scripts/fetch_opencorporates.py \
--query "Microsoft" --out data/oc.csv
```
Without a token the script falls back to scraping the HTML search page.
The fallback is brittle and only fills in `name`, `jurisdiction_code`,
`opencorporates_url` — set the token for serious work.
## 8. Legal & Licensing
- OpenCorporates aggregates public records — the underlying facts are
public domain
- OpenCorporates own database is licensed CC-BY-SA-4.0; attribution required
- API ToS prohibits redistributing the full dataset; per-record reference
is fine
## 9. References
- API docs: https://api.opencorporates.com/documentation/API-Reference
- Jurisdiction codes: https://api.opencorporates.com/v0.4/jurisdictions.json
- Schema: https://opencorporates.com/info/our_data
optional-skills/research/osint-investigation/references/sources/sec-edgar.md
+83
View File
@@ -0,0 +1,83 @@
# SEC EDGAR — Corporate Filings
## 1. Summary
EDGAR (Electronic Data Gathering, Analysis, and Retrieval) is the SEC's system
for corporate disclosure filings: 10-K (annual), 10-Q (quarterly), 8-K (current
events), DEF 14A (proxy), Form 4 (insider trading), 13F (institutional holdings).
## 2. Access Methods
- **API:** `https://data.sec.gov/submissions/CIK<10-digit-padded>.json` (no auth)
- **Filing index:** `https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=...`
- **Full-text search:** `https://efts.sec.gov/LATEST/search-index?q=...`
- **Auth:** None — requires `User-Agent` header with contact info per SEC policy
- **Rate limit:** 10 requests/second per IP (enforced)
## 3. Data Schema
Key fields emitted by `fetch_sec_edgar.py` (filings index):
| Column | Type | Description |
|--------|------|-------------|
| `cik` | str | Central Index Key (10-digit padded) |
| `company_name` | str | Registrant name |
| `form_type` | str | 10-K, 10-Q, 8-K, etc. |
| `filing_date` | str | YYYY-MM-DD |
| `accession_number` | str | Filing accession (e.g. 0000320193-24-000123) |
| `primary_document` | str | Filename of main document |
| `filing_url` | str | Direct URL to filing index |
| `reporting_period` | str | Period of report (where applicable) |
## 4. Coverage
- All public US registrants from 1993 → present
- 1993-2000 has spotty coverage of older filings (paper-to-electronic migration)
- ~12M filings cumulative
- Updated within minutes of filing acceptance
## 5. Cross-Reference Potential
- **USAspending** ↔ `company_name` (public companies as federal contractors)
- **Senate LD** ↔ `company_name` (public companies hire lobbyists)
- **OFAC SDN** ↔ `company_name` (sanctions screening of public registrants)
Join key: company name OR CIK if you have it. CIK is canonical and stable.
## 6. Data Quality
- Subsidiaries often filed under parent CIK — be careful with name matches
- Name changes over time (rebrands, acquisitions) — CIK remains constant
- 10-K Item 1A Risk Factors are free-form text — useful for `web_extract`-style
parsing, not structured queries
- Foreign private issuers file 20-F instead of 10-K
## 7. Acquisition Script
Path: `scripts/fetch_sec_edgar.py`
```bash
# By CIK
python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
--types 10-K,10-Q --out data/edgar_filings.csv
# By company name (resolves to CIK first via name search)
python3 SKILL_DIR/scripts/fetch_sec_edgar.py --company "APPLE INC" \
--types 8-K --since 2024-01-01 --out data/edgar_filings.csv
```
Set `SEC_USER_AGENT` env var with your contact email (SEC requirement).
Example: `SEC_USER_AGENT="Research example@example.com"`.
## 8. Legal & Licensing
- Public record under SEC Rule 24b-2 / 17 CFR § 230.401
- No commercial use restrictions on filing content
- SEC asks all bulk users to include a `User-Agent` with contact info and to
respect 10 req/s — failure to do so can result in IP blocking
## 9. References
- Developer docs: https://www.sec.gov/edgar/sec-api-documentation
- EDGAR full-text search: https://efts.sec.gov/LATEST/search-index
- Fair access policy: https://www.sec.gov/os/accessing-edgar-data
optional-skills/research/osint-investigation/references/sources/senate-ld.md
+89
View File
@@ -0,0 +1,89 @@
# Senate LD — Lobbying Disclosure (LD-1 / LD-2)
## 1. Summary
The Senate Office of Public Records publishes lobbying disclosures under the
Lobbying Disclosure Act of 1995 (LDA, as amended by HLOGA 2007). LD-1 is
registration of a new client-lobbyist relationship; LD-2 is the quarterly
activity report.
## 2. Access Methods
- **API:** `https://lda.senate.gov/api/v1/` (no auth required for read-only)
- **Bulk download:** `https://lda.senate.gov/api/v1/filings/?format=csv` (paginated)
- **Auth:** Token required for >120 req/hour — register at https://lda.senate.gov/api/auth/register/
- **Rate limit:** 120 req/hour unauthenticated, 1,200 req/hour authenticated
## 3. Data Schema
Key fields emitted by `fetch_senate_ld.py`:
| Column | Type | Description |
|--------|------|-------------|
| `filing_uuid` | str | Unique filing ID |
| `filing_type` | str | LD-1, LD-2, LD-203, etc. |
| `filing_year` | int | Year |
| `filing_period` | str | Q1/Q2/Q3/Q4 or annual |
| `registrant_name` | str | Lobbying firm or organization |
| `registrant_id` | str | Senate-assigned registrant ID |
| `client_name` | str | Client being represented |
| `client_id` | str | Senate-assigned client ID |
| `client_general_description` | str | Client industry / business |
| `income` | float | LD-2 income from client this quarter (USD) |
| `expenses` | float | LD-2 expenses (in-house lobbying) |
| `lobbyists` | str | Semicolon-separated lobbyist names |
| `issues` | str | Semicolon-separated issue areas |
| `government_entities` | str | Agencies/chambers contacted |
| `filing_date` | str | YYYY-MM-DD |
## 4. Coverage
- US federal lobbying only (state lobbying handled by individual state ethics offices)
- 1999 → present (full electronic coverage from 2008)
- Quarterly reporting cycle (LD-2)
- ~1M+ filings cumulative
## 5. Cross-Reference Potential
- **USAspending** ↔ `client_name` (clients lobbying for contracts)
- **SEC EDGAR** ↔ `client_name` (public companies as lobbying clients)
- **OFAC SDN** ↔ `client_name` (sanctions screening of lobbying clients)
Join key: normalized client_name. registrant_id and client_id are canonical
when joining Senate-internal records.
## 6. Data Quality
- Many lobbyist names appear in multiple registrants over time (job changes)
- `issues` and `government_entities` are free-text — Inconsistent capitalization
- Foreign agents register under FARA (Department of Justice), NOT here
- Income/expenses are reported in $10,000 brackets in some older filings
## 7. Acquisition Script
Path: `scripts/fetch_senate_ld.py`
```bash
# By client
python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
--year 2024 --out data/lobbying.csv
# By registrant (lobbying firm)
python3 SKILL_DIR/scripts/fetch_senate_ld.py --registrant "BIG K STREET LLP" \
--year 2024 --out data/lobbying.csv
```
Set `SENATE_LDA_TOKEN` env var if you have one (or pass `--token`).
Defaults to anonymous (120 req/hour).
## 8. Legal & Licensing
- Public record under 2 U.S.C. § 1604 (LDA)
- No commercial use restrictions
- Reuse is unconditional — see Senate Public Records Office disclaimer
## 9. References
- API docs: https://lda.senate.gov/api/redoc/v1/
- LDA guidance: https://lobbyingdisclosure.house.gov/ld_guidance.pdf
- Senate Public Records: https://lda.senate.gov/
optional-skills/research/osint-investigation/references/sources/usaspending.md
+97
View File
@@ -0,0 +1,97 @@
# USAspending — Federal Government Contracts and Grants
## 1. Summary
USAspending.gov is the official source of federal spending data. Coverage:
contracts, grants, loans, direct payments, sub-awards. Required by the DATA Act
of 2014 — all federal agencies must report to a single schema.
## 2. Access Methods
- **API v2:** `https://api.usaspending.gov/api/v2/` (no auth, no key)
- **Bulk:** `https://files.usaspending.gov/` (CSV / Parquet by award type)
- **Auth:** None
- **Rate limit:** Not strictly enforced, but be polite — keep to <10 req/s
## 3. Data Schema
Key fields emitted by `fetch_usaspending.py` (prime awards):
| Column | Type | Description |
|--------|------|-------------|
| `award_id` | str | Federal award ID (PIID for contracts, FAIN for grants) |
| `recipient_name` | str | Awardee legal name |
| `recipient_uei` | str | Unique Entity Identifier (replaced DUNS in 2022) |
| `recipient_duns` | str | Legacy DUNS number (historical only) |
| `recipient_parent_name` | str | Ultimate parent organization |
| `recipient_state` | str | Recipient state |
| `awarding_agency` | str | Department / agency name |
| `awarding_sub_agency` | str | Sub-tier (e.g. DoD → Army) |
| `award_type` | str | Contract / Grant / Loan / Direct Payment |
| `award_amount` | float | Current total obligation in USD |
| `award_date` | str | Action / signed date YYYY-MM-DD |
| `period_of_performance_start` | str | YYYY-MM-DD |
| `period_of_performance_end` | str | YYYY-MM-DD |
| `naics_code` | str | Industry classification |
| `psc_code` | str | Product / Service Code |
| `competition_extent` | str | Full / limited / sole-source |
| `description` | str | Award description (free-text) |
## 4. Coverage
- US federal awards only (state/local not included)
- FY 2008 → present (full coverage from FY 2017)
- Updated bi-weekly from agency reporting
- ~100M+ transaction records cumulative
## 5. Cross-Reference Potential
- **SEC EDGAR** ↔ `recipient_name` (public companies as contractors)
- **Senate LD** ↔ `recipient_name` (lobbying clients winning contracts)
- **OFAC SDN** ↔ `recipient_name` (sanctions screening of contractors — must be
filtered out by SAM.gov but verify)
- **ICIJ Offshore** ↔ `recipient_name` (offshore-linked contractors)
Join key: normalized recipient name. UEI is canonical when present.
## 6. Data Quality
- DUNS → UEI transition (April 2022) — old records have DUNS, new records have UEI
- Some sub-awards aren't reported (FFATA threshold is $30k)
- Award amount changes over time (mod actions) — fetch script reports current total
- `competition_extent` field is free-text in older records — `fetch_usaspending.py`
normalizes to canonical values
- Recipient name variations are extensive — "ACME LLC", "Acme L.L.C.", "ACME, INC"
all appear. Use `entity_resolution.py`.
## 7. Acquisition Script
Path: `scripts/fetch_usaspending.py`
```bash
# By recipient name
python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
--fy 2024 --out data/contracts.csv
# By awarding agency
python3 SKILL_DIR/scripts/fetch_usaspending.py --agency "Department of Defense" \
--fy 2024 --out data/contracts.csv
# Filter to sole-source only
python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
--fy 2024 --sole-source-only --out data/contracts.csv
```
## 8. Legal & Licensing
- Public record under the Federal Funding Accountability and Transparency Act
(FFATA, 2006) and DATA Act (2014)
- No commercial use restrictions on the data
- Personal information of award recipients (e.g. small business owners' addresses
in some grants) should be handled per the source agency's privacy notice
## 9. References
- API docs: https://api.usaspending.gov/
- Data dictionary: https://www.usaspending.gov/data-dictionary
- Award schema: https://files.usaspending.gov/docs/Data_Dictionary_Crosswalk.xlsx
optional-skills/research/osint-investigation/references/sources/wayback.md
+93
View File
@@ -0,0 +1,93 @@
# Wayback Machine — Internet Archive CDX
## 1. Summary
The Internet Archive's Wayback Machine has captured ~900B+ web pages since
1996. The CDX server API indexes those captures by URL, timestamp, and
content hash. Free, anonymous, no auth.
## 2. Access Methods
- **CDX server:** `https://web.archive.org/cdx/search/cdx`
- **Wayback URL:** `https://web.archive.org/web/<timestamp>/<url>`
- **Save Page Now (write):** `https://web.archive.org/save/<url>` (different API)
- **Auth:** None
- **Rate limit:** Generous; be polite (~1 req/s)
## 3. Data Schema
Key fields emitted by `fetch_wayback.py`:
| Column | Type | Description |
|--------|------|-------------|
| `url` | str | Original URL captured |
| `timestamp` | str | YYYYMMDDHHMMSS (CDX format) |
| `wayback_url` | str | Direct replay URL |
| `mimetype` | str | Content-type at capture |
| `status` | str | HTTP status (typically 200) |
| `digest` | str | SHA1 of capture content (collapse-friendly) |
| `length` | str | Byte length of capture |
## 4. Coverage
- 1996 → present
- ~900B+ captures across ~700M domains
- Updated continuously by automated crawls + manual saves
- Some domains have aggressive coverage (news), others sparse (private)
## 5. Cross-Reference Potential
- **Wikipedia** ↔ Reverse-lookup pages cited as references that have since
disappeared
- **News URLs** ↔ Original article content when present-day URLs 404
- **Corporate websites** ↔ Historical "About" pages, executive bios that
have been scrubbed
The Wayback CDX is most useful as a **content-recovery** layer when other
sources point to URLs that no longer exist.
## 6. Data Quality
- robots.txt-blocked domains may have spotty or no coverage
- Captures vary in completeness (HTML may be saved without CSS/JS)
- Some content is excluded by domain owner request (DMCA, etc.)
- Coverage of "deep links" (URLs with query strings) is uneven
- Time resolution is per-capture, not continuous — gaps are common
## 7. Acquisition Script
Path: `scripts/fetch_wayback.py`
```bash
# All captures of a specific URL
python3 SKILL_DIR/scripts/fetch_wayback.py --url "https://example.com/page" \
--out data/wb.csv
# All captures of a host
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
--match host --out data/wb.csv
# All captures of a domain + subdomains
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
--match domain --out data/wb.csv
# Only unique-content captures within a date window
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
--match host --collapse digest \
--from-date 2020-01-01 --to-date 2023-12-31 \
--out data/wb.csv
```
## 8. Legal & Licensing
- Internet Archive captures are made under fair-use research provisions
- Replay URLs are stable references — citing them is encouraged
- Internet Archive non-profit terms of use govern content
- Some content is rights-restricted; replay may be blocked even if the
CDX entry shows it as captured
## 9. References
- CDX server docs: https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md
- Wayback API: https://archive.org/help/wayback_api.php
- Internet Archive: https://archive.org/
optional-skills/research/osint-investigation/references/sources/wikipedia.md
+107
View File
@@ -0,0 +1,107 @@
# Wikipedia + Wikidata
## 1. Summary
Wikipedia is the canonical narrative-bio source for notable people, places,
and organizations. Wikidata is its structured-data counterpart: ~110M
items, each with claims, dates, identifiers, and cross-references to
external authorities (VIAF, ISNI, ORCID, GRID, etc.).
Together they're a high-precision entity-resolution layer — the bar for
inclusion is real, but anything past that bar is well-cross-referenced.
## 2. Access Methods
- **Wikipedia OpenSearch:** `https://en.wikipedia.org/w/api.php?action=opensearch`
- **Wikipedia REST summary:** `https://en.wikipedia.org/api/rest_v1/page/summary/<title>`
- **Wikidata Action API:** `https://www.wikidata.org/w/api.php?action=wbgetentities`
- **Wikidata SPARQL:** `https://query.wikidata.org/sparql` (more powerful but aggressively rate-limited)
- **Auth:** None, but **a meaningful User-Agent is required**
Set `HERMES_OSINT_UA` to something identifying (e.g. `your-app/1.0 (you@example.com)`).
Wikimedia returns HTTP 429 to generic UAs.
## 3. Data Schema
Key fields emitted by `fetch_wikipedia.py`:
| Column | Type | Description |
|--------|------|-------------|
| `source` | str | `wikipedia` or `wikipedia+wikidata` |
| `label` | str | Wikipedia article title |
| `description` | str | Short Wikidata description |
| `qid` | str | Wikidata QID (e.g. Q2283 for Microsoft) |
| `wikipedia_title`, `wikipedia_url` | str | Article identifier + URL |
| `wikidata_url` | str | Wikidata entity URL |
| `instance_of` | str | What kind of thing it is (P31) |
| `country` | str | Country (P17 for orgs/places, P27 for people) |
| `occupation` | str | P106 |
| `employer` | str | P108 |
| `date_of_birth` | str | P569, YYYY-MM-DD |
| `place_of_birth` | str | P19 |
| `summary` | str | Wikipedia REST extract (~1000 chars) |
The fetch script uses Wikidata's Action API (NOT SPARQL) for structured
facts — far more lenient on rate limits.
## 4. Coverage
- Wikipedia EN: ~7M articles
- Wikidata: ~110M items, ~1.5B statements
- Updated continuously; abuse filters and bots run constantly
- High notability bar — most private individuals are not in Wikipedia
## 5. Cross-Reference Potential
- **All sources** ↔ `label` (entity identity resolution)
- **SEC EDGAR** ↔ `label` (public companies)
- **CourtListener** ↔ `label` (parties to notable litigation)
- **Wikidata external identifiers** (not currently in this fetcher's output)
link to VIAF, ISNI, ORCID, GRID, GitHub, Twitter, IMDb, ...
Join key: Wikidata QID is canonical. Wikipedia titles are stable for
most articles but can be renamed.
## 6. Data Quality
- Notability filter — only notable entities (criteria vary by topic)
- Recency lag — current events take days to weeks to be reflected
- POV / vandalism — moderated, but edits between sweeps can be bad
- Living-persons biographies have stricter sourcing requirements
- Wikidata claims have qualifiers and references — the fetch script
doesn't currently export them
## 7. Acquisition Script
Path: `scripts/fetch_wikipedia.py`
```bash
# Look up a notable entity
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --out data/wp.csv
# A specific person
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" --out data/wp_bg.csv
# Skip the Wikidata enrichment for speed
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --no-wikidata \
--limit 5 --out data/wp.csv
```
The OpenSearch is fuzzy — `--limit 5` returns the top 5 Wikipedia article
matches. Each is enriched with the QID + structured facts unless
`--no-wikidata` is passed.
## 8. Legal & Licensing
- Wikipedia text: CC-BY-SA-3.0 / GFDL
- Wikidata claims: CC0 (public domain)
- API ToS: respect rate limits, identify your agent
- Commercial use allowed with attribution
## 9. References
- Wikipedia OpenSearch: https://www.mediawiki.org/wiki/API:Opensearch
- Wikipedia REST: https://en.wikipedia.org/api/rest_v1/
- Wikidata Action API: https://www.wikidata.org/wiki/Wikidata:Data_access
- Wikidata SPARQL: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service
- User-Agent policy: https://meta.wikimedia.org/wiki/User-Agent_policy