forked from Zakaria/hermes-agent
Hermes-agent
This commit is contained in:
Zakaria
@@ -0,0 +1,235 @@
|
||||
---
|
||||
name: bioinformatics
|
||||
description: Gateway to 400+ bioinformatics skills from bioSkills and ClawBio. Covers genomics, transcriptomics, single-cell, variant calling, pharmacogenomics, metagenomics, structural biology, and more. Fetches domain-specific reference material on demand.
|
||||
version: 1.0.0
|
||||
platforms: [linux, macos]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [bioinformatics, genomics, sequencing, biology, research, science]
|
||||
category: research
|
||||
---
|
||||
|
||||
# Bioinformatics Skills Gateway
|
||||
|
||||
Use when asked about bioinformatics, genomics, sequencing, variant calling, gene expression, single-cell analysis, protein structure, pharmacogenomics, metagenomics, phylogenetics, or any computational biology task.
|
||||
|
||||
This skill is a gateway to two open-source bioinformatics skill libraries. Instead of bundling hundreds of domain-specific skills, it indexes them and fetches what you need on demand.
|
||||
|
||||
## Sources
|
||||
|
||||
◆ **bioSkills** — 385 reference skills (code patterns, parameter guides, decision trees)
|
||||
Repo: https://github.com/GPTomics/bioSkills
|
||||
Format: SKILL.md per topic with code examples. Python/R/CLI.
|
||||
|
||||
◆ **ClawBio** — 33 runnable pipeline skills (executable scripts, reproducibility bundles)
|
||||
Repo: https://github.com/ClawBio/ClawBio
|
||||
Format: Python scripts with demos. Each analysis exports report.md + commands.sh + environment.yml.
|
||||
|
||||
## How to fetch and use a skill
|
||||
|
||||
1. Identify the domain and skill name from the index below.
|
||||
2. Clone the relevant repo (shallow clone to save time):
|
||||
```bash
|
||||
# bioSkills (reference material)
|
||||
git clone --depth 1 https://github.com/GPTomics/bioSkills.git /tmp/bioSkills
|
||||
|
||||
# ClawBio (runnable pipelines)
|
||||
git clone --depth 1 https://github.com/ClawBio/ClawBio.git /tmp/ClawBio
|
||||
```
|
||||
3. Read the specific skill:
|
||||
```bash
|
||||
# bioSkills — each skill is at: <category>/<skill-name>/SKILL.md
|
||||
cat /tmp/bioSkills/variant-calling/gatk-variant-calling/SKILL.md
|
||||
|
||||
# ClawBio — each skill is at: skills/<skill-name>/
|
||||
cat /tmp/ClawBio/skills/pharmgx-reporter/README.md
|
||||
```
|
||||
4. Follow the fetched skill as reference material. These are NOT Hermes-format skills — treat them as expert domain guides. They contain correct parameters, proper tool flags, and validated pipelines.
|
||||
|
||||
## Skill Index by Domain
|
||||
|
||||
### Sequence Fundamentals
|
||||
bioSkills:
|
||||
sequence-io/ — read-sequences, write-sequences, format-conversion, batch-processing, compressed-files, fastq-quality, filter-sequences, paired-end-fastq, sequence-statistics
|
||||
sequence-manipulation/ — seq-objects, reverse-complement, transcription-translation, motif-search, codon-usage, sequence-properties, sequence-slicing
|
||||
ClawBio:
|
||||
seq-wrangler — Sequence QC, alignment, and BAM processing (wraps FastQC, BWA, SAMtools)
|
||||
|
||||
### Read QC & Alignment
|
||||
bioSkills:
|
||||
read-qc/ — quality-reports, fastp-workflow, adapter-trimming, quality-filtering, umi-processing, contamination-screening, rnaseq-qc
|
||||
read-alignment/ — bwa-alignment, star-alignment, hisat2-alignment, bowtie2-alignment
|
||||
alignment-files/ — sam-bam-basics, alignment-sorting, alignment-filtering, bam-statistics, duplicate-handling, pileup-generation
|
||||
|
||||
### Variant Calling & Annotation
|
||||
bioSkills:
|
||||
variant-calling/ — gatk-variant-calling, deepvariant, variant-calling (bcftools), joint-calling, structural-variant-calling, filtering-best-practices, variant-annotation, variant-normalization, vcf-basics, vcf-manipulation, vcf-statistics, consensus-sequences, clinical-interpretation
|
||||
ClawBio:
|
||||
vcf-annotator — VEP + ClinVar + gnomAD annotation with ancestry-aware context
|
||||
variant-annotation — Variant annotation pipeline
|
||||
|
||||
### Differential Expression (Bulk RNA-seq)
|
||||
bioSkills:
|
||||
differential-expression/ — deseq2-basics, edger-basics, batch-correction, de-results, de-visualization, timeseries-de
|
||||
rna-quantification/ — alignment-free-quant (Salmon/kallisto), featurecounts-counting, tximport-workflow, count-matrix-qc
|
||||
expression-matrix/ — counts-ingest, gene-id-mapping, metadata-joins, sparse-handling
|
||||
ClawBio:
|
||||
rnaseq-de — Full DE pipeline with QC, normalization, and visualization
|
||||
diff-visualizer — Rich visualization and reporting for DE results
|
||||
|
||||
### Single-Cell RNA-seq
|
||||
bioSkills:
|
||||
single-cell/ — preprocessing, clustering, batch-integration, cell-annotation, cell-communication, doublet-detection, markers-annotation, trajectory-inference, multimodal-integration, perturb-seq, scatac-analysis, lineage-tracing, metabolite-communication, data-io
|
||||
ClawBio:
|
||||
scrna-orchestrator — Full Scanpy pipeline (QC, clustering, markers, annotation)
|
||||
scrna-embedding — scVI-based latent embedding and batch integration
|
||||
|
||||
### Spatial Transcriptomics
|
||||
bioSkills:
|
||||
spatial-transcriptomics/ — spatial-data-io, spatial-preprocessing, spatial-domains, spatial-deconvolution, spatial-communication, spatial-neighbors, spatial-statistics, spatial-visualization, spatial-multiomics, spatial-proteomics, image-analysis
|
||||
|
||||
### Epigenomics
|
||||
bioSkills:
|
||||
chip-seq/ — peak-calling, differential-binding, motif-analysis, peak-annotation, chipseq-qc, chipseq-visualization, super-enhancers
|
||||
atac-seq/ — atac-peak-calling, atac-qc, differential-accessibility, footprinting, motif-deviation, nucleosome-positioning
|
||||
methylation-analysis/ — bismark-alignment, methylation-calling, dmr-detection, methylkit-analysis
|
||||
hi-c-analysis/ — hic-data-io, tad-detection, loop-calling, compartment-analysis, contact-pairs, matrix-operations, hic-visualization, hic-differential
|
||||
ClawBio:
|
||||
methylation-clock — Epigenetic age estimation
|
||||
|
||||
### Pharmacogenomics & Clinical
|
||||
bioSkills:
|
||||
clinical-databases/ — clinvar-lookup, gnomad-frequencies, dbsnp-queries, pharmacogenomics, polygenic-risk, hla-typing, variant-prioritization, somatic-signatures, tumor-mutational-burden, myvariant-queries
|
||||
ClawBio:
|
||||
pharmgx-reporter — PGx report from 23andMe/AncestryDNA (12 genes, 31 SNPs, 51 drugs)
|
||||
drug-photo — Photo of medication → personalized PGx dosage card (via vision)
|
||||
clinpgx — ClinPGx API for gene-drug data and CPIC guidelines
|
||||
gwas-lookup — Federated variant lookup across 9 genomic databases
|
||||
gwas-prs — Polygenic risk scores from consumer genetic data
|
||||
nutrigx_advisor — Personalized nutrition from consumer genetic data
|
||||
|
||||
### Population Genetics & GWAS
|
||||
bioSkills:
|
||||
population-genetics/ — association-testing (PLINK GWAS), plink-basics, population-structure, linkage-disequilibrium, scikit-allel-analysis, selection-statistics
|
||||
causal-genomics/ — mendelian-randomization, fine-mapping, colocalization-analysis, mediation-analysis, pleiotropy-detection
|
||||
phasing-imputation/ — haplotype-phasing, genotype-imputation, imputation-qc, reference-panels
|
||||
ClawBio:
|
||||
claw-ancestry-pca — Ancestry PCA against SGDP reference panel
|
||||
|
||||
### Metagenomics & Microbiome
|
||||
bioSkills:
|
||||
metagenomics/ — kraken-classification, metaphlan-profiling, abundance-estimation, functional-profiling, amr-detection, strain-tracking, metagenome-visualization
|
||||
microbiome/ — amplicon-processing, diversity-analysis, differential-abundance, taxonomy-assignment, functional-prediction, qiime2-workflow
|
||||
ClawBio:
|
||||
claw-metagenomics — Shotgun metagenomics profiling (taxonomy, resistome, functional pathways)
|
||||
|
||||
### Genome Assembly & Annotation
|
||||
bioSkills:
|
||||
genome-assembly/ — hifi-assembly, long-read-assembly, short-read-assembly, metagenome-assembly, assembly-polishing, assembly-qc, scaffolding, contamination-detection
|
||||
genome-annotation/ — eukaryotic-gene-prediction, prokaryotic-annotation, functional-annotation, ncrna-annotation, repeat-annotation, annotation-transfer
|
||||
long-read-sequencing/ — basecalling, long-read-alignment, long-read-qc, clair3-variants, structural-variants, medaka-polishing, nanopore-methylation, isoseq-analysis
|
||||
|
||||
### Structural Biology & Chemoinformatics
|
||||
bioSkills:
|
||||
structural-biology/ — alphafold-predictions, modern-structure-prediction, structure-io, structure-navigation, structure-modification, geometric-analysis
|
||||
chemoinformatics/ — molecular-io, molecular-descriptors, similarity-searching, substructure-search, virtual-screening, admet-prediction, reaction-enumeration
|
||||
ClawBio:
|
||||
struct-predictor — Local AlphaFold/Boltz/Chai structure prediction with comparison
|
||||
|
||||
### Proteomics
|
||||
bioSkills:
|
||||
proteomics/ — data-import, peptide-identification, protein-inference, quantification, differential-abundance, dia-analysis, ptm-analysis, proteomics-qc, spectral-libraries
|
||||
ClawBio:
|
||||
proteomics-de — Proteomics differential expression
|
||||
|
||||
### Pathway Analysis & Gene Networks
|
||||
bioSkills:
|
||||
pathway-analysis/ — go-enrichment, gsea, kegg-pathways, reactome-pathways, wikipathways, enrichment-visualization
|
||||
gene-regulatory-networks/ — scenic-regulons, coexpression-networks, differential-networks, multiomics-grn, perturbation-simulation
|
||||
|
||||
### Immunoinformatics
|
||||
bioSkills:
|
||||
immunoinformatics/ — mhc-binding-prediction, epitope-prediction, neoantigen-prediction, immunogenicity-scoring, tcr-epitope-binding
|
||||
tcr-bcr-analysis/ — mixcr-analysis, scirpy-analysis, immcantation-analysis, repertoire-visualization, vdjtools-analysis
|
||||
|
||||
### CRISPR & Genome Engineering
|
||||
bioSkills:
|
||||
crispr-screens/ — mageck-analysis, jacks-analysis, hit-calling, screen-qc, library-design, crispresso-editing, base-editing-analysis, batch-correction
|
||||
genome-engineering/ — grna-design, off-target-prediction, hdr-template-design, base-editing-design, prime-editing-design
|
||||
|
||||
### Workflow Management
|
||||
bioSkills:
|
||||
workflow-management/ — snakemake-workflows, nextflow-pipelines, cwl-workflows, wdl-workflows
|
||||
ClawBio:
|
||||
repro-enforcer — Export any analysis as reproducibility bundle (Conda env + Singularity + checksums)
|
||||
galaxy-bridge — Access 8,000+ Galaxy tools from usegalaxy.org
|
||||
|
||||
### Specialized Domains
|
||||
bioSkills:
|
||||
alternative-splicing/ — splicing-quantification, differential-splicing, isoform-switching, sashimi-plots, single-cell-splicing, splicing-qc
|
||||
ecological-genomics/ — edna-metabarcoding, landscape-genomics, conservation-genetics, biodiversity-metrics, community-ecology, species-delimitation
|
||||
epidemiological-genomics/ — pathogen-typing, variant-surveillance, phylodynamics, transmission-inference, amr-surveillance
|
||||
liquid-biopsy/ — cfdna-preprocessing, ctdna-mutation-detection, fragment-analysis, tumor-fraction-estimation, methylation-based-detection, longitudinal-monitoring
|
||||
epitranscriptomics/ — m6a-peak-calling, m6a-differential, m6anet-analysis, merip-preprocessing, modification-visualization
|
||||
metabolomics/ — xcms-preprocessing, metabolite-annotation, normalization-qc, statistical-analysis, pathway-mapping, lipidomics, targeted-analysis, msdial-preprocessing
|
||||
flow-cytometry/ — fcs-handling, gating-analysis, compensation-transformation, clustering-phenotyping, differential-analysis, cytometry-qc, doublet-detection, bead-normalization
|
||||
systems-biology/ — flux-balance-analysis, metabolic-reconstruction, gene-essentiality, context-specific-models, model-curation
|
||||
rna-structure/ — secondary-structure-prediction, ncrna-search, structure-probing
|
||||
|
||||
### Data Visualization & Reporting
|
||||
bioSkills:
|
||||
data-visualization/ — ggplot2-fundamentals, heatmaps-clustering, volcano-customization, circos-plots, genome-browser-tracks, interactive-visualization, multipanel-figures, network-visualization, upset-plots, color-palettes, specialized-omics-plots, genome-tracks
|
||||
reporting/ — rmarkdown-reports, quarto-reports, jupyter-reports, automated-qc-reports, figure-export
|
||||
ClawBio:
|
||||
profile-report — Analysis profile reporting
|
||||
data-extractor — Extract numerical data from scientific figure images (via vision)
|
||||
lit-synthesizer — PubMed/bioRxiv search, summarization, citation graphs
|
||||
pubmed-summariser — Gene/disease PubMed search with structured briefing
|
||||
|
||||
### Database Access
|
||||
bioSkills:
|
||||
database-access/ — entrez-search, entrez-fetch, entrez-link, blast-searches, local-blast, sra-data, geo-data, uniprot-access, batch-downloads, interaction-databases, sequence-similarity
|
||||
ClawBio:
|
||||
ukb-navigator — Semantic search across 12,000+ UK Biobank fields
|
||||
clinical-trial-finder — Clinical trial discovery
|
||||
|
||||
### Experimental Design
|
||||
bioSkills:
|
||||
experimental-design/ — power-analysis, sample-size, batch-design, multiple-testing
|
||||
|
||||
### Machine Learning for Omics
|
||||
bioSkills:
|
||||
machine-learning/ — omics-classifiers, biomarker-discovery, survival-analysis, model-validation, prediction-explanation, atlas-mapping
|
||||
ClawBio:
|
||||
claw-semantic-sim — Semantic similarity index for disease literature (PubMedBERT)
|
||||
omics-target-evidence-mapper — Aggregate target-level evidence across omics sources
|
||||
|
||||
## Environment Setup
|
||||
|
||||
These skills assume a bioinformatics workstation. Common dependencies:
|
||||
|
||||
```bash
|
||||
# Python
|
||||
pip install biopython pysam cyvcf2 pybedtools pyBigWig scikit-allel anndata scanpy mygene
|
||||
|
||||
# R/Bioconductor
|
||||
Rscript -e 'BiocManager::install(c("DESeq2","edgeR","Seurat","clusterProfiler","methylKit"))'
|
||||
|
||||
# CLI tools (Ubuntu/Debian)
|
||||
sudo apt install samtools bcftools ncbi-blast+ minimap2 bedtools
|
||||
|
||||
# CLI tools (macOS)
|
||||
brew install samtools bcftools blast minimap2 bedtools
|
||||
|
||||
# Or via Conda (recommended for reproducibility)
|
||||
conda install -c bioconda samtools bcftools blast minimap2 bedtools fastp kraken2
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- The fetched skills are NOT in Hermes SKILL.md format. They use their own structure (bioSkills: code pattern cookbooks; ClawBio: README + Python scripts). Read them as expert reference material.
|
||||
- bioSkills are reference guides — they show correct parameters and code patterns but aren't executable pipelines.
|
||||
- ClawBio skills are executable — many have `--demo` flags and can be run directly.
|
||||
- Both repos assume bioinformatics tools are installed. Check prerequisites before running pipelines.
|
||||
- For ClawBio, run `pip install -r requirements.txt` in the cloned repo first.
|
||||
- Genomic data files can be very large. Be mindful of disk space when downloading reference genomes, SRA datasets, or building indices.
|
||||
@@ -0,0 +1,199 @@
|
||||
---
|
||||
name: darwinian-evolver
|
||||
description: Evolve prompts/regex/SQL/code with Imbue's evolution loop.
|
||||
version: 0.1.0
|
||||
author: Bihruze (Asahi0x), Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux, macos]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [evolution, optimization, prompt-engineering, research]
|
||||
related_skills: [arxiv, jupyter-live-kernel]
|
||||
---
|
||||
|
||||
# Darwinian Evolver
|
||||
|
||||
Run Imbue's [darwinian_evolver](https://github.com/imbue-ai/darwinian_evolver) — an
|
||||
LLM-driven evolutionary search loop — to optimize a **prompt, regex, SQL query,
|
||||
or small code snippet** against a fitness function.
|
||||
|
||||
Status: thin wrapper around the upstream tool. The skill installs it, walks the
|
||||
agent through writing a `Problem` definition (organism + evaluator + mutator),
|
||||
and drives the loop via the upstream CLI or a small custom Python driver.
|
||||
|
||||
**License:** the upstream tool is **AGPL-3.0**. The skill ONLY ever invokes it
|
||||
via the upstream CLI or a `subprocess`/`uv run` call (mere aggregation). Do NOT
|
||||
import upstream classes into Hermes itself.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User says "optimize this prompt", "evolve a regex for X", "auto-improve this
|
||||
code/SQL", "search for a better instruction".
|
||||
- You have a scorer (exact match, regex pass-rate, unit test, LLM-judge, runtime
|
||||
metric) AND a starting candidate (organism). If you don't have a scorer, stop
|
||||
and define one first — that's the hard part.
|
||||
- Cost is OK: a typical run is 50–500 LLM calls. On gpt-4o-mini that's pennies;
|
||||
on Claude Sonnet it can be a few dollars.
|
||||
|
||||
Do **not** use this when:
|
||||
- The optimization target is differentiable (use gradient descent / DSPy).
|
||||
- You only need to try 2–3 variants — just write them by hand.
|
||||
- The fitness signal is purely subjective with no measurable criterion.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python ≥3.11
|
||||
- `git`, `uv` (or `pip`)
|
||||
- One of: `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`
|
||||
|
||||
The skill ships a small `parrot_openrouter.py` driver that uses `OPENROUTER_API_KEY`
|
||||
via the OpenAI SDK, so any model on OpenRouter works. The upstream CLI itself
|
||||
hardcodes Anthropic and needs `ANTHROPIC_API_KEY`.
|
||||
|
||||
## Install (One-Time)
|
||||
|
||||
Run via the `terminal` tool:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.hermes/cache/darwinian-evolver && cd ~/.hermes/cache/darwinian-evolver
|
||||
[ -d darwinian_evolver ] || git clone --depth 1 https://github.com/imbue-ai/darwinian_evolver.git
|
||||
cd darwinian_evolver && uv sync
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
cd ~/.hermes/cache/darwinian-evolver/darwinian_evolver \
|
||||
&& uv run darwinian_evolver --help | head -5
|
||||
```
|
||||
|
||||
## Quick Start — The Built-In Parrot Example
|
||||
|
||||
Tiny smoke test (requires `ANTHROPIC_API_KEY`):
|
||||
|
||||
```bash
|
||||
cd ~/.hermes/cache/darwinian-evolver/darwinian_evolver
|
||||
uv run darwinian_evolver parrot \
|
||||
--num_iterations 2 \
|
||||
--num_parents_per_iteration 2 \
|
||||
--mutator_concurrency 2 --evaluator_concurrency 2 \
|
||||
--output_dir /tmp/parrot_demo
|
||||
```
|
||||
|
||||
Outputs:
|
||||
- `/tmp/parrot_demo/snapshots/iteration_N.pkl` — pickled population per iteration
|
||||
- `/tmp/parrot_demo/<jsonl>` — per-iteration JSON log (path printed at end)
|
||||
|
||||
Open `~/.hermes/cache/darwinian-evolver/darwinian_evolver/darwinian_evolver/lineage_visualizer.html`
|
||||
in a browser and load the JSON log to see the evolutionary tree.
|
||||
|
||||
## Quick Start — OpenRouter Driver (No Anthropic Key)
|
||||
|
||||
The skill ships `scripts/parrot_openrouter.py` — same parrot problem, but the
|
||||
LLM call goes through OpenRouter so any provider works.
|
||||
|
||||
```bash
|
||||
# From wherever the skill is installed:
|
||||
SKILL_DIR=~/.hermes/skills/research/darwinian-evolver
|
||||
DE_DIR=~/.hermes/cache/darwinian-evolver/darwinian_evolver
|
||||
|
||||
cd "$DE_DIR" && \
|
||||
EVOLVER_MODEL='openai/gpt-4o-mini' \
|
||||
uv run --with openai python "$SKILL_DIR/scripts/parrot_openrouter.py" \
|
||||
--num_iterations 3 --num_parents_per_iteration 2 \
|
||||
--output_dir /tmp/parrot_or
|
||||
```
|
||||
|
||||
Inspect the result with `scripts/show_snapshot.py`:
|
||||
|
||||
```bash
|
||||
uv run --with openai python "$SKILL_DIR/scripts/show_snapshot.py" \
|
||||
/tmp/parrot_or/snapshots/iteration_3.pkl
|
||||
```
|
||||
|
||||
Expected output: 7 evolved prompt templates ranked by score, with the best
|
||||
landing around 0.6–0.8 (the seed `Say {{ phrase }}` scored 0.000).
|
||||
|
||||
## Defining a Custom Problem
|
||||
|
||||
The skill ships `templates/custom_problem_template.py` — copy, edit, run.
|
||||
Three things you must define:
|
||||
|
||||
1. **`Organism`** — a Pydantic `BaseModel` subclass holding the artifact being
|
||||
evolved (`prompt_template: str`, `regex_pattern: str`, `sql_query: str`,
|
||||
`code_block: str`, etc.). Add a `run(*args)` method that exercises it.
|
||||
|
||||
2. **`Evaluator`** — `.evaluate(organism) -> EvaluationResult(score=..., trainable_failure_cases=[...], holdout_failure_cases=[...], is_viable=True)`.
|
||||
- **`score`** is in `[0, 1]`. Higher is better.
|
||||
- **`trainable_failure_cases`** — what the mutator sees. Include enough
|
||||
context (input, expected, actual) for the LLM to diagnose.
|
||||
- **`holdout_failure_cases`** — kept out of the mutator's view. Use these
|
||||
to detect overfitting.
|
||||
- **`is_viable=True`** unless the organism is completely broken (raises,
|
||||
returns None, etc.). A 0-score viable organism is fine — it just gets
|
||||
down-weighted in parent selection.
|
||||
|
||||
3. **`Mutator`** — `.mutate(organism, failure_cases, learning_log_entries) -> list[Organism]`.
|
||||
Typically: build an LLM prompt that includes the current organism + a
|
||||
failure case + an ask to propose a fix; parse the LLM's response; return
|
||||
a new `Organism`. Return `[]` on parse failure — the loop handles it.
|
||||
|
||||
Then write a driver script that wires `Problem(initial_organism, evaluator, [mutators])`
|
||||
into `EvolveProblemLoop` and iterates over `loop.run(num_iterations=N)` — the
|
||||
shipped `scripts/parrot_openrouter.py` is the reference.
|
||||
|
||||
## Hyperparameters That Actually Matter
|
||||
|
||||
| flag | default | when to change |
|
||||
|---|---|---|
|
||||
| `--num_iterations` | 5 | bump to 10–20 once you trust the evaluator |
|
||||
| `--num_parents_per_iteration` | 4 | drop to 2 for cheap exploration |
|
||||
| `--mutator_concurrency` | 10 | drop to 2–4 to avoid rate limits |
|
||||
| `--evaluator_concurrency` | 10 | same; evaluator hits the LLM too |
|
||||
| `--batch_size` | 1 | raise to 3–5 once your mutator handles multiple failures |
|
||||
| `--verify_mutations` | off | turn on once mutator is wasteful (>10× cost saving on later runs per Imbue) |
|
||||
| `--midpoint_score` | `p75` | leave alone unless scores cluster |
|
||||
| `--sharpness` | 10 | leave alone |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **`Initial organism must be viable`** — set `is_viable=True` in your
|
||||
`EvaluationResult` even on a 0-score seed. The loop refuses non-viable
|
||||
organisms because they imply the loop has nothing to evolve from.
|
||||
2. **Provider content filters kill runs.** Azure-backed OpenRouter models
|
||||
reject phrases like "ignore previous instructions" with HTTP 400. Wrap
|
||||
the LLM call in `try/except` and return `f"<LLM_ERROR: {e}>"` — the
|
||||
evolver will just score that organism 0 and move on.
|
||||
3. **`loop.run()` is a generator** — calling it doesn't run anything until
|
||||
you iterate. Use `for snap in loop.run(num_iterations=N):`.
|
||||
4. **Snapshots are nested pickles.** `iteration_N.pkl` contains a dict with
|
||||
`population_snapshot` (more pickled bytes). To unpickle you must have the
|
||||
`Organism` class importable under the same dotted path it was pickled at.
|
||||
5. **Concurrency defaults are aggressive.** 10/10 will hit rate limits on
|
||||
most providers. Start with 2/2.
|
||||
6. **CLI is hardcoded to Anthropic.** `uv run darwinian_evolver <problem>`
|
||||
reaches for `ANTHROPIC_API_KEY` and uses Claude Sonnet. To use any other
|
||||
provider, write a driver like `parrot_openrouter.py`.
|
||||
7. **AGPL.** Never `from darwinian_evolver import ...` inside Hermes core.
|
||||
Custom driver scripts under `~/.hermes/skills/...` are user-side and fine.
|
||||
8. **No PyPI package.** `pip install darwinian-evolver` will pull the wrong
|
||||
thing. Always install from the GitHub repo.
|
||||
|
||||
## Verification
|
||||
|
||||
After install + a parrot run, exit code 0 from this is sufficient:
|
||||
|
||||
```bash
|
||||
DE_DIR=~/.hermes/cache/darwinian-evolver/darwinian_evolver
|
||||
ls "$DE_DIR/darwinian_evolver/lineage_visualizer.html" >/dev/null && \
|
||||
cd "$DE_DIR" && uv run darwinian_evolver --help >/dev/null && \
|
||||
echo "darwinian-evolver: OK"
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Imbue research post](https://imbue.com/research/2026-02-27-darwinian-evolver/)
|
||||
- [ARC-AGI-2 results](https://imbue.com/research/2026-02-27-arc-agi-2-evolution/)
|
||||
- [imbue-ai/darwinian_evolver](https://github.com/imbue-ai/darwinian_evolver) (AGPL-3.0)
|
||||
- [Darwin Gödel Machines](https://arxiv.org/abs/2505.22954)
|
||||
- [PromptBreeder](https://arxiv.org/abs/2309.16797)
|
||||
@@ -0,0 +1,218 @@
|
||||
"""
|
||||
parrot_openrouter: same as the upstream `parrot` example but the LLM call goes
|
||||
through OpenRouter (OpenAI SDK) instead of Anthropic native. Lets us run an
|
||||
end-to-end evolution with whatever model the user already has paid access to.
|
||||
|
||||
Run with:
|
||||
uv --project darwinian_evolver run python parrot_openrouter.py \
|
||||
--num_iterations 3 --output_dir /tmp/parrot_out
|
||||
|
||||
Reads `OPENROUTER_API_KEY` from the environment.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import jinja2
|
||||
from openai import OpenAI
|
||||
|
||||
# Vendored problem types from upstream (AGPL — only run via subprocess in production)
|
||||
from darwinian_evolver.cli_common import build_hyperparameter_config_from_args
|
||||
from darwinian_evolver.cli_common import register_hyperparameter_args
|
||||
from darwinian_evolver.cli_common import parse_learning_log_view_type
|
||||
from darwinian_evolver.evolve_problem_loop import EvolveProblemLoop
|
||||
from darwinian_evolver.learning_log import LearningLogEntry
|
||||
from darwinian_evolver.problem import EvaluationFailureCase
|
||||
from darwinian_evolver.problem import EvaluationResult
|
||||
from darwinian_evolver.problem import Evaluator
|
||||
from darwinian_evolver.problem import Mutator
|
||||
from darwinian_evolver.problem import Organism
|
||||
from darwinian_evolver.problem import Problem
|
||||
|
||||
DEFAULT_MODEL = os.environ.get("EVOLVER_MODEL", "openai/gpt-4o-mini")
|
||||
|
||||
|
||||
def _client() -> OpenAI:
|
||||
key = os.environ.get("OPENROUTER_API_KEY")
|
||||
if not key:
|
||||
sys.exit("OPENROUTER_API_KEY is not set")
|
||||
return OpenAI(api_key=key, base_url="https://openrouter.ai/api/v1")
|
||||
|
||||
|
||||
def _prompt_llm(prompt: str) -> str:
|
||||
try:
|
||||
r = _client().chat.completions.create(
|
||||
model=DEFAULT_MODEL,
|
||||
max_tokens=1024,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
return r.choices[0].message.content or ""
|
||||
except Exception as e:
|
||||
# Treat any provider error (rate limit, content filter, schema reject)
|
||||
# as a failed response. The evolver will simply see this as a low score
|
||||
# on this organism and move on — much friendlier than killing the run.
|
||||
return f"<LLM_ERROR: {type(e).__name__}: {e}>"
|
||||
|
||||
|
||||
class ParrotOrganism(Organism):
|
||||
prompt_template: str
|
||||
|
||||
def run(self, phrase: str) -> str:
|
||||
try:
|
||||
prompt = jinja2.Template(self.prompt_template).render(phrase=phrase)
|
||||
except jinja2.exceptions.TemplateError as e:
|
||||
return f"Error rendering prompt: {e}"
|
||||
if not prompt:
|
||||
return ""
|
||||
return _prompt_llm(prompt)
|
||||
|
||||
|
||||
class ParrotEvaluationFailureCase(EvaluationFailureCase):
|
||||
phrase: str
|
||||
response: str
|
||||
|
||||
|
||||
class ImproveParrotMutator(Mutator[ParrotOrganism, ParrotEvaluationFailureCase]):
|
||||
IMPROVEMENT_PROMPT_TEMPLATE = """
|
||||
We want to build a prompt that causes an LLM to repeat back a given phrase verbatim.
|
||||
|
||||
The current prompt template is:
|
||||
```
|
||||
{{ organism.prompt_template }}
|
||||
```
|
||||
|
||||
Unfortunately, on this phrase:
|
||||
```
|
||||
{{ failure_case.phrase }}
|
||||
```
|
||||
the LLM responded with:
|
||||
```
|
||||
{{ failure_case.response }}
|
||||
```
|
||||
|
||||
Diagnose what went wrong, then propose an improved prompt template. Put the new
|
||||
template in the LAST triple-backtick block of your response.
|
||||
""".strip()
|
||||
|
||||
def mutate(
|
||||
self,
|
||||
organism: ParrotOrganism,
|
||||
failure_cases: list[ParrotEvaluationFailureCase],
|
||||
learning_log_entries: list[LearningLogEntry],
|
||||
) -> list[ParrotOrganism]:
|
||||
fc = failure_cases[0]
|
||||
prompt = jinja2.Template(self.IMPROVEMENT_PROMPT_TEMPLATE).render(
|
||||
organism=organism, failure_case=fc
|
||||
)
|
||||
try:
|
||||
resp = _prompt_llm(prompt)
|
||||
parts = resp.split("```")
|
||||
if len(parts) < 3:
|
||||
return []
|
||||
new_tpl = parts[-2].strip()
|
||||
return [ParrotOrganism(prompt_template=new_tpl)]
|
||||
except Exception as e:
|
||||
print(f"mutate error: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
|
||||
class ParrotEvaluator(Evaluator[ParrotOrganism, EvaluationResult, ParrotEvaluationFailureCase]):
|
||||
TRAINABLE_PHRASES = [
|
||||
"Hello world.",
|
||||
"bla",
|
||||
"Bla",
|
||||
"bla.",
|
||||
'"bla bla".',
|
||||
"Just say 'foo' once with no extra words.",
|
||||
]
|
||||
HOLDOUT_PHRASES = [
|
||||
"bla, but only once.",
|
||||
"'bla'",
|
||||
]
|
||||
|
||||
def evaluate(self, organism: ParrotOrganism) -> EvaluationResult:
|
||||
train_fails: list[ParrotEvaluationFailureCase] = []
|
||||
hold_fails: list[ParrotEvaluationFailureCase] = []
|
||||
for i, p in enumerate(self.TRAINABLE_PHRASES):
|
||||
r = organism.run(p)
|
||||
if r != p:
|
||||
train_fails.append(ParrotEvaluationFailureCase(
|
||||
phrase=p, response=r, data_point_id=f"trainable_{i}"))
|
||||
for i, p in enumerate(self.HOLDOUT_PHRASES):
|
||||
r = organism.run(p)
|
||||
if r != p:
|
||||
hold_fails.append(ParrotEvaluationFailureCase(
|
||||
phrase=p, response=r, data_point_id=f"holdout_{i}"))
|
||||
n_total = len(self.TRAINABLE_PHRASES) + len(self.HOLDOUT_PHRASES)
|
||||
n_ok = n_total - len(train_fails) - len(hold_fails)
|
||||
return EvaluationResult(
|
||||
score=n_ok / n_total,
|
||||
trainable_failure_cases=train_fails,
|
||||
holdout_failure_cases=hold_fails,
|
||||
# Always viable. Even a 0-score seed is a valid starting point; the
|
||||
# mutator should still get a chance to fix it.
|
||||
is_viable=True,
|
||||
)
|
||||
|
||||
|
||||
def make_problem() -> Problem:
|
||||
return Problem[ParrotOrganism, EvaluationResult, ParrotEvaluationFailureCase](
|
||||
evaluator=ParrotEvaluator(),
|
||||
mutators=[ImproveParrotMutator()],
|
||||
initial_organism=ParrotOrganism(prompt_template="Say {{ phrase }}"),
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
register_hyperparameter_args(ap.add_argument_group("hyperparameters"))
|
||||
ap.add_argument("--num_iterations", type=int, default=3)
|
||||
ap.add_argument("--mutator_concurrency", type=int, default=4)
|
||||
ap.add_argument("--evaluator_concurrency", type=int, default=4)
|
||||
ap.add_argument("--output_dir", type=str, required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
out = Path(args.output_dir)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
hp = build_hyperparameter_config_from_args(args)
|
||||
loop = EvolveProblemLoop(
|
||||
problem=make_problem(),
|
||||
learning_log_view_type=parse_learning_log_view_type(hp.learning_log_view_type),
|
||||
num_parents_per_iteration=hp.num_parents_per_iteration,
|
||||
mutator_concurrency=args.mutator_concurrency,
|
||||
evaluator_concurrency=args.evaluator_concurrency,
|
||||
fixed_midpoint_score=hp.fixed_midpoint_score,
|
||||
midpoint_score_percentile=hp.midpoint_score_percentile,
|
||||
sharpness=hp.sharpness,
|
||||
novelty_weight=hp.novelty_weight,
|
||||
batch_size=hp.batch_size,
|
||||
should_verify_mutations=hp.verify_mutations,
|
||||
)
|
||||
|
||||
import json
|
||||
log_path = out / "results.jsonl"
|
||||
snap_dir = out / "snapshots"
|
||||
snap_dir.mkdir(exist_ok=True)
|
||||
print("Evaluating initial organism...")
|
||||
for snap in loop.run(num_iterations=args.num_iterations):
|
||||
(snap_dir / f"iteration_{snap.iteration}.pkl").write_bytes(snap.snapshot)
|
||||
_, best_eval = snap.best_organism_result
|
||||
print(f"iter={snap.iteration} pop={snap.population_size} "
|
||||
f"best_score={best_eval.score:.3f}")
|
||||
with log_path.open("a") as f:
|
||||
f.write(json.dumps({
|
||||
"iteration": snap.iteration,
|
||||
"best_score": best_eval.score,
|
||||
"pop_size": snap.population_size,
|
||||
"score_percentiles": {str(k): v for k, v in snap.score_percentiles.items()},
|
||||
}) + "\n")
|
||||
print(f"\nDone. Results in: {out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,92 @@
|
||||
"""
|
||||
show_snapshot.py — Dump the population from a darwinian-evolver snapshot pickle.
|
||||
|
||||
Usage:
|
||||
python show_snapshot.py PATH/TO/iteration_N.pkl [--field prompt_template]
|
||||
|
||||
The script is intentionally Organism-agnostic: it walks `org.__dict__` and prints
|
||||
all str fields. By default it shows `prompt_template` if present; pass --field to
|
||||
target a different attribute (e.g. `regex_pattern`, `sql_query`, `code_block`).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import pickle
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("snapshot", type=Path)
|
||||
ap.add_argument(
|
||||
"--field",
|
||||
default=None,
|
||||
help="Organism attribute to display. Defaults to the first str field found.",
|
||||
)
|
||||
ap.add_argument("--top", type=int, default=None, help="Show only top N by score.")
|
||||
ap.add_argument(
|
||||
"--i-trust-this-file",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Required acknowledgement that the snapshot is from a trusted source. "
|
||||
"pickle.loads executes arbitrary code embedded in the file (RCE) and "
|
||||
"must NEVER be run on snapshots received from untrusted parties."
|
||||
),
|
||||
)
|
||||
args = ap.parse_args()
|
||||
|
||||
if not args.snapshot.exists():
|
||||
sys.exit(f"snapshot not found: {args.snapshot}")
|
||||
|
||||
if not args.i_trust_this_file:
|
||||
sys.exit(
|
||||
"refusing to unpickle: pickle.loads is equivalent to executing arbitrary "
|
||||
"code from the snapshot file. Only proceed if you created/control this "
|
||||
"file, then re-run with --i-trust-this-file.\n"
|
||||
f" file: {args.snapshot}"
|
||||
)
|
||||
|
||||
print(
|
||||
f"WARNING: unpickling {args.snapshot} — this executes code embedded in the "
|
||||
"file. Only safe for snapshots you produced yourself.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
# The outer pickle wraps a dict; the inner pickle contains the actual organism
|
||||
# objects, which must be importable under their original dotted path. If you
|
||||
# ran a custom driver, make sure its module is on sys.path before calling this.
|
||||
outer = pickle.loads(args.snapshot.read_bytes()) # noqa: S301 — gated by --i-trust-this-file
|
||||
if not isinstance(outer, dict) or "population_snapshot" not in outer:
|
||||
sys.exit("not a darwinian-evolver snapshot (no population_snapshot key)")
|
||||
inner = pickle.loads(outer["population_snapshot"]) # noqa: S301 — gated by --i-trust-this-file
|
||||
pairs = inner["organisms"] # list of (Organism, EvaluationResult)
|
||||
|
||||
print(f"# organisms: {len(pairs)}\n")
|
||||
ranked = sorted(pairs, key=lambda p: getattr(p[1], "score", 0) or 0, reverse=True)
|
||||
if args.top:
|
||||
ranked = ranked[: args.top]
|
||||
|
||||
for i, (org, res) in enumerate(ranked):
|
||||
score = getattr(res, "score", float("nan"))
|
||||
print(f"=== rank {i} score={score:.3f} ===")
|
||||
# pick field
|
||||
field = args.field
|
||||
if field is None:
|
||||
for k, v in vars(org).items():
|
||||
if isinstance(v, str) and not k.startswith("_") and k not in {"id",}:
|
||||
field = k
|
||||
break
|
||||
val = getattr(org, field, None) if field else None
|
||||
if val is None:
|
||||
print(f" (no string field; org fields: {list(vars(org).keys())})")
|
||||
else:
|
||||
print(f" {field} ({len(val)} chars):")
|
||||
for ln in val.splitlines()[:30]:
|
||||
print(f" {ln}")
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,240 @@
|
||||
"""
|
||||
Template: a custom darwinian-evolver problem.
|
||||
|
||||
Copy this file, fill in the THREE marked spots (Organism, Evaluator, Mutator),
|
||||
then run it as a driver script. The skeleton handles all the wiring so you only
|
||||
write the domain-specific logic.
|
||||
|
||||
To run:
|
||||
cd ~/.hermes/cache/darwinian-evolver/darwinian_evolver
|
||||
OPENROUTER_API_KEY=... uv run --with openai python /path/to/this_file.py \
|
||||
--num_iterations 3 --num_parents_per_iteration 2 \
|
||||
--output_dir /tmp/my_problem
|
||||
|
||||
The pattern mirrors `scripts/parrot_openrouter.py` (the working reference).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
# Upstream types (AGPL — invoked via subprocess in production; importing here
|
||||
# is fine for skill-side driver scripts the user owns).
|
||||
from darwinian_evolver.cli_common import (
|
||||
build_hyperparameter_config_from_args,
|
||||
parse_learning_log_view_type,
|
||||
register_hyperparameter_args,
|
||||
)
|
||||
from darwinian_evolver.evolve_problem_loop import EvolveProblemLoop
|
||||
from darwinian_evolver.learning_log import LearningLogEntry
|
||||
from darwinian_evolver.problem import (
|
||||
EvaluationFailureCase,
|
||||
EvaluationResult,
|
||||
Evaluator,
|
||||
Mutator,
|
||||
Organism,
|
||||
Problem,
|
||||
)
|
||||
|
||||
DEFAULT_MODEL = os.environ.get("EVOLVER_MODEL", "openai/gpt-4o-mini")
|
||||
|
||||
|
||||
def _client() -> OpenAI:
|
||||
key = os.environ.get("OPENROUTER_API_KEY")
|
||||
if not key:
|
||||
sys.exit("OPENROUTER_API_KEY is not set")
|
||||
return OpenAI(api_key=key, base_url="https://openrouter.ai/api/v1")
|
||||
|
||||
|
||||
def _prompt_llm(prompt: str, max_tokens: int = 1024) -> str:
|
||||
try:
|
||||
r = _client().chat.completions.create(
|
||||
model=DEFAULT_MODEL,
|
||||
max_tokens=max_tokens,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
return r.choices[0].message.content or ""
|
||||
except Exception as e:
|
||||
# Never let one bad LLM response kill the run.
|
||||
return f"<LLM_ERROR: {type(e).__name__}: {e}>"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. ORGANISM — what you are evolving.
|
||||
# ---------------------------------------------------------------------------
|
||||
class MyOrganism(Organism):
|
||||
# TODO: replace with your artifact field. Common shapes:
|
||||
# prompt_template: str
|
||||
# regex_pattern: str
|
||||
# sql_query: str
|
||||
# code_block: str
|
||||
artifact: str
|
||||
|
||||
def run(self, *inputs) -> str:
|
||||
"""Exercise the organism on a test input. Return whatever your
|
||||
evaluator wants to score."""
|
||||
# TODO: implement. For prompt evolution this typically calls _prompt_llm
|
||||
# with the artifact rendered against the input. For regex/SQL it would
|
||||
# call `re.findall(self.artifact, input)` / execute SQL / etc.
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. EVALUATOR — score organisms and surface failures the mutator can learn from.
|
||||
# ---------------------------------------------------------------------------
|
||||
class MyFailureCase(EvaluationFailureCase):
|
||||
# TODO: include enough context for the LLM to diagnose the failure.
|
||||
input: str
|
||||
expected: str
|
||||
actual: str
|
||||
|
||||
|
||||
class MyEvaluator(Evaluator[MyOrganism, EvaluationResult, MyFailureCase]):
|
||||
# Split your dataset. Mutator only sees trainable; holdout detects overfitting.
|
||||
TRAINABLE = [
|
||||
# TODO: list of (input, expected) tuples
|
||||
# ("input1", "expected1"),
|
||||
]
|
||||
HOLDOUT = [
|
||||
# TODO: separate set the mutator never sees
|
||||
]
|
||||
|
||||
def evaluate(self, organism: MyOrganism) -> EvaluationResult:
|
||||
train_fails: list[MyFailureCase] = []
|
||||
hold_fails: list[MyFailureCase] = []
|
||||
for i, (inp, expected) in enumerate(self.TRAINABLE):
|
||||
actual = organism.run(inp)
|
||||
if actual != expected:
|
||||
train_fails.append(MyFailureCase(
|
||||
input=inp, expected=expected, actual=actual,
|
||||
data_point_id=f"trainable_{i}",
|
||||
))
|
||||
for i, (inp, expected) in enumerate(self.HOLDOUT):
|
||||
actual = organism.run(inp)
|
||||
if actual != expected:
|
||||
hold_fails.append(MyFailureCase(
|
||||
input=inp, expected=expected, actual=actual,
|
||||
data_point_id=f"holdout_{i}",
|
||||
))
|
||||
n_total = len(self.TRAINABLE) + len(self.HOLDOUT)
|
||||
n_ok = n_total - len(train_fails) - len(hold_fails)
|
||||
return EvaluationResult(
|
||||
score=n_ok / n_total if n_total else 0.0,
|
||||
trainable_failure_cases=train_fails,
|
||||
holdout_failure_cases=hold_fails,
|
||||
# Always-viable. The evolver only blocks completely-broken organisms;
|
||||
# a 0-score organism is fine and will simply be sampled less often.
|
||||
is_viable=True,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. MUTATOR — LLM proposes an improved organism from a failure case.
|
||||
# ---------------------------------------------------------------------------
|
||||
class MyMutator(Mutator[MyOrganism, MyFailureCase]):
|
||||
PROMPT = """
|
||||
The current artifact is:
|
||||
```
|
||||
{artifact}
|
||||
```
|
||||
|
||||
On this input:
|
||||
```
|
||||
{input}
|
||||
```
|
||||
it produced:
|
||||
```
|
||||
{actual}
|
||||
```
|
||||
but we wanted:
|
||||
```
|
||||
{expected}
|
||||
```
|
||||
|
||||
Diagnose what went wrong, then propose an improved version of the artifact.
|
||||
Put the new version in the LAST triple-backtick block of your response.
|
||||
""".strip()
|
||||
|
||||
def mutate(
|
||||
self,
|
||||
organism: MyOrganism,
|
||||
failure_cases: list[MyFailureCase],
|
||||
learning_log_entries: list[LearningLogEntry],
|
||||
) -> list[MyOrganism]:
|
||||
fc = failure_cases[0]
|
||||
prompt = self.PROMPT.format(
|
||||
artifact=organism.artifact,
|
||||
input=fc.input,
|
||||
actual=fc.actual,
|
||||
expected=fc.expected,
|
||||
)
|
||||
resp = _prompt_llm(prompt)
|
||||
parts = resp.split("```")
|
||||
if len(parts) < 3:
|
||||
return []
|
||||
new_artifact = parts[-2].strip()
|
||||
# Strip an opening language tag like "python\n" or "sql\n"
|
||||
if "\n" in new_artifact:
|
||||
first_line, rest = new_artifact.split("\n", 1)
|
||||
if first_line and not first_line.startswith(" ") and len(first_line) < 20:
|
||||
new_artifact = rest
|
||||
return [MyOrganism(artifact=new_artifact)]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Driver — fills in the EvolveProblemLoop boilerplate. You shouldn't need to
|
||||
# touch anything below this line for a typical run.
|
||||
# ---------------------------------------------------------------------------
|
||||
def make_problem() -> Problem:
|
||||
initial = MyOrganism(artifact="TODO: starting artifact here") # TODO
|
||||
return Problem[MyOrganism, EvaluationResult, MyFailureCase](
|
||||
evaluator=MyEvaluator(),
|
||||
mutators=[MyMutator()],
|
||||
initial_organism=initial,
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
register_hyperparameter_args(ap.add_argument_group("hyperparameters"))
|
||||
ap.add_argument("--num_iterations", type=int, default=3)
|
||||
ap.add_argument("--mutator_concurrency", type=int, default=2)
|
||||
ap.add_argument("--evaluator_concurrency", type=int, default=2)
|
||||
ap.add_argument("--output_dir", type=str, required=True)
|
||||
args = ap.parse_args()
|
||||
|
||||
out = Path(args.output_dir)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
(out / "snapshots").mkdir(exist_ok=True)
|
||||
|
||||
hp = build_hyperparameter_config_from_args(args)
|
||||
loop = EvolveProblemLoop(
|
||||
problem=make_problem(),
|
||||
learning_log_view_type=parse_learning_log_view_type(hp.learning_log_view_type),
|
||||
num_parents_per_iteration=hp.num_parents_per_iteration,
|
||||
mutator_concurrency=args.mutator_concurrency,
|
||||
evaluator_concurrency=args.evaluator_concurrency,
|
||||
fixed_midpoint_score=hp.fixed_midpoint_score,
|
||||
midpoint_score_percentile=hp.midpoint_score_percentile,
|
||||
sharpness=hp.sharpness,
|
||||
novelty_weight=hp.novelty_weight,
|
||||
batch_size=hp.batch_size,
|
||||
should_verify_mutations=hp.verify_mutations,
|
||||
)
|
||||
|
||||
print("Evaluating initial organism...")
|
||||
for snap in loop.run(num_iterations=args.num_iterations):
|
||||
(out / "snapshots" / f"iteration_{snap.iteration}.pkl").write_bytes(snap.snapshot)
|
||||
_, best = snap.best_organism_result
|
||||
print(f"iter={snap.iteration} pop={snap.population_size} best_score={best.score:.3f}")
|
||||
|
||||
print(f"\nDone. Results in: {out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: domain-intel
|
||||
description: Passive domain reconnaissance using Python stdlib. Subdomain discovery, SSL certificate inspection, WHOIS lookups, DNS records, domain availability checks, and bulk multi-domain analysis. No API keys required.
|
||||
platforms: [linux, macos, windows]
|
||||
---
|
||||
|
||||
# Domain Intelligence — Passive OSINT
|
||||
|
||||
Passive domain reconnaissance using only Python stdlib.
|
||||
**Zero dependencies. Zero API keys. Works on Linux, macOS, and Windows.**
|
||||
|
||||
## Helper script
|
||||
|
||||
This skill includes `scripts/domain_intel.py` — a complete CLI tool for all domain intelligence operations.
|
||||
|
||||
```bash
|
||||
# Subdomain discovery via Certificate Transparency logs
|
||||
python3 SKILL_DIR/scripts/domain_intel.py subdomains example.com
|
||||
|
||||
# SSL certificate inspection (expiry, cipher, SANs, issuer)
|
||||
python3 SKILL_DIR/scripts/domain_intel.py ssl example.com
|
||||
|
||||
# WHOIS lookup (registrar, dates, name servers — 100+ TLDs)
|
||||
python3 SKILL_DIR/scripts/domain_intel.py whois example.com
|
||||
|
||||
# DNS records (A, AAAA, MX, NS, TXT, CNAME)
|
||||
python3 SKILL_DIR/scripts/domain_intel.py dns example.com
|
||||
|
||||
# Domain availability check (passive: DNS + WHOIS + SSL signals)
|
||||
python3 SKILL_DIR/scripts/domain_intel.py available coolstartup.io
|
||||
|
||||
# Bulk analysis — multiple domains, multiple checks in parallel
|
||||
python3 SKILL_DIR/scripts/domain_intel.py bulk example.com github.com google.com
|
||||
python3 SKILL_DIR/scripts/domain_intel.py bulk example.com github.com --checks ssl,dns
|
||||
```
|
||||
|
||||
`SKILL_DIR` is the directory containing this SKILL.md file. All output is structured JSON.
|
||||
|
||||
## Available commands
|
||||
|
||||
| Command | What it does | Data source |
|
||||
|---------|-------------|-------------|
|
||||
| `subdomains` | Find subdomains from certificate logs | crt.sh (HTTPS) |
|
||||
| `ssl` | Inspect TLS certificate details | Direct TCP:443 to target |
|
||||
| `whois` | Registration info, registrar, dates | WHOIS servers (TCP:43) |
|
||||
| `dns` | A, AAAA, MX, NS, TXT, CNAME records | System DNS + Google DoH |
|
||||
| `available` | Check if domain is registered | DNS + WHOIS + SSL signals |
|
||||
| `bulk` | Run multiple checks on multiple domains | All of the above |
|
||||
|
||||
## When to use this vs built-in tools
|
||||
|
||||
- **Use this skill** for infrastructure questions: subdomains, SSL certs, WHOIS, DNS records, availability
|
||||
- **Use `web_search`** for general research about what a domain/company does
|
||||
- **Use `web_extract`** to get the actual content of a webpage
|
||||
- **Use `terminal` with `curl -I`** for a simple "is this URL reachable" check
|
||||
|
||||
| Task | Better tool | Why |
|
||||
|------|-------------|-----|
|
||||
| "What does example.com do?" | `web_extract` | Gets page content, not DNS/WHOIS data |
|
||||
| "Find info about a company" | `web_search` | General research, not domain-specific |
|
||||
| "Is this website safe?" | `web_search` | Reputation checks need web context |
|
||||
| "Check if a URL is reachable" | `terminal` with `curl -I` | Simple HTTP check |
|
||||
| "Find subdomains of X" | **This skill** | Only passive source for this |
|
||||
| "When does the SSL cert expire?" | **This skill** | Built-in tools can't inspect TLS |
|
||||
| "Who registered this domain?" | **This skill** | WHOIS data not in web search |
|
||||
| "Is coolstartup.io available?" | **This skill** | Passive availability via DNS+WHOIS+SSL |
|
||||
|
||||
## Platform compatibility
|
||||
|
||||
Pure Python stdlib (`socket`, `ssl`, `urllib`, `json`, `concurrent.futures`).
|
||||
Works identically on Linux, macOS, and Windows with no dependencies.
|
||||
|
||||
- **crt.sh queries** use HTTPS (port 443) — works behind most firewalls
|
||||
- **WHOIS queries** use TCP port 43 — may be blocked on restrictive networks
|
||||
- **DNS queries** use Google DoH (HTTPS) for MX/NS/TXT — firewall-friendly
|
||||
- **SSL checks** connect to the target on port 443 — the only "active" operation
|
||||
|
||||
## Data sources
|
||||
|
||||
All queries are **passive** — no port scanning, no vulnerability testing:
|
||||
|
||||
- **crt.sh** — Certificate Transparency logs (subdomain discovery, HTTPS only)
|
||||
- **WHOIS servers** — Direct TCP to 100+ authoritative TLD registrars
|
||||
- **Google DNS-over-HTTPS** — MX, NS, TXT, CNAME resolution (firewall-friendly)
|
||||
- **System DNS** — A/AAAA record resolution
|
||||
- **SSL check** is the only "active" operation (TCP connection to target:443)
|
||||
|
||||
## Notes
|
||||
|
||||
- WHOIS queries use TCP port 43 — may be blocked on restrictive networks
|
||||
- Some WHOIS servers redact registrant info (GDPR) — mention this to the user
|
||||
- crt.sh can be slow for very popular domains (thousands of certs) — set reasonable expectations
|
||||
- The availability check is heuristic-based (3 passive signals) — not authoritative like a registrar API
|
||||
|
||||
---
|
||||
|
||||
*Contributed by [@FurkanL0](https://github.com/FurkanL0)*
|
||||
@@ -0,0 +1,397 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Domain Intelligence — Passive OSINT via Python stdlib.
|
||||
|
||||
Usage:
|
||||
python domain_intel.py subdomains example.com
|
||||
python domain_intel.py ssl example.com
|
||||
python domain_intel.py whois example.com
|
||||
python domain_intel.py dns example.com
|
||||
python domain_intel.py available example.com
|
||||
python domain_intel.py bulk example.com github.com google.com --checks ssl,dns
|
||||
|
||||
All output is structured JSON. No dependencies beyond Python stdlib.
|
||||
Works on Linux, macOS, and Windows.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import socket
|
||||
import ssl
|
||||
import sys
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
# ─── Subdomain Discovery (crt.sh) ──────────────────────────────────────────
|
||||
|
||||
def subdomains(domain, include_expired=False, limit=200):
|
||||
"""Find subdomains via Certificate Transparency logs."""
|
||||
url = f"https://crt.sh/?q=%25.{urllib.parse.quote(domain)}&output=json"
|
||||
req = urllib.request.Request(url, headers={
|
||||
"User-Agent": "domain-intel-skill/1.0", "Accept": "application/json",
|
||||
})
|
||||
with urllib.request.urlopen(req, timeout=15) as r:
|
||||
entries = json.loads(r.read().decode())
|
||||
|
||||
seen, results = set(), []
|
||||
now = datetime.now(timezone.utc)
|
||||
for e in entries:
|
||||
not_after = e.get("not_after", "")
|
||||
if not include_expired and not_after:
|
||||
try:
|
||||
dt = datetime.strptime(not_after[:19], "%Y-%m-%dT%H:%M:%S").replace(tzinfo=timezone.utc)
|
||||
if dt <= now:
|
||||
continue
|
||||
except ValueError:
|
||||
pass
|
||||
for name in e.get("name_value", "").splitlines():
|
||||
name = name.strip().lower()
|
||||
if name and name not in seen:
|
||||
seen.add(name)
|
||||
results.append({
|
||||
"subdomain": name,
|
||||
"issuer": e.get("issuer_name", ""),
|
||||
"not_after": not_after,
|
||||
})
|
||||
|
||||
results.sort(key=lambda r: (r["subdomain"].startswith("*"), r["subdomain"]))
|
||||
return {"domain": domain, "count": min(len(results), limit), "subdomains": results[:limit]}
|
||||
|
||||
|
||||
# ─── SSL Certificate Inspection ────────────────────────────────────────────
|
||||
|
||||
def check_ssl(host, port=443, timeout=10):
|
||||
"""Inspect the TLS certificate of a host."""
|
||||
def flat(rdns):
|
||||
r = {}
|
||||
for rdn in rdns:
|
||||
for item in rdn:
|
||||
if isinstance(item, (list, tuple)) and len(item) == 2:
|
||||
r[item[0]] = item[1]
|
||||
return r
|
||||
|
||||
def parse_date(s):
|
||||
for fmt in ("%b %d %H:%M:%S %Y %Z", "%b %d %H:%M:%S %Y %Z"):
|
||||
try:
|
||||
return datetime.strptime(s, fmt).replace(tzinfo=timezone.utc)
|
||||
except ValueError:
|
||||
pass
|
||||
return None
|
||||
|
||||
warning = None
|
||||
try:
|
||||
ctx = ssl.create_default_context()
|
||||
with socket.create_connection((host, port), timeout=timeout) as sock:
|
||||
with ctx.wrap_socket(sock, server_hostname=host) as s:
|
||||
cert, cipher, proto = s.getpeercert(), s.cipher(), s.version()
|
||||
except ssl.SSLCertVerificationError as e:
|
||||
warning = str(e)
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
with socket.create_connection((host, port), timeout=timeout) as sock:
|
||||
with ctx.wrap_socket(sock, server_hostname=host) as s:
|
||||
cert, cipher, proto = s.getpeercert(), s.cipher(), s.version()
|
||||
|
||||
not_after = parse_date(cert.get("notAfter", ""))
|
||||
now = datetime.now(timezone.utc)
|
||||
days = (not_after - now).days if not_after else None
|
||||
is_expired = days is not None and days < 0
|
||||
|
||||
if is_expired:
|
||||
status = f"EXPIRED ({abs(days)} days ago)"
|
||||
elif days is not None and days <= 14:
|
||||
status = f"CRITICAL — {days} day(s) left"
|
||||
elif days is not None and days <= 30:
|
||||
status = f"WARNING — {days} day(s) left"
|
||||
else:
|
||||
status = f"OK — {days} day(s) remaining" if days is not None else "unknown"
|
||||
|
||||
return {
|
||||
"host": host, "port": port,
|
||||
"subject": flat(cert.get("subject", [])),
|
||||
"issuer": flat(cert.get("issuer", [])),
|
||||
"subject_alt_names": [f"{t}:{v}" for t, v in cert.get("subjectAltName", [])],
|
||||
"not_before": parse_date(cert.get("notBefore", "")).isoformat() if parse_date(cert.get("notBefore", "")) else "",
|
||||
"not_after": not_after.isoformat() if not_after else "",
|
||||
"days_remaining": days, "is_expired": is_expired, "expiry_status": status,
|
||||
"tls_version": proto,
|
||||
"cipher_suite": cipher[0] if cipher else None,
|
||||
"serial_number": cert.get("serialNumber", ""),
|
||||
"verification_warning": warning,
|
||||
}
|
||||
|
||||
|
||||
# ─── WHOIS Lookup ──────────────────────────────────────────────────────────
|
||||
|
||||
WHOIS_SERVERS = {
|
||||
"com": "whois.verisign-grs.com", "net": "whois.verisign-grs.com",
|
||||
"org": "whois.pir.org", "io": "whois.nic.io", "co": "whois.nic.co",
|
||||
"ai": "whois.nic.ai", "dev": "whois.nic.google", "app": "whois.nic.google",
|
||||
"tech": "whois.nic.tech", "shop": "whois.nic.shop", "store": "whois.nic.store",
|
||||
"online": "whois.nic.online", "site": "whois.nic.site", "cloud": "whois.nic.cloud",
|
||||
"digital": "whois.nic.digital", "media": "whois.nic.media", "blog": "whois.nic.blog",
|
||||
"info": "whois.afilias.net", "biz": "whois.biz", "me": "whois.nic.me",
|
||||
"tv": "whois.nic.tv", "cc": "whois.nic.cc", "ws": "whois.website.ws",
|
||||
"uk": "whois.nic.uk", "co.uk": "whois.nic.uk", "de": "whois.denic.de",
|
||||
"nl": "whois.domain-registry.nl", "fr": "whois.nic.fr", "it": "whois.nic.it",
|
||||
"es": "whois.nic.es", "pl": "whois.dns.pl", "ru": "whois.tcinet.ru",
|
||||
"se": "whois.iis.se", "no": "whois.norid.no", "fi": "whois.fi",
|
||||
"ch": "whois.nic.ch", "at": "whois.nic.at", "be": "whois.dns.be",
|
||||
"cz": "whois.nic.cz", "br": "whois.registro.br", "ca": "whois.cira.ca",
|
||||
"mx": "whois.mx", "au": "whois.auda.org.au", "jp": "whois.jprs.jp",
|
||||
"cn": "whois.cnnic.cn", "in": "whois.inregistry.net", "kr": "whois.kr",
|
||||
"sg": "whois.sgnic.sg", "hk": "whois.hkirc.hk", "tr": "whois.nic.tr",
|
||||
"ae": "whois.aeda.net.ae", "za": "whois.registry.net.za",
|
||||
"space": "whois.nic.space", "zone": "whois.nic.zone", "ninja": "whois.nic.ninja",
|
||||
"guru": "whois.nic.guru", "rocks": "whois.nic.rocks", "live": "whois.nic.live",
|
||||
"game": "whois.nic.game", "games": "whois.nic.games",
|
||||
}
|
||||
|
||||
|
||||
def whois_lookup(domain):
|
||||
"""Query WHOIS servers for domain registration info."""
|
||||
parts = domain.split(".")
|
||||
server = WHOIS_SERVERS.get(".".join(parts[-2:])) or WHOIS_SERVERS.get(parts[-1])
|
||||
if not server:
|
||||
return {"error": f"No WHOIS server for .{parts[-1]}"}
|
||||
|
||||
try:
|
||||
with socket.create_connection((server, 43), timeout=10) as s:
|
||||
s.sendall((domain + "\r\n").encode())
|
||||
chunks = []
|
||||
while True:
|
||||
c = s.recv(4096)
|
||||
if not c:
|
||||
break
|
||||
chunks.append(c)
|
||||
raw = b"".join(chunks).decode("utf-8", errors="replace")
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
patterns = {
|
||||
"registrar": r"(?:Registrar|registrar):\s*(.+)",
|
||||
"creation_date": r"(?:Creation Date|Created|created):\s*(.+)",
|
||||
"expiration_date": r"(?:Registry Expiry Date|Expiration Date|Expiry Date):\s*(.+)",
|
||||
"updated_date": r"(?:Updated Date|Last Modified):\s*(.+)",
|
||||
"name_servers": r"(?:Name Server|nserver):\s*(.+)",
|
||||
"status": r"(?:Domain Status|status):\s*(.+)",
|
||||
"dnssec": r"DNSSEC:\s*(.+)",
|
||||
}
|
||||
result = {"domain": domain, "whois_server": server}
|
||||
for key, pat in patterns.items():
|
||||
matches = re.findall(pat, raw, re.IGNORECASE)
|
||||
if matches:
|
||||
if key in {"name_servers", "status"}:
|
||||
result[key] = list(dict.fromkeys(m.strip().lower() for m in matches))
|
||||
else:
|
||||
result[key] = matches[0].strip()
|
||||
|
||||
for field in ("creation_date", "expiration_date", "updated_date"):
|
||||
if field in result:
|
||||
for fmt in ("%Y-%m-%dT%H:%M:%S", "%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d"):
|
||||
try:
|
||||
dt = datetime.strptime(result[field][:19], fmt).replace(tzinfo=timezone.utc)
|
||||
result[field] = dt.isoformat()
|
||||
if field == "expiration_date":
|
||||
days = (dt - datetime.now(timezone.utc)).days
|
||||
result["expiration_days_remaining"] = days
|
||||
result["is_expired"] = days < 0
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
# ─── DNS Records ───────────────────────────────────────────────────────────
|
||||
|
||||
def dns_records(domain, types=None):
|
||||
"""Resolve DNS records using system DNS + Google DoH."""
|
||||
if not types:
|
||||
types = ["A", "AAAA", "MX", "NS", "TXT", "CNAME"]
|
||||
records = {}
|
||||
|
||||
for qtype in types:
|
||||
if qtype == "A":
|
||||
try:
|
||||
records["A"] = list(dict.fromkeys(
|
||||
i[4][0] for i in socket.getaddrinfo(domain, None, socket.AF_INET)
|
||||
))
|
||||
except Exception:
|
||||
records["A"] = []
|
||||
elif qtype == "AAAA":
|
||||
try:
|
||||
records["AAAA"] = list(dict.fromkeys(
|
||||
i[4][0] for i in socket.getaddrinfo(domain, None, socket.AF_INET6)
|
||||
))
|
||||
except Exception:
|
||||
records["AAAA"] = []
|
||||
else:
|
||||
url = f"https://dns.google/resolve?name={urllib.parse.quote(domain)}&type={qtype}"
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "domain-intel-skill/1.0"})
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
data = json.loads(r.read())
|
||||
records[qtype] = [
|
||||
a.get("data", "").strip().rstrip(".")
|
||||
for a in data.get("Answer", []) if a.get("data")
|
||||
]
|
||||
except Exception:
|
||||
records[qtype] = []
|
||||
|
||||
return {"domain": domain, "records": records}
|
||||
|
||||
|
||||
# ─── Domain Availability Check ─────────────────────────────────────────────
|
||||
|
||||
def check_available(domain):
|
||||
"""Check domain availability using passive signals (DNS + WHOIS + SSL)."""
|
||||
signals = {}
|
||||
|
||||
# DNS
|
||||
try:
|
||||
a = [i[4][0] for i in socket.getaddrinfo(domain, None, socket.AF_INET)]
|
||||
except Exception:
|
||||
a = []
|
||||
|
||||
try:
|
||||
ns_url = f"https://dns.google/resolve?name={urllib.parse.quote(domain)}&type=NS"
|
||||
req = urllib.request.Request(ns_url, headers={"User-Agent": "domain-intel-skill/1.0"})
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
ns = [x.get("data", "") for x in json.loads(r.read()).get("Answer", [])]
|
||||
except Exception:
|
||||
ns = []
|
||||
|
||||
signals["dns_a"] = a
|
||||
signals["dns_ns"] = ns
|
||||
dns_exists = bool(a or ns)
|
||||
|
||||
# SSL
|
||||
ssl_up = False
|
||||
try:
|
||||
ctx = ssl.create_default_context()
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
with socket.create_connection((domain, 443), timeout=3) as s:
|
||||
with ctx.wrap_socket(s, server_hostname=domain):
|
||||
ssl_up = True
|
||||
except Exception:
|
||||
pass
|
||||
signals["ssl_reachable"] = ssl_up
|
||||
|
||||
# WHOIS (quick check)
|
||||
tld = domain.rsplit(".", 1)[-1]
|
||||
server = WHOIS_SERVERS.get(tld)
|
||||
whois_avail = None
|
||||
whois_note = ""
|
||||
if server:
|
||||
try:
|
||||
with socket.create_connection((server, 43), timeout=10) as s:
|
||||
s.sendall((domain + "\r\n").encode())
|
||||
raw = b""
|
||||
while True:
|
||||
c = s.recv(4096)
|
||||
if not c:
|
||||
break
|
||||
raw += c
|
||||
raw = raw.decode("utf-8", errors="replace").lower()
|
||||
if any(p in raw for p in ["no match", "not found", "no data found", "status: free"]):
|
||||
whois_avail = True
|
||||
whois_note = "WHOIS: not found"
|
||||
elif "registrar:" in raw or "creation date:" in raw:
|
||||
whois_avail = False
|
||||
whois_note = "WHOIS: registered"
|
||||
else:
|
||||
whois_note = "WHOIS: inconclusive"
|
||||
except Exception as e:
|
||||
whois_note = f"WHOIS error: {e}"
|
||||
|
||||
signals["whois_available"] = whois_avail
|
||||
signals["whois_note"] = whois_note
|
||||
|
||||
if not dns_exists and whois_avail is True:
|
||||
verdict, conf = "LIKELY AVAILABLE", "high"
|
||||
elif dns_exists or whois_avail is False or ssl_up:
|
||||
verdict, conf = "REGISTERED / IN USE", "high"
|
||||
elif not dns_exists and whois_avail is None:
|
||||
verdict, conf = "POSSIBLY AVAILABLE", "medium"
|
||||
else:
|
||||
verdict, conf = "UNCERTAIN", "low"
|
||||
|
||||
return {"domain": domain, "verdict": verdict, "confidence": conf, "signals": signals}
|
||||
|
||||
|
||||
# ─── Bulk Analysis ─────────────────────────────────────────────────────────
|
||||
|
||||
COMMAND_MAP = {
|
||||
"subdomains": subdomains,
|
||||
"ssl": check_ssl,
|
||||
"whois": whois_lookup,
|
||||
"dns": dns_records,
|
||||
"available": check_available,
|
||||
}
|
||||
|
||||
|
||||
def bulk_check(domains, checks=None, max_workers=5):
|
||||
"""Run multiple checks across multiple domains in parallel."""
|
||||
if not checks:
|
||||
checks = ["ssl", "whois", "dns"]
|
||||
|
||||
def run_one(d):
|
||||
entry = {"domain": d}
|
||||
for check in checks:
|
||||
fn = COMMAND_MAP.get(check)
|
||||
if fn:
|
||||
try:
|
||||
entry[check] = fn(d)
|
||||
except Exception as e:
|
||||
entry[check] = {"error": str(e)}
|
||||
return entry
|
||||
|
||||
results = []
|
||||
with ThreadPoolExecutor(max_workers=min(max_workers, 10)) as ex:
|
||||
futures = {ex.submit(run_one, d): d for d in domains[:20]}
|
||||
for f in as_completed(futures):
|
||||
results.append(f.result())
|
||||
|
||||
return {"total": len(results), "checks": checks, "results": results}
|
||||
|
||||
|
||||
# ─── CLI Entry Point ───────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print(__doc__)
|
||||
sys.exit(1)
|
||||
|
||||
command = sys.argv[1].lower()
|
||||
args = sys.argv[2:]
|
||||
|
||||
if command == "bulk":
|
||||
# Parse --checks flag
|
||||
checks = None
|
||||
domains = []
|
||||
i = 0
|
||||
while i < len(args):
|
||||
if args[i] == "--checks" and i + 1 < len(args):
|
||||
checks = [c.strip() for c in args[i + 1].split(",")]
|
||||
i += 2
|
||||
else:
|
||||
domains.append(args[i])
|
||||
i += 1
|
||||
result = bulk_check(domains, checks)
|
||||
elif command in COMMAND_MAP:
|
||||
result = COMMAND_MAP[command](args[0])
|
||||
else:
|
||||
print(f"Unknown command: {command}")
|
||||
print(f"Available: {', '.join(COMMAND_MAP.keys())}, bulk")
|
||||
sys.exit(1)
|
||||
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
name: drug-discovery
|
||||
description: >
|
||||
Pharmaceutical research assistant for drug discovery workflows. Search
|
||||
bioactive compounds on ChEMBL, calculate drug-likeness (Lipinski Ro5, QED,
|
||||
TPSA, synthetic accessibility), look up drug-drug interactions via
|
||||
OpenFDA, interpret ADMET profiles, and assist with lead optimization.
|
||||
Use for medicinal chemistry questions, molecule property analysis, clinical
|
||||
pharmacology, and open-science drug research.
|
||||
platforms: [linux, macos, windows]
|
||||
version: 1.0.0
|
||||
author: bennytimz
|
||||
license: MIT
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [science, chemistry, pharmacology, research, health]
|
||||
prerequisites:
|
||||
commands: [curl, python3]
|
||||
---
|
||||
|
||||
# Drug Discovery & Pharmaceutical Research
|
||||
|
||||
You are an expert pharmaceutical scientist and medicinal chemist with deep
|
||||
knowledge of drug discovery, cheminformatics, and clinical pharmacology.
|
||||
Use this skill for all pharma/chemistry research tasks.
|
||||
|
||||
## Core Workflows
|
||||
|
||||
### 1 — Bioactive Compound Search (ChEMBL)
|
||||
|
||||
Search ChEMBL (the world's largest open bioactivity database) for compounds
|
||||
by target, activity, or molecule name. No API key required.
|
||||
|
||||
```bash
|
||||
# Search compounds by target name (e.g. "EGFR", "COX-2", "ACE")
|
||||
TARGET="$1"
|
||||
ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$TARGET")
|
||||
curl -s "https://www.ebi.ac.uk/chembl/api/data/target/search?q=${ENCODED}&format=json" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
data=json.load(sys.stdin)
|
||||
targets=data.get('targets',[])[:5]
|
||||
for t in targets:
|
||||
print(f\"ChEMBL ID : {t.get('target_chembl_id')}\")
|
||||
print(f\"Name : {t.get('pref_name')}\")
|
||||
print(f\"Type : {t.get('target_type')}\")
|
||||
print()
|
||||
"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Get bioactivity data for a ChEMBL target ID
|
||||
TARGET_ID="$1" # e.g. CHEMBL203
|
||||
curl -s "https://www.ebi.ac.uk/chembl/api/data/activity?target_chembl_id=${TARGET_ID}&pchembl_value__gte=6&limit=10&format=json" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
data=json.load(sys.stdin)
|
||||
acts=data.get('activities',[])
|
||||
print(f'Found {len(acts)} activities (pChEMBL >= 6):')
|
||||
for a in acts:
|
||||
print(f\" Molecule: {a.get('molecule_chembl_id')} | {a.get('standard_type')}: {a.get('standard_value')} {a.get('standard_units')} | pChEMBL: {a.get('pchembl_value')}\")
|
||||
"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Look up a specific molecule by ChEMBL ID
|
||||
MOL_ID="$1" # e.g. CHEMBL25 (aspirin)
|
||||
curl -s "https://www.ebi.ac.uk/chembl/api/data/molecule/${MOL_ID}?format=json" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
m=json.load(sys.stdin)
|
||||
props=m.get('molecule_properties',{}) or {}
|
||||
print(f\"Name : {m.get('pref_name','N/A')}\")
|
||||
print(f\"SMILES : {m.get('molecule_structures',{}).get('canonical_smiles','N/A') if m.get('molecule_structures') else 'N/A'}\")
|
||||
print(f\"MW : {props.get('full_mwt','N/A')} Da\")
|
||||
print(f\"LogP : {props.get('alogp','N/A')}\")
|
||||
print(f\"HBD : {props.get('hbd','N/A')}\")
|
||||
print(f\"HBA : {props.get('hba','N/A')}\")
|
||||
print(f\"TPSA : {props.get('psa','N/A')} Ų\")
|
||||
print(f\"Ro5 violations: {props.get('num_ro5_violations','N/A')}\")
|
||||
print(f\"QED : {props.get('qed_weighted','N/A')}\")
|
||||
"
|
||||
```
|
||||
|
||||
### 2 — Drug-Likeness Calculation (Lipinski Ro5 + Veber)
|
||||
|
||||
Assess any molecule against established oral bioavailability rules using
|
||||
PubChem's free property API — no RDKit install needed.
|
||||
|
||||
```bash
|
||||
COMPOUND="$1"
|
||||
ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$COMPOUND")
|
||||
curl -s "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/${ENCODED}/property/MolecularWeight,XLogP,HBondDonorCount,HBondAcceptorCount,RotatableBondCount,TPSA,InChIKey/JSON" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
data=json.load(sys.stdin)
|
||||
props=data['PropertyTable']['Properties'][0]
|
||||
mw = float(props.get('MolecularWeight', 0))
|
||||
logp = float(props.get('XLogP', 0))
|
||||
hbd = int(props.get('HBondDonorCount', 0))
|
||||
hba = int(props.get('HBondAcceptorCount', 0))
|
||||
rot = int(props.get('RotatableBondCount', 0))
|
||||
tpsa = float(props.get('TPSA', 0))
|
||||
print('=== Lipinski Rule of Five (Ro5) ===')
|
||||
print(f' MW {mw:.1f} Da {\"✓\" if mw<=500 else \"✗ VIOLATION (>500)\"}')
|
||||
print(f' LogP {logp:.2f} {\"✓\" if logp<=5 else \"✗ VIOLATION (>5)\"}')
|
||||
print(f' HBD {hbd} {\"✓\" if hbd<=5 else \"✗ VIOLATION (>5)\"}')
|
||||
print(f' HBA {hba} {\"✓\" if hba<=10 else \"✗ VIOLATION (>10)\"}')
|
||||
viol = sum([mw>500, logp>5, hbd>5, hba>10])
|
||||
print(f' Violations: {viol}/4 {\"→ Likely orally bioavailable\" if viol<=1 else \"→ Poor oral bioavailability predicted\"}')
|
||||
print()
|
||||
print('=== Veber Oral Bioavailability Rules ===')
|
||||
print(f' TPSA {tpsa:.1f} Ų {\"✓\" if tpsa<=140 else \"✗ VIOLATION (>140)\"}')
|
||||
print(f' Rot. bonds {rot} {\"✓\" if rot<=10 else \"✗ VIOLATION (>10)\"}')
|
||||
print(f' Both rules met: {\"Yes → good oral absorption predicted\" if tpsa<=140 and rot<=10 else \"No → reduced oral absorption\"}')
|
||||
"
|
||||
```
|
||||
|
||||
### 3 — Drug Interaction & Safety Lookup (OpenFDA)
|
||||
|
||||
```bash
|
||||
DRUG="$1"
|
||||
ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$DRUG")
|
||||
curl -s "https://api.fda.gov/drug/label.json?search=drug_interactions:\"${ENCODED}\"&limit=3" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
data=json.load(sys.stdin)
|
||||
results=data.get('results',[])
|
||||
if not results:
|
||||
print('No interaction data found in FDA labels.')
|
||||
sys.exit()
|
||||
for r in results[:2]:
|
||||
brand=r.get('openfda',{}).get('brand_name',['Unknown'])[0]
|
||||
generic=r.get('openfda',{}).get('generic_name',['Unknown'])[0]
|
||||
interactions=r.get('drug_interactions',['N/A'])[0]
|
||||
print(f'--- {brand} ({generic}) ---')
|
||||
print(interactions[:800])
|
||||
print()
|
||||
"
|
||||
```
|
||||
|
||||
```bash
|
||||
DRUG="$1"
|
||||
ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$DRUG")
|
||||
curl -s "https://api.fda.gov/drug/event.json?search=patient.drug.medicinalproduct:\"${ENCODED}\"&count=patient.reaction.reactionmeddrapt.exact&limit=10" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
data=json.load(sys.stdin)
|
||||
results=data.get('results',[])
|
||||
if not results:
|
||||
print('No adverse event data found.')
|
||||
sys.exit()
|
||||
print(f'Top adverse events reported:')
|
||||
for r in results[:10]:
|
||||
print(f\" {r['count']:>5}x {r['term']}\")
|
||||
"
|
||||
```
|
||||
|
||||
### 4 — PubChem Compound Search
|
||||
|
||||
```bash
|
||||
COMPOUND="$1"
|
||||
ENCODED=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$COMPOUND")
|
||||
CID=$(curl -s "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/${ENCODED}/cids/TXT" | head -1 | tr -d '[:space:]')
|
||||
echo "PubChem CID: $CID"
|
||||
curl -s "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/cid/${CID}/property/IsomericSMILES,InChIKey,IUPACName/JSON" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
p=json.load(sys.stdin)['PropertyTable']['Properties'][0]
|
||||
print(f\"IUPAC Name : {p.get('IUPACName','N/A')}\")
|
||||
print(f\"SMILES : {p.get('IsomericSMILES','N/A')}\")
|
||||
print(f\"InChIKey : {p.get('InChIKey','N/A')}\")
|
||||
"
|
||||
```
|
||||
|
||||
### 5 — Target & Disease Literature (OpenTargets)
|
||||
|
||||
```bash
|
||||
GENE="$1"
|
||||
curl -s -X POST "https://api.platform.opentargets.org/api/v4/graphql" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"query\":\"{ search(queryString: \\\"${GENE}\\\", entityNames: [\\\"target\\\"], page: {index: 0, size: 1}) { hits { id score object { ... on Target { id approvedSymbol approvedName associatedDiseases(page: {index: 0, size: 5}) { count rows { score disease { id name } } } } } } } }\"}" \
|
||||
| python3 -c "
|
||||
import json,sys
|
||||
data=json.load(sys.stdin)
|
||||
hits=data.get('data',{}).get('search',{}).get('hits',[])
|
||||
if not hits:
|
||||
print('Target not found.')
|
||||
sys.exit()
|
||||
obj=hits[0]['object']
|
||||
print(f\"Target: {obj.get('approvedSymbol')} — {obj.get('approvedName')}\")
|
||||
assoc=obj.get('associatedDiseases',{})
|
||||
print(f\"Associated with {assoc.get('count',0)} diseases. Top associations:\")
|
||||
for row in assoc.get('rows',[]):
|
||||
print(f\" Score {row['score']:.3f} | {row['disease']['name']}\")
|
||||
"
|
||||
```
|
||||
|
||||
## Reasoning Guidelines
|
||||
|
||||
When analysing drug-likeness or molecular properties, always:
|
||||
|
||||
1. **State raw values first** — MW, LogP, HBD, HBA, TPSA, RotBonds
|
||||
2. **Apply rule sets** — Ro5 (Lipinski), Veber, Ghose filter where relevant
|
||||
3. **Flag liabilities** — metabolic hotspots, hERG risk, high TPSA for CNS penetration
|
||||
4. **Suggest optimizations** — bioisosteric replacements, prodrug strategies, ring truncation
|
||||
5. **Cite the source API** — ChEMBL, PubChem, OpenFDA, or OpenTargets
|
||||
|
||||
For ADMET questions, reason through Absorption, Distribution, Metabolism, Excretion, Toxicity systematically. See references/ADMET_REFERENCE.md for detailed guidance.
|
||||
|
||||
## Important Notes
|
||||
|
||||
- All APIs are free, public, require no authentication
|
||||
- ChEMBL rate limits: add sleep 1 between batch requests
|
||||
- FDA data reflects reported adverse events, not necessarily causation
|
||||
- Always recommend consulting a licensed pharmacist or physician for clinical decisions
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | API | Endpoint |
|
||||
|------|-----|----------|
|
||||
| Find target | ChEMBL | `/api/data/target/search?q=` |
|
||||
| Get bioactivity | ChEMBL | `/api/data/activity?target_chembl_id=` |
|
||||
| Molecule properties | PubChem | `/rest/pug/compound/name/{name}/property/` |
|
||||
| Drug interactions | OpenFDA | `/drug/label.json?search=drug_interactions:` |
|
||||
| Adverse events | OpenFDA | `/drug/event.json?search=...&count=reaction` |
|
||||
| Gene-disease | OpenTargets | GraphQL POST `/api/v4/graphql` |
|
||||
@@ -0,0 +1,66 @@
|
||||
# ADMET Reference Guide
|
||||
|
||||
Comprehensive reference for Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET) analysis in drug discovery.
|
||||
|
||||
## Drug-Likeness Rule Sets
|
||||
|
||||
### Lipinski's Rule of Five (Ro5)
|
||||
|
||||
| Property | Threshold |
|
||||
|----------|-----------|
|
||||
| Molecular Weight (MW) | ≤ 500 Da |
|
||||
| Lipophilicity (LogP) | ≤ 5 |
|
||||
| H-Bond Donors (HBD) | ≤ 5 |
|
||||
| H-Bond Acceptors (HBA) | ≤ 10 |
|
||||
|
||||
Reference: Lipinski et al., Adv. Drug Deliv. Rev. 23, 3–25 (1997).
|
||||
|
||||
### Veber's Oral Bioavailability Rules
|
||||
|
||||
| Property | Threshold |
|
||||
|----------|-----------|
|
||||
| TPSA | ≤ 140 Ų |
|
||||
| Rotatable Bonds | ≤ 10 |
|
||||
|
||||
Reference: Veber et al., J. Med. Chem. 45, 2615–2623 (2002).
|
||||
|
||||
### CNS Penetration (BBB)
|
||||
|
||||
| Property | CNS-Optimal |
|
||||
|----------|-------------|
|
||||
| MW | ≤ 400 Da |
|
||||
| LogP | 1–3 |
|
||||
| TPSA | < 90 Ų |
|
||||
| HBD | ≤ 3 |
|
||||
|
||||
## CYP450 Metabolism
|
||||
|
||||
| Isoform | % Drugs | Notable inhibitors |
|
||||
|---------|---------|-------------------|
|
||||
| CYP3A4 | ~50% | Grapefruit, ketoconazole |
|
||||
| CYP2D6 | ~25% | Fluoxetine, paroxetine |
|
||||
| CYP2C9 | ~15% | Fluconazole, amiodarone |
|
||||
| CYP2C19 | ~10% | Omeprazole, fluoxetine |
|
||||
| CYP1A2 | ~5% | Fluvoxamine, ciprofloxacin |
|
||||
|
||||
## hERG Cardiac Toxicity Risk
|
||||
|
||||
Structural alerts: basic nitrogen (pKa 7–9) + aromatic ring + hydrophobic moiety, LogP > 3.5 + basic amine.
|
||||
|
||||
Mitigation: reduce basicity, introduce polar groups, break planarity.
|
||||
|
||||
## Common Bioisosteric Replacements
|
||||
|
||||
| Original | Bioisostere | Purpose |
|
||||
|----------|-------------|---------|
|
||||
| -COOH | -tetrazole, -SO₂NH₂ | Improve permeability |
|
||||
| -OH (phenol) | -F, -CN | Reduce glucuronidation |
|
||||
| Phenyl | Pyridine, thiophene | Reduce LogP |
|
||||
| Ester | -CONHR | Reduce hydrolysis |
|
||||
|
||||
## Key APIs
|
||||
|
||||
- ChEMBL: https://www.ebi.ac.uk/chembl/api/data/
|
||||
- PubChem: https://pubchem.ncbi.nlm.nih.gov/rest/pug/
|
||||
- OpenFDA: https://api.fda.gov/drug/
|
||||
- OpenTargets GraphQL: https://api.platform.opentargets.org/api/v4/graphql
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
chembl_target.py — Search ChEMBL for a target and retrieve top active compounds.
|
||||
Usage: python3 chembl_target.py "EGFR" --min-pchembl 7 --limit 20
|
||||
No external dependencies.
|
||||
"""
|
||||
import sys, json, time, argparse
|
||||
import urllib.request, urllib.parse
|
||||
|
||||
BASE = "https://www.ebi.ac.uk/chembl/api/data"
|
||||
|
||||
def get(endpoint):
|
||||
try:
|
||||
req = urllib.request.Request(f"{BASE}{endpoint}", headers={"Accept":"application/json"})
|
||||
with urllib.request.urlopen(req, timeout=15) as r:
|
||||
return json.loads(r.read())
|
||||
except Exception as e:
|
||||
print(f"API error: {e}", file=sys.stderr); return None
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="ChEMBL target → active compounds")
|
||||
parser.add_argument("target")
|
||||
parser.add_argument("--min-pchembl", type=float, default=6.0)
|
||||
parser.add_argument("--limit", type=int, default=10)
|
||||
args = parser.parse_args()
|
||||
|
||||
enc = urllib.parse.quote(args.target)
|
||||
data = get(f"/target/search?q={enc}&limit=5&format=json")
|
||||
if not data or not data.get("targets"):
|
||||
print("No targets found."); sys.exit(1)
|
||||
|
||||
t = data["targets"][0]
|
||||
tid = t.get("target_chembl_id","")
|
||||
print(f"\nTarget: {t.get('pref_name')} ({tid})")
|
||||
print(f"Type: {t.get('target_type')} | Organism: {t.get('organism','N/A')}")
|
||||
print(f"\nFetching compounds with pChEMBL ≥ {args.min_pchembl}...\n")
|
||||
|
||||
acts = get(f"/activity?target_chembl_id={tid}&pchembl_value__gte={args.min_pchembl}&assay_type=B&limit={args.limit}&order_by=-pchembl_value&format=json")
|
||||
if not acts or not acts.get("activities"):
|
||||
print("No activities found."); sys.exit(0)
|
||||
|
||||
print(f"{'Molecule':<18} {'pChEMBL':>8} {'Type':<12} {'Value':<10} {'Units'}")
|
||||
print("-"*65)
|
||||
seen = set()
|
||||
for a in acts["activities"]:
|
||||
mid = a.get("molecule_chembl_id","N/A")
|
||||
if mid in seen: continue
|
||||
seen.add(mid)
|
||||
print(f"{mid:<18} {str(a.get('pchembl_value','N/A')):>8} {str(a.get('standard_type','N/A')):<12} {str(a.get('standard_value','N/A')):<10} {a.get('standard_units','N/A')}")
|
||||
time.sleep(0.1)
|
||||
print(f"\nTotal: {len(seen)} unique molecules")
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ro5_screen.py — Batch Lipinski Ro5 + Veber screening via PubChem API.
|
||||
Usage: python3 ro5_screen.py aspirin ibuprofen paracetamol
|
||||
No external dependencies beyond stdlib.
|
||||
"""
|
||||
import sys, json, time
|
||||
import urllib.request, urllib.parse
|
||||
|
||||
BASE = "https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name"
|
||||
PROPS = "MolecularWeight,XLogP,HBondDonorCount,HBondAcceptorCount,RotatableBondCount,TPSA"
|
||||
|
||||
def fetch(name):
|
||||
url = f"{BASE}/{urllib.parse.quote(name)}/property/{PROPS}/JSON"
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=10) as r:
|
||||
return json.loads(r.read())["PropertyTable"]["Properties"][0]
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def check(p):
|
||||
mw,logp,hbd,hba,rot,tpsa = float(p.get("MolecularWeight",0)),float(p.get("XLogP",0)),int(p.get("HBondDonorCount",0)),int(p.get("HBondAcceptorCount",0)),int(p.get("RotatableBondCount",0)),float(p.get("TPSA",0))
|
||||
v = sum([mw>500,logp>5,hbd>5,hba>10])
|
||||
return dict(mw=mw,logp=logp,hbd=hbd,hba=hba,rot=rot,tpsa=tpsa,violations=v,ro5=v<=1,veber=tpsa<=140 and rot<=10,ok=v<=1 and tpsa<=140 and rot<=10)
|
||||
|
||||
def report(name, r):
|
||||
if not r: print(f"✗ {name:30s} — not found"); return
|
||||
s = "✓ PASS" if r["ok"] else "✗ FAIL"
|
||||
flags = (f" [Ro5 violations:{r['violations']}]" if not r["ro5"] else "") + (" [Veber fail]" if not r["veber"] else "")
|
||||
print(f"{s} {name:28s} MW={r['mw']:.0f} LogP={r['logp']:.2f} HBD={r['hbd']} HBA={r['hba']} TPSA={r['tpsa']:.0f} RotB={r['rot']}{flags}")
|
||||
|
||||
def main():
|
||||
compounds = sys.stdin.read().splitlines() if len(sys.argv)<2 or sys.argv[1]=="-" else sys.argv[1:]
|
||||
print(f"\n{'Status':<8} {'Compound':<30} Properties\n" + "-"*85)
|
||||
passed = 0
|
||||
for name in compounds:
|
||||
props = fetch(name.strip())
|
||||
result = check(props) if props else None
|
||||
report(name.strip(), result)
|
||||
if result and result["ok"]: passed += 1
|
||||
time.sleep(0.3)
|
||||
print(f"\nSummary: {passed}/{len(compounds)} passed Ro5 + Veber.\n")
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,238 @@
|
||||
---
|
||||
name: duckduckgo-search
|
||||
description: Free web search via DuckDuckGo — text, news, images, videos. No API key needed. Prefer the `ddgs` CLI when installed; use the Python DDGS library only after verifying that `ddgs` is available in the current runtime.
|
||||
version: 1.3.0
|
||||
author: gamedevCloudy
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [search, duckduckgo, web-search, free, fallback]
|
||||
related_skills: [arxiv]
|
||||
fallback_for_toolsets: [web]
|
||||
---
|
||||
|
||||
# DuckDuckGo Search
|
||||
|
||||
Free web search using DuckDuckGo. **No API key required.**
|
||||
|
||||
Preferred when `web_search` is unavailable or unsuitable (for example when `FIRECRAWL_API_KEY` is not set). Can also be used as a standalone search path when DuckDuckGo results are specifically desired.
|
||||
|
||||
## Detection Flow
|
||||
|
||||
Check what is actually available before choosing an approach:
|
||||
|
||||
```bash
|
||||
# Check CLI availability
|
||||
command -v ddgs >/dev/null && echo "DDGS_CLI=installed" || echo "DDGS_CLI=missing"
|
||||
```
|
||||
|
||||
Decision tree:
|
||||
1. If `ddgs` CLI is installed, prefer `terminal` + `ddgs`
|
||||
2. If `ddgs` CLI is missing, do not assume `execute_code` can import `ddgs`
|
||||
3. If the user wants DuckDuckGo specifically, install `ddgs` first in the relevant environment
|
||||
4. Otherwise fall back to built-in web/browser tools
|
||||
|
||||
Important runtime note:
|
||||
- Terminal and `execute_code` are separate runtimes
|
||||
- A successful shell install does not guarantee `execute_code` can import `ddgs`
|
||||
- Never assume third-party Python packages are preinstalled inside `execute_code`
|
||||
|
||||
## Installation
|
||||
|
||||
Install `ddgs` only when DuckDuckGo search is specifically needed and the runtime does not already provide it.
|
||||
|
||||
```bash
|
||||
# Python package + CLI entrypoint
|
||||
pip install ddgs
|
||||
|
||||
# Verify CLI
|
||||
ddgs --help
|
||||
```
|
||||
|
||||
If a workflow depends on Python imports, verify that same runtime can import `ddgs` before using `from ddgs import DDGS`.
|
||||
|
||||
## Method 1: CLI Search (Preferred)
|
||||
|
||||
Use the `ddgs` command via `terminal` when it exists. This is the preferred path because it avoids assuming the `execute_code` sandbox has the `ddgs` Python package installed.
|
||||
|
||||
```bash
|
||||
# Text search
|
||||
ddgs text -q "python async programming" -m 5
|
||||
|
||||
# News search
|
||||
ddgs news -q "artificial intelligence" -m 5
|
||||
|
||||
# Image search
|
||||
ddgs images -q "landscape photography" -m 10
|
||||
|
||||
# Video search
|
||||
ddgs videos -q "python tutorial" -m 5
|
||||
|
||||
# With region filter
|
||||
ddgs text -q "best restaurants" -m 5 -r us-en
|
||||
|
||||
# Recent results only (d=day, w=week, m=month, y=year)
|
||||
ddgs text -q "latest AI news" -m 5 -t w
|
||||
|
||||
# JSON output for parsing
|
||||
ddgs text -q "fastapi tutorial" -m 5 -o json
|
||||
```
|
||||
|
||||
### CLI Flags
|
||||
|
||||
| Flag | Description | Example |
|
||||
|------|-------------|---------|
|
||||
| `-q` | Query — **required** | `-q "search terms"` |
|
||||
| `-m` | Max results | `-m 5` |
|
||||
| `-r` | Region | `-r us-en` |
|
||||
| `-t` | Time limit | `-t w` (week) |
|
||||
| `-s` | Safe search | `-s off` |
|
||||
| `-o` | Output format | `-o json` |
|
||||
|
||||
## Method 2: Python API (Only After Verification)
|
||||
|
||||
Use the `DDGS` class in `execute_code` or another Python runtime only after verifying that `ddgs` is installed there. Do not assume `execute_code` includes third-party packages by default.
|
||||
|
||||
Safe wording:
|
||||
- "Use `execute_code` with `ddgs` after installing or verifying the package if needed"
|
||||
|
||||
Avoid saying:
|
||||
- "`execute_code` includes `ddgs`"
|
||||
- "DuckDuckGo search works by default in `execute_code`"
|
||||
|
||||
**Important:** `max_results` must always be passed as a **keyword argument** — positional usage raises an error on all methods.
|
||||
|
||||
### Text Search
|
||||
|
||||
Best for: general research, companies, documentation.
|
||||
|
||||
```python
|
||||
from ddgs import DDGS
|
||||
|
||||
with DDGS() as ddgs:
|
||||
for r in ddgs.text("python async programming", max_results=5):
|
||||
print(r["title"])
|
||||
print(r["href"])
|
||||
print(r.get("body", "")[:200])
|
||||
print()
|
||||
```
|
||||
|
||||
Returns: `title`, `href`, `body`
|
||||
|
||||
### News Search
|
||||
|
||||
Best for: current events, breaking news, latest updates.
|
||||
|
||||
```python
|
||||
from ddgs import DDGS
|
||||
|
||||
with DDGS() as ddgs:
|
||||
for r in ddgs.news("AI regulation 2026", max_results=5):
|
||||
print(r["date"], "-", r["title"])
|
||||
print(r.get("source", ""), "|", r["url"])
|
||||
print(r.get("body", "")[:200])
|
||||
print()
|
||||
```
|
||||
|
||||
Returns: `date`, `title`, `body`, `url`, `image`, `source`
|
||||
|
||||
### Image Search
|
||||
|
||||
Best for: visual references, product images, diagrams.
|
||||
|
||||
```python
|
||||
from ddgs import DDGS
|
||||
|
||||
with DDGS() as ddgs:
|
||||
for r in ddgs.images("semiconductor chip", max_results=5):
|
||||
print(r["title"])
|
||||
print(r["image"])
|
||||
print(r.get("thumbnail", ""))
|
||||
print(r.get("source", ""))
|
||||
print()
|
||||
```
|
||||
|
||||
Returns: `title`, `image`, `thumbnail`, `url`, `height`, `width`, `source`
|
||||
|
||||
### Video Search
|
||||
|
||||
Best for: tutorials, demos, explainers.
|
||||
|
||||
```python
|
||||
from ddgs import DDGS
|
||||
|
||||
with DDGS() as ddgs:
|
||||
for r in ddgs.videos("FastAPI tutorial", max_results=5):
|
||||
print(r["title"])
|
||||
print(r.get("content", ""))
|
||||
print(r.get("duration", ""))
|
||||
print(r.get("provider", ""))
|
||||
print(r.get("published", ""))
|
||||
print()
|
||||
```
|
||||
|
||||
Returns: `title`, `content`, `description`, `duration`, `provider`, `published`, `statistics`, `uploader`
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| Method | Use When | Key Fields |
|
||||
|--------|----------|------------|
|
||||
| `text()` | General research, companies | title, href, body |
|
||||
| `news()` | Current events, updates | date, title, source, body, url |
|
||||
| `images()` | Visuals, diagrams | title, image, thumbnail, url |
|
||||
| `videos()` | Tutorials, demos | title, content, duration, provider |
|
||||
|
||||
## Workflow: Search then Extract
|
||||
|
||||
DuckDuckGo returns titles, URLs, and snippets — not full page content. To get full page content, search first and then extract the most relevant URL with `web_extract`, browser tools, or curl.
|
||||
|
||||
CLI example:
|
||||
|
||||
```bash
|
||||
ddgs text -q "fastapi deployment guide" -m 3 -o json
|
||||
```
|
||||
|
||||
Python example, only after verifying `ddgs` is installed in that runtime:
|
||||
|
||||
```python
|
||||
from ddgs import DDGS
|
||||
|
||||
with DDGS() as ddgs:
|
||||
results = list(ddgs.text("fastapi deployment guide", max_results=3))
|
||||
for r in results:
|
||||
print(r["title"], "->", r["href"])
|
||||
```
|
||||
|
||||
Then extract the best URL with `web_extract` or another content-retrieval tool.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Rate limiting**: DuckDuckGo may throttle after many rapid requests. Add a short delay between searches if needed.
|
||||
- **No content extraction**: `ddgs` returns snippets, not full page content. Use `web_extract`, browser tools, or curl for the full article/page.
|
||||
- **Results quality**: Generally good but less configurable than Firecrawl's search.
|
||||
- **Availability**: DuckDuckGo may block requests from some cloud IPs. If searches return empty, try different keywords or wait a few seconds.
|
||||
- **Field variability**: Return fields may vary between results or `ddgs` versions. Use `.get()` for optional fields to avoid `KeyError`.
|
||||
- **Separate runtimes**: A successful `ddgs` install in terminal does not automatically mean `execute_code` can import it.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Likely Cause | What To Do |
|
||||
|---------|--------------|------------|
|
||||
| `ddgs: command not found` | CLI not installed in the shell environment | Install `ddgs`, or use built-in web/browser tools instead |
|
||||
| `ModuleNotFoundError: No module named 'ddgs'` | Python runtime does not have the package installed | Do not use Python DDGS there until that runtime is prepared |
|
||||
| Search returns nothing | Temporary rate limiting or poor query | Wait a few seconds, retry, or adjust the query |
|
||||
| CLI works but `execute_code` import fails | Terminal and `execute_code` are different runtimes | Keep using CLI, or separately prepare the Python runtime |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`max_results` is keyword-only**: `ddgs.text("query", 5)` raises an error. Use `ddgs.text("query", max_results=5)`.
|
||||
- **Do not assume the CLI exists**: Check `command -v ddgs` before using it.
|
||||
- **Do not assume `execute_code` can import `ddgs`**: `from ddgs import DDGS` may fail with `ModuleNotFoundError` unless that runtime was prepared separately.
|
||||
- **Package name**: The package is `ddgs` (previously `duckduckgo-search`). Install with `pip install ddgs`.
|
||||
- **Don't confuse `-q` and `-m`** (CLI): `-q` is for the query, `-m` is for max results count.
|
||||
- **Empty results**: If `ddgs` returns nothing, it may be rate-limited. Wait a few seconds and retry.
|
||||
|
||||
## Validated With
|
||||
|
||||
Validated examples against `ddgs==9.11.2` semantics. Skill guidance now treats CLI availability and Python import availability as separate concerns so the documented workflow matches actual runtime behavior.
|
||||
@@ -0,0 +1,28 @@
|
||||
#!/bin/bash
|
||||
# DuckDuckGo Search Helper Script
|
||||
# Wrapper around ddgs CLI with sensible defaults
|
||||
# Usage: ./duckduckgo.sh <query> [max_results]
|
||||
|
||||
set -e
|
||||
|
||||
QUERY="$1"
|
||||
MAX_RESULTS="${2:-5}"
|
||||
|
||||
if [ -z "$QUERY" ]; then
|
||||
echo "Usage: $0 <query> [max_results]"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 'python async programming' 5"
|
||||
echo " $0 'latest AI news' 10"
|
||||
echo ""
|
||||
echo "Requires: pip install ddgs"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if ddgs is available
|
||||
if ! command -v ddgs &> /dev/null; then
|
||||
echo "Error: ddgs not found. Install with: pip install ddgs"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ddgs text -q "$QUERY" -m "$MAX_RESULTS"
|
||||
@@ -0,0 +1,214 @@
|
||||
---
|
||||
name: gitnexus-explorer
|
||||
description: Index a codebase with GitNexus and serve an interactive knowledge graph via web UI + Cloudflare tunnel.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent + Teknium
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [gitnexus, code-intelligence, knowledge-graph, visualization]
|
||||
related_skills: [native-mcp, codebase-inspection]
|
||||
---
|
||||
|
||||
# GitNexus Explorer
|
||||
|
||||
Index any codebase into a knowledge graph and serve an interactive web UI for exploring
|
||||
symbols, call chains, clusters, and execution flows. Tunneled via Cloudflare for remote access.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User wants to visually explore a codebase's architecture
|
||||
- User asks for a knowledge graph / dependency graph of a repo
|
||||
- User wants to share an interactive codebase explorer with someone
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Node.js** (v18+) — required for GitNexus and the proxy
|
||||
- **git** — repo must have a `.git` directory
|
||||
- **cloudflared** — for tunneling (auto-installed to ~/.local/bin if missing)
|
||||
|
||||
## Size Warning
|
||||
|
||||
The web UI renders all nodes in the browser. Repos under ~5,000 files work well. Large
|
||||
repos (30k+ nodes) will be sluggish or crash the browser tab. The CLI/MCP tools work
|
||||
at any scale — only the web visualization has this limit.
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Clone and Build GitNexus (one-time setup)
|
||||
|
||||
```bash
|
||||
GITNEXUS_DIR="${GITNEXUS_DIR:-$HOME/.local/share/gitnexus}"
|
||||
|
||||
if [ ! -d "$GITNEXUS_DIR/gitnexus-web/dist" ]; then
|
||||
git clone https://github.com/abhigyanpatwari/GitNexus.git "$GITNEXUS_DIR"
|
||||
cd "$GITNEXUS_DIR/gitnexus-shared" && npm install && npm run build
|
||||
cd "$GITNEXUS_DIR/gitnexus-web" && npm install
|
||||
fi
|
||||
```
|
||||
|
||||
### 2. Patch the Web UI for Remote Access
|
||||
|
||||
The web UI defaults to `localhost:4747` for API calls. Patch it to use same-origin
|
||||
so it works through a tunnel/proxy:
|
||||
|
||||
**File: `$GITNEXUS_DIR/gitnexus-web/src/config/ui-constants.ts`**
|
||||
Change:
|
||||
```typescript
|
||||
export const DEFAULT_BACKEND_URL = 'http://localhost:4747';
|
||||
```
|
||||
To:
|
||||
```typescript
|
||||
export const DEFAULT_BACKEND_URL = typeof window !== 'undefined' && window.location.hostname !== 'localhost' ? window.location.origin : 'http://localhost:4747';
|
||||
```
|
||||
|
||||
**File: `$GITNEXUS_DIR/gitnexus-web/vite.config.ts`**
|
||||
Add `allowedHosts: true` inside the `server: { }` block (only needed if running dev
|
||||
mode instead of production build):
|
||||
```typescript
|
||||
server: {
|
||||
allowedHosts: true,
|
||||
// ... existing config
|
||||
},
|
||||
```
|
||||
|
||||
Then build the production bundle:
|
||||
```bash
|
||||
cd "$GITNEXUS_DIR/gitnexus-web" && npx vite build
|
||||
```
|
||||
|
||||
### 3. Index the Target Repo
|
||||
|
||||
```bash
|
||||
cd /path/to/target-repo
|
||||
npx gitnexus analyze --skip-agents-md
|
||||
rm -rf .claude/ # remove Claude Code-specific artifacts
|
||||
```
|
||||
|
||||
Add `--embeddings` for semantic search (slower — minutes instead of seconds).
|
||||
|
||||
The index lives in `.gitnexus/` inside the repo (auto-gitignored).
|
||||
|
||||
### 4. Create the Proxy Script
|
||||
|
||||
Write this to a file (e.g., `$GITNEXUS_DIR/proxy.mjs`). It serves the production
|
||||
web UI and proxies `/api/*` to the GitNexus backend — same origin, no CORS issues,
|
||||
no sudo, no nginx.
|
||||
|
||||
```javascript
|
||||
import http from 'node:http';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
const API_PORT = parseInt(process.env.API_PORT || '4747');
|
||||
const DIST_DIR = process.argv[2] || './dist';
|
||||
const PORT = parseInt(process.argv[3] || '8888');
|
||||
|
||||
const MIME = {
|
||||
'.html': 'text/html', '.js': 'application/javascript', '.css': 'text/css',
|
||||
'.json': 'application/json', '.png': 'image/png', '.svg': 'image/svg+xml',
|
||||
'.ico': 'image/x-icon', '.woff2': 'font/woff2', '.woff': 'font/woff',
|
||||
'.wasm': 'application/wasm',
|
||||
};
|
||||
|
||||
function proxyToApi(req, res) {
|
||||
const opts = {
|
||||
hostname: '127.0.0.1', port: API_PORT,
|
||||
path: req.url, method: req.method, headers: req.headers,
|
||||
};
|
||||
const proxy = http.request(opts, (upstream) => {
|
||||
res.writeHead(upstream.statusCode, upstream.headers);
|
||||
upstream.pipe(res, { end: true });
|
||||
});
|
||||
proxy.on('error', () => { res.writeHead(502); res.end('Backend unavailable'); });
|
||||
req.pipe(proxy, { end: true });
|
||||
}
|
||||
|
||||
function serveStatic(req, res) {
|
||||
let filePath = path.join(DIST_DIR, req.url === '/' ? 'index.html' : req.url.split('?')[0]);
|
||||
if (!fs.existsSync(filePath)) filePath = path.join(DIST_DIR, 'index.html');
|
||||
const ext = path.extname(filePath);
|
||||
const mime = MIME[ext] || 'application/octet-stream';
|
||||
try {
|
||||
const data = fs.readFileSync(filePath);
|
||||
res.writeHead(200, { 'Content-Type': mime, 'Cache-Control': 'public, max-age=3600' });
|
||||
res.end(data);
|
||||
} catch { res.writeHead(404); res.end('Not found'); }
|
||||
}
|
||||
|
||||
http.createServer((req, res) => {
|
||||
if (req.url.startsWith('/api')) proxyToApi(req, res);
|
||||
else serveStatic(req, res);
|
||||
}).listen(PORT, () => console.log(`GitNexus proxy on http://localhost:${PORT}`));
|
||||
```
|
||||
|
||||
### 5. Start the Services
|
||||
|
||||
```bash
|
||||
# Terminal 1: GitNexus backend API
|
||||
npx gitnexus serve &
|
||||
|
||||
# Terminal 2: Proxy (web UI + API on one port)
|
||||
node "$GITNEXUS_DIR/proxy.mjs" "$GITNEXUS_DIR/gitnexus-web/dist" 8888 &
|
||||
```
|
||||
|
||||
Verify: `curl -s http://localhost:8888/api/repos` should return the indexed repo(s).
|
||||
|
||||
### 6. Tunnel with Cloudflare (optional — for remote access)
|
||||
|
||||
```bash
|
||||
# Install cloudflared if needed (no sudo)
|
||||
if ! command -v cloudflared &>/dev/null; then
|
||||
mkdir -p ~/.local/bin
|
||||
curl -sL https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 \
|
||||
-o ~/.local/bin/cloudflared
|
||||
chmod +x ~/.local/bin/cloudflared
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
fi
|
||||
|
||||
# Start tunnel (--config /dev/null avoids conflicts with existing named tunnels)
|
||||
cloudflared tunnel --config /dev/null --url http://localhost:8888 --no-autoupdate --protocol http2
|
||||
```
|
||||
|
||||
The tunnel URL (e.g., `https://random-words.trycloudflare.com`) is printed to stderr.
|
||||
Share it — anyone with the link can explore the graph.
|
||||
|
||||
### 7. Cleanup
|
||||
|
||||
```bash
|
||||
# Stop services
|
||||
pkill -f "gitnexus serve"
|
||||
pkill -f "proxy.mjs"
|
||||
pkill -f cloudflared
|
||||
|
||||
# Remove index from the target repo
|
||||
cd /path/to/target-repo
|
||||
npx gitnexus clean
|
||||
rm -rf .claude/
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`--config /dev/null` is required for cloudflared** if the user has an existing
|
||||
named tunnel config at `~/.cloudflared/config.yml`. Without it, the catch-all
|
||||
ingress rule in the config returns 404 for all quick tunnel requests.
|
||||
|
||||
- **Production build is mandatory for tunneling.** The Vite dev server blocks
|
||||
non-localhost hosts by default (`allowedHosts`). The production build + Node
|
||||
proxy avoids this entirely.
|
||||
|
||||
- **The web UI does NOT create `.claude/` or `CLAUDE.md`.** Those are created by
|
||||
`npx gitnexus analyze`. Use `--skip-agents-md` to suppress the markdown files,
|
||||
then `rm -rf .claude/` for the rest. These are Claude Code integrations that
|
||||
hermes-agent users don't need.
|
||||
|
||||
- **Browser memory limit.** The web UI loads the entire graph into browser memory.
|
||||
Repos with 5k+ files may be sluggish. 30k+ files will likely crash the tab.
|
||||
|
||||
- **Embeddings are optional.** `--embeddings` enables semantic search but takes
|
||||
minutes on large repos. Skip it for quick exploration; add it if you want
|
||||
natural language queries via the AI chat panel.
|
||||
|
||||
- **Multiple repos.** `gitnexus serve` serves ALL indexed repos. Index several
|
||||
repos, start serve once, and the web UI lets you switch between them.
|
||||
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* GitNexus reverse proxy — serves production web UI + proxies /api/* to backend.
|
||||
* Zero dependencies, Node.js built-ins only.
|
||||
*
|
||||
* Usage: node proxy.mjs <dist-dir> [port]
|
||||
* dist-dir: path to gitnexus-web/dist (production build)
|
||||
* port: listen port (default: 8888)
|
||||
*
|
||||
* Environment:
|
||||
* API_PORT: GitNexus serve backend port (default: 4747)
|
||||
*/
|
||||
import http from 'node:http';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
const API_PORT = parseInt(process.env.API_PORT || '4747');
|
||||
const DIST_DIR = process.argv[2] || './dist';
|
||||
const PORT = parseInt(process.argv[3] || '8888');
|
||||
|
||||
const MIME = {
|
||||
'.html': 'text/html',
|
||||
'.js': 'application/javascript',
|
||||
'.css': 'text/css',
|
||||
'.json': 'application/json',
|
||||
'.png': 'image/png',
|
||||
'.svg': 'image/svg+xml',
|
||||
'.ico': 'image/x-icon',
|
||||
'.woff2': 'font/woff2',
|
||||
'.woff': 'font/woff',
|
||||
'.wasm': 'application/wasm',
|
||||
'.ttf': 'font/ttf',
|
||||
'.map': 'application/json',
|
||||
};
|
||||
|
||||
function proxyToApi(req, res) {
|
||||
const opts = {
|
||||
hostname: '127.0.0.1',
|
||||
port: API_PORT,
|
||||
path: req.url,
|
||||
method: req.method,
|
||||
headers: { ...req.headers, host: `127.0.0.1:${API_PORT}` },
|
||||
};
|
||||
const proxy = http.request(opts, (upstream) => {
|
||||
res.writeHead(upstream.statusCode, upstream.headers);
|
||||
upstream.pipe(res, { end: true });
|
||||
});
|
||||
proxy.on('error', () => {
|
||||
res.writeHead(502, { 'Content-Type': 'text/plain' });
|
||||
res.end('GitNexus backend unavailable — is `npx gitnexus serve` running?');
|
||||
});
|
||||
req.pipe(proxy, { end: true });
|
||||
}
|
||||
|
||||
function serveStatic(req, res) {
|
||||
const urlPath = req.url.split('?')[0];
|
||||
let filePath = path.join(DIST_DIR, urlPath === '/' ? 'index.html' : urlPath);
|
||||
|
||||
// SPA fallback: if file doesn't exist and isn't a static asset, serve index.html
|
||||
if (!fs.existsSync(filePath) && !path.extname(filePath)) {
|
||||
filePath = path.join(DIST_DIR, 'index.html');
|
||||
}
|
||||
|
||||
const ext = path.extname(filePath);
|
||||
const mime = MIME[ext] || 'application/octet-stream';
|
||||
|
||||
try {
|
||||
const data = fs.readFileSync(filePath);
|
||||
res.writeHead(200, {
|
||||
'Content-Type': mime,
|
||||
'Cache-Control': ext === '.html' ? 'no-cache' : 'public, max-age=86400',
|
||||
});
|
||||
res.end(data);
|
||||
} catch {
|
||||
res.writeHead(404, { 'Content-Type': 'text/plain' });
|
||||
res.end('Not found');
|
||||
}
|
||||
}
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.url.startsWith('/api')) {
|
||||
proxyToApi(req, res);
|
||||
} else {
|
||||
serveStatic(req, res);
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(PORT, () => {
|
||||
console.log(`GitNexus proxy listening on http://localhost:${PORT}`);
|
||||
console.log(` Web UI: http://localhost:${PORT}/`);
|
||||
console.log(` API: http://localhost:${PORT}/api/repos`);
|
||||
console.log(` Backend: http://127.0.0.1:${API_PORT}`);
|
||||
});
|
||||
@@ -0,0 +1,277 @@
|
||||
---
|
||||
name: osint-investigation
|
||||
description: Public-records OSINT investigation framework — SEC EDGAR filings, USAspending contracts, Senate lobbying, OFAC sanctions, ICIJ offshore leaks, NYC property records (ACRIS), OpenCorporates registries, CourtListener court records, Wayback Machine archives, Wikipedia + Wikidata, GDELT news monitoring. Entity resolution across sources, cross-link analysis, timing correlation, evidence chains. Python stdlib only.
|
||||
version: 0.1.0
|
||||
platforms: [linux, macos, windows]
|
||||
author: Hermes Agent (adapted from ShinMegamiBoson/OpenPlanter, MIT)
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [osint, investigation, public-records, sec, sanctions, corporate-registry, property, courts, due-diligence, journalism]
|
||||
category: research
|
||||
related_skills: [domain-intel, arxiv]
|
||||
---
|
||||
|
||||
# OSINT Investigation — Public Records Cross-Reference
|
||||
|
||||
Investigative framework for public-records OSINT: government contracts,
|
||||
corporate filings, lobbying, sanctions, offshore leaks, property records,
|
||||
court records, web archives, knowledge bases, and global news. Resolve
|
||||
entities across heterogeneous sources, build cross-links with explicit
|
||||
confidence, run statistical timing tests, and produce structured evidence
|
||||
chains.
|
||||
|
||||
**Python stdlib only.** Zero install. Works on Linux, macOS, Windows. Most
|
||||
sources work with no API key (OpenCorporates has an optional free token
|
||||
that raises rate limits).
|
||||
|
||||
Adapted from the MIT-licensed ShinMegamiBoson/OpenPlanter project; expanded
|
||||
to cover identity / property / litigation / archives / news sources that
|
||||
the original didn't address.
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Use when the user asks for:
|
||||
|
||||
- "follow the money" — government contracts, lobbying → legislation, sanctions
|
||||
- corporate due diligence — who controls company X, where are they
|
||||
incorporated, who serves on their boards, what filings have they made
|
||||
- sanctions screening — is entity X on OFAC SDN, ICIJ offshore leaks
|
||||
- pay-to-play investigation — contractors with offshore ties, lobbying
|
||||
clients winning awards
|
||||
- property ownership — find recorded deeds/mortgages by name or address
|
||||
(NYC; for other counties point users at the relevant recorder)
|
||||
- litigation history — find federal + state court opinions and PACER dockets
|
||||
- multi-source entity resolution where naming varies (LLC suffixes, abbreviations)
|
||||
- evidence-chain construction with explicit confidence levels
|
||||
- "what's been said about X" — international news (GDELT) + Wikipedia
|
||||
narrative + Wayback Machine to recover dead URLs
|
||||
|
||||
Do NOT use this skill for:
|
||||
|
||||
- general web research → `web_search` / `web_extract`
|
||||
- domain/infrastructure OSINT → `domain-intel` skill
|
||||
- academic literature → `arxiv` skill
|
||||
- social-media profile discovery → `sherlock` skill (optional)
|
||||
- US **federal** campaign finance — FEC is intentionally NOT covered here
|
||||
(the API is unreliable for ad-hoc contributor-name queries on the free
|
||||
DEMO_KEY tier). For federal donations, point users at
|
||||
https://www.fec.gov/data/ directly.
|
||||
|
||||
## Workflow
|
||||
|
||||
The agent runs scripts via the `terminal` tool. `SKILL_DIR` is the directory
|
||||
holding this SKILL.md.
|
||||
|
||||
### 1. Identify which sources apply
|
||||
|
||||
Read the data-source wiki entries to plan the investigation:
|
||||
|
||||
```
|
||||
ls SKILL_DIR/references/sources/
|
||||
|
||||
# Federal financial / regulatory
|
||||
cat SKILL_DIR/references/sources/sec-edgar.md # corporate filings
|
||||
cat SKILL_DIR/references/sources/usaspending.md # federal contracts
|
||||
cat SKILL_DIR/references/sources/senate-ld.md # lobbying
|
||||
cat SKILL_DIR/references/sources/ofac-sdn.md # sanctions
|
||||
cat SKILL_DIR/references/sources/icij-offshore.md # offshore leaks
|
||||
|
||||
# Identity / property / litigation / archives / news
|
||||
cat SKILL_DIR/references/sources/nyc-acris.md # NYC property records
|
||||
cat SKILL_DIR/references/sources/opencorporates.md # global corporate registry
|
||||
cat SKILL_DIR/references/sources/courtlistener.md # court records (federal + state)
|
||||
cat SKILL_DIR/references/sources/wayback.md # Wayback Machine archives
|
||||
cat SKILL_DIR/references/sources/wikipedia.md # Wikipedia + Wikidata
|
||||
cat SKILL_DIR/references/sources/gdelt.md # global news monitoring
|
||||
```
|
||||
|
||||
Each entry follows a 9-section template: summary, access, schema, coverage,
|
||||
cross-reference keys, data quality, acquisition, legal, references.
|
||||
|
||||
The **cross-reference potential** section maps join keys between sources — read
|
||||
those first to pick the right pair.
|
||||
|
||||
### 2. Acquire data
|
||||
|
||||
Each source has a stdlib-only fetch script in `SKILL_DIR/scripts/`:
|
||||
|
||||
**Federal financial / regulatory**
|
||||
|
||||
```bash
|
||||
# SEC EDGAR filings (corporate disclosures)
|
||||
python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
|
||||
--types 10-K,10-Q --out data/edgar_filings.csv
|
||||
|
||||
# USAspending federal contracts
|
||||
python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
|
||||
--fy 2024 --out data/contracts.csv
|
||||
|
||||
# Senate LD-1 / LD-2 lobbying disclosures
|
||||
python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
|
||||
--year 2024 --out data/lobbying.csv
|
||||
|
||||
# OFAC SDN sanctions list (full snapshot)
|
||||
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
|
||||
|
||||
# ICIJ Offshore Leaks — downloads ~70 MB bulk CSV on first use,
|
||||
# then searches it locally. Cached for 30 days under
|
||||
# $HERMES_OSINT_CACHE/icij/ (default: ~/.cache/hermes-osint/icij/).
|
||||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
|
||||
--out data/icij.csv
|
||||
```
|
||||
|
||||
**Identity / property / litigation / archives / news**
|
||||
|
||||
```bash
|
||||
# NYC property records (deeds, mortgages, liens) — ACRIS via Socrata
|
||||
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "SMITH, JOHN" \
|
||||
--out data/acris.csv
|
||||
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" \
|
||||
--out data/acris_addr.csv
|
||||
|
||||
# OpenCorporates — 130+ jurisdiction corporate registry
|
||||
# (free token required; set OPENCORPORATES_API_TOKEN or pass --token)
|
||||
python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
|
||||
--jurisdiction us_ny --out data/opencorporates.csv
|
||||
|
||||
# CourtListener — federal + state court opinions, PACER dockets
|
||||
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Smith v. Example Corp" \
|
||||
--type opinions --out data/courts.csv
|
||||
|
||||
# Wayback Machine — historical web captures
|
||||
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
|
||||
--match host --collapse digest --out data/wayback.csv
|
||||
|
||||
# Wikipedia + Wikidata — narrative bio + structured facts
|
||||
# Set HERMES_OSINT_UA=your-app/1.0 (your@email) to identify yourself
|
||||
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" \
|
||||
--out data/wp.csv
|
||||
|
||||
# GDELT — global news in 100+ languages, ~2015→present
|
||||
python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Example Corp"' \
|
||||
--timespan 1y --out data/gdelt.csv
|
||||
```
|
||||
|
||||
All outputs are normalized CSV with a header row. Re-run scripts idempotently.
|
||||
|
||||
When a private individual won't be in a source (e.g. SEC EDGAR for a non-public-
|
||||
company person, USAspending for someone who isn't a federal contractor, Senate
|
||||
LDA for someone who isn't a lobbying client), the script returns 0 rows with a
|
||||
clear warning rather than silently writing an empty CSV. EDGAR specifically
|
||||
flags when the company-name resolver matched an individual Form 3/4/5 filer
|
||||
rather than a corporate registrant.
|
||||
|
||||
Rate-limit notes are in each source's wiki entry. Default fetchers sleep
|
||||
politely between paginated requests. **API keys raise rate limits** for
|
||||
sources that support them (`SEC_USER_AGENT`, `SENATE_LDA_TOKEN`,
|
||||
`OPENCORPORATES_API_TOKEN`, `COURTLISTENER_TOKEN`). All scripts surface
|
||||
429 responses immediately with the upstream's quota message so the user
|
||||
knows to slow down or supply a key.
|
||||
|
||||
### 3. Resolve entities across sources
|
||||
|
||||
Normalize names and find matches between two CSV files:
|
||||
|
||||
```bash
|
||||
# Match lobbying clients (Senate LDA) against contract recipients (USAspending)
|
||||
python3 SKILL_DIR/scripts/entity_resolution.py \
|
||||
--left data/lobbying.csv --left-name-col client_name \
|
||||
--right data/contracts.csv --right-name-col recipient_name \
|
||||
--out data/cross_links.csv
|
||||
```
|
||||
|
||||
Three matching tiers with explicit confidence:
|
||||
|
||||
| Tier | Method | Confidence |
|
||||
|------|--------|------------|
|
||||
| `exact` | Normalized strings equal after suffix/punctuation strip | high |
|
||||
| `fuzzy` | Sorted-token equality (word-bag match) | medium |
|
||||
| `token_overlap` | ≥60% token overlap, ≥2 shared tokens, tokens ≥4 chars | low |
|
||||
|
||||
Output `cross_links.csv` columns: `match_type, confidence, left_name,
|
||||
right_name, left_normalized, right_normalized, left_row, right_row`.
|
||||
|
||||
### 4. Statistical timing correlation (optional)
|
||||
|
||||
Test whether two time series cluster suspiciously close together — e.g.
|
||||
lobbying filings near contract awards — using a permutation test:
|
||||
|
||||
```bash
|
||||
python3 SKILL_DIR/scripts/timing_analysis.py \
|
||||
--donations data/lobbying.csv --donation-date-col filing_date \
|
||||
--donation-amount-col income --donation-donor-col client_name \
|
||||
--donation-recipient-col registrant_name \
|
||||
--contracts data/contracts.csv --contract-date-col award_date \
|
||||
--contract-vendor-col recipient_name \
|
||||
--cross-links data/cross_links.csv \
|
||||
--permutations 1000 \
|
||||
--out data/timing.json
|
||||
```
|
||||
|
||||
The script's column flags are intentionally generic — the original tool was
|
||||
written for donations vs awards, but it works for any (event, payee) time
|
||||
series joined through cross-links. Null hypothesis: event timing is
|
||||
independent of award dates. One-tailed p-value = fraction of permutations
|
||||
with mean nearest-award distance ≤ observed. Minimum 3 events per (payer,
|
||||
vendor) pair to run the test.
|
||||
|
||||
### 5. Build the findings JSON (evidence chain)
|
||||
|
||||
```bash
|
||||
python3 SKILL_DIR/scripts/build_findings.py \
|
||||
--cross-links data/cross_links.csv \
|
||||
--timing data/timing.json \
|
||||
--out data/findings.json
|
||||
```
|
||||
|
||||
Every finding has `id, title, severity, confidence, summary, evidence[], sources[]`.
|
||||
Each evidence item points back to a specific row in a source CSV. The user (or a
|
||||
follow-up agent) can verify every claim against its source.
|
||||
|
||||
## Confidence and evidence discipline
|
||||
|
||||
This is the load-bearing rule of the skill. Tell the user:
|
||||
|
||||
- Every claim must trace to a record. No naked assertions.
|
||||
- Confidence tier travels with the claim. `match_type=fuzzy` is "probable",
|
||||
not "confirmed."
|
||||
- Entity resolution produces candidates, NOT conclusions. A `fuzzy` match
|
||||
between "ACME LLC" and "Acme Holdings Group" is a lead, not a fact.
|
||||
- Statistical significance ≠ wrongdoing. p < 0.05 means the timing pattern
|
||||
is unlikely under the null. It does not establish corruption.
|
||||
- All data sources here are public records. They may still contain
|
||||
inaccuracies, stale info, or redactions (GDPR, sealed records).
|
||||
|
||||
## Adding a new data source
|
||||
|
||||
Use the template:
|
||||
|
||||
```bash
|
||||
cp SKILL_DIR/templates/source-template.md \
|
||||
SKILL_DIR/references/sources/<your-source>.md
|
||||
```
|
||||
|
||||
Fill in all 9 sections. Write a `fetch_<source>.py` script in `scripts/` that
|
||||
uses stdlib only and writes a normalized CSV. Update the source list in the
|
||||
"When to use" section above.
|
||||
|
||||
## Tools and their limits
|
||||
|
||||
- `entity_resolution.py` does NOT use external fuzzy libraries (no rapidfuzz,
|
||||
no jellyfish). Token-bag matching is the upper bound here. If you need
|
||||
Levenshtein, transliteration, or phonetic matching, pip-install separately.
|
||||
- `timing_analysis.py` uses Python's `random` for permutations. For
|
||||
reproducibility, pass `--seed N`.
|
||||
- `fetch_*.py` scripts use `urllib.request` and respect `Retry-After`. Heavy
|
||||
bulk usage may still violate ToS — read each source's legal section first.
|
||||
|
||||
## Legal note
|
||||
|
||||
All Phase-1 sources are public records. Bulk acquisition is permitted under
|
||||
their respective access terms (FOIA, public records law, ICIJ explicit
|
||||
publication, OFAC public data). However:
|
||||
|
||||
- Some sources rate-limit aggressively. Respect their headers.
|
||||
- Some redact registrant info (GDPR on WHOIS, sealed filings).
|
||||
- Cross-referencing public records to identify private individuals can have
|
||||
ethical implications. The skill produces evidence chains, not accusations.
|
||||
@@ -0,0 +1,98 @@
|
||||
# CourtListener — Free Law Project
|
||||
|
||||
## 1. Summary
|
||||
|
||||
CourtListener (Free Law Project) aggregates court opinions, dockets, oral
|
||||
arguments, and judge data. Covers ~10M federal and state court opinions
|
||||
back to colonial America, plus PACER docket data from RECAP submissions.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **REST API v4:** `https://www.courtlistener.com/api/rest/v4/`
|
||||
- **Auth:** Anonymous reads allowed on most endpoints; token raises rate
|
||||
limits and unlocks bulk export
|
||||
- **Rate limit:** ~5,000 req/hour unauthenticated for search; higher with token
|
||||
|
||||
Set `COURTLISTENER_TOKEN` env var. Get a free token at
|
||||
https://www.courtlistener.com/sign-in/ then create an API key.
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_courtlistener.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `case_name` | str | Case name |
|
||||
| `court` | str | Court name |
|
||||
| `court_id` | str | Court ID (e.g. `nysd`, `scotus`, `ca9`) |
|
||||
| `date_filed` | str | YYYY-MM-DD |
|
||||
| `docket_number` | str | Court docket number |
|
||||
| `judge` | str | Judge name(s) |
|
||||
| `citation` | str | Reporter citation(s) |
|
||||
| `result_type` | str | opinions / dockets / oral / people |
|
||||
| `snippet` | str | Search-match snippet (up to 500 chars) |
|
||||
| `absolute_url` | str | Direct CourtListener URL |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- Federal: all circuit and district courts, SCOTUS
|
||||
- State: all 50 state supreme/appellate courts, many trial courts
|
||||
- Opinions: ~10M back to 1600s (colonial), full coverage 1950 → present
|
||||
- Dockets via RECAP: ~3M+ from user-submitted PACER PDFs
|
||||
- Updated continuously
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **OpenCorporates** ↔ `case_name` (corporate litigation)
|
||||
- **SEC EDGAR** ↔ `case_name` (securities class actions)
|
||||
- **OFAC SDN** ↔ `case_name` (sanctions-related civil/criminal cases)
|
||||
|
||||
Join key: party name from `case_name`. Note: `case_name` often abbreviates
|
||||
("Smith v. Jones" rather than full party names) — use the full case URL
|
||||
to get all parties.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Older opinions (pre-1990) often lack docket numbers and judges
|
||||
- State coverage is more uneven than federal
|
||||
- PACER docket coverage depends on RECAP user submissions — not exhaustive
|
||||
- Sealed documents are excluded
|
||||
- Party names in case captions don't always match filing names exactly
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_courtlistener.py`
|
||||
|
||||
```bash
|
||||
# Search opinions for a party / keyword
|
||||
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
|
||||
--out data/cl.csv
|
||||
|
||||
# PACER dockets (best for recent litigation)
|
||||
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
|
||||
--type dockets --out data/cl_dockets.csv
|
||||
|
||||
# Restrict to a court
|
||||
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Microsoft" \
|
||||
--court ca9 --out data/cl_9th.csv
|
||||
|
||||
# Date range
|
||||
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
|
||||
--date-from 2020-01-01 --date-to 2024-12-31 --out data/cl.csv
|
||||
```
|
||||
|
||||
Pass `--token` or set `COURTLISTENER_TOKEN`.
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Court opinions are public domain
|
||||
- Free Law Project provides the data under CC0 / public domain dedication
|
||||
- No commercial use restrictions on opinion text or metadata
|
||||
- Some PACER PDFs have copyright on layout (not text) — fair use applies
|
||||
|
||||
## 9. References
|
||||
|
||||
- API docs: https://www.courtlistener.com/help/api/rest/
|
||||
- Court IDs: https://www.courtlistener.com/api/jurisdictions/
|
||||
- RECAP archive: https://www.courtlistener.com/recap/
|
||||
- Bulk data: https://www.courtlistener.com/help/api/bulk-data/
|
||||
@@ -0,0 +1,104 @@
|
||||
# GDELT — Global News Monitoring
|
||||
|
||||
## 1. Summary
|
||||
|
||||
GDELT (Global Database of Events, Language, and Tone) monitors world news
|
||||
in 100+ languages with full-text indexing. Updated every 15 minutes.
|
||||
~2015 → present, ~1B+ articles indexed. Free anonymous access.
|
||||
|
||||
GDELT is wider than Google News (more international, more long-tail
|
||||
sources) and indexed by tone/sentiment, themes (CAMEO codes), people, and
|
||||
organizations.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **DOC 2.0 API:** `https://api.gdeltproject.org/api/v2/doc/doc`
|
||||
- **Events / GKG 2.0:** `https://api.gdeltproject.org/api/v2/events/events`
|
||||
- **Auth:** None
|
||||
- **Rate limit:** **1 request per 5 seconds** for the DOC API — strict
|
||||
|
||||
The fetch script automatically retries after a 6-second sleep when a
|
||||
429 is received.
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_gdelt.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `title` | str | Article title |
|
||||
| `url` | str | Article URL |
|
||||
| `seen_date` | str | When GDELT first saw the article (UTC) |
|
||||
| `domain` | str | Publisher domain |
|
||||
| `language` | str | Source language |
|
||||
| `source_country` | str | 2-letter country code |
|
||||
| `tone` | str | GDELT-computed tone score (negative = negative coverage) |
|
||||
| `social_image` | str | Open Graph image URL when available |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- Worldwide news in 100+ languages
|
||||
- ~2015 → present (Events back to 1979 via a separate stream)
|
||||
- Update frequency: 15 minutes
|
||||
- Bias: heavily Anglophone in volume but very wide source list overall
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **All sources** ↔ `title` / `url` (news context for any subject)
|
||||
- **Wikipedia** ↔ event timeline for notable entities
|
||||
- **Wayback Machine** ↔ recover articles whose URLs have died
|
||||
- **OFAC SDN** ↔ news context for sanctions designations
|
||||
- **SEC EDGAR** ↔ news context for 8-K material events
|
||||
|
||||
Join key: entity name appearing in article title or full-text. GDELT also
|
||||
extracts named entities into a separate stream (GKG) not exposed by this
|
||||
fetcher — query GDELT directly for entity-level filtering.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Title extraction is automated and can be wrong (sometimes captures the
|
||||
site name + delimiter + article title; sometimes a generic page title)
|
||||
- Sentiment / tone is computed by GDELT, not source-supplied
|
||||
- Some domains are oversampled (newswires, aggregators)
|
||||
- Source country is inferred from domain registration / TLD — can be
|
||||
wrong for international news sites with country-neutral domains
|
||||
- Article URLs can rot — pair with Wayback Machine to preserve content
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_gdelt.py`
|
||||
|
||||
```bash
|
||||
# Recent news mentioning an entity
|
||||
python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Nous Research" \
|
||||
--timespan 6m --out data/gdelt.csv
|
||||
|
||||
# Phrase-exact (use double quotes inside single quotes for the shell)
|
||||
python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Dillon Rolnick"' \
|
||||
--timespan 1y --out data/gdelt.csv
|
||||
|
||||
# Filter to a country / language
|
||||
python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Microsoft" \
|
||||
--source-country US --source-lang English --out data/gdelt.csv
|
||||
|
||||
# Date range
|
||||
python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Microsoft" \
|
||||
--start 2024-01-01 --end 2024-12-31 --out data/gdelt.csv
|
||||
```
|
||||
|
||||
GDELT supports its own query operators: phrase quoting, AND/OR/NOT,
|
||||
`sourcecountry:US`, `theme:ECON_BANKRUPTCY`, `tone<-5`, etc.
|
||||
See https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/ for syntax.
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- GDELT data is provided free for academic and journalistic use
|
||||
- Article URLs link out to original publishers — copyright remains with
|
||||
the publisher
|
||||
- GDELT is NOT a content archive; it's a metadata index
|
||||
|
||||
## 9. References
|
||||
|
||||
- DOC 2.0 API: https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/
|
||||
- Themes & query syntax: https://blog.gdeltproject.org/gkg-2-0-our-global-knowledge-graph-2-0-amazing-data-at-your-fingertips/
|
||||
- Project home: https://www.gdeltproject.org/
|
||||
@@ -0,0 +1,104 @@
|
||||
# ICIJ Offshore Leaks Database
|
||||
|
||||
## 1. Summary
|
||||
|
||||
The International Consortium of Investigative Journalists (ICIJ) publishes a
|
||||
combined database of offshore entities from the Panama Papers, Paradise Papers,
|
||||
Pandora Papers, Bahamas Leaks, and Offshore Leaks. ~800,000+ offshore entities
|
||||
with their officers, intermediaries, and addresses.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **Bulk download (primary):** `https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip` (~70 MB ZIP, refreshed periodically)
|
||||
- **Search UI (human):** `https://offshoreleaks.icij.org/`
|
||||
- **Auth:** None
|
||||
- **Note:** The previous Open Refine reconciliation endpoint at
|
||||
`/reconcile` now returns 404. ICIJ has removed it. The bulk ZIP is the
|
||||
remaining stable access path. The skill's `fetch_icij_offshore.py` caches
|
||||
the ZIP locally (default `~/.cache/hermes-osint/icij/`, refreshes after
|
||||
30 days) and searches it offline.
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_icij_offshore.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `node_id` | int | ICIJ canonical node ID |
|
||||
| `name` | str | Entity / officer / intermediary name |
|
||||
| `node_type` | str | entity / officer / intermediary / address |
|
||||
| `country_codes` | str | Semicolon-separated ISO codes |
|
||||
| `countries` | str | Country names |
|
||||
| `jurisdiction` | str | Offshore jurisdiction (BVI, Panama, etc.) |
|
||||
| `incorporation_date` | str | YYYY-MM-DD |
|
||||
| `inactivation_date` | str | YYYY-MM-DD (if struck) |
|
||||
| `source` | str | Panama Papers / Paradise Papers / Pandora Papers / etc. |
|
||||
| `entity_url` | str | Link to ICIJ page |
|
||||
| `connections` | str | Semicolon-separated node IDs of related entities |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- Worldwide offshore entity records
|
||||
- Earliest records: 1970s (Bahamas Leaks). Most data 1990–2018.
|
||||
- NOT updated in real-time — new leaks added when ICIJ publishes them
|
||||
- ~810,000 offshore entities + ~750,000 officers + ~150,000 intermediaries
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **SEC EDGAR** ↔ `name` (public companies with offshore arms)
|
||||
- **USAspending** ↔ `name` (federal contractors with offshore structure)
|
||||
- **OFAC SDN** ↔ `name` (sanctioned entities using offshore vehicles)
|
||||
|
||||
Join key: normalized entity/officer name. `node_id` is canonical for cross-
|
||||
referencing within ICIJ. Connections graph traversal is in-script (BFS over
|
||||
`connections`).
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Offshore entity names sometimes appear in multiple leaks with slight variations
|
||||
- Officers may be nominees (front persons), not beneficial owners
|
||||
- Some entries have minimal info (just a name + jurisdiction)
|
||||
- The connections graph is incomplete — some relationships are documented in
|
||||
source materials but not in the structured database
|
||||
- Inactive/struck-off entities are still included with `inactivation_date`
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_icij_offshore.py`
|
||||
|
||||
```bash
|
||||
# Search by entity name (case-insensitive substring across the bulk DB)
|
||||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
|
||||
--out data/icij.csv
|
||||
|
||||
# Search by officer (individual person)
|
||||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH JOHN" \
|
||||
--out data/icij.csv
|
||||
|
||||
# Search by jurisdiction (filter on cached results)
|
||||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH" \
|
||||
--jurisdiction "BRITISH VIRGIN ISLANDS" --out data/icij_bvi.csv
|
||||
|
||||
# Force a fresh download (default refresh window is 30 days)
|
||||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
|
||||
--force-refresh --out data/icij.csv
|
||||
```
|
||||
|
||||
First call downloads the ~70 MB ZIP under `~/.cache/hermes-osint/icij/`
|
||||
(or `$HERMES_OSINT_CACHE/icij/`). Subsequent calls reuse the cache for 30 days.
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Public record as published by ICIJ under explicit publication
|
||||
- No copyright on the underlying facts (entity names, jurisdictions)
|
||||
- ICIJ asks for attribution if used in derivative reporting
|
||||
- **Ethical note**: Presence in this database does NOT imply wrongdoing. Many
|
||||
offshore structures are legal. The database is a research tool, not a list of
|
||||
criminals.
|
||||
|
||||
## 9. References
|
||||
|
||||
- Database: https://offshoreleaks.icij.org/
|
||||
- About the data: https://offshoreleaks.icij.org/pages/about
|
||||
- Methodology: https://www.icij.org/investigations/panama-papers/
|
||||
- API hints: Open Refine reconciliation endpoint at `https://offshoreleaks.icij.org/reconcile`
|
||||
@@ -0,0 +1,90 @@
|
||||
# NYC ACRIS — NYC Real Property Records
|
||||
|
||||
## 1. Summary
|
||||
|
||||
The Automated City Register Information System (ACRIS) is NYC's index of
|
||||
recorded property documents: deeds, mortgages, satisfactions, liens, UCC
|
||||
filings. Covers Manhattan, Bronx, Brooklyn, Queens, Staten Island.
|
||||
Published as 4 linked Socrata datasets on the NYC Open Data portal.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **Socrata API:** `https://data.cityofnewyork.us/resource/636b-3b5g.json` (Parties)
|
||||
- **Other datasets:** `bnx9-e6tj` (Master), `8h5j-fqxa` (Legal), `uqqa-hym2` (References)
|
||||
- **Auth:** None for read access (Socrata `$app_token` raises rate limits if needed)
|
||||
- **Rate limit:** Generous (~1000 req/hour unauthenticated)
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_nyc_acris.py` (Parties joined to Master):
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `document_id` | str | ACRIS document ID |
|
||||
| `name` | str | Party name as recorded (often "LAST, FIRST" but varies) |
|
||||
| `party_type` | str | 1=grantor, 2=grantee, 3=other |
|
||||
| `party_role` | str | Human-readable role label |
|
||||
| `address_1` | str | Property or party address line 1 |
|
||||
| `city`, `state`, `zip`, `country` | str | Address parts |
|
||||
| `doc_type` | str | DEED, MTGE (mortgage), SAT (satisfaction), AGMT, etc. |
|
||||
| `doc_date`, `recorded_date` | str | YYYY-MM-DD |
|
||||
| `borough` | str | Manhattan / Bronx / Brooklyn / Queens / Staten Island |
|
||||
| `amount` | str | Document amount (USD, when applicable) |
|
||||
| `filing_url` | str | Direct ACRIS DocumentImageView link |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- NYC 5 boroughs only — other counties have their own recorders
|
||||
- 1966 → present (older filings exist on microfilm at the County Clerk)
|
||||
- Updated nightly
|
||||
- ~70M+ party records cumulative
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **SEC EDGAR** ↔ `name` (insider filers with NYC property)
|
||||
- **USAspending** ↔ `name` (federal contractors with NYC property)
|
||||
- **Senate LDA** ↔ `name` (lobbyists / clients with NYC property)
|
||||
- **ICIJ Offshore** ↔ `name` (NYC properties owned via offshore vehicles)
|
||||
|
||||
Join key: normalized party name. NYC property records typically store names
|
||||
as "LAST, FIRST" or full LLC names — use `entity_resolution.py`.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Same person appears with multiple name formats over time
|
||||
- LLC and trust ownership obscures beneficial owners
|
||||
- Recording lag can be 2-4 weeks after closing
|
||||
- Older documents have spottier address data
|
||||
- Sealed records (e.g. domestic violence shelters) are excluded by law
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_nyc_acris.py`
|
||||
|
||||
```bash
|
||||
# By party name
|
||||
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "ROLNICK" --out data/acris.csv
|
||||
|
||||
# By address (useful when you know the property but not the names)
|
||||
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" --out data/acris.csv
|
||||
|
||||
# Restrict to grantees (buyers / mortgagees)
|
||||
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "ROLNICK" --party-type 2 \
|
||||
--out data/acris_buyers.csv
|
||||
```
|
||||
|
||||
The script joins Parties → Master to populate doc_type, dates, borough, and
|
||||
amount. Pass `--no-enrich` to skip the join (faster, fewer columns).
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Public record under NYS Real Property Law and NYC Charter
|
||||
- No commercial use restrictions on the data
|
||||
- All ACRIS data is public information by statute
|
||||
|
||||
## 9. References
|
||||
|
||||
- ACRIS portal: https://a836-acris.nyc.gov/CP/
|
||||
- NYC Open Data: https://data.cityofnewyork.us/
|
||||
- Parties dataset: https://data.cityofnewyork.us/City-Government/ACRIS-Real-Property-Parties/636b-3b5g
|
||||
- Document type codes: https://www1.nyc.gov/site/finance/taxes/acris.page
|
||||
@@ -0,0 +1,92 @@
|
||||
# OFAC SDN — Specially Designated Nationals List
|
||||
|
||||
## 1. Summary
|
||||
|
||||
The Office of Foreign Assets Control (OFAC) publishes the Specially Designated
|
||||
Nationals and Blocked Persons List (SDN). US persons are generally prohibited
|
||||
from dealing with individuals and entities on this list. Also published:
|
||||
non-SDN consolidated lists (BIS Denied Persons, FSE, etc.).
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **Full XML:** `https://www.treasury.gov/ofac/downloads/sdn.xml`
|
||||
- **Delimited:** `https://www.treasury.gov/ofac/downloads/sdn.csv`
|
||||
- **Consolidated:** `https://www.treasury.gov/ofac/downloads/consolidated/consolidated.xml`
|
||||
- **Auth:** None
|
||||
- **Rate limit:** None (static file downloads). Updated continuously.
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_ofac_sdn.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `entity_id` | int | OFAC unique ID |
|
||||
| `name` | str | Primary name |
|
||||
| `entity_type` | str | individual / entity / vessel / aircraft |
|
||||
| `program_list` | str | Semicolon-separated sanctions programs (e.g. SDGT;IRAN) |
|
||||
| `title` | str | For individuals: title/role |
|
||||
| `nationalities` | str | Semicolon-separated country codes |
|
||||
| `aka_list` | str | Semicolon-separated "also known as" names |
|
||||
| `addresses` | str | Semicolon-separated known addresses |
|
||||
| `dob` | str | Date of birth (individuals) |
|
||||
| `pob` | str | Place of birth (individuals) |
|
||||
| `remarks` | str | OFAC's free-text remarks |
|
||||
| `last_updated` | str | YYYY-MM-DD (publication date) |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- Worldwide — all entities sanctioned by US Treasury
|
||||
- ~10,000 entries on SDN, ~15,000 on consolidated lists
|
||||
- Updated continuously (sometimes daily during active enforcement)
|
||||
- Includes AKAs (very common, can be 10+ per entity)
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **SEC EDGAR** ↔ `name` (public companies sanctioned)
|
||||
- **USAspending** ↔ `name` (sanctioned entity as federal contractor — should
|
||||
be impossible but verify)
|
||||
- **ICIJ Offshore** ↔ `name` (offshore entities also sanctioned)
|
||||
|
||||
Join key: normalized name. **CRITICAL**: must match against `aka_list` too.
|
||||
Many sanctioned entities are caught only via aliases.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Names are transliterated from many scripts — multiple romanizations possible
|
||||
- AKAs often differ wildly from primary name
|
||||
- Some entries have minimal info (no DOB, no address) for individuals
|
||||
- Free-text `remarks` contain critical context — read them
|
||||
- "Specially Designated Global Terrorists" (SDGT) and "Cyber-related" (CYBER2)
|
||||
programs add and remove entries frequently
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_ofac_sdn.py`
|
||||
|
||||
```bash
|
||||
# Full snapshot
|
||||
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
|
||||
|
||||
# Filter to specific program
|
||||
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --program SDGT --out data/sdn_sdgt.csv
|
||||
|
||||
# Entities only (skip individuals, vessels, aircraft)
|
||||
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --entity-type entity --out data/sdn_entities.csv
|
||||
```
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Public record under Executive Order authority and statutory sanctions programs
|
||||
- US persons MUST screen against this list — it is enforced
|
||||
- No restrictions on the data itself; restrictions are on transactions with
|
||||
the listed entities
|
||||
- ZERO penalty for "over-matching" — false positives must be cleared but are not
|
||||
prohibited
|
||||
|
||||
## 9. References
|
||||
|
||||
- OFAC home: https://ofac.treasury.gov/
|
||||
- SDN list: https://ofac.treasury.gov/specially-designated-nationals-and-blocked-persons-list-sdn-human-readable-lists
|
||||
- Data formats: https://ofac.treasury.gov/sdn-list/sanctions-list-search-tool
|
||||
- Compliance guidance: https://ofac.treasury.gov/recent-actions
|
||||
@@ -0,0 +1,103 @@
|
||||
# OpenCorporates — Global Corporate Registry
|
||||
|
||||
## 1. Summary
|
||||
|
||||
OpenCorporates aggregates corporate registry data from 130+ jurisdictions
|
||||
worldwide (~200M companies). Covers US state-level filings (NY DOS, Delaware
|
||||
DOC, California SOS, etc.), UK Companies House, EU registries, and most
|
||||
common-law jurisdictions.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **REST API:** `https://api.opencorporates.com/v0.4/`
|
||||
- **HTML fallback:** `https://opencorporates.com/companies?q=...`
|
||||
- **Auth:** API token required (free tier 500 calls/month, paid plans available)
|
||||
- **Rate limit:** Token-bound; un-tokened requests return 401
|
||||
|
||||
Set `OPENCORPORATES_API_TOKEN` env var. Get a free token at
|
||||
https://opencorporates.com/api_accounts/new.
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_opencorporates.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `name` | str | Company legal name |
|
||||
| `company_number` | str | Registry-assigned number |
|
||||
| `jurisdiction_code` | str | e.g. `us_ny`, `us_de`, `gb` |
|
||||
| `jurisdiction_name` | str | Human-readable jurisdiction |
|
||||
| `incorporation_date` | str | YYYY-MM-DD |
|
||||
| `dissolution_date` | str | YYYY-MM-DD (empty if active) |
|
||||
| `company_type` | str | Domestic LLC / Foreign Corp / etc. |
|
||||
| `status` | str | Active / Inactive / Dissolved |
|
||||
| `registered_address` | str | Registered office address |
|
||||
| `opencorporates_url` | str | Link to OpenCorporates entity page |
|
||||
| `officers_count` | str | Total officers on record |
|
||||
| `source` | str | `api`, `html`, or `html-fallback` |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- US: all 50 states + DC at state level (LLCs, corps, LPs)
|
||||
- International: UK, EU, Canada, Australia, NZ, many APAC + LATAM jurisdictions
|
||||
- ~200M company records cumulative
|
||||
- Update frequency varies by jurisdiction (UK CH is near-realtime; some
|
||||
state registries lag months)
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **NYC ACRIS** ↔ `name` (LLC/corp owners of NYC property)
|
||||
- **USAspending** ↔ `name` (corporate federal contractors)
|
||||
- **SEC EDGAR** ↔ `name` (public companies + their subsidiaries)
|
||||
- **ICIJ Offshore** ↔ `name` (international corporate structures)
|
||||
|
||||
Join key: normalized company name. Some entries have `previous_names` arrays
|
||||
which are not currently exported by the fetch script — query OC directly
|
||||
for that.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Company-name spellings vary across re-incorporations and renames
|
||||
- Officer records are spottier than company records (many jurisdictions
|
||||
don't require officer disclosure)
|
||||
- Beneficial-ownership data is generally NOT here — most jurisdictions
|
||||
don't require it. UK Companies House has PSC (people with significant
|
||||
control) but that's not universal.
|
||||
- Cross-jurisdictional links (parent / subsidiary) are based on registry
|
||||
filings only; corporate trees are often incomplete
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_opencorporates.py`
|
||||
|
||||
```bash
|
||||
# Search globally by name
|
||||
python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
|
||||
--out data/oc.csv
|
||||
|
||||
# Restrict to a jurisdiction
|
||||
python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
|
||||
--jurisdiction us_ny --out data/oc_ny.csv
|
||||
|
||||
# Set token via env or flag
|
||||
OPENCORPORATES_API_TOKEN=xxx python3 SKILL_DIR/scripts/fetch_opencorporates.py \
|
||||
--query "Microsoft" --out data/oc.csv
|
||||
```
|
||||
|
||||
Without a token the script falls back to scraping the HTML search page.
|
||||
The fallback is brittle and only fills in `name`, `jurisdiction_code`,
|
||||
`opencorporates_url` — set the token for serious work.
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- OpenCorporates aggregates public records — the underlying facts are
|
||||
public domain
|
||||
- OpenCorporates own database is licensed CC-BY-SA-4.0; attribution required
|
||||
- API ToS prohibits redistributing the full dataset; per-record reference
|
||||
is fine
|
||||
|
||||
## 9. References
|
||||
|
||||
- API docs: https://api.opencorporates.com/documentation/API-Reference
|
||||
- Jurisdiction codes: https://api.opencorporates.com/v0.4/jurisdictions.json
|
||||
- Schema: https://opencorporates.com/info/our_data
|
||||
@@ -0,0 +1,83 @@
|
||||
# SEC EDGAR — Corporate Filings
|
||||
|
||||
## 1. Summary
|
||||
|
||||
EDGAR (Electronic Data Gathering, Analysis, and Retrieval) is the SEC's system
|
||||
for corporate disclosure filings: 10-K (annual), 10-Q (quarterly), 8-K (current
|
||||
events), DEF 14A (proxy), Form 4 (insider trading), 13F (institutional holdings).
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **API:** `https://data.sec.gov/submissions/CIK<10-digit-padded>.json` (no auth)
|
||||
- **Filing index:** `https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=...`
|
||||
- **Full-text search:** `https://efts.sec.gov/LATEST/search-index?q=...`
|
||||
- **Auth:** None — requires `User-Agent` header with contact info per SEC policy
|
||||
- **Rate limit:** 10 requests/second per IP (enforced)
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_sec_edgar.py` (filings index):
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `cik` | str | Central Index Key (10-digit padded) |
|
||||
| `company_name` | str | Registrant name |
|
||||
| `form_type` | str | 10-K, 10-Q, 8-K, etc. |
|
||||
| `filing_date` | str | YYYY-MM-DD |
|
||||
| `accession_number` | str | Filing accession (e.g. 0000320193-24-000123) |
|
||||
| `primary_document` | str | Filename of main document |
|
||||
| `filing_url` | str | Direct URL to filing index |
|
||||
| `reporting_period` | str | Period of report (where applicable) |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- All public US registrants from 1993 → present
|
||||
- 1993-2000 has spotty coverage of older filings (paper-to-electronic migration)
|
||||
- ~12M filings cumulative
|
||||
- Updated within minutes of filing acceptance
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **USAspending** ↔ `company_name` (public companies as federal contractors)
|
||||
- **Senate LD** ↔ `company_name` (public companies hire lobbyists)
|
||||
- **OFAC SDN** ↔ `company_name` (sanctions screening of public registrants)
|
||||
|
||||
Join key: company name OR CIK if you have it. CIK is canonical and stable.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Subsidiaries often filed under parent CIK — be careful with name matches
|
||||
- Name changes over time (rebrands, acquisitions) — CIK remains constant
|
||||
- 10-K Item 1A Risk Factors are free-form text — useful for `web_extract`-style
|
||||
parsing, not structured queries
|
||||
- Foreign private issuers file 20-F instead of 10-K
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_sec_edgar.py`
|
||||
|
||||
```bash
|
||||
# By CIK
|
||||
python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
|
||||
--types 10-K,10-Q --out data/edgar_filings.csv
|
||||
|
||||
# By company name (resolves to CIK first via name search)
|
||||
python3 SKILL_DIR/scripts/fetch_sec_edgar.py --company "APPLE INC" \
|
||||
--types 8-K --since 2024-01-01 --out data/edgar_filings.csv
|
||||
```
|
||||
|
||||
Set `SEC_USER_AGENT` env var with your contact email (SEC requirement).
|
||||
Example: `SEC_USER_AGENT="Research example@example.com"`.
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Public record under SEC Rule 24b-2 / 17 CFR § 230.401
|
||||
- No commercial use restrictions on filing content
|
||||
- SEC asks all bulk users to include a `User-Agent` with contact info and to
|
||||
respect 10 req/s — failure to do so can result in IP blocking
|
||||
|
||||
## 9. References
|
||||
|
||||
- Developer docs: https://www.sec.gov/edgar/sec-api-documentation
|
||||
- EDGAR full-text search: https://efts.sec.gov/LATEST/search-index
|
||||
- Fair access policy: https://www.sec.gov/os/accessing-edgar-data
|
||||
@@ -0,0 +1,89 @@
|
||||
# Senate LD — Lobbying Disclosure (LD-1 / LD-2)
|
||||
|
||||
## 1. Summary
|
||||
|
||||
The Senate Office of Public Records publishes lobbying disclosures under the
|
||||
Lobbying Disclosure Act of 1995 (LDA, as amended by HLOGA 2007). LD-1 is
|
||||
registration of a new client-lobbyist relationship; LD-2 is the quarterly
|
||||
activity report.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **API:** `https://lda.senate.gov/api/v1/` (no auth required for read-only)
|
||||
- **Bulk download:** `https://lda.senate.gov/api/v1/filings/?format=csv` (paginated)
|
||||
- **Auth:** Token required for >120 req/hour — register at https://lda.senate.gov/api/auth/register/
|
||||
- **Rate limit:** 120 req/hour unauthenticated, 1,200 req/hour authenticated
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_senate_ld.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `filing_uuid` | str | Unique filing ID |
|
||||
| `filing_type` | str | LD-1, LD-2, LD-203, etc. |
|
||||
| `filing_year` | int | Year |
|
||||
| `filing_period` | str | Q1/Q2/Q3/Q4 or annual |
|
||||
| `registrant_name` | str | Lobbying firm or organization |
|
||||
| `registrant_id` | str | Senate-assigned registrant ID |
|
||||
| `client_name` | str | Client being represented |
|
||||
| `client_id` | str | Senate-assigned client ID |
|
||||
| `client_general_description` | str | Client industry / business |
|
||||
| `income` | float | LD-2 income from client this quarter (USD) |
|
||||
| `expenses` | float | LD-2 expenses (in-house lobbying) |
|
||||
| `lobbyists` | str | Semicolon-separated lobbyist names |
|
||||
| `issues` | str | Semicolon-separated issue areas |
|
||||
| `government_entities` | str | Agencies/chambers contacted |
|
||||
| `filing_date` | str | YYYY-MM-DD |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- US federal lobbying only (state lobbying handled by individual state ethics offices)
|
||||
- 1999 → present (full electronic coverage from 2008)
|
||||
- Quarterly reporting cycle (LD-2)
|
||||
- ~1M+ filings cumulative
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **USAspending** ↔ `client_name` (clients lobbying for contracts)
|
||||
- **SEC EDGAR** ↔ `client_name` (public companies as lobbying clients)
|
||||
- **OFAC SDN** ↔ `client_name` (sanctions screening of lobbying clients)
|
||||
|
||||
Join key: normalized client_name. registrant_id and client_id are canonical
|
||||
when joining Senate-internal records.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Many lobbyist names appear in multiple registrants over time (job changes)
|
||||
- `issues` and `government_entities` are free-text — Inconsistent capitalization
|
||||
- Foreign agents register under FARA (Department of Justice), NOT here
|
||||
- Income/expenses are reported in $10,000 brackets in some older filings
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_senate_ld.py`
|
||||
|
||||
```bash
|
||||
# By client
|
||||
python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
|
||||
--year 2024 --out data/lobbying.csv
|
||||
|
||||
# By registrant (lobbying firm)
|
||||
python3 SKILL_DIR/scripts/fetch_senate_ld.py --registrant "BIG K STREET LLP" \
|
||||
--year 2024 --out data/lobbying.csv
|
||||
```
|
||||
|
||||
Set `SENATE_LDA_TOKEN` env var if you have one (or pass `--token`).
|
||||
Defaults to anonymous (120 req/hour).
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Public record under 2 U.S.C. § 1604 (LDA)
|
||||
- No commercial use restrictions
|
||||
- Reuse is unconditional — see Senate Public Records Office disclaimer
|
||||
|
||||
## 9. References
|
||||
|
||||
- API docs: https://lda.senate.gov/api/redoc/v1/
|
||||
- LDA guidance: https://lobbyingdisclosure.house.gov/ld_guidance.pdf
|
||||
- Senate Public Records: https://lda.senate.gov/
|
||||
@@ -0,0 +1,97 @@
|
||||
# USAspending — Federal Government Contracts and Grants
|
||||
|
||||
## 1. Summary
|
||||
|
||||
USAspending.gov is the official source of federal spending data. Coverage:
|
||||
contracts, grants, loans, direct payments, sub-awards. Required by the DATA Act
|
||||
of 2014 — all federal agencies must report to a single schema.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **API v2:** `https://api.usaspending.gov/api/v2/` (no auth, no key)
|
||||
- **Bulk:** `https://files.usaspending.gov/` (CSV / Parquet by award type)
|
||||
- **Auth:** None
|
||||
- **Rate limit:** Not strictly enforced, but be polite — keep to <10 req/s
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_usaspending.py` (prime awards):
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `award_id` | str | Federal award ID (PIID for contracts, FAIN for grants) |
|
||||
| `recipient_name` | str | Awardee legal name |
|
||||
| `recipient_uei` | str | Unique Entity Identifier (replaced DUNS in 2022) |
|
||||
| `recipient_duns` | str | Legacy DUNS number (historical only) |
|
||||
| `recipient_parent_name` | str | Ultimate parent organization |
|
||||
| `recipient_state` | str | Recipient state |
|
||||
| `awarding_agency` | str | Department / agency name |
|
||||
| `awarding_sub_agency` | str | Sub-tier (e.g. DoD → Army) |
|
||||
| `award_type` | str | Contract / Grant / Loan / Direct Payment |
|
||||
| `award_amount` | float | Current total obligation in USD |
|
||||
| `award_date` | str | Action / signed date YYYY-MM-DD |
|
||||
| `period_of_performance_start` | str | YYYY-MM-DD |
|
||||
| `period_of_performance_end` | str | YYYY-MM-DD |
|
||||
| `naics_code` | str | Industry classification |
|
||||
| `psc_code` | str | Product / Service Code |
|
||||
| `competition_extent` | str | Full / limited / sole-source |
|
||||
| `description` | str | Award description (free-text) |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- US federal awards only (state/local not included)
|
||||
- FY 2008 → present (full coverage from FY 2017)
|
||||
- Updated bi-weekly from agency reporting
|
||||
- ~100M+ transaction records cumulative
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **SEC EDGAR** ↔ `recipient_name` (public companies as contractors)
|
||||
- **Senate LD** ↔ `recipient_name` (lobbying clients winning contracts)
|
||||
- **OFAC SDN** ↔ `recipient_name` (sanctions screening of contractors — must be
|
||||
filtered out by SAM.gov but verify)
|
||||
- **ICIJ Offshore** ↔ `recipient_name` (offshore-linked contractors)
|
||||
|
||||
Join key: normalized recipient name. UEI is canonical when present.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- DUNS → UEI transition (April 2022) — old records have DUNS, new records have UEI
|
||||
- Some sub-awards aren't reported (FFATA threshold is $30k)
|
||||
- Award amount changes over time (mod actions) — fetch script reports current total
|
||||
- `competition_extent` field is free-text in older records — `fetch_usaspending.py`
|
||||
normalizes to canonical values
|
||||
- Recipient name variations are extensive — "ACME LLC", "Acme L.L.C.", "ACME, INC"
|
||||
all appear. Use `entity_resolution.py`.
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_usaspending.py`
|
||||
|
||||
```bash
|
||||
# By recipient name
|
||||
python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
|
||||
--fy 2024 --out data/contracts.csv
|
||||
|
||||
# By awarding agency
|
||||
python3 SKILL_DIR/scripts/fetch_usaspending.py --agency "Department of Defense" \
|
||||
--fy 2024 --out data/contracts.csv
|
||||
|
||||
# Filter to sole-source only
|
||||
python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
|
||||
--fy 2024 --sole-source-only --out data/contracts.csv
|
||||
```
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Public record under the Federal Funding Accountability and Transparency Act
|
||||
(FFATA, 2006) and DATA Act (2014)
|
||||
- No commercial use restrictions on the data
|
||||
- Personal information of award recipients (e.g. small business owners' addresses
|
||||
in some grants) should be handled per the source agency's privacy notice
|
||||
|
||||
## 9. References
|
||||
|
||||
- API docs: https://api.usaspending.gov/
|
||||
- Data dictionary: https://www.usaspending.gov/data-dictionary
|
||||
- Award schema: https://files.usaspending.gov/docs/Data_Dictionary_Crosswalk.xlsx
|
||||
@@ -0,0 +1,93 @@
|
||||
# Wayback Machine — Internet Archive CDX
|
||||
|
||||
## 1. Summary
|
||||
|
||||
The Internet Archive's Wayback Machine has captured ~900B+ web pages since
|
||||
1996. The CDX server API indexes those captures by URL, timestamp, and
|
||||
content hash. Free, anonymous, no auth.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **CDX server:** `https://web.archive.org/cdx/search/cdx`
|
||||
- **Wayback URL:** `https://web.archive.org/web/<timestamp>/<url>`
|
||||
- **Save Page Now (write):** `https://web.archive.org/save/<url>` (different API)
|
||||
- **Auth:** None
|
||||
- **Rate limit:** Generous; be polite (~1 req/s)
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_wayback.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `url` | str | Original URL captured |
|
||||
| `timestamp` | str | YYYYMMDDHHMMSS (CDX format) |
|
||||
| `wayback_url` | str | Direct replay URL |
|
||||
| `mimetype` | str | Content-type at capture |
|
||||
| `status` | str | HTTP status (typically 200) |
|
||||
| `digest` | str | SHA1 of capture content (collapse-friendly) |
|
||||
| `length` | str | Byte length of capture |
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- 1996 → present
|
||||
- ~900B+ captures across ~700M domains
|
||||
- Updated continuously by automated crawls + manual saves
|
||||
- Some domains have aggressive coverage (news), others sparse (private)
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **Wikipedia** ↔ Reverse-lookup pages cited as references that have since
|
||||
disappeared
|
||||
- **News URLs** ↔ Original article content when present-day URLs 404
|
||||
- **Corporate websites** ↔ Historical "About" pages, executive bios that
|
||||
have been scrubbed
|
||||
|
||||
The Wayback CDX is most useful as a **content-recovery** layer when other
|
||||
sources point to URLs that no longer exist.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- robots.txt-blocked domains may have spotty or no coverage
|
||||
- Captures vary in completeness (HTML may be saved without CSS/JS)
|
||||
- Some content is excluded by domain owner request (DMCA, etc.)
|
||||
- Coverage of "deep links" (URLs with query strings) is uneven
|
||||
- Time resolution is per-capture, not continuous — gaps are common
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_wayback.py`
|
||||
|
||||
```bash
|
||||
# All captures of a specific URL
|
||||
python3 SKILL_DIR/scripts/fetch_wayback.py --url "https://example.com/page" \
|
||||
--out data/wb.csv
|
||||
|
||||
# All captures of a host
|
||||
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
|
||||
--match host --out data/wb.csv
|
||||
|
||||
# All captures of a domain + subdomains
|
||||
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
|
||||
--match domain --out data/wb.csv
|
||||
|
||||
# Only unique-content captures within a date window
|
||||
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
|
||||
--match host --collapse digest \
|
||||
--from-date 2020-01-01 --to-date 2023-12-31 \
|
||||
--out data/wb.csv
|
||||
```
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Internet Archive captures are made under fair-use research provisions
|
||||
- Replay URLs are stable references — citing them is encouraged
|
||||
- Internet Archive non-profit terms of use govern content
|
||||
- Some content is rights-restricted; replay may be blocked even if the
|
||||
CDX entry shows it as captured
|
||||
|
||||
## 9. References
|
||||
|
||||
- CDX server docs: https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md
|
||||
- Wayback API: https://archive.org/help/wayback_api.php
|
||||
- Internet Archive: https://archive.org/
|
||||
@@ -0,0 +1,107 @@
|
||||
# Wikipedia + Wikidata
|
||||
|
||||
## 1. Summary
|
||||
|
||||
Wikipedia is the canonical narrative-bio source for notable people, places,
|
||||
and organizations. Wikidata is its structured-data counterpart: ~110M
|
||||
items, each with claims, dates, identifiers, and cross-references to
|
||||
external authorities (VIAF, ISNI, ORCID, GRID, etc.).
|
||||
|
||||
Together they're a high-precision entity-resolution layer — the bar for
|
||||
inclusion is real, but anything past that bar is well-cross-referenced.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- **Wikipedia OpenSearch:** `https://en.wikipedia.org/w/api.php?action=opensearch`
|
||||
- **Wikipedia REST summary:** `https://en.wikipedia.org/api/rest_v1/page/summary/<title>`
|
||||
- **Wikidata Action API:** `https://www.wikidata.org/w/api.php?action=wbgetentities`
|
||||
- **Wikidata SPARQL:** `https://query.wikidata.org/sparql` (more powerful but aggressively rate-limited)
|
||||
- **Auth:** None, but **a meaningful User-Agent is required**
|
||||
|
||||
Set `HERMES_OSINT_UA` to something identifying (e.g. `your-app/1.0 (you@example.com)`).
|
||||
Wikimedia returns HTTP 429 to generic UAs.
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields emitted by `fetch_wikipedia.py`:
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `source` | str | `wikipedia` or `wikipedia+wikidata` |
|
||||
| `label` | str | Wikipedia article title |
|
||||
| `description` | str | Short Wikidata description |
|
||||
| `qid` | str | Wikidata QID (e.g. Q2283 for Microsoft) |
|
||||
| `wikipedia_title`, `wikipedia_url` | str | Article identifier + URL |
|
||||
| `wikidata_url` | str | Wikidata entity URL |
|
||||
| `instance_of` | str | What kind of thing it is (P31) |
|
||||
| `country` | str | Country (P17 for orgs/places, P27 for people) |
|
||||
| `occupation` | str | P106 |
|
||||
| `employer` | str | P108 |
|
||||
| `date_of_birth` | str | P569, YYYY-MM-DD |
|
||||
| `place_of_birth` | str | P19 |
|
||||
| `summary` | str | Wikipedia REST extract (~1000 chars) |
|
||||
|
||||
The fetch script uses Wikidata's Action API (NOT SPARQL) for structured
|
||||
facts — far more lenient on rate limits.
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- Wikipedia EN: ~7M articles
|
||||
- Wikidata: ~110M items, ~1.5B statements
|
||||
- Updated continuously; abuse filters and bots run constantly
|
||||
- High notability bar — most private individuals are not in Wikipedia
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
- **All sources** ↔ `label` (entity identity resolution)
|
||||
- **SEC EDGAR** ↔ `label` (public companies)
|
||||
- **CourtListener** ↔ `label` (parties to notable litigation)
|
||||
- **Wikidata external identifiers** (not currently in this fetcher's output)
|
||||
link to VIAF, ISNI, ORCID, GRID, GitHub, Twitter, IMDb, ...
|
||||
|
||||
Join key: Wikidata QID is canonical. Wikipedia titles are stable for
|
||||
most articles but can be renamed.
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
- Notability filter — only notable entities (criteria vary by topic)
|
||||
- Recency lag — current events take days to weeks to be reflected
|
||||
- POV / vandalism — moderated, but edits between sweeps can be bad
|
||||
- Living-persons biographies have stricter sourcing requirements
|
||||
- Wikidata claims have qualifiers and references — the fetch script
|
||||
doesn't currently export them
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_wikipedia.py`
|
||||
|
||||
```bash
|
||||
# Look up a notable entity
|
||||
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --out data/wp.csv
|
||||
|
||||
# A specific person
|
||||
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" --out data/wp_bg.csv
|
||||
|
||||
# Skip the Wikidata enrichment for speed
|
||||
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --no-wikidata \
|
||||
--limit 5 --out data/wp.csv
|
||||
```
|
||||
|
||||
The OpenSearch is fuzzy — `--limit 5` returns the top 5 Wikipedia article
|
||||
matches. Each is enriched with the QID + structured facts unless
|
||||
`--no-wikidata` is passed.
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Wikipedia text: CC-BY-SA-3.0 / GFDL
|
||||
- Wikidata claims: CC0 (public domain)
|
||||
- API ToS: respect rate limits, identify your agent
|
||||
- Commercial use allowed with attribution
|
||||
|
||||
## 9. References
|
||||
|
||||
- Wikipedia OpenSearch: https://www.mediawiki.org/wiki/API:Opensearch
|
||||
- Wikipedia REST: https://en.wikipedia.org/api/rest_v1/
|
||||
- Wikidata Action API: https://www.wikidata.org/wiki/Wikidata:Data_access
|
||||
- Wikidata SPARQL: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service
|
||||
- User-Agent policy: https://meta.wikimedia.org/wiki/User-Agent_policy
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Tiny stdlib HTTP helper used by fetch_*.py scripts.
|
||||
|
||||
Provides polite retry + JSON convenience + User-Agent enforcement.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
DEFAULT_UA = (
|
||||
"hermes-osint-investigation/0.2 "
|
||||
"(+https://github.com/NousResearch/hermes-agent; "
|
||||
"set HERMES_OSINT_UA env var to identify yourself per "
|
||||
"Wikimedia / SEC fair-use guidance)"
|
||||
)
|
||||
|
||||
|
||||
def get(
|
||||
url: str,
|
||||
*,
|
||||
params: dict | None = None,
|
||||
headers: dict | None = None,
|
||||
user_agent: str | None = None,
|
||||
max_retries: int = 3,
|
||||
backoff: float = 1.5,
|
||||
timeout: float = 30.0,
|
||||
) -> bytes:
|
||||
"""GET with retry on 5xx and Retry-After honoring.
|
||||
|
||||
429 (rate-limit) is raised IMMEDIATELY with a clear message — retrying
|
||||
when the upstream says "you're over quota" just wastes time. The caller
|
||||
should slow down or supply real credentials.
|
||||
"""
|
||||
if params:
|
||||
sep = "&" if "?" in url else "?"
|
||||
url = f"{url}{sep}{urllib.parse.urlencode(params)}"
|
||||
h = {"User-Agent": user_agent or os.environ.get("HERMES_OSINT_UA", DEFAULT_UA)}
|
||||
if headers:
|
||||
h.update(headers)
|
||||
|
||||
last_err: Exception | None = None
|
||||
for attempt in range(max_retries + 1):
|
||||
req = urllib.request.Request(url, headers=h)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
return resp.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 429:
|
||||
# Surface immediately. Read the body so the caller sees the
|
||||
# provider's actual message ("OVER_RATE_LIMIT" etc.).
|
||||
try:
|
||||
body = e.read(2048).decode("utf-8", errors="replace")
|
||||
except Exception: # noqa: BLE001
|
||||
body = ""
|
||||
raise RuntimeError(
|
||||
f"HTTP 429 rate-limited by {urllib.parse.urlsplit(url).netloc}. "
|
||||
f"Slow down or supply a real API key. Body: {body[:300]}"
|
||||
) from e
|
||||
if e.code in {500, 502, 503, 504} and attempt < max_retries:
|
||||
retry_after = e.headers.get("Retry-After") if e.headers else None
|
||||
wait = float(retry_after) if (retry_after and retry_after.isdigit()) else backoff ** (attempt + 1)
|
||||
time.sleep(wait)
|
||||
last_err = e
|
||||
continue
|
||||
raise
|
||||
except urllib.error.URLError as e:
|
||||
if attempt < max_retries:
|
||||
time.sleep(backoff ** (attempt + 1))
|
||||
last_err = e
|
||||
continue
|
||||
raise
|
||||
if last_err:
|
||||
raise last_err
|
||||
raise RuntimeError("unreachable")
|
||||
|
||||
|
||||
def get_json(url: str, **kwargs) -> dict | list:
|
||||
return json.loads(get(url, **kwargs).decode("utf-8"))
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Shared entity-name normalization helpers (stdlib-only).
|
||||
|
||||
Used by entity_resolution.py and timing_analysis.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Legal suffixes / corporate boilerplate to strip during normalization.
|
||||
_SUFFIX_TOKENS = {
|
||||
"INC", "INCORPORATED", "LLC", "LLP", "LP", "LTD", "LIMITED",
|
||||
"CORP", "CORPORATION", "CO", "COMPANY",
|
||||
"GROUP", "GRP", "HOLDINGS", "HOLDING",
|
||||
"PARTNERS", "ASSOCIATES",
|
||||
"INTERNATIONAL", "INTL",
|
||||
"ENTERPRISES", "ENTERPRISE",
|
||||
"SERVICES", "SERVICE", "SVCS",
|
||||
"SOLUTIONS", "MANAGEMENT", "MGMT", "CONSULTING",
|
||||
"TECHNOLOGY", "TECHNOLOGIES", "TECH",
|
||||
"INDUSTRIES", "INDUSTRY",
|
||||
"AMERICA", "AMERICAN",
|
||||
"USA", "US",
|
||||
"PLLC", "PC",
|
||||
"TRUST", "FOUNDATION",
|
||||
}
|
||||
|
||||
_PUNCT_RE = re.compile(r"[^\w\s]")
|
||||
_WS_RE = re.compile(r"\s+")
|
||||
|
||||
|
||||
def normalize_name(name: str | None) -> str:
|
||||
"""Standard normalization: uppercase, strip suffixes, drop punctuation."""
|
||||
if not name:
|
||||
return ""
|
||||
s = _PUNCT_RE.sub(" ", name.upper())
|
||||
s = _WS_RE.sub(" ", s).strip()
|
||||
tokens = [t for t in s.split() if t and t not in _SUFFIX_TOKENS]
|
||||
return " ".join(tokens)
|
||||
|
||||
|
||||
def normalize_aggressive(name: str | None) -> str:
|
||||
"""Aggressive normalization: sorted unique tokens (word-bag)."""
|
||||
base = normalize_name(name)
|
||||
if not base:
|
||||
return ""
|
||||
return " ".join(sorted(set(base.split())))
|
||||
|
||||
|
||||
def name_tokens(name: str | None, min_len: int = 4) -> set[str]:
|
||||
"""Token set used for overlap matching."""
|
||||
base = normalize_name(name)
|
||||
if not base:
|
||||
return set()
|
||||
return {t for t in base.split() if len(t) >= min_len}
|
||||
|
||||
|
||||
def token_overlap_ratio(left: str | None, right: str | None) -> tuple[float, int]:
|
||||
"""Return (jaccard-like ratio, shared token count) over min-len tokens."""
|
||||
a = name_tokens(left)
|
||||
b = name_tokens(right)
|
||||
if not a or not b:
|
||||
return 0.0, 0
|
||||
shared = a & b
|
||||
if not shared:
|
||||
return 0.0, 0
|
||||
union = a | b
|
||||
return len(shared) / len(union), len(shared)
|
||||
@@ -0,0 +1,221 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a structured findings.json with evidence chains (stdlib-only).
|
||||
|
||||
Aggregates cross_links.csv (entity_resolution output) and an optional
|
||||
timing.json (timing_analysis output) into a single evidence-chain document.
|
||||
|
||||
Output structure:
|
||||
{
|
||||
"metadata": {...},
|
||||
"findings": [
|
||||
{
|
||||
"id": "F0001",
|
||||
"title": "...",
|
||||
"severity": "HIGH|MEDIUM|LOW",
|
||||
"confidence": "high|medium|low",
|
||||
"summary": "...",
|
||||
"evidence": [
|
||||
{"source": "cross_links.csv", "row": 12, "fields": {...}},
|
||||
...
|
||||
],
|
||||
"sources": ["cross_links.csv", "timing.json"]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Every finding traces to specific source rows. No naked claims.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
CONFIDENCE_ORDER = {"high": 0, "medium": 1, "low": 2}
|
||||
SEVERITY_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
|
||||
|
||||
|
||||
def _read_cross_links(path: str) -> list[dict[str, str]]:
|
||||
with open(path, newline="", encoding="utf-8") as fh:
|
||||
return list(csv.DictReader(fh))
|
||||
|
||||
|
||||
def build_findings(
|
||||
cross_links_path: str,
|
||||
timing_path: str | None = None,
|
||||
out_path: str = "findings.json",
|
||||
bundled_threshold: int = 3,
|
||||
) -> dict:
|
||||
findings: list[dict] = []
|
||||
next_id = 1
|
||||
|
||||
# 1. Match-based findings, grouped by (left_normalized, right_normalized).
|
||||
matches = _read_cross_links(cross_links_path)
|
||||
grouped: dict[tuple[str, str], list[dict[str, str]]] = defaultdict(list)
|
||||
for i, row in enumerate(matches):
|
||||
row["__row__"] = str(i)
|
||||
grouped[(row.get("left_normalized", ""), row.get("right_normalized", ""))].append(row)
|
||||
|
||||
for (left_norm, right_norm), rows in grouped.items():
|
||||
if not left_norm or not right_norm:
|
||||
continue
|
||||
# Use the highest-confidence match for the finding's overall confidence.
|
||||
best = min(rows, key=lambda r: CONFIDENCE_ORDER.get(r.get("confidence", "low"), 2))
|
||||
finding_id = f"F{next_id:04d}"
|
||||
next_id += 1
|
||||
evidence = [
|
||||
{
|
||||
"source": "cross_links.csv",
|
||||
"row": int(r["__row__"]),
|
||||
"fields": {
|
||||
"match_type": r.get("match_type", ""),
|
||||
"confidence": r.get("confidence", ""),
|
||||
"left_name": r.get("left_name", ""),
|
||||
"right_name": r.get("right_name", ""),
|
||||
"overlap_ratio": r.get("overlap_ratio", ""),
|
||||
"shared_tokens": r.get("shared_tokens", ""),
|
||||
},
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
findings.append(
|
||||
{
|
||||
"id": finding_id,
|
||||
"title": f"Entity match: {best.get('left_name', '')} ↔ {best.get('right_name', '')}",
|
||||
"severity": "MEDIUM" if best.get("confidence") == "high" else "LOW",
|
||||
"confidence": best.get("confidence", "low"),
|
||||
"summary": (
|
||||
f"{len(rows)} cross-link record(s) tie "
|
||||
f"'{best.get('left_name', '')}' to "
|
||||
f"'{best.get('right_name', '')}' "
|
||||
f"(best tier: {best.get('match_type', '')})."
|
||||
),
|
||||
"evidence": evidence,
|
||||
"sources": ["cross_links.csv"],
|
||||
}
|
||||
)
|
||||
|
||||
# 2. Bundled-donations findings (if cross_links carries donor↔candidate pattern).
|
||||
# Heuristic: many distinct left names sharing the same right name.
|
||||
by_right: dict[str, set[str]] = defaultdict(set)
|
||||
by_right_rows: dict[str, list[dict[str, str]]] = defaultdict(list)
|
||||
for r in matches:
|
||||
right = r.get("right_normalized", "")
|
||||
left_raw = r.get("left_name", "").strip()
|
||||
if right and left_raw:
|
||||
by_right[right].add(left_raw)
|
||||
by_right_rows[right].append(r)
|
||||
for right_norm, lefts in by_right.items():
|
||||
if len(lefts) < bundled_threshold:
|
||||
continue
|
||||
rows = by_right_rows[right_norm]
|
||||
right_raw = rows[0].get("right_name", "")
|
||||
findings.append(
|
||||
{
|
||||
"id": f"F{next_id:04d}",
|
||||
"title": f"Bundled cross-links: {len(lefts)} distinct left entities ↔ '{right_raw}'",
|
||||
"severity": "HIGH",
|
||||
"confidence": "medium",
|
||||
"summary": (
|
||||
f"{len(lefts)} distinct left-side entities link to "
|
||||
f"'{right_raw}'. Pattern suggests coordinated relationship "
|
||||
f"(e.g. bundled donations, multi-vendor employer)."
|
||||
),
|
||||
"evidence": [
|
||||
{
|
||||
"source": "cross_links.csv",
|
||||
"row": int(r.get("__row__", "0")),
|
||||
"fields": {
|
||||
"left_name": r.get("left_name", ""),
|
||||
"match_type": r.get("match_type", ""),
|
||||
},
|
||||
}
|
||||
for r in rows
|
||||
],
|
||||
"sources": ["cross_links.csv"],
|
||||
}
|
||||
)
|
||||
next_id += 1
|
||||
|
||||
# 3. Timing-based findings.
|
||||
if timing_path and Path(timing_path).exists():
|
||||
timing = json.loads(Path(timing_path).read_text())
|
||||
for r in timing.get("results", []):
|
||||
if not r.get("significant"):
|
||||
continue
|
||||
findings.append(
|
||||
{
|
||||
"id": f"F{next_id:04d}",
|
||||
"title": (
|
||||
f"Donation timing significantly clusters near awards: "
|
||||
f"{r['donor']} ↔ {r['recipient']}"
|
||||
),
|
||||
"severity": "HIGH" if r["p_value"] < 0.01 else "MEDIUM",
|
||||
"confidence": "medium",
|
||||
"summary": (
|
||||
f"Mean nearest-award distance {r['observed_mean_days']} days "
|
||||
f"(null {r['null_mean_days']} days). p={r['p_value']}, "
|
||||
f"effect size {r['effect_size_sd']} SD. "
|
||||
f"{r['n_donations']} donations, {r['n_award_dates']} awards."
|
||||
),
|
||||
"evidence": [
|
||||
{
|
||||
"source": "timing.json",
|
||||
"row": None,
|
||||
"fields": r,
|
||||
}
|
||||
],
|
||||
"sources": ["timing.json"],
|
||||
}
|
||||
)
|
||||
next_id += 1
|
||||
|
||||
# Sort: severity → confidence → id.
|
||||
findings.sort(
|
||||
key=lambda f: (
|
||||
SEVERITY_ORDER.get(f["severity"], 3),
|
||||
CONFIDENCE_ORDER.get(f["confidence"], 3),
|
||||
f["id"],
|
||||
)
|
||||
)
|
||||
|
||||
payload = {
|
||||
"metadata": {
|
||||
"n_findings": len(findings),
|
||||
"cross_links_path": cross_links_path,
|
||||
"timing_path": timing_path,
|
||||
"bundled_threshold": bundled_threshold,
|
||||
},
|
||||
"findings": findings,
|
||||
}
|
||||
Path(out_path).write_text(json.dumps(payload, indent=2))
|
||||
return payload
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--cross-links", required=True)
|
||||
p.add_argument("--timing", help="Optional timing.json from timing_analysis.py")
|
||||
p.add_argument("--out", default="findings.json")
|
||||
p.add_argument(
|
||||
"--bundled-threshold",
|
||||
type=int,
|
||||
default=3,
|
||||
help="Minimum distinct left entities to flag as bundled (default 3)",
|
||||
)
|
||||
a = p.parse_args()
|
||||
|
||||
payload = build_findings(
|
||||
cross_links_path=a.cross_links,
|
||||
timing_path=a.timing,
|
||||
out_path=a.out,
|
||||
bundled_threshold=a.bundled_threshold,
|
||||
)
|
||||
print(f"Wrote {payload['metadata']['n_findings']} findings to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,228 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cross-source entity resolution (stdlib-only).
|
||||
|
||||
Given two CSV files with name columns, find candidate matches using three
|
||||
tiers of normalization:
|
||||
|
||||
1. exact — normalized strings equal
|
||||
2. fuzzy — sorted-token (word-bag) match
|
||||
3. token_overlap — >=60% Jaccard overlap on >=4-char tokens, >=2 shared
|
||||
|
||||
Adapted from ShinMegamiBoson/OpenPlanter (MIT) but generalized: no Boston-
|
||||
specific record types, no contribution-code filters, no fixed schemas.
|
||||
|
||||
Output CSV columns:
|
||||
match_type, confidence, left_name, right_name,
|
||||
left_normalized, right_normalized, left_row, right_row,
|
||||
overlap_ratio, shared_tokens
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Allow running directly or as a module.
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _normalize import ( # noqa: E402
|
||||
normalize_name,
|
||||
normalize_aggressive,
|
||||
token_overlap_ratio,
|
||||
)
|
||||
|
||||
CONFIDENCE = {
|
||||
"exact": "high",
|
||||
"fuzzy": "medium",
|
||||
"token_overlap": "low",
|
||||
}
|
||||
|
||||
|
||||
def _read_csv(path: str, name_col: str) -> list[dict[str, str]]:
|
||||
rows = []
|
||||
with open(path, newline="", encoding="utf-8") as fh:
|
||||
reader = csv.DictReader(fh)
|
||||
if name_col not in (reader.fieldnames or []):
|
||||
raise SystemExit(
|
||||
f"Column {name_col!r} not in {path}. "
|
||||
f"Available: {reader.fieldnames}"
|
||||
)
|
||||
for i, row in enumerate(reader):
|
||||
row["__row__"] = str(i)
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def _build_index(rows: list[dict[str, str]], name_col: str):
|
||||
"""Index by exact-normalized and aggressive (sorted-token) form."""
|
||||
exact: dict[str, list[dict[str, str]]] = {}
|
||||
aggressive: dict[str, list[dict[str, str]]] = {}
|
||||
for row in rows:
|
||||
raw = row.get(name_col, "")
|
||||
n = normalize_name(raw)
|
||||
if n:
|
||||
exact.setdefault(n, []).append(row)
|
||||
a = normalize_aggressive(raw)
|
||||
if a:
|
||||
aggressive.setdefault(a, []).append(row)
|
||||
return exact, aggressive
|
||||
|
||||
|
||||
def _emit(
|
||||
out_rows: list[dict[str, str]],
|
||||
seen: set[tuple],
|
||||
match_type: str,
|
||||
left_row: dict[str, str],
|
||||
right_row: dict[str, str],
|
||||
left_col: str,
|
||||
right_col: str,
|
||||
ratio: float = 0.0,
|
||||
shared: int = 0,
|
||||
):
|
||||
left_raw = left_row.get(left_col, "")
|
||||
right_raw = right_row.get(right_col, "")
|
||||
key = (
|
||||
left_row["__row__"],
|
||||
right_row["__row__"],
|
||||
match_type,
|
||||
)
|
||||
if key in seen:
|
||||
return
|
||||
seen.add(key)
|
||||
out_rows.append(
|
||||
{
|
||||
"match_type": match_type,
|
||||
"confidence": CONFIDENCE[match_type],
|
||||
"left_name": left_raw,
|
||||
"right_name": right_raw,
|
||||
"left_normalized": normalize_name(left_raw),
|
||||
"right_normalized": normalize_name(right_raw),
|
||||
"left_row": left_row["__row__"],
|
||||
"right_row": right_row["__row__"],
|
||||
"overlap_ratio": f"{ratio:.3f}" if ratio else "",
|
||||
"shared_tokens": str(shared) if shared else "",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def resolve(
|
||||
left_path: str,
|
||||
left_col: str,
|
||||
right_path: str,
|
||||
right_col: str,
|
||||
out_path: str,
|
||||
overlap_threshold: float = 0.60,
|
||||
min_shared: int = 2,
|
||||
skip_overlap: bool = False,
|
||||
) -> int:
|
||||
left_rows = _read_csv(left_path, left_col)
|
||||
right_rows = _read_csv(right_path, right_col)
|
||||
|
||||
right_exact, right_aggressive = _build_index(right_rows, right_col)
|
||||
|
||||
out_rows: list[dict[str, str]] = []
|
||||
seen: set[tuple] = set()
|
||||
|
||||
# Pass 1+2: exact / fuzzy via index lookup.
|
||||
for lrow in left_rows:
|
||||
raw = lrow.get(left_col, "")
|
||||
n = normalize_name(raw)
|
||||
if not n:
|
||||
continue
|
||||
for rrow in right_exact.get(n, []):
|
||||
_emit(out_rows, seen, "exact", lrow, rrow, left_col, right_col)
|
||||
a = normalize_aggressive(raw)
|
||||
if a:
|
||||
for rrow in right_aggressive.get(a, []):
|
||||
_emit(out_rows, seen, "fuzzy", lrow, rrow, left_col, right_col)
|
||||
|
||||
if not skip_overlap:
|
||||
# Pass 3: token overlap (O(N*M) — expensive; allow opt-out).
|
||||
for lrow in left_rows:
|
||||
l_raw = lrow.get(left_col, "")
|
||||
if not normalize_name(l_raw):
|
||||
continue
|
||||
for rrow in right_rows:
|
||||
ratio, shared = token_overlap_ratio(
|
||||
l_raw, rrow.get(right_col, "")
|
||||
)
|
||||
if ratio >= overlap_threshold and shared >= min_shared:
|
||||
_emit(
|
||||
out_rows,
|
||||
seen,
|
||||
"token_overlap",
|
||||
lrow,
|
||||
rrow,
|
||||
left_col,
|
||||
right_col,
|
||||
ratio=ratio,
|
||||
shared=shared,
|
||||
)
|
||||
|
||||
fieldnames = [
|
||||
"match_type",
|
||||
"confidence",
|
||||
"left_name",
|
||||
"right_name",
|
||||
"left_normalized",
|
||||
"right_normalized",
|
||||
"left_row",
|
||||
"right_row",
|
||||
"overlap_ratio",
|
||||
"shared_tokens",
|
||||
]
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
writer = csv.DictWriter(fh, fieldnames=fieldnames)
|
||||
writer.writeheader()
|
||||
writer.writerows(out_rows)
|
||||
return len(out_rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--left", required=True, help="Left CSV path")
|
||||
p.add_argument(
|
||||
"--left-name-col", required=True, help="Name column in left CSV"
|
||||
)
|
||||
p.add_argument("--right", required=True, help="Right CSV path")
|
||||
p.add_argument(
|
||||
"--right-name-col",
|
||||
required=True,
|
||||
help="Name column in right CSV",
|
||||
)
|
||||
p.add_argument("--out", required=True, help="Output CSV path")
|
||||
p.add_argument(
|
||||
"--overlap-threshold",
|
||||
type=float,
|
||||
default=0.60,
|
||||
help="Jaccard overlap threshold for token_overlap tier (default 0.60)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--min-shared",
|
||||
type=int,
|
||||
default=2,
|
||||
help="Minimum shared tokens for token_overlap tier (default 2)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--skip-overlap",
|
||||
action="store_true",
|
||||
help="Skip the O(N*M) token_overlap pass (much faster on large CSVs)",
|
||||
)
|
||||
args = p.parse_args()
|
||||
|
||||
count = resolve(
|
||||
left_path=args.left,
|
||||
left_col=args.left_name_col,
|
||||
right_path=args.right,
|
||||
right_col=args.right_name_col,
|
||||
out_path=args.out,
|
||||
overlap_threshold=args.overlap_threshold,
|
||||
min_shared=args.min_shared,
|
||||
skip_overlap=args.skip_overlap,
|
||||
)
|
||||
print(f"Wrote {count} match rows to {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,149 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search court records via CourtListener (Free Law Project).
|
||||
|
||||
Covers ~10M federal and state court opinions, plus PACER docket data
|
||||
where available. Public REST API v4 supports anonymous read access for
|
||||
search; some endpoints require a token (free at courtlistener.com).
|
||||
|
||||
Set COURTLISTENER_TOKEN to authenticate (raises rate limits).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get_json # noqa: E402
|
||||
|
||||
BASE = "https://www.courtlistener.com/api/rest/v4/search/"
|
||||
|
||||
COLUMNS = [
|
||||
"case_name",
|
||||
"court",
|
||||
"court_id",
|
||||
"date_filed",
|
||||
"docket_number",
|
||||
"judge",
|
||||
"citation",
|
||||
"result_type",
|
||||
"snippet",
|
||||
"absolute_url",
|
||||
]
|
||||
|
||||
SEARCH_TYPES = {
|
||||
"opinions": "o", # Court opinions
|
||||
"dockets": "r", # PACER dockets (may require auth depending on coverage)
|
||||
"oral": "oa", # Oral arguments
|
||||
"people": "p", # Judges / people
|
||||
"recap": "r", # Same as dockets in v4
|
||||
}
|
||||
|
||||
|
||||
def fetch(
|
||||
query: str,
|
||||
search_type: str,
|
||||
court: str | None,
|
||||
date_from: str | None,
|
||||
date_to: str | None,
|
||||
token: str | None,
|
||||
limit: int,
|
||||
out_path: str,
|
||||
) -> int:
|
||||
type_code = SEARCH_TYPES.get(search_type, search_type)
|
||||
params = {
|
||||
"q": query,
|
||||
"type": type_code,
|
||||
}
|
||||
if court:
|
||||
params["court"] = court
|
||||
if date_from:
|
||||
params["filed_after"] = date_from
|
||||
if date_to:
|
||||
params["filed_before"] = date_to
|
||||
headers = {"Authorization": f"Token {token}"} if token else None
|
||||
|
||||
rows: list[dict[str, str]] = []
|
||||
next_url: str | None = f"{BASE}?{urllib.parse.urlencode(params)}"
|
||||
while next_url and len(rows) < limit:
|
||||
try:
|
||||
payload = get_json(next_url, headers=headers)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"CourtListener error: {e}", file=sys.stderr)
|
||||
break
|
||||
if not isinstance(payload, dict):
|
||||
break
|
||||
results = payload.get("results", [])
|
||||
for r in results:
|
||||
if len(rows) >= limit:
|
||||
break
|
||||
rows.append(
|
||||
{
|
||||
"case_name": r.get("caseName", "") or r.get("case_name", "") or "",
|
||||
"court": r.get("court", "") or "",
|
||||
"court_id": r.get("court_id", "") or "",
|
||||
"date_filed": (r.get("dateFiled", "") or r.get("date_filed", "") or "")[:10],
|
||||
"docket_number": r.get("docketNumber", "") or r.get("docket_number", "") or "",
|
||||
"judge": r.get("judge", "") or "",
|
||||
"citation": "; ".join(r.get("citation", []) or []) if isinstance(r.get("citation"), list) else (r.get("citation") or ""),
|
||||
"result_type": search_type,
|
||||
"snippet": (r.get("snippet", "") or "").replace("\n", " ")[:500],
|
||||
"absolute_url": (
|
||||
f"https://www.courtlistener.com{r.get('absolute_url', '')}"
|
||||
if r.get("absolute_url", "").startswith("/")
|
||||
else r.get("absolute_url", "")
|
||||
),
|
||||
}
|
||||
)
|
||||
next_url = payload.get("next")
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
if not rows:
|
||||
print(
|
||||
f"CourtListener: 0 results for type={search_type!r} q={query!r}. "
|
||||
"Most private individuals don't appear in published court records "
|
||||
"unless they were party to a federal or state appellate case.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--query", required=True, help="Search query (party name, case name, keyword)")
|
||||
p.add_argument(
|
||||
"--type",
|
||||
default="opinions",
|
||||
choices=list(SEARCH_TYPES.keys()),
|
||||
help="Search type (default: opinions)",
|
||||
)
|
||||
p.add_argument("--court", help="Court ID filter (e.g. 'nysd' = SDNY, 'scotus' = Supreme Court)")
|
||||
p.add_argument("--date-from", help="Filed-after date YYYY-MM-DD")
|
||||
p.add_argument("--date-to", help="Filed-before date YYYY-MM-DD")
|
||||
p.add_argument("--token", default=os.environ.get("COURTLISTENER_TOKEN"))
|
||||
p.add_argument("--limit", type=int, default=100)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
n = fetch(
|
||||
query=a.query,
|
||||
search_type=a.type,
|
||||
court=a.court,
|
||||
date_from=a.date_from,
|
||||
date_to=a.date_to,
|
||||
token=a.token,
|
||||
limit=a.limit,
|
||||
out_path=a.out,
|
||||
)
|
||||
print(f"Wrote {n} CourtListener rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search the GDELT 2.0 DOC API for news mentions.
|
||||
|
||||
GDELT monitors world news in 100+ languages and indexes the full text.
|
||||
Free, anonymous, ~15-minute update frequency. Covers ~2015→present.
|
||||
|
||||
Useful for surfacing news mentions of a person, company, or topic across
|
||||
international media — much wider net than Google News.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get_json # noqa: E402
|
||||
|
||||
BASE = "https://api.gdeltproject.org/api/v2/doc/doc"
|
||||
|
||||
COLUMNS = [
|
||||
"title",
|
||||
"url",
|
||||
"seen_date",
|
||||
"domain",
|
||||
"language",
|
||||
"source_country",
|
||||
"tone",
|
||||
"social_image",
|
||||
]
|
||||
|
||||
|
||||
def fetch(
|
||||
query: str,
|
||||
mode: str,
|
||||
timespan: str | None,
|
||||
start_datetime: str | None,
|
||||
end_datetime: str | None,
|
||||
source_country: str | None,
|
||||
source_lang: str | None,
|
||||
limit: int,
|
||||
out_path: str,
|
||||
) -> int:
|
||||
params: dict[str, str] = {
|
||||
"query": query,
|
||||
"mode": mode,
|
||||
"format": "json",
|
||||
"maxrecords": str(min(limit, 250)),
|
||||
"sort": "datedesc",
|
||||
}
|
||||
if timespan:
|
||||
params["timespan"] = timespan
|
||||
if start_datetime:
|
||||
params["startdatetime"] = start_datetime.replace("-", "").replace(":", "").replace(" ", "")
|
||||
if end_datetime:
|
||||
params["enddatetime"] = end_datetime.replace("-", "").replace(":", "").replace(" ", "")
|
||||
if source_country:
|
||||
params["sourcecountry"] = source_country
|
||||
if source_lang:
|
||||
params["sourcelang"] = source_lang
|
||||
|
||||
url = f"{BASE}?{urllib.parse.urlencode(params)}"
|
||||
payload: dict | list = {}
|
||||
for attempt in range(3):
|
||||
try:
|
||||
payload = get_json(url)
|
||||
break
|
||||
except RuntimeError as e:
|
||||
# GDELT requires 1 request per 5 seconds; back off and retry.
|
||||
if "429" in str(e) and attempt < 2:
|
||||
print(
|
||||
f"GDELT throttle hit; sleeping 6s before retry "
|
||||
f"(attempt {attempt + 1}/3)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
time.sleep(6)
|
||||
continue
|
||||
print(f"GDELT error: {e}", file=sys.stderr)
|
||||
payload = {}
|
||||
break
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"GDELT error: {e}", file=sys.stderr)
|
||||
payload = {}
|
||||
break
|
||||
|
||||
rows: list[dict[str, str]] = []
|
||||
if isinstance(payload, dict):
|
||||
articles = payload.get("articles", []) or []
|
||||
for a in articles[:limit]:
|
||||
seen = (a.get("seendate") or "")
|
||||
# GDELT format: 20260319T083000Z → 2026-03-19 08:30:00Z
|
||||
if len(seen) == 16 and "T" in seen:
|
||||
seen = f"{seen[0:4]}-{seen[4:6]}-{seen[6:8]} {seen[9:11]}:{seen[11:13]}:{seen[13:15]}Z"
|
||||
rows.append(
|
||||
{
|
||||
"title": (a.get("title") or "").replace("\n", " ").strip(),
|
||||
"url": a.get("url") or "",
|
||||
"seen_date": seen,
|
||||
"domain": a.get("domain") or "",
|
||||
"language": a.get("language") or "",
|
||||
"source_country": a.get("sourcecountry") or "",
|
||||
"tone": str(a.get("tone") or ""),
|
||||
"social_image": a.get("socialimage") or "",
|
||||
}
|
||||
)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
if not rows:
|
||||
print(
|
||||
f"GDELT: 0 articles for query={query!r}. "
|
||||
"GDELT indexes ~2015→present. Try widening the timespan or "
|
||||
"checking the query syntax (https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/).",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--query", required=True, help='Search query (supports GDELT operators: quoted phrases, AND/OR/NOT, sourcecountry:, theme:)')
|
||||
p.add_argument(
|
||||
"--mode",
|
||||
default="ArtList",
|
||||
choices=["ArtList", "ImageCollage", "TimelineVol", "TimelineTone", "ToneChart"],
|
||||
help="GDELT mode (default ArtList for article list)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--timespan",
|
||||
help="Relative window: e.g. '1d', '1w', '1m', '3m', '1y' (overrides start/end)",
|
||||
)
|
||||
p.add_argument("--start", help="Absolute start YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS")
|
||||
p.add_argument("--end", help="Absolute end YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS")
|
||||
p.add_argument("--source-country", help="2-letter source country (e.g. US, UK)")
|
||||
p.add_argument("--source-lang", help="Source language (e.g. English, Spanish)")
|
||||
p.add_argument("--limit", type=int, default=100)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
n = fetch(
|
||||
query=a.query,
|
||||
mode=a.mode,
|
||||
timespan=a.timespan,
|
||||
start_datetime=a.start,
|
||||
end_datetime=a.end,
|
||||
source_country=a.source_country,
|
||||
source_lang=a.source_lang,
|
||||
limit=a.limit,
|
||||
out_path=a.out,
|
||||
)
|
||||
print(f"Wrote {n} GDELT article rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search ICIJ Offshore Leaks via the bulk CSV database.
|
||||
|
||||
The old reconcile endpoint (https://offshoreleaks.icij.org/reconcile) returns
|
||||
404 — ICIJ has removed it. The remaining stable access path is the public
|
||||
bulk download:
|
||||
|
||||
https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip
|
||||
|
||||
~70 MB, ~6 CSVs inside (nodes-entities, nodes-officers, nodes-intermediaries,
|
||||
nodes-addresses, relationships, ...). We cache it under
|
||||
$HERMES_OSINT_CACHE/icij/ (default: ~/.cache/hermes-osint/icij/) and search
|
||||
locally so the agent doesn't re-download for every query.
|
||||
|
||||
Output CSV columns match the original `fetch_icij_offshore.py` contract.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import io
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
BULK_URL = "https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip"
|
||||
|
||||
COLUMNS = [
|
||||
"node_id",
|
||||
"name",
|
||||
"node_type",
|
||||
"country_codes",
|
||||
"countries",
|
||||
"jurisdiction",
|
||||
"incorporation_date",
|
||||
"inactivation_date",
|
||||
"source",
|
||||
"entity_url",
|
||||
"connections",
|
||||
]
|
||||
|
||||
|
||||
def _cache_dir() -> Path:
|
||||
base = os.environ.get("HERMES_OSINT_CACHE")
|
||||
if base:
|
||||
return Path(base) / "icij"
|
||||
return Path.home() / ".cache" / "hermes-osint" / "icij"
|
||||
|
||||
|
||||
def _download(dest: Path, force: bool = False) -> Path:
|
||||
"""Download (or reuse cached) ICIJ bulk ZIP."""
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
zip_path = dest / "full-oldb.zip"
|
||||
if zip_path.exists() and not force:
|
||||
# Re-check age: refetch if older than 30 days.
|
||||
age_days = (time.time() - zip_path.stat().st_mtime) / 86400
|
||||
if age_days < 30:
|
||||
return zip_path
|
||||
print(f"Downloading ICIJ bulk database (~70 MB) to {zip_path}", file=sys.stderr)
|
||||
req = urllib.request.Request(
|
||||
BULK_URL,
|
||||
headers={"User-Agent": "hermes-agent osint-investigation skill"},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=120) as resp: # noqa: S310
|
||||
tmp = zip_path.with_suffix(".zip.tmp")
|
||||
with open(tmp, "wb") as fh:
|
||||
while True:
|
||||
chunk = resp.read(1 << 16)
|
||||
if not chunk:
|
||||
break
|
||||
fh.write(chunk)
|
||||
tmp.replace(zip_path)
|
||||
return zip_path
|
||||
|
||||
|
||||
def _open_csv(zf: zipfile.ZipFile, name_pattern: str):
|
||||
"""Open the first CSV matching name_pattern (case-insensitive substring)."""
|
||||
for info in zf.infolist():
|
||||
if name_pattern.lower() in info.filename.lower() and info.filename.lower().endswith(".csv"):
|
||||
return zf.open(info), info.filename
|
||||
return None, None
|
||||
|
||||
|
||||
def _match(needle_norm: str, hay: str) -> bool:
|
||||
return needle_norm in (hay or "").upper()
|
||||
|
||||
|
||||
def _normalize_query(s: str) -> str:
|
||||
s = s.upper()
|
||||
s = re.sub(r"[^\w\s]", " ", s)
|
||||
s = re.sub(r"\s+", " ", s).strip()
|
||||
return s
|
||||
|
||||
|
||||
def fetch(
|
||||
entity: str | None,
|
||||
officer: str | None,
|
||||
jurisdiction: str | None,
|
||||
out_path: str,
|
||||
cache_dir: Path,
|
||||
force_refresh: bool = False,
|
||||
limit: int = 500,
|
||||
) -> int:
|
||||
zip_path = _download(cache_dir, force=force_refresh)
|
||||
rows: list[dict[str, str]] = []
|
||||
needles: list[tuple[str, str]] = [] # (kind, normalized needle)
|
||||
if entity:
|
||||
needles.append(("Entity", _normalize_query(entity)))
|
||||
if officer:
|
||||
needles.append(("Officer", _normalize_query(officer)))
|
||||
jur_norm = _normalize_query(jurisdiction) if jurisdiction else None
|
||||
|
||||
targets = [
|
||||
("Entity", "nodes-entities"),
|
||||
("Officer", "nodes-officers"),
|
||||
("Intermediary", "nodes-intermediaries"),
|
||||
]
|
||||
|
||||
with zipfile.ZipFile(zip_path) as zf:
|
||||
for node_type, csv_substring in targets:
|
||||
relevant_needles = [n for (k, n) in needles if k in {node_type, "Entity", "Officer"}] or []
|
||||
# Only scan a CSV if we have a needle that could plausibly match it,
|
||||
# or if we have ONLY a jurisdiction filter.
|
||||
applicable_needles = [n for (k, n) in needles if k == node_type]
|
||||
if needles and not applicable_needles and not jur_norm:
|
||||
continue
|
||||
stream, fname = _open_csv(zf, csv_substring)
|
||||
if not stream:
|
||||
continue
|
||||
with stream:
|
||||
text = io.TextIOWrapper(stream, encoding="utf-8", errors="replace")
|
||||
reader = csv.DictReader(text)
|
||||
for row in reader:
|
||||
name = (row.get("name") or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
name_u = name.upper()
|
||||
matched = False
|
||||
for n in applicable_needles or relevant_needles:
|
||||
if _match(n, name_u):
|
||||
matched = True
|
||||
break
|
||||
if not needles:
|
||||
matched = True # jurisdiction-only sweep
|
||||
if not matched:
|
||||
continue
|
||||
jur = (row.get("jurisdiction_description") or row.get("country_codes") or "").strip()
|
||||
if jur_norm and jur_norm not in jur.upper() and jur_norm not in (row.get("countries") or "").upper():
|
||||
continue
|
||||
node_id = (row.get("node_id") or "").strip()
|
||||
rows.append(
|
||||
{
|
||||
"node_id": node_id,
|
||||
"name": name,
|
||||
"node_type": node_type,
|
||||
"country_codes": row.get("country_codes", "") or "",
|
||||
"countries": row.get("countries", "") or "",
|
||||
"jurisdiction": jur,
|
||||
"incorporation_date": row.get("incorporation_date", "") or "",
|
||||
"inactivation_date": row.get("inactivation_date", "") or "",
|
||||
"source": row.get("sourceID", "") or row.get("source", "") or "",
|
||||
"entity_url": (
|
||||
f"https://offshoreleaks.icij.org/nodes/{node_id}" if node_id else ""
|
||||
),
|
||||
"connections": "",
|
||||
}
|
||||
)
|
||||
if len(rows) >= limit:
|
||||
break
|
||||
if len(rows) >= limit:
|
||||
break
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
if not rows:
|
||||
bits = []
|
||||
if entity:
|
||||
bits.append(f"entity={entity!r}")
|
||||
if officer:
|
||||
bits.append(f"officer={officer!r}")
|
||||
if jurisdiction:
|
||||
bits.append(f"jurisdiction={jurisdiction!r}")
|
||||
print(
|
||||
f"ICIJ: 0 matches for {', '.join(bits)}. "
|
||||
"The bulk database covers offshore leaks (Panama, Paradise, Pandora, "
|
||||
"Bahamas, Offshore Leaks). Most private US individuals are NOT in it.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--entity", help="Search by entity name (substring, case-insensitive)")
|
||||
p.add_argument("--officer", help="Search by officer / individual name (substring, case-insensitive)")
|
||||
p.add_argument("--jurisdiction", help="Filter results by jurisdiction substring")
|
||||
p.add_argument("--limit", type=int, default=500)
|
||||
p.add_argument("--out", required=True)
|
||||
p.add_argument(
|
||||
"--cache-dir",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Override cache directory (default: $HERMES_OSINT_CACHE/icij or ~/.cache/hermes-osint/icij)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--force-refresh",
|
||||
action="store_true",
|
||||
help="Re-download the bulk ZIP even if a recent cached copy exists.",
|
||||
)
|
||||
a = p.parse_args()
|
||||
if not (a.entity or a.officer or a.jurisdiction):
|
||||
p.error("must supply at least one of --entity / --officer / --jurisdiction")
|
||||
n = fetch(
|
||||
entity=a.entity,
|
||||
officer=a.officer,
|
||||
jurisdiction=a.jurisdiction,
|
||||
out_path=a.out,
|
||||
cache_dir=a.cache_dir or _cache_dir(),
|
||||
force_refresh=a.force_refresh,
|
||||
limit=a.limit,
|
||||
)
|
||||
print(f"Wrote {n} ICIJ Offshore Leaks rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search NYC property records via ACRIS (Automated City Register Information System).
|
||||
|
||||
Uses the city's Socrata-backed open data API. No auth required for read access.
|
||||
|
||||
Datasets:
|
||||
bnx9-e6tj — Real Property Master (one row per recorded document)
|
||||
636b-3b5g — Real Property Parties (names — grantor, grantee, etc.)
|
||||
8h5j-fqxa — Real Property Legal (lot / property identifiers)
|
||||
uqqa-hym2 — Real Property References
|
||||
|
||||
The Parties dataset has the names. We search by name and optionally join to
|
||||
Master to get the doc type and date.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get_json # noqa: E402
|
||||
|
||||
PARTIES_URL = "https://data.cityofnewyork.us/resource/636b-3b5g.json"
|
||||
MASTER_URL = "https://data.cityofnewyork.us/resource/bnx9-e6tj.json"
|
||||
|
||||
PARTY_TYPE = {
|
||||
"1": "grantor (seller / mortgagor / debtor)",
|
||||
"2": "grantee (buyer / mortgagee / creditor)",
|
||||
"3": "other party",
|
||||
}
|
||||
|
||||
BOROUGH = {
|
||||
"1": "Manhattan",
|
||||
"2": "Bronx",
|
||||
"3": "Brooklyn",
|
||||
"4": "Queens",
|
||||
"5": "Staten Island",
|
||||
}
|
||||
|
||||
COLUMNS = [
|
||||
"document_id",
|
||||
"name",
|
||||
"party_type",
|
||||
"party_role",
|
||||
"address_1",
|
||||
"address_2",
|
||||
"city",
|
||||
"state",
|
||||
"zip",
|
||||
"country",
|
||||
"doc_type",
|
||||
"doc_date",
|
||||
"recorded_date",
|
||||
"borough",
|
||||
"amount",
|
||||
"filing_url",
|
||||
]
|
||||
|
||||
|
||||
def _filing_url(document_id: str) -> str:
|
||||
if not document_id:
|
||||
return ""
|
||||
return (
|
||||
f"https://a836-acris.nyc.gov/DS/DocumentSearch/DocumentImageView?doc_id={document_id}"
|
||||
)
|
||||
|
||||
|
||||
def fetch(
|
||||
name: str | None,
|
||||
address: str | None,
|
||||
party_type: str | None,
|
||||
limit: int,
|
||||
out_path: str,
|
||||
enrich: bool = True,
|
||||
) -> int:
|
||||
if not (name or address):
|
||||
raise SystemExit("must supply --name or --address")
|
||||
|
||||
where_clauses: list[str] = []
|
||||
if name:
|
||||
safe = name.upper().replace("'", "''")
|
||||
where_clauses.append(f"upper(name) like '%{safe}%'")
|
||||
if address:
|
||||
safe_addr = address.upper().replace("'", "''")
|
||||
where_clauses.append(f"upper(address_1) like '%{safe_addr}%'")
|
||||
if party_type and party_type in {"1", "2", "3"}:
|
||||
where_clauses.append(f"party_type='{party_type}'")
|
||||
|
||||
params = {
|
||||
"$where": " AND ".join(where_clauses),
|
||||
"$limit": str(limit),
|
||||
}
|
||||
url = f"{PARTIES_URL}?{urllib.parse.urlencode(params)}"
|
||||
parties = get_json(url)
|
||||
if not isinstance(parties, list):
|
||||
raise SystemExit(f"Unexpected ACRIS response: {parties!r}")
|
||||
|
||||
# Enrich with master record (doc_type, dates, borough, amount).
|
||||
doc_ids: list[str] = sorted({
|
||||
d for d in (p.get("document_id") for p in parties) if d
|
||||
})
|
||||
masters: dict[str, dict] = {}
|
||||
if enrich and doc_ids:
|
||||
# Batch up to 100 doc_ids per request (Socrata IN-list is fine for this).
|
||||
for i in range(0, len(doc_ids), 100):
|
||||
chunk = doc_ids[i : i + 100]
|
||||
id_list = ",".join(f"'{d}'" for d in chunk)
|
||||
master_params = {
|
||||
"$where": f"document_id in ({id_list})",
|
||||
"$limit": "100",
|
||||
}
|
||||
url = f"{MASTER_URL}?{urllib.parse.urlencode(master_params)}"
|
||||
try:
|
||||
rows = get_json(url)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"ACRIS master lookup failed for chunk: {e}", file=sys.stderr)
|
||||
continue
|
||||
if isinstance(rows, list):
|
||||
for r in rows:
|
||||
did = r.get("document_id", "")
|
||||
if did:
|
||||
masters[did] = r
|
||||
|
||||
out_rows: list[dict[str, str]] = []
|
||||
for p in parties:
|
||||
did = p.get("document_id", "") or ""
|
||||
m = masters.get(did, {})
|
||||
out_rows.append(
|
||||
{
|
||||
"document_id": did,
|
||||
"name": p.get("name", "") or "",
|
||||
"party_type": p.get("party_type", "") or "",
|
||||
"party_role": PARTY_TYPE.get(p.get("party_type", ""), ""),
|
||||
"address_1": p.get("address_1", "") or "",
|
||||
"address_2": p.get("address_2", "") or "",
|
||||
"city": p.get("city", "") or "",
|
||||
"state": p.get("state", "") or "",
|
||||
"zip": p.get("zip", "") or "",
|
||||
"country": p.get("country", "") or "",
|
||||
"doc_type": m.get("doc_type", "") or "",
|
||||
"doc_date": (m.get("document_date", "") or "")[:10],
|
||||
"recorded_date": (m.get("recorded_datetime", "") or "")[:10],
|
||||
"borough": BOROUGH.get(m.get("recorded_borough", ""), m.get("recorded_borough", "")),
|
||||
"amount": m.get("document_amt", "") or "",
|
||||
"filing_url": _filing_url(did),
|
||||
}
|
||||
)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(out_rows)
|
||||
|
||||
if not out_rows:
|
||||
filters = []
|
||||
if name:
|
||||
filters.append(f"name={name!r}")
|
||||
if address:
|
||||
filters.append(f"address={address!r}")
|
||||
print(
|
||||
f"NYC ACRIS: 0 records for {', '.join(filters)}. "
|
||||
"ACRIS covers ONLY NYC (5 boroughs). For property records elsewhere, "
|
||||
"search the relevant county recorder directly.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(out_rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--name", help="Party name substring (case-insensitive)")
|
||||
p.add_argument("--address", help="Address line 1 substring")
|
||||
p.add_argument(
|
||||
"--party-type",
|
||||
choices=["1", "2", "3"],
|
||||
help="Filter party type: 1=grantor (seller/mortgagor), 2=grantee (buyer/mortgagee), 3=other",
|
||||
)
|
||||
p.add_argument("--limit", type=int, default=200)
|
||||
p.add_argument(
|
||||
"--no-enrich",
|
||||
action="store_true",
|
||||
help="Skip the master-document lookup that adds doc_type/date/amount",
|
||||
)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
n = fetch(
|
||||
name=a.name,
|
||||
address=a.address,
|
||||
party_type=a.party_type,
|
||||
limit=a.limit,
|
||||
out_path=a.out,
|
||||
enrich=not a.no_enrich,
|
||||
)
|
||||
print(f"Wrote {n} NYC ACRIS rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,175 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch OFAC SDN list (CSV format) and normalize.
|
||||
|
||||
Public endpoint: https://www.treasury.gov/ofac/downloads/sdn.csv
|
||||
Format reference: https://ofac.treasury.gov/specially-designated-nationals-and-blocked-persons-list-sdn-human-readable-lists
|
||||
|
||||
The SDN CSV uses a specific 12-column format with no header row:
|
||||
ent_num, sdn_name, sdn_type, program, title, call_sign, vess_type,
|
||||
tonnage, grt, vess_flag, vess_owner, remarks
|
||||
Address and AKA records live in separate files. We fetch all three and join.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import io
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get # noqa: E402
|
||||
|
||||
SDN_URL = "https://www.treasury.gov/ofac/downloads/sdn.csv"
|
||||
ADD_URL = "https://www.treasury.gov/ofac/downloads/add.csv"
|
||||
ALT_URL = "https://www.treasury.gov/ofac/downloads/alt.csv"
|
||||
|
||||
SDN_COLS = [
|
||||
"ent_num", "sdn_name", "sdn_type", "program", "title",
|
||||
"call_sign", "vess_type", "tonnage", "grt", "vess_flag",
|
||||
"vess_owner", "remarks",
|
||||
]
|
||||
ADD_COLS = [
|
||||
"ent_num", "add_num", "address", "city_state_zip", "country", "add_remarks",
|
||||
]
|
||||
ALT_COLS = [
|
||||
"ent_num", "alt_num", "alt_type", "alt_name", "alt_remarks",
|
||||
]
|
||||
|
||||
COLUMNS = [
|
||||
"entity_id",
|
||||
"name",
|
||||
"entity_type",
|
||||
"program_list",
|
||||
"title",
|
||||
"nationalities",
|
||||
"aka_list",
|
||||
"addresses",
|
||||
"dob",
|
||||
"pob",
|
||||
"remarks",
|
||||
"last_updated",
|
||||
]
|
||||
|
||||
_TYPE_MAP = {
|
||||
"individual": "individual",
|
||||
"entity": "entity",
|
||||
"vessel": "vessel",
|
||||
"aircraft": "aircraft",
|
||||
}
|
||||
|
||||
|
||||
def _read_csv(url: str, columns: list[str]) -> list[dict[str, str]]:
|
||||
body = get(url, timeout=60).decode("latin-1", errors="replace")
|
||||
reader = csv.reader(io.StringIO(body))
|
||||
out = []
|
||||
for row in reader:
|
||||
if not row:
|
||||
continue
|
||||
# Pad/truncate to expected width.
|
||||
row = row[: len(columns)] + [""] * (len(columns) - len(row))
|
||||
out.append(dict(zip(columns, row)))
|
||||
return out
|
||||
|
||||
|
||||
def _strip_quotes(s: str) -> str:
|
||||
s = s.strip()
|
||||
if s.startswith('"') and s.endswith('"'):
|
||||
s = s[1:-1]
|
||||
if s == "-0-":
|
||||
return ""
|
||||
return s
|
||||
|
||||
|
||||
def fetch(
|
||||
program: str | None,
|
||||
entity_type: str | None,
|
||||
out_path: str,
|
||||
) -> int:
|
||||
sdn = _read_csv(SDN_URL, SDN_COLS)
|
||||
addresses = _read_csv(ADD_URL, ADD_COLS)
|
||||
akas = _read_csv(ALT_URL, ALT_COLS)
|
||||
|
||||
addr_by_ent: dict[str, list[str]] = defaultdict(list)
|
||||
for a in addresses:
|
||||
ent = _strip_quotes(a["ent_num"])
|
||||
parts = [
|
||||
_strip_quotes(a[c])
|
||||
for c in ("address", "city_state_zip", "country")
|
||||
if _strip_quotes(a[c])
|
||||
]
|
||||
if parts:
|
||||
addr_by_ent[ent].append(", ".join(parts))
|
||||
|
||||
aka_by_ent: dict[str, list[str]] = defaultdict(list)
|
||||
for k in akas:
|
||||
ent = _strip_quotes(k["ent_num"])
|
||||
name = _strip_quotes(k["alt_name"])
|
||||
if name:
|
||||
aka_by_ent[ent].append(name)
|
||||
|
||||
rows: list[dict[str, str]] = []
|
||||
for r in sdn:
|
||||
ent_num = _strip_quotes(r["ent_num"])
|
||||
if not ent_num:
|
||||
continue
|
||||
sdn_type = _TYPE_MAP.get(_strip_quotes(r["sdn_type"]).lower(), _strip_quotes(r["sdn_type"]))
|
||||
if entity_type and sdn_type != entity_type:
|
||||
continue
|
||||
progs = _strip_quotes(r["program"])
|
||||
if program and program.upper() not in progs.upper().split(";"):
|
||||
continue
|
||||
remarks = _strip_quotes(r["remarks"])
|
||||
# DOB / POB are commonly embedded in remarks for individuals.
|
||||
dob = ""
|
||||
pob = ""
|
||||
if sdn_type == "individual" and remarks:
|
||||
for chunk in remarks.split(";"):
|
||||
ch = chunk.strip()
|
||||
if ch.upper().startswith("DOB"):
|
||||
dob = ch.split(maxsplit=1)[1] if " " in ch else ""
|
||||
elif ch.upper().startswith("POB"):
|
||||
pob = ch.split(maxsplit=1)[1] if " " in ch else ""
|
||||
rows.append(
|
||||
{
|
||||
"entity_id": ent_num,
|
||||
"name": _strip_quotes(r["sdn_name"]),
|
||||
"entity_type": sdn_type,
|
||||
"program_list": "; ".join(p.strip() for p in progs.split(";") if p.strip()),
|
||||
"title": _strip_quotes(r["title"]),
|
||||
"nationalities": "", # not in this CSV; available in XML format
|
||||
"aka_list": "; ".join(aka_by_ent.get(ent_num, [])),
|
||||
"addresses": "; ".join(addr_by_ent.get(ent_num, [])),
|
||||
"dob": dob,
|
||||
"pob": pob,
|
||||
"remarks": remarks,
|
||||
"last_updated": "",
|
||||
}
|
||||
)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("--program", help="Filter to specific sanctions program (e.g. SDGT, IRAN)")
|
||||
p.add_argument(
|
||||
"--entity-type",
|
||||
choices=["individual", "entity", "vessel", "aircraft"],
|
||||
help="Filter to a specific entity type",
|
||||
)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
n = fetch(program=a.program, entity_type=a.entity_type, out_path=a.out)
|
||||
print(f"Wrote {n} OFAC SDN rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,191 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search OpenCorporates company registry data.
|
||||
|
||||
OpenCorporates aggregates ~200M companies from 130+ jurisdictions. The
|
||||
public API requires an API token (free tier: 500 calls/month). Set
|
||||
OPENCORPORATES_API_TOKEN in env or pass --token.
|
||||
|
||||
Without a token, this script falls back to scraping the public HTML
|
||||
search page (limited fields, more brittle, no jurisdiction filter).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get, get_json # noqa: E402
|
||||
|
||||
API_URL = "https://api.opencorporates.com/v0.4/companies/search"
|
||||
HTML_URL = "https://opencorporates.com/companies"
|
||||
|
||||
COLUMNS = [
|
||||
"name",
|
||||
"company_number",
|
||||
"jurisdiction_code",
|
||||
"jurisdiction_name",
|
||||
"incorporation_date",
|
||||
"dissolution_date",
|
||||
"company_type",
|
||||
"status",
|
||||
"registered_address",
|
||||
"opencorporates_url",
|
||||
"officers_count",
|
||||
"source",
|
||||
]
|
||||
|
||||
|
||||
def _via_api(query: str, jurisdiction: str | None, token: str, limit: int) -> list[dict]:
|
||||
params = {
|
||||
"q": query,
|
||||
"api_token": token,
|
||||
"per_page": str(min(limit, 100)),
|
||||
}
|
||||
if jurisdiction:
|
||||
params["jurisdiction_code"] = jurisdiction
|
||||
url = f"{API_URL}?{urllib.parse.urlencode(params)}"
|
||||
payload = get_json(url)
|
||||
if not isinstance(payload, dict):
|
||||
return []
|
||||
results = payload.get("results", {}).get("companies", []) or []
|
||||
return [r.get("company", {}) for r in results if isinstance(r, dict)]
|
||||
|
||||
|
||||
def _via_html(query: str, limit: int) -> list[dict]:
|
||||
"""Best-effort HTML fallback when no API token is available."""
|
||||
params = {"q": query, "utf8": "✓"}
|
||||
url = f"{HTML_URL}?{urllib.parse.urlencode(params)}"
|
||||
body = get(url, user_agent="Mozilla/5.0 hermes-osint").decode("utf-8", errors="replace")
|
||||
# Each result is in <li class="company"> ... </li> with name, url, status
|
||||
pattern = re.compile(
|
||||
r'<li[^>]*class="[^"]*company[^"]*"[^>]*>.*?'
|
||||
r'<a[^>]+href="(?P<url>/companies/[^"]+)"[^>]*>(?P<name>[^<]+)</a>'
|
||||
r'(?:.*?<span[^>]*class="[^"]*jurisdiction[^"]*"[^>]*>(?P<jur>[^<]+)</span>)?'
|
||||
r"(?:.*?<dt[^>]*>(?:Company\s+Number|Number)</dt>\s*<dd[^>]*>(?P<num>[^<]+)</dd>)?",
|
||||
re.DOTALL | re.IGNORECASE,
|
||||
)
|
||||
out = []
|
||||
for m in pattern.finditer(body):
|
||||
if len(out) >= limit:
|
||||
break
|
||||
url_path = m.group("url").strip()
|
||||
out.append(
|
||||
{
|
||||
"name": (m.group("name") or "").strip(),
|
||||
"opencorporates_url": f"https://opencorporates.com{url_path}",
|
||||
"jurisdiction_code": (m.group("jur") or "").strip(),
|
||||
"company_number": (m.group("num") or "").strip(),
|
||||
"_via": "html",
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def fetch(
|
||||
query: str,
|
||||
jurisdiction: str | None,
|
||||
token: str | None,
|
||||
limit: int,
|
||||
out_path: str,
|
||||
) -> int:
|
||||
if token:
|
||||
try:
|
||||
companies = _via_api(query, jurisdiction, token, limit)
|
||||
source_tag = "api"
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(
|
||||
f"OpenCorporates API call failed ({e}); falling back to HTML.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
companies = _via_html(query, limit)
|
||||
source_tag = "html-fallback"
|
||||
else:
|
||||
print(
|
||||
"OPENCORPORATES_API_TOKEN not set — using HTML fallback (limited fields). "
|
||||
"Get a free token at https://opencorporates.com/api_accounts/new",
|
||||
file=sys.stderr,
|
||||
)
|
||||
companies = _via_html(query, limit)
|
||||
source_tag = "html"
|
||||
|
||||
rows: list[dict[str, str]] = []
|
||||
for c in companies[:limit]:
|
||||
if c.get("_via") == "html":
|
||||
rows.append(
|
||||
{
|
||||
"name": c.get("name", ""),
|
||||
"company_number": c.get("company_number", ""),
|
||||
"jurisdiction_code": c.get("jurisdiction_code", ""),
|
||||
"jurisdiction_name": "",
|
||||
"incorporation_date": "",
|
||||
"dissolution_date": "",
|
||||
"company_type": "",
|
||||
"status": "",
|
||||
"registered_address": "",
|
||||
"opencorporates_url": c.get("opencorporates_url", ""),
|
||||
"officers_count": "",
|
||||
"source": source_tag,
|
||||
}
|
||||
)
|
||||
continue
|
||||
addr = c.get("registered_address_in_full") or ""
|
||||
rows.append(
|
||||
{
|
||||
"name": c.get("name", "") or "",
|
||||
"company_number": c.get("company_number", "") or "",
|
||||
"jurisdiction_code": c.get("jurisdiction_code", "") or "",
|
||||
"jurisdiction_name": "",
|
||||
"incorporation_date": c.get("incorporation_date", "") or "",
|
||||
"dissolution_date": c.get("dissolution_date", "") or "",
|
||||
"company_type": c.get("company_type", "") or "",
|
||||
"status": c.get("current_status", "") or c.get("inactive", "") or "",
|
||||
"registered_address": addr,
|
||||
"opencorporates_url": c.get("opencorporates_url", "") or "",
|
||||
"officers_count": str(c.get("officers", {}).get("total_count", "") if c.get("officers") else ""),
|
||||
"source": source_tag,
|
||||
}
|
||||
)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
if not rows:
|
||||
print(
|
||||
f"OpenCorporates: 0 matches for query={query!r}"
|
||||
f"{f' jurisdiction={jurisdiction!r}' if jurisdiction else ''}.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--query", required=True, help="Company name search")
|
||||
p.add_argument(
|
||||
"--jurisdiction",
|
||||
help="Jurisdiction code, e.g. 'us_ny', 'us_de', 'gb', 'sg' (lowercased OpenCorporates style)",
|
||||
)
|
||||
p.add_argument("--limit", type=int, default=50)
|
||||
p.add_argument("--token", default=os.environ.get("OPENCORPORATES_API_TOKEN"))
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
n = fetch(
|
||||
query=a.query,
|
||||
jurisdiction=a.jurisdiction,
|
||||
token=a.token,
|
||||
limit=a.limit,
|
||||
out_path=a.out,
|
||||
)
|
||||
print(f"Wrote {n} OpenCorporates rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,184 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch SEC EDGAR filings index for a given CIK or company name.
|
||||
|
||||
SEC requires a User-Agent header with contact info. Set SEC_USER_AGENT,
|
||||
e.g. SEC_USER_AGENT="Research example@example.com".
|
||||
|
||||
Filings JSON is published at:
|
||||
https://data.sec.gov/submissions/CIK<10-digit-padded>.json
|
||||
|
||||
Company lookup uses:
|
||||
https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&company=<name>&output=atom
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get, get_json # noqa: E402
|
||||
|
||||
SUBMISSIONS_URL = "https://data.sec.gov/submissions/CIK{cik}.json"
|
||||
COLUMNS = [
|
||||
"cik",
|
||||
"company_name",
|
||||
"form_type",
|
||||
"filing_date",
|
||||
"accession_number",
|
||||
"primary_document",
|
||||
"filing_url",
|
||||
"reporting_period",
|
||||
]
|
||||
|
||||
|
||||
def _ua() -> str:
|
||||
ua = os.environ.get("SEC_USER_AGENT", "").strip()
|
||||
if not ua:
|
||||
raise SystemExit(
|
||||
"SEC requires a User-Agent with contact info. "
|
||||
"Set SEC_USER_AGENT='Your Name your@email'."
|
||||
)
|
||||
return ua
|
||||
|
||||
|
||||
def _resolve_cik(company: str) -> tuple[str, str]:
|
||||
"""Resolve a company name to a CIK via EDGAR's atom feed.
|
||||
|
||||
Returns (cik, resolved_company_name). The feed entries also reveal whether
|
||||
the match is an individual filer (Form 3/4/5 only) — surfaced in the
|
||||
return value so callers can warn.
|
||||
"""
|
||||
url = "https://www.sec.gov/cgi-bin/browse-edgar"
|
||||
params = {"action": "getcompany", "company": company, "output": "atom", "owner": "include"}
|
||||
body = get(url, params=params, user_agent=_ua()).decode("utf-8", errors="replace")
|
||||
m = re.search(r"CIK=(\d{10})", body)
|
||||
if not m:
|
||||
raise SystemExit(f"Could not resolve CIK for company={company!r}")
|
||||
cik = m.group(1)
|
||||
name_m = re.search(r"<title>([^<]+)\s*\((\d{10})\)</title>", body)
|
||||
resolved = name_m.group(1).strip() if name_m else ""
|
||||
return cik, resolved
|
||||
|
||||
|
||||
def fetch(
|
||||
cik: str | None,
|
||||
company: str | None,
|
||||
types: list[str],
|
||||
since: str | None,
|
||||
out_path: str,
|
||||
) -> int:
|
||||
resolved_name = ""
|
||||
if not cik and company:
|
||||
try:
|
||||
cik, resolved_name = _resolve_cik(company) # type: ignore[assignment]
|
||||
except SystemExit as e:
|
||||
# Write empty CSV with header so downstream tools still work,
|
||||
# and tell the user clearly.
|
||||
print(f"SEC EDGAR: {e}", file=sys.stderr)
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
csv.DictWriter(fh, fieldnames=COLUMNS).writeheader()
|
||||
return 0
|
||||
if resolved_name:
|
||||
print(
|
||||
f"Resolved company={company!r} → CIK {cik} ({resolved_name})",
|
||||
file=sys.stderr,
|
||||
)
|
||||
if not cik:
|
||||
raise SystemExit("must supply --cik or --company")
|
||||
cik = cik.zfill(10)
|
||||
url = SUBMISSIONS_URL.format(cik=cik)
|
||||
payload = get_json(url, user_agent=_ua())
|
||||
if not isinstance(payload, dict):
|
||||
raise SystemExit(f"Unexpected EDGAR response shape for CIK {cik}")
|
||||
name = payload.get("name", "")
|
||||
recent = (payload.get("filings", {}) or {}).get("recent", {}) or {}
|
||||
form = recent.get("form", [])
|
||||
date = recent.get("filingDate", [])
|
||||
accession = recent.get("accessionNumber", [])
|
||||
primary_doc = recent.get("primaryDocument", [])
|
||||
period = recent.get("reportDate", [])
|
||||
|
||||
# Histogram of available filing types — useful for surfacing why a filter
|
||||
# returned 0 (e.g. user asked for 10-K on an individual Form 4 filer).
|
||||
type_hist: dict[str, int] = {}
|
||||
for ftype in form:
|
||||
type_hist[ftype] = type_hist.get(ftype, 0) + 1
|
||||
|
||||
type_set = {t.strip().upper() for t in types} if types else None
|
||||
rows: list[dict[str, str]] = []
|
||||
for i, ftype in enumerate(form):
|
||||
if type_set and ftype.upper() not in type_set:
|
||||
continue
|
||||
fdate = date[i] if i < len(date) else ""
|
||||
if since and fdate and fdate < since:
|
||||
continue
|
||||
acc = accession[i] if i < len(accession) else ""
|
||||
pdoc = primary_doc[i] if i < len(primary_doc) else ""
|
||||
acc_nodash = acc.replace("-", "")
|
||||
filing_url = (
|
||||
f"https://www.sec.gov/Archives/edgar/data/{int(cik)}/{acc_nodash}/{pdoc}"
|
||||
if acc and pdoc
|
||||
else ""
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"cik": cik,
|
||||
"company_name": name,
|
||||
"form_type": ftype,
|
||||
"filing_date": fdate,
|
||||
"accession_number": acc,
|
||||
"primary_document": pdoc,
|
||||
"filing_url": filing_url,
|
||||
"reporting_period": period[i] if i < len(period) else "",
|
||||
}
|
||||
)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
|
||||
if not rows and type_hist:
|
||||
top = sorted(type_hist.items(), key=lambda kv: -kv[1])[:8]
|
||||
hist_str = ", ".join(f"{t}={n}" for t, n in top)
|
||||
print(
|
||||
f"Warning: SEC EDGAR CIK {cik} ({name}) has {sum(type_hist.values())} "
|
||||
f"recent filings but NONE match types={types}. "
|
||||
f"Available form types: {hist_str}.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
# Insider-filer heuristic: only Form 3/4/5 → individual person, not a company.
|
||||
company_types = {"10-K", "10-Q", "8-K", "20-F", "DEF 14A", "S-1"}
|
||||
if not (set(type_hist.keys()) & company_types):
|
||||
print(
|
||||
f"Note: CIK {cik} appears to be an INDIVIDUAL filer "
|
||||
f"(insider Form 3/4/5 only), not a corporate registrant. "
|
||||
f"The resolver may have matched an officer/director named "
|
||||
f"{company!r} rather than a company.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("--cik", help="Central Index Key (will be 10-digit zero-padded)")
|
||||
p.add_argument("--company", help="Resolve to CIK by company name")
|
||||
p.add_argument("--types", default="", help="Comma-separated form types (e.g. 10-K,10-Q,8-K)")
|
||||
p.add_argument("--since", help="Skip filings before YYYY-MM-DD")
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
types = [t for t in (a.types or "").split(",") if t.strip()]
|
||||
n = fetch(cik=a.cik, company=a.company, types=types, since=a.since, out_path=a.out)
|
||||
print(f"Wrote {n} EDGAR filing rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch Senate Lobbying Disclosure (LD-1 / LD-2) filings.
|
||||
|
||||
Anonymous: 120 req/hour. Token (SENATE_LDA_TOKEN): 1200 req/hour.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get_json # noqa: E402
|
||||
|
||||
ENDPOINT = "https://lda.senate.gov/api/v1/filings/"
|
||||
COLUMNS = [
|
||||
"filing_uuid",
|
||||
"filing_type",
|
||||
"filing_year",
|
||||
"filing_period",
|
||||
"registrant_name",
|
||||
"registrant_id",
|
||||
"client_name",
|
||||
"client_id",
|
||||
"client_general_description",
|
||||
"income",
|
||||
"expenses",
|
||||
"lobbyists",
|
||||
"issues",
|
||||
"government_entities",
|
||||
"filing_date",
|
||||
]
|
||||
|
||||
|
||||
def fetch(
|
||||
client: str | None,
|
||||
registrant: str | None,
|
||||
year: int,
|
||||
token: str | None,
|
||||
out_path: str,
|
||||
page_size: int = 100,
|
||||
max_pages: int = 25,
|
||||
) -> int:
|
||||
params: dict = {"filing_year": year, "page_size": page_size}
|
||||
if client:
|
||||
params["client_name"] = client
|
||||
if registrant:
|
||||
params["registrant_name"] = registrant
|
||||
|
||||
headers = {"Authorization": f"Token {token}"} if token else None
|
||||
rows: list[dict[str, str]] = []
|
||||
url = ENDPOINT
|
||||
page = 0
|
||||
while page < max_pages:
|
||||
try:
|
||||
payload = get_json(url, params=params if page == 0 else None, headers=headers)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"Senate LDA error on page {page + 1}: {e}", file=sys.stderr)
|
||||
break
|
||||
if not isinstance(payload, dict):
|
||||
break
|
||||
results = payload.get("results", [])
|
||||
for r in results:
|
||||
client_obj = r.get("client") or {}
|
||||
registrant_obj = r.get("registrant") or {}
|
||||
lobbying_activities = r.get("lobbying_activities") or []
|
||||
lobbyists = []
|
||||
issues = []
|
||||
entities = []
|
||||
for la in lobbying_activities:
|
||||
for lob in la.get("lobbyists") or []:
|
||||
lob_obj = lob.get("lobbyist") or {}
|
||||
name = " ".join(
|
||||
x for x in (lob_obj.get("first_name", ""), lob_obj.get("last_name", "")) if x
|
||||
)
|
||||
if name:
|
||||
lobbyists.append(name)
|
||||
desc = la.get("description") or ""
|
||||
if desc:
|
||||
issues.append(desc)
|
||||
for ge in la.get("government_entities") or []:
|
||||
nm = ge.get("name") or ""
|
||||
if nm:
|
||||
entities.append(nm)
|
||||
rows.append(
|
||||
{
|
||||
"filing_uuid": r.get("filing_uuid", "") or "",
|
||||
"filing_type": r.get("filing_type", "") or "",
|
||||
"filing_year": str(r.get("filing_year", "") or year),
|
||||
"filing_period": r.get("filing_period", "") or "",
|
||||
"registrant_name": registrant_obj.get("name", "") or "",
|
||||
"registrant_id": str(registrant_obj.get("id", "") or ""),
|
||||
"client_name": client_obj.get("name", "") or "",
|
||||
"client_id": str(client_obj.get("id", "") or ""),
|
||||
"client_general_description": client_obj.get("general_description", "") or "",
|
||||
"income": str(r.get("income", "") or ""),
|
||||
"expenses": str(r.get("expenses", "") or ""),
|
||||
"lobbyists": "; ".join(sorted(set(lobbyists))),
|
||||
"issues": "; ".join(issues),
|
||||
"government_entities": "; ".join(sorted(set(entities))),
|
||||
"filing_date": (r.get("dt_posted") or "")[:10],
|
||||
}
|
||||
)
|
||||
next_url = payload.get("next")
|
||||
if not next_url:
|
||||
break
|
||||
url = next_url
|
||||
page += 1
|
||||
time.sleep(1.0 if not token else 0.3)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("--client", help="Client name filter")
|
||||
p.add_argument("--registrant", help="Registrant (lobbying firm) name filter")
|
||||
p.add_argument("--year", type=int, default=2024)
|
||||
p.add_argument("--token", default=os.environ.get("SENATE_LDA_TOKEN"))
|
||||
p.add_argument("--max-pages", type=int, default=25)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
if not (a.client or a.registrant):
|
||||
p.error("must supply at least one of --client / --registrant")
|
||||
n = fetch(
|
||||
client=a.client,
|
||||
registrant=a.registrant,
|
||||
year=a.year,
|
||||
token=a.token,
|
||||
out_path=a.out,
|
||||
max_pages=a.max_pages,
|
||||
)
|
||||
print(f"Wrote {n} Senate LDA rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,170 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch federal contracts/awards from USAspending.gov API v2.
|
||||
|
||||
No auth required. POST to /api/v2/search/spending_by_award/ with filters.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ENDPOINT = "https://api.usaspending.gov/api/v2/search/spending_by_award/"
|
||||
COLUMNS = [
|
||||
"award_id",
|
||||
"recipient_name",
|
||||
"recipient_uei",
|
||||
"recipient_duns",
|
||||
"recipient_parent_name",
|
||||
"recipient_state",
|
||||
"awarding_agency",
|
||||
"awarding_sub_agency",
|
||||
"award_type",
|
||||
"award_amount",
|
||||
"award_date",
|
||||
"period_of_performance_start",
|
||||
"period_of_performance_end",
|
||||
"naics_code",
|
||||
"psc_code",
|
||||
"competition_extent",
|
||||
"description",
|
||||
]
|
||||
|
||||
# USAspending result column "code" → human label mapping for output.
|
||||
_FIELDS = [
|
||||
"Award ID",
|
||||
"Recipient Name",
|
||||
"Recipient UEI",
|
||||
"Recipient DUNS Number",
|
||||
"Recipient Parent Name",
|
||||
"Recipient State Code",
|
||||
"Awarding Agency",
|
||||
"Awarding Sub Agency",
|
||||
"Award Type",
|
||||
"Award Amount",
|
||||
"Start Date",
|
||||
"End Date",
|
||||
"NAICS Code",
|
||||
"PSC Code",
|
||||
"Type of Set Aside",
|
||||
"Description",
|
||||
]
|
||||
|
||||
|
||||
def _post(body: dict) -> dict:
|
||||
req = urllib.request.Request(
|
||||
ENDPOINT,
|
||||
data=json.dumps(body).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json", "User-Agent": "hermes-agent osint-investigation"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=60) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
|
||||
|
||||
def fetch(
|
||||
recipient: str | None,
|
||||
agency: str | None,
|
||||
fy: int,
|
||||
sole_source_only: bool,
|
||||
out_path: str,
|
||||
page_size: int = 100,
|
||||
max_pages: int = 20,
|
||||
) -> int:
|
||||
filters: dict = {
|
||||
"time_period": [{"start_date": f"{fy - 1}-10-01", "end_date": f"{fy}-09-30"}],
|
||||
# Contracts only by default; adjust award_type_codes for grants/loans.
|
||||
"award_type_codes": ["A", "B", "C", "D"],
|
||||
}
|
||||
if recipient:
|
||||
filters["recipient_search_text"] = [recipient]
|
||||
if agency:
|
||||
filters["agencies"] = [{"type": "awarding", "tier": "toptier", "name": agency}]
|
||||
|
||||
rows: list[dict[str, str]] = []
|
||||
page = 1
|
||||
while page <= max_pages:
|
||||
body = {
|
||||
"filters": filters,
|
||||
"fields": _FIELDS,
|
||||
"page": page,
|
||||
"limit": page_size,
|
||||
"sort": "Award Amount",
|
||||
"order": "desc",
|
||||
}
|
||||
try:
|
||||
payload = _post(body)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"USAspending error on page {page}: {e}", file=sys.stderr)
|
||||
break
|
||||
results = payload.get("results", [])
|
||||
if not results:
|
||||
break
|
||||
for r in results:
|
||||
set_aside = r.get("Type of Set Aside", "") or ""
|
||||
if sole_source_only and "sole" not in set_aside.lower():
|
||||
continue
|
||||
rows.append(
|
||||
{
|
||||
"award_id": r.get("Award ID", "") or "",
|
||||
"recipient_name": r.get("Recipient Name", "") or "",
|
||||
"recipient_uei": r.get("Recipient UEI", "") or "",
|
||||
"recipient_duns": r.get("Recipient DUNS Number", "") or "",
|
||||
"recipient_parent_name": r.get("Recipient Parent Name", "") or "",
|
||||
"recipient_state": r.get("Recipient State Code", "") or "",
|
||||
"awarding_agency": r.get("Awarding Agency", "") or "",
|
||||
"awarding_sub_agency": r.get("Awarding Sub Agency", "") or "",
|
||||
"award_type": r.get("Award Type", "") or "",
|
||||
"award_amount": str(r.get("Award Amount", "") or ""),
|
||||
"award_date": r.get("Start Date", "") or "",
|
||||
"period_of_performance_start": r.get("Start Date", "") or "",
|
||||
"period_of_performance_end": r.get("End Date", "") or "",
|
||||
"naics_code": str(r.get("NAICS Code", "") or ""),
|
||||
"psc_code": str(r.get("PSC Code", "") or ""),
|
||||
"competition_extent": set_aside,
|
||||
"description": r.get("Description", "") or "",
|
||||
}
|
||||
)
|
||||
meta = payload.get("page_metadata", {})
|
||||
if not meta.get("hasNext"):
|
||||
break
|
||||
page += 1
|
||||
time.sleep(0.5)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("--recipient", help="Recipient name search")
|
||||
p.add_argument("--agency", help="Awarding agency (top-tier)")
|
||||
p.add_argument("--fy", type=int, default=2024, help="Federal fiscal year")
|
||||
p.add_argument("--sole-source-only", action="store_true")
|
||||
p.add_argument("--max-pages", type=int, default=20)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
if not (a.recipient or a.agency):
|
||||
p.error("must supply at least one of --recipient / --agency")
|
||||
n = fetch(
|
||||
recipient=a.recipient,
|
||||
agency=a.agency,
|
||||
fy=a.fy,
|
||||
sole_source_only=a.sole_source_only,
|
||||
out_path=a.out,
|
||||
max_pages=a.max_pages,
|
||||
)
|
||||
print(f"Wrote {n} USAspending rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,142 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search the Internet Archive Wayback Machine via the CDX server.
|
||||
|
||||
The CDX API indexes ~900B+ archived web pages. Anonymous read access,
|
||||
no auth required. Useful for finding deleted / changed pages by URL,
|
||||
domain, or substring match.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get_json # noqa: E402
|
||||
|
||||
BASE = "https://web.archive.org/cdx/search/cdx"
|
||||
|
||||
COLUMNS = [
|
||||
"url",
|
||||
"timestamp",
|
||||
"wayback_url",
|
||||
"mimetype",
|
||||
"status",
|
||||
"digest",
|
||||
"length",
|
||||
]
|
||||
|
||||
|
||||
def fetch(
|
||||
url_or_host: str,
|
||||
match_type: str,
|
||||
from_date: str | None,
|
||||
to_date: str | None,
|
||||
status: str | None,
|
||||
mime: str | None,
|
||||
collapse: str | None,
|
||||
limit: int,
|
||||
out_path: str,
|
||||
) -> int:
|
||||
params: dict[str, str] = {
|
||||
"url": url_or_host,
|
||||
"matchType": match_type,
|
||||
"output": "json",
|
||||
"limit": str(limit),
|
||||
}
|
||||
if from_date:
|
||||
params["from"] = from_date.replace("-", "")
|
||||
if to_date:
|
||||
params["to"] = to_date.replace("-", "")
|
||||
if status:
|
||||
params["filter"] = f"statuscode:{status}"
|
||||
if mime:
|
||||
params.setdefault("filter", "")
|
||||
# Multiple filters: CDX accepts repeated filter params via urlencode list
|
||||
params["filter"] = f"mimetype:{mime}"
|
||||
if collapse:
|
||||
params["collapse"] = collapse
|
||||
|
||||
url = f"{BASE}?{urllib.parse.urlencode(params)}"
|
||||
try:
|
||||
payload = get_json(url)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"Wayback CDX error: {e}", file=sys.stderr)
|
||||
payload = []
|
||||
|
||||
rows: list[dict[str, str]] = []
|
||||
if isinstance(payload, list) and len(payload) > 1:
|
||||
header = payload[0]
|
||||
idx = {h: i for i, h in enumerate(header)}
|
||||
for entry in payload[1:]:
|
||||
ts = entry[idx["timestamp"]] if "timestamp" in idx else ""
|
||||
orig = entry[idx["original"]] if "original" in idx else ""
|
||||
rows.append(
|
||||
{
|
||||
"url": orig,
|
||||
"timestamp": ts,
|
||||
"wayback_url": f"https://web.archive.org/web/{ts}/{orig}" if ts and orig else "",
|
||||
"mimetype": entry[idx["mimetype"]] if "mimetype" in idx else "",
|
||||
"status": entry[idx["statuscode"]] if "statuscode" in idx else "",
|
||||
"digest": entry[idx["digest"]] if "digest" in idx else "",
|
||||
"length": entry[idx["length"]] if "length" in idx else "",
|
||||
}
|
||||
)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
if not rows:
|
||||
print(
|
||||
f"Wayback Machine: 0 captures for {url_or_host!r} matchType={match_type}.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--url", required=True, help="URL or host to look up in the archive")
|
||||
p.add_argument(
|
||||
"--match",
|
||||
default="exact",
|
||||
choices=["exact", "prefix", "host", "domain"],
|
||||
help=(
|
||||
"exact: this URL only. "
|
||||
"prefix: this URL's path-prefix. "
|
||||
"host: any URL on this host. "
|
||||
"domain: any URL on this domain or subdomains."
|
||||
),
|
||||
)
|
||||
p.add_argument("--from-date", help="Earliest capture YYYY-MM-DD")
|
||||
p.add_argument("--to-date", help="Latest capture YYYY-MM-DD")
|
||||
p.add_argument("--status", help="HTTP status filter (e.g. 200)")
|
||||
p.add_argument("--mime", help="MIME type filter (e.g. text/html)")
|
||||
p.add_argument(
|
||||
"--collapse",
|
||||
help="Collapse adjacent identical entries (e.g. 'digest' for unique-content captures)",
|
||||
)
|
||||
p.add_argument("--limit", type=int, default=200)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
n = fetch(
|
||||
url_or_host=a.url,
|
||||
match_type=a.match,
|
||||
from_date=a.from_date,
|
||||
to_date=a.to_date,
|
||||
status=a.status,
|
||||
mime=a.mime,
|
||||
collapse=a.collapse,
|
||||
limit=a.limit,
|
||||
out_path=a.out,
|
||||
)
|
||||
print(f"Wrote {n} Wayback capture rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search Wikipedia + Wikidata for an entity (person, company, place, concept).
|
||||
|
||||
Two free APIs:
|
||||
- Wikipedia OpenSearch + REST summary endpoint for narrative bio
|
||||
- Wikidata SPARQL endpoint for structured facts (birth, employer, awards, etc.)
|
||||
|
||||
Both are anonymous-access. Useful for resolving who-is-this-entity questions
|
||||
and surfacing cross-references that other sources can join against.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import re
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from _http import get_json # noqa: E402
|
||||
|
||||
WP_OPENSEARCH = "https://en.wikipedia.org/w/api.php"
|
||||
WP_SUMMARY = "https://en.wikipedia.org/api/rest_v1/page/summary/"
|
||||
WD_ACTION = "https://www.wikidata.org/w/api.php"
|
||||
|
||||
COLUMNS = [
|
||||
"source",
|
||||
"label",
|
||||
"description",
|
||||
"qid",
|
||||
"wikipedia_title",
|
||||
"wikipedia_url",
|
||||
"wikidata_url",
|
||||
"instance_of",
|
||||
"country",
|
||||
"occupation",
|
||||
"employer",
|
||||
"date_of_birth",
|
||||
"place_of_birth",
|
||||
"summary",
|
||||
]
|
||||
|
||||
|
||||
def _wp_search(query: str, limit: int) -> list[dict]:
|
||||
params = {
|
||||
"action": "opensearch",
|
||||
"search": query,
|
||||
"limit": str(min(limit, 20)),
|
||||
"format": "json",
|
||||
}
|
||||
url = f"{WP_OPENSEARCH}?{urllib.parse.urlencode(params)}"
|
||||
data = get_json(url)
|
||||
if not isinstance(data, list) or len(data) < 4:
|
||||
return []
|
||||
titles, descs, urls = data[1], data[2], data[3]
|
||||
out = []
|
||||
for i, title in enumerate(titles):
|
||||
out.append(
|
||||
{
|
||||
"title": title,
|
||||
"description": descs[i] if i < len(descs) else "",
|
||||
"url": urls[i] if i < len(urls) else "",
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def _wp_summary(title: str) -> dict:
|
||||
"""Pull the REST summary for a title — short bio, image, type."""
|
||||
url = f"{WP_SUMMARY}{urllib.parse.quote(title.replace(' ', '_'))}"
|
||||
try:
|
||||
return get_json(url) # type: ignore[return-value]
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"Wikipedia summary lookup for {title!r} failed: {e}", file=sys.stderr)
|
||||
return {}
|
||||
|
||||
|
||||
def _wd_lookup_by_qid(qid: str) -> dict:
|
||||
"""Pull common facts for a QID via Wikidata's Action API (no SPARQL).
|
||||
|
||||
The Action API is far more lenient on rate-limits than the SPARQL Query
|
||||
Service. We get claims as QIDs and then resolve labels in one batch call.
|
||||
"""
|
||||
# Properties of interest. The Action API returns claims as QIDs or
|
||||
# typed literals, so the slot mapping is local-only.
|
||||
interesting = {
|
||||
"P31": "instance_of",
|
||||
"P17": "country", # for orgs / places
|
||||
"P27": "country", # for individuals (country of citizenship)
|
||||
"P106": "occupation",
|
||||
"P108": "employer",
|
||||
"P569": "date_of_birth",
|
||||
"P19": "place_of_birth",
|
||||
}
|
||||
params = {
|
||||
"action": "wbgetentities",
|
||||
"ids": qid,
|
||||
"props": "claims",
|
||||
"format": "json",
|
||||
}
|
||||
url = f"{WD_ACTION}?{urllib.parse.urlencode(params)}"
|
||||
try:
|
||||
data = get_json(url)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"Wikidata wbgetentities for {qid} failed: {e}", file=sys.stderr)
|
||||
return {}
|
||||
if not isinstance(data, dict):
|
||||
return {}
|
||||
claims = (data.get("entities", {}).get(qid, {}) or {}).get("claims", {}) or {}
|
||||
|
||||
# Collect raw values (QIDs or literals) and remember which slot each
|
||||
# came from. Date literals come back as ISO strings; QIDs need a label
|
||||
# resolution pass.
|
||||
qid_to_slots: dict[str, list[str]] = {}
|
||||
facts: dict[str, list[str]] = {}
|
||||
for prop_id, slot in interesting.items():
|
||||
for claim in claims.get(prop_id, []) or []:
|
||||
v = (claim.get("mainsnak", {}) or {}).get("datavalue", {}) or {}
|
||||
vtype = v.get("type")
|
||||
value = v.get("value")
|
||||
if vtype == "wikibase-entityid" and isinstance(value, dict):
|
||||
vqid = value.get("id", "")
|
||||
if vqid:
|
||||
qid_to_slots.setdefault(vqid, [])
|
||||
if slot not in qid_to_slots[vqid]:
|
||||
qid_to_slots[vqid].append(slot)
|
||||
elif vtype == "time" and isinstance(value, dict):
|
||||
raw = value.get("time", "") or ""
|
||||
# +1955-10-28T00:00:00Z → 1955-10-28
|
||||
m = re.search(r"[+-]?(\d{4})-(\d{2})-(\d{2})", raw)
|
||||
if m:
|
||||
facts.setdefault(slot, []).append(
|
||||
f"{m.group(1)}-{m.group(2)}-{m.group(3)}"
|
||||
)
|
||||
elif vtype == "string":
|
||||
facts.setdefault(slot, []).append(str(value))
|
||||
|
||||
# Resolve labels for all referenced QIDs in one batch (up to 50 at a time).
|
||||
qids = list(qid_to_slots)
|
||||
for i in range(0, len(qids), 50):
|
||||
batch = qids[i : i + 50]
|
||||
params = {
|
||||
"action": "wbgetentities",
|
||||
"ids": "|".join(batch),
|
||||
"props": "labels",
|
||||
"languages": "en",
|
||||
"format": "json",
|
||||
}
|
||||
url = f"{WD_ACTION}?{urllib.parse.urlencode(params)}"
|
||||
try:
|
||||
data = get_json(url)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"Wikidata label batch failed: {e}", file=sys.stderr)
|
||||
continue
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
ents = data.get("entities", {}) or {}
|
||||
for vqid, ent in ents.items():
|
||||
label = (ent.get("labels", {}).get("en", {}) or {}).get("value", "") or vqid
|
||||
for slot in qid_to_slots.get(vqid, []):
|
||||
facts.setdefault(slot, []).append(label)
|
||||
|
||||
# Deduplicate per slot, preserving order.
|
||||
deduped: dict[str, list[str]] = {}
|
||||
for slot, vals in facts.items():
|
||||
seen = set()
|
||||
out = []
|
||||
for v in vals:
|
||||
if v in seen:
|
||||
continue
|
||||
seen.add(v)
|
||||
out.append(v)
|
||||
deduped[slot] = out
|
||||
return deduped
|
||||
|
||||
|
||||
def _wd_qid_for_title(title: str) -> str:
|
||||
"""Get the Wikidata QID associated with a Wikipedia article title."""
|
||||
params = {
|
||||
"action": "query",
|
||||
"format": "json",
|
||||
"prop": "pageprops",
|
||||
"ppprop": "wikibase_item",
|
||||
"titles": title,
|
||||
"redirects": 1,
|
||||
}
|
||||
url = f"{WP_OPENSEARCH}?{urllib.parse.urlencode(params)}"
|
||||
try:
|
||||
data = get_json(url)
|
||||
except Exception: # noqa: BLE001
|
||||
return ""
|
||||
if not isinstance(data, dict):
|
||||
return ""
|
||||
pages = data.get("query", {}).get("pages", {}) or {}
|
||||
for page in pages.values():
|
||||
qid = (page.get("pageprops") or {}).get("wikibase_item", "")
|
||||
if qid:
|
||||
return qid
|
||||
return ""
|
||||
|
||||
|
||||
def fetch(query: str, limit: int, no_wikidata: bool, out_path: str) -> int:
|
||||
hits = _wp_search(query, limit)
|
||||
rows: list[dict[str, str]] = []
|
||||
for hit in hits[:limit]:
|
||||
title = hit.get("title", "")
|
||||
if not title:
|
||||
continue
|
||||
summary = _wp_summary(title)
|
||||
qid = _wd_qid_for_title(title) if not no_wikidata else ""
|
||||
facts: dict = {}
|
||||
if qid:
|
||||
facts = _wd_lookup_by_qid(qid)
|
||||
rows.append(
|
||||
{
|
||||
"source": "wikipedia+wikidata" if qid else "wikipedia",
|
||||
"label": title,
|
||||
"description": (summary.get("description") or hit.get("description") or "").strip(),
|
||||
"qid": qid,
|
||||
"wikipedia_title": title,
|
||||
"wikipedia_url": hit.get("url", ""),
|
||||
"wikidata_url": f"https://www.wikidata.org/wiki/{qid}" if qid else "",
|
||||
"instance_of": "; ".join(facts.get("instance_of", [])),
|
||||
"country": "; ".join(facts.get("country", [])),
|
||||
"occupation": "; ".join(facts.get("occupation", [])),
|
||||
"employer": "; ".join(facts.get("employer", [])),
|
||||
"date_of_birth": "; ".join(facts.get("date_of_birth", []))[:10] if facts.get("date_of_birth") else "",
|
||||
"place_of_birth": "; ".join(facts.get("place_of_birth", [])),
|
||||
"summary": (summary.get("extract") or "").replace("\n", " ")[:1000],
|
||||
}
|
||||
)
|
||||
|
||||
Path(out_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out_path, "w", newline="", encoding="utf-8") as fh:
|
||||
w = csv.DictWriter(fh, fieldnames=COLUMNS)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
if not rows:
|
||||
print(
|
||||
f"Wikipedia: 0 articles for query={query!r}. "
|
||||
"Private individuals not notable enough for a Wikipedia article "
|
||||
"won't appear here (the bar is real).",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--query", required=True, help="Entity name (person, company, place, concept)")
|
||||
p.add_argument("--limit", type=int, default=5)
|
||||
p.add_argument(
|
||||
"--no-wikidata",
|
||||
action="store_true",
|
||||
help="Skip the Wikidata SPARQL enrichment (faster, less detail)",
|
||||
)
|
||||
p.add_argument("--out", required=True)
|
||||
a = p.parse_args()
|
||||
n = fetch(query=a.query, limit=a.limit, no_wikidata=a.no_wikidata, out_path=a.out)
|
||||
print(f"Wrote {n} Wikipedia/Wikidata rows to {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,252 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Permutation test for donation/contract timing correlation (stdlib-only).
|
||||
|
||||
For each (donor, vendor) pair, compute the mean number of days between each
|
||||
donation and the nearest contract award. Then shuffle contract award dates
|
||||
N times within the observation window and compute the same statistic. The
|
||||
one-tailed p-value is the fraction of permutations whose mean is <= the
|
||||
observed mean (smaller distance = tighter clustering).
|
||||
|
||||
Adapted from ShinMegamiBoson/OpenPlanter (MIT). Differences:
|
||||
- Pure stdlib (no pandas / numpy)
|
||||
- Domain-agnostic (no snow-vendor / CRITICAL-politician filter)
|
||||
- Configurable column names via flags
|
||||
- Optional --seed for reproducibility
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import datetime as dt
|
||||
import json
|
||||
import random
|
||||
import statistics
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
_DATE_FORMATS = ("%Y-%m-%d", "%m/%d/%Y", "%Y/%m/%d", "%m-%d-%Y", "%Y%m%d")
|
||||
|
||||
|
||||
def parse_date(raw: str) -> dt.date | None:
|
||||
if not raw:
|
||||
return None
|
||||
raw = raw.strip()
|
||||
for fmt in _DATE_FORMATS:
|
||||
try:
|
||||
return dt.datetime.strptime(raw, fmt).date()
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _read(path: str) -> list[dict[str, str]]:
|
||||
with open(path, newline="", encoding="utf-8") as fh:
|
||||
return list(csv.DictReader(fh))
|
||||
|
||||
|
||||
def _nearest_distance(donation_date: dt.date, awards: list[dt.date]) -> int:
|
||||
"""Absolute days to nearest award date."""
|
||||
return min(abs((donation_date - a).days) for a in awards)
|
||||
|
||||
|
||||
def _permute(
|
||||
awards_count: int,
|
||||
donations: list[dt.date],
|
||||
date_min: dt.date,
|
||||
date_max: dt.date,
|
||||
rng: random.Random,
|
||||
) -> float:
|
||||
"""One permutation: draw uniform random award dates, compute mean nearest-distance."""
|
||||
span_days = (date_max - date_min).days or 1
|
||||
rand_awards = [
|
||||
date_min + dt.timedelta(days=rng.randint(0, span_days))
|
||||
for _ in range(awards_count)
|
||||
]
|
||||
distances = [_nearest_distance(d, rand_awards) for d in donations]
|
||||
return statistics.mean(distances)
|
||||
|
||||
|
||||
def analyze(
|
||||
donations_path: str,
|
||||
donation_date_col: str,
|
||||
donation_amount_col: str,
|
||||
donation_donor_col: str,
|
||||
donation_recipient_col: str,
|
||||
contracts_path: str,
|
||||
contract_date_col: str,
|
||||
contract_vendor_col: str,
|
||||
cross_links_path: str | None,
|
||||
n_permutations: int = 1000,
|
||||
min_donations: int = 3,
|
||||
p_threshold: float = 0.05,
|
||||
seed: int | None = None,
|
||||
out_path: str = "timing.json",
|
||||
) -> dict:
|
||||
rng = random.Random(seed)
|
||||
|
||||
donations = _read(donations_path)
|
||||
contracts = _read(contracts_path)
|
||||
|
||||
# Allow optional join through cross_links — donor (left) ↔ vendor (right).
|
||||
# When present, donor strings get mapped to matched vendor names so the
|
||||
# vendor-date index lookup actually finds the contracts.
|
||||
matched_pairs: set[tuple[str, str]] | None = None
|
||||
donor_to_vendors: dict[str, set[str]] = defaultdict(set)
|
||||
if cross_links_path:
|
||||
matched_pairs = set()
|
||||
for row in _read(cross_links_path):
|
||||
left = row.get("left_name", "")
|
||||
right = row.get("right_name", "")
|
||||
matched_pairs.add((left, right))
|
||||
donor_to_vendors[left].add(right)
|
||||
|
||||
# Index contract dates by vendor name.
|
||||
vendor_to_award_dates: dict[str, list[dt.date]] = defaultdict(list)
|
||||
all_award_dates: list[dt.date] = []
|
||||
for row in contracts:
|
||||
d = parse_date(row.get(contract_date_col, ""))
|
||||
if not d:
|
||||
continue
|
||||
vendor_to_award_dates[row.get(contract_vendor_col, "").strip()].append(d)
|
||||
all_award_dates.append(d)
|
||||
|
||||
if not all_award_dates:
|
||||
raise SystemExit(f"No parseable dates in {contracts_path}/{contract_date_col}")
|
||||
global_min = min(all_award_dates)
|
||||
global_max = max(all_award_dates)
|
||||
|
||||
# Group donations by (donor, recipient).
|
||||
grouped: dict[tuple[str, str], list[tuple[dt.date, float]]] = defaultdict(list)
|
||||
for row in donations:
|
||||
donor = row.get(donation_donor_col, "").strip()
|
||||
recip = row.get(donation_recipient_col, "").strip()
|
||||
d = parse_date(row.get(donation_date_col, ""))
|
||||
try:
|
||||
amt = float(row.get(donation_amount_col, "0") or 0)
|
||||
except ValueError:
|
||||
amt = 0.0
|
||||
if not (donor and recip and d):
|
||||
continue
|
||||
grouped[(donor, recip)].append((d, amt))
|
||||
|
||||
results = []
|
||||
skipped = 0
|
||||
for (donor, recip), records in grouped.items():
|
||||
if len(records) < min_donations:
|
||||
skipped += 1
|
||||
continue
|
||||
# Only test if donor appears in cross-links (when provided). The
|
||||
# (donor, candidate) tuple itself is NOT what's in matched_pairs —
|
||||
# cross_links pairs are (donor, vendor). We use the cross-link to
|
||||
# map donor → vendor name(s) so the vendor-date index resolves.
|
||||
if matched_pairs is not None and donor not in donor_to_vendors:
|
||||
skipped += 1
|
||||
continue
|
||||
# Try direct donor→awards first, then go through cross-link vendor names.
|
||||
award_dates = list(vendor_to_award_dates.get(donor, []))
|
||||
if not award_dates:
|
||||
award_dates = list(vendor_to_award_dates.get(recip, []))
|
||||
if not award_dates and donor_to_vendors.get(donor):
|
||||
for vendor_name in donor_to_vendors[donor]:
|
||||
award_dates.extend(vendor_to_award_dates.get(vendor_name, []))
|
||||
if not award_dates:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
donation_dates = [d for (d, _) in records]
|
||||
observed = statistics.mean(
|
||||
_nearest_distance(d, award_dates) for d in donation_dates
|
||||
)
|
||||
|
||||
permuted_means = [
|
||||
_permute(len(award_dates), donation_dates, global_min, global_max, rng)
|
||||
for _ in range(n_permutations)
|
||||
]
|
||||
p_value = sum(1 for m in permuted_means if m <= observed) / n_permutations
|
||||
null_mean = statistics.mean(permuted_means)
|
||||
null_std = statistics.pstdev(permuted_means) or 1.0
|
||||
effect_size = (null_mean - observed) / null_std
|
||||
|
||||
results.append(
|
||||
{
|
||||
"donor": donor,
|
||||
"recipient": recip,
|
||||
"n_donations": len(records),
|
||||
"n_award_dates": len(award_dates),
|
||||
"observed_mean_days": round(observed, 2),
|
||||
"null_mean_days": round(null_mean, 2),
|
||||
"p_value": round(p_value, 4),
|
||||
"effect_size_sd": round(effect_size, 2),
|
||||
"significant": p_value < p_threshold,
|
||||
"total_donation_amount": round(sum(a for (_, a) in records), 2),
|
||||
}
|
||||
)
|
||||
|
||||
results.sort(key=lambda r: r["p_value"])
|
||||
|
||||
payload = {
|
||||
"metadata": {
|
||||
"n_permutations": n_permutations,
|
||||
"min_donations": min_donations,
|
||||
"p_threshold": p_threshold,
|
||||
"seed": seed,
|
||||
"n_pairs_tested": len(results),
|
||||
"n_pairs_skipped": skipped,
|
||||
"n_significant": sum(1 for r in results if r["significant"]),
|
||||
"observation_window": [global_min.isoformat(), global_max.isoformat()],
|
||||
},
|
||||
"results": results,
|
||||
}
|
||||
|
||||
Path(out_path).write_text(json.dumps(payload, indent=2))
|
||||
return payload
|
||||
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--donations", required=True)
|
||||
p.add_argument("--donation-date-col", required=True)
|
||||
p.add_argument("--donation-amount-col", required=True)
|
||||
p.add_argument("--donation-donor-col", required=True)
|
||||
p.add_argument("--donation-recipient-col", required=True)
|
||||
p.add_argument("--contracts", required=True)
|
||||
p.add_argument("--contract-date-col", required=True)
|
||||
p.add_argument("--contract-vendor-col", required=True)
|
||||
p.add_argument(
|
||||
"--cross-links",
|
||||
help="Optional cross_links.csv to restrict (donor, vendor) pairs",
|
||||
)
|
||||
p.add_argument("--permutations", type=int, default=1000)
|
||||
p.add_argument("--min-donations", type=int, default=3)
|
||||
p.add_argument("--p-threshold", type=float, default=0.05)
|
||||
p.add_argument("--seed", type=int)
|
||||
p.add_argument("--out", default="timing.json")
|
||||
a = p.parse_args()
|
||||
|
||||
payload = analyze(
|
||||
donations_path=a.donations,
|
||||
donation_date_col=a.donation_date_col,
|
||||
donation_amount_col=a.donation_amount_col,
|
||||
donation_donor_col=a.donation_donor_col,
|
||||
donation_recipient_col=a.donation_recipient_col,
|
||||
contracts_path=a.contracts,
|
||||
contract_date_col=a.contract_date_col,
|
||||
contract_vendor_col=a.contract_vendor_col,
|
||||
cross_links_path=a.cross_links,
|
||||
n_permutations=a.permutations,
|
||||
min_donations=a.min_donations,
|
||||
p_threshold=a.p_threshold,
|
||||
seed=a.seed,
|
||||
out_path=a.out,
|
||||
)
|
||||
meta = payload["metadata"]
|
||||
print(
|
||||
f"Tested {meta['n_pairs_tested']} pairs ({meta['n_pairs_skipped']} skipped). "
|
||||
f"Significant (p<{meta['p_threshold']}): {meta['n_significant']}. "
|
||||
f"Wrote {a.out}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,59 @@
|
||||
# <Source Name>
|
||||
|
||||
## 1. Summary
|
||||
|
||||
What this data source is, who publishes it, why it matters for investigations.
|
||||
|
||||
## 2. Access Methods
|
||||
|
||||
- API endpoint(s)
|
||||
- Bulk download URLs
|
||||
- Auth requirements (none / API key / OAuth)
|
||||
- Rate limits
|
||||
|
||||
## 3. Data Schema
|
||||
|
||||
Key fields, record types, table relationships. List the columns the fetch
|
||||
script emits.
|
||||
|
||||
## 4. Coverage
|
||||
|
||||
- Jurisdiction
|
||||
- Time range
|
||||
- Update frequency
|
||||
- Data volume (rows / GB)
|
||||
|
||||
## 5. Cross-Reference Potential
|
||||
|
||||
Which other sources can be joined and on what keys. Be explicit:
|
||||
|
||||
- `<source>` ↔ `<column>` (join key: <normalized entity name / EIN / CIK / etc.>)
|
||||
|
||||
## 6. Data Quality
|
||||
|
||||
Known issues — formatting inconsistencies, missing fields, duplicates,
|
||||
historical gaps, redaction.
|
||||
|
||||
## 7. Acquisition Script
|
||||
|
||||
Path: `scripts/fetch_<source>.py`
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
python3 SKILL_DIR/scripts/fetch_<source>.py --<filter> <value> --out data/<source>.csv
|
||||
```
|
||||
|
||||
Output CSV columns: `<col1>, <col2>, ...`
|
||||
|
||||
## 8. Legal & Licensing
|
||||
|
||||
- Public records law / FOIA basis
|
||||
- Terms of use / acceptable use
|
||||
- Attribution requirements (if any)
|
||||
|
||||
## 9. References
|
||||
|
||||
- Official docs: <url>
|
||||
- Data dictionary: <url>
|
||||
- Related coverage / journalism: <url>
|
||||
@@ -0,0 +1,391 @@
|
||||
---
|
||||
name: parallel-cli
|
||||
description: Optional vendor skill for Parallel CLI — agent-native web search, extraction, deep research, enrichment, FindAll, and monitoring. Prefer JSON output and non-interactive flows.
|
||||
version: 1.1.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [Research, Web, Search, Deep-Research, Enrichment, CLI]
|
||||
related_skills: [duckduckgo-search, mcporter]
|
||||
---
|
||||
|
||||
# Parallel CLI
|
||||
|
||||
Use `parallel-cli` when the user explicitly wants Parallel, or when a terminal-native workflow would benefit from Parallel's vendor-specific stack for web search, extraction, deep research, enrichment, entity discovery, or monitoring.
|
||||
|
||||
This is an optional third-party workflow, not a Hermes core capability.
|
||||
|
||||
Important expectations:
|
||||
- Parallel is a paid service with a free tier, not a fully free local tool.
|
||||
- It overlaps with Hermes native `web_search` / `web_extract`, so do not prefer it by default for ordinary lookups.
|
||||
- Prefer this skill when the user mentions Parallel specifically or needs capabilities like Parallel's enrichment, FindAll, or monitor workflows.
|
||||
|
||||
`parallel-cli` is designed for agents:
|
||||
- JSON output via `--json`
|
||||
- Non-interactive command execution
|
||||
- Async long-running jobs with `--no-wait`, `status`, and `poll`
|
||||
- Context chaining with `--previous-interaction-id`
|
||||
- Search, extract, research, enrichment, entity discovery, and monitoring in one CLI
|
||||
|
||||
## When to use it
|
||||
|
||||
Prefer this skill when:
|
||||
- The user explicitly mentions Parallel or `parallel-cli`
|
||||
- The task needs richer workflows than a simple one-shot search/extract pass
|
||||
- You need async deep research jobs that can be launched and polled later
|
||||
- You need structured enrichment, FindAll entity discovery, or monitoring
|
||||
|
||||
Prefer Hermes native `web_search` / `web_extract` for quick one-off lookups when Parallel is not specifically requested.
|
||||
|
||||
## Installation
|
||||
|
||||
Try the least invasive install path available for the environment.
|
||||
|
||||
### Homebrew
|
||||
|
||||
```bash
|
||||
brew install parallel-web/tap/parallel-cli
|
||||
```
|
||||
|
||||
### npm
|
||||
|
||||
```bash
|
||||
npm install -g parallel-web-cli
|
||||
```
|
||||
|
||||
### Python package
|
||||
|
||||
```bash
|
||||
pip install "parallel-web-tools[cli]"
|
||||
```
|
||||
|
||||
### Standalone installer
|
||||
|
||||
```bash
|
||||
curl -fsSL https://parallel.ai/install.sh | bash
|
||||
```
|
||||
|
||||
If you want an isolated Python install, `pipx` can also work:
|
||||
|
||||
```bash
|
||||
pipx install "parallel-web-tools[cli]"
|
||||
pipx ensurepath
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
Interactive login:
|
||||
|
||||
```bash
|
||||
parallel-cli login
|
||||
```
|
||||
|
||||
Headless / SSH / CI:
|
||||
|
||||
```bash
|
||||
parallel-cli login --device
|
||||
```
|
||||
|
||||
API key environment variable:
|
||||
|
||||
```bash
|
||||
export PARALLEL_API_KEY="***"
|
||||
```
|
||||
|
||||
Verify current auth status:
|
||||
|
||||
```bash
|
||||
parallel-cli auth
|
||||
```
|
||||
|
||||
If auth requires browser interaction, run with `pty=true`.
|
||||
|
||||
## Core rule set
|
||||
|
||||
1. Always prefer `--json` when you need machine-readable output.
|
||||
2. Prefer explicit arguments and non-interactive flows.
|
||||
3. For long-running jobs, use `--no-wait` and then `status` / `poll`.
|
||||
4. Cite only URLs returned by the CLI output.
|
||||
5. Save large JSON outputs to a temp file when follow-up questions are likely.
|
||||
6. Use background processes only for genuinely long-running workflows; otherwise run in foreground.
|
||||
7. Prefer Hermes native tools unless the user wants Parallel specifically or needs Parallel-only workflows.
|
||||
|
||||
## Quick reference
|
||||
|
||||
```text
|
||||
parallel-cli
|
||||
├── auth
|
||||
├── login
|
||||
├── logout
|
||||
├── search
|
||||
├── extract / fetch
|
||||
├── research run|status|poll|processors
|
||||
├── enrich run|status|poll|plan|suggest|deploy
|
||||
├── findall run|ingest|status|poll|result|enrich|extend|schema|cancel
|
||||
└── monitor create|list|get|update|delete|events|event-group|simulate
|
||||
```
|
||||
|
||||
## Common flags and patterns
|
||||
|
||||
Commonly useful flags:
|
||||
- `--json` for structured output
|
||||
- `--no-wait` for async jobs
|
||||
- `--previous-interaction-id <id>` for follow-up tasks that reuse earlier context
|
||||
- `--max-results <n>` for search result count
|
||||
- `--mode one-shot|agentic` for search behavior
|
||||
- `--include-domains domain1.com,domain2.com`
|
||||
- `--exclude-domains domain1.com,domain2.com`
|
||||
- `--after-date YYYY-MM-DD`
|
||||
|
||||
Read from stdin when convenient:
|
||||
|
||||
```bash
|
||||
echo "What is the latest funding for Anthropic?" | parallel-cli search - --json
|
||||
echo "Research question" | parallel-cli research run - --json
|
||||
```
|
||||
|
||||
## Search
|
||||
|
||||
Use for current web lookups with structured results.
|
||||
|
||||
```bash
|
||||
parallel-cli search "What is Anthropic's latest AI model?" --json
|
||||
parallel-cli search "SEC filings for Apple" --include-domains sec.gov --json
|
||||
parallel-cli search "bitcoin price" --after-date 2026-01-01 --max-results 10 --json
|
||||
parallel-cli search "latest browser benchmarks" --mode one-shot --json
|
||||
parallel-cli search "AI coding agent enterprise reviews" --mode agentic --json
|
||||
```
|
||||
|
||||
Useful constraints:
|
||||
- `--include-domains` to narrow trusted sources
|
||||
- `--exclude-domains` to strip noisy domains
|
||||
- `--after-date` for recency filtering
|
||||
- `--max-results` when you need broader coverage
|
||||
|
||||
If you expect follow-up questions, save output:
|
||||
|
||||
```bash
|
||||
parallel-cli search "latest React 19 changes" --json -o /tmp/react-19-search.json
|
||||
```
|
||||
|
||||
When summarizing results:
|
||||
- lead with the answer
|
||||
- include dates, names, and concrete facts
|
||||
- cite only returned sources
|
||||
- avoid inventing URLs or source titles
|
||||
|
||||
## Extraction
|
||||
|
||||
Use to pull clean content or markdown from a URL.
|
||||
|
||||
```bash
|
||||
parallel-cli extract https://example.com --json
|
||||
parallel-cli extract https://company.com --objective "Find pricing info" --json
|
||||
parallel-cli extract https://example.com --full-content --json
|
||||
parallel-cli fetch https://example.com --json
|
||||
```
|
||||
|
||||
Use `--objective` when the page is broad and you only need one slice of information.
|
||||
|
||||
## Deep research
|
||||
|
||||
Use for deeper multi-step research tasks that may take time.
|
||||
|
||||
Common processor tiers:
|
||||
- `lite` / `base` for faster, cheaper passes
|
||||
- `core` / `pro` for more thorough synthesis
|
||||
- `ultra` for the heaviest research jobs
|
||||
|
||||
### Synchronous
|
||||
|
||||
```bash
|
||||
parallel-cli research run \
|
||||
"Compare the leading AI coding agents by pricing, model support, and enterprise controls" \
|
||||
--processor core \
|
||||
--json
|
||||
```
|
||||
|
||||
### Async launch + poll
|
||||
|
||||
```bash
|
||||
parallel-cli research run \
|
||||
"Compare the leading AI coding agents by pricing, model support, and enterprise controls" \
|
||||
--processor ultra \
|
||||
--no-wait \
|
||||
--json
|
||||
|
||||
parallel-cli research status trun_xxx --json
|
||||
parallel-cli research poll trun_xxx --json
|
||||
parallel-cli research processors --json
|
||||
```
|
||||
|
||||
### Context chaining / follow-up
|
||||
|
||||
```bash
|
||||
parallel-cli research run "What are the top AI coding agents?" --json
|
||||
parallel-cli research run \
|
||||
"What enterprise controls does the top-ranked one offer?" \
|
||||
--previous-interaction-id trun_xxx \
|
||||
--json
|
||||
```
|
||||
|
||||
Recommended Hermes workflow:
|
||||
1. launch with `--no-wait --json`
|
||||
2. capture the returned run/task ID
|
||||
3. if the user wants to continue other work, keep moving
|
||||
4. later call `status` or `poll`
|
||||
5. summarize the final report with citations from the returned sources
|
||||
|
||||
## Enrichment
|
||||
|
||||
Use when the user has CSV/JSON/tabular inputs and wants additional columns inferred from web research.
|
||||
|
||||
### Suggest columns
|
||||
|
||||
```bash
|
||||
parallel-cli enrich suggest "Find the CEO and annual revenue" --json
|
||||
```
|
||||
|
||||
### Plan a config
|
||||
|
||||
```bash
|
||||
parallel-cli enrich plan -o config.yaml
|
||||
```
|
||||
|
||||
### Inline data
|
||||
|
||||
```bash
|
||||
parallel-cli enrich run \
|
||||
--data '[{"company": "Anthropic"}, {"company": "Mistral"}]' \
|
||||
--intent "Find headquarters and employee count" \
|
||||
--json
|
||||
```
|
||||
|
||||
### Non-interactive file run
|
||||
|
||||
```bash
|
||||
parallel-cli enrich run \
|
||||
--source-type csv \
|
||||
--source companies.csv \
|
||||
--target enriched.csv \
|
||||
--source-columns '[{"name": "company", "description": "Company name"}]' \
|
||||
--intent "Find the CEO and annual revenue"
|
||||
```
|
||||
|
||||
### YAML config run
|
||||
|
||||
```bash
|
||||
parallel-cli enrich run config.yaml
|
||||
```
|
||||
|
||||
### Status / polling
|
||||
|
||||
```bash
|
||||
parallel-cli enrich status <task_group_id> --json
|
||||
parallel-cli enrich poll <task_group_id> --json
|
||||
```
|
||||
|
||||
Use explicit JSON arrays for column definitions when operating non-interactively.
|
||||
Validate the output file before reporting success.
|
||||
|
||||
## FindAll
|
||||
|
||||
Use for web-scale entity discovery when the user wants a discovered dataset rather than a short answer.
|
||||
|
||||
```bash
|
||||
parallel-cli findall run "Find AI coding agent startups with enterprise offerings" --json
|
||||
parallel-cli findall run "AI startups in healthcare" -n 25 --json
|
||||
parallel-cli findall status <run_id> --json
|
||||
parallel-cli findall poll <run_id> --json
|
||||
parallel-cli findall result <run_id> --json
|
||||
parallel-cli findall schema <run_id> --json
|
||||
```
|
||||
|
||||
This is a better fit than ordinary search when the user wants a discovered set of entities that can be reviewed, filtered, or enriched later.
|
||||
|
||||
## Monitor
|
||||
|
||||
Use for ongoing change detection over time.
|
||||
|
||||
```bash
|
||||
parallel-cli monitor list --json
|
||||
parallel-cli monitor get <monitor_id> --json
|
||||
parallel-cli monitor events <monitor_id> --json
|
||||
parallel-cli monitor delete <monitor_id> --json
|
||||
```
|
||||
|
||||
Creation is usually the sensitive part because cadence and delivery matter:
|
||||
|
||||
```bash
|
||||
parallel-cli monitor create --help
|
||||
```
|
||||
|
||||
Use this when the user wants recurring tracking of a page or source rather than a one-time fetch.
|
||||
|
||||
## Recommended Hermes usage patterns
|
||||
|
||||
### Fast answer with citations
|
||||
1. Run `parallel-cli search ... --json`
|
||||
2. Parse titles, URLs, dates, excerpts
|
||||
3. Summarize with inline citations from the returned URLs only
|
||||
|
||||
### URL investigation
|
||||
1. Run `parallel-cli extract URL --json`
|
||||
2. If needed, rerun with `--objective` or `--full-content`
|
||||
3. Quote or summarize the extracted markdown
|
||||
|
||||
### Long research workflow
|
||||
1. Run `parallel-cli research run ... --no-wait --json`
|
||||
2. Store the returned ID
|
||||
3. Continue other work or periodically poll
|
||||
4. Summarize the final report with citations
|
||||
|
||||
### Structured enrichment workflow
|
||||
1. Inspect the input file and columns
|
||||
2. Use `enrich suggest` or provide explicit enriched columns
|
||||
3. Run `enrich run`
|
||||
4. Poll for completion if needed
|
||||
5. Validate the output file before reporting success
|
||||
|
||||
## Error handling and exit codes
|
||||
|
||||
The CLI documents these exit codes:
|
||||
- `0` success
|
||||
- `2` bad input
|
||||
- `3` auth error
|
||||
- `4` API error
|
||||
- `5` timeout
|
||||
|
||||
If you hit auth errors:
|
||||
1. check `parallel-cli auth`
|
||||
2. confirm `PARALLEL_API_KEY` or run `parallel-cli login` / `parallel-cli login --device`
|
||||
3. verify `parallel-cli` is on `PATH`
|
||||
|
||||
## Maintenance
|
||||
|
||||
Check current auth / install state:
|
||||
|
||||
```bash
|
||||
parallel-cli auth
|
||||
parallel-cli --help
|
||||
```
|
||||
|
||||
Update commands:
|
||||
|
||||
```bash
|
||||
parallel-cli update
|
||||
pip install --upgrade parallel-web-tools
|
||||
parallel-cli config auto-update-check off
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Do not omit `--json` unless the user explicitly wants human-formatted output.
|
||||
- Do not cite sources not present in the CLI output.
|
||||
- `login` may require PTY/browser interaction.
|
||||
- Prefer foreground execution for short tasks; do not overuse background processes.
|
||||
- For large result sets, save JSON to `/tmp/*.json` instead of stuffing everything into context.
|
||||
- Do not silently choose Parallel when Hermes native tools are already sufficient.
|
||||
- Remember this is a vendor workflow that usually requires account auth and paid usage beyond the free tier.
|
||||
@@ -0,0 +1,441 @@
|
||||
---
|
||||
name: qmd
|
||||
description: Search personal knowledge bases, notes, docs, and meeting transcripts locally using qmd — a hybrid retrieval engine with BM25, vector search, and LLM reranking. Supports CLI and MCP integration.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent + Teknium
|
||||
license: MIT
|
||||
platforms: [macos, linux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [Search, Knowledge-Base, RAG, Notes, MCP, Local-AI]
|
||||
related_skills: [obsidian, native-mcp, arxiv]
|
||||
---
|
||||
|
||||
# QMD — Query Markup Documents
|
||||
|
||||
Local, on-device search engine for personal knowledge bases. Indexes markdown
|
||||
notes, meeting transcripts, documentation, and any text-based files, then
|
||||
provides hybrid search combining keyword matching, semantic understanding, and
|
||||
LLM-powered reranking — all running locally with no cloud dependencies.
|
||||
|
||||
Created by [Tobi Lütke](https://github.com/tobi/qmd). MIT licensed.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User asks to search their notes, docs, knowledge base, or meeting transcripts
|
||||
- User wants to find something across a large collection of markdown/text files
|
||||
- User wants semantic search ("find notes about X concept") not just keyword grep
|
||||
- User has already set up qmd collections and wants to query them
|
||||
- User asks to set up a local knowledge base or document search system
|
||||
- Keywords: "search my notes", "find in my docs", "knowledge base", "qmd"
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Node.js >= 22 (required)
|
||||
|
||||
```bash
|
||||
# Check version
|
||||
node --version # must be >= 22
|
||||
|
||||
# macOS — install or upgrade via Homebrew
|
||||
brew install node@22
|
||||
|
||||
# Linux — use NodeSource or nvm
|
||||
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
|
||||
sudo apt-get install -y nodejs
|
||||
# or with nvm:
|
||||
nvm install 22 && nvm use 22
|
||||
```
|
||||
|
||||
### SQLite with Extension Support (macOS only)
|
||||
|
||||
macOS system SQLite lacks extension loading. Install via Homebrew:
|
||||
|
||||
```bash
|
||||
brew install sqlite
|
||||
```
|
||||
|
||||
### Install qmd
|
||||
|
||||
```bash
|
||||
npm install -g @tobilu/qmd
|
||||
# or with Bun:
|
||||
bun install -g @tobilu/qmd
|
||||
```
|
||||
|
||||
First run auto-downloads 3 local GGUF models (~2GB total):
|
||||
|
||||
| Model | Purpose | Size |
|
||||
|-------|---------|------|
|
||||
| embeddinggemma-300M-Q8_0 | Vector embeddings | ~300MB |
|
||||
| qwen3-reranker-0.6b-q8_0 | Result reranking | ~640MB |
|
||||
| qmd-query-expansion-1.7B | Query expansion | ~1.1GB |
|
||||
|
||||
### Verify Installation
|
||||
|
||||
```bash
|
||||
qmd --version
|
||||
qmd status
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Command | What It Does | Speed |
|
||||
|---------|-------------|-------|
|
||||
| `qmd search "query"` | BM25 keyword search (no models) | ~0.2s |
|
||||
| `qmd vsearch "query"` | Semantic vector search (1 model) | ~3s |
|
||||
| `qmd query "query"` | Hybrid + reranking (all 3 models) | ~2-3s warm, ~19s cold |
|
||||
| `qmd get <docid>` | Retrieve full document content | instant |
|
||||
| `qmd multi-get "glob"` | Retrieve multiple files | instant |
|
||||
| `qmd collection add <path> --name <n>` | Add a directory as a collection | instant |
|
||||
| `qmd context add <path> "description"` | Add context metadata to improve retrieval | instant |
|
||||
| `qmd embed` | Generate/update vector embeddings | varies |
|
||||
| `qmd status` | Show index health and collection info | instant |
|
||||
| `qmd mcp` | Start MCP server (stdio) | persistent |
|
||||
| `qmd mcp --http --daemon` | Start MCP server (HTTP, warm models) | persistent |
|
||||
|
||||
## Setup Workflow
|
||||
|
||||
### 1. Add Collections
|
||||
|
||||
Point qmd at directories containing your documents:
|
||||
|
||||
```bash
|
||||
# Add a notes directory
|
||||
qmd collection add ~/notes --name notes
|
||||
|
||||
# Add project docs
|
||||
qmd collection add ~/projects/myproject/docs --name project-docs
|
||||
|
||||
# Add meeting transcripts
|
||||
qmd collection add ~/meetings --name meetings
|
||||
|
||||
# List all collections
|
||||
qmd collection list
|
||||
```
|
||||
|
||||
### 2. Add Context Descriptions
|
||||
|
||||
Context metadata helps the search engine understand what each collection
|
||||
contains. This significantly improves retrieval quality:
|
||||
|
||||
```bash
|
||||
qmd context add qmd://notes "Personal notes, ideas, and journal entries"
|
||||
qmd context add qmd://project-docs "Technical documentation for the main project"
|
||||
qmd context add qmd://meetings "Meeting transcripts and action items from team syncs"
|
||||
```
|
||||
|
||||
### 3. Generate Embeddings
|
||||
|
||||
```bash
|
||||
qmd embed
|
||||
```
|
||||
|
||||
This processes all documents in all collections and generates vector
|
||||
embeddings. Re-run after adding new documents or collections.
|
||||
|
||||
### 4. Verify
|
||||
|
||||
```bash
|
||||
qmd status # shows index health, collection stats, model info
|
||||
```
|
||||
|
||||
## Search Patterns
|
||||
|
||||
### Fast Keyword Search (BM25)
|
||||
|
||||
Best for: exact terms, code identifiers, names, known phrases.
|
||||
No models loaded — near-instant results.
|
||||
|
||||
```bash
|
||||
qmd search "authentication middleware"
|
||||
qmd search "handleError async"
|
||||
```
|
||||
|
||||
### Semantic Vector Search
|
||||
|
||||
Best for: natural language questions, conceptual queries.
|
||||
Loads embedding model (~3s first query).
|
||||
|
||||
```bash
|
||||
qmd vsearch "how does the rate limiter handle burst traffic"
|
||||
qmd vsearch "ideas for improving onboarding flow"
|
||||
```
|
||||
|
||||
### Hybrid Search with Reranking (Best Quality)
|
||||
|
||||
Best for: important queries where quality matters most.
|
||||
Uses all 3 models — query expansion, parallel BM25+vector, reranking.
|
||||
|
||||
```bash
|
||||
qmd query "what decisions were made about the database migration"
|
||||
```
|
||||
|
||||
### Structured Multi-Mode Queries
|
||||
|
||||
Combine different search types in a single query for precision:
|
||||
|
||||
```bash
|
||||
# BM25 for exact term + vector for concept
|
||||
qmd query $'lex: rate limiter\nvec: how does throttling work under load'
|
||||
|
||||
# With query expansion
|
||||
qmd query $'expand: database migration plan\nlex: "schema change"'
|
||||
```
|
||||
|
||||
### Query Syntax (lex/BM25 mode)
|
||||
|
||||
| Syntax | Effect | Example |
|
||||
|--------|--------|---------|
|
||||
| `term` | Prefix match | `perf` matches "performance" |
|
||||
| `"phrase"` | Exact phrase | `"rate limiter"` |
|
||||
| `-term` | Exclude term | `performance -sports` |
|
||||
|
||||
### HyDE (Hypothetical Document Embeddings)
|
||||
|
||||
For complex topics, write what you expect the answer to look like:
|
||||
|
||||
```bash
|
||||
qmd query $'hyde: The migration plan involves three phases. First, we add the new columns without dropping the old ones. Then we backfill data. Finally we cut over and remove legacy columns.'
|
||||
```
|
||||
|
||||
### Scoping to Collections
|
||||
|
||||
```bash
|
||||
qmd search "query" --collection notes
|
||||
qmd query "query" --collection project-docs
|
||||
```
|
||||
|
||||
### Output Formats
|
||||
|
||||
```bash
|
||||
qmd search "query" --json # JSON output (best for parsing)
|
||||
qmd search "query" --limit 5 # Limit results
|
||||
qmd get "#abc123" # Get by document ID
|
||||
qmd get "path/to/file.md" # Get by file path
|
||||
qmd get "file.md:50" -l 100 # Get specific line range
|
||||
qmd multi-get "journals/*.md" --json # Batch retrieve by glob
|
||||
```
|
||||
|
||||
## MCP Integration (Recommended)
|
||||
|
||||
qmd exposes an MCP server that provides search tools directly to
|
||||
Hermes Agent via the native MCP client. This is the preferred
|
||||
integration — once configured, the agent gets qmd tools automatically
|
||||
without needing to load this skill.
|
||||
|
||||
### Option A: Stdio Mode (Simple)
|
||||
|
||||
Add to `~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
qmd:
|
||||
command: "qmd"
|
||||
args: ["mcp"]
|
||||
timeout: 30
|
||||
connect_timeout: 45
|
||||
```
|
||||
|
||||
This registers tools: `mcp_qmd_search`, `mcp_qmd_vsearch`,
|
||||
`mcp_qmd_deep_search`, `mcp_qmd_get`, `mcp_qmd_status`.
|
||||
|
||||
**Tradeoff:** Models load on first search call (~19s cold start),
|
||||
then stay warm for the session. Acceptable for occasional use.
|
||||
|
||||
### Option B: HTTP Daemon Mode (Fast, Recommended for Heavy Use)
|
||||
|
||||
Start the qmd daemon separately — it keeps models warm in memory:
|
||||
|
||||
```bash
|
||||
# Start daemon (persists across agent restarts)
|
||||
qmd mcp --http --daemon
|
||||
|
||||
# Runs on http://localhost:8181 by default
|
||||
```
|
||||
|
||||
Then configure Hermes Agent to connect via HTTP:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
qmd:
|
||||
url: "http://localhost:8181/mcp"
|
||||
timeout: 30
|
||||
```
|
||||
|
||||
**Tradeoff:** Uses ~2GB RAM while running, but every query is fast
|
||||
(~2-3s). Best for users who search frequently.
|
||||
|
||||
### Keeping the Daemon Running
|
||||
|
||||
#### macOS (launchd)
|
||||
|
||||
```bash
|
||||
cat > ~/Library/LaunchAgents/com.qmd.daemon.plist << 'EOF'
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
|
||||
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.qmd.daemon</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>qmd</string>
|
||||
<string>mcp</string>
|
||||
<string>--http</string>
|
||||
<string>--daemon</string>
|
||||
</array>
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
<key>KeepAlive</key>
|
||||
<true/>
|
||||
<key>StandardOutPath</key>
|
||||
<string>/tmp/qmd-daemon.log</string>
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/tmp/qmd-daemon.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
|
||||
launchctl load ~/Library/LaunchAgents/com.qmd.daemon.plist
|
||||
```
|
||||
|
||||
#### Linux (systemd user service)
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.config/systemd/user
|
||||
|
||||
cat > ~/.config/systemd/user/qmd-daemon.service << 'EOF'
|
||||
[Unit]
|
||||
Description=QMD MCP Daemon
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
ExecStart=qmd mcp --http --daemon
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
Environment=PATH=/usr/local/bin:/usr/bin:/bin
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
EOF
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now qmd-daemon
|
||||
systemctl --user status qmd-daemon
|
||||
```
|
||||
|
||||
### MCP Tools Reference
|
||||
|
||||
Once connected, these tools are available as `mcp_qmd_*`:
|
||||
|
||||
| MCP Tool | Maps To | Description |
|
||||
|----------|---------|-------------|
|
||||
| `mcp_qmd_search` | `qmd search` | BM25 keyword search |
|
||||
| `mcp_qmd_vsearch` | `qmd vsearch` | Semantic vector search |
|
||||
| `mcp_qmd_deep_search` | `qmd query` | Hybrid search + reranking |
|
||||
| `mcp_qmd_get` | `qmd get` | Retrieve document by ID or path |
|
||||
| `mcp_qmd_status` | `qmd status` | Index health and stats |
|
||||
|
||||
The MCP tools accept structured JSON queries for multi-mode search:
|
||||
|
||||
```json
|
||||
{
|
||||
"searches": [
|
||||
{"type": "lex", "query": "authentication middleware"},
|
||||
{"type": "vec", "query": "how user login is verified"}
|
||||
],
|
||||
"collections": ["project-docs"],
|
||||
"limit": 10
|
||||
}
|
||||
```
|
||||
|
||||
## CLI Usage (Without MCP)
|
||||
|
||||
When MCP is not configured, use qmd directly via terminal:
|
||||
|
||||
```
|
||||
terminal(command="qmd query 'what was decided about the API redesign' --json", timeout=30)
|
||||
```
|
||||
|
||||
For setup and management tasks, always use terminal:
|
||||
|
||||
```
|
||||
terminal(command="qmd collection add ~/Documents/notes --name notes")
|
||||
terminal(command="qmd context add qmd://notes 'Personal research notes and ideas'")
|
||||
terminal(command="qmd embed")
|
||||
terminal(command="qmd status")
|
||||
```
|
||||
|
||||
## How the Search Pipeline Works
|
||||
|
||||
Understanding the internals helps choose the right search mode:
|
||||
|
||||
1. **Query Expansion** — A fine-tuned 1.7B model generates 2 alternative
|
||||
queries. The original gets 2x weight in fusion.
|
||||
2. **Parallel Retrieval** — BM25 (SQLite FTS5) and vector search run
|
||||
simultaneously across all query variants.
|
||||
3. **RRF Fusion** — Reciprocal Rank Fusion (k=60) merges results.
|
||||
Top-rank bonus: #1 gets +0.05, #2-3 get +0.02.
|
||||
4. **LLM Reranking** — qwen3-reranker scores top 30 candidates (0.0-1.0).
|
||||
5. **Position-Aware Blending** — Ranks 1-3: 75% retrieval / 25% reranker.
|
||||
Ranks 4-10: 60/40. Ranks 11+: 40/60 (trusts reranker more for long tail).
|
||||
|
||||
**Smart Chunking:** Documents are split at natural break points (headings,
|
||||
code blocks, blank lines) targeting ~900 tokens with 15% overlap. Code
|
||||
blocks are never split mid-block.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always add context descriptions** — `qmd context add` dramatically
|
||||
improves retrieval accuracy. Describe what each collection contains.
|
||||
2. **Re-embed after adding documents** — `qmd embed` must be re-run when
|
||||
new files are added to collections.
|
||||
3. **Use `qmd search` for speed** — when you need fast keyword lookup
|
||||
(code identifiers, exact names), BM25 is instant and needs no models.
|
||||
4. **Use `qmd query` for quality** — when the question is conceptual or
|
||||
the user needs the best possible results, use hybrid search.
|
||||
5. **Prefer MCP integration** — once configured, the agent gets native
|
||||
tools without needing to load this skill each time.
|
||||
6. **Daemon mode for frequent users** — if the user searches their
|
||||
knowledge base regularly, recommend the HTTP daemon setup.
|
||||
7. **First query in structured search gets 2x weight** — put the most
|
||||
important/certain query first when combining lex and vec.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Models downloading on first run"
|
||||
Normal — qmd auto-downloads ~2GB of GGUF models on first use.
|
||||
This is a one-time operation.
|
||||
|
||||
### Cold start latency (~19s)
|
||||
This happens when models aren't loaded in memory. Solutions:
|
||||
- Use HTTP daemon mode (`qmd mcp --http --daemon`) to keep warm
|
||||
- Use `qmd search` (BM25 only) when models aren't needed
|
||||
- MCP stdio mode loads models on first search, stays warm for session
|
||||
|
||||
### macOS: "unable to load extension"
|
||||
Install Homebrew SQLite: `brew install sqlite`
|
||||
Then ensure it's on PATH before system SQLite.
|
||||
|
||||
### "No collections found"
|
||||
Run `qmd collection add <path> --name <name>` to add directories,
|
||||
then `qmd embed` to index them.
|
||||
|
||||
### Embedding model override (CJK/multilingual)
|
||||
Set `QMD_EMBED_MODEL` environment variable for non-English content:
|
||||
```bash
|
||||
export QMD_EMBED_MODEL="your-multilingual-model"
|
||||
```
|
||||
|
||||
## Data Storage
|
||||
|
||||
- **Index & vectors:** `~/.cache/qmd/index.sqlite`
|
||||
- **Models:** Auto-downloaded to local cache on first run
|
||||
- **No cloud dependencies** — everything runs locally
|
||||
|
||||
## References
|
||||
|
||||
- [GitHub: tobi/qmd](https://github.com/tobi/qmd)
|
||||
- [QMD Changelog](https://github.com/tobi/qmd/blob/main/CHANGELOG.md)
|
||||
@@ -0,0 +1,336 @@
|
||||
---
|
||||
name: scrapling
|
||||
description: Web scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python.
|
||||
version: 1.0.0
|
||||
author: FEUAZUR
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [Web Scraping, Browser, Cloudflare, Stealth, Crawling, Spider]
|
||||
related_skills: [duckduckgo-search, domain-intel]
|
||||
homepage: https://github.com/D4Vinci/Scrapling
|
||||
prerequisites:
|
||||
commands: [scrapling, python]
|
||||
---
|
||||
|
||||
# Scrapling
|
||||
|
||||
[Scrapling](https://github.com/D4Vinci/Scrapling) is a web scraping framework with anti-bot bypass, stealth browser automation, and a spider framework. It provides three fetching strategies (HTTP, dynamic JS, stealth/Cloudflare) and a full CLI.
|
||||
|
||||
**This skill is for educational and research purposes only.** Users must comply with local/international data scraping laws and respect website Terms of Service.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Scraping static HTML pages (faster than browser tools)
|
||||
- Scraping JS-rendered pages that need a real browser
|
||||
- Bypassing Cloudflare Turnstile or bot detection
|
||||
- Crawling multiple pages with a spider
|
||||
- When the built-in `web_extract` tool does not return the data you need
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install "scrapling[all]"
|
||||
scrapling install
|
||||
```
|
||||
|
||||
Minimal install (HTTP only, no browser):
|
||||
```bash
|
||||
pip install scrapling
|
||||
```
|
||||
|
||||
With browser automation only:
|
||||
```bash
|
||||
pip install "scrapling[fetchers]"
|
||||
scrapling install
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Approach | Class | Use When |
|
||||
|----------|-------|----------|
|
||||
| HTTP | `Fetcher` / `FetcherSession` | Static pages, APIs, fast bulk requests |
|
||||
| Dynamic | `DynamicFetcher` / `DynamicSession` | JS-rendered content, SPAs |
|
||||
| Stealth | `StealthyFetcher` / `StealthySession` | Cloudflare, anti-bot protected sites |
|
||||
| Spider | `Spider` | Multi-page crawling with link following |
|
||||
|
||||
## CLI Usage
|
||||
|
||||
### Extract Static Page
|
||||
|
||||
```bash
|
||||
scrapling extract get 'https://example.com' output.md
|
||||
```
|
||||
|
||||
With CSS selector and browser impersonation:
|
||||
|
||||
```bash
|
||||
scrapling extract get 'https://example.com' output.md \
|
||||
--css-selector '.content' \
|
||||
--impersonate 'chrome'
|
||||
```
|
||||
|
||||
### Extract JS-Rendered Page
|
||||
|
||||
```bash
|
||||
scrapling extract fetch 'https://example.com' output.md \
|
||||
--css-selector '.dynamic-content' \
|
||||
--disable-resources \
|
||||
--network-idle
|
||||
```
|
||||
|
||||
### Extract Cloudflare-Protected Page
|
||||
|
||||
```bash
|
||||
scrapling extract stealthy-fetch 'https://protected-site.com' output.html \
|
||||
--solve-cloudflare \
|
||||
--block-webrtc \
|
||||
--hide-canvas
|
||||
```
|
||||
|
||||
### POST Request
|
||||
|
||||
```bash
|
||||
scrapling extract post 'https://example.com/api' output.json \
|
||||
--json '{"query": "search term"}'
|
||||
```
|
||||
|
||||
### Output Formats
|
||||
|
||||
The output format is determined by the file extension:
|
||||
- `.html` -- raw HTML
|
||||
- `.md` -- converted to Markdown
|
||||
- `.txt` -- plain text
|
||||
- `.json` / `.jsonl` -- JSON
|
||||
|
||||
## Python: HTTP Scraping
|
||||
|
||||
### Single Request
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher
|
||||
|
||||
page = Fetcher.get('https://quotes.toscrape.com/')
|
||||
quotes = page.css('.quote .text::text').getall()
|
||||
for q in quotes:
|
||||
print(q)
|
||||
```
|
||||
|
||||
### Session (Persistent Cookies)
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import FetcherSession
|
||||
|
||||
with FetcherSession(impersonate='chrome') as session:
|
||||
page = session.get('https://example.com/', stealthy_headers=True)
|
||||
links = page.css('a::attr(href)').getall()
|
||||
for link in links[:5]:
|
||||
sub = session.get(link)
|
||||
print(sub.css('h1::text').get())
|
||||
```
|
||||
|
||||
### POST / PUT / DELETE
|
||||
|
||||
```python
|
||||
page = Fetcher.post('https://api.example.com/data', json={"key": "value"})
|
||||
page = Fetcher.put('https://api.example.com/item/1', data={"name": "updated"})
|
||||
page = Fetcher.delete('https://api.example.com/item/1')
|
||||
```
|
||||
|
||||
### With Proxy
|
||||
|
||||
```python
|
||||
page = Fetcher.get('https://example.com', proxy='http://user:pass@proxy:8080')
|
||||
```
|
||||
|
||||
## Python: Dynamic Pages (JS-Rendered)
|
||||
|
||||
For pages that require JavaScript execution (SPAs, lazy-loaded content):
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import DynamicFetcher
|
||||
|
||||
page = DynamicFetcher.fetch('https://example.com', headless=True)
|
||||
data = page.css('.js-loaded-content::text').getall()
|
||||
```
|
||||
|
||||
### Wait for Specific Element
|
||||
|
||||
```python
|
||||
page = DynamicFetcher.fetch(
|
||||
'https://example.com',
|
||||
wait_selector=('.results', 'visible'),
|
||||
network_idle=True,
|
||||
)
|
||||
```
|
||||
|
||||
### Disable Resources for Speed
|
||||
|
||||
Blocks fonts, images, media, stylesheets (~25% faster):
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import DynamicSession
|
||||
|
||||
with DynamicSession(headless=True, disable_resources=True, network_idle=True) as session:
|
||||
page = session.fetch('https://example.com')
|
||||
items = page.css('.item::text').getall()
|
||||
```
|
||||
|
||||
### Custom Page Automation
|
||||
|
||||
```python
|
||||
from playwright.sync_api import Page
|
||||
from scrapling.fetchers import DynamicFetcher
|
||||
|
||||
def scroll_and_click(page: Page):
|
||||
page.mouse.wheel(0, 3000)
|
||||
page.wait_for_timeout(1000)
|
||||
page.click('button.load-more')
|
||||
page.wait_for_selector('.extra-results')
|
||||
|
||||
page = DynamicFetcher.fetch('https://example.com', page_action=scroll_and_click)
|
||||
results = page.css('.extra-results .item::text').getall()
|
||||
```
|
||||
|
||||
## Python: Stealth Mode (Anti-Bot Bypass)
|
||||
|
||||
For Cloudflare-protected or heavily fingerprinted sites:
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import StealthyFetcher
|
||||
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://protected-site.com',
|
||||
headless=True,
|
||||
solve_cloudflare=True,
|
||||
block_webrtc=True,
|
||||
hide_canvas=True,
|
||||
)
|
||||
content = page.css('.protected-content::text').getall()
|
||||
```
|
||||
|
||||
### Stealth Session
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import StealthySession
|
||||
|
||||
with StealthySession(headless=True, solve_cloudflare=True) as session:
|
||||
page1 = session.fetch('https://protected-site.com/page1')
|
||||
page2 = session.fetch('https://protected-site.com/page2')
|
||||
```
|
||||
|
||||
## Element Selection
|
||||
|
||||
All fetchers return a `Selector` object with these methods:
|
||||
|
||||
### CSS Selectors
|
||||
|
||||
```python
|
||||
page.css('h1::text').get() # First h1 text
|
||||
page.css('a::attr(href)').getall() # All link hrefs
|
||||
page.css('.quote .text::text').getall() # Nested selection
|
||||
```
|
||||
|
||||
### XPath
|
||||
|
||||
```python
|
||||
page.xpath('//div[@class="content"]/text()').getall()
|
||||
page.xpath('//a/@href').getall()
|
||||
```
|
||||
|
||||
### Find Methods
|
||||
|
||||
```python
|
||||
page.find_all('div', class_='quote') # By tag + attribute
|
||||
page.find_by_text('Read more', tag='a') # By text content
|
||||
page.find_by_regex(r'\$\d+\.\d{2}') # By regex pattern
|
||||
```
|
||||
|
||||
### Similar Elements
|
||||
|
||||
Find elements with similar structure (useful for product listings, etc.):
|
||||
|
||||
```python
|
||||
first_product = page.css('.product')[0]
|
||||
all_similar = first_product.find_similar()
|
||||
```
|
||||
|
||||
### Navigation
|
||||
|
||||
```python
|
||||
el = page.css('.target')[0]
|
||||
el.parent # Parent element
|
||||
el.children # Child elements
|
||||
el.next_sibling # Next sibling
|
||||
el.prev_sibling # Previous sibling
|
||||
```
|
||||
|
||||
## Python: Spider Framework
|
||||
|
||||
For multi-page crawling with link following:
|
||||
|
||||
```python
|
||||
from scrapling.spiders import Spider, Request, Response
|
||||
|
||||
class QuotesSpider(Spider):
|
||||
name = "quotes"
|
||||
start_urls = ["https://quotes.toscrape.com/"]
|
||||
concurrent_requests = 10
|
||||
download_delay = 1
|
||||
|
||||
async def parse(self, response: Response):
|
||||
for quote in response.css('.quote'):
|
||||
yield {
|
||||
"text": quote.css('.text::text').get(),
|
||||
"author": quote.css('.author::text').get(),
|
||||
"tags": quote.css('.tag::text').getall(),
|
||||
}
|
||||
|
||||
next_page = response.css('.next a::attr(href)').get()
|
||||
if next_page:
|
||||
yield response.follow(next_page)
|
||||
|
||||
result = QuotesSpider().start()
|
||||
print(f"Scraped {len(result.items)} quotes")
|
||||
result.items.to_json("quotes.json")
|
||||
```
|
||||
|
||||
### Multi-Session Spider
|
||||
|
||||
Route requests to different fetcher types:
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import FetcherSession, AsyncStealthySession
|
||||
|
||||
class SmartSpider(Spider):
|
||||
name = "smart"
|
||||
start_urls = ["https://example.com/"]
|
||||
|
||||
def configure_sessions(self, manager):
|
||||
manager.add("fast", FetcherSession(impersonate="chrome"))
|
||||
manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
|
||||
|
||||
async def parse(self, response: Response):
|
||||
for link in response.css('a::attr(href)').getall():
|
||||
if "protected" in link:
|
||||
yield Request(link, sid="stealth")
|
||||
else:
|
||||
yield Request(link, sid="fast", callback=self.parse)
|
||||
```
|
||||
|
||||
### Pause/Resume Crawling
|
||||
|
||||
```python
|
||||
spider = QuotesSpider(crawldir="./crawl_checkpoint")
|
||||
spider.start() # Ctrl+C to pause, re-run to resume from checkpoint
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Browser install required**: run `scrapling install` after pip install -- without it, `DynamicFetcher` and `StealthyFetcher` will fail
|
||||
- **Timeouts**: DynamicFetcher/StealthyFetcher timeout is in **milliseconds** (default 30000), Fetcher timeout is in **seconds**
|
||||
- **Cloudflare bypass**: `solve_cloudflare=True` adds 5-15 seconds to fetch time -- only enable when needed
|
||||
- **Resource usage**: StealthyFetcher runs a real browser -- limit concurrent usage
|
||||
- **Legal**: always check robots.txt and website ToS before scraping. This library is for educational and research purposes
|
||||
- **Python version**: requires Python 3.10+
|
||||
@@ -0,0 +1,212 @@
|
||||
---
|
||||
name: searxng-search
|
||||
description: Free meta-search via SearXNG — aggregates results from 70+ search engines. Self-hosted or use a public instance. No API key needed. Falls back automatically when the web search toolset is unavailable.
|
||||
version: 1.0.0
|
||||
author: hermes-agent
|
||||
license: MIT
|
||||
platforms: [linux, macos]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [search, searxng, meta-search, self-hosted, free, fallback]
|
||||
related_skills: [duckduckgo-search, domain-intel]
|
||||
fallback_for_toolsets: [web]
|
||||
---
|
||||
|
||||
# SearXNG Search
|
||||
|
||||
Free meta-search using [SearXNG](https://searxng.org/) — a privacy-respecting, self-hosted search aggregator that queries 70+ search engines simultaneously.
|
||||
|
||||
**No API key required** when using a public instance. Can also be self-hosted for full control. Automatically appears as a fallback when the main web search toolset (`FIRECRAWL_API_KEY`) is not configured.
|
||||
|
||||
## Configuration
|
||||
|
||||
SearXNG requires a `SEARXNG_URL` environment variable pointing to your SearXNG instance:
|
||||
|
||||
```bash
|
||||
# Public instances (no setup required)
|
||||
SEARXNG_URL=https://searxng.example.com
|
||||
|
||||
# Self-hosted SearXNG
|
||||
SEARXNG_URL=http://localhost:8888
|
||||
```
|
||||
|
||||
If no instance is configured, this skill is unavailable and the agent falls back to other search options.
|
||||
|
||||
## Detection Flow
|
||||
|
||||
Check what is actually available before choosing an approach:
|
||||
|
||||
```bash
|
||||
# Check if SEARXNG_URL is set and the instance is reachable
|
||||
curl -s --max-time 5 "${SEARXNG_URL}/search?q=test&format=json" | head -c 200
|
||||
```
|
||||
|
||||
Decision tree:
|
||||
1. If `SEARXNG_URL` is set and the instance responds, use SearXNG
|
||||
2. If `SEARXNG_URL` is unset or unreachable, fall back to other available search tools
|
||||
3. If the user wants SearXNG specifically, help them set up an instance or find a public one
|
||||
|
||||
## Method 1: CLI via curl (Preferred)
|
||||
|
||||
Use `curl` via `terminal` to call the SearXNG JSON API. This avoids assuming any particular Python package is installed.
|
||||
|
||||
```bash
|
||||
# Text search (JSON output)
|
||||
curl -s --max-time 10 \
|
||||
"${SEARXNG_URL}/search?q=python+async+programming&format=json&engines=google,bing&limit=10"
|
||||
|
||||
# With Safesearch off
|
||||
curl -s --max-time 10 \
|
||||
"${SEARXNG_URL}/search?q=example&format=json&safesearch=0"
|
||||
|
||||
# Specific categories (general, news, science, etc.)
|
||||
curl -s --max-time 10 \
|
||||
"${SEARXNG_URL}/search?q=AI+news&format=json&categories=news"
|
||||
```
|
||||
|
||||
### Common CLI Flags
|
||||
|
||||
| Flag | Description | Example |
|
||||
|------|-------------|---------|
|
||||
| `q` | Query string (URL-encoded) | `q=python+async` |
|
||||
| `format` | Output format: `json`, `csv`, `rss` | `format=json` |
|
||||
| `engines` | Comma-separated engine names | `engines=google,bing,ddg` |
|
||||
| `limit` | Max results per engine (default 10) | `limit=5` |
|
||||
| `categories` | Filter by category | `categories=news,science` |
|
||||
| `safesearch` | 0=none, 1=moderate, 2=strict | `safesearch=0` |
|
||||
| `time_range` | Filter: `day`, `week`, `month`, `year` | `time_range=week` |
|
||||
|
||||
### Parsing JSON Results
|
||||
|
||||
```bash
|
||||
# Extract titles and URLs from JSON
|
||||
curl -s --max-time 10 "${SEARXNG_URL}/search?q=fastapi&format=json&limit=5" \
|
||||
| python3 -c "
|
||||
import json, sys
|
||||
data = json.load(sys.stdin)
|
||||
for r in data.get('results', []):
|
||||
print(r.get('title',''))
|
||||
print(r.get('url',''))
|
||||
print(r.get('content','')[:200])
|
||||
print()
|
||||
"
|
||||
```
|
||||
|
||||
Returns per result: `title`, `url`, `content` (snippet), `engine`, `parsed_url`, `img_src`, `thumbnail`, `author`, `published_date`
|
||||
|
||||
## Method 2: Python API via `requests`
|
||||
|
||||
Use the SearXNG REST API directly from Python with the `requests` library:
|
||||
|
||||
```python
|
||||
import os, requests, urllib.parse
|
||||
|
||||
base_url = os.environ.get("SEARXNG_URL", "")
|
||||
if not base_url:
|
||||
raise RuntimeError("SEARXNG_URL is not set")
|
||||
|
||||
query = "fastapi deployment guide"
|
||||
params = {
|
||||
"q": query,
|
||||
"format": "json",
|
||||
"limit": 5,
|
||||
"engines": "google,bing",
|
||||
}
|
||||
|
||||
resp = requests.get(f"{base_url}/search", params=params, timeout=10)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
for r in data.get("results", []):
|
||||
print(r["title"])
|
||||
print(r["url"])
|
||||
print(r.get("content", "")[:200])
|
||||
print()
|
||||
```
|
||||
|
||||
## Method 3: searxng-data Python Package
|
||||
|
||||
For more structured access, install the `searxng-data` package:
|
||||
|
||||
```bash
|
||||
pip install searxng-data
|
||||
```
|
||||
|
||||
```python
|
||||
from searxng_data import engines
|
||||
|
||||
# List available engines
|
||||
print(engines.list_engines())
|
||||
```
|
||||
|
||||
Note: This package only provides engine metadata, not the search API itself.
|
||||
|
||||
## Self-Hosting SearXNG
|
||||
|
||||
To run your own SearXNG instance:
|
||||
|
||||
```bash
|
||||
# Using Docker
|
||||
docker run -d -p 8888:8080 \
|
||||
-v $(pwd)/searxng:/etc/searxng \
|
||||
searxng/searxng:latest
|
||||
|
||||
# Then set
|
||||
SEARXNG_URL=http://localhost:8888
|
||||
```
|
||||
|
||||
Or install via pip:
|
||||
```bash
|
||||
pip install searxng
|
||||
# Edit /etc/searxng/settings.yml
|
||||
searxng-run
|
||||
```
|
||||
|
||||
Public SearXNG instances are available at:
|
||||
- `https://searxng.example.com` (replace with any public instance)
|
||||
|
||||
## Workflow: Search then Extract
|
||||
|
||||
SearXNG returns titles, URLs, and snippets — not full page content. To get full page content, search first and then extract the most relevant URL with `web_extract`, browser tools, or `curl`.
|
||||
|
||||
```bash
|
||||
# Search for relevant pages
|
||||
curl -s "${SEARXNG_URL}/search?q=fastapi+deployment&format=json&limit=3"
|
||||
# Output: list of results with titles and URLs
|
||||
|
||||
# Then extract the best URL with web_extract
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Instance availability**: If the SearXNG instance is down or unreachable, search fails. Always check `SEARXNG_URL` is set and the instance is reachable.
|
||||
- **No content extraction**: SearXNG returns snippets, not full page content. Use `web_extract`, browser tools, or `curl` for full articles.
|
||||
- **Rate limiting**: Some public instances limit requests. Self-hosting avoids this.
|
||||
- **Engine coverage**: Available engines depend on the SearXNG instance configuration. Some engines may be disabled.
|
||||
- **Results freshness**: Meta-search aggregates external engines — result freshness depends on those engines.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Likely Cause | What To Do |
|
||||
|---------|--------------|------------|
|
||||
| `SEARXNG_URL` not set | No instance configured | Use a public SearXNG instance or set up your own |
|
||||
| Connection refused | Instance not running or wrong URL | Check the URL is correct and the instance is running |
|
||||
| Empty results | Instance blocks the query | Try a different instance or self-host |
|
||||
| Slow responses | Public instance under load | Self-host or use a less-loaded public instance |
|
||||
| `json` format not supported | Old SearXNG version | Try `format=rss` or upgrade SearXNG |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Always set `SEARXNG_URL`**: Without it, the skill cannot function.
|
||||
- **URL-encode queries**: Spaces and special characters must be URL-encoded in curl, or use `urllib.parse.quote()` in Python.
|
||||
- **Use `format=json`**: The default format may not be machine-readable. Always request JSON explicitly.
|
||||
- **Set a timeout**: Always use `--max-time` or `timeout=` to avoid hanging on unreachable instances.
|
||||
- **Self-hosting is best**: Public instances may go down, rate-limit, or block. A self-hosted instance is reliable.
|
||||
|
||||
## Instance Discovery
|
||||
|
||||
If `SEARXNG_URL` is not set and the user asks about SearXNG, help them either:
|
||||
1. Find a public SearXNG instance (search for "public searxng instance")
|
||||
2. Set up their own with Docker or pip
|
||||
|
||||
Public instances are listed at: https://searxng.org/
|
||||
@@ -0,0 +1,22 @@
|
||||
#!/bin/bash
|
||||
# Usage: ./searxng.sh <query> [max_results] [engines]
|
||||
# Example: ./searxng.sh "python async" 10 "google,bing"
|
||||
|
||||
QUERY="${1:-}"
|
||||
MAX="${2:-5}"
|
||||
ENGINES="${3:-google,bing}"
|
||||
|
||||
if [ -z "$SEARXNG_URL" ]; then
|
||||
echo "Error: SEARXNG_URL is not set"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -z "$QUERY" ]; then
|
||||
echo "Usage: $0 <query> [max_results] [engines]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ENCODED_QUERY=$(echo "$QUERY" | sed 's/ /+/g')
|
||||
|
||||
curl -s --max-time 10 \
|
||||
"${SEARXNG_URL}/search?q=${ENCODED_QUERY}&format=json&limit=${MAX}&engines=${ENGINES}"
|
||||
Reference in New Issue
Block a user