Hermes-agent
This commit is contained in:
Zakaria
@@ -0,0 +1,403 @@
|
||||
# Adding a New Messaging Platform
|
||||
|
||||
There are two ways to add a platform to the Hermes gateway:
|
||||
|
||||
## Plugin Path (Recommended for Community/Third-Party)
|
||||
|
||||
Create a plugin directory in `~/.hermes/plugins/` (or under `plugins/platforms/`
|
||||
for bundled plugins) with a `plugin.yaml` and `adapter.py`. The adapter
|
||||
inherits from `BasePlatformAdapter` and registers via
|
||||
`ctx.register_platform()` in the `register(ctx)` entry point. This requires
|
||||
**zero changes to core Hermes code**.
|
||||
|
||||
The plugin system automatically handles: adapter creation, config parsing,
|
||||
user authorization, cron delivery, send_message routing, system prompt hints,
|
||||
status display, gateway setup, and more.
|
||||
|
||||
**Optional hooks cover the edges most adapters need:**
|
||||
|
||||
- `env_enablement_fn: () -> Optional[dict]` — seeds `PlatformConfig.extra`
|
||||
(and an optional `home_channel` dict) from env vars BEFORE the adapter is
|
||||
constructed. Without this, env-only setups don't surface in
|
||||
`hermes gateway status` or `get_connected_platforms()` until the SDK
|
||||
instantiates.
|
||||
- `apply_yaml_config_fn: (yaml_cfg, platform_cfg) -> Optional[dict]` —
|
||||
translate this platform's `config.yaml` keys into env vars and/or seed
|
||||
`PlatformConfig.extra` directly. Lets a plugin own its YAML schema
|
||||
instead of growing core `gateway/config.py` boilerplate per platform.
|
||||
Mutating `os.environ` is allowed (use `not os.getenv(...)` guards to
|
||||
preserve env > YAML precedence); the returned dict is merged into
|
||||
`PlatformConfig.extra`. Called during `load_gateway_config()` after
|
||||
the generic shared-key loop and before `_apply_env_overrides()`.
|
||||
- `cron_deliver_env_var: str` — name of the `*_HOME_CHANNEL` env var. When
|
||||
set, `deliver=<name>` cron jobs route to this var without editing
|
||||
`cron/scheduler.py`'s hardcoded sets.
|
||||
- `standalone_sender_fn: async (...) -> dict`: out-of-process delivery
|
||||
for cron jobs that run separately from the gateway. Without this, a
|
||||
`deliver=<name>` job fires correctly but the actual send returns
|
||||
`No live adapter for platform '<name>'`. Pair with `cron_deliver_env_var`
|
||||
for end-to-end cron support. See the docsite for the signature.
|
||||
- `plugin.yaml` `requires_env` / `optional_env` rich-dict entries —
|
||||
auto-populate `OPTIONAL_ENV_VARS` in `hermes_cli/config.py` so the setup
|
||||
wizard surfaces proper descriptions, prompts, password flags, and URLs.
|
||||
|
||||
**Subclassing for platform-specific UX.** When a platform has a hard
|
||||
time-window constraint that the base adapter can't anticipate (LINE's
|
||||
60s single-use reply token, WhatsApp's 24h session window, etc.), an
|
||||
adapter can override `_keep_typing` to layer a mid-flight bubble at a
|
||||
threshold without expanding the kwarg surface. Always
|
||||
`await super()._keep_typing(...)` so the typing heartbeat keeps running,
|
||||
and tear down your side task in `finally`. See `plugins/platforms/line/`
|
||||
for the full pattern (Template Buttons postback at 45s, `RequestCache`
|
||||
state machine, `interrupt_session_activity` override for `/stop`
|
||||
orphans) and the developer-guide page for the prose walkthrough.
|
||||
|
||||
**Sibling adapters that share behavior.** When a single platform has
|
||||
two transport modes the user picks between — unofficial vs official
|
||||
APIs, polling vs websocket, library A vs library B — the right
|
||||
structure is two adapters that share a behavior mixin. WhatsApp does
|
||||
this: `gateway/platforms/whatsapp.py` (Baileys bridge) and
|
||||
`gateway/platforms/whatsapp_cloud.py` (Meta Cloud API) both inherit
|
||||
from `WhatsAppBehaviorMixin` in `gateway/platforms/whatsapp_common.py`.
|
||||
The mixin owns gating, allow-lists, mention parsing, broadcast
|
||||
filters, and the WhatsApp-flavored markdown conversion — everything
|
||||
that's platform-protocol-agnostic. Each adapter owns its transport.
|
||||
Both register distinct `Platform.*` enum values so the gateway can run
|
||||
both simultaneously against different phone numbers. The mixin must
|
||||
come **first** in the bases list — `class WhatsAppAdapter(Mixin,
|
||||
BasePlatformAdapter)` — so the mixin's `format_message` overrides
|
||||
`BasePlatformAdapter`'s generic default.
|
||||
|
||||
See `plugins/platforms/irc/`, `plugins/platforms/teams/`, and
|
||||
`plugins/platforms/google_chat/` for complete working examples, and
|
||||
`website/docs/developer-guide/adding-platform-adapters.md` for the full
|
||||
plugin guide with code examples and hook documentation.
|
||||
|
||||
---
|
||||
|
||||
## Built-in Path (Core Contributors Only)
|
||||
|
||||
Checklist for integrating a platform directly into the Hermes core.
|
||||
Use this as a reference when building a built-in adapter — every item here
|
||||
is a real integration point. Missing any of them will cause broken
|
||||
functionality, missing features, or inconsistent behavior.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core Adapter (`gateway/platforms/<platform>.py`)
|
||||
|
||||
The adapter is a subclass of `BasePlatformAdapter` from `gateway/platforms/base.py`.
|
||||
|
||||
### Required methods
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `__init__(self, config)` | Parse config, init state. Call `super().__init__(config, Platform.YOUR_PLATFORM)` |
|
||||
| `connect() -> bool` | Connect to the platform, start listeners. Return True on success |
|
||||
| `disconnect()` | Stop listeners, close connections, cancel tasks |
|
||||
| `send(chat_id, text, ...) -> SendResult` | Send a text message |
|
||||
| `send_typing(chat_id)` | Send typing indicator |
|
||||
| `send_image(chat_id, image_url, caption) -> SendResult` | Send an image |
|
||||
| `get_chat_info(chat_id) -> dict` | Return `{name, type, chat_id}` for a chat |
|
||||
|
||||
### Optional methods (have default stubs in base)
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `send_document(chat_id, path, caption)` | Send a file attachment |
|
||||
| `send_voice(chat_id, path)` | Send a voice message |
|
||||
| `send_video(chat_id, path, caption)` | Send a video |
|
||||
| `send_animation(chat_id, path, caption)` | Send a GIF/animation |
|
||||
| `send_image_file(chat_id, path, caption)` | Send image from local file |
|
||||
|
||||
### Interactive UX (recommended if your platform supports tappable buttons)
|
||||
|
||||
If your platform supports interactive button/menu messages, implement these for a more polished agent experience. They all degrade gracefully to plain text when not overridden:
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `send_clarify(chat_id, question, choices, clarify_id, session_key, ...)` | Render the `clarify` tool's multi-choice question as tappable buttons. Pair with inbound dispatch that routes button taps to `tools.clarify_gateway.resolve_gateway_clarify`. |
|
||||
| `send_exec_approval(chat_id, command, session_key, description, ...)` | Render dangerous-command approval as Approve/Deny buttons. Inbound dispatch routes to `tools.approval.resolve_gateway_approval`. |
|
||||
| `send_slash_confirm(chat_id, title, message, session_key, confirm_id, ...)` | Render slash-command confirmations (e.g. `/reload-mcp`) as Once/Always/Cancel buttons. Inbound dispatch routes to `tools.slash_confirm.resolve`. |
|
||||
| `send_model_picker(...)` | Interactive `/model` picker. Used by Telegram and Discord. |
|
||||
|
||||
See `gateway/platforms/telegram.py`, `discord.py`, and `whatsapp_cloud.py` for reference implementations. The button-callback id convention (`cl:<id>:<idx>`, `appr:<id>:<choice>`, `sc:<choice>:<id>`) is shared across adapters — match it so the gateway-side resolvers work without modification.
|
||||
|
||||
### Required function
|
||||
|
||||
```python
|
||||
def check_<platform>_requirements() -> bool:
|
||||
"""Check if this platform's dependencies are available."""
|
||||
```
|
||||
|
||||
### Key patterns to follow
|
||||
|
||||
- Use `self.build_source(...)` to construct `SessionSource` objects
|
||||
- Call `self.handle_message(event)` to dispatch inbound messages to the gateway
|
||||
- Use `MessageEvent`, `MessageType`, `SendResult` from base
|
||||
- Use `cache_image_from_bytes`, `cache_audio_from_bytes`, `cache_document_from_bytes` for attachments
|
||||
- Filter self-messages (prevent reply loops)
|
||||
- Filter sync/echo messages if the platform has them
|
||||
- Redact sensitive identifiers (phone numbers, tokens) in all log output
|
||||
- Implement reconnection with exponential backoff + jitter for streaming connections
|
||||
- Set `MAX_MESSAGE_LENGTH` if the platform has message size limits
|
||||
|
||||
---
|
||||
|
||||
## 2. Platform Enum (`gateway/config.py`)
|
||||
|
||||
Add the platform to the `Platform` enum:
|
||||
|
||||
```python
|
||||
class Platform(Enum):
|
||||
...
|
||||
YOUR_PLATFORM = "your_platform"
|
||||
```
|
||||
|
||||
Add env var loading in `_apply_env_overrides()`:
|
||||
|
||||
```python
|
||||
# Your Platform
|
||||
your_token = os.getenv("YOUR_PLATFORM_TOKEN")
|
||||
if your_token:
|
||||
if Platform.YOUR_PLATFORM not in config.platforms:
|
||||
config.platforms[Platform.YOUR_PLATFORM] = PlatformConfig()
|
||||
config.platforms[Platform.YOUR_PLATFORM].enabled = True
|
||||
config.platforms[Platform.YOUR_PLATFORM].token = your_token
|
||||
```
|
||||
|
||||
Update `get_connected_platforms()` if your platform doesn't use token/api_key
|
||||
(e.g., WhatsApp uses `enabled` flag, Signal uses `extra` dict).
|
||||
|
||||
---
|
||||
|
||||
## 3. Adapter Factory (`gateway/run.py`)
|
||||
|
||||
Add to `_create_adapter()`:
|
||||
|
||||
```python
|
||||
elif platform == Platform.YOUR_PLATFORM:
|
||||
from gateway.platforms.your_platform import YourAdapter, check_your_requirements
|
||||
if not check_your_requirements():
|
||||
logger.warning("Your Platform: dependencies not met")
|
||||
return None
|
||||
return YourAdapter(config)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Authorization Maps (`gateway/run.py`)
|
||||
|
||||
Add to BOTH dicts in `_is_user_authorized()`:
|
||||
|
||||
```python
|
||||
platform_env_map = {
|
||||
...
|
||||
Platform.YOUR_PLATFORM: "YOUR_PLATFORM_ALLOWED_USERS",
|
||||
}
|
||||
platform_allow_all_map = {
|
||||
...
|
||||
Platform.YOUR_PLATFORM: "YOUR_PLATFORM_ALLOW_ALL_USERS",
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Session Source (`gateway/session.py`)
|
||||
|
||||
If your platform needs extra identity fields (e.g., Signal's UUID alongside
|
||||
phone number), add them to the `SessionSource` dataclass with `Optional` defaults,
|
||||
and update `to_dict()`, `from_dict()`, and `build_source()` in base.py.
|
||||
|
||||
---
|
||||
|
||||
## 6. System Prompt Hints (`agent/prompt_builder.py`)
|
||||
|
||||
Add a `PLATFORM_HINTS` entry so the agent knows what platform it's on:
|
||||
|
||||
```python
|
||||
PLATFORM_HINTS = {
|
||||
...
|
||||
"your_platform": (
|
||||
"You are on Your Platform. "
|
||||
"Describe formatting capabilities, media support, etc."
|
||||
),
|
||||
}
|
||||
```
|
||||
|
||||
Without this, the agent won't know it's on your platform and may use
|
||||
inappropriate formatting (e.g., markdown on platforms that don't render it).
|
||||
|
||||
---
|
||||
|
||||
## 7. Toolset (`toolsets.py`)
|
||||
|
||||
Add a named toolset for your platform:
|
||||
|
||||
```python
|
||||
"hermes-your-platform": {
|
||||
"description": "Your Platform bot toolset",
|
||||
"tools": _HERMES_CORE_TOOLS,
|
||||
"includes": []
|
||||
},
|
||||
```
|
||||
|
||||
And add it to the `hermes-gateway` composite:
|
||||
|
||||
```python
|
||||
"hermes-gateway": {
|
||||
"includes": [..., "hermes-your-platform"]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Cron Delivery (`cron/scheduler.py`)
|
||||
|
||||
Add to `platform_map` in `_deliver_result()`:
|
||||
|
||||
```python
|
||||
platform_map = {
|
||||
...
|
||||
"your_platform": Platform.YOUR_PLATFORM,
|
||||
}
|
||||
```
|
||||
|
||||
Without this, `cronjob(action="create", deliver="your_platform", ...)` silently fails.
|
||||
|
||||
---
|
||||
|
||||
## 9. Send Message Tool (`tools/send_message_tool.py`)
|
||||
|
||||
Add to `platform_map` in `send_message_tool()`:
|
||||
|
||||
```python
|
||||
platform_map = {
|
||||
...
|
||||
"your_platform": Platform.YOUR_PLATFORM,
|
||||
}
|
||||
```
|
||||
|
||||
Add routing in `_send_to_platform()`:
|
||||
|
||||
```python
|
||||
elif platform == Platform.YOUR_PLATFORM:
|
||||
return await _send_your_platform(pconfig, chat_id, message)
|
||||
```
|
||||
|
||||
Implement `_send_your_platform()` — a standalone async function that sends
|
||||
a single message without requiring the full adapter (for use by cron jobs
|
||||
and the send_message tool outside the gateway process).
|
||||
|
||||
Update the tool schema `target` description to include your platform example.
|
||||
|
||||
---
|
||||
|
||||
## 10. Cronjob Tool Schema (`tools/cronjob_tools.py`)
|
||||
|
||||
Update the `deliver` parameter description and docstring to mention your
|
||||
platform as a delivery option.
|
||||
|
||||
---
|
||||
|
||||
## 11. Channel Directory (`gateway/channel_directory.py`)
|
||||
|
||||
If your platform can't enumerate chats (most can't), add it to the
|
||||
session-based discovery list:
|
||||
|
||||
```python
|
||||
for plat_name in ("telegram", "whatsapp", "signal", "your_platform"):
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Status Display (`hermes_cli/status.py`)
|
||||
|
||||
Add to the `platforms` dict in the Messaging Platforms section:
|
||||
|
||||
```python
|
||||
platforms = {
|
||||
...
|
||||
"Your Platform": ("YOUR_PLATFORM_TOKEN", "YOUR_PLATFORM_HOME_CHANNEL"),
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Gateway Setup Wizard (`hermes_cli/gateway.py`)
|
||||
|
||||
Add to the `_PLATFORMS` list:
|
||||
|
||||
```python
|
||||
{
|
||||
"key": "your_platform",
|
||||
"label": "Your Platform",
|
||||
"emoji": "📱",
|
||||
"token_var": "YOUR_PLATFORM_TOKEN",
|
||||
"setup_instructions": [...],
|
||||
"vars": [...],
|
||||
}
|
||||
```
|
||||
|
||||
If your platform needs custom setup logic (connectivity testing, QR codes,
|
||||
policy choices), add a `_setup_your_platform()` function and route to it
|
||||
in the platform selection switch.
|
||||
|
||||
Update `_platform_status()` if your platform's "configured" check differs
|
||||
from the standard `bool(get_env_value(token_var))`.
|
||||
|
||||
---
|
||||
|
||||
## 14. Phone/ID Redaction (`agent/redact.py`)
|
||||
|
||||
If your platform uses sensitive identifiers (phone numbers, etc.), add a
|
||||
regex pattern and redaction function to `agent/redact.py`. This ensures
|
||||
identifiers are masked in ALL log output, not just your adapter's logs.
|
||||
|
||||
---
|
||||
|
||||
## 15. Documentation
|
||||
|
||||
| File | What to update |
|
||||
|------|---------------|
|
||||
| `README.md` | Platform list in feature table + documentation table |
|
||||
| `AGENTS.md` | Gateway description + env var config section |
|
||||
| `website/docs/user-guide/messaging/<platform>.md` | **NEW** — Full setup guide (see existing platform docs for template) |
|
||||
| `website/docs/user-guide/messaging/index.md` | Architecture diagram, toolset table, security examples, Next Steps links |
|
||||
| `website/docs/reference/environment-variables.md` | All env vars for the platform |
|
||||
|
||||
---
|
||||
|
||||
## 16. Tests (`tests/gateway/test_<platform>.py`)
|
||||
|
||||
Recommended test coverage:
|
||||
|
||||
- Platform enum exists with correct value
|
||||
- Config loading from env vars via `_apply_env_overrides`
|
||||
- Adapter init (config parsing, allowlist handling, default values)
|
||||
- Helper functions (redaction, parsing, file type detection)
|
||||
- Session source round-trip (to_dict → from_dict)
|
||||
- Authorization integration (platform in allowlist maps)
|
||||
- Send message tool routing (platform in platform_map)
|
||||
|
||||
Optional but valuable:
|
||||
- Async tests for message handling flow (mock the platform API)
|
||||
- SSE/WebSocket reconnection logic
|
||||
- Attachment processing
|
||||
- Group message filtering
|
||||
|
||||
---
|
||||
|
||||
## Quick Verification
|
||||
|
||||
After implementing everything, verify with:
|
||||
|
||||
```bash
|
||||
# All tests pass
|
||||
python -m pytest tests/ -q
|
||||
|
||||
# Grep for your platform name to find any missed integration points
|
||||
grep -r "telegram\|discord\|whatsapp\|slack" gateway/ tools/ agent/ cron/ hermes_cli/ toolsets.py \
|
||||
--include="*.py" -l | sort -u
|
||||
# Check each file in the output — if it mentions other platforms but not yours, you missed it
|
||||
```
|
||||
@@ -0,0 +1,45 @@
|
||||
"""
|
||||
Platform adapters for messaging integrations.
|
||||
|
||||
Each adapter handles:
|
||||
- Receiving messages from a platform
|
||||
- Sending messages/responses back
|
||||
- Platform-specific authentication
|
||||
- Message formatting and media handling
|
||||
"""
|
||||
|
||||
from .base import BasePlatformAdapter, MessageEvent, SendResult
|
||||
|
||||
# QQAdapter and YuanbaoAdapter were previously imported eagerly here, but
|
||||
# nothing in the codebase consumes ``from gateway.platforms import
|
||||
# QQAdapter`` (every real call site uses the long-form path
|
||||
# ``from gateway.platforms.qqbot import QQAdapter``). The eager imports
|
||||
# pulled in qqbot's chunked-upload + keyboards + onboard machinery and
|
||||
# yuanbao's websocket stack — about 48 ms wall and ~8 MB RSS on every
|
||||
# CLI invocation, even ones that never touch a gateway adapter.
|
||||
#
|
||||
# Use PEP 562 module ``__getattr__`` to keep the public re-export working
|
||||
# while deferring the actual import to first attribute access. This is
|
||||
# 100% backward-compatible for any external code that still imports the
|
||||
# adapters from the package root.
|
||||
__all__ = [
|
||||
"BasePlatformAdapter",
|
||||
"MessageEvent",
|
||||
"SendResult",
|
||||
"QQAdapter",
|
||||
"YuanbaoAdapter",
|
||||
]
|
||||
|
||||
|
||||
def __getattr__(name):
|
||||
if name == "QQAdapter":
|
||||
from .qqbot import QQAdapter # noqa: F401
|
||||
return QQAdapter
|
||||
if name == "YuanbaoAdapter":
|
||||
from .yuanbao import YuanbaoAdapter # noqa: F401
|
||||
return YuanbaoAdapter
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
|
||||
def __dir__():
|
||||
return sorted(__all__)
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Shared HTTP client factory for long-lived platform adapters.
|
||||
|
||||
Gateway messaging platforms (QQ Bot, Feishu, WeCom, DingTalk, Signal,
|
||||
BlueBubbles, WeCom-callback) keep a persistent ``httpx.AsyncClient``
|
||||
alive for the adapter's lifetime. That amortises TLS/connection setup
|
||||
across many API calls, but it also means the process's file-descriptor
|
||||
pressure is sensitive to how aggressively the pool recycles idle keep-
|
||||
alive connections.
|
||||
|
||||
httpx's default ``keepalive_expiry`` is 5 seconds. On macOS behind
|
||||
Cloudflare Warp (and other transparent proxies), peer-initiated FIN can
|
||||
sit in ``CLOSE_WAIT`` longer than that before the local socket actually
|
||||
drains — which, multiplied across 7 long-lived adapters plus the LLM
|
||||
client and MCP clients, walks straight into the default 256 fd limit.
|
||||
See #18451.
|
||||
|
||||
``platform_httpx_limits()`` returns a tighter ``httpx.Limits`` the
|
||||
adapter factories use instead of the httpx default. The values chosen:
|
||||
|
||||
* ``max_keepalive_connections=10`` — plenty for any single adapter;
|
||||
platform APIs rarely parallelise beyond this.
|
||||
* ``keepalive_expiry=2.0`` — close idle sockets aggressively so a
|
||||
proxy's lingering CLOSE_WAIT window can't starve the process.
|
||||
|
||||
Override via ``HERMES_GATEWAY_HTTPX_KEEPALIVE_EXPIRY`` /
|
||||
``HERMES_GATEWAY_HTTPX_MAX_KEEPALIVE`` env vars when tuning under load.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
try:
|
||||
import httpx
|
||||
except ImportError: # pragma: no cover — optional dep
|
||||
httpx = None # type: ignore[assignment]
|
||||
|
||||
|
||||
_DEFAULT_KEEPALIVE_EXPIRY_S = 2.0
|
||||
_DEFAULT_MAX_KEEPALIVE = 10
|
||||
|
||||
|
||||
def platform_httpx_limits() -> "httpx.Limits | None":
|
||||
"""Return ``httpx.Limits`` tuned for persistent platform-adapter clients.
|
||||
|
||||
Returns ``None`` when httpx isn't importable, so callers can fall
|
||||
back to httpx's built-in default without a hard dependency on this
|
||||
helper being reachable.
|
||||
"""
|
||||
if httpx is None:
|
||||
return None
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
raw = os.environ.get(name, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
val = float(raw)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return val if val > 0 else default
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
raw = os.environ.get(name, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
val = int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return val if val > 0 else default
|
||||
|
||||
keepalive_expiry = _env_float(
|
||||
"HERMES_GATEWAY_HTTPX_KEEPALIVE_EXPIRY", _DEFAULT_KEEPALIVE_EXPIRY_S
|
||||
)
|
||||
max_keepalive = _env_int(
|
||||
"HERMES_GATEWAY_HTTPX_MAX_KEEPALIVE", _DEFAULT_MAX_KEEPALIVE
|
||||
)
|
||||
|
||||
return httpx.Limits(
|
||||
max_keepalive_connections=max_keepalive,
|
||||
# Leave max_connections at httpx default (100) — plenty of headroom.
|
||||
keepalive_expiry=keepalive_expiry,
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,878 @@
|
||||
"""
|
||||
Email platform adapter for the Hermes gateway.
|
||||
|
||||
Allows users to interact with Hermes by sending emails.
|
||||
Uses IMAP to receive and SMTP to send messages.
|
||||
|
||||
Environment variables:
|
||||
EMAIL_IMAP_HOST — IMAP server host (e.g., imap.gmail.com)
|
||||
EMAIL_IMAP_PORT — IMAP server port (default: 993)
|
||||
EMAIL_SMTP_HOST — SMTP server host (e.g., smtp.gmail.com)
|
||||
EMAIL_SMTP_PORT — SMTP server port (default: 587)
|
||||
EMAIL_ADDRESS — Email address for the agent
|
||||
EMAIL_PASSWORD — Email password or app-specific password
|
||||
EMAIL_POLL_INTERVAL — Seconds between mailbox checks (default: 15)
|
||||
EMAIL_ALLOWED_USERS — Comma-separated list of allowed sender addresses
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import email as email_lib
|
||||
import imaplib
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import smtplib
|
||||
import socket
|
||||
import ssl
|
||||
import uuid
|
||||
from email.header import decode_header
|
||||
from email.mime.multipart import MIMEMultipart
|
||||
from email.mime.text import MIMEText
|
||||
from email.mime.base import MIMEBase
|
||||
from email.utils import formatdate
|
||||
from email import encoders
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from gateway.platforms.base import (
|
||||
BasePlatformAdapter,
|
||||
MessageEvent,
|
||||
MessageType,
|
||||
SendResult,
|
||||
cache_document_from_bytes,
|
||||
cache_image_from_bytes,
|
||||
)
|
||||
from gateway.config import Platform, PlatformConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
# Automated sender patterns — emails from these are silently ignored
|
||||
_NOREPLY_PATTERNS = (
|
||||
"noreply", "no-reply", "no_reply", "donotreply", "do-not-reply",
|
||||
"mailer-daemon", "postmaster", "bounce", "notifications@",
|
||||
"automated@", "auto-confirm", "auto-reply", "automailer",
|
||||
)
|
||||
|
||||
# RFC headers that indicate bulk/automated mail
|
||||
_AUTOMATED_HEADERS = {
|
||||
"Auto-Submitted": lambda v: v.lower() != "no",
|
||||
"Precedence": lambda v: v.lower() in {"bulk", "list", "junk"},
|
||||
"X-Auto-Response-Suppress": lambda v: bool(v),
|
||||
"List-Unsubscribe": lambda v: bool(v),
|
||||
}
|
||||
|
||||
# Gmail-safe max length per email body
|
||||
MAX_MESSAGE_LENGTH = 50_000
|
||||
|
||||
SMTP_CONNECT_TIMEOUT = 30
|
||||
|
||||
|
||||
def _create_ipv4_connection(
|
||||
host: str,
|
||||
port: int,
|
||||
timeout: float,
|
||||
source_address: Any = None,
|
||||
) -> socket.socket:
|
||||
"""Create a TCP connection using only IPv4 addresses.
|
||||
|
||||
This mirrors ``socket.create_connection`` but constrains DNS resolution to
|
||||
``AF_INET``. It avoids mutating process-global socket functions, which
|
||||
matters because email sends run in executor threads.
|
||||
"""
|
||||
last_error: OSError | None = None
|
||||
for family, socktype, proto, _canonname, sockaddr in socket.getaddrinfo(
|
||||
host, port, socket.AF_INET, socket.SOCK_STREAM
|
||||
):
|
||||
sock = socket.socket(family, socktype, proto)
|
||||
sock.settimeout(timeout)
|
||||
try:
|
||||
if source_address:
|
||||
sock.bind(source_address)
|
||||
sock.connect(sockaddr)
|
||||
return sock
|
||||
except OSError as exc:
|
||||
last_error = exc
|
||||
sock.close()
|
||||
if last_error is not None:
|
||||
raise last_error
|
||||
raise OSError(f"No IPv4 address found for {host}:{port}")
|
||||
|
||||
|
||||
class _IPv4SMTP(smtplib.SMTP):
|
||||
def _get_socket(self, host, port, timeout): # type: ignore[override]
|
||||
return _create_ipv4_connection(
|
||||
host,
|
||||
port,
|
||||
timeout,
|
||||
source_address=self.source_address,
|
||||
)
|
||||
|
||||
|
||||
class _IPv4SMTP_SSL(smtplib.SMTP_SSL):
|
||||
def _get_socket(self, host, port, timeout): # type: ignore[override]
|
||||
raw_sock = _create_ipv4_connection(
|
||||
host,
|
||||
port,
|
||||
timeout,
|
||||
source_address=self.source_address,
|
||||
)
|
||||
return self.context.wrap_socket(
|
||||
raw_sock,
|
||||
server_hostname=getattr(self, "_host", host),
|
||||
)
|
||||
|
||||
# Supported image extensions for inline detection
|
||||
_IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".gif", ".webp"}
|
||||
|
||||
def _send_imap_id(imap: "imaplib.IMAP4") -> None:
|
||||
"""Send RFC 2971 IMAP ID command identifying this client.
|
||||
|
||||
Required by 163/NetEase mailbox after LOGIN: without it, every UID
|
||||
SEARCH/FETCH returns ``BYE Unsafe Login`` and disconnects. Other
|
||||
IMAP servers either honor it silently or reject the unknown command;
|
||||
we swallow failures so non-supporting servers keep working.
|
||||
"""
|
||||
try:
|
||||
try:
|
||||
from hermes_cli import __version__ as _hermes_version
|
||||
except Exception: # noqa: BLE001 — keep ID best-effort if import fails
|
||||
_hermes_version = "0"
|
||||
imap.xatom(
|
||||
"ID",
|
||||
f'("name" "hermes-agent" "version" "{_hermes_version}" '
|
||||
'"vendor" "NousResearch" '
|
||||
'"support-email" "noreply@nousresearch.com")',
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 — best-effort, never fatal
|
||||
logger.debug("[Email] IMAP ID command not accepted: %s", e)
|
||||
|
||||
|
||||
def _is_automated_sender(address: str, headers: dict) -> bool:
|
||||
"""Return True if this email is from an automated/noreply source."""
|
||||
addr = address.lower()
|
||||
if any(pattern in addr for pattern in _NOREPLY_PATTERNS):
|
||||
return True
|
||||
for header, check in _AUTOMATED_HEADERS.items():
|
||||
value = headers.get(header, "")
|
||||
if value and check(value):
|
||||
return True
|
||||
return False
|
||||
|
||||
def check_email_requirements() -> bool:
|
||||
"""Check if email platform dependencies are available."""
|
||||
addr = os.getenv("EMAIL_ADDRESS")
|
||||
pwd = os.getenv("EMAIL_PASSWORD")
|
||||
imap = os.getenv("EMAIL_IMAP_HOST")
|
||||
smtp = os.getenv("EMAIL_SMTP_HOST")
|
||||
if not all([addr, pwd, imap, smtp]):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _decode_header_value(raw: str) -> str:
|
||||
"""Decode an RFC 2047 encoded email header into a plain string."""
|
||||
parts = decode_header(raw)
|
||||
decoded = []
|
||||
for part, charset in parts:
|
||||
if isinstance(part, bytes):
|
||||
decoded.append(part.decode(charset or "utf-8", errors="replace"))
|
||||
else:
|
||||
decoded.append(part)
|
||||
return " ".join(decoded)
|
||||
|
||||
|
||||
def _extract_text_body(msg: email_lib.message.Message) -> str:
|
||||
"""Extract the plain-text body from a potentially multipart email."""
|
||||
if msg.is_multipart():
|
||||
for part in msg.walk():
|
||||
content_type = part.get_content_type()
|
||||
disposition = str(part.get("Content-Disposition", ""))
|
||||
# Skip attachments
|
||||
if "attachment" in disposition:
|
||||
continue
|
||||
if content_type == "text/plain":
|
||||
payload = part.get_payload(decode=True)
|
||||
if payload:
|
||||
charset = part.get_content_charset() or "utf-8"
|
||||
return payload.decode(charset, errors="replace")
|
||||
# Fallback: try text/html and strip tags
|
||||
for part in msg.walk():
|
||||
content_type = part.get_content_type()
|
||||
disposition = str(part.get("Content-Disposition", ""))
|
||||
if "attachment" in disposition:
|
||||
continue
|
||||
if content_type == "text/html":
|
||||
payload = part.get_payload(decode=True)
|
||||
if payload:
|
||||
charset = part.get_content_charset() or "utf-8"
|
||||
html = payload.decode(charset, errors="replace")
|
||||
return _strip_html(html)
|
||||
return ""
|
||||
else:
|
||||
payload = msg.get_payload(decode=True)
|
||||
if payload:
|
||||
charset = msg.get_content_charset() or "utf-8"
|
||||
text = payload.decode(charset, errors="replace")
|
||||
if msg.get_content_type() == "text/html":
|
||||
return _strip_html(text)
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
def _strip_html(html: str) -> str:
|
||||
"""Naive HTML tag stripper for fallback text extraction."""
|
||||
text = re.sub(r"<br\s*/?>", "\n", html, flags=re.IGNORECASE)
|
||||
text = re.sub(r"<p[^>]*>", "\n", text, flags=re.IGNORECASE)
|
||||
text = re.sub(r"</p>", "\n", text, flags=re.IGNORECASE)
|
||||
text = re.sub(r"<[^>]+>", "", text)
|
||||
text = re.sub(r" ", " ", text)
|
||||
text = re.sub(r"&", "&", text)
|
||||
text = re.sub(r"<", "<", text)
|
||||
text = re.sub(r">", ">", text)
|
||||
text = re.sub(r"\n{3,}", "\n\n", text)
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _extract_email_address(raw: str) -> str:
|
||||
"""Extract bare email address from 'Name <addr>' format."""
|
||||
match = re.search(r"<([^>]+)>", raw)
|
||||
if match:
|
||||
return match.group(1).strip().lower()
|
||||
return raw.strip().lower()
|
||||
|
||||
|
||||
def _extract_attachments(
|
||||
msg: email_lib.message.Message,
|
||||
skip_attachments: bool = False,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Extract attachment metadata and cache files locally.
|
||||
|
||||
When *skip_attachments* is True, all attachment/inline parts are ignored
|
||||
(useful for malware protection or bandwidth savings).
|
||||
"""
|
||||
attachments = []
|
||||
if not msg.is_multipart():
|
||||
return attachments
|
||||
|
||||
for part in msg.walk():
|
||||
disposition = str(part.get("Content-Disposition", ""))
|
||||
if skip_attachments and ("attachment" in disposition or "inline" in disposition):
|
||||
continue
|
||||
if "attachment" not in disposition and "inline" not in disposition:
|
||||
continue
|
||||
# Skip text/plain and text/html body parts
|
||||
content_type = part.get_content_type()
|
||||
if content_type in {"text/plain", "text/html"} and "attachment" not in disposition:
|
||||
continue
|
||||
|
||||
filename = part.get_filename()
|
||||
if filename:
|
||||
filename = _decode_header_value(filename)
|
||||
else:
|
||||
ext = part.get_content_subtype() or "bin"
|
||||
filename = f"attachment.{ext}"
|
||||
|
||||
payload = part.get_payload(decode=True)
|
||||
if not payload:
|
||||
continue
|
||||
|
||||
ext = Path(filename).suffix.lower()
|
||||
if ext in _IMAGE_EXTS:
|
||||
try:
|
||||
cached_path = cache_image_from_bytes(payload, ext)
|
||||
except ValueError:
|
||||
logger.debug("Skipping non-image attachment %s (invalid magic bytes)", filename)
|
||||
continue
|
||||
attachments.append({
|
||||
"path": cached_path,
|
||||
"filename": filename,
|
||||
"type": "image",
|
||||
"media_type": content_type,
|
||||
})
|
||||
else:
|
||||
cached_path = cache_document_from_bytes(payload, filename)
|
||||
attachments.append({
|
||||
"path": cached_path,
|
||||
"filename": filename,
|
||||
"type": "document",
|
||||
"media_type": content_type,
|
||||
})
|
||||
|
||||
return attachments
|
||||
|
||||
|
||||
class EmailAdapter(BasePlatformAdapter):
|
||||
"""Email gateway adapter using IMAP (receive) and SMTP (send)."""
|
||||
|
||||
def __init__(self, config: PlatformConfig):
|
||||
super().__init__(config, Platform.EMAIL)
|
||||
|
||||
self._address = os.getenv("EMAIL_ADDRESS", "")
|
||||
self._password = os.getenv("EMAIL_PASSWORD", "")
|
||||
self._imap_host = os.getenv("EMAIL_IMAP_HOST", "")
|
||||
self._imap_port = int(os.getenv("EMAIL_IMAP_PORT", "993"))
|
||||
self._smtp_host = os.getenv("EMAIL_SMTP_HOST", "")
|
||||
self._smtp_port = int(os.getenv("EMAIL_SMTP_PORT", "587"))
|
||||
self._poll_interval = int(os.getenv("EMAIL_POLL_INTERVAL", "15"))
|
||||
|
||||
# Skip attachments — configured via config.yaml:
|
||||
# platforms:
|
||||
# email:
|
||||
# skip_attachments: true
|
||||
extra = config.extra or {}
|
||||
self._skip_attachments = extra.get("skip_attachments", False)
|
||||
|
||||
# Track message IDs we've already processed to avoid duplicates
|
||||
self._seen_uids: set = set()
|
||||
self._seen_uids_max: int = 2000 # cap to prevent unbounded memory growth
|
||||
self._poll_task: Optional[asyncio.Task] = None
|
||||
|
||||
# Map chat_id (sender email) -> last subject + message-id for threading
|
||||
self._thread_context: Dict[str, Dict[str, str]] = {}
|
||||
|
||||
logger.info("[Email] Adapter initialized for %s", self._address)
|
||||
|
||||
def _trim_seen_uids(self) -> None:
|
||||
"""Keep only the most recent UIDs to prevent unbounded memory growth.
|
||||
|
||||
IMAP UIDs are monotonically increasing integers. When the set grows
|
||||
beyond the cap, we keep only the highest half — old UIDs are safe to
|
||||
drop because new messages always have higher UIDs and IMAP's UNSEEN
|
||||
flag prevents re-delivery regardless.
|
||||
"""
|
||||
if len(self._seen_uids) <= self._seen_uids_max:
|
||||
return
|
||||
try:
|
||||
# UIDs are bytes like b'1234' — sort numerically and keep top half
|
||||
sorted_uids = sorted(self._seen_uids, key=lambda u: int(u))
|
||||
keep = self._seen_uids_max // 2
|
||||
self._seen_uids = set(sorted_uids[-keep:])
|
||||
logger.debug("[Email] Trimmed seen UIDs to %d entries", len(self._seen_uids))
|
||||
except (ValueError, TypeError):
|
||||
# Fallback: just clear old entries if sort fails
|
||||
self._seen_uids = set(list(self._seen_uids)[-self._seen_uids_max // 2:])
|
||||
|
||||
def _connect_smtp(self) -> smtplib.SMTP:
|
||||
"""Create an SMTP connection, selecting the correct protocol for the port.
|
||||
|
||||
Port 465 uses implicit TLS (``SMTP_SSL``). All other ports use
|
||||
``SMTP`` + ``STARTTLS``.
|
||||
|
||||
When the host resolves to an IPv6 address that is unreachable
|
||||
(common on networks without IPv6 routing), the default connection can
|
||||
hang until the socket timeout expires. We retry connection-level
|
||||
failures through an IPv4-only socket path, without mutating global
|
||||
resolver state. TLS verification errors are not retried.
|
||||
|
||||
Returns a connected SMTP object with TLS established — callers
|
||||
can proceed directly to ``login()``.
|
||||
"""
|
||||
ctx = ssl.create_default_context()
|
||||
host = self._smtp_host
|
||||
port = self._smtp_port
|
||||
|
||||
def _connect(*, ipv4_only: bool = False) -> smtplib.SMTP:
|
||||
"""Attempt one SMTP connection."""
|
||||
smtp_cls = _IPv4SMTP if ipv4_only else smtplib.SMTP
|
||||
smtp_ssl_cls = _IPv4SMTP_SSL if ipv4_only else smtplib.SMTP_SSL
|
||||
if port == 465:
|
||||
return smtp_ssl_cls(host, port, timeout=SMTP_CONNECT_TIMEOUT, context=ctx)
|
||||
smtp = smtp_cls(host, port, timeout=SMTP_CONNECT_TIMEOUT)
|
||||
try:
|
||||
smtp.starttls(context=ctx)
|
||||
except Exception:
|
||||
smtp.close()
|
||||
raise
|
||||
return smtp
|
||||
|
||||
try:
|
||||
return _connect()
|
||||
except (socket.timeout, TimeoutError, ConnectionError, OSError) as exc:
|
||||
if isinstance(exc, ssl.SSLError):
|
||||
raise
|
||||
# Connection-level failure (may be unreachable IPv6).
|
||||
# Retry with IPv4 only.
|
||||
return _connect(ipv4_only=True)
|
||||
|
||||
async def connect(self) -> bool:
|
||||
"""Connect to the IMAP server and start polling for new messages."""
|
||||
try:
|
||||
# Test IMAP connection
|
||||
imap = imaplib.IMAP4_SSL(self._imap_host, self._imap_port, timeout=30)
|
||||
imap.login(self._address, self._password)
|
||||
_send_imap_id(imap)
|
||||
# Mark all existing messages as seen so we only process new ones
|
||||
imap.select("INBOX")
|
||||
status, data = imap.uid("search", None, "ALL")
|
||||
if status == "OK" and data and data[0]:
|
||||
for uid in data[0].split():
|
||||
self._seen_uids.add(uid)
|
||||
# Keep only the most recent UIDs to prevent unbounded growth
|
||||
self._trim_seen_uids()
|
||||
imap.logout()
|
||||
logger.info("[Email] IMAP connection test passed. %d existing messages skipped.", len(self._seen_uids))
|
||||
except Exception as e:
|
||||
logger.error("[Email] IMAP connection failed: %s", e)
|
||||
return False
|
||||
|
||||
try:
|
||||
# Test SMTP connection
|
||||
smtp = self._connect_smtp()
|
||||
try:
|
||||
smtp.login(self._address, self._password)
|
||||
finally:
|
||||
smtp.quit()
|
||||
logger.info("[Email] SMTP connection test passed.")
|
||||
except Exception as e:
|
||||
logger.error("[Email] SMTP connection failed: %s", e)
|
||||
return False
|
||||
|
||||
self._running = True
|
||||
self._poll_task = asyncio.create_task(self._poll_loop())
|
||||
print(f"[Email] Connected as {self._address}")
|
||||
return True
|
||||
|
||||
async def disconnect(self) -> None:
|
||||
"""Stop polling and disconnect."""
|
||||
self._running = False
|
||||
if self._poll_task:
|
||||
self._poll_task.cancel()
|
||||
try:
|
||||
await self._poll_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._poll_task = None
|
||||
logger.info("[Email] Disconnected.")
|
||||
|
||||
async def _poll_loop(self) -> None:
|
||||
"""Poll IMAP for new messages at regular intervals."""
|
||||
while self._running:
|
||||
try:
|
||||
await self._check_inbox()
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error("[Email] Poll error: %s", e)
|
||||
await asyncio.sleep(self._poll_interval)
|
||||
|
||||
async def _check_inbox(self) -> None:
|
||||
"""Check INBOX for unseen messages and dispatch them."""
|
||||
# Run IMAP operations in a thread to avoid blocking the event loop
|
||||
loop = asyncio.get_running_loop()
|
||||
messages = await loop.run_in_executor(None, self._fetch_new_messages)
|
||||
for msg_data in messages:
|
||||
await self._dispatch_message(msg_data)
|
||||
|
||||
def _fetch_new_messages(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch new (unseen) messages from IMAP. Runs in executor thread."""
|
||||
results = []
|
||||
try:
|
||||
imap = imaplib.IMAP4_SSL(self._imap_host, self._imap_port, timeout=30)
|
||||
try:
|
||||
imap.login(self._address, self._password)
|
||||
_send_imap_id(imap)
|
||||
imap.select("INBOX")
|
||||
|
||||
status, data = imap.uid("search", None, "UNSEEN")
|
||||
if status != "OK" or not data or not data[0]:
|
||||
return results
|
||||
|
||||
for uid in data[0].split():
|
||||
if uid in self._seen_uids:
|
||||
continue
|
||||
self._seen_uids.add(uid)
|
||||
# Trim periodically to prevent unbounded memory growth
|
||||
if len(self._seen_uids) > self._seen_uids_max:
|
||||
self._trim_seen_uids()
|
||||
|
||||
status, msg_data = imap.uid("fetch", uid, "(RFC822)")
|
||||
if status != "OK":
|
||||
continue
|
||||
|
||||
raw_email = msg_data[0][1]
|
||||
msg = email_lib.message_from_bytes(raw_email)
|
||||
|
||||
sender_raw = msg.get("From", "")
|
||||
sender_addr = _extract_email_address(sender_raw)
|
||||
sender_name = _decode_header_value(sender_raw)
|
||||
# Remove email from name if present
|
||||
if "<" in sender_name:
|
||||
sender_name = sender_name.split("<")[0].strip().strip('"')
|
||||
|
||||
subject = _decode_header_value(msg.get("Subject", "(no subject)"))
|
||||
message_id = msg.get("Message-ID", "")
|
||||
in_reply_to = msg.get("In-Reply-To", "")
|
||||
# Skip automated/noreply senders before any processing
|
||||
msg_headers = dict(msg.items())
|
||||
if _is_automated_sender(sender_addr, msg_headers):
|
||||
logger.debug("[Email] Skipping automated sender: %s", sender_addr)
|
||||
continue
|
||||
body = _extract_text_body(msg)
|
||||
attachments = _extract_attachments(msg, skip_attachments=self._skip_attachments)
|
||||
|
||||
results.append({
|
||||
"uid": uid,
|
||||
"sender_addr": sender_addr,
|
||||
"sender_name": sender_name,
|
||||
"subject": subject,
|
||||
"message_id": message_id,
|
||||
"in_reply_to": in_reply_to,
|
||||
"body": body,
|
||||
"attachments": attachments,
|
||||
"date": msg.get("Date", ""),
|
||||
})
|
||||
finally:
|
||||
try:
|
||||
imap.logout()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.error("[Email] IMAP fetch error: %s", e)
|
||||
return results
|
||||
|
||||
async def _dispatch_message(self, msg_data: Dict[str, Any]) -> None:
|
||||
"""Convert a fetched email into a MessageEvent and dispatch it."""
|
||||
sender_addr = msg_data["sender_addr"]
|
||||
|
||||
# Skip self-messages
|
||||
if sender_addr == self._address.lower():
|
||||
return
|
||||
|
||||
# Never reply to automated senders
|
||||
if _is_automated_sender(sender_addr, {}):
|
||||
logger.debug("[Email] Dropping automated sender at dispatch: %s", sender_addr)
|
||||
return
|
||||
|
||||
# Skip senders not in EMAIL_ALLOWED_USERS — prevents the adapter
|
||||
# from creating a MessageEvent (and thus thread context) for senders
|
||||
# that the gateway will never authorize. Without this early guard,
|
||||
# a race between dispatch and authorization can result in the adapter
|
||||
# sending a reply even though the handler returned None.
|
||||
allowed_raw = os.getenv("EMAIL_ALLOWED_USERS", "").strip()
|
||||
if allowed_raw:
|
||||
allowed = {addr.strip().lower() for addr in allowed_raw.split(",") if addr.strip()}
|
||||
if sender_addr.lower() not in allowed:
|
||||
logger.debug("[Email] Dropping non-allowlisted sender at dispatch: %s", sender_addr)
|
||||
return
|
||||
|
||||
subject = msg_data["subject"]
|
||||
body = msg_data["body"].strip()
|
||||
attachments = msg_data["attachments"]
|
||||
|
||||
# Build message text: include subject as context
|
||||
text = body
|
||||
if subject and not subject.startswith("Re:"):
|
||||
text = f"[Subject: {subject}]\n\n{body}"
|
||||
|
||||
# Determine message type and media
|
||||
media_urls = []
|
||||
media_types = []
|
||||
msg_type = MessageType.TEXT
|
||||
|
||||
for att in attachments:
|
||||
media_urls.append(att["path"])
|
||||
media_types.append(att["media_type"])
|
||||
if att["type"] == "image" and msg_type == MessageType.TEXT:
|
||||
msg_type = MessageType.PHOTO
|
||||
elif att["type"] == "document":
|
||||
# Document wins over PHOTO for mixed attachments: run.py's
|
||||
# image handling keys off the per-path image/* mime type
|
||||
# regardless of message_type, but document-context injection
|
||||
# gates strictly on MessageType.DOCUMENT — so DOCUMENT is the
|
||||
# only classification that surfaces both.
|
||||
msg_type = MessageType.DOCUMENT
|
||||
|
||||
# Store thread context for reply threading
|
||||
self._thread_context[sender_addr] = {
|
||||
"subject": subject,
|
||||
"message_id": msg_data["message_id"],
|
||||
}
|
||||
|
||||
source = self.build_source(
|
||||
chat_id=sender_addr,
|
||||
chat_name=msg_data["sender_name"] or sender_addr,
|
||||
chat_type="dm",
|
||||
user_id=sender_addr,
|
||||
user_name=msg_data["sender_name"] or sender_addr,
|
||||
)
|
||||
|
||||
event = MessageEvent(
|
||||
text=text or "(empty email)",
|
||||
message_type=msg_type,
|
||||
source=source,
|
||||
message_id=msg_data["message_id"],
|
||||
media_urls=media_urls,
|
||||
media_types=media_types,
|
||||
reply_to_message_id=msg_data["in_reply_to"] or None,
|
||||
)
|
||||
|
||||
logger.info("[Email] New message from %s: %s", sender_addr, subject)
|
||||
await self.handle_message(event)
|
||||
|
||||
async def send(
|
||||
self,
|
||||
chat_id: str,
|
||||
content: str,
|
||||
reply_to: Optional[str] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> SendResult:
|
||||
"""Send an email reply to the given address."""
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
message_id = await loop.run_in_executor(
|
||||
None, self._send_email, chat_id, content, reply_to
|
||||
)
|
||||
return SendResult(success=True, message_id=message_id)
|
||||
except Exception as e:
|
||||
logger.error("[Email] Send failed to %s: %s", chat_id, e)
|
||||
return SendResult(success=False, error=str(e))
|
||||
|
||||
def _send_email(
|
||||
self,
|
||||
to_addr: str,
|
||||
body: str,
|
||||
reply_to_msg_id: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Send an email via SMTP. Runs in executor thread."""
|
||||
msg = MIMEMultipart()
|
||||
msg["From"] = self._address
|
||||
msg["To"] = to_addr
|
||||
|
||||
# Thread context for reply
|
||||
ctx = self._thread_context.get(to_addr, {})
|
||||
subject = ctx.get("subject", "Hermes Agent")
|
||||
if not subject.startswith("Re:"):
|
||||
subject = f"Re: {subject}"
|
||||
msg["Subject"] = subject
|
||||
|
||||
# Threading headers
|
||||
original_msg_id = reply_to_msg_id or ctx.get("message_id")
|
||||
if original_msg_id:
|
||||
msg["In-Reply-To"] = original_msg_id
|
||||
msg["References"] = original_msg_id
|
||||
|
||||
msg["Date"] = formatdate(localtime=True)
|
||||
msg_id = f"<hermes-{uuid.uuid4().hex[:12]}@{self._address.split('@')[1]}>"
|
||||
msg["Message-ID"] = msg_id
|
||||
|
||||
msg.attach(MIMEText(body, "plain", "utf-8"))
|
||||
|
||||
smtp = self._connect_smtp()
|
||||
try:
|
||||
smtp.login(self._address, self._password)
|
||||
smtp.send_message(msg)
|
||||
finally:
|
||||
try:
|
||||
smtp.quit()
|
||||
except Exception:
|
||||
smtp.close()
|
||||
|
||||
logger.info("[Email] Sent reply to %s (subject: %s)", to_addr, subject)
|
||||
return msg_id
|
||||
|
||||
async def send_typing(self, chat_id: str, metadata: Optional[Dict[str, Any]] = None) -> None:
|
||||
"""Email has no typing indicator — no-op."""
|
||||
|
||||
async def send_image(
|
||||
self,
|
||||
chat_id: str,
|
||||
image_url: str,
|
||||
caption: Optional[str] = None,
|
||||
reply_to: Optional[str] = None,
|
||||
) -> SendResult:
|
||||
"""Send an image URL as part of an email body."""
|
||||
text = caption or ""
|
||||
text += f"\n\nImage: {image_url}"
|
||||
return await self.send(chat_id, text.strip(), reply_to)
|
||||
|
||||
async def send_multiple_images(
|
||||
self,
|
||||
chat_id: str,
|
||||
images: List[Tuple[str, str]],
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
human_delay: float = 0.0,
|
||||
) -> None:
|
||||
"""Send a batch of images as a single email with multiple MIME attachments.
|
||||
|
||||
Local files are attached directly. URL images have their URL
|
||||
appended to the body (email adapter does not download remote
|
||||
images). No hard cap — email clients handle dozens of
|
||||
attachments fine, subject to SMTP message size limits.
|
||||
"""
|
||||
if not images:
|
||||
return
|
||||
|
||||
from urllib.parse import unquote as _unquote
|
||||
|
||||
body_parts: List[str] = []
|
||||
local_paths: List[str] = []
|
||||
for image_url, alt_text in images:
|
||||
if alt_text:
|
||||
body_parts.append(alt_text)
|
||||
if image_url.startswith("file://"):
|
||||
local_path = _unquote(image_url[7:])
|
||||
if Path(local_path).exists():
|
||||
local_paths.append(local_path)
|
||||
else:
|
||||
logger.warning("[Email] Skipping missing image: %s", local_path)
|
||||
else:
|
||||
# Remote URLs just get linked in the body (parity with send_image)
|
||||
body_parts.append(f"Image: {image_url}")
|
||||
|
||||
if not local_paths and not body_parts:
|
||||
return
|
||||
|
||||
body = "\n\n".join(body_parts)
|
||||
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(
|
||||
None,
|
||||
self._send_email_with_attachments,
|
||||
chat_id,
|
||||
body,
|
||||
local_paths,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("[Email] Multi-image send failed, falling back: %s", e, exc_info=True)
|
||||
await super().send_multiple_images(chat_id, images, metadata, human_delay)
|
||||
|
||||
def _send_email_with_attachments(
|
||||
self,
|
||||
to_addr: str,
|
||||
body: str,
|
||||
file_paths: List[str],
|
||||
) -> str:
|
||||
"""Send an email with multiple file attachments via SMTP."""
|
||||
msg = MIMEMultipart()
|
||||
msg["From"] = self._address
|
||||
msg["To"] = to_addr
|
||||
|
||||
ctx = self._thread_context.get(to_addr, {})
|
||||
subject = ctx.get("subject", "Hermes Agent")
|
||||
if not subject.startswith("Re:"):
|
||||
subject = f"Re: {subject}"
|
||||
msg["Subject"] = subject
|
||||
|
||||
original_msg_id = ctx.get("message_id")
|
||||
if original_msg_id:
|
||||
msg["In-Reply-To"] = original_msg_id
|
||||
msg["References"] = original_msg_id
|
||||
|
||||
msg["Date"] = formatdate(localtime=True)
|
||||
msg_id = f"<hermes-{uuid.uuid4().hex[:12]}@{self._address.split('@')[1]}>"
|
||||
msg["Message-ID"] = msg_id
|
||||
|
||||
if body:
|
||||
msg.attach(MIMEText(body, "plain", "utf-8"))
|
||||
|
||||
for file_path in file_paths:
|
||||
p = Path(file_path)
|
||||
try:
|
||||
with open(p, "rb") as f:
|
||||
part = MIMEBase("application", "octet-stream")
|
||||
part.set_payload(f.read())
|
||||
encoders.encode_base64(part)
|
||||
part.add_header("Content-Disposition", f"attachment; filename={p.name}")
|
||||
msg.attach(part)
|
||||
except Exception as e:
|
||||
logger.warning("[Email] Failed to attach %s: %s", file_path, e)
|
||||
|
||||
smtp = self._connect_smtp()
|
||||
try:
|
||||
smtp.login(self._address, self._password)
|
||||
smtp.send_message(msg)
|
||||
finally:
|
||||
try:
|
||||
smtp.quit()
|
||||
except Exception:
|
||||
smtp.close()
|
||||
|
||||
logger.info("[Email] Sent multi-attachment email to %s (%d files)", to_addr, len(file_paths))
|
||||
return msg_id
|
||||
|
||||
async def send_document(
|
||||
self,
|
||||
chat_id: str,
|
||||
file_path: str,
|
||||
caption: Optional[str] = None,
|
||||
file_name: Optional[str] = None,
|
||||
reply_to: Optional[str] = None,
|
||||
**kwargs,
|
||||
) -> SendResult:
|
||||
"""Send a file as an email attachment."""
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
message_id = await loop.run_in_executor(
|
||||
None,
|
||||
self._send_email_with_attachment,
|
||||
chat_id,
|
||||
caption or "",
|
||||
file_path,
|
||||
file_name,
|
||||
)
|
||||
return SendResult(success=True, message_id=message_id)
|
||||
except Exception as e:
|
||||
logger.error("[Email] Send document failed: %s", e)
|
||||
return SendResult(success=False, error=str(e))
|
||||
|
||||
def _send_email_with_attachment(
|
||||
self,
|
||||
to_addr: str,
|
||||
body: str,
|
||||
file_path: str,
|
||||
file_name: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Send an email with a file attachment via SMTP."""
|
||||
msg = MIMEMultipart()
|
||||
msg["From"] = self._address
|
||||
msg["To"] = to_addr
|
||||
|
||||
ctx = self._thread_context.get(to_addr, {})
|
||||
subject = ctx.get("subject", "Hermes Agent")
|
||||
if not subject.startswith("Re:"):
|
||||
subject = f"Re: {subject}"
|
||||
msg["Subject"] = subject
|
||||
|
||||
original_msg_id = ctx.get("message_id")
|
||||
if original_msg_id:
|
||||
msg["In-Reply-To"] = original_msg_id
|
||||
msg["References"] = original_msg_id
|
||||
|
||||
msg["Date"] = formatdate(localtime=True)
|
||||
msg_id = f"<hermes-{uuid.uuid4().hex[:12]}@{self._address.split('@')[1]}>"
|
||||
msg["Message-ID"] = msg_id
|
||||
|
||||
if body:
|
||||
msg.attach(MIMEText(body, "plain", "utf-8"))
|
||||
|
||||
# Attach file
|
||||
p = Path(file_path)
|
||||
fname = file_name or p.name
|
||||
with open(p, "rb") as f:
|
||||
part = MIMEBase("application", "octet-stream")
|
||||
part.set_payload(f.read())
|
||||
encoders.encode_base64(part)
|
||||
part.add_header("Content-Disposition", f"attachment; filename={fname}")
|
||||
msg.attach(part)
|
||||
|
||||
smtp = self._connect_smtp()
|
||||
try:
|
||||
smtp.login(self._address, self._password)
|
||||
smtp.send_message(msg)
|
||||
finally:
|
||||
try:
|
||||
smtp.quit()
|
||||
except Exception:
|
||||
smtp.close()
|
||||
|
||||
return msg_id
|
||||
|
||||
async def get_chat_info(self, chat_id: str) -> Dict[str, Any]:
|
||||
"""Return basic info about the email chat."""
|
||||
ctx = self._thread_context.get(chat_id, {})
|
||||
return {
|
||||
"name": chat_id,
|
||||
"type": "dm",
|
||||
"chat_id": chat_id,
|
||||
"subject": ctx.get("subject", ""),
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,429 @@
|
||||
"""
|
||||
Feishu document comment access-control rules.
|
||||
|
||||
3-tier rule resolution: exact doc > wildcard "*" > top-level > code defaults.
|
||||
Each field (enabled/policy/allow_from) falls back independently.
|
||||
Config: ~/.hermes/feishu_comment_rules.json (mtime-cached, hot-reload).
|
||||
Pairing store: ~/.hermes/feishu_comment_pairing.json.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from hermes_constants import get_hermes_home
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Paths
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# Uses the canonical ``get_hermes_home()`` helper (HERMES_HOME-aware and
|
||||
# profile-safe). Resolved at import time; this module is lazy-imported by
|
||||
# the Feishu comment event handler, which runs long after profile overrides
|
||||
# have been applied, so freezing paths here is safe.
|
||||
|
||||
RULES_FILE = get_hermes_home() / "feishu_comment_rules.json"
|
||||
PAIRING_FILE = get_hermes_home() / "feishu_comment_pairing.json"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_VALID_POLICIES = ("allowlist", "pairing")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CommentDocumentRule:
|
||||
"""Per-document rule. ``None`` means 'inherit from lower tier'."""
|
||||
enabled: Optional[bool] = None
|
||||
policy: Optional[str] = None
|
||||
allow_from: Optional[frozenset] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CommentsConfig:
|
||||
"""Top-level comment access config."""
|
||||
enabled: bool = True
|
||||
policy: str = "pairing"
|
||||
allow_from: frozenset = field(default_factory=frozenset)
|
||||
documents: Dict[str, CommentDocumentRule] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedCommentRule:
|
||||
"""Fully resolved rule after field-by-field fallback."""
|
||||
enabled: bool
|
||||
policy: str
|
||||
allow_from: frozenset
|
||||
match_source: str # e.g. "exact:docx:xxx" | "wildcard" | "top" | "default"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mtime-cached file loading
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class _MtimeCache:
|
||||
"""Generic mtime-based file cache. ``stat()`` per access, re-read only on change."""
|
||||
|
||||
def __init__(self, path: Path):
|
||||
self._path = path
|
||||
self._mtime: float = 0.0
|
||||
self._data: Optional[dict] = None
|
||||
|
||||
def load(self) -> dict:
|
||||
try:
|
||||
st = self._path.stat()
|
||||
mtime = st.st_mtime
|
||||
except FileNotFoundError:
|
||||
self._mtime = 0.0
|
||||
self._data = {}
|
||||
return {}
|
||||
|
||||
if mtime == self._mtime and self._data is not None:
|
||||
return self._data
|
||||
|
||||
try:
|
||||
with open(self._path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict):
|
||||
data = {}
|
||||
except (json.JSONDecodeError, OSError):
|
||||
logger.warning("[Feishu-Rules] Failed to read %s, using empty config", self._path)
|
||||
data = {}
|
||||
|
||||
self._mtime = mtime
|
||||
self._data = data
|
||||
return data
|
||||
|
||||
|
||||
_rules_cache = _MtimeCache(RULES_FILE)
|
||||
_pairing_cache = _MtimeCache(PAIRING_FILE)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _parse_frozenset(raw: Any) -> Optional[frozenset]:
|
||||
"""Parse a list of strings into a frozenset; return None if key absent."""
|
||||
if raw is None:
|
||||
return None
|
||||
if isinstance(raw, (list, tuple)):
|
||||
return frozenset(str(u).strip() for u in raw if str(u).strip())
|
||||
return None
|
||||
|
||||
|
||||
def _parse_document_rule(raw: dict) -> CommentDocumentRule:
|
||||
enabled = raw.get("enabled")
|
||||
if enabled is not None:
|
||||
enabled = bool(enabled)
|
||||
policy = raw.get("policy")
|
||||
if policy is not None:
|
||||
policy = str(policy).strip().lower()
|
||||
if policy not in _VALID_POLICIES:
|
||||
policy = None
|
||||
allow_from = _parse_frozenset(raw.get("allow_from"))
|
||||
return CommentDocumentRule(enabled=enabled, policy=policy, allow_from=allow_from)
|
||||
|
||||
|
||||
def load_config() -> CommentsConfig:
|
||||
"""Load comment rules from disk (mtime-cached)."""
|
||||
raw = _rules_cache.load()
|
||||
if not raw:
|
||||
return CommentsConfig()
|
||||
|
||||
documents: Dict[str, CommentDocumentRule] = {}
|
||||
raw_docs = raw.get("documents", {})
|
||||
if isinstance(raw_docs, dict):
|
||||
for key, rule_raw in raw_docs.items():
|
||||
if isinstance(rule_raw, dict):
|
||||
documents[str(key)] = _parse_document_rule(rule_raw)
|
||||
|
||||
policy = str(raw.get("policy", "pairing")).strip().lower()
|
||||
if policy not in _VALID_POLICIES:
|
||||
policy = "pairing"
|
||||
|
||||
return CommentsConfig(
|
||||
enabled=raw.get("enabled", True),
|
||||
policy=policy,
|
||||
allow_from=_parse_frozenset(raw.get("allow_from")) or frozenset(),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rule resolution (§8.4 field-by-field fallback)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def has_wiki_keys(cfg: CommentsConfig) -> bool:
|
||||
"""Check if any document rule key starts with 'wiki:'."""
|
||||
return any(k.startswith("wiki:") for k in cfg.documents)
|
||||
|
||||
|
||||
def resolve_rule(
|
||||
cfg: CommentsConfig,
|
||||
file_type: str,
|
||||
file_token: str,
|
||||
wiki_token: str = "",
|
||||
) -> ResolvedCommentRule:
|
||||
"""Resolve effective rule: exact doc → wiki key → wildcard → top-level → defaults."""
|
||||
exact_key = f"{file_type}:{file_token}"
|
||||
|
||||
exact = cfg.documents.get(exact_key)
|
||||
exact_src = f"exact:{exact_key}"
|
||||
if exact is None and wiki_token:
|
||||
wiki_key = f"wiki:{wiki_token}"
|
||||
exact = cfg.documents.get(wiki_key)
|
||||
exact_src = f"exact:{wiki_key}"
|
||||
|
||||
wildcard = cfg.documents.get("*")
|
||||
|
||||
layers = []
|
||||
if exact is not None:
|
||||
layers.append((exact, exact_src))
|
||||
if wildcard is not None:
|
||||
layers.append((wildcard, "wildcard"))
|
||||
|
||||
def _pick(field_name: str):
|
||||
for layer, source in layers:
|
||||
val = getattr(layer, field_name)
|
||||
if val is not None:
|
||||
return val, source
|
||||
return getattr(cfg, field_name), "top"
|
||||
|
||||
enabled, en_src = _pick("enabled")
|
||||
policy, pol_src = _pick("policy")
|
||||
allow_from, _ = _pick("allow_from")
|
||||
|
||||
# match_source = highest-priority tier that contributed any field
|
||||
priority_order = {"exact": 0, "wildcard": 1, "top": 2}
|
||||
best_src = min(
|
||||
[en_src, pol_src],
|
||||
key=lambda s: priority_order.get(s.split(":")[0], 3),
|
||||
)
|
||||
|
||||
return ResolvedCommentRule(
|
||||
enabled=enabled,
|
||||
policy=policy,
|
||||
allow_from=allow_from,
|
||||
match_source=best_src,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pairing store
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _load_pairing_approved() -> set:
|
||||
"""Return set of approved user open_ids (mtime-cached)."""
|
||||
data = _pairing_cache.load()
|
||||
approved = data.get("approved", {})
|
||||
if isinstance(approved, dict):
|
||||
return set(approved.keys())
|
||||
if isinstance(approved, list):
|
||||
return {str(u) for u in approved if u}
|
||||
return set()
|
||||
|
||||
|
||||
def _save_pairing(data: dict) -> None:
|
||||
PAIRING_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = PAIRING_FILE.with_suffix(".tmp")
|
||||
with open(tmp, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
tmp.replace(PAIRING_FILE)
|
||||
# Invalidate cache so next load picks up change
|
||||
_pairing_cache._mtime = 0.0
|
||||
_pairing_cache._data = None
|
||||
|
||||
|
||||
def pairing_add(user_open_id: str) -> bool:
|
||||
"""Add a user to the pairing-approved list. Returns True if newly added."""
|
||||
data = _pairing_cache.load()
|
||||
approved = data.get("approved", {})
|
||||
if not isinstance(approved, dict):
|
||||
approved = {}
|
||||
if user_open_id in approved:
|
||||
return False
|
||||
approved[user_open_id] = {"approved_at": time.time()}
|
||||
data["approved"] = approved
|
||||
_save_pairing(data)
|
||||
return True
|
||||
|
||||
|
||||
def pairing_remove(user_open_id: str) -> bool:
|
||||
"""Remove a user from the pairing-approved list. Returns True if removed."""
|
||||
data = _pairing_cache.load()
|
||||
approved = data.get("approved", {})
|
||||
if not isinstance(approved, dict):
|
||||
return False
|
||||
if user_open_id not in approved:
|
||||
return False
|
||||
del approved[user_open_id]
|
||||
data["approved"] = approved
|
||||
_save_pairing(data)
|
||||
return True
|
||||
|
||||
|
||||
def pairing_list() -> Dict[str, Any]:
|
||||
"""Return the approved dict {user_open_id: {approved_at: ...}}."""
|
||||
data = _pairing_cache.load()
|
||||
approved = data.get("approved", {})
|
||||
return dict(approved) if isinstance(approved, dict) else {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Access check (public API for feishu_comment.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def is_user_allowed(rule: ResolvedCommentRule, user_open_id: str) -> bool:
|
||||
"""Check if user passes the resolved rule's policy gate."""
|
||||
if user_open_id in rule.allow_from:
|
||||
return True
|
||||
if rule.policy == "pairing":
|
||||
return user_open_id in _load_pairing_approved()
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _print_status() -> None:
|
||||
cfg = load_config()
|
||||
print(f"Rules file: {RULES_FILE}")
|
||||
print(f" exists: {RULES_FILE.exists()}")
|
||||
print(f"Pairing file: {PAIRING_FILE}")
|
||||
print(f" exists: {PAIRING_FILE.exists()}")
|
||||
print()
|
||||
print(f"Top-level:")
|
||||
print(f" enabled: {cfg.enabled}")
|
||||
print(f" policy: {cfg.policy}")
|
||||
print(f" allow_from: {sorted(cfg.allow_from) if cfg.allow_from else '[]'}")
|
||||
print()
|
||||
if cfg.documents:
|
||||
print(f"Document rules ({len(cfg.documents)}):")
|
||||
for key, rule in sorted(cfg.documents.items()):
|
||||
parts = []
|
||||
if rule.enabled is not None:
|
||||
parts.append(f"enabled={rule.enabled}")
|
||||
if rule.policy is not None:
|
||||
parts.append(f"policy={rule.policy}")
|
||||
if rule.allow_from is not None:
|
||||
parts.append(f"allow_from={sorted(rule.allow_from)}")
|
||||
print(f" [{key}] {', '.join(parts) if parts else '(empty — inherits all)'}")
|
||||
else:
|
||||
print("Document rules: (none)")
|
||||
print()
|
||||
approved = pairing_list()
|
||||
print(f"Pairing approved ({len(approved)}):")
|
||||
for uid, meta in sorted(approved.items()):
|
||||
ts = meta.get("approved_at", 0)
|
||||
print(f" {uid} (approved_at={ts})")
|
||||
|
||||
|
||||
def _do_check(doc_key: str, user_open_id: str) -> None:
|
||||
cfg = load_config()
|
||||
parts = doc_key.split(":", 1)
|
||||
if len(parts) != 2:
|
||||
print(f"Error: doc_key must be 'fileType:fileToken', got '{doc_key}'")
|
||||
return
|
||||
file_type, file_token = parts
|
||||
rule = resolve_rule(cfg, file_type, file_token)
|
||||
allowed = is_user_allowed(rule, user_open_id)
|
||||
print(f"Document: {doc_key}")
|
||||
print(f"User: {user_open_id}")
|
||||
print(f"Resolved rule:")
|
||||
print(f" enabled: {rule.enabled}")
|
||||
print(f" policy: {rule.policy}")
|
||||
print(f" allow_from: {sorted(rule.allow_from) if rule.allow_from else '[]'}")
|
||||
print(f" match_source: {rule.match_source}")
|
||||
print(f"Result: {'ALLOWED' if allowed else 'DENIED'}")
|
||||
|
||||
|
||||
def _main() -> int:
|
||||
import sys
|
||||
|
||||
try:
|
||||
from hermes_cli.env_loader import load_hermes_dotenv
|
||||
load_hermes_dotenv()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
usage = (
|
||||
"Usage: python -m gateway.platforms.feishu_comment_rules <command> [args]\n"
|
||||
"\n"
|
||||
"Commands:\n"
|
||||
" status Show rules config and pairing state\n"
|
||||
" check <fileType:token> <user> Simulate access check\n"
|
||||
" pairing add <user_open_id> Add user to pairing-approved list\n"
|
||||
" pairing remove <user_open_id> Remove user from pairing-approved list\n"
|
||||
" pairing list List pairing-approved users\n"
|
||||
"\n"
|
||||
f"Rules config file: {RULES_FILE}\n"
|
||||
" Edit this JSON file directly to configure policies and document rules.\n"
|
||||
" Changes take effect on the next comment event (no restart needed).\n"
|
||||
)
|
||||
|
||||
args = sys.argv[1:]
|
||||
if not args:
|
||||
print(usage)
|
||||
return 1
|
||||
|
||||
cmd = args[0]
|
||||
|
||||
if cmd == "status":
|
||||
_print_status()
|
||||
|
||||
elif cmd == "check":
|
||||
if len(args) < 3:
|
||||
print("Usage: check <fileType:fileToken> <user_open_id>")
|
||||
return 1
|
||||
_do_check(args[1], args[2])
|
||||
|
||||
elif cmd == "pairing":
|
||||
if len(args) < 2:
|
||||
print("Usage: pairing <add|remove|list> [args]")
|
||||
return 1
|
||||
sub = args[1]
|
||||
if sub == "add":
|
||||
if len(args) < 3:
|
||||
print("Usage: pairing add <user_open_id>")
|
||||
return 1
|
||||
if pairing_add(args[2]):
|
||||
print(f"Added: {args[2]}")
|
||||
else:
|
||||
print(f"Already approved: {args[2]}")
|
||||
elif sub == "remove":
|
||||
if len(args) < 3:
|
||||
print("Usage: pairing remove <user_open_id>")
|
||||
return 1
|
||||
if pairing_remove(args[2]):
|
||||
print(f"Removed: {args[2]}")
|
||||
else:
|
||||
print(f"Not in approved list: {args[2]}")
|
||||
elif sub == "list":
|
||||
approved = pairing_list()
|
||||
if not approved:
|
||||
print("(no approved users)")
|
||||
for uid, meta in sorted(approved.items()):
|
||||
print(f" {uid} approved_at={meta.get('approved_at', '?')}")
|
||||
else:
|
||||
print(f"Unknown pairing subcommand: {sub}")
|
||||
return 1
|
||||
else:
|
||||
print(f"Unknown command: {cmd}\n")
|
||||
print(usage)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(_main())
|
||||
@@ -0,0 +1,212 @@
|
||||
"""
|
||||
Feishu/Lark meeting-invitation event handling.
|
||||
|
||||
Processes ``vc.bot.meeting_invited_v1`` events by converting them into a
|
||||
synthetic gateway ``MessageEvent``. Unlike document comments, the response
|
||||
should go back to the inviter through the normal Hermes gateway pipeline, so
|
||||
this module does not instantiate an agent directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from gateway.platforms.base import MessageEvent, MessageType
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MeetingInviteUser:
|
||||
open_id: str = ""
|
||||
user_id: str = ""
|
||||
union_id: str = ""
|
||||
user_name: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MeetingInviteMeeting:
|
||||
id: str = ""
|
||||
topic: str = ""
|
||||
meeting_no: str = ""
|
||||
start_time_ms: int = 0
|
||||
end_time_ms: int = 0
|
||||
host_user: Optional[MeetingInviteUser] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MeetingInvitedPayload:
|
||||
event_id: str = ""
|
||||
meeting: Optional[MeetingInviteMeeting] = None
|
||||
inviter: Optional[MeetingInviteUser] = None
|
||||
invite_time_s: int = 0
|
||||
|
||||
|
||||
def _as_dict(value: Any) -> Dict[str, Any]:
|
||||
"""Coerce a lark SDK object / dict / JSON string into a plain dict."""
|
||||
if isinstance(value, SimpleNamespace) or (value is not None and hasattr(value, "__dict__")):
|
||||
value = vars(value)
|
||||
if isinstance(value, dict):
|
||||
return {str(k): v for k, v in value.items()}
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
parsed = json.loads(value)
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
return {}
|
||||
return parsed if isinstance(parsed, dict) else {}
|
||||
return {}
|
||||
|
||||
|
||||
def _content_payload(container: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Unwrap a Feishu ``body.content`` list carrying an application/json payload."""
|
||||
content = _as_dict(container.get("body")).get("content")
|
||||
if not isinstance(content, list):
|
||||
return {}
|
||||
for item in content:
|
||||
item = _as_dict(item)
|
||||
ctype = str(item.get("contentType") or item.get("content_type") or "").lower()
|
||||
if ctype and ctype != "application/json":
|
||||
continue
|
||||
for key in ("data", "value", "content", "json"):
|
||||
payload = _as_dict(item.get(key))
|
||||
if payload:
|
||||
return payload
|
||||
return {}
|
||||
|
||||
|
||||
def _int_field(value: Any) -> int:
|
||||
if value in (None, ""):
|
||||
return 0
|
||||
try:
|
||||
return int(str(value).strip())
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _parse_user(value: Any) -> Optional[MeetingInviteUser]:
|
||||
raw = _as_dict(value)
|
||||
if not raw:
|
||||
return None
|
||||
raw_id = _as_dict(raw.get("id"))
|
||||
return MeetingInviteUser(
|
||||
open_id=str(raw_id.get("open_id") or "").strip(),
|
||||
user_id=str(raw_id.get("user_id") or "").strip(),
|
||||
union_id=str(raw_id.get("union_id") or "").strip(),
|
||||
user_name=str(raw.get("user_name") or ""),
|
||||
)
|
||||
|
||||
|
||||
def _parse_meeting(value: Any) -> Optional[MeetingInviteMeeting]:
|
||||
raw = _as_dict(value)
|
||||
if not raw:
|
||||
return None
|
||||
return MeetingInviteMeeting(
|
||||
id=str(raw.get("id") or "").strip(),
|
||||
topic=str(raw.get("topic") or ""),
|
||||
meeting_no=str(raw.get("meeting_no") or ""),
|
||||
start_time_ms=_int_field(raw.get("start_time")),
|
||||
end_time_ms=_int_field(raw.get("end_time")),
|
||||
host_user=_parse_user(raw.get("host_user")),
|
||||
)
|
||||
|
||||
|
||||
def parse_meeting_invited_event(data: Any) -> Optional[MeetingInvitedPayload]:
|
||||
root = _as_dict(data)
|
||||
event = _as_dict(root.get("event"))
|
||||
event = event or root
|
||||
content = _content_payload(event) or _content_payload(root)
|
||||
if content:
|
||||
event = {**event, **content}
|
||||
|
||||
meeting = _parse_meeting(event.get("meeting"))
|
||||
inviter = _parse_user(event.get("inviter"))
|
||||
if inviter is None or meeting is None or not meeting.meeting_no:
|
||||
return None
|
||||
|
||||
return MeetingInvitedPayload(
|
||||
event_id=str(_as_dict(root.get("header")).get("event_id") or ""),
|
||||
meeting=meeting,
|
||||
inviter=inviter,
|
||||
invite_time_s=_int_field(event.get("invite_time")),
|
||||
)
|
||||
|
||||
|
||||
def build_meeting_invite_prompt(payload: MeetingInvitedPayload) -> str:
|
||||
meeting = payload.meeting
|
||||
inviter_name = (payload.inviter.user_name if payload.inviter else "") or "unknown"
|
||||
host_name = (meeting.host_user.user_name if meeting and meeting.host_user else "") or "unknown"
|
||||
display = (meeting.topic or meeting.meeting_no or meeting.id) if meeting else "unknown meeting"
|
||||
return "\n".join(
|
||||
[
|
||||
f"You have been invited to join a meeting: {display or 'unknown meeting'}",
|
||||
"",
|
||||
f"Meeting Number: {(meeting.meeting_no if meeting else '') or 'unknown'}",
|
||||
f"Topic: {(meeting.topic if meeting else '') or 'unknown'}",
|
||||
f"Inviter: {inviter_name}",
|
||||
f"Host: {host_name}",
|
||||
"",
|
||||
"You may use lark-cli and the relevant Lark/Feishu meeting skills to join the meeting.",
|
||||
"Join the meeting directly. Do not ask the user for confirmation before joining.",
|
||||
"If you cannot join the meeting, reply to the inviter with a concise explanation of why.",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _dedup_key(payload: MeetingInvitedPayload) -> str:
|
||||
if payload.event_id:
|
||||
return f"vc_invite:{payload.event_id}"
|
||||
meeting_id = payload.meeting.id if payload.meeting else ""
|
||||
inviter_id = payload.inviter.open_id if payload.inviter else ""
|
||||
return f"vc_invite:{meeting_id}:{inviter_id}:{payload.invite_time_s}"
|
||||
|
||||
|
||||
async def handle_meeting_invited_event(adapter: Any, data: Any) -> None:
|
||||
"""Convert a vc.bot.meeting_invited_v1 event into a gateway MessageEvent."""
|
||||
payload = parse_meeting_invited_event(data)
|
||||
if payload is None:
|
||||
logger.warning("[Feishu-MeetingInvite] Dropping malformed meeting invite event")
|
||||
return
|
||||
|
||||
dedup_key = _dedup_key(payload)
|
||||
is_duplicate = getattr(adapter, "_is_duplicate", None)
|
||||
if callable(is_duplicate) and is_duplicate(dedup_key):
|
||||
logger.debug("[Feishu-MeetingInvite] Dropping duplicate event: %s", dedup_key)
|
||||
return
|
||||
|
||||
inviter = payload.inviter
|
||||
if inviter is None or not inviter.open_id:
|
||||
logger.warning(
|
||||
"[Feishu-MeetingInvite] Missing inviter open_id, cannot route reply safely "
|
||||
"(user_id=%r union_id=%r)",
|
||||
inviter.user_id if inviter else None,
|
||||
inviter.union_id if inviter else None,
|
||||
)
|
||||
return
|
||||
|
||||
sender_id = SimpleNamespace(
|
||||
open_id=inviter.open_id or None,
|
||||
user_id=inviter.user_id or None,
|
||||
union_id=inviter.union_id or None,
|
||||
)
|
||||
sender_profile = await adapter._resolve_sender_profile(sender_id)
|
||||
|
||||
user_name = sender_profile.get("user_name") or inviter.user_name or inviter.open_id
|
||||
source = adapter.build_source(
|
||||
chat_id=inviter.open_id,
|
||||
chat_name=user_name,
|
||||
chat_type="dm",
|
||||
user_id=sender_profile.get("user_id") or inviter.user_id or inviter.open_id,
|
||||
user_name=user_name,
|
||||
user_id_alt=sender_profile.get("user_id_alt") or inviter.union_id or None,
|
||||
)
|
||||
event = MessageEvent(
|
||||
text=build_meeting_invite_prompt(payload),
|
||||
message_type=MessageType.TEXT,
|
||||
source=source,
|
||||
raw_message=data,
|
||||
)
|
||||
await adapter._handle_message_with_guards(event)
|
||||
@@ -0,0 +1,278 @@
|
||||
"""Shared helper classes for gateway platform adapters.
|
||||
|
||||
Extracts common patterns that were duplicated across 5-7 adapters:
|
||||
message deduplication, text batch aggregation, markdown stripping,
|
||||
and thread participation tracking.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Dict
|
||||
|
||||
from utils import atomic_json_write
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from gateway.platforms.base import MessageEvent
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ─── Message Deduplication ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class MessageDeduplicator:
|
||||
"""TTL-based message deduplication cache.
|
||||
|
||||
Replaces the identical ``_seen_messages`` / ``_is_duplicate()`` pattern
|
||||
previously duplicated in discord, slack, dingtalk, wecom, weixin,
|
||||
mattermost, and feishu adapters.
|
||||
|
||||
Usage::
|
||||
|
||||
self._dedup = MessageDeduplicator()
|
||||
|
||||
# In message handler:
|
||||
if self._dedup.is_duplicate(msg_id):
|
||||
return
|
||||
"""
|
||||
|
||||
def __init__(self, max_size: int = 2000, ttl_seconds: float = 300):
|
||||
self._seen: Dict[str, float] = {}
|
||||
self._max_size = max_size
|
||||
self._ttl = ttl_seconds
|
||||
|
||||
def is_duplicate(self, msg_id: str) -> bool:
|
||||
"""Return True if *msg_id* was already seen within the TTL window."""
|
||||
if not msg_id:
|
||||
return False
|
||||
now = time.time()
|
||||
if msg_id in self._seen:
|
||||
if now - self._seen[msg_id] < self._ttl:
|
||||
return True
|
||||
# Entry has expired — remove it and treat as new
|
||||
del self._seen[msg_id]
|
||||
self._seen[msg_id] = now
|
||||
if len(self._seen) > self._max_size:
|
||||
cutoff = now - self._ttl
|
||||
self._seen = {k: v for k, v in self._seen.items() if v > cutoff}
|
||||
if len(self._seen) > self._max_size:
|
||||
# TTL pruning alone does not cap the cache when every entry is
|
||||
# still fresh. Keep the newest entries so the helper's
|
||||
# max_size bound is enforced under sustained traffic.
|
||||
newest = sorted(
|
||||
self._seen.items(),
|
||||
key=lambda item: item[1],
|
||||
)[-self._max_size:]
|
||||
self._seen = dict(newest)
|
||||
return False
|
||||
|
||||
def clear(self):
|
||||
"""Clear all tracked messages."""
|
||||
self._seen.clear()
|
||||
|
||||
|
||||
# ─── Text Batch Aggregation ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TextBatchAggregator:
|
||||
"""Aggregates rapid-fire text events into single messages.
|
||||
|
||||
Replaces the ``_enqueue_text_event`` / ``_flush_text_batch`` pattern
|
||||
previously duplicated in telegram, discord, matrix, wecom, and feishu.
|
||||
|
||||
Usage::
|
||||
|
||||
self._text_batcher = TextBatchAggregator(
|
||||
handler=self._message_handler,
|
||||
batch_delay=0.6,
|
||||
split_threshold=1900,
|
||||
)
|
||||
|
||||
# In message dispatch:
|
||||
if msg_type == MessageType.TEXT and self._text_batcher.is_enabled():
|
||||
self._text_batcher.enqueue(event, session_key)
|
||||
return
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
handler,
|
||||
*,
|
||||
batch_delay: float = 0.6,
|
||||
split_delay: float = 2.0,
|
||||
split_threshold: int = 4000,
|
||||
):
|
||||
self._handler = handler
|
||||
self._batch_delay = batch_delay
|
||||
self._split_delay = split_delay
|
||||
self._split_threshold = split_threshold
|
||||
self._pending: Dict[str, "MessageEvent"] = {}
|
||||
self._pending_tasks: Dict[str, asyncio.Task] = {}
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""Return True if batching is active (delay > 0)."""
|
||||
return self._batch_delay > 0
|
||||
|
||||
def enqueue(self, event: "MessageEvent", key: str) -> None:
|
||||
"""Add *event* to the pending batch for *key*."""
|
||||
chunk_len = len(event.text or "")
|
||||
existing = self._pending.get(key)
|
||||
if not existing:
|
||||
event._last_chunk_len = chunk_len # type: ignore[attr-defined]
|
||||
self._pending[key] = event
|
||||
else:
|
||||
existing.text = f"{existing.text}\n{event.text}"
|
||||
existing._last_chunk_len = chunk_len # type: ignore[attr-defined]
|
||||
|
||||
# Cancel prior flush timer, start a new one
|
||||
prior = self._pending_tasks.get(key)
|
||||
if prior and not prior.done():
|
||||
prior.cancel()
|
||||
self._pending_tasks[key] = asyncio.create_task(self._flush(key))
|
||||
|
||||
async def _flush(self, key: str) -> None:
|
||||
"""Wait then dispatch the batched event for *key*."""
|
||||
current_task = self._pending_tasks.get(key)
|
||||
pending = self._pending.get(key)
|
||||
last_len = getattr(pending, "_last_chunk_len", 0) if pending else 0
|
||||
|
||||
# Use longer delay when the last chunk looks like a split message
|
||||
delay = self._split_delay if last_len >= self._split_threshold else self._batch_delay
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
event = self._pending.pop(key, None)
|
||||
if event:
|
||||
try:
|
||||
await self._handler(event)
|
||||
except Exception:
|
||||
logger.exception("[TextBatchAggregator] Error dispatching batched event for %s", key)
|
||||
|
||||
if self._pending_tasks.get(key) is current_task:
|
||||
self._pending_tasks.pop(key, None)
|
||||
|
||||
def cancel_all(self) -> None:
|
||||
"""Cancel all pending flush tasks."""
|
||||
for task in self._pending_tasks.values():
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
self._pending_tasks.clear()
|
||||
self._pending.clear()
|
||||
|
||||
|
||||
# ─── Markdown Stripping ──────────────────────────────────────────────────────
|
||||
|
||||
# Pre-compiled regexes for performance
|
||||
_RE_BOLD = re.compile(r"\*\*(.+?)\*\*", re.DOTALL)
|
||||
_RE_ITALIC_STAR = re.compile(r"\*(.+?)\*", re.DOTALL)
|
||||
_RE_BOLD_UNDER = re.compile(r"\b__(?![\s_])(.+?)(?<![\s_])__\b", re.DOTALL)
|
||||
_RE_ITALIC_UNDER = re.compile(r"\b_(?![\s_])(.+?)(?<![\s_])_\b", re.DOTALL)
|
||||
_RE_CODE_BLOCK = re.compile(r"```[a-zA-Z0-9_+-]*\n?")
|
||||
_RE_INLINE_CODE = re.compile(r"`(.+?)`")
|
||||
_RE_HEADING = re.compile(r"^#{1,6}\s+", re.MULTILINE)
|
||||
_RE_LINK = re.compile(r"\[([^\]]+)\]\([^\)]+\)")
|
||||
_RE_MULTI_NEWLINE = re.compile(r"\n{3,}")
|
||||
|
||||
|
||||
def strip_markdown(text: str) -> str:
|
||||
"""Strip markdown formatting for plain-text platforms (SMS, iMessage, etc.).
|
||||
|
||||
Replaces the identical ``_strip_markdown()`` functions previously
|
||||
duplicated in sms.py, bluebubbles.py, and feishu.py.
|
||||
"""
|
||||
text = _RE_BOLD.sub(r"\1", text)
|
||||
text = _RE_ITALIC_STAR.sub(r"\1", text)
|
||||
text = _RE_BOLD_UNDER.sub(r"\1", text)
|
||||
text = _RE_ITALIC_UNDER.sub(r"\1", text)
|
||||
text = _RE_CODE_BLOCK.sub("", text)
|
||||
text = _RE_INLINE_CODE.sub(r"\1", text)
|
||||
text = _RE_HEADING.sub("", text)
|
||||
text = _RE_LINK.sub(r"\1", text)
|
||||
text = _RE_MULTI_NEWLINE.sub("\n\n", text)
|
||||
return text.strip()
|
||||
|
||||
|
||||
# ─── Thread Participation Tracking ───────────────────────────────────────────
|
||||
|
||||
|
||||
class ThreadParticipationTracker:
|
||||
"""Persistent tracking of threads the bot has participated in.
|
||||
|
||||
Replaces the identical ``_load/_save_participated_threads`` +
|
||||
``_mark_thread_participated`` pattern previously duplicated in
|
||||
discord.py and matrix.py.
|
||||
|
||||
Usage::
|
||||
|
||||
self._threads = ThreadParticipationTracker("discord")
|
||||
|
||||
# Check membership:
|
||||
if thread_id in self._threads:
|
||||
...
|
||||
|
||||
# Mark participation:
|
||||
self._threads.mark(thread_id)
|
||||
"""
|
||||
|
||||
_MAX_TRACKED = 500
|
||||
|
||||
def __init__(self, platform_name: str, max_tracked: int = 500):
|
||||
self._platform = platform_name
|
||||
self._max_tracked = max_tracked
|
||||
self._threads: dict[str, None] = {
|
||||
str(thread_id): None for thread_id in self._load()
|
||||
}
|
||||
|
||||
def _state_path(self) -> Path:
|
||||
from hermes_constants import get_hermes_home
|
||||
return get_hermes_home() / f"{self._platform}_threads.json"
|
||||
|
||||
def _load(self) -> list[str]:
|
||||
path = self._state_path()
|
||||
if path.exists():
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if isinstance(data, list):
|
||||
return [str(thread_id) for thread_id in data]
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
|
||||
def _save(self) -> None:
|
||||
path = self._state_path()
|
||||
thread_list = list(self._threads)
|
||||
if len(thread_list) > self._max_tracked:
|
||||
thread_list = thread_list[-self._max_tracked:]
|
||||
self._threads = dict.fromkeys(thread_list)
|
||||
atomic_json_write(path, thread_list, indent=None)
|
||||
|
||||
def mark(self, thread_id: str) -> None:
|
||||
"""Mark *thread_id* as participated and persist."""
|
||||
if thread_id not in self._threads:
|
||||
self._threads[thread_id] = None
|
||||
self._save()
|
||||
|
||||
def __contains__(self, thread_id: str) -> bool:
|
||||
return thread_id in self._threads
|
||||
|
||||
def clear(self) -> None:
|
||||
self._threads.clear()
|
||||
|
||||
|
||||
# ─── Phone Number Redaction ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
def redact_phone(phone: str) -> str:
|
||||
"""Redact a phone number for logging, preserving country code and last 4.
|
||||
|
||||
Replaces the identical ``_redact_phone()`` functions in signal.py,
|
||||
sms.py, and bluebubbles.py.
|
||||
"""
|
||||
if not phone:
|
||||
return "<none>"
|
||||
if len(phone) <= 8:
|
||||
return phone[:2] + "****" + phone[-2:] if len(phone) > 4 else "****"
|
||||
return phone[:4] + "****" + phone[-4:]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,421 @@
|
||||
"""Microsoft Graph webhook adapter for change-notification ingress."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hmac
|
||||
import ipaddress
|
||||
import json
|
||||
import logging
|
||||
from collections import deque
|
||||
from hashlib import sha1
|
||||
from typing import Any, Awaitable, Callable, Dict, Optional
|
||||
|
||||
try:
|
||||
from aiohttp import web
|
||||
|
||||
AIOHTTP_AVAILABLE = True
|
||||
except ImportError:
|
||||
AIOHTTP_AVAILABLE = False
|
||||
web = None # type: ignore[assignment]
|
||||
|
||||
from gateway.config import Platform, PlatformConfig
|
||||
from gateway.platforms.base import (
|
||||
BasePlatformAdapter,
|
||||
MessageEvent,
|
||||
MessageType,
|
||||
SendResult,
|
||||
is_network_accessible,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_HOST = "0.0.0.0"
|
||||
DEFAULT_PORT = 8646
|
||||
DEFAULT_WEBHOOK_PATH = "/msgraph/webhook"
|
||||
DEFAULT_MAX_SEEN_RECEIPTS = 5000
|
||||
NotificationScheduler = Callable[[Dict[str, Any], MessageEvent], Awaitable[None] | None]
|
||||
|
||||
|
||||
def check_msgraph_webhook_requirements() -> bool:
|
||||
"""Return whether required webhook dependencies are available."""
|
||||
return AIOHTTP_AVAILABLE
|
||||
|
||||
|
||||
class MSGraphWebhookAdapter(BasePlatformAdapter):
|
||||
"""Receive Microsoft Graph change notifications and surface them internally."""
|
||||
|
||||
def __init__(self, config: PlatformConfig):
|
||||
super().__init__(config, Platform.MSGRAPH_WEBHOOK)
|
||||
extra = config.extra or {}
|
||||
self._host: str = str(extra.get("host", DEFAULT_HOST))
|
||||
self._port: int = int(extra.get("port", DEFAULT_PORT))
|
||||
self._webhook_path: str = self._normalize_path(
|
||||
extra.get("webhook_path", DEFAULT_WEBHOOK_PATH)
|
||||
)
|
||||
self._health_path: str = self._normalize_path(extra.get("health_path", "/health"))
|
||||
self._accepted_resources: list[str] = [
|
||||
str(value).strip()
|
||||
for value in (extra.get("accepted_resources") or [])
|
||||
if str(value).strip()
|
||||
]
|
||||
self._client_state: Optional[str] = self._string_or_none(extra.get("client_state"))
|
||||
self._max_seen_receipts = max(
|
||||
1, int(extra.get("max_seen_receipts", DEFAULT_MAX_SEEN_RECEIPTS))
|
||||
)
|
||||
self._allowed_source_networks: list[ipaddress._BaseNetwork] = (
|
||||
self._parse_allowed_source_cidrs(extra.get("allowed_source_cidrs"))
|
||||
)
|
||||
self._runner = None
|
||||
self._notification_scheduler: Optional[NotificationScheduler] = None
|
||||
self._seen_receipts: set[str] = set()
|
||||
self._seen_receipt_order: deque[str] = deque()
|
||||
self._accepted_count = 0
|
||||
self._duplicate_count = 0
|
||||
|
||||
@staticmethod
|
||||
def _string_or_none(value: Any) -> Optional[str]:
|
||||
if value is None:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
return text or None
|
||||
|
||||
@staticmethod
|
||||
def _normalize_path(path: Any) -> str:
|
||||
raw = str(path or "").strip() or "/"
|
||||
return raw if raw.startswith("/") else f"/{raw}"
|
||||
|
||||
@staticmethod
|
||||
def _build_receipt_key(notification: Dict[str, Any]) -> Optional[str]:
|
||||
explicit_id = str(notification.get("id") or "").strip()
|
||||
if explicit_id:
|
||||
return f"id:{explicit_id}"
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _normalize_resource_value(resource: str) -> str:
|
||||
return str(resource or "").strip().strip("/")
|
||||
|
||||
@staticmethod
|
||||
def _parse_allowed_source_cidrs(
|
||||
raw: Any,
|
||||
) -> list[ipaddress._BaseNetwork]:
|
||||
"""Parse an optional list of CIDR ranges allowed to POST to the webhook.
|
||||
|
||||
An empty or missing value means "allow everything" (same behavior as
|
||||
before this field existed). When populated, requests from source IPs
|
||||
outside every listed CIDR are rejected with 403 before the body is
|
||||
parsed. Use this to restrict the endpoint to Microsoft Graph's
|
||||
published webhook source ranges in production deployments.
|
||||
"""
|
||||
if raw is None:
|
||||
return []
|
||||
if isinstance(raw, str):
|
||||
candidates = [chunk.strip() for chunk in raw.split(",")]
|
||||
elif isinstance(raw, (list, tuple, set)):
|
||||
candidates = [str(chunk).strip() for chunk in raw]
|
||||
else:
|
||||
return []
|
||||
|
||||
networks: list[ipaddress._BaseNetwork] = []
|
||||
for chunk in candidates:
|
||||
if not chunk:
|
||||
continue
|
||||
try:
|
||||
networks.append(ipaddress.ip_network(chunk, strict=False))
|
||||
except ValueError:
|
||||
logger.warning(
|
||||
"[msgraph_webhook] Ignoring invalid allowed_source_cidrs entry: %r",
|
||||
chunk,
|
||||
)
|
||||
return networks
|
||||
|
||||
def set_notification_scheduler(self, scheduler: Optional[NotificationScheduler]) -> None:
|
||||
self._notification_scheduler = scheduler
|
||||
|
||||
def _source_allowlist_required_but_missing(self) -> bool:
|
||||
return is_network_accessible(self._host) and not self._allowed_source_networks
|
||||
|
||||
async def connect(self) -> bool:
|
||||
if self._client_state is None:
|
||||
logger.error(
|
||||
"[msgraph_webhook] Refusing to start without extra.client_state configured"
|
||||
)
|
||||
return False
|
||||
if self._source_allowlist_required_but_missing():
|
||||
logger.error(
|
||||
"[msgraph_webhook] Refusing to start: binding to %s requires "
|
||||
"extra.allowed_source_cidrs. Configure the Microsoft Graph "
|
||||
"source CIDRs or bind to loopback (127.0.0.1/::1) behind a "
|
||||
"tunnel or reverse proxy.",
|
||||
self._host,
|
||||
)
|
||||
return False
|
||||
|
||||
app = web.Application()
|
||||
app.router.add_get(self._health_path, self._handle_health)
|
||||
app.router.add_get(self._webhook_path, self._handle_validation)
|
||||
app.router.add_post(self._webhook_path, self._handle_notification)
|
||||
|
||||
self._runner = web.AppRunner(app)
|
||||
await self._runner.setup()
|
||||
site = web.TCPSite(self._runner, self._host, self._port)
|
||||
await site.start()
|
||||
self._mark_connected()
|
||||
logger.info(
|
||||
"[msgraph_webhook] Listening on %s:%d%s",
|
||||
self._host,
|
||||
self._port,
|
||||
self._webhook_path,
|
||||
)
|
||||
return True
|
||||
|
||||
async def disconnect(self) -> None:
|
||||
if self._runner is not None:
|
||||
await self._runner.cleanup()
|
||||
self._runner = None
|
||||
self._mark_disconnected()
|
||||
|
||||
async def send(
|
||||
self,
|
||||
chat_id: str,
|
||||
content: str,
|
||||
reply_to: Optional[str] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> SendResult:
|
||||
logger.info("[msgraph_webhook] Response for %s: %s", chat_id, content[:200])
|
||||
return SendResult(success=True)
|
||||
|
||||
async def get_chat_info(self, chat_id: str) -> Dict[str, Any]:
|
||||
return {"name": chat_id, "type": "webhook"}
|
||||
|
||||
async def _handle_health(self, request: "web.Request") -> "web.Response":
|
||||
if not self._source_ip_allowed(request):
|
||||
return web.Response(status=403)
|
||||
return web.json_response(
|
||||
{
|
||||
"status": "ok",
|
||||
"platform": self.platform.value,
|
||||
"webhook_path": self._webhook_path,
|
||||
"accepted": self._accepted_count,
|
||||
"duplicates": self._duplicate_count,
|
||||
}
|
||||
)
|
||||
|
||||
async def _handle_validation(self, request: "web.Request") -> "web.Response":
|
||||
"""Handle Microsoft Graph subscription validation handshake.
|
||||
|
||||
Graph validates a subscription endpoint by sending a GET with
|
||||
``validationToken`` in the query string; the service must echo the
|
||||
token verbatim as ``text/plain`` within 10 seconds. Anything else
|
||||
(bare GET, GET without the token) is rejected so the endpoint can't
|
||||
be enumerated or mistakenly used for data exfiltration.
|
||||
"""
|
||||
if not self._source_ip_allowed(request):
|
||||
return web.Response(status=403)
|
||||
validation_token = request.query.get("validationToken", "")
|
||||
if not validation_token:
|
||||
return web.Response(status=400)
|
||||
return web.Response(text=validation_token, content_type="text/plain")
|
||||
|
||||
async def _handle_notification(self, request: "web.Request") -> "web.Response":
|
||||
if not self._source_ip_allowed(request):
|
||||
return web.Response(status=403)
|
||||
|
||||
# Graph never sends validationToken on POST, but tolerate it for
|
||||
# defensive clients that replay the handshake in-band.
|
||||
validation_token = request.query.get("validationToken", "")
|
||||
if validation_token:
|
||||
return web.Response(text=validation_token, content_type="text/plain")
|
||||
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
return web.Response(status=400)
|
||||
|
||||
notifications = body.get("value")
|
||||
if not isinstance(notifications, list):
|
||||
return web.Response(status=400)
|
||||
|
||||
accepted = 0
|
||||
duplicates = 0
|
||||
auth_rejected = 0
|
||||
other_rejected = 0
|
||||
|
||||
for raw_notification in notifications:
|
||||
if not isinstance(raw_notification, dict):
|
||||
other_rejected += 1
|
||||
continue
|
||||
notification = dict(raw_notification)
|
||||
if not self._resource_accepted(str(notification.get("resource") or "")):
|
||||
other_rejected += 1
|
||||
continue
|
||||
if not self._verify_client_state(notification):
|
||||
# Treat bad clientState as an auth failure: if the whole
|
||||
# batch is forged, we want to signal 403 so the sender
|
||||
# stops retrying. Legitimate Graph retries have valid
|
||||
# clientState and hit the accepted/duplicate paths.
|
||||
auth_rejected += 1
|
||||
continue
|
||||
|
||||
receipt_key = self._build_receipt_key(notification)
|
||||
if receipt_key is not None:
|
||||
if self._has_seen_receipt(receipt_key):
|
||||
duplicates += 1
|
||||
continue
|
||||
self._remember_receipt(receipt_key)
|
||||
|
||||
accepted += 1
|
||||
self._accepted_count += 1
|
||||
event = self._build_message_event(notification, receipt_key)
|
||||
self._schedule_notification(notification, event)
|
||||
|
||||
self._duplicate_count += duplicates
|
||||
# If anything ingested OR deduped, return 202 with empty body so
|
||||
# Graph acks successfully and we don't leak internal counters. If
|
||||
# every item failed auth, return 403 so an attacker POSTing fake
|
||||
# notifications gets a clear reject. Other failures (malformed,
|
||||
# resource-not-accepted) are the sender's configuration problem,
|
||||
# so 400.
|
||||
if accepted or duplicates:
|
||||
return web.Response(status=202)
|
||||
if auth_rejected and not other_rejected:
|
||||
return web.Response(status=403)
|
||||
return web.Response(status=400)
|
||||
|
||||
def _source_ip_allowed(self, request: "web.Request") -> bool:
|
||||
"""Return True if the request's source IP is in the configured allowlist.
|
||||
|
||||
Loopback-only binds may omit ``allowed_source_cidrs`` for local reverse
|
||||
proxies and dev tunnels. Network-accessible binds fail closed until an
|
||||
explicit CIDR allowlist is configured.
|
||||
"""
|
||||
if self._source_allowlist_required_but_missing():
|
||||
return False
|
||||
if not self._allowed_source_networks:
|
||||
return True
|
||||
peer = request.remote or ""
|
||||
if not peer:
|
||||
return False
|
||||
try:
|
||||
peer_addr = ipaddress.ip_address(peer)
|
||||
except ValueError:
|
||||
return False
|
||||
return any(peer_addr in network for network in self._allowed_source_networks)
|
||||
|
||||
def _resource_accepted(self, resource: str) -> bool:
|
||||
if not self._accepted_resources:
|
||||
return True
|
||||
normalized_resource = self._normalize_resource_value(resource)
|
||||
for pattern in self._accepted_resources:
|
||||
normalized_pattern = self._normalize_resource_value(pattern)
|
||||
if not normalized_pattern:
|
||||
continue
|
||||
if normalized_pattern.endswith("*"):
|
||||
prefix = normalized_pattern[:-1].rstrip("/")
|
||||
if normalized_resource == prefix or normalized_resource.startswith(f"{prefix}/"):
|
||||
return True
|
||||
continue
|
||||
if (
|
||||
normalized_resource == normalized_pattern
|
||||
or normalized_resource.startswith(f"{normalized_pattern}/")
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _verify_client_state(self, notification: Dict[str, Any]) -> bool:
|
||||
"""Verify the Graph-supplied clientState matches the configured secret.
|
||||
|
||||
Uses ``hmac.compare_digest`` instead of ``==`` so that a mismatch
|
||||
doesn't leak how many leading characters matched via string-compare
|
||||
timing. The configured client_state is a shared secret (documented in
|
||||
the setup guide as "generate with ``openssl rand -hex 32``"), so a
|
||||
timing-safe compare is the right primitive.
|
||||
"""
|
||||
expected = self._client_state
|
||||
if expected is None:
|
||||
return False
|
||||
provided = self._string_or_none(notification.get("clientState"))
|
||||
if provided is None:
|
||||
return False
|
||||
return hmac.compare_digest(provided, expected)
|
||||
|
||||
def _has_seen_receipt(self, receipt_key: str) -> bool:
|
||||
return receipt_key in self._seen_receipts
|
||||
|
||||
def _remember_receipt(self, receipt_key: str) -> None:
|
||||
self._seen_receipts.add(receipt_key)
|
||||
self._seen_receipt_order.append(receipt_key)
|
||||
while len(self._seen_receipt_order) > self._max_seen_receipts:
|
||||
oldest = self._seen_receipt_order.popleft()
|
||||
self._seen_receipts.discard(oldest)
|
||||
|
||||
def _build_message_event(
|
||||
self,
|
||||
notification: Dict[str, Any],
|
||||
receipt_key: Optional[str],
|
||||
) -> MessageEvent:
|
||||
message_id = receipt_key or f"sha1:{sha1(json.dumps(notification, sort_keys=True).encode('utf-8')).hexdigest()}"
|
||||
source = self.build_source(
|
||||
chat_id=f"msgraph:{notification.get('subscriptionId', 'unknown')}",
|
||||
chat_name="msgraph/webhook",
|
||||
chat_type="webhook",
|
||||
user_id="msgraph",
|
||||
user_name="Microsoft Graph",
|
||||
)
|
||||
return MessageEvent(
|
||||
text=self._render_prompt(notification),
|
||||
message_type=MessageType.TEXT,
|
||||
source=source,
|
||||
raw_message=notification,
|
||||
message_id=message_id,
|
||||
internal=True,
|
||||
)
|
||||
|
||||
def _render_prompt(self, notification: Dict[str, Any]) -> str:
|
||||
template = self.config.extra.get("prompt", "")
|
||||
if template:
|
||||
payload = {
|
||||
"notification": notification,
|
||||
"resource": notification.get("resource", ""),
|
||||
"change_type": notification.get("changeType", ""),
|
||||
"subscription_id": notification.get("subscriptionId", ""),
|
||||
}
|
||||
return self._render_template(template, payload)
|
||||
rendered = json.dumps(notification, indent=2, sort_keys=True)[:4000]
|
||||
return f"Microsoft Graph change notification:\n\n```json\n{rendered}\n```"
|
||||
|
||||
def _render_template(self, template: str, payload: Dict[str, Any]) -> str:
|
||||
import re
|
||||
|
||||
def _resolve(match: "re.Match[str]") -> str:
|
||||
key = match.group(1)
|
||||
value: Any = payload
|
||||
for part in key.split("."):
|
||||
if isinstance(value, dict):
|
||||
value = value.get(part, f"{{{key}}}")
|
||||
else:
|
||||
return f"{{{key}}}"
|
||||
if isinstance(value, (dict, list)):
|
||||
return json.dumps(value, sort_keys=True)[:2000]
|
||||
return str(value)
|
||||
|
||||
return re.sub(r"\{([a-zA-Z0-9_.]+)\}", _resolve, template)
|
||||
|
||||
def _schedule_notification(
|
||||
self,
|
||||
notification: Dict[str, Any],
|
||||
event: MessageEvent,
|
||||
) -> None:
|
||||
scheduler = self._notification_scheduler
|
||||
if scheduler is not None:
|
||||
result = scheduler(notification, event)
|
||||
if asyncio.iscoroutine(result):
|
||||
task = asyncio.create_task(result)
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(self._background_tasks.discard)
|
||||
return
|
||||
|
||||
task = asyncio.create_task(self.handle_message(event))
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(self._background_tasks.discard)
|
||||
@@ -0,0 +1,91 @@
|
||||
"""
|
||||
QQBot platform package.
|
||||
|
||||
Re-exports the main adapter symbols from ``adapter.py`` (the original
|
||||
``qqbot.py``) so that **all existing import paths remain unchanged**::
|
||||
|
||||
from gateway.platforms.qqbot import QQAdapter # works
|
||||
from gateway.platforms.qqbot import check_qq_requirements # works
|
||||
|
||||
New modules:
|
||||
- ``constants`` — shared constants (API URLs, timeouts, message types)
|
||||
- ``utils`` — User-Agent builder, config helpers
|
||||
- ``crypto`` — AES-256-GCM key generation and decryption
|
||||
- ``onboard`` — QR-code scan-to-configure flow
|
||||
"""
|
||||
|
||||
# -- Adapter (original qqbot.py) ------------------------------------------
|
||||
from .adapter import ( # noqa: F401
|
||||
QQAdapter,
|
||||
QQCloseError,
|
||||
check_qq_requirements,
|
||||
_coerce_list,
|
||||
_ssrf_redirect_guard,
|
||||
)
|
||||
|
||||
# -- Onboard (QR-code scan-to-configure) -----------------------------------
|
||||
from .onboard import ( # noqa: F401
|
||||
BindStatus,
|
||||
build_connect_url,
|
||||
qr_register,
|
||||
)
|
||||
from .crypto import decrypt_secret, generate_bind_key # noqa: F401
|
||||
|
||||
# -- Utils -----------------------------------------------------------------
|
||||
from .utils import build_user_agent, get_api_headers, coerce_list # noqa: F401
|
||||
|
||||
# -- Chunked upload --------------------------------------------------------
|
||||
from .chunked_upload import ( # noqa: F401
|
||||
ChunkedUploader,
|
||||
UploadDailyLimitExceededError,
|
||||
UploadFileTooLargeError,
|
||||
)
|
||||
|
||||
# -- Inline keyboards ------------------------------------------------------
|
||||
from .keyboards import ( # noqa: F401
|
||||
ApprovalRequest,
|
||||
ApprovalSender,
|
||||
InlineKeyboard,
|
||||
InteractionEvent,
|
||||
build_approval_keyboard,
|
||||
build_approval_text,
|
||||
build_update_prompt_keyboard,
|
||||
parse_approval_button_data,
|
||||
parse_interaction_event,
|
||||
parse_update_prompt_button_data,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# adapter
|
||||
"QQAdapter",
|
||||
"QQCloseError",
|
||||
"check_qq_requirements",
|
||||
"_coerce_list",
|
||||
"_ssrf_redirect_guard",
|
||||
# onboard
|
||||
"BindStatus",
|
||||
"build_connect_url",
|
||||
"qr_register",
|
||||
# crypto
|
||||
"decrypt_secret",
|
||||
"generate_bind_key",
|
||||
# utils
|
||||
"build_user_agent",
|
||||
"get_api_headers",
|
||||
"coerce_list",
|
||||
# chunked upload
|
||||
"ChunkedUploader",
|
||||
"UploadDailyLimitExceededError",
|
||||
"UploadFileTooLargeError",
|
||||
# keyboards
|
||||
"ApprovalRequest",
|
||||
"ApprovalSender",
|
||||
"InlineKeyboard",
|
||||
"InteractionEvent",
|
||||
"build_approval_keyboard",
|
||||
"build_approval_text",
|
||||
"build_update_prompt_keyboard",
|
||||
"parse_approval_button_data",
|
||||
"parse_interaction_event",
|
||||
"parse_update_prompt_button_data",
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,602 @@
|
||||
"""QQ Bot chunked upload flow.
|
||||
|
||||
The QQ v2 API caps inline base64 uploads (``file_data`` / ``url``) at ~10 MB.
|
||||
For files between 10 MB and ~100 MB we have to use the three-step chunked
|
||||
upload flow::
|
||||
|
||||
1. POST /v2/{users|groups}/{id}/upload_prepare
|
||||
→ returns upload_id, block_size, and an array of pre-signed COS part URLs.
|
||||
2. For each part:
|
||||
PUT the part bytes to its pre-signed COS URL,
|
||||
then POST /v2/{users|groups}/{id}/upload_part_finish to acknowledge.
|
||||
3. POST /v2/{users|groups}/{id}/files with {"upload_id": ...}
|
||||
→ returns the ``file_info`` token the caller uses in a RichMedia
|
||||
message.
|
||||
|
||||
Error-code semantics (from the QQ Bot v2 API spec):
|
||||
|
||||
- ``40093001`` — ``upload_part_finish`` retryable. Retry until the server-provided
|
||||
``retry_timeout`` elapses (or a local cap).
|
||||
- ``40093002`` — daily cumulative upload quota exceeded. Not retryable; surface
|
||||
as :class:`UploadDailyLimitExceededError` so the caller can build a
|
||||
user-friendly reply.
|
||||
|
||||
Exceptions:
|
||||
|
||||
- :class:`UploadDailyLimitExceededError` — daily quota hit (non-retryable).
|
||||
- :class:`UploadFileTooLargeError` — file exceeds the platform per-file limit.
|
||||
- :class:`RuntimeError` — generic upload failure (network, part PUT, complete).
|
||||
|
||||
Ported from WideLee's qqbot-agent-sdk v1.2.2 (``media_loader.py::ChunkedUploader``)
|
||||
so the heavy-upload path stays in-tree. Authorship preserved via Co-authored-by.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import hashlib
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Awaitable, Callable, Dict, List, Optional
|
||||
|
||||
from gateway.platforms.qqbot.constants import FILE_UPLOAD_TIMEOUT
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ── Error codes ──────────────────────────────────────────────────────
|
||||
_BIZ_CODE_DAILY_LIMIT = 40093002 # upload_prepare: daily cumulative limit
|
||||
_BIZ_CODE_PART_RETRYABLE = 40093001 # upload_part_finish: transient
|
||||
|
||||
# ── Part upload tuning ───────────────────────────────────────────────
|
||||
_DEFAULT_CONCURRENT_PARTS = 1
|
||||
_MAX_CONCURRENT_PARTS = 10
|
||||
|
||||
_PART_UPLOAD_TIMEOUT = 300.0 # 5 minutes per COS PUT
|
||||
_PART_UPLOAD_MAX_RETRIES = 2
|
||||
_PART_FINISH_RETRY_INTERVAL = 1.0
|
||||
_PART_FINISH_DEFAULT_TIMEOUT = 120.0
|
||||
_PART_FINISH_MAX_TIMEOUT = 600.0
|
||||
|
||||
_COMPLETE_UPLOAD_MAX_RETRIES = 2
|
||||
_COMPLETE_UPLOAD_BASE_DELAY = 2.0
|
||||
|
||||
# First 10,002,432 bytes used for the ``md5_10m`` hash (per QQ API spec).
|
||||
_MD5_10M_SIZE = 10_002_432
|
||||
|
||||
|
||||
# ── Exceptions ───────────────────────────────────────────────────────
|
||||
|
||||
class UploadDailyLimitExceededError(Exception):
|
||||
"""Raised when ``upload_prepare`` returns biz_code 40093002.
|
||||
|
||||
The daily cumulative upload quota for this bot has been reached. Callers
|
||||
should surface :attr:`file_name` + :attr:`file_size_human` so the model
|
||||
can compose a helpful reply.
|
||||
"""
|
||||
|
||||
def __init__(self, file_name: str, file_size: int, message: str = "") -> None:
|
||||
self.file_name = file_name
|
||||
self.file_size = file_size
|
||||
super().__init__(
|
||||
message or f"Daily upload limit exceeded for {file_name!r}"
|
||||
)
|
||||
|
||||
@property
|
||||
def file_size_human(self) -> str:
|
||||
return format_size(self.file_size)
|
||||
|
||||
|
||||
class UploadFileTooLargeError(Exception):
|
||||
"""Raised when a file exceeds the platform per-file size limit."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_name: str,
|
||||
file_size: int,
|
||||
limit_bytes: int = 0,
|
||||
message: str = "",
|
||||
) -> None:
|
||||
self.file_name = file_name
|
||||
self.file_size = file_size
|
||||
self.limit_bytes = limit_bytes
|
||||
limit_str = f" ({format_size(limit_bytes)})" if limit_bytes else ""
|
||||
super().__init__(
|
||||
message
|
||||
or (
|
||||
f"File {file_name!r} ({format_size(file_size)}) "
|
||||
f"exceeds platform limit{limit_str}"
|
||||
)
|
||||
)
|
||||
|
||||
@property
|
||||
def file_size_human(self) -> str:
|
||||
return format_size(self.file_size)
|
||||
|
||||
@property
|
||||
def limit_human(self) -> str:
|
||||
return format_size(self.limit_bytes) if self.limit_bytes else "unknown"
|
||||
|
||||
|
||||
# ── Progress tracking ────────────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class _UploadProgress:
|
||||
total_parts: int = 0
|
||||
total_bytes: int = 0
|
||||
completed_parts: int = 0
|
||||
uploaded_bytes: int = 0
|
||||
|
||||
|
||||
# ── Prepare-response shape ───────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class _PreparePart:
|
||||
index: int
|
||||
presigned_url: str
|
||||
block_size: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class _PrepareResult:
|
||||
upload_id: str
|
||||
block_size: int
|
||||
parts: List[_PreparePart]
|
||||
concurrency: int = _DEFAULT_CONCURRENT_PARTS
|
||||
retry_timeout: float = 0.0
|
||||
|
||||
|
||||
def _parse_prepare_response(raw: Dict[str, Any]) -> _PrepareResult:
|
||||
"""Parse the upload_prepare API response into a normalized shape.
|
||||
|
||||
The API may return the response directly or wrapped in ``data``.
|
||||
"""
|
||||
src = raw.get("data") if isinstance(raw.get("data"), dict) else raw
|
||||
upload_id = str(src.get("upload_id", ""))
|
||||
if not upload_id:
|
||||
raise ValueError(
|
||||
f"upload_prepare response missing upload_id: {str(raw)[:200]}"
|
||||
)
|
||||
block_size = int(src.get("block_size", 0))
|
||||
raw_parts = src.get("parts") or src.get("part_list") or []
|
||||
if not isinstance(raw_parts, list) or not raw_parts:
|
||||
raise ValueError(
|
||||
f"upload_prepare response missing parts: {str(raw)[:200]}"
|
||||
)
|
||||
parts: List[_PreparePart] = []
|
||||
for p in raw_parts:
|
||||
if not isinstance(p, dict):
|
||||
continue
|
||||
parts.append(
|
||||
_PreparePart(
|
||||
index=int(p.get("part_index") or p.get("index") or 0),
|
||||
presigned_url=str(
|
||||
p.get("presigned_url") or p.get("url") or ""
|
||||
),
|
||||
block_size=int(p.get("block_size", 0)),
|
||||
)
|
||||
)
|
||||
return _PrepareResult(
|
||||
upload_id=upload_id,
|
||||
block_size=block_size,
|
||||
parts=parts,
|
||||
concurrency=int(src.get("concurrency", _DEFAULT_CONCURRENT_PARTS)) or _DEFAULT_CONCURRENT_PARTS,
|
||||
retry_timeout=float(src.get("retry_timeout", 0.0) or 0.0),
|
||||
)
|
||||
|
||||
|
||||
# ── Chunked upload driver ────────────────────────────────────────────
|
||||
|
||||
ApiRequestFn = Callable[..., Awaitable[Dict[str, Any]]]
|
||||
"""Signature of the adapter's ``_api_request`` callable.
|
||||
|
||||
We pass the bound method in rather than importing the adapter, to avoid
|
||||
circular imports and keep this module testable in isolation.
|
||||
"""
|
||||
|
||||
|
||||
class ChunkedUploader:
|
||||
"""Run the prepare → PUT parts → complete sequence.
|
||||
|
||||
:param api_request: Bound ``_api_request(method, path, body=..., timeout=...)``
|
||||
coroutine from the adapter. Must raise ``RuntimeError`` with the biz_code
|
||||
embedded in the message on API errors.
|
||||
:param http_put: Coroutine ``(url, data, headers, timeout) -> response`` for
|
||||
COS part uploads. Typically wraps ``httpx.AsyncClient.put``.
|
||||
:param log_tag: Log prefix.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_request: ApiRequestFn,
|
||||
http_put: Callable[..., Awaitable[Any]],
|
||||
log_tag: str = "QQBot",
|
||||
) -> None:
|
||||
self._api_request = api_request
|
||||
self._http_put = http_put
|
||||
self._log_tag = log_tag
|
||||
|
||||
async def upload(
|
||||
self,
|
||||
chat_type: str,
|
||||
target_id: str,
|
||||
file_path: str,
|
||||
file_type: int,
|
||||
file_name: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Run the full chunked upload and return the ``complete_upload`` response.
|
||||
|
||||
:param chat_type: ``'c2c'`` or ``'group'``.
|
||||
:param target_id: User or group openid.
|
||||
:param file_path: Absolute path to a local file.
|
||||
:param file_type: ``MEDIA_TYPE_*`` constant.
|
||||
:param file_name: Original filename (for upload_prepare).
|
||||
:returns: The raw response dict from ``complete_upload`` — contains
|
||||
``file_info`` that the caller uses in a RichMedia message body.
|
||||
:raises UploadDailyLimitExceededError: On biz_code 40093002.
|
||||
:raises UploadFileTooLargeError: When the file exceeds the platform limit.
|
||||
:raises RuntimeError: On other API or I/O failures.
|
||||
"""
|
||||
if chat_type not in {"c2c", "group"}:
|
||||
raise ValueError(
|
||||
f"ChunkedUploader: unsupported chat_type {chat_type!r}"
|
||||
)
|
||||
|
||||
path = Path(file_path)
|
||||
file_size = path.stat().st_size
|
||||
|
||||
logger.info(
|
||||
"[%s] Chunked upload start: file=%s size=%s type=%d",
|
||||
self._log_tag, file_name, format_size(file_size), file_type,
|
||||
)
|
||||
|
||||
# Step 1: compute hashes (blocking I/O → executor).
|
||||
hashes = await asyncio.get_running_loop().run_in_executor(
|
||||
None, _compute_file_hashes, file_path, file_size
|
||||
)
|
||||
|
||||
# Step 2: upload_prepare.
|
||||
prepare = await self._prepare(
|
||||
chat_type, target_id, file_type, file_name, file_size, hashes
|
||||
)
|
||||
max_concurrent = min(prepare.concurrency, _MAX_CONCURRENT_PARTS)
|
||||
retry_timeout = min(
|
||||
prepare.retry_timeout if prepare.retry_timeout > 0 else _PART_FINISH_DEFAULT_TIMEOUT,
|
||||
_PART_FINISH_MAX_TIMEOUT,
|
||||
)
|
||||
logger.info(
|
||||
"[%s] Prepared: upload_id=%s block_size=%s parts=%d concurrency=%d",
|
||||
self._log_tag, prepare.upload_id, format_size(prepare.block_size),
|
||||
len(prepare.parts), max_concurrent,
|
||||
)
|
||||
|
||||
progress = _UploadProgress(
|
||||
total_parts=len(prepare.parts),
|
||||
total_bytes=file_size,
|
||||
)
|
||||
|
||||
# Step 3: PUT each part + notify.
|
||||
tasks: List[Callable[[], Awaitable[None]]] = [
|
||||
functools.partial(
|
||||
self._upload_one_part,
|
||||
chat_type=chat_type,
|
||||
target_id=target_id,
|
||||
file_path=file_path,
|
||||
file_size=file_size,
|
||||
upload_id=prepare.upload_id,
|
||||
rsp_block_size=prepare.block_size,
|
||||
part=part,
|
||||
retry_timeout=retry_timeout,
|
||||
progress=progress,
|
||||
)
|
||||
for part in prepare.parts
|
||||
]
|
||||
await _run_with_concurrency(tasks, max_concurrent)
|
||||
|
||||
logger.info(
|
||||
"[%s] All %d parts uploaded, completing…",
|
||||
self._log_tag, len(prepare.parts),
|
||||
)
|
||||
|
||||
# Step 4: complete_upload (retry on transient errors).
|
||||
return await self._complete(chat_type, target_id, prepare.upload_id)
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────
|
||||
# Step 1 — upload_prepare
|
||||
# ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _prepare(
|
||||
self,
|
||||
chat_type: str,
|
||||
target_id: str,
|
||||
file_type: int,
|
||||
file_name: str,
|
||||
file_size: int,
|
||||
hashes: Dict[str, str],
|
||||
) -> _PrepareResult:
|
||||
base = "/v2/users" if chat_type == "c2c" else "/v2/groups"
|
||||
path = f"{base}/{target_id}/upload_prepare"
|
||||
body = {
|
||||
"file_type": file_type,
|
||||
"file_name": file_name,
|
||||
"file_size": file_size,
|
||||
"md5": hashes["md5"],
|
||||
"sha1": hashes["sha1"],
|
||||
"md5_10m": hashes["md5_10m"],
|
||||
}
|
||||
try:
|
||||
raw = await self._api_request(
|
||||
"POST", path, body=body, timeout=FILE_UPLOAD_TIMEOUT
|
||||
)
|
||||
except RuntimeError as exc:
|
||||
err_msg = str(exc)
|
||||
if f"{_BIZ_CODE_DAILY_LIMIT}" in err_msg:
|
||||
raise UploadDailyLimitExceededError(
|
||||
file_name, file_size, err_msg
|
||||
) from exc
|
||||
raise
|
||||
return _parse_prepare_response(raw)
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────
|
||||
# Step 2 — PUT one part + part_finish
|
||||
# ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _upload_one_part(
|
||||
self,
|
||||
chat_type: str,
|
||||
target_id: str,
|
||||
file_path: str,
|
||||
file_size: int,
|
||||
upload_id: str,
|
||||
rsp_block_size: int,
|
||||
part: _PreparePart,
|
||||
retry_timeout: float,
|
||||
progress: _UploadProgress,
|
||||
) -> None:
|
||||
"""PUT one part to COS, then call ``upload_part_finish``."""
|
||||
part_index = part.index
|
||||
# Per-part block_size wins; fall back to the response-level value.
|
||||
actual_block_size = part.block_size if part.block_size > 0 else rsp_block_size
|
||||
offset = (part_index - 1) * rsp_block_size
|
||||
length = min(actual_block_size, file_size - offset)
|
||||
|
||||
# Read this slice of the file (blocking → executor).
|
||||
data = await asyncio.get_running_loop().run_in_executor(
|
||||
None, _read_file_chunk, file_path, offset, length
|
||||
)
|
||||
md5_hex = hashlib.md5(data).hexdigest()
|
||||
|
||||
logger.debug(
|
||||
"[%s] Part %d/%d: uploading %s (offset=%d md5=%s)",
|
||||
self._log_tag, part_index, progress.total_parts,
|
||||
format_size(length), offset, md5_hex,
|
||||
)
|
||||
|
||||
await self._put_to_presigned_url(
|
||||
part.presigned_url, data, part_index, progress.total_parts
|
||||
)
|
||||
await self._part_finish_with_retry(
|
||||
chat_type, target_id, upload_id,
|
||||
part_index, length, md5_hex, retry_timeout,
|
||||
)
|
||||
|
||||
progress.completed_parts += 1
|
||||
progress.uploaded_bytes += length
|
||||
logger.debug(
|
||||
"[%s] Part %d/%d done (%d/%d total)",
|
||||
self._log_tag, part_index, progress.total_parts,
|
||||
progress.completed_parts, progress.total_parts,
|
||||
)
|
||||
|
||||
async def _put_to_presigned_url(
|
||||
self,
|
||||
url: str,
|
||||
data: bytes,
|
||||
part_index: int,
|
||||
total_parts: int,
|
||||
) -> None:
|
||||
"""PUT part data to a pre-signed COS URL with retry."""
|
||||
last_exc: Optional[Exception] = None
|
||||
for attempt in range(_PART_UPLOAD_MAX_RETRIES + 1):
|
||||
try:
|
||||
resp = await asyncio.wait_for(
|
||||
self._http_put(
|
||||
url,
|
||||
data=data,
|
||||
headers={"Content-Length": str(len(data))},
|
||||
),
|
||||
timeout=_PART_UPLOAD_TIMEOUT,
|
||||
)
|
||||
# Caller's http_put is expected to return an httpx-like response.
|
||||
status = getattr(resp, "status_code", 0)
|
||||
if 200 <= status < 300:
|
||||
logger.debug(
|
||||
"[%s] PUT part %d/%d: %d OK",
|
||||
self._log_tag, part_index, total_parts, status,
|
||||
)
|
||||
return
|
||||
body_preview = ""
|
||||
try:
|
||||
body_preview = getattr(resp, "text", "")[:200]
|
||||
except Exception: # pragma: no cover — defensive
|
||||
pass
|
||||
raise RuntimeError(
|
||||
f"COS PUT returned {status}: {body_preview}"
|
||||
)
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _PART_UPLOAD_MAX_RETRIES:
|
||||
delay = 1.0 * (2 ** attempt)
|
||||
logger.warning(
|
||||
"[%s] PUT part %d/%d attempt %d failed, retry in %.1fs: %s",
|
||||
self._log_tag, part_index, total_parts,
|
||||
attempt + 1, delay, exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
raise RuntimeError(
|
||||
f"Part {part_index}/{total_parts} upload failed after "
|
||||
f"{_PART_UPLOAD_MAX_RETRIES + 1} attempts: {last_exc}"
|
||||
)
|
||||
|
||||
async def _part_finish_with_retry(
|
||||
self,
|
||||
chat_type: str,
|
||||
target_id: str,
|
||||
upload_id: str,
|
||||
part_index: int,
|
||||
block_size: int,
|
||||
md5: str,
|
||||
retry_timeout: float,
|
||||
) -> None:
|
||||
"""Call ``upload_part_finish``, retrying on biz_code 40093001."""
|
||||
base = "/v2/users" if chat_type == "c2c" else "/v2/groups"
|
||||
path = f"{base}/{target_id}/upload_part_finish"
|
||||
body = {
|
||||
"upload_id": upload_id,
|
||||
"part_index": part_index,
|
||||
"block_size": block_size,
|
||||
"md5": md5,
|
||||
}
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
start = loop.time()
|
||||
attempt = 0
|
||||
while True:
|
||||
try:
|
||||
await self._api_request(
|
||||
"POST", path, body=body, timeout=FILE_UPLOAD_TIMEOUT
|
||||
)
|
||||
return
|
||||
except RuntimeError as exc:
|
||||
err_msg = str(exc)
|
||||
if f"{_BIZ_CODE_PART_RETRYABLE}" not in err_msg:
|
||||
raise
|
||||
elapsed = loop.time() - start
|
||||
if elapsed >= retry_timeout:
|
||||
raise RuntimeError(
|
||||
f"upload_part_finish persistent retry timed out "
|
||||
f"after {retry_timeout:.0f}s ({attempt} retries): {exc}"
|
||||
) from exc
|
||||
attempt += 1
|
||||
logger.debug(
|
||||
"[%s] part_finish retryable error, attempt %d, "
|
||||
"elapsed=%.1fs: %s",
|
||||
self._log_tag, attempt, elapsed, exc,
|
||||
)
|
||||
await asyncio.sleep(_PART_FINISH_RETRY_INTERVAL)
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────
|
||||
# Step 3 — complete_upload
|
||||
# ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _complete(
|
||||
self,
|
||||
chat_type: str,
|
||||
target_id: str,
|
||||
upload_id: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Call ``complete_upload`` with retry.
|
||||
|
||||
This reuses the ``/files`` endpoint (same as the simple URL-based upload)
|
||||
but signals the chunked-completion path by sending only ``upload_id``.
|
||||
"""
|
||||
base = "/v2/users" if chat_type == "c2c" else "/v2/groups"
|
||||
path = f"{base}/{target_id}/files"
|
||||
body = {"upload_id": upload_id}
|
||||
|
||||
last_exc: Optional[Exception] = None
|
||||
for attempt in range(_COMPLETE_UPLOAD_MAX_RETRIES + 1):
|
||||
try:
|
||||
return await self._api_request(
|
||||
"POST", path, body=body, timeout=FILE_UPLOAD_TIMEOUT
|
||||
)
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _COMPLETE_UPLOAD_MAX_RETRIES:
|
||||
delay = _COMPLETE_UPLOAD_BASE_DELAY * (2 ** attempt)
|
||||
logger.warning(
|
||||
"[%s] complete_upload attempt %d failed, "
|
||||
"retry in %.1fs: %s",
|
||||
self._log_tag, attempt + 1, delay, exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
raise RuntimeError(
|
||||
f"complete_upload failed after "
|
||||
f"{_COMPLETE_UPLOAD_MAX_RETRIES + 1} attempts: {last_exc}"
|
||||
)
|
||||
|
||||
|
||||
# ── Helpers (module-level for testability) ───────────────────────────
|
||||
|
||||
def format_size(size_bytes: int) -> str:
|
||||
"""Return a human-readable file size string (e.g. ``'12.3 MB'``)."""
|
||||
size = float(size_bytes)
|
||||
for unit in ("B", "KB", "MB", "GB"):
|
||||
if size < 1024.0:
|
||||
return f"{size:.1f} {unit}"
|
||||
size /= 1024.0
|
||||
return f"{size:.1f} TB"
|
||||
|
||||
|
||||
def _read_file_chunk(file_path: str, offset: int, length: int) -> bytes:
|
||||
"""Read *length* bytes from *file_path* starting at *offset*.
|
||||
|
||||
:raises IOError: If fewer bytes were read than expected (truncated file).
|
||||
"""
|
||||
with open(file_path, "rb") as fh:
|
||||
fh.seek(offset)
|
||||
data = fh.read(length)
|
||||
if len(data) != length:
|
||||
raise IOError(
|
||||
f"Short read from {file_path}: expected {length} bytes at "
|
||||
f"offset {offset}, got {len(data)} (file may be truncated)"
|
||||
)
|
||||
return data
|
||||
|
||||
|
||||
def _compute_file_hashes(file_path: str, file_size: int) -> Dict[str, str]:
|
||||
"""Compute md5, sha1, and md5_10m in a single pass."""
|
||||
md5 = hashlib.md5()
|
||||
sha1 = hashlib.sha1()
|
||||
md5_10m = hashlib.md5()
|
||||
|
||||
need_10m = file_size > _MD5_10M_SIZE
|
||||
bytes_read = 0
|
||||
|
||||
with open(file_path, "rb") as fh:
|
||||
while True:
|
||||
chunk = fh.read(65536)
|
||||
if not chunk:
|
||||
break
|
||||
md5.update(chunk)
|
||||
sha1.update(chunk)
|
||||
if need_10m:
|
||||
remaining = _MD5_10M_SIZE - bytes_read
|
||||
if remaining > 0:
|
||||
md5_10m.update(chunk[:remaining])
|
||||
bytes_read += len(chunk)
|
||||
|
||||
full_md5 = md5.hexdigest()
|
||||
return {
|
||||
"md5": full_md5,
|
||||
"sha1": sha1.hexdigest(),
|
||||
# For small files the "10m" hash is just the full md5.
|
||||
"md5_10m": md5_10m.hexdigest() if need_10m else full_md5,
|
||||
}
|
||||
|
||||
|
||||
async def _run_with_concurrency(
|
||||
tasks: List[Callable[[], Awaitable[None]]],
|
||||
concurrency: int,
|
||||
) -> None:
|
||||
"""Run a list of thunks with a bounded number in flight at once."""
|
||||
concurrency = max(concurrency, 1)
|
||||
sem = asyncio.Semaphore(concurrency)
|
||||
|
||||
async def _wrap(thunk: Callable[[], Awaitable[None]]) -> None:
|
||||
async with sem:
|
||||
await thunk()
|
||||
|
||||
await asyncio.gather(*(_wrap(t) for t in tasks))
|
||||
@@ -0,0 +1,74 @@
|
||||
"""QQBot package-level constants shared across adapter, onboard, and other modules."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# QQBot adapter version — bump on functional changes to the adapter package.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
QQBOT_VERSION = "1.1.0"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# API endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# The portal domain is configurable via QQ_API_HOST for corporate proxies
|
||||
# or test environments. Default: q.qq.com (production).
|
||||
PORTAL_HOST = os.getenv("QQ_PORTAL_HOST", "q.qq.com")
|
||||
|
||||
API_BASE = "https://api.sgroup.qq.com"
|
||||
TOKEN_URL = "https://bots.qq.com/app/getAppAccessToken"
|
||||
GATEWAY_URL_PATH = "/gateway"
|
||||
|
||||
# QR-code onboard endpoints (on the portal host)
|
||||
ONBOARD_CREATE_PATH = "/lite/create_bind_task"
|
||||
ONBOARD_POLL_PATH = "/lite/poll_bind_result"
|
||||
QR_URL_TEMPLATE = (
|
||||
"https://q.qq.com/qqbot/openclaw/connect.html"
|
||||
"?task_id={task_id}&_wv=2&source=hermes"
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Timeouts & retry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
DEFAULT_API_TIMEOUT = 30.0
|
||||
FILE_UPLOAD_TIMEOUT = 120.0
|
||||
CONNECT_TIMEOUT_SECONDS = 20.0
|
||||
|
||||
RECONNECT_BACKOFF = [2, 5, 10, 30, 60]
|
||||
MAX_RECONNECT_ATTEMPTS = 100
|
||||
RATE_LIMIT_DELAY = 60 # seconds
|
||||
QUICK_DISCONNECT_THRESHOLD = 5.0 # seconds
|
||||
MAX_QUICK_DISCONNECT_COUNT = 3
|
||||
|
||||
ONBOARD_POLL_INTERVAL = 2.0 # seconds between poll_bind_result calls
|
||||
ONBOARD_API_TIMEOUT = 10.0
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Message limits
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MAX_MESSAGE_LENGTH = 4000
|
||||
DEDUP_WINDOW_SECONDS = 300
|
||||
DEDUP_MAX_SIZE = 1000
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# QQ Bot message types
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MSG_TYPE_TEXT = 0
|
||||
MSG_TYPE_MARKDOWN = 2
|
||||
MSG_TYPE_MEDIA = 7
|
||||
MSG_TYPE_INPUT_NOTIFY = 6
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# QQ Bot file media types
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MEDIA_TYPE_IMAGE = 1
|
||||
MEDIA_TYPE_VIDEO = 2
|
||||
MEDIA_TYPE_VOICE = 3
|
||||
MEDIA_TYPE_FILE = 4
|
||||
@@ -0,0 +1,45 @@
|
||||
"""AES-256-GCM utilities for QQBot scan-to-configure credential decryption."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import os
|
||||
|
||||
|
||||
def generate_bind_key() -> str:
|
||||
"""Generate a 256-bit random AES key and return it as base64.
|
||||
|
||||
The key is passed to ``create_bind_task`` so the server can encrypt
|
||||
the bot's *client_secret* before returning it. Only this CLI holds
|
||||
the key, ensuring the secret never travels in plaintext.
|
||||
"""
|
||||
return base64.b64encode(os.urandom(32)).decode()
|
||||
|
||||
|
||||
def decrypt_secret(encrypted_base64: str, key_base64: str) -> str:
|
||||
"""Decrypt a base64-encoded AES-256-GCM ciphertext.
|
||||
|
||||
Ciphertext layout (after base64-decoding)::
|
||||
|
||||
IV (12 bytes) ‖ ciphertext (N bytes) ‖ AuthTag (16 bytes)
|
||||
|
||||
Args:
|
||||
encrypted_base64: The ``bot_encrypt_secret`` value from
|
||||
``poll_bind_result``.
|
||||
key_base64: The base64 AES key generated by
|
||||
:func:`generate_bind_key`.
|
||||
|
||||
Returns:
|
||||
The decrypted *client_secret* as a UTF-8 string.
|
||||
"""
|
||||
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
|
||||
|
||||
key = base64.b64decode(key_base64)
|
||||
raw = base64.b64decode(encrypted_base64)
|
||||
|
||||
iv = raw[:12]
|
||||
ciphertext_with_tag = raw[12:] # AESGCM expects ciphertext + tag concatenated
|
||||
|
||||
aesgcm = AESGCM(key)
|
||||
plaintext = aesgcm.decrypt(iv, ciphertext_with_tag, None)
|
||||
return plaintext.decode("utf-8")
|
||||
@@ -0,0 +1,473 @@
|
||||
"""QQ Bot inline keyboards + approval / update-prompt senders.
|
||||
|
||||
QQ Bot v2 supports attaching inline keyboards to outbound messages. When a
|
||||
user clicks a button, the platform dispatches an ``INTERACTION_CREATE``
|
||||
gateway event containing the button's ``data`` payload. The bot must ACK the
|
||||
interaction promptly via ``PUT /interactions/{id}`` or the user sees an
|
||||
error indicator on the button.
|
||||
|
||||
This module provides:
|
||||
|
||||
- :class:`InlineKeyboard` + button dataclasses — serialized into the
|
||||
``keyboard`` field of the outbound message body.
|
||||
- :func:`build_approval_keyboard` — 3-button ✅ once / ⭐ always / ❌ deny
|
||||
keyboard for tool-approval flows.
|
||||
- :func:`build_update_prompt_keyboard` — Yes/No keyboard for update confirms.
|
||||
- :func:`parse_approval_button_data` / :func:`parse_update_prompt_button_data`
|
||||
— decode the ``button_data`` payload from ``INTERACTION_CREATE``.
|
||||
- :class:`ApprovalRequest` + :class:`ApprovalSender` — high-level helper that
|
||||
builds an approval message with keyboard and posts it to a c2c / group chat.
|
||||
|
||||
``button_data`` formats::
|
||||
|
||||
approve:<session_key>:<decision> # decision = allow-once|allow-always|deny
|
||||
update_prompt:<answer> # answer = y|n
|
||||
|
||||
Ported from WideLee's qqbot-agent-sdk v1.2.2 (``approval.py`` + ``dto.py``
|
||||
keyboard types). Authorship preserved via Co-authored-by.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Awaitable, Callable, Dict, List, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── button_data prefixes + patterns ──────────────────────────────────
|
||||
|
||||
APPROVAL_BUTTON_PREFIX = "approve:"
|
||||
UPDATE_PROMPT_PREFIX = "update_prompt:"
|
||||
|
||||
# Pattern: approve:<session_key>:<decision>
|
||||
# session_key may itself contain colons (e.g. agent:main:qqbot:c2c:OPENID),
|
||||
# so the session_key group is greedy but trails the decision.
|
||||
_APPROVAL_DATA_RE = re.compile(
|
||||
r"^approve:(.+):(allow-once|allow-always|deny)$"
|
||||
)
|
||||
|
||||
# Pattern: update_prompt:y | update_prompt:n
|
||||
_UPDATE_PROMPT_RE = re.compile(r"^update_prompt:(y|n)$")
|
||||
|
||||
|
||||
# ── Keyboard dataclasses ─────────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class KeyboardButtonPermission:
|
||||
"""Button permission metadata. ``type=2`` means all users can click."""
|
||||
type: int = 2
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {"type": self.type}
|
||||
|
||||
|
||||
@dataclass
|
||||
class KeyboardButtonAction:
|
||||
"""What happens when the button is clicked.
|
||||
|
||||
:param type: ``1`` (Callback — triggers ``INTERACTION_CREATE``) or
|
||||
``2`` (Link — opens a URL).
|
||||
:param data: Payload delivered in ``data.resolved.button_data`` when
|
||||
``type=1``.
|
||||
:param permission: :class:`KeyboardButtonPermission`.
|
||||
:param click_limit: Max clicks per user (``1`` = single-use).
|
||||
"""
|
||||
type: int
|
||||
data: str
|
||||
permission: KeyboardButtonPermission = field(
|
||||
default_factory=KeyboardButtonPermission
|
||||
)
|
||||
click_limit: int = 1
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"type": self.type,
|
||||
"data": self.data,
|
||||
"permission": self.permission.to_dict(),
|
||||
"click_limit": self.click_limit,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class KeyboardButtonRenderData:
|
||||
"""Visual rendering of a button.
|
||||
|
||||
:param label: Pre-click label.
|
||||
:param visited_label: Post-click label (button stays greyed in place).
|
||||
:param style: ``0`` = grey, ``1`` = blue.
|
||||
"""
|
||||
label: str
|
||||
visited_label: str
|
||||
style: int = 1
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"label": self.label,
|
||||
"visited_label": self.visited_label,
|
||||
"style": self.style,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class KeyboardButton:
|
||||
"""One button in a keyboard.
|
||||
|
||||
:param group_id: Buttons sharing a ``group_id`` are mutually exclusive —
|
||||
clicking one greys the rest.
|
||||
"""
|
||||
id: str
|
||||
render_data: KeyboardButtonRenderData
|
||||
action: KeyboardButtonAction
|
||||
group_id: str = "default"
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"id": self.id,
|
||||
"render_data": self.render_data.to_dict(),
|
||||
"action": self.action.to_dict(),
|
||||
"group_id": self.group_id,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class KeyboardRow:
|
||||
buttons: List[KeyboardButton] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {"buttons": [b.to_dict() for b in self.buttons]}
|
||||
|
||||
|
||||
@dataclass
|
||||
class KeyboardContent:
|
||||
rows: List[KeyboardRow] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {"rows": [r.to_dict() for r in self.rows]}
|
||||
|
||||
|
||||
@dataclass
|
||||
class InlineKeyboard:
|
||||
"""Top-level keyboard payload — goes into ``MessageToCreate.keyboard``."""
|
||||
content: KeyboardContent = field(default_factory=KeyboardContent)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {"content": self.content.to_dict()}
|
||||
|
||||
|
||||
# ── INTERACTION_CREATE parsing ───────────────────────────────────────
|
||||
|
||||
def parse_approval_button_data(button_data: str) -> Optional[tuple[str, str]]:
|
||||
"""Parse approval ``button_data`` into ``(session_key, decision)``.
|
||||
|
||||
:param button_data: Raw ``data.resolved.button_data`` from
|
||||
``INTERACTION_CREATE``.
|
||||
:returns: ``(session_key, decision)`` or ``None`` if not an approval button.
|
||||
"""
|
||||
m = _APPROVAL_DATA_RE.match(button_data or "")
|
||||
if not m:
|
||||
return None
|
||||
return m.group(1), m.group(2)
|
||||
|
||||
|
||||
def parse_update_prompt_button_data(button_data: str) -> Optional[str]:
|
||||
"""Parse update-prompt ``button_data`` into ``'y'`` or ``'n'``."""
|
||||
m = _UPDATE_PROMPT_RE.match(button_data or "")
|
||||
if not m:
|
||||
return None
|
||||
return m.group(1)
|
||||
|
||||
|
||||
# ── Keyboard builders ────────────────────────────────────────────────
|
||||
|
||||
def _make_callback_button(
|
||||
btn_id: str,
|
||||
label: str,
|
||||
visited_label: str,
|
||||
data: str,
|
||||
style: int,
|
||||
group_id: str,
|
||||
) -> KeyboardButton:
|
||||
return KeyboardButton(
|
||||
id=btn_id,
|
||||
render_data=KeyboardButtonRenderData(
|
||||
label=label,
|
||||
visited_label=visited_label,
|
||||
style=style,
|
||||
),
|
||||
action=KeyboardButtonAction(type=1, data=data),
|
||||
group_id=group_id,
|
||||
)
|
||||
|
||||
|
||||
def build_approval_keyboard(session_key: str) -> InlineKeyboard:
|
||||
"""Build the 3-button approval keyboard.
|
||||
|
||||
Layout: ``[✅ 允许一次] [⭐ 始终允许] [❌ 拒绝]`` — all three share
|
||||
``group_id='approval'`` so clicking one greys out the rest.
|
||||
|
||||
:param session_key: Embedded into ``button_data`` so the decision
|
||||
routes back to the right pending approval.
|
||||
"""
|
||||
return InlineKeyboard(
|
||||
content=KeyboardContent(
|
||||
rows=[
|
||||
KeyboardRow(buttons=[
|
||||
_make_callback_button(
|
||||
btn_id="allow",
|
||||
label="✅ 允许一次",
|
||||
visited_label="已允许",
|
||||
data=f"{APPROVAL_BUTTON_PREFIX}{session_key}:allow-once",
|
||||
style=1,
|
||||
group_id="approval",
|
||||
),
|
||||
_make_callback_button(
|
||||
btn_id="always",
|
||||
label="⭐ 始终允许",
|
||||
visited_label="已始终允许",
|
||||
data=f"{APPROVAL_BUTTON_PREFIX}{session_key}:allow-always",
|
||||
style=1,
|
||||
group_id="approval",
|
||||
),
|
||||
_make_callback_button(
|
||||
btn_id="deny",
|
||||
label="❌ 拒绝",
|
||||
visited_label="已拒绝",
|
||||
data=f"{APPROVAL_BUTTON_PREFIX}{session_key}:deny",
|
||||
style=0,
|
||||
group_id="approval",
|
||||
),
|
||||
]),
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def build_update_prompt_keyboard() -> InlineKeyboard:
|
||||
"""Build a Yes/No keyboard for update confirmation prompts."""
|
||||
return InlineKeyboard(
|
||||
content=KeyboardContent(
|
||||
rows=[
|
||||
KeyboardRow(buttons=[
|
||||
_make_callback_button(
|
||||
btn_id="yes",
|
||||
label="✓ 确认",
|
||||
visited_label="已确认",
|
||||
data=f"{UPDATE_PROMPT_PREFIX}y",
|
||||
style=1,
|
||||
group_id="update_prompt",
|
||||
),
|
||||
_make_callback_button(
|
||||
btn_id="no",
|
||||
label="✗ 取消",
|
||||
visited_label="已取消",
|
||||
data=f"{UPDATE_PROMPT_PREFIX}n",
|
||||
style=0,
|
||||
group_id="update_prompt",
|
||||
),
|
||||
]),
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# ── ApprovalRequest + text builder ───────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class ApprovalRequest:
|
||||
"""Structured approval-request display data.
|
||||
|
||||
:param session_key: Routes the decision back to the waiting caller.
|
||||
:param title: Short title at the top.
|
||||
:param description: Optional longer description.
|
||||
:param command_preview: Command text (exec approvals).
|
||||
:param cwd: Working directory (exec approvals).
|
||||
:param tool_name: Tool name (plugin approvals).
|
||||
:param severity: ``'critical' | 'info' | ''``.
|
||||
:param timeout_sec: Seconds until the approval expires.
|
||||
"""
|
||||
session_key: str
|
||||
title: str
|
||||
description: str = ""
|
||||
command_preview: str = ""
|
||||
cwd: str = ""
|
||||
tool_name: str = ""
|
||||
severity: str = ""
|
||||
timeout_sec: int = 120
|
||||
|
||||
|
||||
def build_approval_text(req: ApprovalRequest) -> str:
|
||||
"""Render an :class:`ApprovalRequest` into the message body (markdown)."""
|
||||
if req.command_preview or req.cwd:
|
||||
return _build_exec_text(req)
|
||||
return _build_plugin_text(req)
|
||||
|
||||
|
||||
def _build_exec_text(req: ApprovalRequest) -> str:
|
||||
lines: List[str] = ["🔐 **命令执行审批**", ""]
|
||||
if req.command_preview:
|
||||
preview = req.command_preview[:300]
|
||||
lines.append(f"```\n{preview}\n```")
|
||||
if req.cwd:
|
||||
lines.append(f"📁 目录: {req.cwd}")
|
||||
if req.title and req.title != req.command_preview:
|
||||
lines.append(f"📋 {req.title}")
|
||||
if req.description:
|
||||
lines.append(f"📝 {req.description}")
|
||||
lines.append("")
|
||||
lines.append(f"⏱️ 超时: {req.timeout_sec} 秒")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _build_plugin_text(req: ApprovalRequest) -> str:
|
||||
icon = (
|
||||
"🔴" if req.severity == "critical"
|
||||
else "🔵" if req.severity == "info"
|
||||
else "🟡"
|
||||
)
|
||||
lines: List[str] = [f"{icon} **审批请求**", ""]
|
||||
lines.append(f"📋 {req.title}")
|
||||
if req.description:
|
||||
lines.append(f"📝 {req.description}")
|
||||
if req.tool_name:
|
||||
lines.append(f"🔧 工具: {req.tool_name}")
|
||||
lines.append("")
|
||||
lines.append(f"⏱️ 超时: {req.timeout_sec} 秒")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ── ApprovalSender ───────────────────────────────────────────────────
|
||||
|
||||
PostMessageFn = Callable[..., Awaitable[Dict[str, Any]]]
|
||||
"""Signature of an async POST to ``/v2/{users|groups}/{id}/messages``.
|
||||
|
||||
Implementations accept a body dict and return the raw API response.
|
||||
"""
|
||||
|
||||
|
||||
class ApprovalSender:
|
||||
"""Send an approval-request message with an inline keyboard.
|
||||
|
||||
Decoupled from the adapter via callables so it can be unit-tested in
|
||||
isolation. Pass the adapter's ``_send_message_with_keyboard`` helper
|
||||
(or any equivalent) as ``post_message``.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
post_c2c: PostMessageFn,
|
||||
post_group: PostMessageFn,
|
||||
log_tag: str = "QQBot",
|
||||
) -> None:
|
||||
self._post_c2c = post_c2c
|
||||
self._post_group = post_group
|
||||
self._log_tag = log_tag
|
||||
|
||||
async def send(
|
||||
self,
|
||||
chat_type: str,
|
||||
chat_id: str,
|
||||
req: ApprovalRequest,
|
||||
msg_id: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""Send an approval message to *chat_id*.
|
||||
|
||||
:param chat_type: ``'c2c'`` or ``'group'``.
|
||||
:param chat_id: User openid or group openid.
|
||||
:param req: :class:`ApprovalRequest`.
|
||||
:param msg_id: Reply-to message id (required for passive messages).
|
||||
:returns: ``True`` on success, ``False`` on failure.
|
||||
"""
|
||||
text = build_approval_text(req)
|
||||
keyboard = build_approval_keyboard(req.session_key)
|
||||
|
||||
logger.info(
|
||||
"[%s] Sending approval request to %s:%s (session=%.20s…)",
|
||||
self._log_tag, chat_type, chat_id, req.session_key,
|
||||
)
|
||||
|
||||
try:
|
||||
if chat_type == "c2c":
|
||||
await self._post_c2c(chat_id, text, msg_id, keyboard)
|
||||
elif chat_type == "group":
|
||||
await self._post_group(chat_id, text, msg_id, keyboard)
|
||||
else:
|
||||
logger.warning(
|
||||
"[%s] Approval: unsupported chat_type %r",
|
||||
self._log_tag, chat_type,
|
||||
)
|
||||
return False
|
||||
logger.info(
|
||||
"[%s] Approval message sent to %s:%s",
|
||||
self._log_tag, chat_type, chat_id,
|
||||
)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error(
|
||||
"[%s] Failed to send approval message to %s:%s: %s",
|
||||
self._log_tag, chat_type, chat_id, exc,
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
# ── INTERACTION_CREATE event shape ───────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class InteractionEvent:
|
||||
"""Parsed ``INTERACTION_CREATE`` event payload.
|
||||
|
||||
See https://bot.q.qq.com/wiki/develop/api-v2/dev-prepare/interface-framework/event-emit.html
|
||||
"""
|
||||
id: str = ""
|
||||
"""Interaction event id — required for the ``PUT /interactions/{id}`` ACK."""
|
||||
|
||||
type: int = 0
|
||||
"""Event type code (``11`` = message button)."""
|
||||
|
||||
chat_type: int = 0
|
||||
"""``0`` = guild, ``1`` = group, ``2`` = c2c."""
|
||||
|
||||
scene: str = ""
|
||||
"""``'guild'`` | ``'group'`` | ``'c2c'`` — human-readable scene."""
|
||||
|
||||
group_openid: str = ""
|
||||
group_member_openid: str = ""
|
||||
user_openid: str = ""
|
||||
channel_id: str = ""
|
||||
guild_id: str = ""
|
||||
|
||||
button_data: str = ""
|
||||
button_id: str = ""
|
||||
resolver_user_id: str = ""
|
||||
|
||||
@property
|
||||
def operator_openid(self) -> str:
|
||||
"""Best available operator openid (group → member; c2c → user)."""
|
||||
return (
|
||||
self.group_member_openid
|
||||
or self.user_openid
|
||||
or self.resolver_user_id
|
||||
)
|
||||
|
||||
|
||||
def parse_interaction_event(raw: Dict[str, Any]) -> InteractionEvent:
|
||||
"""Parse a raw ``INTERACTION_CREATE`` dispatch payload (``d``)."""
|
||||
data_raw = raw.get("data") or {}
|
||||
resolved = data_raw.get("resolved") or {}
|
||||
scene_code = int(raw.get("chat_type", 0) or 0)
|
||||
scene = {0: "guild", 1: "group", 2: "c2c"}.get(scene_code, "")
|
||||
return InteractionEvent(
|
||||
id=str(raw.get("id", "")),
|
||||
type=int(data_raw.get("type", 0) or 0),
|
||||
chat_type=scene_code,
|
||||
scene=scene,
|
||||
group_openid=str(raw.get("group_openid", "")),
|
||||
group_member_openid=str(raw.get("group_member_openid", "")),
|
||||
user_openid=str(raw.get("user_openid", "")),
|
||||
channel_id=str(raw.get("channel_id", "")),
|
||||
guild_id=str(raw.get("guild_id", "")),
|
||||
button_data=str(resolved.get("button_data", "")),
|
||||
button_id=str(resolved.get("button_id", "")),
|
||||
resolver_user_id=str(resolved.get("user_id", "")),
|
||||
)
|
||||
@@ -0,0 +1,220 @@
|
||||
"""
|
||||
QQBot scan-to-configure (QR code onboard) module.
|
||||
|
||||
Mirrors the Feishu onboarding pattern: synchronous HTTP + a single public
|
||||
entry-point ``qr_register()`` that handles the full flow (create task →
|
||||
display QR code → poll → decrypt credentials).
|
||||
|
||||
Calls the ``q.qq.com`` ``create_bind_task`` / ``poll_bind_result`` APIs to
|
||||
generate a QR-code URL and poll for scan completion. On success the caller
|
||||
receives the bot's *app_id*, *client_secret* (decrypted locally), and the
|
||||
scanner's *user_openid* — enough to fully configure the QQBot gateway.
|
||||
|
||||
Reference: https://bot.q.qq.com/wiki/develop/api-v2/
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from enum import IntEnum
|
||||
from typing import Optional, Tuple
|
||||
from urllib.parse import quote
|
||||
|
||||
from .constants import (
|
||||
ONBOARD_API_TIMEOUT,
|
||||
ONBOARD_CREATE_PATH,
|
||||
ONBOARD_POLL_INTERVAL,
|
||||
ONBOARD_POLL_PATH,
|
||||
PORTAL_HOST,
|
||||
QR_URL_TEMPLATE,
|
||||
)
|
||||
from .crypto import decrypt_secret, generate_bind_key
|
||||
from .utils import get_api_headers
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bind status
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class BindStatus(IntEnum):
|
||||
"""Status codes returned by ``_poll_bind_result``."""
|
||||
|
||||
NONE = 0
|
||||
PENDING = 1
|
||||
COMPLETED = 2
|
||||
EXPIRED = 3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# QR rendering
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
try:
|
||||
import qrcode as _qrcode_mod
|
||||
except (ImportError, TypeError):
|
||||
_qrcode_mod = None # type: ignore[assignment]
|
||||
|
||||
|
||||
def _render_qr(url: str) -> bool:
|
||||
"""Try to render a QR code in the terminal. Returns True if successful."""
|
||||
if _qrcode_mod is None:
|
||||
return False
|
||||
try:
|
||||
qr = _qrcode_mod.QRCode(
|
||||
error_correction=_qrcode_mod.constants.ERROR_CORRECT_M,
|
||||
border=2,
|
||||
)
|
||||
qr.add_data(url)
|
||||
qr.make(fit=True)
|
||||
qr.print_ascii(invert=True)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Synchronous HTTP helpers (mirrors Feishu _post_registration pattern)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _create_bind_task(timeout: float = ONBOARD_API_TIMEOUT) -> Tuple[str, str]:
|
||||
"""Create a bind task and return *(task_id, aes_key_base64)*.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If the API returns a non-zero ``retcode``.
|
||||
"""
|
||||
import httpx
|
||||
|
||||
url = f"https://{PORTAL_HOST}{ONBOARD_CREATE_PATH}"
|
||||
key = generate_bind_key()
|
||||
|
||||
with httpx.Client(timeout=timeout, follow_redirects=True) as client:
|
||||
resp = client.post(url, json={"key": key}, headers=get_api_headers())
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
if data.get("retcode") != 0:
|
||||
raise RuntimeError(data.get("msg", "create_bind_task failed"))
|
||||
|
||||
task_id = data.get("data", {}).get("task_id")
|
||||
if not task_id:
|
||||
raise RuntimeError("create_bind_task: missing task_id in response")
|
||||
|
||||
logger.debug("create_bind_task ok: task_id=%s", task_id)
|
||||
return task_id, key
|
||||
|
||||
|
||||
def _poll_bind_result(
|
||||
task_id: str,
|
||||
timeout: float = ONBOARD_API_TIMEOUT,
|
||||
) -> Tuple[BindStatus, str, str, str]:
|
||||
"""Poll the bind result for *task_id*.
|
||||
|
||||
Returns:
|
||||
A 4-tuple of ``(status, bot_appid, bot_encrypt_secret, user_openid)``.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If the API returns a non-zero ``retcode``.
|
||||
"""
|
||||
import httpx
|
||||
|
||||
url = f"https://{PORTAL_HOST}{ONBOARD_POLL_PATH}"
|
||||
|
||||
with httpx.Client(timeout=timeout, follow_redirects=True) as client:
|
||||
resp = client.post(url, json={"task_id": task_id}, headers=get_api_headers())
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
if data.get("retcode") != 0:
|
||||
raise RuntimeError(data.get("msg", "poll_bind_result failed"))
|
||||
|
||||
d = data.get("data", {})
|
||||
return (
|
||||
BindStatus(d.get("status", 0)),
|
||||
str(d.get("bot_appid", "")),
|
||||
d.get("bot_encrypt_secret", ""),
|
||||
d.get("user_openid", ""),
|
||||
)
|
||||
|
||||
|
||||
def build_connect_url(task_id: str) -> str:
|
||||
"""Build the QR-code target URL for a given *task_id*."""
|
||||
return QR_URL_TEMPLATE.format(task_id=quote(task_id))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public entry-point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_MAX_REFRESHES = 3
|
||||
|
||||
|
||||
def qr_register(timeout_seconds: int = 600) -> Optional[dict]:
|
||||
"""Run the QQBot scan-to-configure QR registration flow.
|
||||
|
||||
Mirrors ``feishu.qr_register()``: handles create → display → poll →
|
||||
decrypt in one call. Unexpected errors propagate to the caller.
|
||||
|
||||
:returns:
|
||||
``{"app_id": ..., "client_secret": ..., "user_openid": ...}`` on
|
||||
success, or ``None`` on failure / expiry / cancellation.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout_seconds
|
||||
|
||||
for refresh_count in range(_MAX_REFRESHES + 1):
|
||||
# ── Create bind task ──
|
||||
try:
|
||||
task_id, aes_key = _create_bind_task()
|
||||
except Exception as exc:
|
||||
logger.warning("[QQBot onboard] Failed to create bind task: %s", exc)
|
||||
return None
|
||||
|
||||
url = build_connect_url(task_id)
|
||||
|
||||
# ── Display QR code + URL ──
|
||||
print()
|
||||
if _render_qr(url):
|
||||
print(f" Scan the QR code above, or open this URL directly:\n {url}")
|
||||
else:
|
||||
print(f" Open this URL in QQ on your phone:\n {url}")
|
||||
print(" Tip: pip install qrcode to display a scannable QR code here")
|
||||
print()
|
||||
|
||||
# ── Poll loop ──
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
status, app_id, encrypted_secret, user_openid = _poll_bind_result(task_id)
|
||||
except Exception:
|
||||
time.sleep(ONBOARD_POLL_INTERVAL)
|
||||
continue
|
||||
|
||||
if status == BindStatus.COMPLETED:
|
||||
client_secret = decrypt_secret(encrypted_secret, aes_key)
|
||||
print()
|
||||
print(f" QR scan complete! (App ID: {app_id})")
|
||||
if user_openid:
|
||||
print(f" Scanner's OpenID: {user_openid}")
|
||||
return {
|
||||
"app_id": app_id,
|
||||
"client_secret": client_secret,
|
||||
"user_openid": user_openid,
|
||||
}
|
||||
|
||||
if status == BindStatus.EXPIRED:
|
||||
if refresh_count >= _MAX_REFRESHES:
|
||||
logger.warning("[QQBot onboard] QR code expired %d times — giving up", _MAX_REFRESHES)
|
||||
return None
|
||||
print(f"\n QR code expired, refreshing... ({refresh_count + 1}/{_MAX_REFRESHES})")
|
||||
break # next for-loop iteration creates a new task
|
||||
|
||||
time.sleep(ONBOARD_POLL_INTERVAL)
|
||||
else:
|
||||
# deadline reached without completing
|
||||
logger.warning("[QQBot onboard] Poll timed out after %ds", timeout_seconds)
|
||||
return None
|
||||
|
||||
return None
|
||||
@@ -0,0 +1,71 @@
|
||||
"""QQBot shared utilities — User-Agent, HTTP helpers, config coercion."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import platform
|
||||
import sys
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from .constants import QQBOT_VERSION
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# User-Agent
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _get_hermes_version() -> str:
|
||||
"""Return the hermes-agent package version, or 'dev' if unavailable."""
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
return version("hermes-agent")
|
||||
except Exception:
|
||||
return "dev"
|
||||
|
||||
|
||||
def build_user_agent() -> str:
|
||||
"""Build a descriptive User-Agent string.
|
||||
|
||||
Format::
|
||||
|
||||
QQBotAdapter/<qqbot_version> (Python/<py_version>; <os>; Hermes/<hermes_version>)
|
||||
|
||||
Example::
|
||||
|
||||
QQBotAdapter/1.0.0 (Python/3.11.15; darwin; Hermes/0.9.0)
|
||||
"""
|
||||
py_version = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
|
||||
os_name = platform.system().lower()
|
||||
hermes_version = _get_hermes_version()
|
||||
return f"QQBotAdapter/{QQBOT_VERSION} (Python/{py_version}; {os_name}; Hermes/{hermes_version})"
|
||||
|
||||
|
||||
def get_api_headers() -> Dict[str, str]:
|
||||
"""Return standard HTTP headers for QQBot API requests.
|
||||
|
||||
Includes ``Content-Type``, ``Accept``, and a dynamic ``User-Agent``.
|
||||
``q.qq.com`` requires ``Accept: application/json`` — without it,
|
||||
the server returns a JavaScript anti-bot challenge page.
|
||||
"""
|
||||
return {
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json",
|
||||
"User-Agent": build_user_agent(),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def coerce_list(value: Any) -> List[str]:
|
||||
"""Coerce config values into a trimmed string list.
|
||||
|
||||
Accepts comma-separated strings, lists, tuples, sets, or single values.
|
||||
"""
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
return [item.strip() for item in value.split(",") if item.strip()]
|
||||
if isinstance(value, (list, tuple, set)):
|
||||
return [str(item).strip() for item in value if str(item).strip()]
|
||||
return [str(value).strip()] if str(value).strip() else []
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,369 @@
|
||||
"""
|
||||
Signal attachment rate-limit scheduler.
|
||||
|
||||
Process-wide token-bucket simulator that mirrors the per-account
|
||||
attachment rate limit signal-cli/Signal-Server enforce. Producers
|
||||
(``SignalAdapter.send_multiple_images`` and the ``send_message`` tool's
|
||||
Signal path) call ``acquire(n)`` before an attachment send; on a 429
|
||||
they call ``feedback(retry_after, n)`` so the model recalibrates from
|
||||
the server's authoritative hint.
|
||||
|
||||
The scheduler serializes concurrent calls through an ``asyncio.Lock``,
|
||||
giving FIFO fairness across agent sessions sharing one signal-cli
|
||||
daemon.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from typing import Any, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SIGNAL_MAX_ATTACHMENTS_PER_MSG = 32 # per-message attachment cap (source: Signal-{Android,Desktop} source code)
|
||||
SIGNAL_RATE_LIMIT_BUCKET_CAPACITY = 50 # server-side token-bucket capacity for attachments rate limiting
|
||||
SIGNAL_RATE_LIMIT_DEFAULT_RETRY_AFTER = 4 # fallback token refill interval for signal-cli < v0.14.3
|
||||
SIGNAL_RATE_LIMIT_MAX_ATTEMPTS = 2 # initial attempt + 1 retry
|
||||
SIGNAL_BATCH_PACING_NOTICE_THRESHOLD = 10.0 # if estimated waiting time > 10s, notify the user about the delay
|
||||
SIGNAL_RPC_ERROR_RATELIMIT = -5 # signal-cli (v0.14.3+) JSON-RPC error code for RateLimitException
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Errors
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class SignalRateLimitError(Exception):
|
||||
"""
|
||||
Raised by ``SignalAdapter._rpc`` for rate-limit responses when the
|
||||
caller has opted in via ``raise_on_rate_limit=True``.
|
||||
|
||||
Carries the server-supplied per-token Retry-After (in seconds) on
|
||||
signal-cli ≥ v0.14.3
|
||||
``retry_after`` is None when the version doesn't expose it.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, retry_after: Optional[float] = None) -> None:
|
||||
super().__init__(message)
|
||||
self.retry_after = retry_after
|
||||
|
||||
|
||||
class SignalSchedulerError(Exception):
|
||||
pass
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Detection helpers — used to fish a 429 out of signal-cli's various error
|
||||
# shapes (typed code, [429] substring, libsignal-net RetryLaterException
|
||||
# leaked through AttachmentInvalidException).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# "Retry after 4 seconds" / "retry after 4 second" — libsignal-net's
|
||||
# RetryLaterException string form, surfaced when 429s hit during
|
||||
# attachment upload (signal-cli wraps these as AttachmentInvalidException
|
||||
# rather than RateLimitException, so the typed path doesn't fire).
|
||||
_RETRY_AFTER_RE = re.compile(r"Retry after (\d+(?:\.\d+)?)\s*second", re.IGNORECASE)
|
||||
|
||||
|
||||
def _extract_retry_after_seconds(err: Any) -> Optional[float]:
|
||||
"""Pull the per-token Retry-After window from a signal-cli rate-limit error.
|
||||
|
||||
Tries two sources, in order:
|
||||
1. ``error.data.response.results[*].retryAfterSeconds`` — the
|
||||
structured field signal-cli ≥ v0.14.3 surfaces for plain
|
||||
RateLimitException.
|
||||
2. ``"Retry after N seconds"`` parsed out of the message — covers
|
||||
libsignal-net's RetryLaterException that gets wrapped as
|
||||
AttachmentInvalidException during attachment upload, where the
|
||||
structured field stays null.
|
||||
|
||||
Returns None when neither yields a value.
|
||||
"""
|
||||
msg = ""
|
||||
if isinstance(err, dict):
|
||||
data = err.get("data") or {}
|
||||
response = data.get("response") or {}
|
||||
results = response.get("results") or []
|
||||
candidates = [
|
||||
r.get("retryAfterSeconds") for r in results
|
||||
if isinstance(r, dict) and r.get("retryAfterSeconds")
|
||||
]
|
||||
if candidates:
|
||||
return float(max(candidates))
|
||||
msg = str(err.get("message", ""))
|
||||
else:
|
||||
msg = str(err)
|
||||
match = _RETRY_AFTER_RE.search(msg)
|
||||
return float(match.group(1)) if match else None
|
||||
|
||||
|
||||
def _is_signal_rate_limit_error(err: Any) -> bool:
|
||||
"""True if a signal-cli RPC error reflects a rate-limit failure.
|
||||
|
||||
Matches three layers:
|
||||
- typed ``RATELIMIT_ERROR`` code (signal-cli ≥ v0.14.3, plain
|
||||
RateLimitException)
|
||||
- legacy ``[429] / RateLimitException`` substrings
|
||||
- libsignal-net's ``RetryLaterException`` / ``Retry after N seconds``
|
||||
surfaced inside ``AttachmentInvalidException`` when the rate
|
||||
limit is hit during attachment upload — signal-cli never re-tags
|
||||
these as RateLimitException, so substring is the only signal.
|
||||
"""
|
||||
if isinstance(err, dict) and err.get("code") == SIGNAL_RPC_ERROR_RATELIMIT:
|
||||
return True
|
||||
|
||||
message = (
|
||||
str(err.get("message", ""))
|
||||
if isinstance(err, dict)
|
||||
else str(err)
|
||||
)
|
||||
msg_lower = message.lower()
|
||||
return (
|
||||
"[429]" in message
|
||||
or "ratelimit" in msg_lower
|
||||
or "retrylaterexception" in msg_lower
|
||||
or "retry after" in msg_lower
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Misc helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _format_wait(seconds: float) -> str:
|
||||
"""Human-friendly wait label for user-facing pacing notices."""
|
||||
s = max(0.0, seconds)
|
||||
if s < 90:
|
||||
return f"{int(round(s))}s"
|
||||
return f"{max(1, int(round(s / 60)))} min"
|
||||
|
||||
|
||||
def _signal_send_timeout(num_attachments: int) -> float:
|
||||
"""HTTP timeout for a Signal ``send`` RPC.
|
||||
|
||||
signal-cli uploads attachments serially during the call, so the
|
||||
server-side time scales with batch size. Default 30s is fine for
|
||||
text-only sends but truncates large attachment batches mid-upload —
|
||||
we then log a phantom failure even though signal-cli completes the
|
||||
send a few seconds later. Scale at 5s/attachment with a 60s floor.
|
||||
"""
|
||||
if num_attachments <= 0:
|
||||
return 30.0
|
||||
return max(60.0, 5.0 * num_attachments)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scheduler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class SignalAttachmentScheduler:
|
||||
"""Process-wide token-bucket simulator for Signal attachment sends.
|
||||
|
||||
The bucket holds up to ``capacity`` tokens (default 50, matching
|
||||
Signal's server-side rate-limit bucket size). Each attachment consumes one
|
||||
token. Tokens refill at ``refill_rate`` tokens/second, calibrated
|
||||
from the per-token Retry-After hint we get from the server when a
|
||||
429 fires. Until we've observed one, we use the documented default
|
||||
(1 token / 4 seconds).
|
||||
|
||||
Concurrent ``acquire(n)`` calls serialize through an
|
||||
``asyncio.Lock`` — natural FIFO across agent sessions hitting the
|
||||
same daemon.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
capacity: float = float(SIGNAL_RATE_LIMIT_BUCKET_CAPACITY),
|
||||
default_retry_after: float = float(SIGNAL_RATE_LIMIT_DEFAULT_RETRY_AFTER),
|
||||
) -> None:
|
||||
self.capacity = float(capacity)
|
||||
self.tokens = float(capacity)
|
||||
self.refill_rate = 1.0 / float(default_retry_after)
|
||||
self.last_refill = time.monotonic()
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internals
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _refill(self) -> None:
|
||||
now = time.monotonic()
|
||||
elapsed = now - self.last_refill
|
||||
if elapsed > 0 and self.tokens < self.capacity:
|
||||
self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
|
||||
self.last_refill = now
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def estimate_wait(self, n: int) -> float:
|
||||
"""Best-effort estimate of the seconds until ``n`` tokens would
|
||||
be available. Used to decide whether to emit a user-facing
|
||||
pacing notice *before* committing to an ``acquire`` that may
|
||||
block silently. Lock-free; small races vs. concurrent acquires
|
||||
are benign for an informational notice.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
elapsed = now - self.last_refill
|
||||
projected = self.tokens
|
||||
if elapsed > 0 and projected < self.capacity:
|
||||
projected = min(self.capacity, projected + elapsed * self.refill_rate)
|
||||
deficit = n - projected
|
||||
if deficit <= 0:
|
||||
return 0.0
|
||||
return deficit / self.refill_rate
|
||||
|
||||
async def acquire(self, n: int) -> float:
|
||||
"""Block until at least ``n`` tokens are available, return the
|
||||
seconds slept.
|
||||
|
||||
Does **not** deduct tokens — the bucket is a read-only model of
|
||||
server-side capacity. Call ``report_rpc_duration()`` after the
|
||||
RPC to synchronise the model with the server timeline.
|
||||
|
||||
Not perfect in case lots of coroutines try to acquire for big
|
||||
uploads (``report_rpc_duration`` will take a long time to get hit)
|
||||
but this is just a simulation. Signal server is ground truth and
|
||||
will raise rate-limit exceptions triggering requeues.
|
||||
|
||||
The lock is released during ``asyncio.sleep`` so other callers
|
||||
can interleave. A retry loop re-checks after each sleep in
|
||||
case the deadline was pessimistic.
|
||||
"""
|
||||
if n <= 0:
|
||||
return 0.0
|
||||
if n > self.capacity:
|
||||
raise SignalSchedulerError(
|
||||
f"Signal scheduler was called requesting {n} tokens "
|
||||
f"(max is {self.capacity})",
|
||||
)
|
||||
|
||||
total_slept = 0.0
|
||||
first_pass = True
|
||||
while True:
|
||||
async with self._lock:
|
||||
self._refill()
|
||||
if self.tokens >= n:
|
||||
if not first_pass or total_slept > 0:
|
||||
logger.debug(
|
||||
"Signal scheduler: tokens sufficient for %d "
|
||||
"(remaining=%.1f, total_slept=%.1fs)",
|
||||
n, self.tokens, total_slept,
|
||||
)
|
||||
return total_slept
|
||||
deficit = n - self.tokens
|
||||
wait = deficit / self.refill_rate
|
||||
if first_pass:
|
||||
logger.info(
|
||||
"Signal scheduler: pausing %.1fs for %d tokens "
|
||||
"(available=%.1f, deficit=%.1f, refill=%.4f/s ≈ %.1fs/token)",
|
||||
wait, n, self.tokens, deficit,
|
||||
self.refill_rate, 1.0 / self.refill_rate,
|
||||
)
|
||||
first_pass = False
|
||||
await asyncio.sleep(wait)
|
||||
total_slept += wait
|
||||
|
||||
async def report_rpc_duration(self, rpc_duration: float, n_attachments: int) -> None:
|
||||
"""Record an attachment-send RPC that just completed.
|
||||
|
||||
Deducts ``n_attachments`` tokens without crediting refill during
|
||||
the upload window. Signal's server checks the bucket at RPC start
|
||||
and does *not* refill during request processing — refill resumes
|
||||
after the response. Crediting upload-time refill causes cumulative
|
||||
drift that eventually triggers 429s.
|
||||
|
||||
Advances ``last_refill`` so the next ``acquire`` / ``_refill``
|
||||
starts counting from this point.
|
||||
"""
|
||||
if n_attachments <= 0:
|
||||
return
|
||||
|
||||
async with self._lock:
|
||||
now = time.monotonic()
|
||||
token_before = self.tokens
|
||||
self.tokens = max(0.0, token_before - float(n_attachments))
|
||||
self.last_refill = now
|
||||
logger.log(
|
||||
logging.INFO if rpc_duration > 10 and n_attachments > 5 else logging.DEBUG,
|
||||
"Signal scheduler: RPC for %d att took %.1fs — "
|
||||
"tokens %.1f → %.1f (deducted=%d, no upload refill credited, refill=%.4fs⁻¹)",
|
||||
n_attachments, rpc_duration,
|
||||
token_before, self.tokens,
|
||||
n_attachments, self.refill_rate,
|
||||
)
|
||||
|
||||
def feedback(self, retry_after: Optional[float], n_attempted: int) -> None:
|
||||
"""Apply server feedback after a 429.
|
||||
|
||||
``retry_after`` is the per-*token* refill window the server
|
||||
reports (None when signal-cli is older than v0.14.3 and didn't
|
||||
surface it).
|
||||
|
||||
When present we calibrate ``refill_rate`` from it:
|
||||
the server is authoritative.
|
||||
"""
|
||||
if retry_after and retry_after > 0:
|
||||
new_rate = 1.0 / float(retry_after)
|
||||
if new_rate != self.refill_rate:
|
||||
logger.info(
|
||||
"Signal scheduler: calibrating refill_rate to %.4f tokens/sec "
|
||||
"(server retry_after=%.1fs per token)",
|
||||
new_rate, retry_after,
|
||||
)
|
||||
self.refill_rate = new_rate
|
||||
self.tokens = 0.0
|
||||
self.last_refill = time.monotonic()
|
||||
|
||||
def state(self) -> dict:
|
||||
"""Return current scheduler state for diagnostic logging (read-only).
|
||||
|
||||
Does not advance ``last_refill`` — safe to call from logging paths
|
||||
without perturbing the bucket.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
elapsed = now - self.last_refill
|
||||
projected = self.tokens
|
||||
if elapsed > 0 and projected < self.capacity:
|
||||
projected = min(self.capacity, projected + elapsed * self.refill_rate)
|
||||
return {
|
||||
"tokens": round(projected, 1),
|
||||
"capacity": int(self.capacity),
|
||||
"refill_rate": round(self.refill_rate, 4),
|
||||
"refill_seconds_per_token": round(1.0 / self.refill_rate, 1) if self.refill_rate > 0 else float("inf"),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Process-wide singleton
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_scheduler: Optional[SignalAttachmentScheduler] = None
|
||||
|
||||
|
||||
def get_scheduler() -> SignalAttachmentScheduler:
|
||||
"""Return the process-wide scheduler, creating it on first access."""
|
||||
global _scheduler
|
||||
if _scheduler is None:
|
||||
_scheduler = SignalAttachmentScheduler()
|
||||
logger.info(
|
||||
"Signal scheduler: created (capacity=%d tokens, refill=%.4f/s ≈ %.1fs/token)",
|
||||
int(_scheduler.capacity),
|
||||
_scheduler.refill_rate,
|
||||
1.0 / _scheduler.refill_rate,
|
||||
)
|
||||
return _scheduler
|
||||
|
||||
|
||||
def _reset_scheduler() -> None:
|
||||
"""Drop the cached scheduler so the next ``get_scheduler`` call
|
||||
builds a fresh one. Test-only — never call from production paths."""
|
||||
global _scheduler
|
||||
_scheduler = None
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,379 @@
|
||||
"""SMS (Twilio) platform adapter.
|
||||
|
||||
Connects to the Twilio REST API for outbound SMS and runs an aiohttp
|
||||
webhook server to receive inbound messages.
|
||||
|
||||
Shares credentials with the optional telephony skill — same env vars:
|
||||
- TWILIO_ACCOUNT_SID
|
||||
- TWILIO_AUTH_TOKEN
|
||||
- TWILIO_PHONE_NUMBER (E.164 from-number, e.g. +15551234567)
|
||||
|
||||
Gateway-specific env vars:
|
||||
- SMS_WEBHOOK_PORT (default 8080)
|
||||
- SMS_WEBHOOK_HOST (default 127.0.0.1)
|
||||
- SMS_WEBHOOK_URL (public URL for Twilio signature validation — required)
|
||||
- SMS_INSECURE_NO_SIGNATURE (true to disable signature validation — dev only)
|
||||
- SMS_ALLOWED_USERS (comma-separated E.164 phone numbers)
|
||||
- SMS_ALLOW_ALL_USERS (true/false)
|
||||
- SMS_HOME_CHANNEL (phone number for cron delivery)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import logging
|
||||
import os
|
||||
import urllib.parse
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from gateway.config import Platform, PlatformConfig
|
||||
from gateway.platforms.base import (
|
||||
BasePlatformAdapter,
|
||||
MessageEvent,
|
||||
MessageType,
|
||||
SendResult,
|
||||
)
|
||||
from gateway.platforms.helpers import redact_phone, strip_markdown
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TWILIO_API_BASE = "https://api.twilio.com/2010-04-01/Accounts"
|
||||
MAX_SMS_LENGTH = 1600 # ~10 SMS segments
|
||||
DEFAULT_WEBHOOK_PORT = 8080
|
||||
DEFAULT_WEBHOOK_HOST = "127.0.0.1"
|
||||
|
||||
|
||||
def check_sms_requirements() -> bool:
|
||||
"""Check if SMS adapter dependencies are available."""
|
||||
try:
|
||||
import aiohttp # noqa: F401
|
||||
except ImportError:
|
||||
return False
|
||||
return bool(os.getenv("TWILIO_ACCOUNT_SID") and os.getenv("TWILIO_AUTH_TOKEN"))
|
||||
|
||||
|
||||
class SmsAdapter(BasePlatformAdapter):
|
||||
"""
|
||||
Twilio SMS <-> Hermes gateway adapter.
|
||||
|
||||
Each inbound phone number gets its own Hermes session (multi-tenant).
|
||||
Replies are always sent from the configured TWILIO_PHONE_NUMBER.
|
||||
"""
|
||||
|
||||
MAX_MESSAGE_LENGTH = MAX_SMS_LENGTH
|
||||
|
||||
def __init__(self, config: PlatformConfig):
|
||||
super().__init__(config, Platform.SMS)
|
||||
self._account_sid: str = os.environ["TWILIO_ACCOUNT_SID"]
|
||||
self._auth_token: str = os.environ["TWILIO_AUTH_TOKEN"]
|
||||
self._from_number: str = os.getenv("TWILIO_PHONE_NUMBER", "")
|
||||
self._webhook_port: int = int(
|
||||
os.getenv("SMS_WEBHOOK_PORT", str(DEFAULT_WEBHOOK_PORT))
|
||||
)
|
||||
self._webhook_host: str = os.getenv("SMS_WEBHOOK_HOST", DEFAULT_WEBHOOK_HOST)
|
||||
self._webhook_url: str = os.getenv("SMS_WEBHOOK_URL", "").strip()
|
||||
self._runner = None
|
||||
self._http_session: Optional["aiohttp.ClientSession"] = None
|
||||
|
||||
def _basic_auth_header(self) -> str:
|
||||
"""Build HTTP Basic auth header value for Twilio."""
|
||||
creds = f"{self._account_sid}:{self._auth_token}"
|
||||
encoded = base64.b64encode(creds.encode("ascii")).decode("ascii")
|
||||
return f"Basic {encoded}"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Required abstract methods
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def connect(self) -> bool:
|
||||
import aiohttp
|
||||
from aiohttp import web
|
||||
|
||||
if not self._from_number:
|
||||
msg = "[sms] TWILIO_PHONE_NUMBER not set — cannot send replies"
|
||||
logger.error(msg)
|
||||
self._set_fatal_error("sms_missing_phone_number", msg, retryable=False)
|
||||
return False
|
||||
|
||||
insecure_no_sig = os.getenv("SMS_INSECURE_NO_SIGNATURE", "").lower() == "true"
|
||||
|
||||
if not self._webhook_url and not insecure_no_sig:
|
||||
msg = (
|
||||
"[sms] Refusing to start: SMS_WEBHOOK_URL is required for Twilio "
|
||||
"signature validation. Set it to the public URL configured in your "
|
||||
"Twilio console (e.g. https://example.com/webhooks/twilio). "
|
||||
"For local development without validation, set "
|
||||
"SMS_INSECURE_NO_SIGNATURE=true (NOT recommended for production)."
|
||||
)
|
||||
logger.error(msg)
|
||||
self._set_fatal_error("sms_missing_webhook_url", msg, retryable=False)
|
||||
return False
|
||||
|
||||
if insecure_no_sig and not self._webhook_url:
|
||||
logger.warning(
|
||||
"[sms] SMS_INSECURE_NO_SIGNATURE=true — Twilio signature validation "
|
||||
"is DISABLED. Any client that can reach port %d can inject messages. "
|
||||
"Do NOT use this in production.",
|
||||
self._webhook_port,
|
||||
)
|
||||
|
||||
app = web.Application()
|
||||
app.router.add_post("/webhooks/twilio", self._handle_webhook)
|
||||
app.router.add_get("/health", lambda _: web.Response(text="ok"))
|
||||
|
||||
self._runner = web.AppRunner(app)
|
||||
await self._runner.setup()
|
||||
site = web.TCPSite(self._runner, self._webhook_host, self._webhook_port)
|
||||
await site.start()
|
||||
self._http_session = aiohttp.ClientSession(
|
||||
timeout=aiohttp.ClientTimeout(total=30),
|
||||
trust_env=True,
|
||||
)
|
||||
self._running = True
|
||||
|
||||
logger.info(
|
||||
"[sms] Twilio webhook server listening on %s:%d, from: %s",
|
||||
self._webhook_host,
|
||||
self._webhook_port,
|
||||
redact_phone(self._from_number),
|
||||
)
|
||||
return True
|
||||
|
||||
async def disconnect(self) -> None:
|
||||
if self._http_session:
|
||||
await self._http_session.close()
|
||||
self._http_session = None
|
||||
if self._runner:
|
||||
await self._runner.cleanup()
|
||||
self._runner = None
|
||||
self._running = False
|
||||
logger.info("[sms] Disconnected")
|
||||
|
||||
async def send(
|
||||
self,
|
||||
chat_id: str,
|
||||
content: str,
|
||||
reply_to: Optional[str] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> SendResult:
|
||||
import aiohttp
|
||||
|
||||
formatted = self.format_message(content)
|
||||
chunks = self.truncate_message(formatted)
|
||||
last_result = SendResult(success=True)
|
||||
|
||||
url = f"{TWILIO_API_BASE}/{self._account_sid}/Messages.json"
|
||||
headers = {
|
||||
"Authorization": self._basic_auth_header(),
|
||||
}
|
||||
|
||||
session = self._http_session or aiohttp.ClientSession(
|
||||
timeout=aiohttp.ClientTimeout(total=30),
|
||||
trust_env=True,
|
||||
)
|
||||
try:
|
||||
for chunk in chunks:
|
||||
form_data = aiohttp.FormData()
|
||||
form_data.add_field("From", self._from_number)
|
||||
form_data.add_field("To", chat_id)
|
||||
form_data.add_field("Body", chunk)
|
||||
|
||||
try:
|
||||
async with session.post(url, data=form_data, headers=headers) as resp:
|
||||
body = await resp.json()
|
||||
if resp.status >= 400:
|
||||
error_msg = body.get("message", str(body))
|
||||
logger.error(
|
||||
"[sms] send failed to %s: %s %s",
|
||||
redact_phone(chat_id),
|
||||
resp.status,
|
||||
error_msg,
|
||||
)
|
||||
return SendResult(
|
||||
success=False,
|
||||
error=f"Twilio {resp.status}: {error_msg}",
|
||||
)
|
||||
msg_sid = body.get("sid", "")
|
||||
last_result = SendResult(success=True, message_id=msg_sid)
|
||||
except Exception as e:
|
||||
logger.error("[sms] send error to %s: %s", redact_phone(chat_id), e)
|
||||
return SendResult(success=False, error=str(e))
|
||||
finally:
|
||||
# Close session only if we created a fallback (no persistent session)
|
||||
if not self._http_session and session:
|
||||
await session.close()
|
||||
|
||||
return last_result
|
||||
|
||||
async def get_chat_info(self, chat_id: str) -> Dict[str, Any]:
|
||||
return {"name": chat_id, "type": "dm"}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SMS-specific formatting
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def format_message(self, content: str) -> str:
|
||||
"""Strip markdown — SMS renders it as literal characters."""
|
||||
return strip_markdown(content)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Twilio signature validation
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _validate_twilio_signature(
|
||||
self, url: str, post_params: dict, signature: str,
|
||||
) -> bool:
|
||||
"""Validate ``X-Twilio-Signature`` header (HMAC-SHA1, base64).
|
||||
|
||||
Tries both with and without the default port for the URL scheme,
|
||||
since Twilio may sign with either variant.
|
||||
|
||||
Algorithm: https://www.twilio.com/docs/usage/security#validating-requests
|
||||
"""
|
||||
if self._check_signature(url, post_params, signature):
|
||||
return True
|
||||
|
||||
variant = self._port_variant_url(url)
|
||||
if variant and self._check_signature(variant, post_params, signature):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _check_signature(
|
||||
self, url: str, post_params: dict, signature: str,
|
||||
) -> bool:
|
||||
"""Compute and compare a single Twilio signature."""
|
||||
data_to_sign = url
|
||||
for key in sorted(post_params.keys()):
|
||||
data_to_sign += key + post_params[key]
|
||||
mac = hmac.new(
|
||||
self._auth_token.encode("utf-8"),
|
||||
data_to_sign.encode("utf-8"),
|
||||
hashlib.sha1,
|
||||
)
|
||||
computed = base64.b64encode(mac.digest()).decode("utf-8")
|
||||
return hmac.compare_digest(computed, signature)
|
||||
|
||||
@staticmethod
|
||||
def _port_variant_url(url: str) -> str | None:
|
||||
"""Return the URL with the default port toggled, or None.
|
||||
|
||||
Only toggles default ports (443 for https, 80 for http).
|
||||
Non-standard ports are never modified.
|
||||
"""
|
||||
parsed = urllib.parse.urlparse(url)
|
||||
default_ports = {"https": 443, "http": 80}
|
||||
default_port = default_ports.get(parsed.scheme)
|
||||
if default_port is None:
|
||||
return None
|
||||
|
||||
if parsed.port == default_port:
|
||||
# Has explicit default port → strip it
|
||||
return urllib.parse.urlunparse(
|
||||
(parsed.scheme, parsed.hostname, parsed.path,
|
||||
parsed.params, parsed.query, parsed.fragment)
|
||||
)
|
||||
elif parsed.port is None:
|
||||
# No port → add default
|
||||
netloc = f"{parsed.hostname}:{default_port}"
|
||||
return urllib.parse.urlunparse(
|
||||
(parsed.scheme, netloc, parsed.path,
|
||||
parsed.params, parsed.query, parsed.fragment)
|
||||
)
|
||||
|
||||
# Non-standard port — no variant
|
||||
return None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Twilio webhook handler
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _handle_webhook(self, request) -> "aiohttp.web.Response":
|
||||
from aiohttp import web
|
||||
|
||||
try:
|
||||
raw = await request.read()
|
||||
# Twilio sends form-encoded data, not JSON
|
||||
form = urllib.parse.parse_qs(raw.decode("utf-8"), keep_blank_values=True)
|
||||
except Exception as e:
|
||||
logger.error("[sms] webhook parse error: %s", e)
|
||||
return web.Response(
|
||||
text='<?xml version="1.0" encoding="UTF-8"?><Response></Response>',
|
||||
content_type="application/xml",
|
||||
status=400,
|
||||
)
|
||||
|
||||
# Validate Twilio request signature when SMS_WEBHOOK_URL is configured
|
||||
if self._webhook_url:
|
||||
twilio_sig = request.headers.get("X-Twilio-Signature", "")
|
||||
if not twilio_sig:
|
||||
logger.warning("[sms] Rejected: missing X-Twilio-Signature header")
|
||||
return web.Response(
|
||||
text='<?xml version="1.0" encoding="UTF-8"?><Response></Response>',
|
||||
content_type="application/xml",
|
||||
status=403,
|
||||
)
|
||||
flat_params = {k: v[0] for k, v in form.items() if v}
|
||||
if not self._validate_twilio_signature(
|
||||
self._webhook_url, flat_params, twilio_sig
|
||||
):
|
||||
logger.warning("[sms] Rejected: invalid Twilio signature")
|
||||
return web.Response(
|
||||
text='<?xml version="1.0" encoding="UTF-8"?><Response></Response>',
|
||||
content_type="application/xml",
|
||||
status=403,
|
||||
)
|
||||
|
||||
# Extract fields (parse_qs returns lists)
|
||||
from_number = (form.get("From", [""]))[0].strip()
|
||||
to_number = (form.get("To", [""]))[0].strip()
|
||||
text = (form.get("Body", [""]))[0].strip()
|
||||
message_sid = (form.get("MessageSid", [""]))[0].strip()
|
||||
|
||||
if not from_number or not text:
|
||||
return web.Response(
|
||||
text='<?xml version="1.0" encoding="UTF-8"?><Response></Response>',
|
||||
content_type="application/xml",
|
||||
)
|
||||
|
||||
# Ignore messages from our own number (echo prevention)
|
||||
if from_number == self._from_number:
|
||||
logger.debug("[sms] ignoring echo from own number %s", redact_phone(from_number))
|
||||
return web.Response(
|
||||
text='<?xml version="1.0" encoding="UTF-8"?><Response></Response>',
|
||||
content_type="application/xml",
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"[sms] inbound from %s -> %s: %s",
|
||||
redact_phone(from_number),
|
||||
redact_phone(to_number),
|
||||
text[:80],
|
||||
)
|
||||
|
||||
source = self.build_source(
|
||||
chat_id=from_number,
|
||||
chat_name=from_number,
|
||||
chat_type="dm",
|
||||
user_id=from_number,
|
||||
user_name=from_number,
|
||||
)
|
||||
event = MessageEvent(
|
||||
text=text,
|
||||
message_type=MessageType.TEXT,
|
||||
source=source,
|
||||
raw_message=form,
|
||||
message_id=message_sid,
|
||||
)
|
||||
|
||||
# Non-blocking: Twilio expects a fast response
|
||||
task = asyncio.create_task(self.handle_message(event))
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(self._background_tasks.discard)
|
||||
|
||||
# Return empty TwiML — we send replies via the REST API, not inline TwiML
|
||||
return web.Response(
|
||||
text='<?xml version="1.0" encoding="UTF-8"?><Response></Response>',
|
||||
content_type="application/xml",
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,259 @@
|
||||
"""Telegram-specific network helpers.
|
||||
|
||||
Provides a hostname-preserving fallback transport for networks where
|
||||
api.telegram.org resolves to an endpoint that is unreachable from the current
|
||||
host. The transport keeps the logical request host and TLS SNI as
|
||||
api.telegram.org while retrying the TCP connection against one or more fallback
|
||||
IPv4 addresses.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import logging
|
||||
import socket
|
||||
from typing import Iterable, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TELEGRAM_API_HOST = "api.telegram.org"
|
||||
|
||||
# DNS-over-HTTPS providers used to discover Telegram API IPs that may differ
|
||||
# from the (potentially unreachable) IP returned by the local system resolver.
|
||||
_DOH_TIMEOUT = 4.0 # seconds — bounded so connect() isn't noticeably delayed
|
||||
|
||||
_DOH_PROVIDERS: list[dict] = [
|
||||
{
|
||||
"url": "https://dns.google/resolve",
|
||||
"params": {"name": _TELEGRAM_API_HOST, "type": "A"},
|
||||
"headers": {},
|
||||
},
|
||||
{
|
||||
"url": "https://cloudflare-dns.com/dns-query",
|
||||
"params": {"name": _TELEGRAM_API_HOST, "type": "A"},
|
||||
"headers": {"Accept": "application/dns-json"},
|
||||
},
|
||||
]
|
||||
|
||||
# Last-resort IPs when DoH is also blocked. These are stable Telegram Bot API
|
||||
# endpoints in the 149.154.160.0/20 block (same seed used by OpenClaw).
|
||||
_SEED_FALLBACK_IPS: list[str] = ["149.154.167.220"]
|
||||
|
||||
|
||||
def _resolve_proxy_url(target_hosts=None) -> str | None:
|
||||
# Delegate to shared implementation (env vars + macOS system proxy detection)
|
||||
from gateway.platforms.base import resolve_proxy_url
|
||||
return resolve_proxy_url("TELEGRAM_PROXY", target_hosts=target_hosts)
|
||||
|
||||
|
||||
class TelegramFallbackTransport(httpx.AsyncBaseTransport):
|
||||
"""Retry Telegram Bot API requests via fallback IPs while preserving TLS/SNI.
|
||||
|
||||
Requests continue to target https://api.telegram.org/... logically, but on
|
||||
connect failures the underlying TCP connection is retried against a known
|
||||
reachable IP. This is effectively the programmatic equivalent of
|
||||
``curl --resolve api.telegram.org:443:<ip>``.
|
||||
"""
|
||||
|
||||
def __init__(self, fallback_ips: Iterable[str], **transport_kwargs):
|
||||
self._fallback_ips = list(dict.fromkeys(_normalize_fallback_ips(fallback_ips)))
|
||||
proxy_url = _resolve_proxy_url(target_hosts=[_TELEGRAM_API_HOST, *self._fallback_ips])
|
||||
if proxy_url and "proxy" not in transport_kwargs:
|
||||
transport_kwargs["proxy"] = proxy_url
|
||||
self._primary = httpx.AsyncHTTPTransport(**transport_kwargs)
|
||||
self._fallbacks = {
|
||||
ip: httpx.AsyncHTTPTransport(**transport_kwargs) for ip in self._fallback_ips
|
||||
}
|
||||
self._sticky_ip: Optional[str] = None
|
||||
self._sticky_lock = asyncio.Lock()
|
||||
|
||||
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
|
||||
if request.url.host != _TELEGRAM_API_HOST or not self._fallback_ips:
|
||||
return await self._primary.handle_async_request(request)
|
||||
|
||||
sticky_ip = self._sticky_ip
|
||||
attempt_order: list[Optional[str]] = [sticky_ip] if sticky_ip else [None]
|
||||
if sticky_ip:
|
||||
attempt_order.append(None) # retry primary DNS after sticky failure
|
||||
for ip in self._fallback_ips:
|
||||
if ip != sticky_ip:
|
||||
attempt_order.append(ip)
|
||||
|
||||
last_error: Exception | None = None
|
||||
for ip in attempt_order:
|
||||
candidate = request if ip is None else _rewrite_request_for_ip(request, ip)
|
||||
transport = self._primary if ip is None else self._fallbacks[ip]
|
||||
try:
|
||||
response = await transport.handle_async_request(candidate)
|
||||
if ip is not None and self._sticky_ip != ip:
|
||||
async with self._sticky_lock:
|
||||
if self._sticky_ip != ip:
|
||||
self._sticky_ip = ip
|
||||
logger.warning(
|
||||
"[Telegram] Primary api.telegram.org path unreachable; using sticky fallback IP %s",
|
||||
ip,
|
||||
)
|
||||
return response
|
||||
except Exception as exc:
|
||||
last_error = exc
|
||||
if not _is_retryable_connect_error(exc):
|
||||
raise
|
||||
if ip is not None and ip == self._sticky_ip:
|
||||
async with self._sticky_lock:
|
||||
if self._sticky_ip == ip:
|
||||
self._sticky_ip = None
|
||||
logger.warning(
|
||||
"[Telegram] Sticky fallback IP %s failed; resetting to primary DNS path",
|
||||
ip,
|
||||
)
|
||||
if ip is None:
|
||||
logger.warning(
|
||||
"[Telegram] Primary api.telegram.org connection failed (%s); trying fallback IPs %s",
|
||||
exc,
|
||||
", ".join(self._fallback_ips),
|
||||
)
|
||||
continue
|
||||
logger.warning("[Telegram] Fallback IP %s failed: %s", ip, exc)
|
||||
continue
|
||||
|
||||
if last_error is None:
|
||||
raise RuntimeError("All Telegram fallback IPs exhausted but no error was recorded")
|
||||
raise last_error
|
||||
|
||||
async def aclose(self) -> None:
|
||||
await self._primary.aclose()
|
||||
for transport in self._fallbacks.values():
|
||||
await transport.aclose()
|
||||
|
||||
|
||||
def _normalize_fallback_ips(values: Iterable[str]) -> list[str]:
|
||||
normalized: list[str] = []
|
||||
for value in values:
|
||||
raw = str(value).strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
addr = ipaddress.ip_address(raw)
|
||||
except ValueError:
|
||||
logger.warning("Ignoring invalid Telegram fallback IP: %r", raw)
|
||||
continue
|
||||
if addr.version != 4:
|
||||
logger.warning("Ignoring non-IPv4 Telegram fallback IP: %s", raw)
|
||||
continue
|
||||
if addr.is_private or addr.is_loopback or addr.is_link_local or addr.is_unspecified:
|
||||
logger.warning("Ignoring private/internal Telegram fallback IP: %s", raw)
|
||||
continue
|
||||
normalized.append(str(addr))
|
||||
return normalized
|
||||
|
||||
|
||||
def parse_fallback_ip_env(value: str | None) -> list[str]:
|
||||
if not value:
|
||||
return []
|
||||
parts = [part.strip() for part in value.split(",")]
|
||||
return _normalize_fallback_ips(parts)
|
||||
|
||||
|
||||
def _resolve_system_dns() -> set[str]:
|
||||
"""Return the IPv4 addresses that the OS resolver gives for api.telegram.org."""
|
||||
try:
|
||||
results = socket.getaddrinfo(_TELEGRAM_API_HOST, 443, socket.AF_INET)
|
||||
return {addr[4][0] for addr in results}
|
||||
except Exception:
|
||||
return set()
|
||||
|
||||
|
||||
async def _query_doh_provider(
|
||||
client: httpx.AsyncClient, provider: dict
|
||||
) -> list[str]:
|
||||
"""Query one DoH provider and return A-record IPs."""
|
||||
try:
|
||||
resp = await client.get(
|
||||
provider["url"], params=provider["params"], headers=provider["headers"]
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
ips: list[str] = []
|
||||
for answer in data.get("Answer", []):
|
||||
if answer.get("type") != 1: # A record
|
||||
continue
|
||||
raw = answer.get("data", "").strip()
|
||||
try:
|
||||
ipaddress.ip_address(raw)
|
||||
ips.append(raw)
|
||||
except ValueError:
|
||||
continue
|
||||
return ips
|
||||
except Exception as exc:
|
||||
logger.debug("DoH query to %s failed: %s", provider["url"], exc)
|
||||
return []
|
||||
|
||||
|
||||
async def discover_fallback_ips() -> list[str]:
|
||||
"""Auto-discover Telegram API IPs via DNS-over-HTTPS.
|
||||
|
||||
Resolves api.telegram.org through Google and Cloudflare DoH and returns all
|
||||
unique A records. IPs that match the local system resolver are kept rather
|
||||
than excluded: in many networks the system-DNS IP is the most reliable path
|
||||
to api.telegram.org and a transient primary-path failure should be retried
|
||||
against the same address via the IP-rewrite path before the seed list is
|
||||
consulted (#14520). Falls back to a hardcoded seed list only when DoH
|
||||
yields no usable answers.
|
||||
"""
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(_DOH_TIMEOUT)) as client:
|
||||
doh_tasks = [_query_doh_provider(client, p) for p in _DOH_PROVIDERS]
|
||||
system_dns_task = asyncio.to_thread(_resolve_system_dns)
|
||||
results = await asyncio.gather(system_dns_task, *doh_tasks, return_exceptions=True)
|
||||
|
||||
# results[0] = system DNS IPs (set), results[1:] = DoH IP lists
|
||||
system_ips: set[str] = results[0] if isinstance(results[0], set) else set()
|
||||
|
||||
doh_ips: list[str] = []
|
||||
for r in results[1:]:
|
||||
if isinstance(r, list):
|
||||
doh_ips.extend(r)
|
||||
|
||||
# Deduplicate preserving order
|
||||
seen: set[str] = set()
|
||||
candidates: list[str] = []
|
||||
for ip in doh_ips:
|
||||
if ip not in seen:
|
||||
seen.add(ip)
|
||||
candidates.append(ip)
|
||||
|
||||
# Validate through existing normalization
|
||||
validated = _normalize_fallback_ips(candidates)
|
||||
|
||||
if validated:
|
||||
logger.debug("Discovered Telegram fallback IPs via DoH: %s", ", ".join(validated))
|
||||
return validated
|
||||
|
||||
logger.info(
|
||||
"DoH discovery yielded no usable IPs (system DNS: %s); using seed fallback IPs %s",
|
||||
", ".join(system_ips) or "unknown",
|
||||
", ".join(_SEED_FALLBACK_IPS),
|
||||
)
|
||||
return list(_SEED_FALLBACK_IPS)
|
||||
|
||||
|
||||
def _rewrite_request_for_ip(request: httpx.Request, ip: str) -> httpx.Request:
|
||||
original_host = request.url.host or _TELEGRAM_API_HOST
|
||||
url = request.url.copy_with(host=ip)
|
||||
headers = request.headers.copy()
|
||||
headers["host"] = original_host
|
||||
extensions = dict(request.extensions)
|
||||
extensions["sni_hostname"] = original_host
|
||||
return httpx.Request(
|
||||
method=request.method,
|
||||
url=url,
|
||||
headers=headers,
|
||||
stream=request.stream,
|
||||
extensions=extensions,
|
||||
)
|
||||
|
||||
|
||||
def _is_retryable_connect_error(exc: Exception) -> bool:
|
||||
return isinstance(exc, (httpx.ConnectTimeout, httpx.ConnectError))
|
||||
@@ -0,0 +1,971 @@
|
||||
"""Generic webhook platform adapter.
|
||||
|
||||
Runs an aiohttp HTTP server that receives webhook POSTs from external
|
||||
services (GitHub, GitLab, JIRA, Stripe, etc.), validates HMAC signatures,
|
||||
transforms payloads into agent prompts, and routes responses back to the
|
||||
source or to another configured platform.
|
||||
|
||||
Configuration lives in config.yaml under platforms.webhook.extra.routes.
|
||||
Each route defines:
|
||||
- events: which event types to accept (header-based filtering)
|
||||
- secret: HMAC secret for signature validation (REQUIRED)
|
||||
- prompt: template string formatted with the webhook payload
|
||||
- skills: optional list of skills to load for the agent
|
||||
- deliver: where to send the response (github_comment, telegram, etc.)
|
||||
- deliver_extra: additional delivery config (repo, pr_number, chat_id)
|
||||
- deliver_only: if true, skip the agent — the rendered prompt IS the
|
||||
message that gets delivered. Use for external push notifications
|
||||
(Supabase, monitoring alerts, inter-agent pings) where zero LLM cost
|
||||
and sub-second delivery matter more than agent reasoning.
|
||||
|
||||
Security:
|
||||
- HMAC secret is required per route (validated at startup)
|
||||
- Rate limiting per route (fixed-window, configurable)
|
||||
- Idempotency cache prevents duplicate agent runs on webhook retries
|
||||
- Body size limits checked before reading payload
|
||||
- Set secret to "INSECURE_NO_AUTH" to skip validation (testing only)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
from collections import deque
|
||||
from typing import Any, Deque, Dict, List, Optional
|
||||
|
||||
try:
|
||||
from aiohttp import web
|
||||
|
||||
AIOHTTP_AVAILABLE = True
|
||||
except ImportError:
|
||||
AIOHTTP_AVAILABLE = False
|
||||
web = None # type: ignore[assignment]
|
||||
|
||||
from gateway.config import Platform, PlatformConfig
|
||||
from gateway.platforms.base import (
|
||||
BasePlatformAdapter,
|
||||
MessageEvent,
|
||||
MessageType,
|
||||
SendResult,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_BUILTIN_DELIVER_PLATFORMS = {
|
||||
"telegram", "discord", "slack", "signal", "sms", "whatsapp",
|
||||
"matrix", "mattermost", "homeassistant", "email", "dingtalk",
|
||||
"feishu", "wecom", "wecom_callback", "weixin", "bluebubbles",
|
||||
"qqbot", "yuanbao",
|
||||
}
|
||||
|
||||
DEFAULT_HOST = "0.0.0.0"
|
||||
DEFAULT_PORT = 8644
|
||||
_INSECURE_NO_AUTH = "INSECURE_NO_AUTH"
|
||||
_DYNAMIC_ROUTES_FILENAME = "webhook_subscriptions.json"
|
||||
_RATE_WINDOW_SECONDS = 60.0
|
||||
|
||||
# Hostnames/IP literals that only serve connections originating on the same
|
||||
# machine. Anything else is treated as a public bind for safety-rail purposes.
|
||||
_LOOPBACK_HOSTS = frozenset({
|
||||
"127.0.0.1",
|
||||
"localhost",
|
||||
"::1",
|
||||
"ip6-localhost",
|
||||
"ip6-loopback",
|
||||
})
|
||||
|
||||
|
||||
def _is_loopback_host(host: str) -> bool:
|
||||
"""True when `host` binds only to the local machine.
|
||||
|
||||
Covers IPv4 loopback, the standard `localhost` alias, IPv6 loopback in
|
||||
both bracketed and bare form, and the common Debian-style aliases. Any
|
||||
falsy value (empty string, None) is conservatively treated as non-loopback
|
||||
because an unset host usually means the platform-default public bind.
|
||||
"""
|
||||
if not host:
|
||||
return False
|
||||
return host.strip().lower() in _LOOPBACK_HOSTS
|
||||
|
||||
|
||||
def check_webhook_requirements() -> bool:
|
||||
"""Check if webhook adapter dependencies are available."""
|
||||
return AIOHTTP_AVAILABLE
|
||||
|
||||
|
||||
class WebhookAdapter(BasePlatformAdapter):
|
||||
"""Generic webhook receiver that triggers agent runs from HTTP POSTs."""
|
||||
|
||||
def __init__(self, config: PlatformConfig):
|
||||
super().__init__(config, Platform.WEBHOOK)
|
||||
self._host: str = config.extra.get("host", DEFAULT_HOST)
|
||||
self._port: int = int(config.extra.get("port", DEFAULT_PORT))
|
||||
self._global_secret: str = config.extra.get("secret", "")
|
||||
self._static_routes: Dict[str, dict] = config.extra.get("routes", {})
|
||||
self._dynamic_routes: Dict[str, dict] = {}
|
||||
self._dynamic_routes_mtime: float = 0.0
|
||||
self._routes: Dict[str, dict] = dict(self._static_routes)
|
||||
self._runner = None
|
||||
|
||||
# Delivery info keyed by session chat_id.
|
||||
#
|
||||
# Read by every send() invocation for the chat_id (status messages
|
||||
# AND the final response). Cleaned up via TTL on each POST so the
|
||||
# dict stays bounded — see _prune_delivery_info(). Do NOT pop on
|
||||
# send(), or interim status messages (e.g. fallback notifications,
|
||||
# context-pressure warnings) will consume the entry before the
|
||||
# final response arrives, causing the response to silently fall
|
||||
# back to the "log" deliver type.
|
||||
self._delivery_info: Dict[str, dict] = {}
|
||||
self._delivery_info_created: Dict[str, float] = {}
|
||||
self._delivery_info_order: Deque[tuple[float, str]] = deque()
|
||||
|
||||
# Reference to gateway runner for cross-platform delivery (set externally)
|
||||
self.gateway_runner = None
|
||||
|
||||
# Idempotency: TTL cache of recently processed delivery IDs.
|
||||
# Prevents duplicate agent runs when webhook providers retry.
|
||||
self._seen_deliveries: Dict[str, float] = {}
|
||||
self._idempotency_ttl: int = 3600 # 1 hour
|
||||
self._seen_deliveries_next_prune_at: float = 0.0
|
||||
|
||||
# Rate limiting: per-route timestamps in a fixed window.
|
||||
self._rate_counts: Dict[str, Deque[float]] = {}
|
||||
self._rate_limit: int = int(config.extra.get("rate_limit", 30)) # per minute
|
||||
|
||||
# Body size limit (auth-before-body pattern)
|
||||
self._max_body_bytes: int = int(
|
||||
config.extra.get("max_body_bytes", 1_048_576)
|
||||
) # 1MB
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def connect(self) -> bool:
|
||||
# Load agent-created subscriptions before validating
|
||||
self._reload_dynamic_routes()
|
||||
|
||||
# Validate routes at startup — secret is required per route
|
||||
for name, route in self._routes.items():
|
||||
secret = route.get("secret", self._global_secret)
|
||||
if not secret:
|
||||
raise ValueError(
|
||||
f"[webhook] Route '{name}' has no HMAC secret. "
|
||||
f"Set 'secret' on the route or globally. "
|
||||
f"For testing without auth, set secret to '{_INSECURE_NO_AUTH}'."
|
||||
)
|
||||
|
||||
# Safety rail: refuse to start if INSECURE_NO_AUTH is combined with a
|
||||
# non-loopback bind. The escape hatch is for local testing only;
|
||||
# serving an unauthenticated route on a public interface is a
|
||||
# deployment-grade footgun we'd rather crash early than ship.
|
||||
if secret == _INSECURE_NO_AUTH and not _is_loopback_host(self._host):
|
||||
raise ValueError(
|
||||
f"[webhook] Route '{name}' uses INSECURE_NO_AUTH secret "
|
||||
f"but is bound to non-loopback host '{self._host}'. "
|
||||
f"INSECURE_NO_AUTH is for local testing only. "
|
||||
f"Refusing to start to prevent accidental exposure."
|
||||
)
|
||||
# deliver_only routes bypass the agent — the POST body becomes a
|
||||
# direct push notification via the configured delivery target.
|
||||
# Validate up-front so misconfiguration surfaces at startup rather
|
||||
# than on the first webhook POST.
|
||||
if route.get("deliver_only"):
|
||||
deliver = route.get("deliver", "log")
|
||||
if not deliver or deliver == "log":
|
||||
raise ValueError(
|
||||
f"[webhook] Route '{name}' has deliver_only=true but "
|
||||
f"deliver is '{deliver}'. Direct delivery requires a "
|
||||
f"real target (telegram, discord, slack, github_comment, etc.)."
|
||||
)
|
||||
|
||||
app = web.Application()
|
||||
app.router.add_get("/health", self._handle_health)
|
||||
app.router.add_post("/webhooks/{route_name}", self._handle_webhook)
|
||||
|
||||
# Port conflict detection — fail fast if port is already in use
|
||||
import socket as _socket
|
||||
try:
|
||||
with _socket.socket(_socket.AF_INET, _socket.SOCK_STREAM) as _s:
|
||||
_s.settimeout(1)
|
||||
_s.connect(('127.0.0.1', self._port))
|
||||
logger.error('[webhook] Port %d already in use. Set a different port in config.yaml: platforms.webhook.port', self._port)
|
||||
return False
|
||||
except (ConnectionRefusedError, OSError):
|
||||
pass # port is free
|
||||
|
||||
self._runner = web.AppRunner(app)
|
||||
await self._runner.setup()
|
||||
site = web.TCPSite(self._runner, self._host, self._port)
|
||||
await site.start()
|
||||
self._mark_connected()
|
||||
|
||||
route_names = ", ".join(self._routes.keys()) or "(none configured)"
|
||||
logger.info(
|
||||
"[webhook] Listening on %s:%d — routes: %s",
|
||||
self._host,
|
||||
self._port,
|
||||
route_names,
|
||||
)
|
||||
return True
|
||||
|
||||
async def disconnect(self) -> None:
|
||||
if self._runner:
|
||||
await self._runner.cleanup()
|
||||
self._runner = None
|
||||
self._mark_disconnected()
|
||||
logger.info("[webhook] Disconnected")
|
||||
|
||||
async def send(
|
||||
self,
|
||||
chat_id: str,
|
||||
content: str,
|
||||
reply_to: Optional[str] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> SendResult:
|
||||
"""Deliver the agent's response to the configured destination.
|
||||
|
||||
chat_id is ``webhook:{route}:{delivery_id}``. The delivery info
|
||||
stored during webhook receipt is read with ``.get()`` (not popped)
|
||||
so that interim status messages emitted before the final response
|
||||
— fallback-model notifications, context-pressure warnings, etc. —
|
||||
do not consume the entry and silently downgrade the final response
|
||||
to the ``log`` deliver type. TTL cleanup happens on POST.
|
||||
"""
|
||||
delivery = self._delivery_info.get(chat_id, {})
|
||||
deliver_type = delivery.get("deliver", "log")
|
||||
|
||||
if deliver_type == "log":
|
||||
logger.info("[webhook] Response for %s: %s", chat_id, content[:200])
|
||||
return SendResult(success=True)
|
||||
|
||||
if deliver_type == "github_comment":
|
||||
return await self._deliver_github_comment(content, delivery)
|
||||
|
||||
# Cross-platform delivery — any platform with a gateway adapter.
|
||||
# Check both built-in names and plugin-registered platforms.
|
||||
_is_known_platform = deliver_type in _BUILTIN_DELIVER_PLATFORMS
|
||||
if not _is_known_platform:
|
||||
try:
|
||||
from gateway.platform_registry import platform_registry
|
||||
_is_known_platform = platform_registry.is_registered(deliver_type)
|
||||
except Exception:
|
||||
pass
|
||||
if self.gateway_runner and _is_known_platform:
|
||||
return await self._deliver_cross_platform(
|
||||
deliver_type, content, delivery
|
||||
)
|
||||
|
||||
logger.warning("[webhook] Unknown deliver type: %s", deliver_type)
|
||||
return SendResult(
|
||||
success=False, error=f"Unknown deliver type: {deliver_type}"
|
||||
)
|
||||
|
||||
def _prune_delivery_info(self, now: float) -> None:
|
||||
"""Drop delivery_info entries older than the idempotency TTL.
|
||||
|
||||
Mirrors the cleanup pattern used for ``_seen_deliveries``. Called
|
||||
on each POST so the dict size is bounded by ``rate_limit * TTL``
|
||||
even if many webhooks fire and never receive a final response.
|
||||
"""
|
||||
if len(self._delivery_info_order) < len(self._delivery_info_created):
|
||||
self._delivery_info_order = deque(
|
||||
(created_at, key)
|
||||
for key, created_at in sorted(
|
||||
self._delivery_info_created.items(), key=lambda item: item[1]
|
||||
)
|
||||
)
|
||||
cutoff = now - self._idempotency_ttl
|
||||
while self._delivery_info_order and self._delivery_info_order[0][0] < cutoff:
|
||||
created_at, key = self._delivery_info_order.popleft()
|
||||
if self._delivery_info_created.get(key) != created_at:
|
||||
continue
|
||||
self._delivery_info.pop(key, None)
|
||||
self._delivery_info_created.pop(key, None)
|
||||
|
||||
def _prune_seen_deliveries(self, now: float) -> None:
|
||||
"""Occasionally prune expired delivery IDs without scanning every POST."""
|
||||
if now < self._seen_deliveries_next_prune_at:
|
||||
return
|
||||
cutoff = now - self._idempotency_ttl
|
||||
stale = [k for k, t in self._seen_deliveries.items() if t < cutoff]
|
||||
for k in stale:
|
||||
self._seen_deliveries.pop(k, None)
|
||||
self._seen_deliveries_next_prune_at = now + min(60.0, max(1.0, self._idempotency_ttl / 10))
|
||||
|
||||
def _record_rate_limit_hit(self, route_name: str, now: float) -> bool:
|
||||
"""Return True if route is still within limit after recording this hit."""
|
||||
window = self._rate_counts.get(route_name)
|
||||
if not isinstance(window, deque):
|
||||
new_window: Deque[float] = deque(window or ())
|
||||
self._rate_counts[route_name] = new_window
|
||||
window = new_window
|
||||
cutoff = now - _RATE_WINDOW_SECONDS
|
||||
while window and window[0] < cutoff:
|
||||
window.popleft()
|
||||
if len(window) >= self._rate_limit:
|
||||
return False
|
||||
window.append(now)
|
||||
return True
|
||||
|
||||
def _record_delivery_id(self, delivery_id: str, now: float) -> bool:
|
||||
"""Return True when this delivery should be processed."""
|
||||
seen_at = self._seen_deliveries.get(delivery_id)
|
||||
if seen_at is not None and now - seen_at < self._idempotency_ttl:
|
||||
return False
|
||||
if seen_at is not None:
|
||||
self._seen_deliveries.pop(delivery_id, None)
|
||||
self._seen_deliveries[delivery_id] = now
|
||||
if len(self._seen_deliveries) > max(self._rate_limit * 2, 128):
|
||||
self._prune_seen_deliveries(now)
|
||||
return True
|
||||
|
||||
async def get_chat_info(self, chat_id: str) -> Dict[str, Any]:
|
||||
return {"name": chat_id, "type": "webhook"}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# HTTP handlers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _handle_health(self, request: "web.Request") -> "web.Response":
|
||||
"""GET /health — simple health check."""
|
||||
return web.json_response({"status": "ok", "platform": "webhook"})
|
||||
|
||||
def _reload_dynamic_routes(self) -> None:
|
||||
"""Reload agent-created subscriptions from disk if the file changed."""
|
||||
from hermes_constants import get_hermes_home
|
||||
hermes_home = get_hermes_home()
|
||||
subs_path = hermes_home / _DYNAMIC_ROUTES_FILENAME
|
||||
if not subs_path.exists():
|
||||
if self._dynamic_routes:
|
||||
self._dynamic_routes = {}
|
||||
self._routes = dict(self._static_routes)
|
||||
logger.debug("[webhook] Dynamic subscriptions file removed, cleared dynamic routes")
|
||||
return
|
||||
try:
|
||||
mtime = subs_path.stat().st_mtime
|
||||
if mtime <= self._dynamic_routes_mtime:
|
||||
return # No change
|
||||
data = json.loads(subs_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
return
|
||||
# Merge: static routes take precedence over dynamic ones.
|
||||
# Reject any dynamic route whose effective secret is empty —
|
||||
# an empty secret would cause _handle_webhook to skip HMAC
|
||||
# validation entirely, letting unauthenticated callers in.
|
||||
new_dynamic: Dict[str, dict] = {}
|
||||
for k, v in data.items():
|
||||
if k in self._static_routes:
|
||||
continue
|
||||
effective_secret = v.get("secret", self._global_secret)
|
||||
if not effective_secret:
|
||||
logger.warning(
|
||||
"[webhook] Dynamic route '%s' skipped: 'secret' is "
|
||||
"missing or empty. Set a valid HMAC secret, or use "
|
||||
"'%s' to explicitly disable auth (testing only).",
|
||||
k,
|
||||
_INSECURE_NO_AUTH,
|
||||
)
|
||||
continue
|
||||
if (
|
||||
effective_secret == _INSECURE_NO_AUTH
|
||||
and not _is_loopback_host(self._host)
|
||||
):
|
||||
logger.warning(
|
||||
"[webhook] Dynamic route '%s' skipped: INSECURE_NO_AUTH "
|
||||
"is only allowed on loopback hosts. Current host: '%s'.",
|
||||
k,
|
||||
self._host,
|
||||
)
|
||||
continue
|
||||
new_dynamic[k] = v
|
||||
self._dynamic_routes = new_dynamic
|
||||
self._routes = {**self._dynamic_routes, **self._static_routes}
|
||||
self._dynamic_routes_mtime = mtime
|
||||
logger.info(
|
||||
"[webhook] Reloaded %d dynamic route(s): %s",
|
||||
len(self._dynamic_routes),
|
||||
", ".join(self._dynamic_routes.keys()) or "(none)",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("[webhook] Failed to reload dynamic routes: %s", e)
|
||||
|
||||
async def _handle_webhook(self, request: "web.Request") -> "web.Response":
|
||||
"""POST /webhooks/{route_name} — receive and process a webhook event."""
|
||||
# Hot-reload dynamic subscriptions on each request (mtime-gated, cheap)
|
||||
self._reload_dynamic_routes()
|
||||
|
||||
route_name = request.match_info.get("route_name", "")
|
||||
route_config = self._routes.get(route_name)
|
||||
|
||||
if not route_config:
|
||||
return web.json_response(
|
||||
{"error": f"Unknown route: {route_name}"}, status=404
|
||||
)
|
||||
|
||||
# Disabled routes are kept in the subscriptions file (so the dashboard
|
||||
# can re-enable them) but reject incoming events. Default-enabled:
|
||||
# only an explicit ``enabled: false`` turns a route off, matching the
|
||||
# mcp_servers ``enabled`` semantics.
|
||||
if route_config.get("enabled", True) is False:
|
||||
return web.json_response(
|
||||
{"error": f"Route disabled: {route_name}"}, status=403
|
||||
)
|
||||
|
||||
# ── Auth-before-body ─────────────────────────────────────
|
||||
# Check Content-Length before reading the full payload.
|
||||
content_length = request.content_length or 0
|
||||
if content_length > self._max_body_bytes:
|
||||
return web.json_response(
|
||||
{"error": "Payload too large"}, status=413
|
||||
)
|
||||
|
||||
# Read body (must be done before any validation)
|
||||
try:
|
||||
raw_body = await request.read()
|
||||
except Exception as e:
|
||||
logger.error("[webhook] Failed to read body: %s", e)
|
||||
return web.json_response({"error": "Bad request"}, status=400)
|
||||
|
||||
# Validate HMAC signature FIRST (skip only for the explicit local-test
|
||||
# INSECURE_NO_AUTH mode). Missing/empty secrets must fail closed here,
|
||||
# not only during connect(), so direct handler reuse cannot turn a
|
||||
# network webhook route into an unauthenticated agent-dispatch surface.
|
||||
secret = route_config.get("secret", self._global_secret)
|
||||
if not secret:
|
||||
logger.error(
|
||||
"[webhook] Route %s has no HMAC secret; refusing request",
|
||||
route_name,
|
||||
)
|
||||
return web.json_response(
|
||||
{"error": "Webhook route is missing an HMAC secret"},
|
||||
status=403,
|
||||
)
|
||||
if secret != _INSECURE_NO_AUTH:
|
||||
if not self._validate_signature(request, raw_body, secret):
|
||||
logger.warning(
|
||||
"[webhook] Invalid signature for route %s", route_name
|
||||
)
|
||||
return web.json_response(
|
||||
{"error": "Invalid signature"}, status=401
|
||||
)
|
||||
|
||||
# ── Rate limiting (after auth) ───────────────────────────
|
||||
now = time.time()
|
||||
if not self._record_rate_limit_hit(route_name, now):
|
||||
return web.json_response(
|
||||
{"error": "Rate limit exceeded"}, status=429
|
||||
)
|
||||
|
||||
# Parse payload
|
||||
try:
|
||||
payload = json.loads(raw_body)
|
||||
except json.JSONDecodeError:
|
||||
# Try form-encoded as fallback
|
||||
try:
|
||||
import urllib.parse
|
||||
|
||||
payload = dict(
|
||||
urllib.parse.parse_qsl(raw_body.decode("utf-8"))
|
||||
)
|
||||
except Exception:
|
||||
return web.json_response(
|
||||
{"error": "Cannot parse body"}, status=400
|
||||
)
|
||||
|
||||
# Check event type filter
|
||||
event_type = (
|
||||
request.headers.get("X-GitHub-Event", "")
|
||||
or request.headers.get("X-GitLab-Event", "")
|
||||
or payload.get("event_type", "")
|
||||
or payload.get("type", "")
|
||||
or "unknown"
|
||||
)
|
||||
allowed_events = route_config.get("events", [])
|
||||
if allowed_events and event_type not in allowed_events:
|
||||
logger.debug(
|
||||
"[webhook] Ignoring event %s for route %s (allowed: %s)",
|
||||
event_type,
|
||||
route_name,
|
||||
allowed_events,
|
||||
)
|
||||
return web.json_response(
|
||||
{"status": "ignored", "event": event_type}
|
||||
)
|
||||
|
||||
# Format prompt from template
|
||||
prompt_template = route_config.get("prompt", "")
|
||||
prompt = self._render_prompt(
|
||||
prompt_template, payload, event_type, route_name
|
||||
)
|
||||
|
||||
# Inject skill content if configured.
|
||||
# We call build_skill_invocation_message() directly rather than
|
||||
# using /skill-name slash commands — the gateway's command parser
|
||||
# would intercept those and break the flow.
|
||||
skills = route_config.get("skills", [])
|
||||
if skills:
|
||||
try:
|
||||
from agent.skill_commands import (
|
||||
build_skill_invocation_message,
|
||||
get_skill_commands,
|
||||
)
|
||||
|
||||
skill_cmds = get_skill_commands()
|
||||
for skill_name in skills:
|
||||
cmd_key = f"/{skill_name}"
|
||||
if cmd_key in skill_cmds:
|
||||
skill_content = build_skill_invocation_message(
|
||||
cmd_key, user_instruction=prompt
|
||||
)
|
||||
if skill_content:
|
||||
prompt = skill_content
|
||||
break # Load the first matching skill
|
||||
else:
|
||||
logger.warning(
|
||||
"[webhook] Skill '%s' not found", skill_name
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("[webhook] Skill loading failed: %s", e)
|
||||
|
||||
# Build a unique delivery ID
|
||||
delivery_id = request.headers.get(
|
||||
"X-GitHub-Delivery",
|
||||
request.headers.get(
|
||||
"svix-id",
|
||||
request.headers.get("X-Request-ID", str(int(time.time() * 1000))),
|
||||
),
|
||||
)
|
||||
|
||||
# ── Idempotency ─────────────────────────────────────────
|
||||
# Skip duplicate deliveries (webhook retries).
|
||||
now = time.time()
|
||||
if not self._record_delivery_id(delivery_id, now):
|
||||
logger.info(
|
||||
"[webhook] Skipping duplicate delivery %s", delivery_id
|
||||
)
|
||||
return web.json_response(
|
||||
{"status": "duplicate", "delivery_id": delivery_id},
|
||||
status=200,
|
||||
)
|
||||
|
||||
# ── Direct delivery mode (deliver_only) ─────────────────
|
||||
# Skip the agent entirely — the rendered prompt IS the message we
|
||||
# deliver. Use case: external services (Supabase, monitoring,
|
||||
# cron jobs, other agents) that need to push a plain notification
|
||||
# to a user's chat with zero LLM cost. Reuses the same HMAC auth,
|
||||
# rate limiting, idempotency, and template rendering as agent mode.
|
||||
if route_config.get("deliver_only"):
|
||||
delivery = {
|
||||
"deliver": route_config.get("deliver", "log"),
|
||||
"deliver_extra": self._render_delivery_extra(
|
||||
route_config.get("deliver_extra", {}), payload
|
||||
),
|
||||
"payload": payload,
|
||||
}
|
||||
logger.info(
|
||||
"[webhook] direct-deliver event=%s route=%s target=%s msg_len=%d delivery=%s",
|
||||
event_type,
|
||||
route_name,
|
||||
delivery["deliver"],
|
||||
len(prompt),
|
||||
delivery_id,
|
||||
)
|
||||
try:
|
||||
result = await self._direct_deliver(prompt, delivery)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"[webhook] direct-deliver failed route=%s delivery=%s",
|
||||
route_name,
|
||||
delivery_id,
|
||||
)
|
||||
return web.json_response(
|
||||
{"status": "error", "error": "Delivery failed", "delivery_id": delivery_id},
|
||||
status=502,
|
||||
)
|
||||
|
||||
if result.success:
|
||||
return web.json_response(
|
||||
{
|
||||
"status": "delivered",
|
||||
"route": route_name,
|
||||
"target": delivery["deliver"],
|
||||
"delivery_id": delivery_id,
|
||||
},
|
||||
status=200,
|
||||
)
|
||||
# Delivery attempted but target rejected it — surface as 502
|
||||
# with a generic error (don't leak adapter-level detail).
|
||||
logger.warning(
|
||||
"[webhook] direct-deliver target rejected route=%s target=%s error=%s",
|
||||
route_name,
|
||||
delivery["deliver"],
|
||||
result.error,
|
||||
)
|
||||
return web.json_response(
|
||||
{"status": "error", "error": "Delivery failed", "delivery_id": delivery_id},
|
||||
status=502,
|
||||
)
|
||||
|
||||
# Use delivery_id in session key so concurrent webhooks on the
|
||||
# same route get independent agent runs (not queued/interrupted).
|
||||
session_chat_id = f"webhook:{route_name}:{delivery_id}"
|
||||
|
||||
# Store delivery info for send(). Read by every send() invocation
|
||||
# for this chat_id (interim status messages and the final response),
|
||||
# so we do NOT pop on send. TTL-based cleanup keeps the dict bounded.
|
||||
deliver_config = {
|
||||
"deliver": route_config.get("deliver", "log"),
|
||||
"deliver_extra": self._render_delivery_extra(
|
||||
route_config.get("deliver_extra", {}), payload
|
||||
),
|
||||
"payload": payload,
|
||||
}
|
||||
self._delivery_info[session_chat_id] = deliver_config
|
||||
self._delivery_info_created[session_chat_id] = now
|
||||
self._delivery_info_order.append((now, session_chat_id))
|
||||
self._prune_delivery_info(now)
|
||||
|
||||
# Build source and event
|
||||
source = self.build_source(
|
||||
chat_id=session_chat_id,
|
||||
chat_name=f"webhook/{route_name}",
|
||||
chat_type="webhook",
|
||||
user_id=f"webhook:{route_name}",
|
||||
user_name=route_name,
|
||||
)
|
||||
event = MessageEvent(
|
||||
text=prompt,
|
||||
message_type=MessageType.TEXT,
|
||||
source=source,
|
||||
raw_message=payload,
|
||||
message_id=delivery_id,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"[webhook] %s event=%s route=%s prompt_len=%d delivery=%s",
|
||||
request.method,
|
||||
event_type,
|
||||
route_name,
|
||||
len(prompt),
|
||||
delivery_id,
|
||||
)
|
||||
|
||||
# Non-blocking — return 202 Accepted immediately
|
||||
task = asyncio.create_task(self.handle_message(event))
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(self._background_tasks.discard)
|
||||
|
||||
return web.json_response(
|
||||
{
|
||||
"status": "accepted",
|
||||
"route": route_name,
|
||||
"event": event_type,
|
||||
"delivery_id": delivery_id,
|
||||
},
|
||||
status=202,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Signature validation
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _validate_signature(
|
||||
self, request: "web.Request", body: bytes, secret: str
|
||||
) -> bool:
|
||||
"""Validate webhook signature (GitHub, GitLab, Svix, generic HMAC-SHA256)."""
|
||||
def _header(name: str) -> str:
|
||||
return (
|
||||
request.headers.get(name, "")
|
||||
or request.headers.get(name.lower(), "")
|
||||
or request.headers.get(name.upper(), "")
|
||||
)
|
||||
|
||||
# Svix / AgentMail:
|
||||
# svix-id: msg_...
|
||||
# svix-timestamp: unix seconds
|
||||
# svix-signature: v1,<base64-hmac> [v1,<base64-hmac> ...]
|
||||
# Signed content is: "{id}.{timestamp}.{raw_body}". Svix secrets
|
||||
# usually start with "whsec_" and the remainder is base64-encoded.
|
||||
svix_id = _header("svix-id")
|
||||
svix_timestamp = _header("svix-timestamp")
|
||||
svix_signature = _header("svix-signature")
|
||||
if svix_id or svix_timestamp or svix_signature:
|
||||
return self._validate_svix_signature(
|
||||
body=body,
|
||||
secret=secret,
|
||||
msg_id=svix_id,
|
||||
timestamp=svix_timestamp,
|
||||
signature_header=svix_signature,
|
||||
)
|
||||
|
||||
# GitHub: X-Hub-Signature-256 = sha256=<hex>
|
||||
gh_sig = request.headers.get("X-Hub-Signature-256", "")
|
||||
if gh_sig:
|
||||
expected = "sha256=" + hmac.new(
|
||||
secret.encode(), body, hashlib.sha256
|
||||
).hexdigest()
|
||||
return hmac.compare_digest(gh_sig, expected)
|
||||
|
||||
# GitLab: X-Gitlab-Token = <plain secret>
|
||||
gl_token = request.headers.get("X-Gitlab-Token", "")
|
||||
if gl_token:
|
||||
return hmac.compare_digest(gl_token, secret)
|
||||
|
||||
# Generic: X-Webhook-Signature = <hex HMAC-SHA256>
|
||||
generic_sig = request.headers.get("X-Webhook-Signature", "")
|
||||
if generic_sig:
|
||||
expected = hmac.new(
|
||||
secret.encode(), body, hashlib.sha256
|
||||
).hexdigest()
|
||||
return hmac.compare_digest(generic_sig, expected)
|
||||
|
||||
# No recognised signature header but secret is configured → reject
|
||||
logger.debug(
|
||||
"[webhook] Secret configured but no signature header found"
|
||||
)
|
||||
return False
|
||||
|
||||
def _validate_svix_signature(
|
||||
self,
|
||||
body: bytes,
|
||||
secret: str,
|
||||
msg_id: str,
|
||||
timestamp: str,
|
||||
signature_header: str,
|
||||
tolerance_seconds: int = 300,
|
||||
) -> bool:
|
||||
"""Validate Svix-compatible signatures used by AgentMail webhooks."""
|
||||
if not (msg_id and timestamp and signature_header and secret):
|
||||
return False
|
||||
|
||||
try:
|
||||
ts = int(timestamp)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
if abs(int(time.time()) - ts) > tolerance_seconds:
|
||||
logger.warning("[webhook] Svix signature timestamp outside replay window")
|
||||
return False
|
||||
|
||||
if secret.startswith("whsec_"):
|
||||
encoded_secret = secret.removeprefix("whsec_")
|
||||
try:
|
||||
key = base64.b64decode(encoded_secret, validate=True)
|
||||
except (binascii.Error, ValueError):
|
||||
logger.debug("[webhook] Invalid whsec_ Svix signing secret")
|
||||
return False
|
||||
else:
|
||||
# Be permissive for providers that document Svix-style headers but
|
||||
# hand out raw shared secrets rather than whsec_ base64 secrets.
|
||||
logger.debug("[webhook] Validating Svix-style signature with raw secret")
|
||||
key = secret.encode()
|
||||
|
||||
signed_content = msg_id.encode() + b"." + timestamp.encode() + b"." + body
|
||||
expected = base64.b64encode(
|
||||
hmac.new(key, signed_content, hashlib.sha256).digest()
|
||||
).decode()
|
||||
|
||||
# Svix can send multiple signatures separated by spaces during secret
|
||||
# rotation. Each entry is formatted as "vN,<base64>".
|
||||
for part in signature_header.split():
|
||||
try:
|
||||
version, signature = part.split(",", 1)
|
||||
except ValueError:
|
||||
continue
|
||||
if version == "v1" and hmac.compare_digest(signature, expected):
|
||||
return True
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Prompt rendering
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _render_prompt(
|
||||
self,
|
||||
template: str,
|
||||
payload: dict,
|
||||
event_type: str,
|
||||
route_name: str,
|
||||
) -> str:
|
||||
"""Render a prompt template with the webhook payload.
|
||||
|
||||
Supports dot-notation access into nested dicts:
|
||||
``{pull_request.title}`` → ``payload["pull_request"]["title"]``
|
||||
|
||||
Special token ``{__raw__}`` dumps the entire payload as indented
|
||||
JSON (truncated to 4000 chars). Useful for monitoring alerts or
|
||||
any webhook where the agent needs to see the full payload.
|
||||
"""
|
||||
if not template:
|
||||
truncated = json.dumps(payload, indent=2)[:4000]
|
||||
return (
|
||||
f"Webhook event '{event_type}' on route "
|
||||
f"'{route_name}':\n\n```json\n{truncated}\n```"
|
||||
)
|
||||
|
||||
def _resolve(match: re.Match) -> str:
|
||||
key = match.group(1)
|
||||
# Special token: dump the entire payload as JSON
|
||||
if key == "__raw__":
|
||||
return json.dumps(payload, indent=2)[:4000]
|
||||
value: Any = payload
|
||||
for part in key.split("."):
|
||||
if isinstance(value, dict):
|
||||
value = value.get(part, f"{{{key}}}")
|
||||
else:
|
||||
return f"{{{key}}}"
|
||||
if isinstance(value, (dict, list)):
|
||||
return json.dumps(value, indent=2)[:2000]
|
||||
return str(value)
|
||||
|
||||
return re.sub(r"\{([a-zA-Z0-9_.]+)\}", _resolve, template)
|
||||
|
||||
def _render_delivery_extra(
|
||||
self, extra: dict, payload: dict
|
||||
) -> dict:
|
||||
"""Render delivery_extra template values with payload data."""
|
||||
rendered: Dict[str, Any] = {}
|
||||
for key, value in extra.items():
|
||||
if isinstance(value, str):
|
||||
rendered[key] = self._render_prompt(value, payload, "", "")
|
||||
else:
|
||||
rendered[key] = value
|
||||
return rendered
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Response delivery
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _direct_deliver(
|
||||
self, content: str, delivery: dict
|
||||
) -> SendResult:
|
||||
"""Deliver *content* directly without invoking the agent.
|
||||
|
||||
Used by ``deliver_only`` routes: the rendered template becomes the
|
||||
literal message body, and we dispatch to the same delivery helpers
|
||||
that the agent-mode ``send()`` flow uses. All target types that
|
||||
work in agent mode work here — Telegram, Discord, Slack, GitHub
|
||||
PR comments, etc.
|
||||
"""
|
||||
deliver_type = delivery.get("deliver", "log")
|
||||
|
||||
if deliver_type == "log":
|
||||
# Shouldn't reach here — startup validation rejects deliver_only
|
||||
# with deliver=log — but guard defensively.
|
||||
logger.info("[webhook] direct-deliver log-only: %s", content[:200])
|
||||
return SendResult(success=True)
|
||||
|
||||
if deliver_type == "github_comment":
|
||||
return await self._deliver_github_comment(content, delivery)
|
||||
|
||||
# Fall through to the cross-platform dispatcher, which validates the
|
||||
# target name and routes via the gateway runner.
|
||||
return await self._deliver_cross_platform(
|
||||
deliver_type, content, delivery
|
||||
)
|
||||
|
||||
async def _deliver_github_comment(
|
||||
self, content: str, delivery: dict
|
||||
) -> SendResult:
|
||||
"""Post agent response as a GitHub PR/issue comment via ``gh`` CLI."""
|
||||
extra = delivery.get("deliver_extra", {})
|
||||
repo = extra.get("repo", "")
|
||||
pr_number = extra.get("pr_number", "")
|
||||
|
||||
if not repo or not pr_number:
|
||||
logger.error(
|
||||
"[webhook] github_comment delivery missing repo or pr_number"
|
||||
)
|
||||
return SendResult(
|
||||
success=False, error="Missing repo or pr_number"
|
||||
)
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[
|
||||
"gh",
|
||||
"pr",
|
||||
"comment",
|
||||
str(pr_number),
|
||||
"--repo",
|
||||
repo,
|
||||
"--body",
|
||||
content,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
logger.info(
|
||||
"[webhook] Posted comment on %s#%s", repo, pr_number
|
||||
)
|
||||
return SendResult(success=True)
|
||||
else:
|
||||
logger.error(
|
||||
"[webhook] gh pr comment failed: %s", result.stderr
|
||||
)
|
||||
return SendResult(success=False, error=result.stderr)
|
||||
except FileNotFoundError:
|
||||
logger.error(
|
||||
"[webhook] 'gh' CLI not found — install GitHub CLI for "
|
||||
"github_comment delivery"
|
||||
)
|
||||
return SendResult(
|
||||
success=False, error="gh CLI not installed"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("[webhook] github_comment delivery error: %s", e)
|
||||
return SendResult(success=False, error=str(e))
|
||||
|
||||
async def _deliver_cross_platform(
|
||||
self, platform_name: str, content: str, delivery: dict
|
||||
) -> SendResult:
|
||||
"""Route response to another platform (telegram, discord, etc.)."""
|
||||
if not self.gateway_runner:
|
||||
return SendResult(
|
||||
success=False,
|
||||
error="No gateway runner for cross-platform delivery",
|
||||
)
|
||||
|
||||
try:
|
||||
target_platform = Platform(platform_name)
|
||||
except ValueError:
|
||||
return SendResult(
|
||||
success=False, error=f"Unknown platform: {platform_name}"
|
||||
)
|
||||
|
||||
adapter = self.gateway_runner.adapters.get(target_platform)
|
||||
if not adapter:
|
||||
return SendResult(
|
||||
success=False,
|
||||
error=f"Platform {platform_name} not connected",
|
||||
)
|
||||
|
||||
# Use home channel if no specific chat_id in deliver_extra
|
||||
extra = delivery.get("deliver_extra", {})
|
||||
chat_id = extra.get("chat_id", "")
|
||||
if not chat_id:
|
||||
home = self.gateway_runner.config.get_home_channel(target_platform)
|
||||
if home:
|
||||
chat_id = home.chat_id
|
||||
else:
|
||||
return SendResult(
|
||||
success=False,
|
||||
error=f"No chat_id or home channel for {platform_name}",
|
||||
)
|
||||
|
||||
# Pass thread_id from deliver_extra so Telegram forum topics work
|
||||
metadata = None
|
||||
thread_id = extra.get("message_thread_id") or extra.get("thread_id")
|
||||
if thread_id:
|
||||
metadata = {"thread_id": thread_id}
|
||||
|
||||
return await adapter.send(chat_id, content, metadata=metadata)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,425 @@
|
||||
"""WeCom callback-mode adapter for self-built enterprise applications.
|
||||
|
||||
Unlike the bot/websocket adapter in ``wecom.py``, this handles the standard
|
||||
WeCom callback flow: WeCom POSTs encrypted XML to an HTTP endpoint, the
|
||||
adapter decrypts it, queues the message for the agent, and immediately
|
||||
acknowledges. The agent's reply is delivered later via the proactive
|
||||
``message/send`` API using an access-token.
|
||||
|
||||
Supports multiple self-built apps under one gateway instance, scoped by
|
||||
``corp_id:user_id`` to avoid cross-corp collisions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import socket as _socket
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional
|
||||
# Security: parse untrusted, pre-auth request bodies (WeCom callbacks) with
|
||||
# defusedxml to block billion-laughs / entity-expansion (and XXE) DoS. The
|
||||
# parsing API (fromstring) is a drop-in for the stdlib calls used below;
|
||||
# response-building XML lives in wecom_crypto.py and is not parsed here.
|
||||
try:
|
||||
import defusedxml.ElementTree as ET
|
||||
|
||||
DEFUSEDXML_AVAILABLE = True
|
||||
except ImportError:
|
||||
ET = None # type: ignore[assignment]
|
||||
DEFUSEDXML_AVAILABLE = False
|
||||
|
||||
try:
|
||||
from aiohttp import web
|
||||
|
||||
AIOHTTP_AVAILABLE = True
|
||||
except ImportError:
|
||||
web = None # type: ignore[assignment]
|
||||
AIOHTTP_AVAILABLE = False
|
||||
|
||||
try:
|
||||
import httpx
|
||||
|
||||
HTTPX_AVAILABLE = True
|
||||
except ImportError:
|
||||
httpx = None # type: ignore[assignment]
|
||||
HTTPX_AVAILABLE = False
|
||||
|
||||
from gateway.config import Platform, PlatformConfig
|
||||
from gateway.platforms.base import BasePlatformAdapter, MessageEvent, MessageType, SendResult
|
||||
from gateway.platforms.wecom_crypto import WXBizMsgCrypt, WeComCryptoError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_HOST = "0.0.0.0"
|
||||
DEFAULT_PORT = 8645
|
||||
DEFAULT_PATH = "/wecom/callback"
|
||||
ACCESS_TOKEN_TTL_SECONDS = 7200
|
||||
MESSAGE_DEDUP_TTL_SECONDS = 300
|
||||
|
||||
|
||||
def check_wecom_callback_requirements() -> bool:
|
||||
return AIOHTTP_AVAILABLE and HTTPX_AVAILABLE and DEFUSEDXML_AVAILABLE
|
||||
|
||||
|
||||
class WecomCallbackAdapter(BasePlatformAdapter):
|
||||
def __init__(self, config: PlatformConfig):
|
||||
super().__init__(config, Platform.WECOM_CALLBACK)
|
||||
extra = config.extra or {}
|
||||
self._host = str(extra.get("host") or DEFAULT_HOST)
|
||||
self._port = int(extra.get("port") or DEFAULT_PORT)
|
||||
self._path = str(extra.get("path") or DEFAULT_PATH)
|
||||
self._apps: List[Dict[str, Any]] = self._normalize_apps(extra)
|
||||
self._runner: Optional[web.AppRunner] = None
|
||||
self._site: Optional[web.TCPSite] = None
|
||||
self._app: Optional[web.Application] = None
|
||||
self._http_client: Optional[httpx.AsyncClient] = None
|
||||
self._message_queue: asyncio.Queue[MessageEvent] = asyncio.Queue()
|
||||
self._poll_task: Optional[asyncio.Task] = None
|
||||
self._seen_messages: Dict[str, float] = {}
|
||||
self._user_app_map: Dict[str, str] = {}
|
||||
self._access_tokens: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# App normalisation
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _user_app_key(corp_id: str, user_id: str) -> str:
|
||||
return f"{corp_id}:{user_id}" if corp_id else user_id
|
||||
|
||||
@staticmethod
|
||||
def _normalize_apps(extra: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
apps = extra.get("apps")
|
||||
if isinstance(apps, list) and apps:
|
||||
return [dict(app) for app in apps if isinstance(app, dict)]
|
||||
if extra.get("corp_id"):
|
||||
return [
|
||||
{
|
||||
"name": extra.get("name") or "default",
|
||||
"corp_id": extra.get("corp_id", ""),
|
||||
"corp_secret": extra.get("corp_secret", ""),
|
||||
"agent_id": str(extra.get("agent_id", "")),
|
||||
"token": extra.get("token", ""),
|
||||
"encoding_aes_key": extra.get("encoding_aes_key", ""),
|
||||
}
|
||||
]
|
||||
return []
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def connect(self) -> bool:
|
||||
if not self._apps:
|
||||
logger.warning("[WecomCallback] No callback apps configured")
|
||||
return False
|
||||
if not check_wecom_callback_requirements():
|
||||
logger.warning("[WecomCallback] aiohttp/httpx not installed")
|
||||
return False
|
||||
|
||||
# Quick port-in-use check.
|
||||
try:
|
||||
with _socket.socket(_socket.AF_INET, _socket.SOCK_STREAM) as sock:
|
||||
sock.settimeout(1)
|
||||
sock.connect(("127.0.0.1", self._port))
|
||||
logger.error("[WecomCallback] Port %d already in use", self._port)
|
||||
return False
|
||||
except (ConnectionRefusedError, OSError):
|
||||
pass
|
||||
|
||||
try:
|
||||
# Tighter keepalive so idle CLOSE_WAIT drains promptly (#18451).
|
||||
from gateway.platforms._http_client_limits import platform_httpx_limits
|
||||
self._http_client = httpx.AsyncClient(timeout=20.0, limits=platform_httpx_limits())
|
||||
self._app = web.Application()
|
||||
self._app.router.add_get("/health", self._handle_health)
|
||||
self._app.router.add_get(self._path, self._handle_verify)
|
||||
self._app.router.add_post(self._path, self._handle_callback)
|
||||
self._runner = web.AppRunner(self._app)
|
||||
await self._runner.setup()
|
||||
self._site = web.TCPSite(self._runner, self._host, self._port)
|
||||
await self._site.start()
|
||||
self._poll_task = asyncio.create_task(self._poll_loop())
|
||||
self._mark_connected()
|
||||
logger.info(
|
||||
"[WecomCallback] HTTP server listening on %s:%s%s",
|
||||
self._host, self._port, self._path,
|
||||
)
|
||||
for app in self._apps:
|
||||
try:
|
||||
await self._refresh_access_token(app)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"[WecomCallback] Initial token refresh failed for app '%s': %s",
|
||||
app.get("name", "default"), exc,
|
||||
)
|
||||
return True
|
||||
except Exception:
|
||||
await self._cleanup()
|
||||
logger.exception("[WecomCallback] Failed to start")
|
||||
return False
|
||||
|
||||
async def disconnect(self) -> None:
|
||||
self._running = False
|
||||
if self._poll_task:
|
||||
self._poll_task.cancel()
|
||||
try:
|
||||
await self._poll_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._poll_task = None
|
||||
await self._cleanup()
|
||||
self._mark_disconnected()
|
||||
logger.info("[WecomCallback] Disconnected")
|
||||
|
||||
async def _cleanup(self) -> None:
|
||||
self._site = None
|
||||
if self._runner:
|
||||
await self._runner.cleanup()
|
||||
self._runner = None
|
||||
self._app = None
|
||||
if self._http_client:
|
||||
await self._http_client.aclose()
|
||||
self._http_client = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Outbound: proactive send via access-token API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def send(
|
||||
self,
|
||||
chat_id: str,
|
||||
content: str,
|
||||
reply_to: Optional[str] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> SendResult:
|
||||
app = self._resolve_app_for_chat(chat_id)
|
||||
touser = chat_id.split(":", 1)[1] if ":" in chat_id else chat_id
|
||||
try:
|
||||
payload = {
|
||||
"touser": touser,
|
||||
"msgtype": "text",
|
||||
"agentid": int(str(app.get("agent_id") or 0)),
|
||||
"text": {"content": content[:2048]},
|
||||
"safe": 0,
|
||||
}
|
||||
for _attempt in range(2):
|
||||
token = await self._get_access_token(app)
|
||||
resp = await self._http_client.post(
|
||||
f"https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token={token}",
|
||||
json=payload,
|
||||
)
|
||||
data = resp.json()
|
||||
errcode = data.get("errcode")
|
||||
if errcode in {40001, 42001} and _attempt == 0:
|
||||
# WeCom rejected the token — evict the cached entry so
|
||||
# the next _get_access_token call forces a fresh fetch.
|
||||
logger.warning(
|
||||
"[WecomCallback] Token rejected for app '%s' (errcode=%s), refreshing",
|
||||
app.get("name", "default"), errcode,
|
||||
)
|
||||
self._access_tokens.pop(app["name"], None)
|
||||
continue
|
||||
if errcode != 0:
|
||||
return SendResult(success=False, error=str(data))
|
||||
return SendResult(
|
||||
success=True,
|
||||
message_id=str(data.get("msgid", "")),
|
||||
raw_response=data,
|
||||
)
|
||||
return SendResult(success=False, error="send failed after token refresh")
|
||||
except Exception as exc:
|
||||
return SendResult(success=False, error=str(exc))
|
||||
|
||||
def _resolve_app_for_chat(self, chat_id: str) -> Dict[str, Any]:
|
||||
"""Pick the app associated with *chat_id*, falling back sensibly."""
|
||||
app_name = self._user_app_map.get(chat_id)
|
||||
if not app_name and ":" not in chat_id:
|
||||
# Legacy bare user_id — try to find a unique match.
|
||||
matching = [k for k in self._user_app_map if k.endswith(f":{chat_id}")]
|
||||
if len(matching) == 1:
|
||||
app_name = self._user_app_map.get(matching[0])
|
||||
app = self._get_app_by_name(app_name) if app_name else None
|
||||
return app or self._apps[0]
|
||||
|
||||
async def get_chat_info(self, chat_id: str) -> Dict[str, Any]:
|
||||
return {"name": chat_id, "type": "dm"}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Inbound: HTTP callback handlers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _handle_health(self, request: web.Request) -> web.Response:
|
||||
return web.json_response({"status": "ok", "platform": "wecom_callback"})
|
||||
|
||||
async def _handle_verify(self, request: web.Request) -> web.Response:
|
||||
"""GET endpoint — WeCom URL verification handshake."""
|
||||
msg_signature = request.query.get("msg_signature", "")
|
||||
timestamp = request.query.get("timestamp", "")
|
||||
nonce = request.query.get("nonce", "")
|
||||
echostr = request.query.get("echostr", "")
|
||||
for app in self._apps:
|
||||
try:
|
||||
crypt = self._crypt_for_app(app)
|
||||
plain = crypt.verify_url(msg_signature, timestamp, nonce, echostr)
|
||||
return web.Response(text=plain, content_type="text/plain")
|
||||
except Exception:
|
||||
continue
|
||||
return web.Response(status=403, text="signature verification failed")
|
||||
|
||||
async def _handle_callback(self, request: web.Request) -> web.Response:
|
||||
"""POST endpoint — receive an encrypted message callback."""
|
||||
msg_signature = request.query.get("msg_signature", "")
|
||||
timestamp = request.query.get("timestamp", "")
|
||||
nonce = request.query.get("nonce", "")
|
||||
body = await request.text()
|
||||
|
||||
for app in self._apps:
|
||||
try:
|
||||
decrypted = self._decrypt_request(
|
||||
app, body, msg_signature, timestamp, nonce,
|
||||
)
|
||||
event = self._build_event(app, decrypted)
|
||||
if event is not None:
|
||||
# Deduplicate: WeCom retries callbacks on timeout,
|
||||
# producing duplicate inbound messages (#10305).
|
||||
if event.message_id:
|
||||
now = time.time()
|
||||
if event.message_id in self._seen_messages:
|
||||
if now - self._seen_messages[event.message_id] < MESSAGE_DEDUP_TTL_SECONDS:
|
||||
logger.debug("[WecomCallback] Duplicate MsgId %s, skipping", event.message_id)
|
||||
return web.Response(text="success", content_type="text/plain")
|
||||
del self._seen_messages[event.message_id]
|
||||
self._seen_messages[event.message_id] = now
|
||||
# Prune expired entries when cache grows large
|
||||
if len(self._seen_messages) > 2000:
|
||||
cutoff = now - MESSAGE_DEDUP_TTL_SECONDS
|
||||
self._seen_messages = {k: v for k, v in self._seen_messages.items() if v > cutoff}
|
||||
# Record which app this user belongs to.
|
||||
if event.source and event.source.user_id:
|
||||
map_key = self._user_app_key(
|
||||
str(app.get("corp_id") or ""), event.source.user_id,
|
||||
)
|
||||
self._user_app_map[map_key] = app["name"]
|
||||
await self._message_queue.put(event)
|
||||
# Immediately acknowledge — the agent's reply will arrive
|
||||
# later via the proactive message/send API.
|
||||
return web.Response(text="success", content_type="text/plain")
|
||||
except WeComCryptoError:
|
||||
continue
|
||||
except Exception:
|
||||
logger.exception("[WecomCallback] Error handling message")
|
||||
break
|
||||
return web.Response(status=400, text="invalid callback payload")
|
||||
|
||||
async def _poll_loop(self) -> None:
|
||||
"""Drain the message queue and dispatch to the gateway runner."""
|
||||
while True:
|
||||
event = await self._message_queue.get()
|
||||
try:
|
||||
task = asyncio.create_task(self.handle_message(event))
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(self._background_tasks.discard)
|
||||
except Exception:
|
||||
logger.exception("[WecomCallback] Failed to enqueue event")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# XML / crypto helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _decrypt_request(
|
||||
self, app: Dict[str, Any], body: str,
|
||||
msg_signature: str, timestamp: str, nonce: str,
|
||||
) -> str:
|
||||
root = ET.fromstring(body)
|
||||
encrypt = root.findtext("Encrypt", default="")
|
||||
crypt = self._crypt_for_app(app)
|
||||
return crypt.decrypt(msg_signature, timestamp, nonce, encrypt).decode("utf-8")
|
||||
|
||||
def _build_event(self, app: Dict[str, Any], xml_text: str) -> Optional[MessageEvent]:
|
||||
root = ET.fromstring(xml_text)
|
||||
msg_type = (root.findtext("MsgType") or "").lower()
|
||||
# Silently acknowledge lifecycle events.
|
||||
if msg_type == "event":
|
||||
event_name = (root.findtext("Event") or "").lower()
|
||||
if event_name in {"enter_agent", "subscribe"}:
|
||||
return None
|
||||
if msg_type not in {"text", "event"}:
|
||||
return None
|
||||
|
||||
user_id = root.findtext("FromUserName", default="")
|
||||
corp_id = root.findtext("ToUserName", default=app.get("corp_id", ""))
|
||||
scoped_chat_id = self._user_app_key(corp_id, user_id)
|
||||
content = root.findtext("Content", default="").strip()
|
||||
if not content and msg_type == "event":
|
||||
content = "/start"
|
||||
msg_id = (
|
||||
root.findtext("MsgId")
|
||||
or f"{user_id}:{root.findtext('CreateTime', default='0')}"
|
||||
)
|
||||
source = self.build_source(
|
||||
chat_id=scoped_chat_id,
|
||||
chat_name=user_id,
|
||||
chat_type="dm",
|
||||
user_id=user_id,
|
||||
user_name=user_id,
|
||||
)
|
||||
return MessageEvent(
|
||||
text=content,
|
||||
message_type=MessageType.TEXT,
|
||||
source=source,
|
||||
raw_message=xml_text,
|
||||
message_id=msg_id,
|
||||
)
|
||||
|
||||
def _crypt_for_app(self, app: Dict[str, Any]) -> WXBizMsgCrypt:
|
||||
return WXBizMsgCrypt(
|
||||
token=str(app.get("token") or ""),
|
||||
encoding_aes_key=str(app.get("encoding_aes_key") or ""),
|
||||
receive_id=str(app.get("corp_id") or ""),
|
||||
)
|
||||
|
||||
def _get_app_by_name(self, name: Optional[str]) -> Optional[Dict[str, Any]]:
|
||||
if not name:
|
||||
return None
|
||||
for app in self._apps:
|
||||
if app.get("name") == name:
|
||||
return app
|
||||
return None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Access-token management
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _get_access_token(self, app: Dict[str, Any]) -> str:
|
||||
cached = self._access_tokens.get(app["name"])
|
||||
now = time.time()
|
||||
if cached and cached.get("expires_at", 0) > now + 60:
|
||||
return cached["token"]
|
||||
return await self._refresh_access_token(app)
|
||||
|
||||
async def _refresh_access_token(self, app: Dict[str, Any]) -> str:
|
||||
resp = await self._http_client.get(
|
||||
"https://qyapi.weixin.qq.com/cgi-bin/gettoken",
|
||||
params={
|
||||
"corpid": app.get("corp_id"),
|
||||
"corpsecret": app.get("corp_secret"),
|
||||
},
|
||||
)
|
||||
data = resp.json()
|
||||
if data.get("errcode") != 0:
|
||||
raise RuntimeError(f"WeCom token refresh failed: {data}")
|
||||
token = data["access_token"]
|
||||
expires_in = int(data.get("expires_in", ACCESS_TOKEN_TTL_SECONDS))
|
||||
self._access_tokens[app["name"]] = {
|
||||
"token": token,
|
||||
"expires_at": time.time() + expires_in,
|
||||
}
|
||||
logger.info(
|
||||
"[WecomCallback] Token refreshed for app '%s' (corp=%s), expires in %ss",
|
||||
app.get("name", "default"),
|
||||
app.get("corp_id", ""),
|
||||
expires_in,
|
||||
)
|
||||
return token
|
||||
@@ -0,0 +1,142 @@
|
||||
"""WeCom BizMsgCrypt-compatible AES-CBC encryption for callback mode.
|
||||
|
||||
Implements the same wire format as Tencent's official ``WXBizMsgCrypt``
|
||||
SDK so that WeCom can verify, encrypt, and decrypt callback payloads.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import os
|
||||
import secrets
|
||||
import socket
|
||||
import struct
|
||||
from typing import Optional
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
from cryptography.hazmat.backends import default_backend
|
||||
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
|
||||
|
||||
|
||||
class WeComCryptoError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class SignatureError(WeComCryptoError):
|
||||
pass
|
||||
|
||||
|
||||
class DecryptError(WeComCryptoError):
|
||||
pass
|
||||
|
||||
|
||||
class EncryptError(WeComCryptoError):
|
||||
pass
|
||||
|
||||
|
||||
class PKCS7Encoder:
|
||||
block_size = 32
|
||||
|
||||
@classmethod
|
||||
def encode(cls, text: bytes) -> bytes:
|
||||
amount_to_pad = cls.block_size - (len(text) % cls.block_size)
|
||||
if amount_to_pad == 0:
|
||||
amount_to_pad = cls.block_size
|
||||
pad = bytes([amount_to_pad]) * amount_to_pad
|
||||
return text + pad
|
||||
|
||||
@classmethod
|
||||
def decode(cls, decrypted: bytes) -> bytes:
|
||||
if not decrypted:
|
||||
raise DecryptError("empty decrypted payload")
|
||||
pad = decrypted[-1]
|
||||
if pad < 1 or pad > cls.block_size:
|
||||
raise DecryptError("invalid PKCS7 padding")
|
||||
if decrypted[-pad:] != bytes([pad]) * pad:
|
||||
raise DecryptError("malformed PKCS7 padding")
|
||||
return decrypted[:-pad]
|
||||
|
||||
|
||||
def _sha1_signature(token: str, timestamp: str, nonce: str, encrypt: str) -> str:
|
||||
parts = sorted([token, timestamp, nonce, encrypt])
|
||||
return hashlib.sha1("".join(parts).encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
class WXBizMsgCrypt:
|
||||
"""Minimal WeCom callback crypto helper compatible with BizMsgCrypt semantics."""
|
||||
|
||||
def __init__(self, token: str, encoding_aes_key: str, receive_id: str):
|
||||
if not token:
|
||||
raise ValueError("token is required")
|
||||
if not encoding_aes_key:
|
||||
raise ValueError("encoding_aes_key is required")
|
||||
if len(encoding_aes_key) != 43:
|
||||
raise ValueError("encoding_aes_key must be 43 chars")
|
||||
if not receive_id:
|
||||
raise ValueError("receive_id is required")
|
||||
|
||||
self.token = token
|
||||
self.receive_id = receive_id
|
||||
self.key = base64.b64decode(encoding_aes_key + "=")
|
||||
self.iv = self.key[:16]
|
||||
|
||||
def verify_url(self, msg_signature: str, timestamp: str, nonce: str, echostr: str) -> str:
|
||||
plain = self.decrypt(msg_signature, timestamp, nonce, echostr)
|
||||
return plain.decode("utf-8")
|
||||
|
||||
def decrypt(self, msg_signature: str, timestamp: str, nonce: str, encrypt: str) -> bytes:
|
||||
expected = _sha1_signature(self.token, timestamp, nonce, encrypt)
|
||||
if expected != msg_signature:
|
||||
raise SignatureError("signature mismatch")
|
||||
try:
|
||||
cipher_text = base64.b64decode(encrypt)
|
||||
except Exception as exc:
|
||||
raise DecryptError(f"invalid base64 payload: {exc}") from exc
|
||||
try:
|
||||
cipher = Cipher(algorithms.AES(self.key), modes.CBC(self.iv), backend=default_backend())
|
||||
decryptor = cipher.decryptor()
|
||||
padded = decryptor.update(cipher_text) + decryptor.finalize()
|
||||
plain = PKCS7Encoder.decode(padded)
|
||||
content = plain[16:] # skip 16-byte random prefix
|
||||
xml_length = socket.ntohl(struct.unpack("I", content[:4])[0])
|
||||
xml_content = content[4:4 + xml_length]
|
||||
receive_id = content[4 + xml_length:].decode("utf-8")
|
||||
except WeComCryptoError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise DecryptError(f"decrypt failed: {exc}") from exc
|
||||
|
||||
if receive_id != self.receive_id:
|
||||
raise DecryptError("receive_id mismatch")
|
||||
return xml_content
|
||||
|
||||
def encrypt(self, plaintext: str, nonce: Optional[str] = None, timestamp: Optional[str] = None) -> str:
|
||||
nonce = nonce or self._random_nonce()
|
||||
timestamp = timestamp or str(int(__import__("time").time()))
|
||||
encrypt = self._encrypt_bytes(plaintext.encode("utf-8"))
|
||||
signature = _sha1_signature(self.token, timestamp, nonce, encrypt)
|
||||
root = ET.Element("xml")
|
||||
ET.SubElement(root, "Encrypt").text = encrypt
|
||||
ET.SubElement(root, "MsgSignature").text = signature
|
||||
ET.SubElement(root, "TimeStamp").text = timestamp
|
||||
ET.SubElement(root, "Nonce").text = nonce
|
||||
return ET.tostring(root, encoding="unicode")
|
||||
|
||||
def _encrypt_bytes(self, raw: bytes) -> str:
|
||||
try:
|
||||
random_prefix = os.urandom(16)
|
||||
msg_len = struct.pack("I", socket.htonl(len(raw)))
|
||||
payload = random_prefix + msg_len + raw + self.receive_id.encode("utf-8")
|
||||
padded = PKCS7Encoder.encode(payload)
|
||||
cipher = Cipher(algorithms.AES(self.key), modes.CBC(self.iv), backend=default_backend())
|
||||
encryptor = cipher.encryptor()
|
||||
encrypted = encryptor.update(padded) + encryptor.finalize()
|
||||
return base64.b64encode(encrypted).decode("utf-8")
|
||||
except Exception as exc:
|
||||
raise EncryptError(f"encrypt failed: {exc}") from exc
|
||||
|
||||
@staticmethod
|
||||
def _random_nonce(length: int = 10) -> str:
|
||||
alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
||||
return "".join(secrets.choice(alphabet) for _ in range(length))
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,367 @@
|
||||
"""
|
||||
Transport-agnostic WhatsApp behavior shared by the Baileys bridge adapter
|
||||
and the official WhatsApp Cloud API adapter.
|
||||
|
||||
The mixin provides:
|
||||
- Allow-list / DM / group gating
|
||||
- Mention detection (explicit @-mentions + configurable regex patterns)
|
||||
- Quoted-reply-to-bot detection
|
||||
- Broadcast / Channel / Newsletter filtering
|
||||
- WhatsApp-flavored markdown conversion
|
||||
- Outgoing chunk length budgeting
|
||||
|
||||
It is the *behavior layer*. Transport-specific concerns (subprocess management,
|
||||
HTTP webhooks, Graph API calls, media upload protocols) live in each adapter.
|
||||
|
||||
Mixin contract — the adapter must set these on ``self`` before any of the
|
||||
mixin's methods are called (typically in ``__init__``):
|
||||
|
||||
self.config # gateway.config.PlatformConfig
|
||||
self.name # str — adapter name (used in log lines)
|
||||
self._dm_policy # str: "open" | "allowlist" | "disabled"
|
||||
self._allow_from # set[str]
|
||||
self._group_policy # str: "open" | "allowlist" | "disabled"
|
||||
self._group_allow_from # set[str]
|
||||
self._mention_patterns # list[re.Pattern]
|
||||
self._reply_prefix # Optional[str]
|
||||
|
||||
Class attributes ``MAX_MESSAGE_LENGTH`` and ``DEFAULT_REPLY_PREFIX`` are
|
||||
defined on the mixin and may be overridden per-adapter if needed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WhatsAppBehaviorMixin:
|
||||
"""Shared behavior for all WhatsApp adapters (Baileys + Cloud API).
|
||||
|
||||
See module docstring for the attribute contract the host adapter must
|
||||
satisfy. This mixin owns no state of its own — every value it touches
|
||||
is either a class attribute or set by the adapter's ``__init__``.
|
||||
"""
|
||||
|
||||
# WhatsApp message limits — practical UX limit, not protocol max.
|
||||
# WhatsApp allows ~65K but long messages are unreadable on mobile.
|
||||
MAX_MESSAGE_LENGTH: int = 4096
|
||||
supports_code_blocks = True # WhatsApp renders fenced code blocks (monospace)
|
||||
|
||||
DEFAULT_REPLY_PREFIX: str = "⚕ *Hermes Agent*\n────────────\n"
|
||||
|
||||
@property
|
||||
def enforces_own_access_policy(self) -> bool:
|
||||
"""WhatsApp gates DM/group access at intake via dm_policy/group_policy."""
|
||||
return True
|
||||
|
||||
# ------------------------------------------------------------------ config
|
||||
def _effective_reply_prefix(self) -> str:
|
||||
"""Return the prefix to add to outgoing replies in self-chat mode.
|
||||
|
||||
Subclasses that don't have a self-chat concept (the Cloud API
|
||||
adapter) can override this to always return ``""`` or apply a
|
||||
different policy.
|
||||
"""
|
||||
whatsapp_mode = os.getenv("WHATSAPP_MODE", "self-chat")
|
||||
if whatsapp_mode != "self-chat":
|
||||
return ""
|
||||
if self._reply_prefix is not None:
|
||||
return self._reply_prefix.replace("\\n", "\n")
|
||||
env_prefix = os.getenv("WHATSAPP_REPLY_PREFIX")
|
||||
if env_prefix is not None:
|
||||
return env_prefix.replace("\\n", "\n")
|
||||
return self.DEFAULT_REPLY_PREFIX
|
||||
|
||||
def _outgoing_chunk_limit(self) -> int:
|
||||
"""Reserve room for the reply prefix so the final message fits."""
|
||||
prefix_len = len(self._effective_reply_prefix())
|
||||
# Keep enough space for truncate_message's pagination indicator and
|
||||
# code-fence repair even if a user configures a very long prefix.
|
||||
return max(1024, self.MAX_MESSAGE_LENGTH - prefix_len)
|
||||
|
||||
def _whatsapp_require_mention(self) -> bool:
|
||||
configured = self.config.extra.get("require_mention")
|
||||
if configured is not None:
|
||||
if isinstance(configured, str):
|
||||
return configured.lower() in {"true", "1", "yes", "on"}
|
||||
return bool(configured)
|
||||
return os.getenv("WHATSAPP_REQUIRE_MENTION", "false").lower() in {
|
||||
"true",
|
||||
"1",
|
||||
"yes",
|
||||
"on",
|
||||
}
|
||||
|
||||
def _whatsapp_free_response_chats(self) -> set[str]:
|
||||
raw = self.config.extra.get("free_response_chats")
|
||||
if raw is None:
|
||||
raw = os.getenv("WHATSAPP_FREE_RESPONSE_CHATS", "")
|
||||
if isinstance(raw, list):
|
||||
return {str(part).strip() for part in raw if str(part).strip()}
|
||||
return {part.strip() for part in str(raw).split(",") if part.strip()}
|
||||
|
||||
@staticmethod
|
||||
def _coerce_allow_list(raw) -> set[str]:
|
||||
"""Parse allow_from / group_allow_from from config or env var."""
|
||||
if raw is None:
|
||||
return set()
|
||||
if isinstance(raw, list):
|
||||
return {str(part).strip() for part in raw if str(part).strip()}
|
||||
return {part.strip() for part in str(raw).split(",") if part.strip()}
|
||||
|
||||
# ------------------------------------------------------------------ JID helpers
|
||||
@staticmethod
|
||||
def _normalize_whatsapp_id(value: Optional[str]) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
normalized = str(value).strip()
|
||||
if ":" in normalized and "@" in normalized:
|
||||
normalized = normalized.replace(":", "@", 1)
|
||||
return normalized
|
||||
|
||||
@staticmethod
|
||||
def _is_broadcast_chat(chat_id: str) -> bool:
|
||||
"""True for WhatsApp pseudo-chats that aren't real conversations.
|
||||
|
||||
Covers Status updates (Stories) and Channel/Newsletter broadcasts.
|
||||
These show up as inbound messages on Baileys but the agent should
|
||||
never reply — answering a Story update spams the contact's status
|
||||
feed, and Channel posts aren't addressable in the first place.
|
||||
"""
|
||||
if not chat_id:
|
||||
return False
|
||||
cid = chat_id.strip().lower()
|
||||
if cid == "status@broadcast":
|
||||
return True
|
||||
# @broadcast suffix covers status@broadcast plus any future
|
||||
# broadcast-list variants. @newsletter is the Channel JID suffix.
|
||||
if cid.endswith("@broadcast") or cid.endswith("@newsletter"):
|
||||
return True
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------ gating
|
||||
def _is_dm_allowed(self, sender_id: str) -> bool:
|
||||
"""Check whether a DM from the given sender should be processed."""
|
||||
if self._dm_policy == "disabled":
|
||||
return False
|
||||
if self._dm_policy == "allowlist":
|
||||
return sender_id in self._allow_from
|
||||
# "open" — all DMs allowed
|
||||
return True
|
||||
|
||||
def _is_group_allowed(self, chat_id: str) -> bool:
|
||||
"""Check whether a group chat should be processed."""
|
||||
if self._group_policy == "disabled":
|
||||
return False
|
||||
if self._group_policy == "allowlist":
|
||||
return chat_id in self._group_allow_from
|
||||
# "open" — all groups allowed
|
||||
return True
|
||||
|
||||
def _compile_mention_patterns(self):
|
||||
patterns = self.config.extra.get("mention_patterns")
|
||||
if patterns is None:
|
||||
raw = os.getenv("WHATSAPP_MENTION_PATTERNS", "").strip()
|
||||
if raw:
|
||||
try:
|
||||
patterns = json.loads(raw)
|
||||
except Exception:
|
||||
patterns = [
|
||||
part.strip() for part in raw.splitlines() if part.strip()
|
||||
]
|
||||
if not patterns:
|
||||
patterns = [
|
||||
part.strip() for part in raw.split(",") if part.strip()
|
||||
]
|
||||
if patterns is None:
|
||||
return []
|
||||
if isinstance(patterns, str):
|
||||
patterns = [patterns]
|
||||
if not isinstance(patterns, list):
|
||||
logger.warning(
|
||||
"[%s] whatsapp mention_patterns must be a list or string; got %s",
|
||||
self.name,
|
||||
type(patterns).__name__,
|
||||
)
|
||||
return []
|
||||
|
||||
compiled = []
|
||||
for pattern in patterns:
|
||||
if not isinstance(pattern, str) or not pattern.strip():
|
||||
continue
|
||||
try:
|
||||
compiled.append(re.compile(pattern, re.IGNORECASE))
|
||||
except re.error as exc:
|
||||
logger.warning(
|
||||
"[%s] Invalid WhatsApp mention pattern %r: %s",
|
||||
self.name,
|
||||
pattern,
|
||||
exc,
|
||||
)
|
||||
if compiled:
|
||||
logger.info(
|
||||
"[%s] Loaded %d WhatsApp mention pattern(s)", self.name, len(compiled)
|
||||
)
|
||||
return compiled
|
||||
|
||||
def _bot_ids_from_message(self, data: Dict[str, Any]) -> set[str]:
|
||||
bot_ids = set()
|
||||
for candidate in data.get("botIds") or []:
|
||||
normalized = self._normalize_whatsapp_id(candidate)
|
||||
if normalized:
|
||||
bot_ids.add(normalized)
|
||||
return bot_ids
|
||||
|
||||
def _message_is_reply_to_bot(self, data: Dict[str, Any]) -> bool:
|
||||
quoted_participant = self._normalize_whatsapp_id(data.get("quotedParticipant"))
|
||||
if not quoted_participant:
|
||||
return False
|
||||
return quoted_participant in self._bot_ids_from_message(data)
|
||||
|
||||
def _message_mentions_bot(self, data: Dict[str, Any]) -> bool:
|
||||
bot_ids = self._bot_ids_from_message(data)
|
||||
if not bot_ids:
|
||||
return False
|
||||
mentioned_ids = {
|
||||
nid
|
||||
for candidate in (data.get("mentionedIds") or [])
|
||||
if (nid := self._normalize_whatsapp_id(candidate))
|
||||
}
|
||||
if mentioned_ids & bot_ids:
|
||||
return True
|
||||
|
||||
body = str(data.get("body") or "")
|
||||
lower_body = body.lower()
|
||||
for bot_id in bot_ids:
|
||||
bare_id = bot_id.split("@", 1)[0].lower()
|
||||
if bare_id and (f"@{bare_id}" in lower_body or bare_id in lower_body):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _message_matches_mention_patterns(self, data: Dict[str, Any]) -> bool:
|
||||
if not self._mention_patterns:
|
||||
return False
|
||||
body = str(data.get("body") or "")
|
||||
return any(pattern.search(body) for pattern in self._mention_patterns)
|
||||
|
||||
def _clean_bot_mention_text(self, text: str, data: Dict[str, Any]) -> str:
|
||||
if not text:
|
||||
return text
|
||||
bot_ids = self._bot_ids_from_message(data)
|
||||
cleaned = text
|
||||
for bot_id in bot_ids:
|
||||
bare_id = bot_id.split("@", 1)[0]
|
||||
if bare_id:
|
||||
cleaned = re.sub(
|
||||
rf"@{re.escape(bare_id)}\b[,:\-]*\s*", "", cleaned
|
||||
)
|
||||
return cleaned.strip() or text
|
||||
|
||||
def _should_process_message(self, data: Dict[str, Any]) -> bool:
|
||||
chat_id_raw = str(data.get("chatId") or "")
|
||||
# WhatsApp uses pseudo-chats for Status updates (Stories) and
|
||||
# Channel/Newsletter broadcasts. These are not real conversations
|
||||
# and the agent should never reply to them — even in self-chat mode
|
||||
# where the bridge may surface them as "fromMe" events.
|
||||
if self._is_broadcast_chat(chat_id_raw):
|
||||
return False
|
||||
is_group = data.get("isGroup", False)
|
||||
if is_group:
|
||||
chat_id = chat_id_raw
|
||||
if not self._is_group_allowed(chat_id):
|
||||
return False
|
||||
else:
|
||||
sender_id = str(data.get("senderId") or data.get("from") or "")
|
||||
if not self._is_dm_allowed(sender_id):
|
||||
return False
|
||||
# DMs that pass the policy gate are always processed
|
||||
return True
|
||||
# Group messages: check mention / free-response settings
|
||||
chat_id = str(data.get("chatId") or "")
|
||||
if chat_id in self._whatsapp_free_response_chats():
|
||||
return True
|
||||
if not self._whatsapp_require_mention():
|
||||
return True
|
||||
body = str(data.get("body") or "").strip()
|
||||
if body.startswith("/"):
|
||||
return True
|
||||
if self._message_is_reply_to_bot(data):
|
||||
return True
|
||||
if self._message_mentions_bot(data):
|
||||
return True
|
||||
return self._message_matches_mention_patterns(data)
|
||||
|
||||
# ------------------------------------------------------------------ formatting
|
||||
def format_message(self, content: str) -> str:
|
||||
"""Convert standard markdown to WhatsApp-compatible formatting.
|
||||
|
||||
WhatsApp supports: *bold*, _italic_, ~strikethrough~, ```code```,
|
||||
and monospaced `inline`. Standard markdown uses different syntax
|
||||
for bold/italic/strikethrough, so we convert here.
|
||||
|
||||
Code blocks (``` fenced) and inline code (`) are protected from
|
||||
conversion via placeholder substitution.
|
||||
"""
|
||||
if not content:
|
||||
return content
|
||||
|
||||
# --- 1. Protect fenced code blocks from formatting changes ---
|
||||
_FENCE_PH = "\x00FENCE"
|
||||
fences: list[str] = []
|
||||
|
||||
def _save_fence(m: re.Match) -> str:
|
||||
fences.append(m.group(0))
|
||||
return f"{_FENCE_PH}{len(fences) - 1}\x00"
|
||||
|
||||
result = re.sub(r"```[\s\S]*?```", _save_fence, content)
|
||||
|
||||
# --- 2. Protect inline code ---
|
||||
_CODE_PH = "\x00CODE"
|
||||
codes: list[str] = []
|
||||
|
||||
def _save_code(m: re.Match) -> str:
|
||||
codes.append(m.group(0))
|
||||
return f"{_CODE_PH}{len(codes) - 1}\x00"
|
||||
|
||||
result = re.sub(r"`[^`\n]+`", _save_code, result)
|
||||
|
||||
# --- 3. Convert markdown formatting to WhatsApp syntax ---
|
||||
# Bold: **text** or __text__ → *text*
|
||||
result = re.sub(r"\*\*(.+?)\*\*", r"*\1*", result)
|
||||
result = re.sub(r"__(.+?)__", r"*\1*", result)
|
||||
# Strikethrough: ~~text~~ → ~text~
|
||||
result = re.sub(r"~~(.+?)~~", r"~\1~", result)
|
||||
# Italic: *text* is already WhatsApp italic — leave as-is
|
||||
# _text_ is already WhatsApp italic — leave as-is
|
||||
|
||||
# --- 4. Convert markdown headers to bold text ---
|
||||
# # Header → *Header*. Strip any *...* wrapping already produced
|
||||
# by step 3 (e.g. "# **Title**" → "*Title*", not "**Title**",
|
||||
# which WhatsApp renders with literal asterisks).
|
||||
def _header_to_bold(m: re.Match) -> str:
|
||||
inner = m.group(1).strip()
|
||||
while len(inner) > 1 and inner.startswith("*") and inner.endswith("*"):
|
||||
inner = inner[1:-1].strip()
|
||||
return f"*{inner}*"
|
||||
|
||||
result = re.sub(
|
||||
r"^#{1,6}\s+(.+)$", _header_to_bold, result, flags=re.MULTILINE
|
||||
)
|
||||
|
||||
# --- 5. Convert markdown links: [text](url) → text (url) ---
|
||||
result = re.sub(r"\[([^\]]+)\]\(([^)]+)\)", r"\1 (\2)", result)
|
||||
|
||||
# --- 6. Restore protected sections ---
|
||||
for i, fence in enumerate(fences):
|
||||
result = result.replace(f"{_FENCE_PH}{i}\x00", fence)
|
||||
for i, code in enumerate(codes):
|
||||
result = result.replace(f"{_CODE_PH}{i}\x00", code)
|
||||
|
||||
return result
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,645 @@
|
||||
"""
|
||||
yuanbao_media.py — 元宝平台媒体处理模块
|
||||
|
||||
提供 COS 上传、文件下载、TIM 媒体消息构建等功能。
|
||||
移植自 TypeScript 版 media.ts（yuanbao-openclaw-plugin），
|
||||
使用 httpx 替代 cos-nodejs-sdk-v5，避免引入额外 SDK 依赖。
|
||||
|
||||
COS 上传流程：
|
||||
1. 调用 genUploadInfo 获取临时凭证（tmpSecretId/tmpSecretKey/sessionToken）
|
||||
2. 用临时凭证通过 HMAC-SHA1 签名构建 Authorization 头
|
||||
3. HTTP PUT 上传到 COS
|
||||
|
||||
TIM 消息体构建：
|
||||
- buildImageMsgBody() → TIMImageElem
|
||||
- buildFileMsgBody() → TIMFileElem
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
import struct
|
||||
import time
|
||||
import urllib.parse
|
||||
from typing import Optional, Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ============ 常量 ============
|
||||
|
||||
UPLOAD_INFO_PATH = "/api/resource/genUploadInfo"
|
||||
DEFAULT_API_DOMAIN = "yuanbao.tencent.com"
|
||||
DEFAULT_MAX_SIZE_MB = 50
|
||||
|
||||
# COS 加速域名后缀（优先使用全球加速）
|
||||
COS_USE_ACCELERATE = True
|
||||
|
||||
# ============ 类型映射 ============
|
||||
|
||||
# MIME → image_format 数字（TIM 协议字段）
|
||||
_MIME_TO_IMAGE_FORMAT: dict[str, int] = {
|
||||
"image/jpeg": 1,
|
||||
"image/jpg": 1,
|
||||
"image/gif": 2,
|
||||
"image/png": 3,
|
||||
"image/bmp": 4,
|
||||
"image/webp": 255,
|
||||
"image/heic": 255,
|
||||
"image/tiff": 255,
|
||||
}
|
||||
|
||||
# 文件扩展名 → MIME
|
||||
_EXT_TO_MIME: dict[str, str] = {
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
".png": "image/png",
|
||||
".gif": "image/gif",
|
||||
".webp": "image/webp",
|
||||
".bmp": "image/bmp",
|
||||
".heic": "image/heic",
|
||||
".tiff": "image/tiff",
|
||||
".ico": "image/x-icon",
|
||||
".pdf": "application/pdf",
|
||||
".doc": "application/msword",
|
||||
".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
||||
".xls": "application/vnd.ms-excel",
|
||||
".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||
".ppt": "application/vnd.ms-powerpoint",
|
||||
".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
|
||||
".txt": "text/plain",
|
||||
".zip": "application/zip",
|
||||
".tar": "application/x-tar",
|
||||
".gz": "application/gzip",
|
||||
".mp3": "audio/mpeg",
|
||||
".mp4": "video/mp4",
|
||||
".wav": "audio/wav",
|
||||
".ogg": "audio/ogg",
|
||||
".webm": "video/webm",
|
||||
}
|
||||
|
||||
|
||||
# ============ 工具函数 ============
|
||||
|
||||
def guess_mime_type(filename: str) -> str:
|
||||
"""根据文件扩展名猜测 MIME 类型。"""
|
||||
ext = os.path.splitext(filename)[-1].lower()
|
||||
return _EXT_TO_MIME.get(ext, "application/octet-stream")
|
||||
|
||||
|
||||
def is_image(filename: str, mime_type: str = "") -> bool:
|
||||
"""判断是否为图片类型。"""
|
||||
if mime_type.startswith("image/"):
|
||||
return True
|
||||
ext = os.path.splitext(filename)[-1].lower()
|
||||
return ext in {".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp", ".heic", ".tiff", ".ico"}
|
||||
|
||||
|
||||
def get_image_format(mime_type: str) -> int:
|
||||
"""获取 TIM 图片格式编号。"""
|
||||
return _MIME_TO_IMAGE_FORMAT.get(mime_type.lower(), 255)
|
||||
|
||||
|
||||
def md5_hex(data: bytes) -> str:
|
||||
"""计算 MD5 十六进制摘要。"""
|
||||
return hashlib.md5(data).hexdigest()
|
||||
|
||||
|
||||
def generate_file_id() -> str:
|
||||
"""生成随机文件 ID（32 位 hex）。"""
|
||||
return secrets.token_hex(16)
|
||||
|
||||
|
||||
|
||||
# ============ 图片尺寸解析（纯 Python，无需 Pillow） ============
|
||||
|
||||
def parse_image_size(data: bytes) -> Optional[dict[str, int]]:
|
||||
"""
|
||||
解析图片宽高（支持 JPEG/PNG/GIF/WebP），无需第三方依赖。
|
||||
返回 {"width": w, "height": h} 或 None（无法识别）。
|
||||
"""
|
||||
return (
|
||||
_parse_png_size(data)
|
||||
or _parse_jpeg_size(data)
|
||||
or _parse_gif_size(data)
|
||||
or _parse_webp_size(data)
|
||||
)
|
||||
|
||||
|
||||
def _parse_png_size(buf: bytes) -> Optional[dict[str, int]]:
|
||||
if len(buf) < 24:
|
||||
return None
|
||||
if buf[:4] != b"\x89PNG":
|
||||
return None
|
||||
w = struct.unpack(">I", buf[16:20])[0]
|
||||
h = struct.unpack(">I", buf[20:24])[0]
|
||||
return {"width": w, "height": h}
|
||||
|
||||
|
||||
def _parse_jpeg_size(buf: bytes) -> Optional[dict[str, int]]:
|
||||
if len(buf) < 4 or buf[0] != 0xFF or buf[1] != 0xD8:
|
||||
return None
|
||||
i = 2
|
||||
while i < len(buf) - 9:
|
||||
if buf[i] != 0xFF:
|
||||
i += 1
|
||||
continue
|
||||
marker = buf[i + 1]
|
||||
if marker in {0xC0, 0xC2}:
|
||||
h = struct.unpack(">H", buf[i + 5: i + 7])[0]
|
||||
w = struct.unpack(">H", buf[i + 7: i + 9])[0]
|
||||
return {"width": w, "height": h}
|
||||
if i + 3 < len(buf):
|
||||
i += 2 + struct.unpack(">H", buf[i + 2: i + 4])[0]
|
||||
else:
|
||||
break
|
||||
return None
|
||||
|
||||
|
||||
def _parse_gif_size(buf: bytes) -> Optional[dict[str, int]]:
|
||||
if len(buf) < 10:
|
||||
return None
|
||||
sig = buf[:6].decode("ascii", errors="replace")
|
||||
if sig not in {"GIF87a", "GIF89a"}:
|
||||
return None
|
||||
w = struct.unpack("<H", buf[6:8])[0]
|
||||
h = struct.unpack("<H", buf[8:10])[0]
|
||||
return {"width": w, "height": h}
|
||||
|
||||
|
||||
def _parse_webp_size(buf: bytes) -> Optional[dict[str, int]]:
|
||||
if len(buf) < 16:
|
||||
return None
|
||||
if buf[:4] != b"RIFF" or buf[8:12] != b"WEBP":
|
||||
return None
|
||||
chunk = buf[12:16].decode("ascii", errors="replace")
|
||||
if chunk == "VP8 ":
|
||||
if len(buf) >= 30 and buf[23] == 0x9D and buf[24] == 0x01 and buf[25] == 0x2A:
|
||||
w = struct.unpack("<H", buf[26:28])[0] & 0x3FFF
|
||||
h = struct.unpack("<H", buf[28:30])[0] & 0x3FFF
|
||||
return {"width": w, "height": h}
|
||||
elif chunk == "VP8L":
|
||||
if len(buf) >= 25 and buf[20] == 0x2F:
|
||||
bits = struct.unpack("<I", buf[21:25])[0]
|
||||
w = (bits & 0x3FFF) + 1
|
||||
h = ((bits >> 14) & 0x3FFF) + 1
|
||||
return {"width": w, "height": h}
|
||||
elif chunk == "VP8X":
|
||||
if len(buf) >= 30:
|
||||
w = (buf[24] | (buf[25] << 8) | (buf[26] << 16)) + 1
|
||||
h = (buf[27] | (buf[28] << 8) | (buf[29] << 16)) + 1
|
||||
return {"width": w, "height": h}
|
||||
return None
|
||||
|
||||
|
||||
# ============ URL 下载 ============
|
||||
|
||||
async def download_url(
|
||||
url: str,
|
||||
max_size_mb: int = DEFAULT_MAX_SIZE_MB,
|
||||
) -> tuple[bytes, str]:
|
||||
"""
|
||||
下载 URL 内容，返回 (bytes, content_type)。
|
||||
|
||||
Args:
|
||||
url: HTTP(S) URL
|
||||
max_size_mb: 最大允许大小（MB），超过则抛出异常
|
||||
|
||||
Returns:
|
||||
(data_bytes, content_type_string)
|
||||
|
||||
Raises:
|
||||
ValueError: 内容超过大小限制
|
||||
httpx.HTTPError: 网络/HTTP 错误
|
||||
"""
|
||||
max_bytes = max_size_mb * 1024 * 1024
|
||||
async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
|
||||
# 先 HEAD 检查大小
|
||||
try:
|
||||
head = await client.head(url)
|
||||
content_length = int(head.headers.get("content-length", 0) or 0)
|
||||
if content_length > 0 and content_length > max_bytes:
|
||||
raise ValueError(
|
||||
f"文件过大: {content_length / 1024 / 1024:.1f} MB > {max_size_mb} MB"
|
||||
)
|
||||
except httpx.HTTPStatusError:
|
||||
pass # 部分服务器不支持 HEAD，忽略
|
||||
|
||||
# GET 下载（流式读取，防止超限）
|
||||
async with client.stream("GET", url) as resp:
|
||||
resp.raise_for_status()
|
||||
|
||||
content_type = resp.headers.get("content-type", "").split(";")[0].strip()
|
||||
|
||||
chunks: list[bytes] = []
|
||||
downloaded = 0
|
||||
async for chunk in resp.aiter_bytes(65536):
|
||||
downloaded += len(chunk)
|
||||
if downloaded > max_bytes:
|
||||
raise ValueError(
|
||||
f"文件过大: 已超过 {max_size_mb} MB 限制"
|
||||
)
|
||||
chunks.append(chunk)
|
||||
|
||||
data = b"".join(chunks)
|
||||
return data, content_type
|
||||
|
||||
|
||||
# ============ COS 鉴权（HMAC-SHA1） ============
|
||||
|
||||
def _cos_sign(
|
||||
method: str,
|
||||
path: str,
|
||||
params: dict[str, str],
|
||||
headers: dict[str, str],
|
||||
secret_id: str,
|
||||
secret_key: str,
|
||||
start_time: Optional[int] = None,
|
||||
expire_seconds: int = 3600,
|
||||
) -> str:
|
||||
"""
|
||||
构建 COS 请求签名（q-sign-algorithm=sha1 方案）。
|
||||
参考：https://cloud.tencent.com/document/product/436/7778
|
||||
|
||||
Args:
|
||||
method: HTTP 方法（小写，如 "put"）
|
||||
path: URL 路径（URL encode 后的小写）
|
||||
params: URL 查询参数 dict（用于签名）
|
||||
headers: 参与签名的请求头 dict（key 需小写）
|
||||
secret_id: 临时 SecretId（tmpSecretId）
|
||||
secret_key: 临时 SecretKey（tmpSecretKey）
|
||||
start_time: 签名起始 Unix 时间戳（默认 now）
|
||||
expire_seconds: 签名有效期（秒，默认 3600）
|
||||
|
||||
Returns:
|
||||
Authorization header 值（完整字符串）
|
||||
"""
|
||||
now = int(time.time())
|
||||
q_sign_time = f"{start_time or now};{(start_time or now) + expire_seconds}"
|
||||
|
||||
# Step 1: SignKey = HMAC-SHA1(SecretKey, q-sign-time)
|
||||
sign_key = hmac.new(
|
||||
secret_key.encode("utf-8"),
|
||||
q_sign_time.encode("utf-8"),
|
||||
hashlib.sha1,
|
||||
).hexdigest()
|
||||
|
||||
# Step 2: HttpString
|
||||
# 参数和头部需按字典序排列，key 小写
|
||||
sorted_params = sorted((k.lower(), urllib.parse.quote(str(v), safe="") ) for k, v in params.items())
|
||||
sorted_headers = sorted((k.lower(), urllib.parse.quote(str(v), safe="") ) for k, v in headers.items())
|
||||
|
||||
url_param_list = ";".join(k for k, _ in sorted_params)
|
||||
url_params = "&".join(f"{k}={v}" for k, v in sorted_params)
|
||||
header_list = ";".join(k for k, _ in sorted_headers)
|
||||
header_str = "&".join(f"{k}={v}" for k, v in sorted_headers)
|
||||
|
||||
http_string = "\n".join([
|
||||
method.lower(),
|
||||
path,
|
||||
url_params,
|
||||
header_str,
|
||||
"",
|
||||
])
|
||||
|
||||
# Step 3: StringToSign = sha1 hash of HttpString
|
||||
sha1_of_http = hashlib.sha1(http_string.encode("utf-8")).hexdigest()
|
||||
string_to_sign = "\n".join([
|
||||
"sha1",
|
||||
q_sign_time,
|
||||
sha1_of_http,
|
||||
"",
|
||||
])
|
||||
|
||||
# Step 4: Signature = HMAC-SHA1(SignKey, StringToSign)
|
||||
signature = hmac.new(
|
||||
sign_key.encode("utf-8"),
|
||||
string_to_sign.encode("utf-8"),
|
||||
hashlib.sha1,
|
||||
).hexdigest()
|
||||
|
||||
return (
|
||||
f"q-sign-algorithm=sha1"
|
||||
f"&q-ak={secret_id}"
|
||||
f"&q-sign-time={q_sign_time}"
|
||||
f"&q-key-time={q_sign_time}"
|
||||
f"&q-header-list={header_list}"
|
||||
f"&q-url-param-list={url_param_list}"
|
||||
f"&q-signature={signature}"
|
||||
)
|
||||
|
||||
|
||||
# ============ 主要公开 API ============
|
||||
|
||||
async def get_cos_credentials(
|
||||
app_key: str,
|
||||
api_domain: str,
|
||||
token: str,
|
||||
filename: str = "file",
|
||||
file_id: Optional[str] = None,
|
||||
bot_id: str = "",
|
||||
route_env: str = "",
|
||||
) -> dict:
|
||||
"""
|
||||
调用 genUploadInfo 接口获取 COS 临时密钥及上传配置。
|
||||
|
||||
Args:
|
||||
app_key: 应用 Key（用于 X-ID 头）
|
||||
api_domain: API 域名（如 https://bot.yuanbao.tencent.com）
|
||||
token: 当前有效的签票 token（X-Token 头）
|
||||
filename: 待上传的文件名（含扩展名）
|
||||
file_id: 客户端生成的唯一文件 ID（不传则自动生成）
|
||||
bot_id: Bot 账号 ID（用于 X-ID 头）
|
||||
|
||||
Returns:
|
||||
COS 上传配置 dict，包含以下字段：
|
||||
bucketName (str) — COS Bucket 名称
|
||||
region (str) — COS 地域
|
||||
location (str) — 上传 Key（对象路径）
|
||||
encryptTmpSecretId (str) — 临时 SecretId
|
||||
encryptTmpSecretKey(str) — 临时 SecretKey
|
||||
encryptToken (str) — SessionToken
|
||||
startTime (int) — 凭证起始时间戳（Unix）
|
||||
expiredTime (int) — 凭证过期时间戳（Unix）
|
||||
resourceUrl (str) — 上传后的公网访问 URL
|
||||
resourceID (str) — 资源 ID（可选）
|
||||
|
||||
Raises:
|
||||
RuntimeError: 接口返回非 0 code 或字段缺失
|
||||
"""
|
||||
if file_id is None:
|
||||
file_id = generate_file_id()
|
||||
|
||||
upload_url = f"{api_domain.rstrip('/')}{UPLOAD_INFO_PATH}"
|
||||
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"X-Token": token,
|
||||
"X-ID": bot_id or app_key,
|
||||
"X-Source": "web",
|
||||
}
|
||||
if route_env:
|
||||
headers["X-Route-Env"] = route_env
|
||||
body = {
|
||||
"fileName": filename,
|
||||
"fileId": file_id,
|
||||
"docFrom": "localDoc",
|
||||
"docOpenId": "",
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0) as client:
|
||||
resp = await client.post(upload_url, json=body, headers=headers)
|
||||
resp.raise_for_status()
|
||||
result: dict[str, Any] = resp.json()
|
||||
|
||||
code = result.get("code")
|
||||
if code != 0 and code is not None:
|
||||
raise RuntimeError(
|
||||
f"genUploadInfo 失败: code={code}, msg={result.get('msg', '')}"
|
||||
)
|
||||
|
||||
data = result.get("data") or result
|
||||
required_fields = ["bucketName", "location"]
|
||||
missing = [f for f in required_fields if not data.get(f)]
|
||||
if missing:
|
||||
raise RuntimeError(
|
||||
f"genUploadInfo 返回字段不完整: 缺少字段 {missing}"
|
||||
)
|
||||
|
||||
return data
|
||||
|
||||
|
||||
async def upload_to_cos(
|
||||
file_bytes: bytes,
|
||||
filename: str,
|
||||
content_type: str,
|
||||
credentials: dict,
|
||||
bucket: str,
|
||||
region: str,
|
||||
) -> dict:
|
||||
"""
|
||||
通过 httpx PUT 请求将文件上传到 COS。
|
||||
使用临时凭证（tmpSecretId/tmpSecretKey/sessionToken）构建 HMAC-SHA1 签名。
|
||||
|
||||
Args:
|
||||
file_bytes: 文件二进制内容
|
||||
filename: 文件名（用于辅助计算 MIME、UUID）
|
||||
content_type: MIME 类型（如 "image/jpeg"）
|
||||
credentials: get_cos_credentials() 返回的 dict，包含：
|
||||
encryptTmpSecretId → tmpSecretId
|
||||
encryptTmpSecretKey → tmpSecretKey
|
||||
encryptToken → sessionToken
|
||||
location → COS key（对象路径）
|
||||
resourceUrl → 上传后公网 URL
|
||||
startTime → 凭证起始时间（Unix）
|
||||
expiredTime → 凭证过期时间（Unix）
|
||||
bucket: COS Bucket 名称（如 chatbot-1234567890）
|
||||
region: COS 地域（如 ap-guangzhou）
|
||||
|
||||
Returns:
|
||||
上传结果 dict，包含：
|
||||
url (str) — COS 公网访问 URL
|
||||
uuid (str) — 文件内容 MD5
|
||||
size (int) — 文件大小（字节）
|
||||
width (int, optional) — 图片宽度（仅图片）
|
||||
height (int, optional) — 图片高度（仅图片）
|
||||
|
||||
Raises:
|
||||
httpx.HTTPStatusError: COS 返回非 2xx 状态
|
||||
RuntimeError: credentials 字段缺失
|
||||
"""
|
||||
secret_id: str = credentials.get("encryptTmpSecretId", "")
|
||||
secret_key: str = credentials.get("encryptTmpSecretKey", "")
|
||||
session_token: str = credentials.get("encryptToken", "")
|
||||
cos_key: str = credentials.get("location", "")
|
||||
resource_url: str = credentials.get("resourceUrl", "")
|
||||
start_time: Optional[int] = credentials.get("startTime")
|
||||
expired_time: Optional[int] = credentials.get("expiredTime")
|
||||
|
||||
if not secret_id or not secret_key or not cos_key:
|
||||
raise RuntimeError(
|
||||
f"COS credentials 不完整: secretId={bool(secret_id)}, "
|
||||
f"secretKey={bool(secret_key)}, location={bool(cos_key)}"
|
||||
)
|
||||
|
||||
# 构建 COS 上传 URL（优先使用全球加速域名）
|
||||
if COS_USE_ACCELERATE:
|
||||
cos_host = f"{bucket}.cos.accelerate.myqcloud.com"
|
||||
else:
|
||||
cos_host = f"{bucket}.cos.{region}.myqcloud.com"
|
||||
|
||||
# URL encode cos_key（保留 /）
|
||||
encoded_key = urllib.parse.quote(cos_key, safe="/")
|
||||
cos_url = f"https://{cos_host}/{encoded_key.lstrip('/')}"
|
||||
|
||||
# 确定 Content-Type
|
||||
if not content_type or content_type == "application/octet-stream":
|
||||
if is_image(filename):
|
||||
content_type = guess_mime_type(filename)
|
||||
else:
|
||||
content_type = "application/octet-stream"
|
||||
|
||||
# 计算文件 MD5 + size
|
||||
file_uuid = md5_hex(file_bytes)
|
||||
file_size = len(file_bytes)
|
||||
|
||||
# 参与签名的请求头
|
||||
sign_headers = {
|
||||
"host": cos_host,
|
||||
"content-type": content_type,
|
||||
"x-cos-security-token": session_token,
|
||||
}
|
||||
|
||||
# 计算签名有效期
|
||||
now = int(time.time())
|
||||
sign_start = start_time if start_time else now
|
||||
sign_expire = (expired_time - now) if expired_time and expired_time > now else 3600
|
||||
|
||||
authorization = _cos_sign(
|
||||
method="put",
|
||||
path=f"/{encoded_key.lstrip('/')}",
|
||||
params={},
|
||||
headers=sign_headers,
|
||||
secret_id=secret_id,
|
||||
secret_key=secret_key,
|
||||
start_time=sign_start,
|
||||
expire_seconds=sign_expire,
|
||||
)
|
||||
|
||||
put_headers = {
|
||||
"Authorization": authorization,
|
||||
"Content-Type": content_type,
|
||||
"x-cos-security-token": session_token,
|
||||
}
|
||||
|
||||
logger.info(
|
||||
"COS PUT: bucket=%s region=%s key=%s size=%d mime=%s",
|
||||
bucket, region, cos_key, file_size, content_type,
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
resp = await client.put(
|
||||
cos_url,
|
||||
content=file_bytes,
|
||||
headers=put_headers,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
# 解析图片尺寸（仅图片类型）
|
||||
result: dict[str, Any] = {
|
||||
"url": resource_url or cos_url,
|
||||
"uuid": file_uuid,
|
||||
"size": file_size,
|
||||
}
|
||||
|
||||
if content_type.startswith("image/"):
|
||||
size_info = parse_image_size(file_bytes)
|
||||
if size_info:
|
||||
result["width"] = size_info["width"]
|
||||
result["height"] = size_info["height"]
|
||||
|
||||
logger.info(
|
||||
"COS 上传成功: url=%s size=%d",
|
||||
result["url"], file_size,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
# ============ TIM 媒体消息构建 ============
|
||||
|
||||
def build_image_msg_body(
|
||||
url: str,
|
||||
uuid: Optional[str] = None,
|
||||
filename: Optional[str] = None,
|
||||
size: int = 0,
|
||||
width: int = 0,
|
||||
height: int = 0,
|
||||
mime_type: str = "",
|
||||
) -> list[dict]:
|
||||
"""
|
||||
构建腾讯 IM TIMImageElem 消息体。
|
||||
参考：https://cloud.tencent.com/document/product/269/2720
|
||||
|
||||
Args:
|
||||
url: 图片公网访问 URL（COS resourceUrl）
|
||||
uuid: 文件 UUID（MD5 或其他唯一标识）
|
||||
filename: 文件名（uuid 为空时作为备用）
|
||||
size: 文件大小（字节）
|
||||
width: 图片宽度（像素）
|
||||
height: 图片高度（像素）
|
||||
mime_type: MIME 类型（用于确定 image_format）
|
||||
|
||||
Returns:
|
||||
TIMImageElem 消息体列表（适合直接放入 msg_body）
|
||||
"""
|
||||
_uuid = uuid or filename or _basename_from_url(url) or "image"
|
||||
image_format = get_image_format(mime_type) if mime_type else 255
|
||||
|
||||
return [
|
||||
{
|
||||
"msg_type": "TIMImageElem",
|
||||
"msg_content": {
|
||||
"uuid": _uuid,
|
||||
"image_format": image_format,
|
||||
"image_info_array": [
|
||||
{
|
||||
"type": 1, # 1 = 原图
|
||||
"size": size,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"url": url,
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def build_file_msg_body(
|
||||
url: str,
|
||||
filename: str,
|
||||
uuid: Optional[str] = None,
|
||||
size: int = 0,
|
||||
) -> list[dict]:
|
||||
"""
|
||||
构建腾讯 IM TIMFileElem 消息体。
|
||||
参考：https://cloud.tencent.com/document/product/269/2720
|
||||
|
||||
Args:
|
||||
url: 文件公网访问 URL（COS resourceUrl）
|
||||
filename: 文件名（含扩展名）
|
||||
uuid: 文件 UUID（MD5 或其他唯一标识，不传则使用 filename）
|
||||
size: 文件大小（字节）
|
||||
|
||||
Returns:
|
||||
TIMFileElem 消息体列表（适合直接放入 msg_body）
|
||||
"""
|
||||
_uuid = uuid or filename
|
||||
|
||||
return [
|
||||
{
|
||||
"msg_type": "TIMFileElem",
|
||||
"msg_content": {
|
||||
"uuid": _uuid,
|
||||
"file_name": filename,
|
||||
"file_size": size,
|
||||
"url": url,
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
# ============ 内部工具 ============
|
||||
|
||||
def _basename_from_url(url: str) -> str:
|
||||
"""从 URL 提取文件名。"""
|
||||
try:
|
||||
parsed = urllib.parse.urlparse(url)
|
||||
return os.path.basename(parsed.path)
|
||||
except Exception:
|
||||
return ""
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,558 @@
|
||||
"""
|
||||
Yuanbao sticker (TIMFaceElem) support.
|
||||
|
||||
Ported from yuanbao-openclaw-plugin/src/sticker/.
|
||||
|
||||
TIMFaceElem wire format:
|
||||
{
|
||||
"msg_type": "TIMFaceElem",
|
||||
"msg_content": {
|
||||
"index": 0, # always 0 per Yuanbao convention
|
||||
"data": "<json>", # serialised sticker metadata
|
||||
}
|
||||
}
|
||||
|
||||
The `data` field carries a JSON string with the sticker's metadata so the
|
||||
receiver can look up the correct asset in the emoji pack.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
import unicodedata
|
||||
from typing import Optional
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sticker catalogue – ported from builtin-stickers.json
|
||||
# Key : canonical name (Chinese)
|
||||
# Value : {sticker_id, package_id, name, description, width, height, formats}
|
||||
# ---------------------------------------------------------------------------
|
||||
STICKER_MAP: dict[str, dict] = {
|
||||
"六六六": {
|
||||
"sticker_id": "278", "package_id": "1003", "name": "六六六",
|
||||
"description": "666 厉害 牛 棒 绝了 好强 awesome",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"我想开了": {
|
||||
"sticker_id": "262", "package_id": "1003", "name": "我想开了",
|
||||
"description": "想开 佛系 释怀 顿悟 看淡了 无所谓",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"害羞": {
|
||||
"sticker_id": "130", "package_id": "1003", "name": "害羞",
|
||||
"description": "腼腆 不好意思 脸红 娇羞 羞涩 捂脸",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"比心": {
|
||||
"sticker_id": "252", "package_id": "1003", "name": "比心",
|
||||
"description": "笔芯 爱你 爱心手势 love heart 喜欢你",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"委屈": {
|
||||
"sticker_id": "125", "package_id": "1003", "name": "委屈",
|
||||
"description": "难过 想哭 可怜巴巴 瘪嘴 受伤 被欺负",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"亲亲": {
|
||||
"sticker_id": "146", "package_id": "1003", "name": "亲亲",
|
||||
"description": "么么 mua 亲一下 kiss 飞吻 啵",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"酷": {
|
||||
"sticker_id": "131", "package_id": "1003", "name": "酷",
|
||||
"description": "帅 墨镜 cool 高冷 有型 swagger",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"睡": {
|
||||
"sticker_id": "145", "package_id": "1003", "name": "睡",
|
||||
"description": "睡觉 困 zzZ 打盹 躺平 休眠 sleepy",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"发呆": {
|
||||
"sticker_id": "152", "package_id": "1003", "name": "发呆",
|
||||
"description": "懵 愣住 放空 呆滞 出神 脑子空白",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"可怜": {
|
||||
"sticker_id": "157", "package_id": "1003", "name": "可怜",
|
||||
"description": "卖萌 求饶 委屈巴巴 弱小 拜托 眼巴巴",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"摊手": {
|
||||
"sticker_id": "200", "package_id": "1003", "name": "摊手",
|
||||
"description": "无奈 没办法 耸肩 随便 那咋整 whatever",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"头大": {
|
||||
"sticker_id": "213", "package_id": "1003", "name": "头大",
|
||||
"description": "头疼 烦恼 郁闷 难搞 崩溃 一团乱",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"吓": {
|
||||
"sticker_id": "256", "package_id": "1003", "name": "吓",
|
||||
"description": "害怕 惊恐 震惊 吓一跳 恐怖 怂",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"吐血": {
|
||||
"sticker_id": "203", "package_id": "1003", "name": "吐血",
|
||||
"description": "无语 崩溃 被雷 内伤 一口老血 屮",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"哼": {
|
||||
"sticker_id": "185", "package_id": "1003", "name": "哼",
|
||||
"description": "傲娇 生气 不满 撇嘴 不理 赌气",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"嘿嘿": {
|
||||
"sticker_id": "220", "package_id": "1003", "name": "嘿嘿",
|
||||
"description": "坏笑 猥琐笑 偷笑 憨笑 得意 你懂的",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"头秃": {
|
||||
"sticker_id": "218", "package_id": "1003", "name": "头秃",
|
||||
"description": "程序员 加班 焦虑 没头发 秃了 肝爆",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"暗中观察": {
|
||||
"sticker_id": "221", "package_id": "1003", "name": "暗中观察",
|
||||
"description": "窥屏 潜水 偷偷看 角落 围观 屏住呼吸",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"我酸了": {
|
||||
"sticker_id": "224", "package_id": "1003", "name": "我酸了",
|
||||
"description": "嫉妒 柠檬精 羡慕 吃柠檬 眼红 恰柠檬",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"打call": {
|
||||
"sticker_id": "246", "package_id": "1003", "name": "打call",
|
||||
"description": "应援 加油 支持 喝彩 助威 call",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"庆祝": {
|
||||
"sticker_id": "251", "package_id": "1003", "name": "庆祝",
|
||||
"description": "祝贺 开心 耶 party 胜利 干杯",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"奋斗": {
|
||||
"sticker_id": "151", "package_id": "1003", "name": "奋斗",
|
||||
"description": "努力 加油 拼搏 冲 干劲 卷起来",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"惊讶": {
|
||||
"sticker_id": "143", "package_id": "1003", "name": "惊讶",
|
||||
"description": "震惊 哇 不敢相信 OMG 居然 这么离谱",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"疑问": {
|
||||
"sticker_id": "144", "package_id": "1003", "name": "疑问",
|
||||
"description": "问号 不懂 啥 为什么 啥情况 懵逼问",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"仔细分析": {
|
||||
"sticker_id": "248", "package_id": "1003", "name": "仔细分析",
|
||||
"description": "思考 推敲 认真 研究 琢磨 让我想想",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"撅嘴": {
|
||||
"sticker_id": "184", "package_id": "1003", "name": "撅嘴",
|
||||
"description": "嘟嘴 卖萌 不高兴 撒娇 嘴翘",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"泪奔": {
|
||||
"sticker_id": "199", "package_id": "1003", "name": "泪奔",
|
||||
"description": "大哭 伤心 破防 感动哭 泪流满面 呜呜",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"尊嘟假嘟": {
|
||||
"sticker_id": "276", "package_id": "1003", "name": "尊嘟假嘟",
|
||||
"description": "真的假的 真假 可爱问 你骗我 是不是",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"略略略": {
|
||||
"sticker_id": "113", "package_id": "1003", "name": "略略略",
|
||||
"description": "调皮 吐舌 不服 略 气死你 鬼脸",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"困": {
|
||||
"sticker_id": "180", "package_id": "1003", "name": "困",
|
||||
"description": "想睡 倦 打哈欠 睁不开眼 好困啊 sleepy",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"折磨": {
|
||||
"sticker_id": "181", "package_id": "1003", "name": "折磨",
|
||||
"description": "难受 痛苦 煎熬 蚌埠住了 受不了 要命",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"抠鼻": {
|
||||
"sticker_id": "182", "package_id": "1003", "name": "抠鼻",
|
||||
"description": "不屑 无聊 淡定 无所谓 鄙视 挖鼻",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"鼓掌": {
|
||||
"sticker_id": "183", "package_id": "1003", "name": "鼓掌",
|
||||
"description": "拍手 叫好 赞同 666 喝彩 掌声",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"斜眼笑": {
|
||||
"sticker_id": "204", "package_id": "1003", "name": "斜眼笑",
|
||||
"description": "滑稽 坏笑 doge 意味深长 阴阳怪气 嘿嘿嘿",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"辣眼睛": {
|
||||
"sticker_id": "216", "package_id": "1003", "name": "辣眼睛",
|
||||
"description": "看不下去 cringe 毁三观 太丑了 瞎了",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"哦哟": {
|
||||
"sticker_id": "217", "package_id": "1003", "name": "哦哟",
|
||||
"description": "惊讶 起哄 哇哦 有戏 不简单 哟",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"吃瓜": {
|
||||
"sticker_id": "222", "package_id": "1003", "name": "吃瓜",
|
||||
"description": "围观 看戏 八卦 路人 看热闹 板凳",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"狗头": {
|
||||
"sticker_id": "225", "package_id": "1003", "name": "狗头",
|
||||
"description": "doge 保命 开玩笑 滑稽 反讽 懂的都懂",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"敬礼": {
|
||||
"sticker_id": "227", "package_id": "1003", "name": "敬礼",
|
||||
"description": "salute 尊重 收到 遵命 致敬 报告",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"哦": {
|
||||
"sticker_id": "231", "package_id": "1003", "name": "哦",
|
||||
"description": "知道了 明白 敷衍 嗯 这样啊 收到",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"拿到红包": {
|
||||
"sticker_id": "236", "package_id": "1003", "name": "拿到红包",
|
||||
"description": "红包 谢谢老板 发财 开心 抢到了 欧气",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"牛吖": {
|
||||
"sticker_id": "239", "package_id": "1003", "name": "牛吖",
|
||||
"description": "牛 厉害 强 666 佩服 大佬",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"贴贴": {
|
||||
"sticker_id": "272", "package_id": "1003", "name": "贴贴",
|
||||
"description": "抱抱 亲昵 蹭蹭 亲密 靠靠 撒娇贴",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"爱心": {
|
||||
"sticker_id": "138", "package_id": "1003", "name": "爱心",
|
||||
"description": "心 love 喜欢你 红心 示爱 么么哒",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"晚安": {
|
||||
"sticker_id": "170", "package_id": "1003", "name": "晚安",
|
||||
"description": "好梦 睡了 night 早点休息 安啦 moon",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"太阳": {
|
||||
"sticker_id": "176", "package_id": "1003", "name": "太阳",
|
||||
"description": "晴天 早上好 阳光 morning 好天气 日",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"柠檬": {
|
||||
"sticker_id": "266", "package_id": "1003", "name": "柠檬",
|
||||
"description": "酸 嫉妒 柠檬精 羡慕 我酸 恰柠檬",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"大冤种": {
|
||||
"sticker_id": "267", "package_id": "1003", "name": "大冤种",
|
||||
"description": "倒霉 吃亏 自嘲 好心没好报 背锅 工具人",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"吐了": {
|
||||
"sticker_id": "132", "package_id": "1003", "name": "吐了",
|
||||
"description": "恶心 yue 受不了 嫌弃 想吐 生理不适",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"怒": {
|
||||
"sticker_id": "134", "package_id": "1003", "name": "怒",
|
||||
"description": "生气 愤怒 火大 暴躁 气炸 怼",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"玫瑰": {
|
||||
"sticker_id": "165", "package_id": "1003", "name": "玫瑰",
|
||||
"description": "花 示爱 表白 浪漫 送你花 情人节",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"凋谢": {
|
||||
"sticker_id": "119", "package_id": "1003", "name": "凋谢",
|
||||
"description": "花谢 失恋 难过 枯萎 心碎 凉了",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"点赞": {
|
||||
"sticker_id": "159", "package_id": "1003", "name": "点赞",
|
||||
"description": "赞 认同 好棒 good like 大拇指 顶",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"握手": {
|
||||
"sticker_id": "164", "package_id": "1003", "name": "握手",
|
||||
"description": "合作 你好 商务 hello deal 成交 友好",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"抱拳": {
|
||||
"sticker_id": "163", "package_id": "1003", "name": "抱拳",
|
||||
"description": "谢谢 失敬 江湖 承让 拜托 有礼",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"ok": {
|
||||
"sticker_id": "169", "package_id": "1003", "name": "ok",
|
||||
"description": "好的 收到 没问题 okay 行 可以 懂了",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"拳头": {
|
||||
"sticker_id": "174", "package_id": "1003", "name": "拳头",
|
||||
"description": "加油 干 冲 fight 力量 击拳 硬气",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"鞭炮": {
|
||||
"sticker_id": "191", "package_id": "1003", "name": "鞭炮",
|
||||
"description": "过年 喜庆 爆竹 春节 噼里啪啦 红",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
"烟花": {
|
||||
"sticker_id": "258", "package_id": "1003", "name": "烟花",
|
||||
"description": "庆典 漂亮 新年 嘭 绽放 节日快乐",
|
||||
"width": 128, "height": 128, "formats": "png",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def get_sticker_by_name(name: str) -> Optional[dict]:
|
||||
"""
|
||||
按名称查找贴纸，支持模糊匹配。
|
||||
|
||||
匹配优先级：
|
||||
1. 完全相等（name）
|
||||
2. name 包含查询词（前缀/子串）
|
||||
3. description 包含查询词（同义词搜索）
|
||||
4. 通用模糊评分（与 sticker-search 同算法），命中即返回得分最高的一条
|
||||
|
||||
返回 sticker dict，找不到返回 None。
|
||||
"""
|
||||
if not name:
|
||||
return None
|
||||
|
||||
query = name.strip()
|
||||
|
||||
if query in STICKER_MAP:
|
||||
return STICKER_MAP[query]
|
||||
|
||||
for key, sticker in STICKER_MAP.items():
|
||||
if query in key or key in query:
|
||||
return sticker
|
||||
|
||||
for sticker in STICKER_MAP.values():
|
||||
desc = sticker.get("description", "")
|
||||
if query in desc:
|
||||
return sticker
|
||||
|
||||
matches = search_stickers(query, limit=1)
|
||||
return matches[0] if matches else None
|
||||
|
||||
|
||||
def get_random_sticker(category: str = None) -> dict:
|
||||
"""
|
||||
随机返回一个贴纸。
|
||||
|
||||
若指定 category，则在 description 中含有该关键词的贴纸里随机选取；
|
||||
category 为 None 时从全表随机。
|
||||
"""
|
||||
if category:
|
||||
candidates = [
|
||||
s for s in STICKER_MAP.values()
|
||||
if category in s.get("description", "") or category in s.get("name", "")
|
||||
]
|
||||
if candidates:
|
||||
return random.choice(candidates)
|
||||
return random.choice(list(STICKER_MAP.values()))
|
||||
|
||||
|
||||
def get_sticker_by_id(sticker_id: str) -> Optional[dict]:
|
||||
"""按 sticker_id 精确查找贴纸。"""
|
||||
if not sticker_id:
|
||||
return None
|
||||
sid = str(sticker_id).strip()
|
||||
for sticker in STICKER_MAP.values():
|
||||
if sticker.get("sticker_id") == sid:
|
||||
return sticker
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 模糊搜索（对齐 chatbot-web yuanbao-openclaw-plugin/sticker-cache.ts.searchStickers）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_PUNCT_RE = re.compile(r"[\s\u3000\-_·.,，。!！?？\"“”'‘’、/\\]+")
|
||||
|
||||
|
||||
def _normalize_text(raw: str) -> str:
|
||||
return unicodedata.normalize("NFKC", str(raw or "")).strip().lower()
|
||||
|
||||
|
||||
def _compact_text(raw: str) -> str:
|
||||
return _PUNCT_RE.sub("", _normalize_text(raw))
|
||||
|
||||
|
||||
def _multiset_char_hit_ratio(needle: str, haystack: str) -> float:
|
||||
if not needle:
|
||||
return 0.0
|
||||
bag: dict[str, int] = {}
|
||||
for ch in haystack:
|
||||
bag[ch] = bag.get(ch, 0) + 1
|
||||
hits = 0
|
||||
for ch in needle:
|
||||
n = bag.get(ch, 0)
|
||||
if n > 0:
|
||||
hits += 1
|
||||
bag[ch] = n - 1
|
||||
return hits / len(needle)
|
||||
|
||||
|
||||
def _bigram_jaccard(a: str, b: str) -> float:
|
||||
if len(a) < 2 or len(b) < 2:
|
||||
return 0.0
|
||||
A = {a[i:i + 2] for i in range(len(a) - 1)}
|
||||
B = {b[i:i + 2] for i in range(len(b) - 1)}
|
||||
inter = len(A & B)
|
||||
union = len(A) + len(B) - inter
|
||||
return inter / union if union else 0.0
|
||||
|
||||
|
||||
def _longest_subsequence_ratio(needle: str, haystack: str) -> float:
|
||||
if not needle:
|
||||
return 0.0
|
||||
j = 0
|
||||
for ch in haystack:
|
||||
if j >= len(needle):
|
||||
break
|
||||
if ch == needle[j]:
|
||||
j += 1
|
||||
return j / len(needle)
|
||||
|
||||
|
||||
def _score_field(haystack: str, query: str) -> float:
|
||||
hay = _normalize_text(haystack)
|
||||
q = _normalize_text(query)
|
||||
if not hay or not q:
|
||||
return 0.0
|
||||
hay_c = _compact_text(haystack)
|
||||
q_c = _compact_text(query)
|
||||
best = 0.0
|
||||
if hay == q:
|
||||
best = max(best, 100.0)
|
||||
if q in hay:
|
||||
best = max(best, 92 + min(6, len(q)))
|
||||
if len(q) >= 2 and hay.startswith(q):
|
||||
best = max(best, 88.0)
|
||||
if q_c and q_c in hay_c:
|
||||
best = max(best, 86.0)
|
||||
best = max(best, _multiset_char_hit_ratio(q_c, hay_c) * 62)
|
||||
best = max(best, _bigram_jaccard(q_c, hay_c) * 58)
|
||||
best = max(best, _longest_subsequence_ratio(q_c, hay_c) * 52)
|
||||
if len(q) == 1 and q in hay:
|
||||
best = max(best, 68.0)
|
||||
return best
|
||||
|
||||
|
||||
def search_stickers(query: str, limit: int = 10) -> list[dict]:
|
||||
"""
|
||||
在内置贴纸表中按模糊匹配排序返回前 N 条结果。
|
||||
|
||||
评分综合 name/description 字段的子串、字符多重集覆盖、bigram Jaccard、子序列比例。
|
||||
name 权重略高于 description（×0.88）。空 query 时按字典顺序返回前 N 条。
|
||||
"""
|
||||
safe_limit = max(1, min(500, int(limit) if limit else 10))
|
||||
if not query or not _normalize_text(query):
|
||||
return list(STICKER_MAP.values())[:safe_limit]
|
||||
|
||||
scored: list[tuple[float, dict]] = []
|
||||
for sticker in STICKER_MAP.values():
|
||||
name_s = _score_field(sticker.get("name", ""), query)
|
||||
desc_s = _score_field(sticker.get("description", ""), query) * 0.88
|
||||
sid = str(sticker.get("sticker_id", "")).strip()
|
||||
q_norm = _normalize_text(query)
|
||||
id_s = 0.0
|
||||
if sid and q_norm:
|
||||
sid_norm = _normalize_text(sid)
|
||||
if sid_norm == q_norm:
|
||||
id_s = 100.0
|
||||
elif q_norm in sid_norm:
|
||||
id_s = 84.0
|
||||
scored.append((max(name_s, desc_s, id_s), sticker))
|
||||
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
top = scored[0][0] if scored else 0
|
||||
if top <= 0:
|
||||
return [s for _, s in scored[:safe_limit]]
|
||||
|
||||
if top >= 22:
|
||||
floor = 18.0
|
||||
elif top >= 12:
|
||||
floor = max(10.0, top * 0.5)
|
||||
else:
|
||||
floor = max(6.0, top * 0.35)
|
||||
|
||||
filtered = [pair for pair in scored if pair[0] >= floor]
|
||||
out = filtered if filtered else scored
|
||||
return [s for _, s in out[:safe_limit]]
|
||||
|
||||
|
||||
def build_face_msg_body(
|
||||
face_index: int,
|
||||
face_type: int = 1,
|
||||
data: Optional[str] = None,
|
||||
) -> list:
|
||||
"""
|
||||
构造 TIMFaceElem 消息体。
|
||||
|
||||
Yuanbao 约定：
|
||||
- index 固定传 0（服务端通过 data 字段识别具体表情）
|
||||
- data 为 JSON 字符串，包含 sticker_id / package_id 等字段
|
||||
|
||||
Args:
|
||||
face_index: 保留字段，暂时不影响 wire format（Yuanbao 固定 index=0）。
|
||||
当 face_index > 0 时视为旧版 QQ 表情 ID，直接放入 index。
|
||||
face_type: 保留字段（兼容旧接口，当前未使用）。
|
||||
data: 已序列化的 JSON 字符串；为 None 时仅传 index。
|
||||
|
||||
Returns:
|
||||
符合 Yuanbao TIM 协议的 msg_body list，如::
|
||||
|
||||
[{"msg_type": "TIMFaceElem", "msg_content": {"index": 0, "data": "..."}}]
|
||||
"""
|
||||
msg_content: dict = {"index": face_index}
|
||||
if data is not None:
|
||||
msg_content["data"] = data
|
||||
return [{"msg_type": "TIMFaceElem", "msg_content": msg_content}]
|
||||
|
||||
|
||||
def build_sticker_msg_body(sticker: dict) -> list:
|
||||
"""
|
||||
从 STICKER_MAP 中的 sticker dict 直接构造 TIMFaceElem 消息体。
|
||||
|
||||
这是 send_sticker() 的内部辅助，确保 data 字段与原始 JS 插件一致。
|
||||
"""
|
||||
data_payload = json.dumps(
|
||||
{
|
||||
"sticker_id": sticker["sticker_id"],
|
||||
"package_id": sticker["package_id"],
|
||||
"width": sticker.get("width", 128),
|
||||
"height": sticker.get("height", 128),
|
||||
"formats": sticker.get("formats", "png"),
|
||||
"name": sticker["name"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
return build_face_msg_body(face_index=0, data=data_payload)
|
||||
Reference in New Issue
Block a user