CTI-Automator collects threat intelligence from AlienVault OTX, Abuse.ch ThreatFox, PDF reports and authorized GitHub repositories. It validates IoCs and MITRE ATT&CK mappings, optionally enriches public indicators, and exports STIX 2.1 bundles to files or OpenCTI.
The same operations are available through a Python CLI and a standalone FastMCP server. Docker Compose supports scheduled ingestion with optional profiles for local AI, historical memory and monitoring.
Privacy-first, not remote-service-free: Ollama is the default analysis provider. Hosted AI and VirusTotal/Shodan enrichment require explicit configuration or invocation. Production credentials, provider quotas and OpenCTI permissions must be validated in your own environment.
- Architecture and capabilities
- Installation and configuration
- Using the CLI
- Using MCP
- Data protection (DLP)
- Detection as Code
- Historical memory (RAG)
- SecOps and ChatOps
- Docker Compose deployment
- Logs, metrics and monitoring
- Development and releases
OTX / ThreatFox ── API normalization ────────────────────────┐
│
PDF / authorized repository reports ── Text + AI analysis ──┤
▼
Validated Pydantic models
│
Optional public-IoC enrichment
(VirusTotal / Shodan)
│
STIX 2.1 → file / OpenCTI
CLI / MCP / Compose ingestion loop
JSON events + Prometheus metrics
The stack uses Python 3.11+, Requests, pdfplumber, Pydantic, Ollama or an OpenAI-compatible adapter, FastMCP, stix2 and pycti. Ruff, strict mypy, pytest and GitHub Actions validate the application and its container.
- OTX: subscribed pulses, with modification-time filtering and pagination bounded to 100 pages.
- ThreatFox: recent IoCs from a 1–7 day window.
ip:portvalues become IP observables; the original value remains in internal source metadata. - STIX/OpenCTI: normalized indicators and ATT&CK mappings become STIX 2.1 objects with deterministic IDs. Repeated imports reuse those identities; actual merge behavior depends on the OpenCTI version and permissions. Missing source timestamps use the Unix epoch instead of an invented observation time. Unsupported indicator types fail export explicitly.
The OpenCTI adapter uses pycti's JSON import API with a configured timeout and rejects incomplete or partially rejected imports rather than reporting success.
PDF extraction rejects missing, unreadable, empty or non-PDF input. Report text is passed to the configured LLM as untrusted evidence, not instructions. Pydantic validates the resulting IoCs, confidence, evidence, rationale and MITRE ATT&CK technique mappings. Treat generated intelligence as analysis requiring review, not a guarantee that every model conclusion is correct.
Repository ingestion uses the GitHub REST API with an explicit allowlist:
- Only regular UTF-8
.md,.txtand.rstfiles underreports/,docs/, or the rootREADME.mdare accepted. Sample/binary directories, archives, symlinks and executable files are excluded; fetched content is never executed. - A commit snapshot is pinned and Git blob integrity is checked. Documents retain repository, path, commit, digest, source URL and retrieval time.
- Defaults are 20 changed documents per repository/run, 100000 bytes/document and 2 MB/API response. Remaining changes are processed on later runs; truncated repository trees fail explicitly.
- Atomic, file-locked checkpoints advance only after successful analysis,
requested delivery and saving the validated result. Results are stored with
owner-only permissions in the checkpoint's sibling
.results/directory. Earlier successes survive later failures; concurrent ingestion is rejected.
Telegram ingestion is not currently implemented.
Enrichment is disabled by default. --enrich or MCP's provider arguments
authorize sending the selected public IoC values to external providers.
VirusTotal supports public IPs, domains and valid MD5/SHA-1/SHA-256 hashes. Shodan supports IPs only; when both are selected, non-IP indicators go only to VirusTotal. Results preserve provider, retrieval time, status, confidence and reference URL. STIX external references retain that provenance; source confidence is not overwritten.
Before any request in a batch, the query policy rejects email addresses,
URLs, private/reserved IPs, internal or reserved domains, invalid hashes and
type mismatches. Configure restricted corporate domains too. A mixed batch
containing disallowed values fails as a whole; use MCP's enrich_indicators
with a reviewed public subset if needed. This policy is not a full DLP module.
The process-local cache has a configurable TTL and a 1000-entry limit.
Queries are deduplicated. Local quota exhaustion and HTTP 429 raise
QuotaError; HTTP 404 is a cached not-found. There are no unbounded retries
or silent success fallbacks. Budgets are not shared across cluster replicas.
Install from the repository in an isolated Python environment:
python3 -m venv .venv
source .venv/bin/activate
pip install .
# Required by pycti on Debian; the Docker image already includes it:
sudo apt-get install libmagic1Alternatively, install a wheel from a GitHub Release with pip or pipx.
The package is not assumed to be available on PyPI.
Copy the environment template, insert your values and load it:
cp .env.example .env
chmod 600 .env
# Edit .env locally; never commit real credentials.
set -a
source .env
set +aThe package reads environment variables; it does not automatically load
.env. Only source a file you trust.
| Purpose | Environment variables |
|---|---|
| OTX / ThreatFox | OTX_API_KEY, ABUSECH_API_KEY; optional OTX_BASE_URL, THREATFOX_BASE_URL |
| OpenCTI import | OPENCTI_URL, OPENCTI_TOKEN |
| Local analysis | AI_PROVIDER=ollama, OLLAMA_BASE_URL, OLLAMA_MODEL |
| Hosted analysis | AI_PROVIDER=openai or groq, AI_API_KEY, AI_BASE_URL, AI_MODEL |
| Enrichment credentials | VIRUSTOTAL_API_KEY, SHODAN_API_KEY |
| Enrichment policy | CTI_RESTRICTED_DOMAINS (comma-separated), CTI_ENRICHMENT_CACHE_TTL=3600, CTI_ENRICHMENT_RPM=4 |
| Repository access | CTI_REPOSITORY_ALLOWLIST (comma-separated owner/repo), optional GITHUB_TOKEN |
| Repository limits/state | CTI_REPOSITORY_MAX_FILES=20, CTI_REPOSITORY_MAX_BYTES=100000, CTI_REPOSITORY_CHECKPOINT=/state/repositories.json |
| Network and logs | CTI_REQUEST_TIMEOUT=30, CTI_VERIFY_TLS=true, CTI_LOG_LEVEL=INFO |
OPENCTI_VERIFY_TLS is a compatibility alias for the shared TLS setting;
when both variables are set, it takes precedence over CTI_VERIFY_TLS.
Keep TLS verification enabled outside a controlled test environment.
Ollama is the default and should point to a trusted local service. Selecting a hosted provider sends report text to that provider. For example:
AI_PROVIDER=openai
AI_API_KEY=your-protected-key
AI_BASE_URL=https://api.openai.com/v1
AI_MODEL=gpt-4o-miniFor repository access, use a GitHub token with minimal read-only content permissions. Public repositories can be queried without a token, subject to GitHub's unauthenticated rate limits. Checkpoints never store credentials.
cti-automator --help
cti-automator doctor
cti-automator ui --host 127.0.0.1 --port 8080
cti-automator ingest-otx --modified-since 2026-01-01
cti-automator ingest-threatfox --days 1
cti-automator ingest-otx --opencti
cti-automator ingest-threatfox --days 7 --opencticti-automator doctor checks whether OpenCTI (6.9.29), TheHive (5.5.0-1),
Ollama (0.5.12) and Qdrant (v1.13.4) are running locally and outputs
exact Docker installation proposals when any of them is missing.
cti-automator ui launches a loopback-only local web interface to inspect stack
health, upload and analyze PDFs, filter IoCs/TTPs, preview YARA/SIGMA proposals
and export STIX 2.1 bundles.
Without --opencti, API commands return normalized JSON. Import mode writes
counts to stdout rather than IoC content.
cti-automator analyze-pdf report.pdf --output report.analysis.json
cti-automator export-pdf report.analysis.json --stix-output report.bundle.json
cti-automator export-pdf report.analysis.json --opencti
export CTI_REPOSITORY_ALLOWLIST=owner/reports
export CTI_REPOSITORY_CHECKPOINT="$HOME/.local/state/cti-automator/repositories.json"
cti-automator ingest-repositories
cti-automator ingest-repositories --openctiChoose a writable checkpoint location outside the deployment. Fetching or analyzing a report does not itself authorize an OpenCTI import. Successfully checkpointed repository content is not reprocessed automatically; choose a separate state path if you intentionally need a new processing/delivery pass.
cti-automator ingest-threatfox --enrich virustotal --opencti
cti-automator export-pdf report.pdf --enrich virustotal --stix-output report.jsonThese commands require the selected provider credentials and a batch that passes the public-indicator policy described above.
Enable configurable PII redaction before any LLM call or embedding:
export CTI_DLP_ENABLED=true
export CTI_DLP_EMAIL_DOMAINS=corp.example
export CTI_DLP_INTERNAL_CIDRS=10.0.0.0/8,192.168.0.0/16
export CTI_DLP_NAMES="Alice Smith"
export CTI_DLP_ALLOWED_VALUES=203.0.113.7 # public IoCs that must survive
cti-automator analyze-pdf report.pdfProtected values are replaced with stable <CTI_EMAIL_n>, <CTI_IP_n> and
<CTI_NAME_n> placeholders for the whole execution; counts per category are
exported through the cti_redactions_total metric without logging any
original value. A sanitization failure blocks the LLM call instead of sending
unprotected text. Detection is heuristic (regex and CIDR membership); review
the residual risk documented for ambiguous names and unusual formats.
Generate YARA and SIGMA proposals from an analyzed report:
cti-automator generate-detections report.pdf \
--yara-dir detections --sigma-dir detections
cti-automator validate-detections detections # CI GitOps gateOptional pull-request export requires an explicit allowlist and a token;
without --submit the exporter only prints a dry-run plan:
export CTI_DETECTION_PR_ALLOWLIST=acme/detections
export GITHUB_TOKEN=ghp_...
cti-automator generate-detections report.pdf --pr-repo acme/detections
cti-automator generate-detections report.pdf --pr-repo acme/detections --submitRules always carry status="proposal" (YARA) or the
cti-automator-proposal marker (SIGMA); merging requires human review
through the Detection rules GitOps gate workflow, which validates every
change to detections/. Compile YARA rules with yara and check
SIGMA rules with sigma-cli before deployment. Nothing is deployed
automatically.
Disabled by default; local Qdrant plus local Ollama embeddings keep data on the host:
docker run -d -p 6333:6333 qdrant/qdrant
ollama pull nomic-embed-text
export CTI_MEMORY_ENABLED=true
cti-automator memory-remember report.pdf
cti-automator memory-recall "same infrastructure earlier?" --limit 5Chunks carry tenant, source document, techniques and creation time; recall is
tenant-isolated and returns cited fragments. DLP sanitization is applied
before embeddings, retention is enforced with
CTI_MEMORY_RETENTION_DAYS, and remote embeddings require the explicit
CTI_MEMORY_REMOTE_EMBEDDINGS=true opt-in plus AI_* credentials.
Create one idempotent case per report in TheHive or Cortex XSOAR; dry-run is always available:
export CTI_THEHIVE_URL=https://thehive.internal
export CTI_THEHIVE_TOKEN=...
cti-automator create-secops-case report.pdf --destination thehive --dry-run
cti-automator create-secops-case report.pdf --destination thehive --severity highCases are keyed by a deterministic idempotency key (document + indicators +
techniques): retries reconcile ambiguous responses instead of duplicating,
with bounded attempts (CTI_SECOPS_RETRY_ATTEMPTS). ChatOps notifications
are deduplicated, HMAC-signed webhooks configured as name=url pairs:
export CTI_CHATOPS_WEBHOOKS='soc=https://hooks.example/abcd'
export CTI_CHATOPS_SECRETS='soc=signing-secret'
cti-automator notify-chatops report.pdf --destination soc --dry-run
cti-automator notify-chatops report.pdf --destination socHTTPS public destinations require no extra consent; internal or plain-HTTP
hosts must be listed in CTI_SECOPS_ALLOWED_HOSTS (SSRF protection). No
automatic response actions are performed.
cti-automator mcp
cti-automator mcp --transport streamable-http --host 127.0.0.1 --port 8000The default transport is stdio. HTTP serves MCP at /mcp, process health at
/healthz and Prometheus metrics at /metrics. --host and --port require
an HTTP transport; ports must be between 1 and 65535.
| Tool | Operation |
|---|---|
analyze_pdf |
Return validated PDF intelligence |
export_pdf_to_stix |
Return a PDF-derived STIX bundle |
export_pdf_to_opencti |
Analyze a PDF and import its bundle |
get_otx_pulses |
Fetch pulses; optional opencti=true and enrich providers |
get_threatfox_iocs |
Fetch recent IoCs; optional opencti=true and enrich providers |
enrich_indicators |
Enrich a reviewed list using explicit providers |
push_to_opencti |
Import normalized indicators |
ingest_repositories |
Analyze allowlisted reports; optional opencti=true |
generate_detections |
Return YARA/SIGMA proposal summaries (no writes) |
remember_analysis |
Store one analysis in historical memory |
recall_memory |
Recall cited fragments for a query |
create_secops_case |
Plan or create a TheHive/XSOAR case; dry_run defaults to true |
send_chatops_notification |
Plan or send a signed notification; dry_run defaults to true |
PDF tools also accept optional enrich providers. File paths are resolved on
the server, not on the client: make required reports accessible to that process.
Remote authentication and authorization are not implemented. Keep MCP on localhost or a trusted internal network; do not expose it through a public Ingress or LoadBalancer.
compose.yaml replaces the previous K3s manifests: it runs the
MCP server and a scheduled ingestion loop, with optional profiles for local
AI (local-ai), historical memory (memory) and monitoring (monitoring).
The container workflow validates builds on pull requests and main. Version
tags publish ghcr.io/kta41/cti-automator:<version> and stable latest after
validation. Pin an immutable version or digest in compose.yaml for
production; the default latest is a bootstrap reference. To build locally
instead, change the anchor image to build: .:
docker compose build 2>/dev/null || docker build -t cti-automator:local .You can bootstrap .env (with auto-generated local tokens/passwords and 0600
permissions), create ./state, start the Docker Compose services (including the
Web UI on http://127.0.0.1:8085 and MCP on http://127.0.0.1:8000/mcp), and
pull the required Ollama models with a single command:
# Standard stack: Web UI + MCP + Ingest + Ollama + Qdrant (reuses host services if already running)
cti-automator up
# Full lab stack: adds Prometheus, Grafana, TheHive (5.5.0-1) and OpenCTI (6.9.29)
cti-automator up --mode full
# Inspect stack health and local endpoints
cti-automator status
# Stop all stack profiles cleanly
cti-automator downIf you prefer running docker compose directly, COMPOSE_PROFILES=local-ai,memory
is pre-set in .env.example, so a single docker compose up -d starts the
core services plus local AI and Qdrant:
docker compose up -d
docker compose ps
curl --fail http://127.0.0.1:8000/healthzContainers run as UID/GID 10001 with dropped capabilities, read-only root
filesystems, no-new-privileges and bounded writable /tmp. The MCP port is
published on loopback only — remote authentication is not implemented.
Optional components are enabled with profiles:
docker compose --profile local-ai up -d # local Ollama at 127.0.0.1:11434
docker compose --profile memory up -d # local Qdrant for RAG memory
docker compose --profile monitoring up -d # Prometheus (9090) + Grafana (3000)The ingest service replaces the previous CronJob: it imports OTX, then
ThreatFox, then repository reports if the allowlist is non-empty, every 30
minutes, without enabling enrichment. Repository checkpoints persist in
./state (bind mount); back that directory up.
Health checks test process responsiveness, not source APIs, credentials,
Ollama or OpenCTI. The mcp service healthcheck polls /healthz.
To run an import manually with the same configuration:
docker compose run --rm ingest sh -c \
'cti-automator ingest-otx --opencti && cti-automator ingest-threatfox --opencti'For failures, inspect service logs and restart policies:
docker compose logs -f mcp ingest
docker compose restart ingestOperational events are JSON on stderr, preserving CLI JSON/STIX stdout
and MCP stdio. CTI_LOG_LEVEL controls verbosity. Each MCP invocation gets
a fresh run UUID; nested operations share it. Events record start/end/error,
duration, counts where available and exception category, without arguments,
IoCs, report text, tokens or exception messages.
Root-logger library events omit messages, metadata and tracebacks. MCP errors are masked; framework tracebacks, HTTP access logs and server banners are suppressed. Libraries with independent output can still emit diagnostics: restrict access and retention of all logs and stored analysis artifacts.
/metrics exposes counters for operations/outcomes, processed items and quota
failures, plus latency histograms with bounded operation labels. Metrics are
process-local: the static monitoring target scrapes MCP, not short-lived CLI
Jobs. Aggregate Job JSON events in your log backend; these metrics do not
claim cluster-wide Job totals.
monitoring/ supplies the Prometheus configuration, four alert
rules and a Grafana dashboard used by the Compose monitoring profile:
docker compose --profile monitoring up -d
docker compose logs -f prometheus grafanaPrometheus is reachable on 127.0.0.1:9090 and Grafana on 127.0.0.1:3000
(default password changeme — override GF_SECURITY_ADMIN_PASSWORD in
.env). Alerts cover endpoint unavailability, error ratio above 20%, quota
exhaustion and p95 latency above 120 seconds. Edit thresholds/durations in
monitoring/rules.yml and restart Prometheus to
reload. Retention is 7 days/800 MB on the prometheus_data volume; use a
bind mount for durable history. Configure Alertmanager separately for
notification delivery.
The dashboard is provisioned automatically from monitoring/grafana/dashboards/cti-automator.json with the Prometheus datasource. All monitoring ports stay on loopback.
src/cti_automator/
cli.py CLI entrypoint
mcp_server.py MCP tools, health and metrics routes
config.py, models.py Runtime configuration and validated models
connectors/ OTX and ThreatFox HTTP clients
parsers/, analyzer.py PDF extraction and report analysis
llm.py Local and hosted AI adapters
enrichment.py Optional VirusTotal/Shodan queries
repositories.py Authorized reports and checkpoints
exporters/ STIX and OpenCTI delivery
sanitizer.py Configurable DLP redaction before LLM/embeddings
detections.py YARA/SIGMA proposals, validation and PR export
memory.py Qdrant-backed historical memory (RAG)
secops.py TheHive/XSOAR cases and signed ChatOps webhooks
observability.py Structured events and metrics
tests/ Regression tests and runtime/alert fixtures
detections/ Committed rule proposals under GitOps review
compose.yaml Docker Compose deployment (mcp, ingest, profiles)
monitoring/ Prometheus and Grafana configuration for Compose
.github/workflows/ Releases, container, PyPI publishing and rule gates
Dockerfile Multistage, non-root runtime with libmagic
pyproject.toml Packaging, dependencies and development tooling
.env.example Configuration reference
Install development dependencies and run checks:
source .venv/bin/activate
pip install -e '.[dev]'
ruff check src tests
mypy src
pytest
python -m buildThe verified integration contracts require FastMCP 2.14.7+ and pycti 6.9.29+, within the major-version bounds declared in the package. Automated tests use simulated services and the runtime smoke test exercises the real pycti JSON adapter/splitter with a simulated object importer. Validate real OpenCTI merges, provider credentials and LLM behavior separately before production use.
For a release, update the project version, validate and commit the intended
changes, then create and push a matching semantic tag such as v0.2.0.
Tags matching v* trigger package validation/build and a GitHub Release with
wheel and source archive. The container workflow separately publishes validated
images to GHCR.
Optional Python distribution uses the manual Publish Python distribution
workflow with PyPI Trusted Publishing (OIDC), minimal permissions, build
provenance attestations, tag/version consistency checks and secret-hygiene
checks on the built artifacts. Rehearse on TestPyPI first (the default
target); promoting to PyPI requires the protected pypi environment and
explicit authorization. The base install is fully functional — all
integrations share the same requests-based core, so no optional extras are
required at runtime beyond dev for development.
Developed and maintained by Tomás Vinuesa Lago. Contributions, suggestions and feedback are welcome.