Turn raw chat exports (Discord / Telegram / Lark / any JSON) into structured community-intelligence reports — passively, without running a survey. Combines NLP heuristics, local embeddings (BGE-M3 + FAISS), structured LLM fact extraction, deterministic analytics, and multi-model synthesis into one reproducible pipeline. User IDs are SHA-256-hashed by default; no raw quotes leak into outputs.
Multi-language and multi-region out of the box — pick your target market with --target-language / --region, or add your own LanguageProfile + RegionProfile. Ships with 13 languages and 7 regions as fully-worked examples.
Option A — Scaffold a new project:
pip install parallax
parallax init my-project --platform discord
cd my-project
# place your export in data/, then:
parallax-analyze --input data/export.json --platform discord --out out/ -vOption B — From source:
git clone https://github.com/lunalunaa/parallax.git
cd parallax
# Install (pick one):
uv sync --extra dev # uv (recommended)
# — or —
python3 -m venv .venv && source .venv/bin/activate && pip install -e ".[dev]"
# Run:
parallax-analyze --input ~/Downloads/discord_export.json --platform discord \
--out ./out -v
# Optional: LLM topic tagging + cross-tabulations
parallax-topics --input-chat ~/Downloads/discord_export.json --platform discord \
--out ./out/topics.json
parallax-crosstabs --users-json ./out/users.json --out ./out/crosstabs.jsonFirst install is slow (~1–2 GB for
sentence-transformers/faiss-cpu/torch). This is expected.
Editor / Jupyter kernel setup (Zed, VS Code, JupyterLab)
This repo ships a pyrightconfig.json pointing at .venv. If your editor shows "missing import" errors or Kernel error: No module named ipykernel_launcher, it's using the wrong interpreter. Fix:
pip install -e ".[dev]" # or: uv pip install -e ".[dev]"
python -m ipykernel install --user --name=parallax --display-name="parallax (.venv)"Then select "parallax (.venv)" in your editor's kernel/interpreter picker. Verify with basedpyright . (should report 0 errors).
chat_export.jsonl
│
┌──────────┼──────────┐
│ │ │
Stream B Stream C Stream D
Semantic Structured Deterministic
retrieval fact extract analytics
BGE-M3 + LLM per (no LLM)
FAISS ~100-msg
chunk
│ │ │
└──────────┼──────────┘
│
narrative_synthesis.py
(one large-context LLM call)
│
build_final_report.py
│
final_report.md
Stream A (topics.py) runs first as broad LLM topic classification. See docs/PIPELINE.md for the full writeup, per-stream I/O, and a real cost breakdown (~110 min, ~$3.65 for 28K messages).
The remaining stream scripts are one-shot modules (no main()):
python -m parallax.streams.semantic_retrieval
python -m parallax.streams.fact_extraction
python -m parallax.streams.deterministic_analytics
python -m parallax.streams.narrative_synthesis
python -m parallax.streams.build_final_report
python -m parallax.streams.post_analysis--target-language |
Language | Detection | --region |
Region |
|---|---|---|---|---|
none (default) |
— | Disables language split | global (default) |
Global/English-default |
zh |
Chinese | CJK script ratio | cn |
Greater China |
ja |
Japanese | Hiragana/Katakana + CJK | jp |
Japan |
ko |
Korean | Hangul script ratio | kr |
Korea |
ru |
Russian | Cyrillic script ratio | ru |
Russia/CIS |
ar |
Arabic | Arabic script ratio | latam |
Latin America |
he |
Hebrew | Hebrew script ratio | mena |
Middle East/N. Africa |
th |
Thai | Thai script ratio | ||
vi |
Vietnamese | Vietnamese-diacritic | ||
es fr de pt id |
Latin-script | Stopword-frequency (lower precision) |
Add your own market by adding one entry to src/parallax/core/languages.py — no code changes elsewhere.
All stream scripts read from environment variables — no hardcoded paths to edit:
| Variable | Purpose | Default |
|---|---|---|
CHAT_JSONL |
Path to raw NDJSON export | ./data/pages.jsonl |
OUT_DIR |
Output directory for all streams | ./out |
SALT_FILE |
User-ID hashing salt | ./user_hash_salt.key |
TARGET_LANGUAGE |
Which query set + LLM prompt language | en |
TS_UTC_OFFSET_HOURS |
UTC offset for tz-less display timestamps | 0 (UTC) |
GROUND_TRUTH_HUMANS / GROUND_TRUTH_BOTS |
Manually-verified live membership | 0 |
LLM_PROVIDER / LLM_MODEL / LLM_MODEL_PRO |
LLM CLI provider/model | example defaults |
COMMUNITY_NAME / CORPUS_DESCRIPTION / GROUND_TRUTH_SUMMARY |
Context for synthesis prompt | generic |
REPORT_TITLE / REPORT_SUBJECT / REPORT_PERIOD |
Report front-matter | generic |
See docs/PIPELINE.md for the full list and usage.
src/parallax/
├── core/
│ ├── analyze.py # main pipeline (parallax-analyze)
│ ├── languages.py # language/region profile registry
│ ├── keywords.py # keyword dictionaries — tune for your product
│ ├── crosstabs.py # cross-tabulation helper (parallax-crosstabs)
│ ├── config.py # YAML config loader (fact_schema, queries, etc.)
│ ├── diff.py # stats comparison tool (parallax-diff)
│ ├── init.py # project scaffolder (parallax-init)
│ └── state.py # SQLite state store for --incremental
├── config/ # YAML config files (override via PARALLAX_CONFIG_DIR)
│ ├── fact_schema.yaml # Stream C extraction schema
│ ├── queries.yaml # Stream B retrieval queries by language
│ ├── brand_patterns.yaml # post_analysis brand audit patterns
│ ├── url_domains.yaml # URL classification domain lists
│ └── canonical_schema.json # JSON Schema for --platform canonical
├── streams/
│ ├── topics.py # Stream A: LLM topic-tagging (parallax-topics)
│ ├── semantic_retrieval.py # Stream B: embeddings + FAISS semantic search
│ ├── fact_extraction.py # Stream C: structured LLM fact extraction (tolerant JSON)
│ ├── deterministic_analytics.py # Stream D: ground-truth-anchored analytics
│ ├── narrative_synthesis.py # feeds all streams into one LLM synthesis call
│ ├── build_final_report.py # assembles methodology + findings + recommendations
│ └── post_analysis.py # brand audit, cost drill, CSV exports
└── templates/
└── report-template.md # report skeleton with {{stats.xxx}} placeholders
| Platform | --platform |
How to export |
|---|---|---|
| Discord | discord |
DiscordChatExporter → JSON |
| Telegram | telegram |
Telegram Desktop → group → Export chat history → JSON |
| Lark/Feishu | lark |
@larksuite/cli im +chat-messages-list |
| Slack | slack |
Slack workspace → Settings → Export workspace data (zip or directory of JSONs) |
| CSV (generic) | csv |
Any tool that produces a CSV with a content column (plus optional author, timestamp, channel) |
| Anything else | canonical |
Match the canonical JSON schema |
Re-running parallax-analyze on an updated export normally processes everything from scratch. With --incremental, only new messages (by message ID) are processed, and results are merged into the existing stats.json:
# First run (full):
parallax-analyze --input export.json --platform discord --out ./out
# Subsequent runs (incremental):
parallax-analyze --input export_updated.json --platform discord --out ./out --incremental -vA SQLite state store (parallax_state.db in the output dir) tracks which message IDs have been seen. Counter-type stats (providers, friction, features, etc.) are summed; user counts take the max (conservative, since the same user may appear in both runs).
Use parallax-diff to compare two stats.json files:
parallax-diff --old ./run1/stats.json --new ./run2/stats.json
# or JSON output:
parallax-diff --old ./run1/stats.json --new ./run2/stats.json --jsonShows KPI deltas (messages, users, questions, friction, answered rate), per-counter changes with new/gone/changed status, and percentage deltas.
Generate a self-contained interactive HTML dashboard from stats.json:
# Single run with drill-down (click any bar to see matching messages):
python -m parallax.streams.generate_dashboard \
--stats out/stats.json --users out/users.json --chat data/export.json \
--out out/dashboard.html
# Trend view (compare multiple runs):
python -m parallax.streams.generate_dashboard \
--stats run1/stats.json run2/stats.json run3/stats.json \
--out out/dashboard.htmlFeatures:
- KPI cards with delta indicators (↑ 12% vs prev) when comparing runs
- Bar lists for providers, messaging platforms, friction, features, language, install paths, shadow communities, URL categories, location proxy, competitors
- Drill-down — click any bar to expand a panel showing matching messages (timestamp, sender, content)
- Trend view — bar chart comparing total messages across runs, with per-bar deltas
- Filter controls — run selector dropdown, cohort toggle (All / Target-lang), view toggle (Current / Trend)
- Self-contained — no server, no build step, no external JS dependencies. Works fully offline.
See examples/quickstart.ipynb for a full walkthrough.
parallax-analyze supports two output formats for stats:
# JSON (default — writes stats.json):
parallax-analyze --input export.json --platform discord --out ./out
# CSV (writes stats.csv — flat key-value layout for spreadsheets/BI tools):
parallax-analyze --input export.json --platform discord --out ./out --format csvCSV layout: category,key,value — one row per metric (e.g. metadata,total_messages,226).
- User IDs are SHA-256-hashed with a local salt before reaching any output file
--keep-namesis OFF by default; display names are<redacted>- Salt is auto-generated with
secrets.token_hex(32), chmod0600, gitignored
MIT — see LICENSE.