diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 0000000..c18da2e --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,27 @@ +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +# Required +version: 2 + +# Set the OS, Python version, and other tools you might need +build: + os: ubuntu-24.04 + tools: + python: "3.13" + +# Build documentation in the "docs/" directory with Sphinx +sphinx: + configuration: docs/source/conf.py + + +format: + - html + - pdf +# Optionally, but recommended, +# declare the Python requirements required to build your documentation +# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html +python: + install: + - requirements: docs/source/requirements.txt + diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md deleted file mode 100644 index a85ea59..0000000 --- a/docs/ARCHITECTURE.md +++ /dev/null @@ -1,138 +0,0 @@ -# Architecture - -*multiplex* is a modular framework for prototyping LLM-based mutation testing. -One run mutates **one method/function** in a target project and evaluates the -generated mutants against that project's test suite. The source language is set -by `project.language` (`java` — the default — or `python`); see the language -registry below. - -## Pipeline - -Orchestrated top-to-bottom by `multiplex/__main__.py`: - -``` -config.yml - │ - ▼ -1. Load YAML config; resolve the approach's system_prompts and the language - (languages.get_language(project.language), default "java") - │ - ▼ -2. Back up target source file to .orig - Delete and recreate /output/ ← destroys previous results - │ - ▼ -3. util/extract_method.py (tree-sitter, grammar from the LanguageSpec) - Find method/function by name + start line - → writes output/original_method. (.java / .py) - → returns (start_byte, end_byte) offsets into the source file - │ - ▼ -4. approach//controller.main(model, output_path, prompts, language) - One or more LLM calls (via model.Model / LiteLLM) - → writes mutant files to output/-mutants/mutant_N. - │ - ▼ -5. execute/{maven,defects4j,pytest_runner}.run_mutants(...) - For each mutant file: - a. restore source file from .orig backup - b. util/rewrite_method.py splices mutant text into the file - at the saved byte offsets - c. checks/compilable.py — tree-sitter parse-error scan - checks/syntactic_equivalence.py — AST equality vs original (ignores comments) - d. run the project's tests (Defects4J backend only; see Status below) - → writes output/-mutants/mutant_summary.csv (Defects4J) - │ - ▼ -6. (optional, --marv flag) util/marv.py output_marv(output_path, approach) - Reads original_method.java + mutant files + mutant_summary.csv - → writes output/marv.json (Marv mutations schema; pure Python, no marv binary) - │ - ▼ -7. Restore original source file from .orig backup -``` - -Key mechanism: the method's **byte offsets** captured in step 3 are reused in -step 5b to splice each mutant into a pristine copy of the file. Anything that -changes the file between those steps invalidates the offsets. - -## Mutant-generation approaches (`multiplex/approach/`) - -Each approach is a package with a `controller.py` exposing -`main(model, output_dir, prompts, language)`. Dispatch is an if/elif chain in -`__main__.py` keyed on `mutation.approach`. - -| Approach | Strategy | LLM calls | -|----------|----------|-----------| -| `basic` | Single system prompt, ask for a mutant 10 times | 10 | -| `hazop` | Chain: describe method line-by-line → mutate descriptions using HAZOP guidewords (NO/MORE/LESS/...) → implement each deviation | 2 + 1 per deviation | -| `stpa` | Chain: describe control flow as GraphViz DOT → identify Unsafe Control Actions (UCAs) → implement each UCA | 2 + 1 per UCA | -| `mutahunter` | Single call with AST + line-numbered source; LLM returns YAML of line-level mutations spliced in locally (prompts not bundled — licensing) | 1 | -| `llmorpheus` | tree-sitter query finds mutation sites (conditions, loop headers, call args), each replaced by ``; LLM proposes 3 replacements per site | 1 per placeholder | - -All five approaches are language-aware via the `language` argument (file -extension, code fence, prompt noun). llmorpheus additionally selects its -mutation-site query per language from `MUTATION_QUERIES` in -`approach/llmorpheus/placeholders.py`. - -Intermediate artifacts are files in `output/` — approaches communicate between -their own steps via files, not in-memory state (see artifact table below). - -## Execution backends (`multiplex/execute/`) - -Selected by `project.runtool`: - -- `d4j` → `defects4j.py` — the complete backend. Baselines the unmutated - project first (aborts if its tests fail), times the baseline run, then runs - each compilable mutant's tests with a timeout of 5× baseline. A surviving - mutant is one whose test run reports `Failing tests: 0`. Requires the - `defects4j` CLI and a `JDK_11` env var (see DEVELOPMENT.md). -- `mvn` → `maven.py` — self-contained backend for plain Maven projects. Runs - `mvn -f clean test`; a mutant survives if the build passes - (`BUILD SUCCESS`). Baselines the original first (aborts if its tests fail), - then per mutant does the same equivalence → rewrite → compilable → test → - summary flow as the Defects4J backend. Used by the runnable Java example under - `examples/` (see DEVELOPMENT.md § Example). -- `pytest` → `pytest_runner.py` — self-contained backend for Python projects. - Runs `sys.executable -m pytest -q ` (pytest under multiplex's own - interpreter); a mutant survives if pytest exits 0 (all tests pass). Same - baseline → per-mutant flow as the Maven backend. Drives the runnable Python - example (`examples/config-python.yml`). Named `pytest_runner` so it does not - shadow the installed `pytest` package. - -## Language registry (`multiplex/languages/`) - -`get_language(project.language)` returns a `LanguageSpec` (grammar, definition -node types, comment node types, extension, code-fence tag, prompt noun, -mutahunter label). It is resolved once in `__main__.py` and threaded into -`extract_method`, every approach's `main`, the checks, and the execution -backend — so the pipeline is language-agnostic and adding a language is a -registry entry plus prompts and an example (see docs/EXTENDING.md § Add a -language). `project.language` defaults to `java`. - -## Output artifacts - -Everything lands under `/output/` (wiped at the start of each run): - -| Artifact | Written by | -|----------|-----------| -| `original_method.` | extract_method (step 3); read by every approach (`.java`/`.py`) | -| `hazop-descriptions.txt`, `hazop-mutated-descriptions.txt` | hazop chain steps | -| `control_diagram.txt`, `ucas.csv` | stpa chain steps | -| `placeholders/{N_placeholder., N_orig., placeholders.json}` | llmorpheus site finder | -| `-mutants/mutant_N.` | every approach's final step | -| `-mutants/mutant_summary.csv` | maven/defects4j/pytest backends; columns `MUTANT, EQUIVALENCE, COMPILABLE, SURVIVES` | -| `-test/_test.txt` | defects4j backend; per-mutant test output | -| `marv.json` | util/marv.py (`--marv` flag only); Marv-schema view of every mutant | - -## LLM access (`multiplex/model.py`) - -All LLM traffic goes through `Model.make_request(messages)`, a thin wrapper -over LiteLLM `completion()`. Model/endpoint come from config; the API key is -read from the environment variable *named* by `llm.token_env_var` (never stored -in config). Azure endpoints get `AZURE_AI_API_BASE`/`AZURE_AI_API_KEY` set -specially. To support another provider, replace or extend `model.py`. - -LLM responses are treated as fenced code: approaches strip a leading -```` ``` ```` fence (the tag is `language.fence`, e.g. `java`/`python`) and -keep everything before the next ```` ``` ```` fence. diff --git a/docs/CONFIG.md b/docs/CONFIG.md deleted file mode 100644 index a15633e..0000000 --- a/docs/CONFIG.md +++ /dev/null @@ -1,61 +0,0 @@ -# Configuration Reference - -*multiplex* takes a single YAML config file: -`uv run ./multiplex ./path/to/config.yml`. -Templates: [`examples/config-java.yml`](../examples/config-java.yml), -[`examples/config-python.yml`](../examples/config-python.yml). - -Only the keys the selected `mutation.approach` needs are required. `project`, -`mutation`, and `llm` keys are always required; `system_prompts` keys are -required per-approach (see that section). Missing required keys fail fast at -startup with an actionable `SystemExit`, before any output is wiped. - -## `project` - -| Key | Type | Meaning | -|-----|------|---------| -| `language` | `java` \| `python` | Source language of the file under test. **Optional; defaults to `java`** (existing configs need no change). Selects the tree-sitter grammar, the definition node types extracted, the artifact/mutant file extension, and the code-fence/prompt wording. An unknown value fails fast at startup with a `SystemExit`. | -| `projectroot` | path | Root of the project under test. `output/` is created (and **wiped every run**) inside it. | -| `filename` | path | The source file containing the method/function under test (`.java` or `.py`). Backed up to `.orig` during the run. | -| `method` | string | Name of the method/constructor (Java) or function (Python) to mutate. | -| `line` | int | 1-based line number of the method/function **name identifier** in `filename` (not annotations/decorators above it). Both `method` and `line` must match for extraction to succeed; disambiguates overloads. | -| `runtool` | `mvn` \| `d4j` \| `pytest` | Execution backend. `mvn` runs a plain Maven project (`mvn clean test`); `d4j` targets a Defects4J checkout (needs the `defects4j` CLI + `JDK_11`); `pytest` runs pytest under multiplex's own interpreter (`sys.executable -m pytest `) — a mutant survives if pytest exits 0. `mvn` drives the Java `examples/` setup and `pytest` the Python one. | - -## `mutation` - -| Key | Type | Meaning | -|-----|------|---------| -| `approach` | `basic` \| `hazop` \| `stpa` \| `mutahunter` \| `llmorpheus` | Mutant-generation approach. An unknown value fails fast at startup with a `SystemExit` listing the valid approaches. | - -## `llm` - -| Key | Type | Meaning | -|-----|------|---------| -| `model` | string | LiteLLM model string, e.g. `ollama/gpt-oss:20b`. | -| `endpoint` | URL | LLM API endpoint, e.g. `http://127.0.0.1:11434`. If the string contains `azure`, Azure env vars are set automatically. | -| `token_env_var` | string or empty | **Name** of the environment variable holding the API key — never the key itself. Leave empty for keyless local endpoints (e.g. Ollama). The named var must exist or startup fails with KeyError. | - -## `system_prompts` - -Multi-line YAML strings (typically `|` blocks). Only the keys used by the -selected `mutation.approach` are required — prompts for other approaches may be -omitted. The approach→required-keys mapping is `APPROACH_PROMPT_KEYS` in -`multiplex/prompts.py`; a missing or empty required key raises a `SystemExit` -naming it. User prompts are constructed in code; only system prompts live in -config. - -| Key | Used by | -|-----|---------| -| `basic_generate_mutants` | basic | -| `hazop_describe_process` | hazop step 1 (line-by-line description) | -| `hazop_identify_deviations` | hazop step 2 (guideword mutation; must instruct CSV output `number, original_rule, GUIDEWORD, changed_rule`) | -| `hazop_implement_deviations` | hazop step 3 (code generation) | -| `stpa_describe_control_flow` | stpa step 1 (GraphViz DOT control diagram) | -| `stpa_identify_ucas` | stpa step 2 (numbered UCA list) | -| `stpa_implement_ucas` | stpa step 3 (code generation) | -| `mutahunter_generate_mutants` | mutahunter — **not bundled**; users must paste Mutahunter's own prompt text (licensing restriction). Response must be YAML with `mutants: [{mutated_code, line_number}]`. | -| `llmorpheus_system` | llmorpheus (per-placeholder replacement generation) | - -Output-format expectations matter: downstream parsing is brittle string/CSV/ -YAML/regex handling (see MODULE_REFERENCE.md), so prompts must pin the exact -output format the next step expects. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md deleted file mode 100644 index 3c0623e..0000000 --- a/docs/DEVELOPMENT.md +++ /dev/null @@ -1,116 +0,0 @@ -# Development Guide - -## Setup and commands - -Dependency management is [uv](https://github.com/astral-sh/uv); Python >= 3.13. - -```bash -uv sync -p 3.13 # install deps incl. dev group (pytest, ruff, pylint) -uv run pytest tests # all tests — run from the REPO ROOT -uv run pytest tests/checks/test_compiles.py # one file -uv run pytest tests/checks/test_compiles.py::test_name # one test -uv run ruff check . # lint (CI uses --output-format=github) -uv run ./multiplex ./path/to/config.yml # run the tool -``` - -- **Pin Python 3.13** (as CI does): `requires-python` allows >= 3.13, but - litellm's `tiktoken` dependency fails to build on 3.14. -- Tests must run from the repo root: they reference `tests/resources/...` - relatively. -- Running the tool needs a reachable LLM endpoint and a target project (Java or - Python, per `project.language`). `mvn` runs need Maven + a JDK; `pytest` runs - need pytest for the active interpreter; `d4j` runs additionally need the - `defects4j` CLI (expected under `./defects4j/framework/bin`) and a `JDK_11` env - var pointing at a JDK 11 home. - -CI (GitHub Actions, on push/PR to `main`): `test.yml` runs -`uv run pytest tests` on Python 3.13; `ruff.yml` runs `uv run ruff check`. - -## Example - -Self-contained end-to-end examples live under `examples/`: - -```bash -uv run ./multiplex ./examples/config-java.yml # Java (from the repo root) -uv run ./multiplex ./examples/config-python.yml # Python -``` - -- Target: `examples/project/java-example/` — a tiny Maven project with one method - (`com.example.Classifier.classify`) and JUnit tests pinning its behavior. -- `examples/config-java.yml` uses the `basic` approach and the `mvn` runtool. -- Prerequisites: `mvn` + a JDK 11+ on PATH (the project targets Java 11; if - `mvn` picks up an older JDK via `JAVA_HOME` the compile fails with "release - version 11 not supported"), and a running Ollama serving the model - named in `llm.model` (default `gpt-oss:20b`; point it at any model you have, - e.g. `ollama pull gpt-oss:20b`). Any LiteLLM-supported endpoint works if you - edit `llm`. -- Output lands in `examples/project/java-example/output/basic-mutants/` - (`mutant_N.java` plus `mutant_summary.csv`). Run artifacts (`output/`, - `*.orig`, Maven `target/`) are git-ignored. -- The `basic` approach makes 10 LLM calls and the `mvn` backend runs the test - suite once per compilable mutant, so a full run takes a few minutes (longer - on a slow/local model). -- Python counterpart: `examples/config-python.yml` (`project.language: python`, - `pytest` runtool) mutates `classify` in - `examples/project/python-example/classifier.py` and evaluates each mutant with - pytest (run under multiplex's own interpreter). Same output layout - (`output/basic-mutants/mutant_N.py` + - `mutant_summary.csv`). Needs pytest (already provided by `uv run`) and the same - Ollama/LLM endpoint. - -## Import convention (critical) - -The tool is executed as a directory (`uv run ./multiplex`), which puts -`multiplex/` itself on `sys.path`: - -- **Inside `multiplex/`**: flat imports, no package prefix — - `from model import Model`, `from util.io import write_to_file`, - `import approach.basic.controller as basic`. -- **Inside `tests/`**: package-prefixed imports — - `from multiplex.checks.syntactic_equivalence import check_mutant_equivalent`. - -Using the wrong style in either place breaks at import time. This also means -`multiplex/` modules are not importable as `multiplex.x` **and** `x` in the -same process; keep the two worlds separate. - -## Conventions - -- tree-sitter is the universal source-analysis tool (extraction, compilability, - equivalence, placeholder finding); the grammar comes from the run's - `languages.LanguageSpec`. Versions are pinned (`tree-sitter==0.23.2`, - `tree-sitter-java==0.23.5`, `tree-sitter-python` 0.23.x); the code depends on - that API — do not bump casually. -- Approaches communicate between their own steps via files in `output/`, not - in-memory state; artifact names are prefixed with the approach name. -- LLM code responses are unfenced by stripping a leading ```` ``` ```` - fence (the tag is `language.fence`, e.g. `java`/`python`) and truncating at the - next ```` ``` ```` fence (`removeprefix` + `split`, see EXTENDING.md for the - exact two lines). -- Mutant files are named `mutant_.java` in `output/-mutants/`. -- Status/progress reporting is `print()`-based throughout (no logging config, - one `logging.warning` in extract_method). - -## Known issues and gotchas (verified 2026-07) - -Bugs — fix or work around, don't replicate: - -_No open bugs currently tracked._ - -Behavioral gotchas — by design (or at least current design), be aware: - -- `/output/` is **deleted without prompting** at every run start - (the confirmation prompt in `__main__.py` is commented out, answer - hard-coded to `"y"`). -- The target source file is mutated **in place** during execution; it is - backed up to `.orig` and restored at run end, but a crash mid-run - can leave a mutant in the working tree (the `.orig` file remains for manual - restore). -- "Compilable" means *parses without tree-sitter ERROR nodes* — no compiler - runs; type errors and unresolved symbols count as compilable. -- An invalid `mutation.approach` value, or a missing/empty required - `system_prompts` key for the chosen approach, raises `SystemExit` at startup - (before the output wipe). Only the selected approach's prompt keys are - required — see `multiplex/prompts.py`. -- Mutahunter prompts are not bundled (licensing); the user pastes them into - the config. Running the `mutahunter` approach without them fails fast with a - clear message. diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md deleted file mode 100644 index 87755bd..0000000 --- a/docs/EXTENDING.md +++ /dev/null @@ -1,139 +0,0 @@ -# Extending multiplex - -Checklists for adding modules. Follow the import convention: inside -`multiplex/`, import siblings **without** the `multiplex.` prefix -(DEVELOPMENT.md § Import convention). - -## Add a mutant-generation approach - -Use `approach/basic/` as the single-prompt template or `approach/hazop/` as -the multi-prompt-chain template. - -1. **Create the package** `multiplex/approach//` with `__init__.py` and - `controller.py` exposing: - ```python - def main(model, output_dir, prompts, language): - ... - ``` - - `model` is a `model.Model`; call `model.make_request(messages)` with - OpenAI-style message dicts. - - `language` is a `languages.LanguageSpec` (resolved in `__main__.py`); use it - for anything language-specific — `language.extension`, `language.fence`, - `language.noun` — so your approach works for every registered language. - - Read the method under test via - `from approach.util import get_method_under_test` — call it as - `get_method_under_test(output_dir, language)`. - - Pass intermediate results between steps as files in `output_dir` - (convention: prefix them with your approach name). - - **Required output**: write final mutants to - `Path(output_dir, "-mutants/") / f"mutant_{count}{language.extension}"`. - Each file must contain the complete replacement method body (it is spliced - verbatim over the original method's byte range). Strip LLM code fences: - ```python - mutant = mutant.removeprefix("```" + language.fence) - mutant = mutant.split("```", 1)[0] - ``` -2. **Register the prompt key(s)**: add a `"": [...]` entry to - `APPROACH_PROMPT_KEYS` in `multiplex/prompts.py` listing the - `system_prompts` keys your approach reads. Only the selected approach's keys - are required at runtime, so users running other approaches need not define - yours (and vice versa). Document the keys in `docs/CONFIG.md`; adding them to - `examples/config-java.yml` (a commented stub is fine) is optional but helpful. -3. **Register the dispatch branch**: add an `elif config['mutation']['approach'] - == "":` branch in `multiplex/__main__.py` calling - `.main(model, output_path, prompts, language)`, plus the corresponding - `import approach..controller as `. -4. The `-mutants` directory name must equal the `mutation.approach` config - value — the execution backends reconstruct the path as - `output/-mutants/`. - -## Add an execution/evaluation backend - -Model on `multiplex/execute/defects4j.py` or `multiplex/execute/maven.py` -(both follow the same flow; `maven.py` is the simpler, self-contained one). - -1. Create `multiplex/execute/.py` exposing: - ```python - def run_mutants(project_root, original_file, output_path, - method_start_byte, method_end_byte, duplicate, approach, - language): - ``` - `language` is a `languages.LanguageSpec` — use it for the artifact path and - the checks below. -2. The expected loop, per mutant file in `output/-mutants/`: - - restore the pristine source: `shutil.copy2(duplicate, original_file)` - (guard on `duplicate.exists()`); - - `rewrite_method(original_file, method_start_byte, method_end_byte, - mutant_path)` — note: **4 arguments**; - - `check_mutant_equivalent(mutant_path, - language.original_method_path(output_path), language)` and - `check_mutant_compilable(original_file, language)` from `checks/`; - - if compilable, run the project's tests with your build tool and decide - survived/killed; - - collect rows and finish with `write_mutant_summary(mutants_dir, rows)` - (header row: `["MUTANT", "EQUIVALENCE", "COMPILABLE", "SURVIVES"]`). -3. Register it in `multiplex/__main__.py` under a new `project.runtool` value - (pass `language` through to `run_mutants`). -4. Useful pattern from defects4j: run the unmutated project first as a - baseline (abort if it fails), and use a multiple of its wall-clock time as - the per-mutant test timeout to catch infinite-loop mutants. - -## Add a language - -The pipeline is language-agnostic: the parser, checks, approaches and execution -backends all take a `languages.LanguageSpec` resolved once in `__main__.py` from -the `project.language` config value. Java and Python ship in -`multiplex/languages/__init__.py`. - -To add a language: - -1. **Add the tree-sitter grammar** dependency (e.g. `tree-sitter-`) to - `pyproject.toml` and add a `LanguageSpec` entry to `_REGISTRY` in - `multiplex/languages/__init__.py`: - ```python - "": LanguageSpec( - name="", - extension=".", - ts_language=Language(ts_.language()), - def_node_types=frozenset({...}), # nodes that denote a definition - comment_node_types=frozenset({...}), # ignored by the equivalence check - fence="", # ``` fence tag - noun=" function", # prompt wording - label="", # mutahunter language string - ), - ``` - `def_node_types` are the node types `extract_method` matches (their child - `identifier` is compared to `project.method`); `comment_node_types` are - dropped when comparing ASTs for syntactic equivalence. Inspect a grammar with - a few lines of tree-sitter to find the right node types. -2. **Provide a prompt set** for the language in the config's `system_prompts` - (see `examples/config-python.yml`). Prompt *keys* are shared across languages; - only the wording differs. mutahunter prompts remain user-supplied. -3. **Add a mutation-site query for llmorpheus** if you want that approach: - add a `"": ` entry to `MUTATION_QUERIES` in - `multiplex/approach/llmorpheus/placeholders.py` (keyed by `language.name`). - The other four approaches work with no per-approach changes. -4. **Pick an execution backend**: reuse an existing `project.runtool` - (`mvn`/`d4j`/`pytest`) or add one (see the section above). `pytest_runner.py` - is the template for a script/test-command backend. -5. **Add an end-to-end example** under `examples/` (a small project + a config - with `project.language: `), mirroring `examples/project/python-example/`. - -Existing configs are unaffected: `project.language` defaults to `java`. - -## Add / change the LLM provider - -`multiplex/model.py` wraps LiteLLM, so most providers work by editing only the -config (`llm.model`, `llm.endpoint`, `llm.token_env_var`). If a provider needs -extra setup (custom env vars, headers), extend `Model.__init__` — see the -existing Azure special-case. Keep the `make_request(messages) -> str` interface -unchanged; every approach depends on it. - -## Testing your module - -Put tests under `tests//test_.py` with fixture files in -`tests/resources/`. Import production code **with** the package prefix -(`from multiplex.checks.compilable import ...`). Run from the repo root: -`uv run pytest tests`. LLM calls are not mocked anywhere yet — keep pure logic -(parsing, splicing, checks) in functions separate from `model.make_request` -call sites so it is testable without an LLM. diff --git a/docs/MODULE_REFERENCE.md b/docs/MODULE_REFERENCE.md deleted file mode 100644 index ad2868e..0000000 --- a/docs/MODULE_REFERENCE.md +++ /dev/null @@ -1,269 +0,0 @@ -# Module Reference - -Function-level reference for every module. Paths are relative to `multiplex/`. -Signatures are exact; use this instead of opening source files. - -Note on imports: inside `multiplex/` modules import each other without the -package prefix (e.g. `from util.io import write_to_file`). See DEVELOPMENT.md. - -Most pipeline functions take a `language` argument — a `languages.LanguageSpec` -resolved once in `__main__.py` from `project.language` (default `java`). - -## languages/__init__.py - -```python -@dataclass(frozen=True) -class LanguageSpec: - name: str; extension: str; ts_language: Language - def_node_types: frozenset; comment_node_types: frozenset - fence: str; noun: str; label: str - def original_method_path(self, output_dir) -> Path # output_dir/original_method - -get_language(name=None) -> LanguageSpec # default "java"; SystemExit on unknown -DEFAULT_LANGUAGE = "java" -``` - -- `_REGISTRY` holds the `java` and `python` specs. `name` matching is - case-insensitive. Raises `SystemExit` (actionable, lists valid languages) for - an unknown name — same fail-fast contract as `prompts.resolve_prompts`. - -## model.py - -```python -class Model: - def __init__(self, model=None, endpoint=None, api_key_var=None) - def current_model(self) -> str - def make_request(self, messages) -> str # returns response.choices[0].message.content -``` - -- `api_key_var` is the **name** of an env var; its value is read from - `os.environ` (KeyError if the named var is unset). -- If `"azure" in endpoint`, sets `AZURE_AI_API_BASE` and `AZURE_AI_API_KEY`. -- `messages` is a LiteLLM/OpenAI-style list of `{"role": ..., "content": ...}`. - -## prompts.py - -```python -APPROACH_PROMPT_KEYS: dict[str, list[str]] # approach name -> required system_prompts keys -resolve_prompts(config, approach) -> dict # {key: prompt} for the approach's keys -``` - -- `resolve_prompts` reads only the selected approach's keys from - `config["system_prompts"]`. Raises `SystemExit` (actionable message) if - `approach` is unknown or any required key is missing/empty. Pure function, no - I/O — called by `__main__.py` before any destructive step. Importable in tests - as `from multiplex.prompts import ...` (no sibling imports, so it works under - both the flat runtime import and the package-prefixed test import). - -## util/io.py - -```python -write_to_file(output_file_path, output) # plain text write (mode "w") -write_ucas(ucas_output_path, ucas_output) # writes str line-by-line -write_mutant_summary(mutants_dir, mutants) # writes mutants_dir/mutant_summary.csv from list-of-rows -read_input_to_str(input_file_path) -> str -read_ucas(input_file_path) -> (list[str], int) # (lines, line count) -read_hazop_mutations(input_file_path) -> (list[str], int) # 4th CSV column of each line -reset_source_code(duplicate_file_path, filename) -``` - -- `reset_source_code`: if `.orig` exists, moves it over `filename` - (restore); then always re-copies `filename` → `.orig`. Called at run start - and end by `__main__.py`. - -## util/extract_method.py - -```python -extract_method_from_file(file_path, method_name, output_dir, start_line, language) - -> (start_byte, end_byte) | None -``` - -- tree-sitter walk (grammar from `language`) for a node in - `language.def_node_types` whose identifier text == `method_name` **and** whose - identifier is on line `start_line` (1-based). Both must match — `start_line` is - the line of the method/function *name*, not of annotations/decorators above it. -- Writes the method source to `language.original_method_path(output_dir)` - (`output_dir/original_method.`). -- Returns `None` if not found (logs a warning). The caller in `__main__.py` - checks for `None` and exits with a clear `SystemExit` naming - `project.method`/`project.line`. - -## util/rewrite_method.py - -```python -rewrite_method(orig_path, start_byte, end_byte, mutant_file_path) -``` - -- Reads the pristine file from `orig_path + ".orig"` (the backup), splices the - mutant file's stripped text over bytes `[start_byte:end_byte)`, writes the - result to `orig_path`. Byte offsets come from `extract_method_from_file`. - -## util/parser.py - -```python -parse_output(input) -> dict | None # strips ```yaml fences, yaml.safe_load; None on parse error -add_mutant_to_method(numbered_src, mutant, line_number) -> str -``` - -- `add_mutant_to_method` takes source whose lines are prefixed `"N: "`, - replaces line `line_number` with `mutant` (preserving indentation), and - returns un-numbered source. Used by the mutahunter approach. - -## util/marv.py - -```python -output_marv(output_dir, approach) # writes output_dir/marv.json (Marv schema); no-op unless --marv -``` - -- Called from `__main__.py` only when the `--marv` flag is set, after execution. - Reads `output_dir/original_method.java` and every - `-mutants/mutant_N.java`; raises `FileNotFoundError` if the original - method file or the mutants dir is missing. Pure Python — does **not** invoke - the `marv` binary. -- Per-mutant `Status` comes from `-mutants/mutant_summary.csv`: - `equivalent` → `IGNORED`, non-compilable → `CRASHED`, `survives` → `SURVIVED`, - otherwise `KILLED`; `PENDING` when no summary row exists. -- `Operation` is `REPLACE_METHOD`, except under the `hazop` approach where it is - the guideword from `hazop-mutated-descriptions.txt` (zipped to mutants in - sorted order). -- Each mutation's `Start`/`End` are the line/char bounds of the code that - **differs** between the original method and that mutant, located with `difflib` - (`_file_span`). The enclosing `MutantRegion` spans the whole - `original_method.java`. -- Emits JSON mapping `"original_method.java"` → one region → its mutations, - matching Marv's mutations schema. - -## util/marv_model.py - -Dataclasses mirroring [Marv's mutations schema](https://github.com/SecretSheppy/marv/blob/main/api/marv-mutations-schema.json), -serialized to JSON by `util/marv.py`: - -- `Status` — enum: `KILLED`, `SURVIVED`, `CRASHED`, `TIMEOUT`, `NO_COVERAGE`, - `PENDING`, `IGNORED`. -- `Pos(Line, Char)` — a source position; both fields 0-indexed. -- `Mutation(ID, Description, Operation, Start, End, Status, Replacement, FrameworkMutantID=None)`. -- `MutantRegion(ID, StartLine, EndLine, Mutations=[])` — a conflict region - (`StartLine` inclusive, `EndLine` non-inclusive, both 0-indexed). -- `MarvOutput(files={})` — map of source path → list of `MutantRegion`. - -## checks/compilable.py - -```python -check_mutant_compilable(filename, language) -> bool -``` - -- **Not a real compile.** tree-sitter (grammar from `language`) parses the file - and returns False if any `ERROR` or missing node exists. Catches syntax errors - only — type errors, missing symbols, etc. pass. - -## checks/syntactic_equivalence.py - -```python -check_mutant_equivalent(mutant_filename, original_filename, language) -> bool -``` - -- Serializes both files' tree-sitter ASTs to strings, dropping - `language.comment_node_types`; True iff the strings are identical. Detects - mutants that only change comments/formatting. Prints a `SequenceMatcher` - similarity ratio as a side effect (not used in the decision). - -## approach/util.py - -```python -get_method_under_test(output_dir, language) -> str # reads language.original_method_path(output_dir); FileNotFoundError if absent -``` - -## approach/*/controller.py (all five) - -```python -main(model, output_dir, prompts, language) # prompts: the full dict built in __main__.py -``` - -Each approach's `generate_code` uses `language` for the mutant file extension, -the ```` ``` ````-fence tag it strips, and the prompt noun -(`Java method`/`Python function`). - -Each controller just calls its package's step functions in order with the -relevant `prompts[...]` key(s): - -- `basic`: `code_generator.generate_code` — 10 requests, each response saved as - `basic-mutants/mutant_N.java`. -- `hazop`: `method_explainer.describe_method` → `hazop-descriptions.txt`; - `description_mutator.mutate_descriptions` → `hazop-mutated-descriptions.txt` - (CSV rows `number, original_rule, GUIDEWORD, changed_rule`); - `code_generator.generate_code` → one mutant per changed rule. -- `stpa`: `control_diagram.create_control_diagram` → `control_diagram.txt` (DOT); - `uca_generator.generate_ucas` → `ucas.csv`; - `code_generator.generate_code` → one mutant per UCA - (loop is `range(0, ucas_count)` — one mutant for every UCA). -- `mutahunter`: `code_generator.generate_code` — one request containing the - method's AST plus line-numbered source; expects YAML back with - `mutants: [{mutated_code, line_number}, ...]`, spliced locally via - `util.parser.add_mutant_to_method`. The user-prompt template is intentionally - gutted (licensing) — users must fill in Mutahunter's own prompt text. -- `llmorpheus`: `placeholders.create_placeholders` — the tree-sitter query for - `language.name` (from `MUTATION_QUERIES`) captures mutation sites (Java: - if/while/do/switch conditions, for-loop parts, loop headers, call - names/receivers/args; Python: if/while conditions, for target/iterable, - comparison/boolean/binary operators, call function/args); each capture produces - `placeholders/N_placeholder.` (method with `` substituted), - `placeholders/N_orig.` (original fragment), and an entry in - `placeholders/placeholders.json` (`{tag, start_byte, end_byte, text}`). - `code_generator.generate_code` — one request per placeholder asking for 3 - replacements ("Option 1/2/3" fenced blocks, extracted with the regex - ```` \n```\n(.*?)\n``` ```` under `re.DOTALL`); each replacement spliced into - the method at the placeholder's byte offsets → - `llmorpheus-mutants/mutant_N.`. - -## execute/defects4j.py - -```python -run_mutants(project_root, original_file, output_path, - method_start_byte, method_end_byte, duplicate, approach, language) -``` - -- `_execute(project_root, file=None, approach=None, timer=None) -> bool`: - deletes `.classes_instrumented`/`target`, runs `defects4j test -w - ` with `PATH += ./defects4j/framework/bin` and - `JAVA_HOME = $JDK_11`; True iff output contains `Failing tests: 0`. - With `timer` set it enforces a subprocess timeout and writes test output to - `output/-test/_test.txt`. Kills stray Java processes after. -- `run_mutants` baselines the original project (raises IOError if its tests - fail), sets per-mutant timeout = 5× baseline duration, then per mutant: - restore from backup → equivalence check → rewrite → compilable check → - (if compilable) run tests. Appends `[name, equivalent, compilable, survives]` - rows and writes `mutant_summary.csv`. - -## execute/maven.py - -```python -run_mutants(project_root, filename, output_path, - method_start_byte, method_end_byte, duplicate, approach, language) -``` - -- `_execute(project_root) -> bool`: runs `mvn -f clean test`; - True if the output contains `BUILD SUCCESS` (all tests green → mutant - survived), False otherwise (mutant killed, incl. compile failures Maven - catches that tree-sitter does not). -- `run_mutants`: same flow as the Defects4J backend — baseline the original - project (raises `IOError` if its tests fail), then per mutant: restore from - backup → equivalence check → rewrite → compilable check → (if compilable) run - tests → append `[name, equivalent, compilable, survives]` → write - `mutant_summary.csv`. Drives the runnable Java example under `examples/`. - -## execute/pytest_runner.py - -```python -run_mutants(project_root, original_file, output_path, - method_start_byte, method_end_byte, duplicate, approach, language) -``` - -- `_execute(project_root) -> bool`: runs `sys.executable -m pytest -q - ` (argv list, no shell) — pytest runs under multiplex's own - interpreter, not a `python` resolved from PATH. True iff pytest exits 0 (all - tests pass → mutant survived), False otherwise (mutant killed, incl. - collection/syntax errors). If pytest is not installed for that interpreter, - raises a `SystemExit`. -- `run_mutants`: identical baseline → per-mutant flow as the Maven backend. - Drives the runnable Python example (`examples/config-python.yml`). Module named - `pytest_runner` so importing it does not shadow the installed `pytest` package. diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..faf5089 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= uv run sphinx-build +SOURCEDIR = source +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index 39c218a..0000000 --- a/docs/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# docs/ — Agent Documentation Index - -Documentation for humans and AI agents working on *multiplex*. Each file is -self-contained: **load only the file(s) relevant to your task** rather than -reading source code or every doc. - -| File | Read it when you need to... | -|------|-----------------------------| -| [ARCHITECTURE.md](ARCHITECTURE.md) | Understand the pipeline end-to-end: data flow, output artifacts, how mutants are injected and evaluated. | -| [MODULE_REFERENCE.md](MODULE_REFERENCE.md) | Call or modify an existing function — every public function's signature, inputs, outputs, and side effects, without opening source files. | -| [CONFIG.md](CONFIG.md) | Read, write, or validate a `config.yml` — full schema, every key. | -| [EXTENDING.md](EXTENDING.md) | Add a new mutant-generation approach, execution backend, language, or LLM provider — exact checklists of files to touch. | -| [DEVELOPMENT.md](DEVELOPMENT.md) | Set up, run commands, run tests, follow conventions, or avoid known gotchas and open bugs. | - -Quick orientation (enough for trivial tasks, no further reading needed): - -- *multiplex* mutates **one method/function** per run (Java or Python, set by - `project.language`): extract it with tree-sitter, ask an LLM to generate - mutants, splice each mutant back into the source file, run the project's tests, - and record which mutants survive. -- Entry point: `multiplex/__main__.py`, run as `uv run ./multiplex ./config.yml`. -- Python >= 3.13 (pin 3.13 — see DEVELOPMENT.md), deps via `uv`, lint via `ruff`, - tests via `pytest` from the repo root. -- Code inside `multiplex/` imports siblings **without** the `multiplex.` prefix - (`from model import Model`); tests import **with** it - (`from multiplex.checks... import ...`). See DEVELOPMENT.md before editing imports. diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000..dc1312a --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=source +set BUILDDIR=build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/source/_static/custom.css b/docs/source/_static/custom.css new file mode 100644 index 0000000..0d031f8 --- /dev/null +++ b/docs/source/_static/custom.css @@ -0,0 +1,84 @@ +/* ========================================================================== + Read the Docs (sphinx_rtd_theme) Custom Theme Overrides + ========================================================================== */ + +:root { + --accent-green: #3FD55C; + --accent-green-hover: #35b84f; + --bg-dark: #1E242B; + --bg-hover: #252B31; + --text-light: #F0F4F8; +} + +/* -------------------------------------------------------------------------- + Navigation Sidebar & Top Bar Backgrounds + -------------------------------------------------------------------------- */ + +.wy-nav-side, +.wy-nav-top { + background: var(--bg-dark); +} + +/* Search Box Focus Ring */ +.wy-side-nav-search input[type="text"]:focus { + border-color: var(--accent-green); +} + +/* -------------------------------------------------------------------------- + Sidebar Menu Styles + -------------------------------------------------------------------------- */ + +/* Default Sidebar Item Hover */ +.wy-menu-vertical a:hover { + background-color: var(--bg-hover); + color: #ffffff; +} + +/* Active / Expanded Section Container & Header ("Included Modules") */ +.wy-menu-vertical li.current > a, +.wy-menu-vertical li.on > a { + background: var(--bg-hover); + color: #ffffff !important; + border-right: 4px solid var(--accent-green); + + border-top: none !important; + border-bottom: none !important; + box-shadow: none !important; +} + +.wy-menu-vertical li.current > a:hover, +.wy-menu-vertical li.on > a:hover { + background: var(--bg-hover); + color: #ffffff !important; +} + +/* Expand / Collapse Icon for Active Section */ +.wy-menu-vertical li.current > a span.toctree-expand, +.wy-menu-vertical li.on > a span.toctree-expand { + color: var(--accent-green) !important; +} + +/* -------------------------------------------------------------------------- + Main Content Area Links & Buttons + -------------------------------------------------------------------------- */ + +/* Inline Links */ +.rst-content a, +.rst-content a:visited { + color: var(--accent-green); +} + +.rst-content a:hover { + color: var(--accent-green-hover); +} + +/* Navigation Buttons */ +.btn-neutral { + background-color: var(--bg-hover) !important; + color: #ffffff !important; +} + +.btn-neutral:hover { + background-color: var(--accent-green) !important; + color: var(--bg-dark) !important; +} diff --git a/docs/source/_static/images/favicon.png b/docs/source/_static/images/favicon.png new file mode 100644 index 0000000..66b3546 Binary files /dev/null and b/docs/source/_static/images/favicon.png differ diff --git a/docs/source/_static/images/logo.png b/docs/source/_static/images/logo.png new file mode 100644 index 0000000..7aa57ab Binary files /dev/null and b/docs/source/_static/images/logo.png differ diff --git a/docs/source/adding-execution-and-evaluation.rst b/docs/source/adding-execution-and-evaluation.rst new file mode 100644 index 0000000..6869b31 --- /dev/null +++ b/docs/source/adding-execution-and-evaluation.rst @@ -0,0 +1,2 @@ +Adding an Execution and Evaluation Module +========================================= diff --git a/docs/source/adding-language.rst b/docs/source/adding-language.rst new file mode 100644 index 0000000..05f6200 --- /dev/null +++ b/docs/source/adding-language.rst @@ -0,0 +1,2 @@ +Adding a Programming Language +============================= diff --git a/docs/source/adding-mutant-generation.rst b/docs/source/adding-mutant-generation.rst new file mode 100644 index 0000000..3cd557e --- /dev/null +++ b/docs/source/adding-mutant-generation.rst @@ -0,0 +1,2 @@ +Adding a Mutant Generation Approach +=================================== diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst new file mode 100644 index 0000000..3837742 --- /dev/null +++ b/docs/source/architecture.rst @@ -0,0 +1,2 @@ +Architecture +============ diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..876991e --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,35 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = "multiplex" +copyright = "2026, Megan Maton" +author = "Megan Maton" + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = [] + +templates_path = ["_templates"] +exclude_patterns = [] + + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = "sphinx_rtd_theme" +html_static_path = ["_static"] +html_logo = "_static/images/logo.png" +html_favicon = "_static/images/favicon.png" + +html_theme_options = { + "style_nav_header_background": "#1E242B", +} +html_css_files = [ + "custom.css", +] diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst new file mode 100644 index 0000000..6e79ebd --- /dev/null +++ b/docs/source/configuration.rst @@ -0,0 +1,2 @@ +Configuration +============= diff --git a/docs/source/existing-modules.rst b/docs/source/existing-modules.rst new file mode 100644 index 0000000..f93521e --- /dev/null +++ b/docs/source/existing-modules.rst @@ -0,0 +1,45 @@ +Included Modules +================ + +Languages +--------- +*multiplex* currently includes Java and Python compatibility. +These are selected per run through the configuration file through ``project.language``. +Instructions for adding additional language compatibility can be found in :doc:`adding-language`. + +Mutant Generation +----------------- +*multiplex* currently includes the following five mutant generation modules: + ++-------------+--------------------+--------------------------------------------+ +| Approach | *multiplex* config | Description | ++=============+====================+============================================+ +| STPA | stpa | Implements Systems Theoretic Process | +| | | Analysis mutant generation approach | ++-------------+--------------------+--------------------------------------------+ +| HAZOP | hazop | Implements HAZard and OPerability study | +| | | mutant generation approach | ++-------------+--------------------+--------------------------------------------+ +| Unguided | basic | Implements an unguided mutant generation | +| | | approach | ++-------------+--------------------+--------------------------------------------+ +| LLMorpheus | llmorpheus | Implements the LLMorpheus approach | ++-------------+--------------------+--------------------------------------------+ +| Mutahunter | mutahunter | Implements the mutahunter approach | ++-------------+--------------------+--------------------------------------------+ + +Execution and Evaluation +------------------------ +*multiplex* currently offers the following for mutant execution and evaluation. + ++-------------+--------------------+--------------------------------------------+ +| Framework | *multiplex* config | Description | ++=============+====================+============================================+ +| Maven | mvn | Execute mutants using a Maven runner | ++-------------+--------------------+--------------------------------------------+ +| Defects4J | d4j | Framework to execute test suites from | ++-------------+--------------------+--------------------------------------------+ +| PyTest | pytest | Execute mutants using PyTest runner | ++-------------+--------------------+--------------------------------------------+ + + diff --git a/docs/source/getting-started.rst b/docs/source/getting-started.rst new file mode 100644 index 0000000..5ef60cc --- /dev/null +++ b/docs/source/getting-started.rst @@ -0,0 +1,148 @@ +Getting Started +=============== + +Requirements +------------ +Using the *multiplex* tool requires the following: + +- UV Dependency Management: Follow installation instructions at `uv`_. +- ``yq`` for parsing yaml config files with bash + +.. _uv: https://github.com/astral-sh/uv + + +Installation +------------ + +1. Clone the repository: + + .. code-block:: bash + + git clone https://github.com/LLM-Mutation/multiplex.git + +2. Install the dependencies and activate environment: + + .. code-block:: bash + + cd multiplex + uv sync + source .venv/bin/activate + cd .. + + +Configuration +------------- + +*multiplex* is configured using a ``yaml`` file. +Users can use the example configuration file in the ``examples/`` to set up their project. +This section overviews the content required in the configuration file. + + +Project +^^^^^^^ + +This section includes the details about the project and method under test, including reference to the module for running and parsing the tests. + +LLM +^^^ + +To configure the LLM, update the config.yml file. +Below is an example for ``gpt-oss:20b``, run locally using Ollama. +To use `gpt-oss:20b`, download Ollama, and run `ollama pull gpt-oss:20b`. +Ensure Ollama is running (``ollama serve``) when using *multiplex* for this example. + +.. code-block:: yaml + + llm: + model: ollama/gpt-oss:20b + endpoint: http://127.0.0.1:11434 + token_env_var: # env var name storing token (NOT TOKEN) + +.. important:: + Security Note: Never hardcode API keys in your config file. Set an environment variable and reference its name in the token_env_var field. + +System Prompts +^^^^^^^^^^^^^^ + +.. note:: + Mutahunter Module prompts must be added by user due to licensing restrictions. - `User Prompt Link`_ / `System Prompt Link`_. + +.. _User Prompt Link:: https://github.com/codeintegrity-ai/mutahunter/blob/main/src/mutahunter/core/templates/mutant_generation/mutator_user.txt + +.. _System Prompt Link:: https://github.com/codeintegrity-ai/mutahunter/blob/main/src/mutahunter/core/templates/mutant_generation/mutator_system.txt + +The System Prompts are included in the configuration file for easy updating. +The user prompts are constructed within *multiplex* so users should modify or create modules to alter these. + +Running *multiplex* +------------------- +Once configured and modules are set up, users can run *multiplex* using the following command: + + +.. code-block:: bash + + uv run /path/to/multiplex ./path/to/config.yml + + +Included examples +^^^^^^^^^^^^^^^^^ + +Self-contained examples are included for both supported languages, each mutating +a small project with the ``basic`` approach. With a running Ollama instance +(``ollama pull gpt-oss:20b``, ``ollama serve`` and edit ``llm.model`` in the config), run from the repo +root: + +.. code-block:: bash + + uv run ./multiplex ./examples/config-java.yml # Java (needs mvn + a JDK on PATH) + uv run ./multiplex ./examples/config-python.yml # Python (uses pytest via uv run) + + +Results are written to the project's ``output/-mutants/mutant_summary.csv``. See `examples/README.md`_ for +details. + +.. examples/README.md:: https://github.com/LLM-Mutation/multiplex/blob/main/examples/README.md + +Optional: Marv output +^^^^^^^^^^^^^^^^^^^^^ +If you want Marv-compatible output, install Marv and make sure its binary is +on your ``PATH`` before running ``multiplex``: + +.. code-block:: bash + + go install github.com/SecretSheppy/marv@latest + export PATH="$HOME/go/bin:$PATH" + marv --version + +Inside the project, run + +.. code-block:: bash + + marv init -f generic + + +and update the .marv.yml file to the following: + +.. code-block:: yaml + marv: + port: 8080 + output: + path: .marv + merge: false + review-dir: .marv/reviews + generic: + framework: "multiplex" + marv-json: "output/marv.json" + src-dir: "output" + + +Then run *multiplex* with the ``--marv`` flag to generate ``output/marv.json`` +alongside the usual mutant files and summary: + +.. code-block:: bash + + uv run multiplex --marv ./examples/config.yml + + +Marv can then read the generated ``marv.json`` from the project output folder by running the ``marv`` command. + diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..67acf64 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,59 @@ +.. multiplex documentation main file, created by + sphinx-quickstart on Fri Aug 7 10:42:38 2026. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to *multiplex* +======================= + +A tool for prototyping and comparing LLM-based mutation testing techniques. + +*multiplex* has a modular design to enable researchers to build, design and test LLM-based Mutation testing approaches in an easy to use framework. + +Multiplex is currently designed to enable mutation at the method level, but +prompting and further modularisation could support other context levels. +Fork this project and add your own modules. + +Questions and Community +----------------------- +Have a question or found a bug? We’d love to hear from you. Please open an issue in the **Issue Tracker** and we'll get back to you! + +Citing this tool +---------------------- +If you use this tool in your work or research, please cite as follows: + +.. code-block:: latex + + @inproceedings{Maton2026a, + author = {Maton, Megan and Kapfhammer, Gregory M. and McMinn, Phil}, + title = {multiplex: A Modular LLM-based Mutation Framework}, + booktitle = {Proceedings of the International Conference on Automated Software Engineering (ASE) - Tools and Datasets Track}, + year = {2026}, + } + +If you are specifically interested in Hazard Analysis approaches for guiding LLM-based mutant generation, please consider reading (and if relevant, citing): + +.. code-block:: latex + + @inproceedings{Maton2026, + author = "Maton, Megan and Kapfhammer, Gregory M. and McMinn, Phil", + title = "Empirically Comparing Hazard-Guided LLM Mutation Techniques with Existing LLM- and + Rule-Based Approaches", + booktitle = "International Conference on Evaluation and Assessment in Software Engineering (EASE)", + year = "2026" + } + +.. toctree:: + :hidden: + :caption: Home + +.. toctree:: + :hidden: + :maxdepth: 3 + :caption: Introduction + + getting-started + existing-modules + configuration + + diff --git a/docs/source/requirements.txt b/docs/source/requirements.txt new file mode 100644 index 0000000..8d129a6 --- /dev/null +++ b/docs/source/requirements.txt @@ -0,0 +1 @@ +sphinx_rtd_theme=1.3.0 diff --git a/multiplex/execute/pytest_runner.py b/multiplex/execute/pytest_runner.py index 518ed31..1dcd319 100644 --- a/multiplex/execute/pytest_runner.py +++ b/multiplex/execute/pytest_runner.py @@ -1,9 +1,5 @@ """Pytest execution backend. -Mirrors ``execute/maven.py``: baseline the original (unmutated) project, then per -mutant restore the source, run the equivalence/compilable checks, splice the -mutant into the source, and run the project's tests to decide survived/killed. - Named ``pytest_runner`` (not ``pytest``) so it does not shadow the installed ``pytest`` package on import. """ @@ -31,8 +27,11 @@ def _execute(project_root, output_path=None, approach=None, label=None): command = [sys.executable, "-m", "pytest", "-q", str(project_root)] try: result = subprocess.run( - command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, - text=True, check=False + command, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + check=False, ) except FileNotFoundError as exc: raise SystemExit( @@ -70,7 +69,7 @@ def run_mutants( mutants = [["MUTANT", "EQUIVALENCE", "COMPILABLE", "SURVIVES"]] if not _execute(project_root, output_path, approach, "ORIGINAL"): - raise IOError( + raise OSError( "The original (unmutated) project did not pass pytest, so mutants " "cannot be evaluated against it. Run it manually to see why: " f"{sys.executable} -m pytest {project_root}" @@ -83,7 +82,9 @@ def run_mutants( shutil.copy2(duplicate, original_file) path = Path(mutants_dir, mutant_file) - mutant_equivalent = check_mutant_equivalent(path, original_method_path, language) + mutant_equivalent = check_mutant_equivalent( + path, original_method_path, language + ) mutant_output.append(mutant_equivalent) rewrite_method(original_file, method_start_byte, method_end_byte, path) diff --git a/multiplex/model.py b/multiplex/model.py index f1c648a..154b59c 100644 --- a/multiplex/model.py +++ b/multiplex/model.py @@ -3,23 +3,21 @@ from litellm import completion - class Model: - def __init__(self, model=None, endpoint=None, api_key_var=None): self.model = model self.endpoint = endpoint - if api_key_var is not None and api_key_var != '': + if api_key_var is not None and api_key_var != "": self.api_key = os.environ[api_key_var] - if "azure" in endpoint: + if "azure" in str(endpoint): os.environ["AZURE_AI_API_BASE"] = endpoint os.environ["AZURE_AI_API_KEY"] = os.environ[api_key_var] - + if "ollama" in self.model: + os.environ["OLLAMA_API_BASE"] = self.endpoint print("Model:", self.model) - def current_model(self): """Return current model""" return self.model diff --git a/pyproject.toml b/pyproject.toml index 52dc08a..a870248 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,4 +18,6 @@ dev = [ "pylint>=3.3.7", "pytest>=8.4.1", "ruff>=0.11.12", + "sphinx>=9.1.0", + "sphinx-rtd-theme>=3.1.0", ] diff --git a/uv.lock b/uv.lock index 466268a..db2221c 100644 --- a/uv.lock +++ b/uv.lock @@ -57,6 +57,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" }, ] +[[package]] +name = "alabaster" +version = "1.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a6/f8/d9c74d0daf3f742840fd818d69cfae176fa332022fd44e3469487d5a9420/alabaster-1.0.0.tar.gz", hash = "sha256:c00dca57bca26fa62a6d7d0a9fcce65f3e026e9bfe33e9c538fd3fbb2144fd9e", size = 24210, upload-time = "2024-07-26T18:15:03.762Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/b3/6b4067be973ae96ba0d615946e314c5ae35f9f993eca561b356540bb0c2b/alabaster-1.0.0-py3-none-any.whl", hash = "sha256:fc6786402dc3fcb2de3cabd5fe455a2db534b371124f1f21de8731783dec828b", size = 13929, upload-time = "2024-07-26T18:15:02.05Z" }, +] + [[package]] name = "annotated-types" version = "0.7.0" @@ -97,6 +106,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/77/06/bb80f5f86020c4551da315d78b3ab75e8228f89f0162f2c3a819e407941a/attrs-25.3.0-py3-none-any.whl", hash = "sha256:427318ce031701fea540783410126f03899a97ffc6f61596ad581ac2e40e3bc3", size = 63815, upload-time = "2025-03-13T11:10:21.14Z" }, ] +[[package]] +name = "babel" +version = "2.18.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" }, +] + [[package]] name = "certifi" version = "2025.8.3" @@ -176,6 +194,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" }, ] +[[package]] +name = "docutils" +version = "0.22.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ae/b6/03bb70946330e88ffec97aefd3ea75ba575cb2e762061e0e62a213befee8/docutils-0.22.4.tar.gz", hash = "sha256:4db53b1fde9abecbb74d91230d32ab626d94f6badfc575d6db9194a49df29968", size = 2291750, upload-time = "2025-12-18T19:00:26.443Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/02/10/5da547df7a391dcde17f59520a231527b8571e6f46fc8efb02ccb370ab12/docutils-0.22.4-py3-none-any.whl", hash = "sha256:d0013f540772d1420576855455d050a2180186c91c15779301ac2ccb3eeb68de", size = 633196, upload-time = "2025-12-18T19:00:18.077Z" }, +] + [[package]] name = "fastuuid" version = "0.14.0" @@ -347,6 +374,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl", hash = "sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3", size = 70442, upload-time = "2024-09-15T18:07:37.964Z" }, ] +[[package]] +name = "imagesize" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6c/e6/7bf14eeb8f8b7251141944835abd42eb20a658d89084b7e1f3e5fe394090/imagesize-2.0.0.tar.gz", hash = "sha256:8e8358c4a05c304f1fccf7ff96f036e7243a189e9e42e90851993c558cfe9ee3", size = 1773045, upload-time = "2026-03-03T14:18:29.941Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5f/53/fb7122b71361a0d121b669dcf3d31244ef75badbbb724af388948de543e2/imagesize-2.0.0-py2.py3-none-any.whl", hash = "sha256:5667c5bbb57ab3f1fa4bc366f4fbc971db3d5ed011fd2715fd8001f782718d96", size = 9441, upload-time = "2026-03-03T14:18:27.892Z" }, +] + [[package]] name = "importlib-metadata" version = "8.7.0" @@ -578,6 +614,8 @@ dev = [ { name = "pylint" }, { name = "pytest" }, { name = "ruff" }, + { name = "sphinx" }, + { name = "sphinx-rtd-theme" }, ] [package.metadata] @@ -595,6 +633,8 @@ dev = [ { name = "pylint", specifier = ">=3.3.7" }, { name = "pytest", specifier = ">=8.4.1" }, { name = "ruff", specifier = ">=0.11.12" }, + { name = "sphinx", specifier = ">=9.1.0" }, + { name = "sphinx-rtd-theme", specifier = ">=3.1.0" }, ] [[package]] @@ -860,6 +900,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7c/e4/56027c4a6b4ae70ca9de302488c5ca95ad4a39e190093d6c1a8ace08341b/requests-2.32.4-py3-none-any.whl", hash = "sha256:27babd3cda2a6d50b30443204ee89830707d396671944c998b5975b031ac2b2c", size = 64847, upload-time = "2025-06-09T16:43:05.728Z" }, ] +[[package]] +name = "roman-numerals" +version = "4.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ae/f9/41dc953bbeb056c17d5f7a519f50fdf010bd0553be2d630bc69d1e022703/roman_numerals-4.1.0.tar.gz", hash = "sha256:1af8b147eb1405d5839e78aeb93131690495fe9da5c91856cb33ad55a7f1e5b2", size = 9077, upload-time = "2025-12-17T18:25:34.381Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/54/6f679c435d28e0a568d8e8a7c0a93a09010818634c3c3907fc98d8983770/roman_numerals-4.1.0-py3-none-any.whl", hash = "sha256:647ba99caddc2cc1e55a51e4360689115551bf4476d90e8162cf8c345fe233c7", size = 7676, upload-time = "2025-12-17T18:25:33.098Z" }, +] + [[package]] name = "rpds-py" version = "0.27.1" @@ -960,6 +1009,123 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, ] +[[package]] +name = "snowballstemmer" +version = "3.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/43/f8/0a71edf031f03c40db17503cb8ca78a69a171254e568e7db241b0ab57ea1/snowballstemmer-3.1.1.tar.gz", hash = "sha256:e07bbc54a0d798fe6010a12398422e62a8bfbba95c394fd0956ef58cb4d3e260", size = 123314, upload-time = "2026-06-03T00:56:40.194Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4c/07/2ebca9b11fb9be7340a818d8d6f63feaebb146be2c4afbd6061701d6df6e/snowballstemmer-3.1.1-py3-none-any.whl", hash = "sha256:7e207fa178741da09cdee59d3ecec3827ad5f92b1fc5c9ff3755b639f71f5752", size = 104164, upload-time = "2026-06-03T00:56:38.614Z" }, +] + +[[package]] +name = "sphinx" +version = "9.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "alabaster" }, + { name = "babel" }, + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "docutils" }, + { name = "imagesize" }, + { name = "jinja2" }, + { name = "packaging" }, + { name = "pygments" }, + { name = "requests" }, + { name = "roman-numerals" }, + { name = "snowballstemmer" }, + { name = "sphinxcontrib-applehelp" }, + { name = "sphinxcontrib-devhelp" }, + { name = "sphinxcontrib-htmlhelp" }, + { name = "sphinxcontrib-jsmath" }, + { name = "sphinxcontrib-qthelp" }, + { name = "sphinxcontrib-serializinghtml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cd/bd/f08eb0f4eed5c83f1ba2a3bd18f7745a2b1525fad70660a1c00224ec468a/sphinx-9.1.0.tar.gz", hash = "sha256:7741722357dd75f8190766926071fed3bdc211c74dd2d7d4df5404da95930ddb", size = 8718324, upload-time = "2025-12-31T15:09:27.646Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/73/f7/b1884cb3188ab181fc81fa00c266699dab600f927a964df02ec3d5d1916a/sphinx-9.1.0-py3-none-any.whl", hash = "sha256:c84fdd4e782504495fe4f2c0b3413d6c2bf388589bb352d439b2a3bb99991978", size = 3921742, upload-time = "2025-12-31T15:09:25.561Z" }, +] + +[[package]] +name = "sphinx-rtd-theme" +version = "3.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "docutils" }, + { name = "sphinx" }, + { name = "sphinxcontrib-jquery" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/84/68/a1bfbf38c0f7bccc9b10bbf76b94606f64acb1552ae394f0b8285bfaea25/sphinx_rtd_theme-3.1.0.tar.gz", hash = "sha256:b44276f2c276e909239a4f6c955aa667aaafeb78597923b1c60babc76db78e4c", size = 7620915, upload-time = "2026-01-12T16:03:31.17Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/87/c7/b5c8015d823bfda1a346adb2c634a2101d50bb75d421eb6dcb31acd25ebc/sphinx_rtd_theme-3.1.0-py2.py3-none-any.whl", hash = "sha256:1785824ae8e6632060490f67cf3a72d404a85d2d9fc26bce3619944de5682b89", size = 7655617, upload-time = "2026-01-12T16:03:28.101Z" }, +] + +[[package]] +name = "sphinxcontrib-applehelp" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/6e/b837e84a1a704953c62ef8776d45c3e8d759876b4a84fe14eba2859106fe/sphinxcontrib_applehelp-2.0.0.tar.gz", hash = "sha256:2f29ef331735ce958efa4734873f084941970894c6090408b079c61b2e1c06d1", size = 20053, upload-time = "2024-07-29T01:09:00.465Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5d/85/9ebeae2f76e9e77b952f4b274c27238156eae7979c5421fba91a28f4970d/sphinxcontrib_applehelp-2.0.0-py3-none-any.whl", hash = "sha256:4cd3f0ec4ac5dd9c17ec65e9ab272c9b867ea77425228e68ecf08d6b28ddbdb5", size = 119300, upload-time = "2024-07-29T01:08:58.99Z" }, +] + +[[package]] +name = "sphinxcontrib-devhelp" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/d2/5beee64d3e4e747f316bae86b55943f51e82bb86ecd325883ef65741e7da/sphinxcontrib_devhelp-2.0.0.tar.gz", hash = "sha256:411f5d96d445d1d73bb5d52133377b4248ec79db5c793ce7dbe59e074b4dd1ad", size = 12967, upload-time = "2024-07-29T01:09:23.417Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/35/7a/987e583882f985fe4d7323774889ec58049171828b58c2217e7f79cdf44e/sphinxcontrib_devhelp-2.0.0-py3-none-any.whl", hash = "sha256:aefb8b83854e4b0998877524d1029fd3e6879210422ee3780459e28a1f03a8a2", size = 82530, upload-time = "2024-07-29T01:09:21.945Z" }, +] + +[[package]] +name = "sphinxcontrib-htmlhelp" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/43/93/983afd9aa001e5201eab16b5a444ed5b9b0a7a010541e0ddfbbfd0b2470c/sphinxcontrib_htmlhelp-2.1.0.tar.gz", hash = "sha256:c9e2916ace8aad64cc13a0d233ee22317f2b9025b9cf3295249fa985cc7082e9", size = 22617, upload-time = "2024-07-29T01:09:37.889Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0a/7b/18a8c0bcec9182c05a0b3ec2a776bba4ead82750a55ff798e8d406dae604/sphinxcontrib_htmlhelp-2.1.0-py3-none-any.whl", hash = "sha256:166759820b47002d22914d64a075ce08f4c46818e17cfc9470a9786b759b19f8", size = 98705, upload-time = "2024-07-29T01:09:36.407Z" }, +] + +[[package]] +name = "sphinxcontrib-jquery" +version = "4.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "sphinx" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/de/f3/aa67467e051df70a6330fe7770894b3e4f09436dea6881ae0b4f3d87cad8/sphinxcontrib-jquery-4.1.tar.gz", hash = "sha256:1620739f04e36a2c779f1a131a2dfd49b2fd07351bf1968ced074365933abc7a", size = 122331, upload-time = "2023-03-14T15:01:01.944Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/76/85/749bd22d1a68db7291c89e2ebca53f4306c3f205853cf31e9de279034c3c/sphinxcontrib_jquery-4.1-py2.py3-none-any.whl", hash = "sha256:f936030d7d0147dd026a4f2b5a57343d233f1fc7b363f68b3d4f1cb0993878ae", size = 121104, upload-time = "2023-03-14T15:01:00.356Z" }, +] + +[[package]] +name = "sphinxcontrib-jsmath" +version = "1.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b2/e8/9ed3830aeed71f17c026a07a5097edcf44b692850ef215b161b8ad875729/sphinxcontrib-jsmath-1.0.1.tar.gz", hash = "sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8", size = 5787, upload-time = "2019-01-21T16:10:16.347Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/42/4c8646762ee83602e3fb3fbe774c2fac12f317deb0b5dbeeedd2d3ba4b77/sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl", hash = "sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178", size = 5071, upload-time = "2019-01-21T16:10:14.333Z" }, +] + +[[package]] +name = "sphinxcontrib-qthelp" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/68/bc/9104308fc285eb3e0b31b67688235db556cd5b0ef31d96f30e45f2e51cae/sphinxcontrib_qthelp-2.0.0.tar.gz", hash = "sha256:4fe7d0ac8fc171045be623aba3e2a8f613f8682731f9153bb2e40ece16b9bbab", size = 17165, upload-time = "2024-07-29T01:09:56.435Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/27/83/859ecdd180cacc13b1f7e857abf8582a64552ea7a061057a6c716e790fce/sphinxcontrib_qthelp-2.0.0-py3-none-any.whl", hash = "sha256:b18a828cdba941ccd6ee8445dbe72ffa3ef8cbe7505d8cd1fa0d42d3f2d5f3eb", size = 88743, upload-time = "2024-07-29T01:09:54.885Z" }, +] + +[[package]] +name = "sphinxcontrib-serializinghtml" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3b/44/6716b257b0aa6bfd51a1b31665d1c205fb12cb5ad56de752dfa15657de2f/sphinxcontrib_serializinghtml-2.0.0.tar.gz", hash = "sha256:e9d912827f872c029017a53f0ef2180b327c3f7fd23c87229f7a8e8b70031d4d", size = 16080, upload-time = "2024-07-29T01:10:09.332Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/52/a7/d2782e4e3f77c8450f727ba74a8f12756d5ba823d81b941f1b04da9d033a/sphinxcontrib_serializinghtml-2.0.0-py3-none-any.whl", hash = "sha256:6e2cb0eef194e10c27ec0023bfeb25badbbb5868244cf5bc5bdc04e4464bf331", size = 92072, upload-time = "2024-07-29T01:10:08.203Z" }, +] + [[package]] name = "tiktoken" version = "0.11.0"