feat: implement Python source extractor

[Shreyaaaash]

Pull Request

Summary

Adds PythonExtractor, which parses .py source files with the stdlib ast module into NormalizedDocuments, giving code ingestion its first source-format extractor.

Linked context

Closes #84

Checked issue #84 and the open PR list before starting — no other contributor has claimed it and no competing PR exists. The only related item is #68 (Markdown extractor, merged), which is a different format and not a duplicate of this work.

PR Size

• Medium (100-499 LOC delta)

PR Type

• New feature
• Test

Context / Motivation

The roadmap calls for source-code ingestion, and Python is the smallest useful first format since the standard library can parse Python structure via ast without adding a dependency. This establishes the extractor pattern for code (as opposed to prose formats like Markdown/text) ahead of any multi-language work.

Changes

• Add PythonExtractor (src/anchor/extractors/python.py), which walks top-level ast nodes and emits one NormalizedDocument per top-level function/class, plus module-typed documents for other top-level statements (docstring, imports, constants, top-level calls), in file order.
• Each document includes line_start/line_end, char_start/char_end (exact round-trip offsets into the original file), and code-aware metadata: symbol_type (function / async_function / class / module) and symbol_name for defs/classes.
• Decorators are included in a function/class's content and line range.
• Add tests/fixtures/ingestion/sample.py, a fixture with a module docstring, imports, a top-level function, a class, and a trailing top-level statement.
• Add 15 unit tests in tests/unit/test_python_extractor.py covering functions, async functions, classes, decorators, nested methods, module docstring/imports, interleaved module/function ordering, empty files, and char-offset round-tripping.

How to test

1. pip install -e ".[ingestion]"
2. pytest -m "not eval" — full deterministic suite passes (135 passed, 36 deselected)
3. pytest tests/unit/test_python_extractor.py -v -m unit — 15/15 pass in isolation
4. Optional: ruff check / ruff format --check / mypy src/anchor/extractors/python.py all clean

Prompt Change Eval Results

N/A — no prompt changes in this PR.

Checklist

• I ran the relevant local checks.
• I updated documentation or confirmed no docs changes were needed.
• I linked related issues or pull requests and confirmed this is not duplicate work.
• This change stays within the stated scope of the PR.

Notes for reviewers

• Not wired into detect.py or ingestor.py — public file-ingestion routing was explicitly out of scope for [task] Add Python source-code extractor #84, since format detection ([task] Add source-code format detection #83) landed separately and this issue is scoped to the extractor only.
• Module-level content is intentionally split into its own module-typed document(s) rather than dropped or merged into adjacent symbol docs, mirroring how the Markdown extractor handles pre-heading content — happy to discuss if a different grouping is preferred.

Summary by CodeRabbit

• New Features

  • Added Python source extraction support to generate structured documents for top-level functions, async functions, classes, and module-level content.
  • Includes metadata such as symbol type/name and line/character ranges, with source format marked as Python.

• Bug Fixes

  • Ensures decorators are included in extracted definitions.
  • Skips empty or whitespace-only extracted sections.

• Tests

  • Added unit coverage for malformed Python inputs, verifying a ValueError with the file path and a SyntaxError cause.
  • Added fixture coverage for extracting constants, functions, and classes.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds PythonExtractor, which parses Python files with ast and emits NormalizedDocument objects for top-level symbols and module content, including source metadata. Adds a fixture module and unit tests for extraction behavior and malformed input.

Changes

Python Extractor Implementation and Tests

Layer / File(s)	Summary
Core extractor implementation src/anchor/extractors/python.py	Adds AST-based extraction for top-level functions, async functions, classes, and module statements with decorator-aware line ranges, character offsets, symbol metadata, and syntax-error conversion.
Extraction fixture module tests/fixtures/ingestion/sample.py	Adds module-level statements, a greet function, a Greeter class, and an import-time print statement.
Extractor behavior and edge-case tests tests/unit/test_python_extractor.py	Covers symbol and module documents, ordering, decorators, nested methods, metadata offsets, empty files, multiple functions, and malformed Python errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant PythonExtractor
  participant ast
  participant NormalizedDocument

  Caller->>PythonExtractor: extract(path)
  PythonExtractor->>ast: parse(source)
  ast-->>PythonExtractor: top-level AST nodes
  PythonExtractor->>NormalizedDocument: build symbol or module documents
  PythonExtractor-->>Caller: return extracted documents

Loading

Possibly related PRs

• Codelab-Davis/Anchor#68: Establishes the shared NormalizedDocument and extractor pattern used by this Python extractor.

Suggested labels: area: source, area: tests, needs: duplicate-review

Suggested reviewers: AlexanderGardiner

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR implements the Python extractor and tests for functions, classes, module content, and metadata as requested in #84.
Out of Scope Changes check	✅ Passed	The changes stay within the extractor, fixture, and tests, with no unrelated language extractors or ingestion wiring added.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding a Python source extractor.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

src/anchor/extractors/python.py (1)

28-31: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Consider handling SyntaxError from ast.parse.

When the extractor is eventually wired into the ingestion pipeline, a malformed .py file will cause ast.parse to raise an unhandled SyntaxError, crashing the extraction step. Since the PR objectives note this is not yet wired into detect.py/ingestor.py, this is acceptable for now, but adding a try/except that either returns an empty list or logs and skips would make the extractor more resilient when integrated.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/anchor/extractors/python.py` around lines 28 - 31, The
PythonExtractor.extract method currently calls ast.parse directly, so a
malformed .py file will raise an uncaught SyntaxError when this extractor is
used. Update extract in PythonExtractor to catch SyntaxError around the
ast.parse call and either return an empty list or log-and-skip the file so the
ingestion pipeline can continue gracefully; use the extract method and ast.parse
as the key locations to change.

tests/fixtures/ingestion/sample.py (1)

1-28: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Fixture file is not referenced by any test in this PR.

All 15 tests in test_python_extractor.py use run_extract with inline string content rather than loading this fixture. Consider either adding a test that exercises the fixture (e.g., an integration-style test that reads and extracts it) or removing it to avoid dead code. If it's intended for future use, a brief comment explaining its purpose would help.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/fixtures/ingestion/sample.py` around lines 1 - 28, The sample Python
fixture is currently unused, so either add a test in test_python_extractor.py
that actually loads and runs extraction against sample.py (using run_extract or
a similar helper) or remove the fixture if it is not needed. If you keep it,
reference its purpose near the fixture or in the relevant test so its presence
is clearly intentional and tied to the Python extractor coverage.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/anchor/extractors/python.py`:
- Around line 28-31: The PythonExtractor.extract method currently calls
ast.parse directly, so a malformed .py file will raise an uncaught SyntaxError
when this extractor is used. Update extract in PythonExtractor to catch
SyntaxError around the ast.parse call and either return an empty list or
log-and-skip the file so the ingestion pipeline can continue gracefully; use the
extract method and ast.parse as the key locations to change.

In `@tests/fixtures/ingestion/sample.py`:
- Around line 1-28: The sample Python fixture is currently unused, so either add
a test in test_python_extractor.py that actually loads and runs extraction
against sample.py (using run_extract or a similar helper) or remove the fixture
if it is not needed. If you keep it, reference its purpose near the fixture or
in the relevant test so its presence is clearly intentional and tied to the
Python extractor coverage.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f0ed6754-f287-4337-9470-dfe77db38ddd

📥 Commits

Reviewing files that changed from the base of the PR and between 6a085c3 and 85de13b.

📒 Files selected for processing (3)

• src/anchor/extractors/python.py
• tests/fixtures/ingestion/sample.py
• tests/unit/test_python_extractor.py

[github-actions]

Thanks for opening your first pull request in this repository, @Shreyaaaash.

Before review starts, make sure the PR checklist is complete and the fastest relevant local checks have been run. The expected workflow is documented in CONTRIBUTING.md.

[github-actions]

Potentially related open or recent pull requests:

• feat: implement Markdown extractor with mistune #68 feat: implement Markdown extractor with mistune (merged)

If one of these replaces this PR, a maintainer can close this with /superseded-by #123.

[AlexanderGardiner]

Thanks for contributing! This looks good! Could you update the PR title so the CI for PR Hygiene passes? Also this issue from CodeRabbit is valid and needs a quick fix:

Consider handling SyntaxError from ast.parse.

When the extractor is eventually wired into the ingestion pipeline, a malformed .py file will cause ast.parse to raise an unhandled SyntaxError, crashing the extraction step.

[AlexanderGardiner]

Thanks! Sorry for the delay, I've merged now.