Repository navigation
fix(wiki-parser): exclude untracked and gitignored files from the wiki scan - #707
Open
royweisfeld wants to merge 1 commit into
Open
royweisfeld wants to merge 1 commit into
royweisfeld wants to merge 1 commit into
Conversation
The Karpathy wiki parser scanned every *.md file under the wiki root via rglob, including locally generated, untracked, or gitignored files. When a vault sits inside a git work tree, such files (generated cross-platform index files, private overlay pages, scratch notes) could enter the scan manifest — and their link targets (e.g. private-tier page paths) leaked into the knowledge graph metadata even though the target files were never traversed. parse-knowledge-base.py now filters scan candidates through git ls-files when the wiki root is inside a git work tree, at all scan sites (format detection, name-resolution map, article ID precompute, article node building). Exclusions are surfaced as a manifest warning. Non-git vaults keep the previous scan-everything behavior. Adds tests for both paths: untracked files excluded (with warning) in a git repo, and the fallback including all files without git.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
parse-knowledge-base.pydiscovered wiki articles withwiki_root.rglob("*.md")— every markdown file under the wiki root, including untracked files (locally generated cross-platform index pages, gitignored private-overlay mounts, scratch notes).This had two failure modes observed on a real vault:
rglobdoes not follow symlinked private mounts), but their paths and page titles landed inscan-manifest.jsonasknowledgeMeta.wikilinksand flowed intoknowledge-graph.json, which downstream consumers (context hooks, semantic queries) surface.Change
git_tracked_files()+list_wiki_md_files()helpers: when the wiki root is inside a git work tree, scan only git-tracked files; surface the excluded count as a manifest warning (git-filter: excluded N untracked markdown file(s) …).detect_format,build_name_to_stem_map, article-ID precompute, and article-node building — so leaked targets can't re-enter via nodes, name maps, or link resolution.Tests
test_untracked_files_excluded_when_git_repo— a git repo with a tracked wiki plus an untracked generated index linking to../_private/...: asserts the index is excluded, its private link target appears nowhere in the manifest, and the exclusion warning is present.test_fallback_includes_all_files_without_git— non-git vaults keep the old behavior.tests/skill/knowledge/test_parse_knowledge_base.py10 passed, 1 skipped.Notes
pnpm testshows 1 pre-existing failure on cleanmain(tests/skill/understand/test_scan_project.test.mjs›.understandignorenegation re-include) — reproduced with this branch stashed, so unrelated to this change.