Skip to content

fix(skills): align skill catalog names with read_skill resolution#250

Merged
initializ-mk merged 2 commits into
mainfrom
fix/skill-lookup-naming
Jul 7, 2026
Merged

fix(skills): align skill catalog names with read_skill resolution#250
initializ-mk merged 2 commits into
mainfrom
fix/skill-lookup-naming

Conversation

@initializ-mk

Copy link
Copy Markdown
Contributor

Problem

An agent with a skill whose loadable name differs from its internal ## Tool: heading could never load that skill. Observed live: the LLM was shown k8s_triage in the skill catalog and called read_skill{"name":"k8s_triage"}, but the skill's loadable name (frontmatter name / directory, also what the agent card advertises) was k8s-incident-triage. read_skill resolves by filesystem path, so it returned {"error":"skill \"k8s_triage\" not found"}, and the model abandoned the skill and improvised raw kubectl.

Root cause: three divergent identifiers for one skill.

Layer Source Value
Agent card + directory frontmatter name: k8s-incident-triage
LLM catalog (what the model calls with) ## Tool: heading (entry.Name) k8s_triage
read_skill resolution filesystem path skills/<arg>/SKILL.md (+ _-) needs k8s-incident-triage

The catalog advertised the internal tool heading while read_skill resolved by the skill's loadable name, and the two never had to agree. Unit tests never crossed the catalog→read_skill boundary, so nothing caught it.

Changes

buildSkillCatalog (forge-cli/runtime/runner.go) — now emits one line per skill keyed by the loadable name (frontmatter name, falling back to directory/file) — the exact string read_skill resolves and the agent card advertises. Each line also carries a provides: <tool names> list so the LLM still sees the skill's capabilities for tool selection, without mistaking them for the read_skill argument. Script/binary-backed ## Tool: entries are excluded from provides because they are registered as first-class callable tools (registerSkillTools) and invoked directly by name.

## Available Skills

To use a skill, call `read_skill` with the skill name (the identifier before the colon)
to load its full instructions, then follow them. `provides:` lists the capabilities inside
a skill — they are documentation loaded with the skill, not tools you call directly.

- k8s-incident-triage: Read-only Kubernetes incident triage using kubectl. [provides: k8s_triage] (uses cli_execute)

read_skill (forge-core/tools/builtins/read_skill.go) — resolves via a frontmatter-name index (keyed by both the frontmatter name and the directory/file name, normalized for case and _/- drift) in addition to the direct path lookup. A miss now returns the available skill names (available_skills) so the model can retry instead of giving up. Directory-traversal guard preserved.

Why both halves

  • The catalog fix makes the model call read_skill with the resolvable name in the common case (directory == frontmatter name).
  • The read_skill index makes resolution robust when the directory differs from the frontmatter name, and turns a dead-end "not found" into an actionable list.

Tests

  • forge-core/tools/builtins/read_skill_test.go — resolves by frontmatter name when the directory differs; normalized variants (k8s_incident_triage, K8S-Incident-Triage); flat skills/<name>.md layout; not-found lists available skills; traversal rejected.
  • forge-cli/runtime/runner_skill_catalog_test.go — catalog keys the line on the loadable name (not the tool heading) while surfacing tool names under provides; and a catalog↔read_skill contract test asserting every advertised name resolves through read_skill (the integration gap that let this ship).

Full forge-core/tools/... and forge-cli/runtime suites green; gofmt -w + golangci-lint run clean on both changed packages.

The system-prompt skill catalog advertised each skill by its internal
'## Tool:' heading name (e.g. 'k8s_triage'), but read_skill resolves a
skill by its loadable name — the frontmatter 'name' / directory (e.g.
'k8s-incident-triage', which is also what the agent card advertises).
When the two differed, the LLM called read_skill with the advertised
tool name and got 'skill not found', then abandoned the skill.

Changes:
- buildSkillCatalog now emits one line per skill keyed by the loadable
  name (frontmatter name, falling back to directory/file) — the exact
  string read_skill resolves and the agent card advertises. Each line
  also carries a 'provides: <tool names>' list so the LLM still sees the
  skill's capabilities for tool selection, without confusing them for
  the read_skill argument. Script/binary-backed '## Tool:' entries are
  excluded from 'provides' because they are registered as first-class
  callable tools (registerSkillTools) and invoked directly by name.
- read_skill resolves via a frontmatter-name index (keyed by both the
  frontmatter name and the directory/file name, normalized for case and
  underscore/hyphen drift) in addition to the direct path lookup; a miss
  now returns the available skill names instead of a bare error.

Regression tests: read_skill resolves by frontmatter name when the
directory differs, normalized variants, flat layout, not-found lists
available skills; catalog advertises the loadable name (not the tool
heading) while surfacing tool names under 'provides'; and a
catalog<->read_skill contract test asserting every advertised name
resolves (the gap that let this ship).
…ference)

Skills ship more than SKILL.md — helper scripts (shell, python,
javascript), reference folders, and additional markdown all live in the
skill's directory and reach the running agent via COPY . . at build
time, but were invisible to the LLM because read_skill returned only
SKILL.md.

read_skill now appends a '## Skill files' listing of every other file in
a subdirectory skill's folder, relative to the agent root and annotated
with the script language, so the model can read or execute them as the
skill's steps describe. Flat single-file skills are unaffected.

Also documents why skillEntryHasScript (the catalog 'provides' exclusion)
stays aligned to registerSkillTools' .sh-only registration: a py/js tool
is not yet a callable tool, so it correctly stays in 'provides' and its
script is surfaced by the new file listing. Broadening the extension set
must wait for registerSkillTools to register those languages, or the
tools would vanish from both the tool list and the catalog.
@initializ-mk

Copy link
Copy Markdown
Contributor Author

Folded in a follow-up (46b91a3): read_skill now surfaces the skill's directory files.

Skills ship more than SKILL.md — helper scripts (shell / python / javascript), reference folders, and extra markdown all live in the skill's directory and reach the running agent (COPY . .), but were invisible because read_skill returned only SKILL.md. It now appends a ## Skill files listing of every other file in a subdirectory skill's folder, relative to the agent root and annotated with the script language:

## Skill files
This skill ships the following supporting files (relative to the agent root). Read or execute them via your file/exec tools as the steps above describe:
- skills/k8s-incident-triage/reference/runbook.md
- skills/k8s-incident-triage/scripts/collect.sh — shell
- skills/k8s-incident-triage/scripts/triage.py — python

Flat single-file skills (skills/<name>.md) are unaffected.

On the catalog side, the provides: exclusion (skillEntryHasScript) is deliberately kept aligned to registerSkillTools' .sh-only registration: a .py/.js-backed tool is not yet registered as a directly-callable tool, so it correctly stays in provides: and its script is surfaced by the new file listing. Broadening the extension set is a separate change that must land together with registerSkillTools learning to execute those languages — otherwise such tools would vanish from both the tool list and the catalog. The helper now documents that coupling.

Tests added: file listing includes py/js/sh scripts + reference files and excludes SKILL.md; flat skills append nothing.

@initializ-mk

Copy link
Copy Markdown
Contributor Author

Follow-up tracked in #251 — skill-relative file resolution (read referenced markdown and execute referenced .sh/.py/.js scripts from a skill's own directory). This PR lays the groundwork: read_skill now resolves a skill by its loadable name and lists the skill's directory files (with languages); #251 closes the loop so those referenced files actually read/execute relative to the skill dir.

@initializ-mk initializ-mk merged commit 18deda8 into main Jul 7, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant