check_skill_frontmatter: also catch name/description length and name-directory mismatch

[antonyprasad-db]

Problem or opportunity

check_skill_frontmatter in scripts/skillsgen/validators.py guards one silent-drop failure today: an unquoted : in a description, which strict-YAML loaders read as a mapping separator and discard. Its docstring names the failure well — the skill is dropped at load time and the author is never told.

Three more failures behave identically and are not checked. Consuming loaders enforce constraints beyond "the YAML parses":

• name outside 1–64 characters
• description longer than 1024 characters
• name not equal to the skill's directory name

A loader that hits any of these drops the skill with at most a warning in its own log. Nothing in this repo's CI notices, and the skill author sees no error — the skill just silently stops being available in that agent.

The name/directory mismatch is not hypothetical. Two skills published elsewhere in the ecosystem are dropped for exactly this reason when installed into an agent that enforces it:

name: analyzing-mlflow-trace     directory: analyze-mlflow-trace
name: analyzing-mlflow-session   directory: analyze-mlflow-chat-session

No skill in this repo violates any of the three rules today, so this is preventive rather than a live bug. The margin is thinner than it looks though — measured with folded blocks resolved:

skill	description length	of 1024
databricks-app-design	845	82.5%
databricks-dbsql	737	72.0%
databricks-model-serving	717	70.0%

One ordinary edit to either of the top two would push it over, and the skill would disappear from any agent enforcing that ceiling with no signal here.

Proposed change

Extend check_skill_frontmatter with those three checks, alongside the existing unquoted-colon check, so the same class of failure is caught in one place.

One implementation detail is worth calling out because it is easy to get wrong: measuring the description with a first-line-only regex under-counts folded and literal blocks badly. skills/databricks-dbsql/SKILL.md uses description: >- with the text on indented continuation lines, so a naive regex measures it as 2 characters rather than 737 — which would let an over-long folded description pass the very check being added. The measurement needs to resolve >, >- and | blocks as well as plain and quoted scalars, and stay regex-based rather than yaml.safe_load to keep the package stdlib-only per the existing note about the CI runner having no PyPI.

Affected skill or area

scripts/skillsgen/validators.py (check_skill_frontmatter), scripts/skills.py (re-export of the two limit constants), tests/skills_generator_test.py (SkillFrontmatterTest).

No skill content changes.

Additional context

I have this implemented and tested against the repo, since PR creation here is restricted to collaborators. Happy to hand over the patch, or to land it in the internal source of truth myself once my Universe access is set up — whichever is less work for you.

Test results with the change applied: 126 tests pass (7 added), including the existing test_repo_skills_are_clean, which confirms all 32 shipped skills already satisfy the new rules. The added cases cover one per rule, the description boundary at exactly the ceiling, and folded descriptions both over and within the limit.

Constraints referenced above were read from a consuming loader's shipped implementation and confirmed by observing which skills it accepted and rejected in practice, rather than from documentation.

[antonyprasad-db]

Correction: the name/directory mismatch claim above is wrong

I need to retract one of the three rules I proposed. On closer reading of a consuming loader's implementation, a name that disagrees with the directory name does not cause the skill to be dropped. It emits a warning and the skill loads normally.

The validator's last statement is a comma expression:

return frontMatter.name !== dirName && report("skill.name.mismatch", {…}),
       { frontMatter, content }

It evaluates the warning, discards the result, and returns a truthy object. The caller does if (result) return { type: "skill", … }, so the skill is registered. Contrast the preceding branches, which each do report(…); return — returning undefined, which the caller treats as "skip":

condition	outcome
frontmatter missing	dropped
name or description empty	dropped
name outside 1–64 chars	dropped
description over 1024 chars	dropped
name != directory name	warning only — skill loads

So the two skills I cited are not silently unavailable. They log a warning on every scan and work fine. My description of them as "dropped for exactly this reason" was incorrect, and I should have traced the control flow before asserting it.

What still stands

The two length rules are unaffected and remain the substance of this proposal:

• name outside 1–64 characters → skill dropped
• description over 1024 characters → skill dropped

Both are genuine silent-drop failures with no signal in this repo's CI, and the headroom is real: databricks-app-design at 845 of 1024 (82.5%) and databricks-dbsql at 737 (72.0%), the latter only measurable correctly if folded blocks are resolved.

On the third check

A name/directory mismatch is still worth flagging in my view, but as consistency hygiene, not as a silent-drop bug — the honest justification is that the two disagreeing is confusing and tools may reasonably assume they match, not that anything currently breaks. Reasonable to drop that check from the proposal if you would rather keep the validator strictly to failures that lose functionality. Happy either way; the length ceilings are the part that matters.

[antonyprasad-db]

Implemented — the two length checks

Both length rules are now enforced in check_skill_frontmatter, with tests. The change is in review against the internal source of truth this repo publishes from, so it should appear here through the normal publish once it merges.

What it adds:

• name longer than 64 characters → error
• description longer than 1024 characters → error
• an absent or empty name → error (previously unchecked, and an empty name is a silent drop too)

Description length is measured through the existing extract_description_from_skill in skillsgen/discovery.py rather than a new regex. That function already resolves folded and literal blocks via _BLOCK_SCALAR_INDICATORS, so databricks-dbsql's description: >- is measured at its real length instead of the 2 characters a first-line regex sees. Reusing it seemed better than introducing a second, subtly different parser for the same field.

Tests: 7 cases — over-limit and at-limit for both fields, so the comparison is > and not >=; an absent name; and the folded-scalar case with a short control, so the check can't pass by always firing. The existing test_repo_skills_are_clean still passes across all 32 skills, which confirms this is a regression guard rather than a fix: nothing violates either limit today.

The name/directory check is not included

Per the correction above, that rule only warns and the skill still loads. There's a second, independent reason to leave it out: check_skill_frontmatter has no warning channel. Every string it returns becomes a hard CI failure via ok = False in the validate path, so adding this check would fail the build for something no loader actually rejects. Introducing a warning severity is a broader change to the CLI plumbing and didn't belong in the same change.

Happy to revisit it if a warning channel is ever added.

[antonyprasad-db]

Correction: the length check had four gaps, and my reasoning above was wrong

In my previous comment I said measuring the description through the existing extract_description_from_skill "seemed better than introducing a second, subtly different parser for the same field." That was the wrong call, and reusing it inherited its blind spots. Review found four ways an over-limit description still passed the check. Three are concrete and reproducible:

Shape	Real length	Measured as
description: "start --- <long tail>"	1210	5
description: >- # note + indented body	~1500	21
plain scalar continued on indented lines	~1531	16

The causes:

1. Truncation at a mid-line ---. That helper ends the frontmatter at the first --- anywhere, whereas _read_frontmatter and the real file shape only close on a delimiter line. So a description containing a literal --- was cut before measurement.
2. Unrecognised block-scalar headers. It matches only the six bare forms |, |-, |+, >, >-, >+. YAML also permits an explicit indentation indicator and a trailing comment, so >2 and >- # note fell through as plain scalars and measured as the header token.
3. Plain multi-line scalars. An unquoted value continued onto indented lines was measured by its first line only, so a short opening line hid an over-limit folded body.
4. Folding is approximated, not implemented. This one can't be fixed here: the package is stdlib-only because the protected CI runner has no pypi, so there's no YAML parser available.

Gaps 1 to 3 are fixed. The measurement now resolves from the frontmatter block the check already holds, and folds continuation lines for plain and quoted scalars as well as block scalars. Gap 4 is documented on the function rather than papered over — blank lines inside block scalars are now preserved as separators, which narrows it, and the residual error direction is under-counting, so the failure mode is a missed violation rather than a false alarm on a valid skill.

I deliberately left extract_description_from_skill itself alone, because it also feeds manifest and marketplace generation and changing its parsing is a wider change than this lint fix should make. Worth noting though that manifest descriptions inherit the same truncation, so that helper arguably deserves its own fix.

Tests cover each gap with a matching short control, including a sibling key after a multi-line description to prove the fold stops at column 0.

Nothing about the two length limits themselves has changed — name at 64 and description at 1024 still stand, and no skill in the repo violates either.

[antonyprasad-db]

Shipped — closing.

Both length checks are live in this repo now. scripts/skillsgen/validators.py defines MAX_SKILL_NAME_LEN = 64 and MAX_SKILL_DESCRIPTION_LEN = 1024 and enforces them in the frontmatter validation path; tests/skills_generator_test.py covers both boundaries plus the folded-scalar case my first attempt measured wrong.

Scope as landed, against what I originally proposed:

• description over 1024 chars → flagged. Shipped.
• name over 64 chars → flagged. Shipped.
• name != directory name → not shipped. Retracted above: third-party loaders only warn on this, they don't drop the skill, so it doesn't belong in a hard check.

Closing. Thanks for bearing with the two corrections along the way.