Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions .agents/skills/framework-modernizer/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
---
name: framework-modernizer
description: >-
Use this skill when the user asks to migrate, upgrade, or modernize a
Node.js codebase from Express 4 to Express 5 — including phrasings like
"bump express to v5", "upgrade express", "migrate to express 5",
"express deprecation warnings", "we're stuck on express 4", or when
preparing a major-version dependency PR involving express ^4.x.
Also fires when the user mentions removed Express APIs (app.del, res.send(status),
the deprecated body-parser bundling), discontinued path-to-regexp 0.x
patterns (e.g. unnamed wildcards `*` or optional segments `:id?`), or
asks for a migration plan / breaking changes audit on an Express 4 repo.
This skill audits, classifies (SAFE / AUTOFIX / MANUAL), applies safe
autofixes, and emits a phased migration plan grounded in the official
Express 5 migration guide. It does NOT run the consumer's tests; it
emits a plan the team executes.
license: UNLICENSED
allowed-tools: Read, Grep, Glob, Edit
---

# framework-modernizer

> **Reference skill — Express 4 → Express 5 migration.**
> Built with [Genesis](https://github.com/DevExpGbb/genesis) as a worked example for the workshop. Read it, fork the pattern, build your own (Next 13→14, React 17→18, Angular 16→17…).

## When to use this skill

- The repo's `package.json` has a top-level `express` dependency on `^4.x` and the user wants to move to `^5.x`.
- The user reports deprecation warnings from Express 4 they want resolved.
- A major-version Dependabot/Renovate PR is open and a human asks "is this safe to merge?"

**Do NOT use this skill when:**
- The repo is not Express (Fastify, Koa, Hapi → wrong tool).
- The Express version is already `^5.x` (no work).
- The user wants you to also write/run tests post-migration (out of scope — emit the plan, the team validates).

## What this skill does

1. **Discover.** `Glob` for `package.json` files. `Read` each, confirm `express` is a direct dependency on `^4.x`. Stop early if not found.
2. **Scan.** For each Express 4 app rooted in a discovered `package.json`:
- `Grep -n` the breaking-change patterns from [`references/express-4-to-5-breaking-changes.md`](references/express-4-to-5-breaking-changes.md) across `**/*.{js,mjs,cjs,ts}`.
- Collect each hit with its file path, line number, and matched pattern ID (e.g. `BC-001`).
3. **Classify** every finding via [`references/classifier-rubric.md`](references/classifier-rubric.md):
- **SAFE** — no behavior change in v5; informational only.
- **AUTOFIX** — mechanical replacement; this skill applies it via `Edit`.
- **MANUAL** — semantics changed; emit a TODO comment + reference link, do **not** edit.
4. **Apply autofixes.** For each AUTOFIX finding, perform the exact `Edit` specified in the catalog. Print a one-line diff summary per edit.
5. **Emit migration plan.** Write `MIGRATION-PLAN.md` at the repo root using [`references/phased-plan-template.md`](references/phased-plan-template.md). Include three phases: **Phase 1 — Autofixed (this skill)**, **Phase 2 — Manual edits required**, **Phase 3 — Validation checklist**. Cross-reference every MANUAL item back to the official Express 5 migration guide.

## Outputs

| Artifact | Where | When |
|---|---|---|
| Per-file `Edit`s for AUTOFIX class | In-place | Step 4 |
| `MIGRATION-PLAN.md` at repo root | New file | Step 5 |
| Console summary: `N safe, M autofixed, K manual` | stdout | End |

## Constraints

- **Source-grounded only.** Every finding must trace to a pattern in [`references/express-4-to-5-breaking-changes.md`](references/express-4-to-5-breaking-changes.md). Do not invent breaking changes from memory — Express 5 is the reference, not your training data.
- **No package.json bump in this skill.** The migration plan instructs the team to bump `express` to `^5.0.0`; the skill does not rewrite it. (Reason: bumping invalidates the lockfile and triggers a npm install side-effect; that's a deliberate human gate.)
- **No test runs.** Emitting the plan is the deliverable. The team's CI is the oracle.
- **Idempotent.** Re-running the skill on an already-migrated repo emits `0 findings` and no edits.

## Examples

### Invocation

> "Migrate `services/api/` from Express 4 to Express 5."

### Expected end-state

```
Discovered: services/api/package.json (express ^4.18.2)
Scanned: 14 files
Findings:
SAFE × 2 (informational)
AUTOFIX × 3 → applied
MANUAL × 4 → see MIGRATION-PLAN.md

Wrote: services/api/MIGRATION-PLAN.md
```

## How this was designed

This skill went through the full [Genesis](https://github.com/DevExpGbb/genesis) 8-step process. The handoff packet is in [`references/DESIGN.md`](references/DESIGN.md). Reproducing it for your own framework migration (Next 13→14, React 17→18, etc.):

1. **Step 1 — intent.** Single capability, single framework pair. Don't try to migrate 5 frameworks in one skill.
2. **Step 2 — components.** PIPELINE pattern: scan → classify → autofix → plan. No fan-out, no panel.
3. **Step 5 — Architecture artifacts.** Three files do the heavy lifting: the **catalog** (cited breaking changes) is loaded as context; the **rubric** (SAFE/AUTOFIX/MANUAL classifier) is the decision boundary; the **skill** orchestrates them. The eval runner is the regression harness — change a regex, watch CI.

## Evals

Two layers, each catching different failures (see [`evals/README.md`](evals/README.md) for the full recipe):

1. **Behavior evals** ([`evals/evals.json`](evals/evals.json), [`evals/triggers.json`](evals/triggers.json)) — inference-based, per [agentskills.io spec](https://agentskills.io/skill-creation/evaluating-skills). Each case runs twice (with_skill / without_skill); the value delta is the evidence. Manual / harness-driven; not in CI.
2. **Catalog regression test** ([`evals/run.js`](evals/run.js)) — pure-Node script that locks the catalog regexes against a deliberate fixture. CI-friendly; exits non-zero on drift.

Both layers were scaffolded via the [Genesis](https://github.com/DevExpGbb/genesis) skill.
75 changes: 75 additions & 0 deletions .agents/skills/framework-modernizer/evals/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Evals — framework-modernizer

This directory holds **two distinct test layers**. Both are valuable; they catch different failures.

| Layer | What it tests | Output | Runs in CI? |
|---|---|---|---|
| **Behavior evals** ([`evals.json`](./evals.json), [`triggers.json`](./triggers.json)) | The skill's actual LLM behavior. Each case runs twice: with the skill loaded and without it, so the value delta is measurable. | `<skill>-workspace/iteration-N/<case>/{with_skill,without_skill}/{outputs,timing.json,grading.json}` | No — manual / harness-driven. See "How to run" below. |
| **Catalog regression test** ([`run.js`](./run.js)) | The deterministic substrate the skill depends on. Locks the catalog regexes against the fixture so the team can't silently break detection by editing a regex. | Pure-Node script, exits `0` on green, `1` on drift. | Yes — fast, hermetic, no LLM. |

The behavior evals are what [agentskills.io calls "evals"](https://agentskills.io/skill-creation/evaluating-skills). The catalog test is a unit test for the catalog file. Both were scaffolded via the [Genesis](https://github.com/DevExpGbb/genesis) skill — Genesis's step-6 EVALS PLAN produces `evals.json` + `triggers.json`; the catalog regression script is independent contributor tooling.

---

## Layer 1 — Behavior evals (the real evals)

`evals.json` defines 3 content evals + the iteration workflow. `triggers.json` defines ~20 trigger queries (60/40 train/val) for the dispatch description.

### How to run

The agentskills.io spec describes the structure but does NOT prescribe a runner — execution is harness-driven. The current canonical recipe:

1. **Per case in `evals.json`**, spawn TWO subagent runs (or two clean sessions if your harness has no subagent isolation):
- **with_skill**: skill is loaded into the subagent's context.
- **without_skill**: same prompt, no skill — establishes the baseline.
2. Capture each run's outputs into `framework-modernizer-workspace/iteration-N/<case-id>/{with_skill|without_skill}/outputs/`. Capture `timing.json` (`total_tokens`, `duration_ms` — your harness should provide these) alongside.
3. Grade per the case's `assertions[]`. Script-graded assertions (`file_exists`, `contains_all_of`, `no_invented_bc_ids`, `skill_activated`, `skill_declines`, `no_file_writes`) are mechanical. `llm_judge` assertions get a fresh LLM call with the rubric. Write `grading.json`.
4. Aggregate to `framework-modernizer-workspace/iteration-N/benchmark.json`.
5. **Ship gate**: all 3 cases PASS with_skill AND show measurable delta vs without_skill. If `with_skill ≈ without_skill`, the skill is not adding value — redesign or delete.

For trigger evals: run via the harness's dispatcher (NOT the skill body). The question is whether the dispatch description matches the query. Validation split is the gate (≥0.5 on should-trigger, <0.5 on near-miss should-not-trigger).

### Iteration discipline (per agentskills.io)

- **Add assertions AFTER the first run.** You don't know what "good" looks like until you've seen actual output. Iteration 1 is exploratory; iteration 2 hardens with assertions.
- **Iterate the skill body, not the assertions, when assertions fail.** If you're tweaking assertions to pass, you've inverted the test.

---

## Layer 2 — Catalog regression test

```bash
node .apm/skills/framework-modernizer/evals/run.js
```

Exit `0` = all expected findings matched; exit `1` = drift (catalog regex changed, fixture changed, or expected file out of date). CI-friendly.

### What's here (Layer 2 files)

| Path | Purpose |
|---|---|
| `run.js` | Pure-Node runner. Re-implements catalog regexes line-by-line and diffs vs `expected/findings.txt`. |
| `fixtures/express4-app/server.js` | Deliberate Express 4 mini-app exercising 8 of the 12 catalog patterns (BC-001, BC-002, BC-006, BC-007, BC-101, BC-102, BC-201, BC-202). Also used by Layer 1 case `fm-c1-explicit-migration`. |
| `fixtures/express4-app/package.json` | Pins `express ^4.18.2` so the fixture is unambiguously v4. |
| `expected/findings.txt` | Ground truth — `BC-ID<TAB>file<TAB>line` rows the runner must reproduce exactly. |

### Why it works this way

- **The catalog is the contract.** Every detection regex in [`../references/express-4-to-5-breaking-changes.md`](../references/express-4-to-5-breaking-changes.md) must have a corresponding entry in `run.js` and (for any pattern exercised by the fixture) a row in `expected/findings.txt`.
- **The fixture is intentionally broken.** Don't "fix" the v4 patterns — the file's whole purpose is to fail v5 detection so the eval has something to assert on.
- **Annotations use `EXPECT-NNN` form**, not the literal pattern, so comments don't trigger false positives in the regex pass.

### Extending — add a new BC-NNN pattern

1. Add the pattern to [`../references/express-4-to-5-breaking-changes.md`](../references/express-4-to-5-breaking-changes.md) with: ID, classification, source citation, detect regex, fix.
2. Add `['BC-NNN', /your-regex/]` to the `PATTERNS` array in `run.js`.
3. Add a triggering example to `fixtures/express4-app/server.js` (annotated `// EXPECT-NNN ...`).
4. Add the expected `BC-NNN<TAB>server.js<TAB><line>` row to `expected/findings.txt`.
5. Run `node .apm/skills/framework-modernizer/evals/run.js` and adjust line numbers if needed.
6. (If the new pattern materially changes skill behavior on the fixture) extend `evals.json` case `fm-c1-explicit-migration` so Layer 1 covers it too.

---

## Forking the pattern (Next 13→14, React 17→18, etc.)

Same two-layer structure, swap the catalog and fixture. See [`../references/DESIGN.md`](../references/DESIGN.md) for the Genesis handoff packet, and ask Genesis to scaffold the evals for your new skill — it will produce `evals.json` + `triggers.json` per the spec.
48 changes: 48 additions & 0 deletions .agents/skills/framework-modernizer/evals/evals.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{
"$schema": "https://agentskills.io/schemas/evals.json",
"_comment": "Content evals for framework-modernizer per agentskills.io spec. Each case runs TWICE: with the skill loaded (with_skill) and without (without_skill) to make the value delta measurable. Assertions are added AFTER the first iteration (you don't know what 'good' looks like until you see actual output). Grading is LLM-judge for holistic checks + script for mechanical checks. Workspace: framework-modernizer-workspace/iteration-N/<case-id>/{with_skill,without_skill}/{outputs,timing.json,grading.json}",
"skill": "framework-modernizer",
"cases": [
{
"id": "fm-c1-explicit-migration",
"prompt": "Migrate this Express 4 app to Express 5. Audit for breaking changes, apply safe fixes in place, and write a migration plan I can hand to my team.",
"files": ["fixtures/express4-app/"],
"expected_output": "Discovers ≥8 known patterns from references/express-4-to-5-breaking-changes.md (BC-001, BC-002, BC-006, BC-007, BC-101, BC-102, BC-201, BC-202). Applies AUTOFIX edits in place for the 4 AUTOFIX patterns. Writes MIGRATION-PLAN.md at fixture root with three phases. Every MANUAL item cites the official Express 5 migration guide URL. Console summary reports counts per class.",
"assertions": [
{"type": "file_exists", "path": "MIGRATION-PLAN.md", "kind": "script"},
{"type": "contains_all_of", "field": "outputs/MIGRATION-PLAN.md", "values": ["Phase 1", "Phase 2", "Phase 3", "expressjs.com/en/guide/migrating-5"], "kind": "script"},
{"type": "no_invented_bc_ids", "valid_ids_source": "references/express-4-to-5-breaking-changes.md", "kind": "script"},
{"type": "llm_judge", "rubric": "Are all autofix edits semantically correct (e.g. app.del → app.delete, res.sendfile → res.sendFile)? PASS only if all 4 AUTOFIX patterns are correctly transformed without breaking surrounding code."}
]
},
{
"id": "fm-c2-implicit-migration",
"prompt": "We're stuck on Express 4 and the deprecation warnings keep piling up. What would it take to move to v5?",
"files": ["fixtures/express4-app/"],
"expected_output": "Skill activates despite indirect phrasing (no 'migrate' verb). Produces a structured plan grounded in the catalog. Does NOT fabricate breaking-change IDs from training data. The without_skill baseline is expected to produce vague advice or a hallucinated breaking-change list — that delta is the evidence the skill is adding value.",
"assertions": [
{"type": "skill_activated", "kind": "script"},
{"type": "no_invented_bc_ids", "valid_ids_source": "references/express-4-to-5-breaking-changes.md", "kind": "script"},
{"type": "llm_judge", "rubric": "Compare with_skill vs without_skill output blind. Which one would a senior engineer trust more for an actual migration? PASS if with_skill is judged more grounded, more specific, and less likely to mislead."}
]
},
{
"id": "fm-c3-near-miss-decline",
"prompt": "Audit our Fastify codebase for upgrade issues from v4 to v5.",
"files": ["fixtures/express4-app/"],
"expected_output": "Skill correctly declines (per the 'Do NOT use this skill when' clause: 'The repo is not Express'). Surfaces that this skill is Express-only and suggests the user check for a Fastify-specific migration tool. Does NOT attempt to apply Express patterns to Fastify code.",
"assertions": [
{"type": "skill_declines", "kind": "script"},
{"type": "no_file_writes", "files": ["MIGRATION-PLAN.md"], "kind": "script"},
{"type": "llm_judge", "rubric": "Did the skill cleanly hand off without contaminating the user's Fastify context with Express-specific advice? PASS if the decline is explicit and useful."}
]
}
],
"iteration_workflow": {
"_comment": "Per agentskills.io: run each case twice (with_skill, without_skill) in subagent-isolated contexts. Capture outputs/, timing.json, grading.json per run. Aggregate to benchmark.json. Add assertions only AFTER the first iteration when you've seen what real output looks like. Iterate the skill body, NOT the assertions, when assertions fail.",
"workspace": "framework-modernizer-workspace/iteration-{N}/{case-id}/{with_skill|without_skill}/",
"outputs_per_run": ["outputs/", "timing.json", "grading.json"],
"aggregate": "framework-modernizer-workspace/iteration-{N}/benchmark.json",
"ship_gate": "All 3 cases PASS with_skill AND show measurable delta vs without_skill. If with_skill ≈ without_skill, the skill is not adding value — redesign or delete."
}
}
10 changes: 10 additions & 0 deletions .agents/skills/framework-modernizer/evals/expected/findings.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Expected findings on the fixture
# Format: BC-ID<TAB>file<TAB>line
BC-201 server.js 8
BC-202 server.js 11
BC-001 server.js 14
BC-006 server.js 20
BC-007 server.js 25
BC-002 server.js 30
BC-101 server.js 34
BC-102 server.js 39
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"name": "express4-app-fixture",
"version": "0.0.1",
"private": true,
"description": "Deliberate Express 4 fixture with 8 known breaking patterns (3 SAFE, 3 AUTOFIX, 2 MANUAL). Used by framework-modernizer eval runner.",
"main": "server.js",
"dependencies": {
"express": "^4.18.2"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// Deliberate Express 4 fixture for the framework-modernizer eval suite.
// Comment annotations use the EXPECT-* form to avoid accidental regex hits.

const express = require('express');
const app = express();

// EXPECT-201 SAFE: urlencoded extended default flipped in v5
app.use(express.urlencoded());

// EXPECT-202 SAFE: static dotfiles default flipped in v5
app.use(express.static('public'));

// EXPECT-001 AUTOFIX: app.del removed
app.del('/user/:id', (req, res) => {
res.send(`DELETE /user/${req.params.id}`);
});

// EXPECT-006 AUTOFIX: magic redirect string removed
app.get('/back', (req, res) => {
res.redirect('back');
});

// EXPECT-007 AUTOFIX: lowercase method renamed
app.get('/file', (req, res) => {
res.sendfile(__dirname + '/public/index.html');
});

// EXPECT-002 AUTOFIX: numeric-only send removed
app.get('/notfound', (req, res) => {
res.send(404);
});

// EXPECT-101 MANUAL: unnamed wildcard
app.get('/*', (req, res) => {
res.send('catch-all');
});

// EXPECT-102 MANUAL: optional segment
app.get('/file/:name.:ext?', (req, res) => {
res.send(`name=${req.params.name} ext=${req.params.ext}`);
});

module.exports = app;
Loading