Repository navigation
Release 0.8.0: macOS 27.2 fm support and accurate decode metrics - #9
Conversation
macOS 27.2 renamed `fm available` to `fm models`; `system` only looked available because the deprecation warning put the word next to the model name. Read availability and model identity from one `fm models` call (with an `available` fallback), drop fm `warning:` lines before parsing, and keep full reasons such as the Private Cloud Compute "use the Terminal app" hint. Measurement fixes found on real hardware: - TPOT/decode now use only tokens after the first delivery (fm emits ~20 tokens in its first chunk), and write bursts under 5 ms are one delivery. - Output token counts drop the constant framing token count-tokens adds, calibrated once per run. - The CV column is run-to-run CV per prompt instead of the pooled spread across different prompts; DECODE/S is token-weighted. - One warmup per model by default. Also: model identity in reports/compare, doctor license check, command and flag typo suggestions, macOS 27.2 fixtures, and a 27.2-shaped fake fm. Co-authored-by: Cursor <cursoragent@cursor.com>
…hing - CI matrix covers Node 22, 24, and 26 (Node 20 is end of life), smoke-tests both fm surfaces, and reports one aggregate ci status. - Least-privilege permissions, SHA-pinned actions, job timeouts, no dependency cache in publish jobs. - Release publishes via npm trusted publishing (OIDC) with an NPM_TOKEN fallback, still writes release notes when publishing fails but then fails visibly, and installs the published package to verify it. - Version dispatches Release, since a GITHUB_TOKEN tag push never did. - Links and release notes point at dvnold/fm-bench. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d9b191d96
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Compare the run-to-run CV only when both reports carry it; otherwise fall | ||
| // back to the suite-wide latency CV on both sides so the definitions match. | ||
| function comparableCv(before, after) { | ||
| const bothStable = before && after && Object.hasOwn(before, 'stabilityCv') && Object.hasOwn(after, 'stabilityCv'); | ||
| return bothStable | ||
| ? [before.stabilityCv, after.stabilityCv] | ||
| : [before?.latency?.cv, after?.latency?.cv]; |
There was a problem hiding this comment.
When a 0.7.x report and a 0.8.0 report use identical suite settings (for example, both explicitly set --warmup 1), suiteMatch remains true and --strict accepts the comparison. This fallback aligns the changed CV definition, but TPOT is still compared even though 0.7.x used outputTokens - 1 and process-exit timing while 0.8.0 uses tokens after the first delivery and ends at the last delivery; output token rates also changed through framing-overhead removal. The resulting percentages can therefore be presented as regressions or improvements with no compatibility warning, so cross-version comparisons should be warned/rejected or those incompatible fields suppressed.
Useful? React with 👍 / 👎.
Summary
Verified on real hardware: macOS 27.2 (26B5091g), Apple M5 Pro,
/usr/bin/fm.fm: availability and model identity (system = AFM 3 Core Advanced) come fromfm models, with a fallback toavailableon 27.0. Previouslysystemonly looked available because theavailabledeprecation warning sat next to the model name. Full reasons are kept (for examplepcc: "…Please use the Terminal app."),doctorchecksfm license --status, andcomparewarns when the model identity changes.count-tokensadds.modles→models,--rnus→--runs);doctoroutput is aligned.See
CHANGELOG.mdfor details.Verification
npm run checkon Node 22, 24, and 26 (175 tests)actionlintfm:doctor,models,--runs 3,--profile interactive|throughput,--sweep-concurrency 1,2 --ciwith SLOs,--output-dir --export-html,validate,export,compare,history, thepccunavailable path, typo guardCHANGELOG.mdand docs updatedMade with Cursor