Skip to content

Add assembly-time flow-counter static analysis (closes #18) - #55

Merged
michaelkamprath merged 24 commits into
mainfrom
flow-counters
Aug 2, 2026
Merged

michaelkamprath merged 24 commits into
mainfrom
flow-counters

Conversation

@michaelkamprath

Copy link
Copy Markdown
Owner

Summary

This branch adds flow counters: opt-in, assembly-time static analysis that tracks scalar quantities — hardware stack depth, cycle costs, or any other per-instruction budget — through a program's control flow. It closes issue #18 (named stack-slot offsets that survive push/pop edits) and builds the general machinery around it.

An ISA opts in by declaring a flow_counters section and per-instruction effect metadata in its configuration. Source code then opens tracking regions and lets the assembler do the bookkeeping:

divide32:
#track stack mode=called
    _arg_dividend := COORDINATE(stack, 3)   ; caller's slot at sp+3
    phs4                                    ; stack 0 -> 4
    ms4q _arg_dividend,_argX4               ; emits sp+7: offset tracks pushes
    pls4                                    ; stack 4 -> 0
    rts                                     ; verified balanced before return

What the analyzer does

  • CFG path analysis: follows direct branches and joins, verifies every return path independently, requires equal values where paths rejoin, and accepts net-zero loops. Multiple named counter instances track concurrently.
  • Counter coordinates: name := COORDINATE(counter, N) names a physical position; a bare reference to that symbol later resolves to its current offset at that program point. Inserting or removing a push updates every reference with no source edits. Crossed (popped) slots are permanently invalidated — a stale reference is a compile error, not a wrong byte.
  • Entry conventions & terminals: mode=called/mode=jumped entry modes, #entry roots for alternate entries, and per-class return-terminal reconciliation (before_effect/after_effect) verify routine-owned stack balance before control returns through a stack-saved address.
  • Manual control: #assert (flow-aware), #set, #suspend/#resume, instance bounds (min=/max=, including memory-map-derived label expressions), operand-dependent deltas (add sp, N), call-effect summaries, and watched memory-mapped writes that invalidate anchors.

Guarantees

  • Emission-inert: flow metadata changes no generated bytecode. A strip-equivalence invariant (enforced by tests) requires byte-identical output when directives are removed and references hand-resolved.
  • Dependency-driven: analysis runs only when checks are enabled or an emitted value needs it; non-flow ISAs and non-flow source pay nothing, and flow keywords are reserved only for ISAs that opt in.
  • --no-flow-checks/-A suppresses verification while still resolving emitted COUNTER() and coordinate values identically.

Language surface

#track/#endtrack, #entry, #assert, #set, #suspend/#resume, := COORDINATE() declarations, COUNTER(), bare coordinate references, and the __FLOW_COUNTERS_AVAILABLE__ capability probe.

Tooling & collateral

  • Generated Sublime/VSCode/Vim extensions highlight the flow syntax with a distinct coral palette; bare coordinate usages are tagged contextually (Sublime semantic regions, VSCode semantic tokens) since they are lexically indistinguishable from labels. Hover documentation covers every directive and operator; non-flow ISAs ship none of it.
  • Listing output gains a flow column showing per-line counter transitions and resolved coordinate offsets.
  • Generated ISA documentation renders counter-class contracts and per- instruction effects; wiki pages, CHANGELOG, and the requirements spec (docs/flow-counters-requirements.md) are updated.
  • Worked examples for Intel 8085 and Minimal 64x4 (original and redux), including a 32-bit math library tracking real MinOS stack usage.

Testing

~960 tests pass, including milestone acceptance suites (M0–M5), dev verification harnesses with byte-identity checks against hand-stripped twins, editor-generation tests for both enabled and non-flow ISAs, and regression tests for the semantic-highlighting render path. flake8 clean.

michaelkamprath and others added 19 commits July 16, 2026 20:55
- add counter-coordinate declarations and COORDINATE/OFFSET evaluation
- demonstrate stack-parameter tracking and configurable coordinate bounds
- rename LabelScope to SymbolScope and relocate flow symbol types
- centralize canonical global, file, and local symbol patterns
- align VS Code, Sublime, and Vim syntax and hover generation
- add M2 development examples, documentation, and test coverage
- Merge variant analysis semantics deeply via a shared
  merge_instruction_semantics() helper: nested metadata (documentation,
  flow_effects) now merges key-by-key instead of a variant override
  discarding root sibling keys; a variant can never remove a root key
  (explicit null is ignored); bytecode/operands still replace wholesale to
  mirror the emission machinery; the raw variants list no longer leaks into
  merged semantics; retained records and _effective_instruction_configs()
  are guaranteed to agree.
- Guard every FlowLinearAnalyzer error path so a future
  accumulate-and-continue DiagnosticReporter degrades to recorded
  diagnostics instead of crashing on invalid state.
- Stop shipping the := hover-detection literal in extensions generated
  from non-flow ISAs (templated via ##DECLARATION_OPERATOR##); add the :=
  docs to ALL_DIRECTIVE_DOCS and enforce coverage in docsgen tests.
- Document decisions and known future work in the requirements: M4
  context-validation rework, the top-level-parts assumption behind
  flow_expression_nodes, and zero-offset coordinate invalidation
  orientation. Record the newly reserved global keywords (#track,
  #endtrack, COUNTER, OFFSET, COORDINATE) as a breaking change in the
  changelog.
- Expand test coverage: nonfatal-reporter regression tests, flow-metadata
  validation on ISAs without a flow_counters section, register-plus-
  expression operand records, frozen expression tree shape, and diagnostic
  filename attribution; rename test_flow_configgen_m1.py to
  test_flow_configgen.py to match its M1+M2 scope.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- add named #track entry modes and explicit overrides
- support before- and after-effect terminal reconciliation
- evaluate and validate ARG(n) operand-dependent effects
- add M3 stack-parameter demos, documentation, and tests
- add general and flow-aware #assert with optional colored messages
- add #set, #suspend, and #resume for scalar counters
- support concurrent named counter instances and overlapping windows
- clarify #if, #assert, and #require semantics
- update syntax tooling, documentation, demos, and acceptance tests
- A live counter not mapped on a return terminal can no longer silently
  cross it: the return has no fall-through successor, so an unmapped live
  counter now errors on the terminal line with #endtrack guidance and its
  path closes, making code after the return uniformly unreachable for
  every counter (previously it kept accruing effects from dead code and
  COUNTER() resolved there, inconsistent with the solo-tracked case).
- #set now permanently invalidates the counter's prior coordinates, like
  #resume, per the dead-coordinate rule; its bounds check is pinned as
  deliberate. #resume to an already-invalidated coordinate now emits a
  flow warning instead of silently re-anchoring.
- Flow-directive value expressions (#set/#resume right-hand sides,
  #track/#endtrack init=/exit=) now accept every compile-time constant
  kind - =/EQU constants, #define macros, and ISA predefined constants -
  matching #assert. The name-vs-value rule is context-universal: names
  passed to COUNTER()/OFFSET()/COORDINATE() and the as=/mode= identifiers
  stay literal everywhere, including ordinary instruction operands and :=
  declarations, which previously had their flow names macro-substituted
  by the line factory's whole-line resolution. One shared
  protect-and-resolve helper now serves #assert, the flow directives, and
  the line factory.
- Preprocessor directive case-sensitivity codified per requirements
  clarification: dead IGNORECASE flags removed from the flow directive
  patterns and #TRACK pinned as unknown.
- AssertLine robustness: flow-dependent asserts no longer compute a
  throwaway general result, and general asserts are enforced exactly once
  (parse time), removing a double report under a non-fatal reporter.
- Test coverage: unreachable-line reads and completeness-check exemption,
  nonfatal-reporter regression tests extended with M3/M4 error sites,
  per-window #endtrack exit attribution, cross-class violation naming,
  and flow-assert hover-doc suffix presence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implement M5 branch, loop, call, entry-point, and path-consistency
analysis. Add flow-aware listings, development examples, editor
collateral, documentation, and regression coverage for CFG diagnostics.
- generate flow-counter ISA documentation and instruction hover details
- add instance bounds and counter-anchor invalidation support
- configure flow analysis for Intel 8085 and Minimal 64x4 Redux
- improve editor scopes for flow expressions and comparison operators
- expand milestone demos, validation, and regression coverage
- reset the required release version to 0.8.0
- replace static-analysis flags with -a/--flow-checks and -A/--no-flow-checks
- resolve emitted COUNTER() and OFFSET() values when checks are disabled
- add Minimal 64x4 stack metadata, tracked routines, and demonstration code
- update flow requirements, development harnesses, and regression coverage
A symbol declared with `:= COORDINATE(...)` now resolves to its current
counter-relative offset when referenced directly (`lds .arg`), mirroring
how address labels work and making flow-counter adoption a one-line
declaration change. The OFFSET() operator is removed from the expression
grammar, reserved keywords, and all analysis paths.

Assembler:
- resolve bare T_LABEL leaves against coordinate scopes during the
  analysis pass; ordinary labels/constants and the numeric fallback
  outrank a coordinate sharing their spelling
- carry coordinate-candidate nodes alongside definite flow nodes so the
  dependency-driven gating and CFG observation classification still
  trigger when emitted values reference declared coordinates
- enforce rejected contexts (layout, #if, constants) at resolution time
  via FlowSymbolError, since bare references are lexically invisible
- defer #assert operands containing preprocessor-unresolvable names to
  the flow pass; the bare-name shorthand falls back to coordinate
  offsets; #resume keeps its saved-anchor form for top-level coordinates
- dedupe coordinate-reference diagnostics across CFG worklist revisits

Editor tooling:
- drop OFFSET rules from Sublime/VSCode/Vim grammars and hover docs
- tag bare coordinate usages contextually instead: the Sublime plugin's
  semantic-region pass and a new VSCode `flowCoordinate` semantic token
  scope them as variable.other.flow.coordinate.usage (Vim has no
  contextual engine and retains label styling)
- give the coordinate-usage color rule the same low-alpha background as
  label/constant usages so Sublime regions render as a tint + underline
  rather than an opaque inverse-video fill, with a regression test
- rework the flow syntax palette around a coral family distinct from
  the macro/instruction oranges

Docs, spec, examples, dev harnesses, and tests updated to the bare
reference syntax; strip-equivalence and byte-identity guarantees are
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repository owner deleted a comment from chatgpt-codex-connector Bot Aug 1, 2026
michaelkamprath and others added 5 commits August 1, 2026 00:11
A call target address holding no program content is now treated as an
external callee (e.g. a MinOS ROM routine named by a predefined
constant): the declared flow_call_effects summary governs the
caller-visible effect, consistent with its taken-on-faith contract.
Previously every call target had to resolve to executable code inside
the assembled program, making ROM calls impossible in tracked regions.
A call target landing on emitted data remains an error, and hard-coded
numeric targets inside the program still resolve to the instruction at
that address (regression-tested for the interrupt-vector use case).

stringlib.min64x4 now assembles under flow analysis:
- cstr_reverse brackets its runtime-length push/pop loops with
  #suspend stack / #resume stack = 0 (depth has no static value there)
- uint32_to_decimal_cstr ends its region at the never-returning
  .err_buffer_size label, keeping the inline-string _Print convention
  (whose physical fall-through lands in string data) outside analysis

Spec updated to document external-callee semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…counters in Minimal 64x4 examples

A source-level macro invocation now behaves like a single instruction for
flow-value observation: COUNTER() values and counter-coordinate references
in its operand expressions resolve against the state at the invocation,
for every expanded constituent, instead of re-resolving between the
constituents' own effects. This makes macros whose spelled operand
arithmetic pre-compensates for internal counter motion (the Minimal 64x4
phs4s/phs2s stack-copy pattern: read a stack byte, push it, read the next
byte at a pre-adjusted offset) emit identical bytes for constant and
coordinate arguments. Previously a coordinate argument double-counted the
compensation and emitted operands off by one per constituent — including
4 wrong bytes already committed with stringlib's annotations. The listing
already displayed macro rows as one aggregate transition; value
resolution now agrees with it. Regression-tested by asserting a
push-compensating macro emits byte-identical output for both spellings.

Example updates, all verified byte-identical to their pre-flow baselines:
- Minimal 64x4: is_prime32 gains a tracked region with .arg_value /
  .arg_result coordinates replacing hand-maintained stack offsets
- Minimal 64x4 Redux: port the full flow-counter adoption from the 64x4
  set — primes (is_prime32 region), math32lib (compare/multiply/divide
  regions, #entry for multiply_int32), and stringlib (cstr_reverse
  #suspend/#resume span, uint32_to_decimal_cstr coordinates with the
  never-returning error tail closed by #endtrack, print_uint32) — with
  redux mnemonics preserved

Also: preprocessor directives may be indented (nested #if blocks), so the
Sublime, VSCode, and Vim grammars now tolerate leading whitespace before
the # instead of anchoring it to column 0; the assembler already accepted
the indented form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olding

Unit tests must depend only on files under test/ and src/. The milestone
acceptance harnesses the flow tests consume — ISA configs, .asm fixtures,
and the verify_m*.py scripts they execute — move from dev/flow-counters-m*
to test/flow_harnesses/m*, with the nine dependent test files repointed
and verify_m5_1.py's repo-root depth corrected for its new location.

The remaining dev/flow-counters-m* content (milestone READMEs and design
notes) is development scaffolding not intended for the public repository:
it is removed from tracking and suppressed via a scoped dev/.gitignore,
while remaining on disk locally. dev/color-scheme stays tracked — the
requirements doc references its palette demo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@michaelkamprath
michaelkamprath merged commit e5f566e into main Aug 2, 2026
4 checks passed
@michaelkamprath
michaelkamprath deleted the flow-counters branch August 2, 2026 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant