session-start-hook.sh parses the payload in the same locale as its three siblings, not just with the same parser (#177)

[fdaviddpt]

session-start-hook.sh:32 parses the hook payload with jit_json_fields + jit_session_key out of common.sh — the same functions pre-prompt, pre-tool and pre-path use — and the comment above it explains why: a second, simpler regex would be a second answer to "what is a session id", and the two would drift. Those three pin LC_ALL=C. This one did not. The same parser in a different locale is not the same parser, so the comment was a claim the code did not carry out.

What was measured

3 engines x 2 locales, on a payload whose session_id carries a lone 0xE9, reading the value jit_session_key() returns:

	C	en_US.UTF-8
one-true-awk 20200816	"" refused	"" refused, plus a suppressed towc: multibyte conversion failure
gawk 5.4.1	"" refused	the id ACCEPTED, 0xE9 and all
mawk 1.3.4	"" refused	"" refused

gawk in a multibyte locale does not match a lone 0xE9 against [^A-Za-z0-9_-], so the bare-name check quietly stopped being a bare-name check — on the awk that is awk on most Linux boxes and on ubuntu-latest.

Bounded honestly. The separators are single-byte and still matched, verified independently:

LC_ALL=en_US.UTF-8 gawk 'BEGIN{k="jbad\351/x"; print (k ~ /[^A-Za-z0-9_-]/)?"MATCHED":"NOT"}'  -> MATCHED

So nothing ever left the state directory and this was never a traversal. What was lost was agreement: the matching hooks refused that id and kept no marker, while this hook built two marker names out of it and cleared files nothing had written.

The test, and why it is shaped oddly

The obvious fixture — plant the marker the unpinned parse would clear, assert it survives — cannot be built on macOS, because APFS enforces valid UTF-8 in file names. The first draft was red on all three engines for that reason alone, with the redirection error on stderr as the only tell. That is a vacuous negative and would have shipped as coverage.

Section J instead asks a question that has an answer on every filesystem: what was rm handed? A shimmed rm first on PATH logs its arguments. A refused id never reaches the if [ -n "$SESSION_ID" ] branch, so no marker path is built at all.

Red, before the pin:

  FAIL: [gawk] a malformed session id was treated as a session id
    2 marker path(s) reached rm; the three matching hooks refuse this id
      rm arg: .../state/vocab-shown-jbad?id.txt
      rm arg: .../state/path-shown-jbad?id.txt
56 passed, 1 failed

Green after: 57 passed, 0 failed. Every negative leg is paired with a positive control in the same fixture, engine and locale.

The second commit gives section J the third state: this file already had one (section I drives exit 2, which run-all.sh keeps apart from both), and without it a host with no UTF-8 locale printed N passed, 0 failed and exited 0 — byte-identical to a run that drove the gawk cell. Driven all three ways: rc=0 normal, rc=2 with locale shimmed to report US-ASCII, rc=1 with JIT_TESTS_REQUIRE_UTF8_LOCALE=1.

The $( ) NUL-drop at the same line

Measured rather than argued about: it costs nothing. one-true-awk truncates the record at the NUL, gawk carries it and the class check matches it, mawk refuses it — no path puts a NUL into the capture on any engine in either locale. Written into the comment so nobody re-measures it.

Scope — the sweep was done, the pins were not

I drove all nine unpinned sites, not just this one. Five of nine diverge. Full table in the report note; the invariant it found is sharper than "every awk that reads untrusted bytes is pinned":

An awk diverges by locale exactly when it matches a regex against a record that can carry an invalid byte. Sites doing only index(), substr() and == are identical in all six cells.

That predicts all nine rows. The four other divergent sites are deliberately not pinned here — they are build tooling under the opposite contract (tooling.md: fail loudly), and for three of them pinning would convert a loud one-engine abort into a quiet pass-through that relies on rebuild-tsv.sh's bad-byte reporter — which the issue itself documents as broken on mawk/ubuntu-latest. Pinning first would trade a loud bug for a quiet one. Details and the prerequisite are in the report.

Two corrections to the issue

• Its per-site table is short by one row: jit-dry-run.sh:452 at 3b8caaa (check_paths_fragment) is absent, and it is one of the five that diverge. The stated count of nine does not match the eight unpinned rows listed.
• README.md:486 is README.md:518 at HEAD. At HEAD, :486 is a sentence about @invocation macros.

The README sentence claimed LC_ALL=C was "pinned on every awk in the plugin". It was not and still is not. It now claims only what holds and what the sweep verified: every awk that reaches the pattern guard — pre-tool-hook.sh, pre-path-hook.sh and jit-dry-run.sh's pattern probes. (pre-prompt-hook.sh composes no $JIT_AWK_GUARD, so it is correctly excluded.)

Closes #177.

Verified by the maintainer

Both corrections to the issue are accepted, and the issue was mine. Its per-site table is short by one row and its README.md citation had rotted by 32 lines — which is, with some irony, the exact class of defect #191 is open about, committed inside the issue asking for a locale sweep.

Red re-run, mine, in a scratch worktree at this branch with only the pin reverted: 57 passed, 0 failed with it, 56 passed, 1 failed without — failing on the gawk leg only, FAIL: [gawk] a malformed session id was treated as a session id, with the awk and mawk legs green in the same run. That one-engine signature is what makes this a locale finding rather than a broken fixture, and it is why the shimmed-rm design matters: the obvious fixture cannot exist on APFS, and the draft that tried it was red on all three engines for a reason that had nothing to do with the bug.

The refusal to pin the other four is the best decision in this PR and I am accepting it in full. The reasoning is that one-true-awk's abort is today's loudness at those sites; pinning to C removes it and hands the bad byte to a reporter that is silent on mawk — the default awk on ubuntu-latest. That is trading the loud bug for the quiet one, and refusing it is worth more than the four-line diff that would have looked like completing the issue.

Filed rather than carried, with the measurement attached:

• rebuild-tsv.sh can write an empty path index and say nothing: one-true-awk aborts under a UTF-8 locale and the append is unchecked #195 — rebuild-tsv.sh can write an empty path index and report success: :452 aborts under a UTF-8 locale, emits 0 rows against 3, and the >> "$tsv" append is unchecked. Plus the :538 comment asserting a pin that does not exist, and :353's right-loudness-wrong-reason message. This is the prerequisite, not a follow-up.
• Pin the four remaining divergent awk sites — measurement is done, and it is blocked on the bad-byte reporter #196 — the four remaining divergent sites, blocked on rebuild-tsv.sh can write an empty path index and say nothing: one-true-awk aborts under a UTF-8 locale and the append is unchecked #195, carrying the full per-site table and the sharper invariant this sweep produced: an awk diverges by locale exactly when it matches a regex against a record that can carry an invalid byte. Sites doing only index(), substr() and == are identical in all six cells — which predicts all nine rows.

One class is not-checked and is recorded as such rather than rounded to a pass: whether Git Bash on windows-latest has a UTF-8 locale, and therefore whether section J runs there or skips. The auditor read the workflow and did not run CI. Section J's third state means a skip is now visible rather than silent, so this PR's own machinery will answer it on the first Windows leg.