Skip to content

fix(parsing): decode \uXXXX escapes in challenge quoted-strings - #247

Open
kriss39 wants to merge 1 commit into
tempoxyz:mainfrom
kriss39:fix/unicode-escape-in-quoted-params
Open

fix(parsing): decode \uXXXX escapes in challenge quoted-strings#247
kriss39 wants to merge 1 commit into
tempoxyz:mainfrom
kriss39:fix/unicode-escape-in-quoted-params

Conversation

@kriss39

@kriss39 kriss39 commented Sep 10, 2026

Copy link
Copy Markdown

Problem

A header value cannot carry characters above Latin-1, so mppx escapes them as \uXXXX when it serializes a challenge and decodes them again when it parses one. pympp treats the backslash as a plain RFC 9110 quoted-pair — drop the backslash, keep the next character — so the escape survives as literal text.

Round-tripping a description through the reference implementation and then through pympp:

mppx emits : description="Payment — 50% off ☕"
mppx parses: "Payment — 50% off ☕"
pympp parses: "Payment u2014 50% off u2615"

mpp-go and mpp-rs decode it the same wrong way; companion PRs are open for both.

description is excluded from the challenge binding, so a corrupted one is cosmetic. The same escape is applied to every quoted parameter though, including realm, which is slot 0 of the challenge ID HMAC — a realm above Latin-1 would come back altered and the binding check would then fail.

Fix

Decode the escape in _unescape_quoted, matching readQuotedAuthParamValue in mppx:

  • \uXXXX becomes that code unit.
  • Astral characters are serialized as a surrogate pair (😀😀), so a high surrogate is combined with the escape that follows it.
  • An unpaired surrogate becomes U+FFFD. JavaScript can hold a lone surrogate in a string and Python nominally can too, but it does not survive encoding, so the replacement character is the closest usable value.
  • A doubled backslash is still consumed first, so \\u2014 stays the literal text — the same reasoning the reference implementation documents for why a bare \u is unambiguous.

Only the parse side changes. Characters at or below Latin-1 are not escaped by the serializer and already parsed correctly; they are covered in the tests to pin that down.

Tests

Eight cases in tests/test_parsing.py: BMP escapes, an astral surrogate pair, raw Latin-1, a mixed value with a quote and a backslash, both unpaired surrogates, a doubled backslash, and a truncated escape. Five of them fail on main.

uv run pytest -m "not integration" passes (960 passed, 11 skipped), ruff check, ruff format --check and pyright are clean.

The expectations were taken from mppx rather than written by hand: each header in the tests is what Challenge.serialize produces, and each expected value is what Challenge.deserialize returns for it.

A header value cannot carry characters above Latin-1, so a challenge escapes
them as `\uXXXX` and the reference implementation decodes them back on parse.
pympp treated the backslash as a plain RFC 9110 quoted-pair, so it dropped it
and kept the rest verbatim: a description serialized as

    description="Payment — 50% off ☕"

parsed as "Payment u2014 50% off u2615" instead of "Payment — 50% off ☕".

Decode the escape in `_unescape_quoted`, recombining the surrogate pairs that
astral characters are serialized as. An unpaired surrogate becomes U+FFFD,
since it has no Python representation that survives encoding. A doubled
backslash is still consumed first, so `\\u2014` stays the literal text.

Characters at or below Latin-1 are not escaped and already parsed correctly;
they are covered here to pin that down.
@kriss39

kriss39 commented Sep 10, 2026

Copy link
Copy Markdown
Author

Companion fixes for the same escape, so the three SDKs land on the reference behaviour together: tempoxyz/mpp-go#154 (Go), tempoxyz/mpp-rs#418 (Rust).

@kriss39 kriss39 changed the title fix: decode \uXXXX escapes in WWW-Authenticate quoted-strings fix(parsing): decode \uXXXX escapes in challenge quoted-strings Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant