Skip to content

fix(mpp): decode \uXXXX escapes in challenge quoted-strings - #154

Closed
kriss39 wants to merge 1 commit into
tempoxyz:mainfrom
kriss39:fix/unicode-escape-in-quoted-params
Closed

kriss39 wants to merge 1 commit into
tempoxyz:mainfrom
kriss39:fix/unicode-escape-in-quoted-params

Conversation

@kriss39

@kriss39 kriss39 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Problem

A header value cannot carry characters above Latin-1, so mppx escapes them as \uXXXX when it serializes a challenge and decodes them again when it parses one. readQuotedAuthParamValue treats the backslash as a plain RFC 9110 quoted-pair — drop the backslash, keep the next byte — so the escape survives as literal text.

Round-tripping a description through the reference implementation and then through mpp-go:

mppx emits  : description="Payment — 50% off ☕"
mppx parses : "Payment — 50% off ☕"
mpp-go parses: "Payment u2014 50% off u2615"

pympp and mpp-rs decode it the same wrong way; companion PRs are open for both.

Description is excluded from the challenge binding, so a corrupted one is cosmetic. The same escape is applied to every quoted parameter though, including realm, which is slot 0 of the challenge ID HMAC — a realm above Latin-1 would come back altered and the binding check would then fail.

Fix

Decode the escape, matching readQuotedAuthParamValue in mppx:

  • \uXXXX becomes that code unit.
  • Astral characters are serialized as a surrogate pair (😀 → 😀), so a high surrogate is combined with the escape that follows it via utf16.DecodeRune.
  • An unpaired surrogate becomes U+FFFD, which is the closest value a Go string can hold.
  • A doubled backslash is still consumed first, so \\u2014 stays the literal text — — the same reasoning the reference implementation documents for why a bare \u is unambiguous.

Only the parse side changes, and only the escape handling: raw bytes still go through WriteByte, so characters at or below Latin-1, which the serializer leaves unescaped, are unaffected. They are covered in the tests to pin that down.

Tests

Seven cases in TestParseChallengeDecodesUnicodeEscapes: BMP escapes, an astral surrogate pair, raw Latin-1, both unpaired surrogates, a doubled backslash, and a truncated escape. Four fail on main.

go test ./... and go vet ./... pass, gofmt is clean.

The expectations were taken from mppx rather than written by hand: the header shape in the tests is what Challenge.serialize produces, and each expected value is what Challenge.deserialize returns for it.

A header value cannot carry characters above Latin-1, so a challenge escapes
them as `\uXXXX` and the reference implementation decodes them back on parse.
readQuotedAuthParamValue treated the backslash as a plain RFC 9110 quoted-pair,
dropping it and keeping the rest verbatim, so a description serialized as

    description="Payment — 50% off ☕"

parsed as "Payment u2014 50% off u2615".

Decode the escape, recombining the surrogate pairs that astral characters are
serialized as. An unpaired surrogate becomes U+FFFD, which is the closest value
a Go string can hold. A doubled backslash is still consumed first, so `\\u2014`
stays the literal text.

Characters at or below Latin-1 travel unescaped and already parsed correctly;
they are covered here to pin that down.
@kriss39

kriss39 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Companion fixes for the same escape, so the three SDKs land on the reference behaviour together: tempoxyz/pympp#247 (Python), tempoxyz/mpp-rs#418 (Rust).

@kriss39 kriss39 changed the title fix: decode \uXXXX escapes in WWW-Authenticate quoted-strings fix(mpp): decode \uXXXX escapes in challenge quoted-strings Sep 10, 2026
@brendanjryan

Copy link
Copy Markdown
Contributor

Superseded by #168, which expands this Unicode fix to parsing and formatting with surrogate, malformed-escape, and round-trip coverage, and is covered by tempoxyz/mpp-tools#240. Thank you for identifying this.

brendanjryan added a commit that referenced this pull request Sep 21, 2026
## Motivation

Quoted challenge parameters encoded by mppx use UTF-16 escapes, but the
Go parser treated them as literal text.

## Summary

- decode BMP and surrogate-pair escapes in quoted parameters
- emit canonical UTF-16 escapes for non-Latin-1 text
- cover literal, truncated, unpaired-surrogate, BMP, astral, and raw
Latin-1 cases

## Key design considerations

- preserve ordinary quoted-pair semantics
- unpaired surrogates decode to U+FFFD
- `description="Pay with 😀"` round-trips through `\ud83d\ude00`
- supersedes [#154](#154)

---------

Co-authored-by: Kanan <93033289+kriss39@users.noreply.github.com>
brendanjryan added a commit to tempoxyz/mpp-tools that referenced this pull request Sep 21, 2026
## Motivation

Challenge-header behavior has drifted across SDKs because Unicode escape
and extension-method cases were absent from the shared vectors.

## Summary

- add BMP, astral, realm, literal escape, truncated escape, and raw
Latin-1 scenarios
- add the canonical extended method identifier scenario
- exercise parse, format, and round-trip behavior where applicable

## Key design considerations

- mppx wire behavior remains the golden reference
- SDK updates: [mpp-rs
#428](tempoxyz/mpp-rs#428), [mpp-go
#168](tempoxyz/mpp-go#168), and [pympp
#256](tempoxyz/pympp#256)
- original reports: [mpp-rs
#414](tempoxyz/mpp-rs#414), [mpp-rs
#418](tempoxyz/mpp-rs#418), [mpp-go
#154](tempoxyz/mpp-go#154), and [pympp
#247](tempoxyz/pympp#247)
- keep this draft until the pinned SDK revisions contain the
corresponding fixes

---------

Co-authored-by: Kanan <93033289+kriss39@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants