Skip to content

[bugfix] Replace CachingResponseWrapper's header-name allowlist with buffer+replay - #6692

Merged
line-o merged 2 commits into
developfrom
dp-header-buffer-redesign
Sep 15, 2026
Merged

line-o merged 2 commits into
developfrom
dp-header-buffer-redesign

Conversation

@duncdrum

@duncdrum duncdrum commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #6691. Supersedes the fixed header-name allowlist that PR added (Content-Length, Content-Type, Last-Modified, ETag, Accept-Ranges) with a structural fix, per design discussion on #6691.

CachingResponseWrapper was conflating two different kinds of header write:

  • Pipeline configuration — a controller.xql <exist:set-header> directive, applied by URLRewrite#setHeaders before doRewrite()'s dispatcher.forward() even runs. Must survive regardless of which pipeline step's output eventually wins.
  • Step output — anything the dispatched step itself sets (Jetty's own resource metadata, or the step's own XQuery response:set-header() calls). Provisional: if a later <exist:view> step replaces this step's output, its headers must not survive either.

The allowlist from #6691 could only close this gap one enumerated header name at a time. This PR replaces it with a structural distinction instead:

  • New HeaderPassthroughResponse interface + HttpResponseWrapper#setPassthroughHeader, used only by URLRewrite#setHeaders, tunneling through however many CachingResponseWrapper layers are stacked to reach the real response immediately. Deliberately distinct from HttpResponseWrapper#setHeader (which backs response:set-header()), so a step's own programmatic header call is not accidentally granted the same bypass.
  • Everything else a step writes is now buffered via a sealed BufferedHeader record hierarchy (setHeader/addHeader/setIntHeader/addIntHeader/setDateHeader/addDateHeader) and replayed verbatim, in original order, from flush() — but only for whichever wrapper actually gets flushed.

Content-Type and Content-Length keep their existing special-casing (untouched): Content-Type via the first-write-wins contentType field, Content-Length derived from the actual buffered byte count at flush time rather than trusted from any step's self-reported value.

close #3913

Test Plan

@github-actions

github-actions Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

📊 XQTS result comparison

Comparison of this run against develop.

Metric develop this run Change
➖ Passed 28,846 (90.67%) 28,846 (90.67%) 0 (0.00 pp)
➖ Failures 1,568 1,568 0
➖ Errors 134 134 0
➖ Skipped 1,267 1,267 0
🧪 Total tests 31,815 31,815 0

Relative to develop: 1 newly passing, 1 newly failing, 0 new errors, 0 newly skipped — counting only tests recorded in both runs whose outcome changed.

🔴 Newly failing tests (1)
  • Constr-inscope-4 (was passing)
🟢 Newly passing tests (1)
  • Constr-inscope-2 (was failing)

Runtime: 529.1s (+231.23s vs develop).

@duncdrum
duncdrum added this pull request to stack #6695 September 10, 2026 13:59
@duncdrum duncdrum added the performance bottlenecks, opportunities for rewriting, optimization label Sep 10, 2026
@duncdrum
duncdrum marked this pull request as ready for review September 10, 2026 14:40
@duncdrum
duncdrum requested a review from a team as a code owner September 10, 2026 14:40
@joewiz

joewiz commented Sep 13, 2026

Copy link
Copy Markdown
Member

[This response was prompted by Joe, drafted by Claude Code, and reviewed by Joe.]

Approved. Replacing the allowlist with the config-versus-step-output distinction is the right call — an enumerated list of header names could only ever be closed one name at a time, and the next surface to set an unlisted header would have reopened it. Separating the two kinds of write structurally means the question stops being "which headers are special" and becomes "which step's output won", which is the question that actually determines whether a header should survive. Keeping setPassthroughHeader deliberately distinct from HttpResponseWrapper#setHeader matters too: it would have been easy to grant response:set-header() the same bypass by accident and quietly reintroduce the leak.

The XQTS comparison above is a false alarm — please don't read the −3 as a regression. Constr-inscope-1, -3 and -4 are flaky in the harness, not affected by this change. The same four tests flip in both directions on PRs that cannot possibly affect conformance:

PR what it changes Constr-inscope delta
#6687 OASIS catalog wiring −1, +2
#6666 documentation only −1
#6661 dependency pinning −2
#6433 install-and-deploy version handling −2, +3

A documentation-only PR cannot change XQuery conformance results. I'll open a separate issue for the flakiness with this evidence, since the real cost is the reverse case — a genuine conformance regression getting waved through as "probably the flaky ones again".

@line-o
line-o force-pushed the dp-header-buffer-redesign branch from f77ea91 to c4d3d6a Compare September 14, 2026 15:43
Base automatically changed from dp-fix-urlrewrite-view-pipeline-jetty12 to develop September 14, 2026 17:28
…lus its counterpart

Two tests scoping the CachingResponseWrapper header-buffering redesign
(Option A) that follows in the next commit:

- URLRewriteResponseSetHeaderViewPipelineTest is red against current
  code: a header an intermediate pipeline step sets programmatically via
  XQuery's response:set-header() -- not through controller.xql's static
  <exist:set-header> config -- currently leaks straight through to the
  client even though a view step discards that step's own output
  afterward. Same shape of bug as Content-Length (#6669), Content-Type,
  and Last-Modified/ETag/Accept-Ranges (already fixed on the branch this
  stacks on), just with no fixed header name an allowlist could ever
  enumerate in advance -- the actual reason to move past "suppress these
  specific known-bad names" to a structural distinction.

- URLRewriteFinalStepResponseSetHeaderSurvivesTest is a characterization
  test, green both before and after: it pins the case that broke a first
  attempt at this redesign (an allowlist-based "suppress anything not in
  controller.xql's static config" approach) -- a header the *final*,
  actually-flushed step sets the same way, with no static config
  representation at all, must still reach the client. cache=true means
  "a view or error-handler exists somewhere in the pipeline," not "this
  step is provisional," so config-presence alone can't tell final from
  intermediate.
Supersedes the fixed allowlist (Content-Length, Content-Type,
Last-Modified, ETag, Accept-Ranges) with a structural fix: split the two
kinds of header write CachingResponseWrapper was conflating.

- Pipeline configuration (controller.xql's <exist:set-header>, applied
  by URLRewrite#setHeaders before doRewrite()'s dispatcher.forward()
  even runs) must survive regardless of which step's output eventually
  wins -- applyViews() discards each step's wrapper for a fresh one on
  the next view step and only ever flushes the last one. This now has
  its own explicit bypass: HeaderPassthroughResponse#setPassthroughHeader,
  implemented by CachingResponseWrapper to tunnel through however many
  wrapper layers are stacked until it reaches a response that isn't one
  of these wrappers, applying the header there immediately.
  HttpResponseWrapper#setPassthroughHeader is new and deliberately
  distinct from #setHeader (which backs the XQuery response:set-header()
  function) so a step's own programmatic header call keeps going through
  the normal path below, not this one.

- Everything else a dispatched step writes -- Jetty's own resource
  metadata, or the step's own response:set-header() calls -- is now
  buffered generically (setHeader/addHeader/setIntHeader/addIntHeader/
  setDateHeader/addDateHeader all record a BufferedHeader entry) and
  replayed verbatim, in original order, from flush() -- but only when
  this wrapper is the one that actually gets flushed. An intermediate
  step's headers are correctly never replayed (its wrapper is discarded
  first); a final step's headers -- config-driven or programmatic, no
  longer distinguished by name -- correctly are.

Content-Type and Content-Length keep their existing special-casing:
Content-Type via the first-write-wins contentType field, Content-Length
by being derived from the actual buffered byte count at flush time
rather than trusted from any step's self-reported value (any buffered
Content-Length entry is skipped during replay for exactly this reason).
Status codes are unaffected -- setStatus()/sendError() already passed
through immediately with a mirrored field, which is correct and load-
bearing (service()/applyViews() read getStatus() right after doRewrite()
returns), not part of this bug class.

Turns URLRewriteResponseSetHeaderViewPipelineTest from the previous
commit green, while URLRewriteFinalStepResponseSetHeaderSurvivesTest
and URLRewriteSetHeaderSurvivesViewPipelineTest -- the safety nets for
this exact redesign, added because a first attempt at it (an allowlist
derived from controller.xql's static config) broke the former -- stay
green throughout.
@line-o
line-o force-pushed the dp-header-buffer-redesign branch from c4d3d6a to 4bfad7b Compare September 14, 2026 17:28
@line-o
line-o merged commit 03449ee into develop Sep 15, 2026
21 of 22 checks passed
@line-o
line-o deleted the dp-header-buffer-redesign branch September 15, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug issue confirmed as bug performance bottlenecks, opportunities for rewriting, optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] last-modified and created header sent twice

4 participants