Skip to content

fix: Report a Spark analysis failure as a 422 naming the subject - #2726

Open
fhnaumann wants to merge 2 commits into
release/server/3.0.0from
issue/2723
Open

fix: Report a Spark analysis failure as a 422 naming the subject#2726
fhnaumann wants to merge 2 commits into
release/server/3.0.0from
issue/2723

Conversation

@fhnaumann

@fhnaumann fhnaumann commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Fixes #2723.

Problem

Every Spark analysis failure raised by a $sql-run or $sql-export subject reached the caller as
an opaque 500:

{"resourceType":"OperationOutcome","issue":[{"severity":"error","code":"processing","diagnostics":"Unexpected error occurred"}]}

AnalysisException extends java.lang.Exception and matches no clause in
ErrorHandlingInterceptor.convertError, so it fell through to the terminal
catch (Throwable) -> internalServerError branch, which discards the cause. An unresolved column,
an unknown function, a missing GROUP BY, an ambiguous reference and a datatype mismatch were all
byte-identical to a genuine infrastructure fault. SqlValidator.validateFunctionName deliberately
leaves unknown function names for Spark's analyser on the grounds that its message "is more helpful
than a synthetic rejection" — but that message never reached anyone.

Change

An analysis failure is now translated inside operations/sql/ into a 422 carrying
issue.code = invalid, expression = ["subject"] and Spark's own analyser message. That is what
sql-run.md's status table already defines 422 to mean: "The subject is of no admitted kind, or
is conformant but cannot be processed."
An unresolved column is a conformant SQLQuery Library
that cannot be processed, and 400 stays reserved for the parameter-level faults that table lists.

Three files, one new helper and two (well, three) call sites:

  • SqlOperationError.executionFailure(Exception) — returns a 422 via the existing
    unprocessable(...) helper for an AnalysisException, and returns anything else unaltered so it
    still renders as a 500. The diagnostics come from getSimpleMessage() rather than
    getMessage(): the latter appends the whole unresolved logical plan, which is unbounded, names
    the internal request-scoped views the dependency graph was materialised under, and tells the
    caller nothing they can act on. The result is length-bounded as well, since the "did you mean"
    list is drawn from the subject's columns and a wide dependency makes it long.
  • SqlRunProvider.runSqlLibrary — wraps pipeline.execute(...). The dataset is analysed inside
    execute before the streaming consumer writes a byte, so the response is uncommitted and the
    status is still free to change.
  • SqlExportExecutor.runSql and runView — the same, beside the ConstraintViolationException
    catch that already demonstrates the pattern.

AnalysisException is a Scala-declared checked exception that no signature on the call path
declares, so it cannot be named in a catch clause. It is matched by type instead, following the
precedent in ExportFileWriter.writeCsv.

Why not ErrorHandlingInterceptor

DeltaAnalysisException extends AnalysisException, so a blanket catch in the interceptor would have
to sit after the existing Delta carve-out and would silently widen behaviour across the import and
Delta paths, where an analysis failure genuinely is a server-side fault. The translation is
deliberately confined to operations/sql/.

Why $sql-export needs no JobProvider change

JobProvider renders a failed job with ErrorHandlingInterceptor.convertError(...), the same
converter as a synchronous request, and its catch (BaseServerResponseException) branch passes a
typed exception through unaltered when its status is non-zero. SqlOperationError produces an
UnprocessableEntityException, so a 422 thrown on the job thread survives the async round-trip
with its status and OperationOutcome intact.

What deliberately does not change

  • Runtime (non-analysis) errors stay 500. 1/0, a failing CAST and friends fire inside the
    streaming consumer, after the response may already be committed, so the status cannot be
    rewritten. Asserted negatively in the tests.
  • The three statically-detected faults stay 400 — malformed SQL, an undeclared table, a
    disallowed operation. Asserted in SqlRunProviderIT.
  • Kick-off validation for $sql-export is unchanged. An analysis error is still accepted with a
    202 and surfaces as a deferred failure; this PR only makes that failure legible. Filed
    separately as $sql-export does not honour its documented kick-off validation guarantee #2724.

Testing

SqlRunProviderTest — an analysis failure becomes a 422 naming the subject with the message
intact; a non-analysis failure propagates untranslated and is not a BaseServerResponseException.

SqlExportExecutorTest — the same pair, with propagatesAFailingSubject strengthened to assert the
untranslated propagation.

SqlRunProviderIT — real Spark analysis failures end to end over the wire: an unresolved column
(asserting the UNRESOLVED_COLUMN condition and the analyser's "did you mean" suggestions reach
diagnostics), an unresolved column nested in a subquery, an unknown function, and a regression
guard that the three statically-detected faults are still 400.

SqlRunProviderTest, SqlExportExecutorTest    38 tests, 0 failures
SqlRunProviderIT, SqlExportProviderIT        29 tests, 0 failures

Verified by hand

Against a locally built image, POST /fhir/$sql-run with an inline SQLQuery Library subject over
an inline ViewDefinition context projecting pid, gender, bd.

SQL Before After
SELECT no_such_col FROM t 500 opaque 422 [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... Did you mean one of the following? [\bd`, `gender`, `pid`]. SQLSTATE: 42703; line 1 pos 7`
SELECT * FROM (SELECT no_such_col FROM t) x LIMIT 0 500 opaque 422 same condition, line 1 pos 22
SELECT no_such_fn(pid) FROM t 500 opaque 422 [UNRESOLVED_ROUTINE] Cannot resolve routine \no_such_fn` ... SQLSTATE: 42883`
SELECT pid FROM t GROUP BY nonexistent 500 opaque 422 [UNRESOLVED_COLUMN.WITH_SUGGESTION] ...
SELECT pid, count(*) FROM t 500 opaque 422 [MISSING_GROUP_BY] The query does not include a GROUP BY clause ...
SELECT pid FROM t a, t b 500 opaque 422 [AMBIGUOUS_REFERENCE] Reference \pid` is ambiguous, could be: [`a`.`pid`, `b`.`pid`]`
SELECT pid[1] FROM t 500 opaque 422 [INVALID_EXTRACT_BASE_FIELD_TYPE] ...
SELEKT pid FROM t 400 400 unchanged, PARSE_SYNTAX_ERROR
SELECT x FROM nope 400 400 unchanged, undeclared table
DROP TABLE t 400 400 unchanged, disallowed operation
SELECT 1/0 FROM t 500 500 unchanged — runtime, out of scope
SELECT CAST(pid AS INT) FROM t 500 500 unchanged — runtime, out of scope
SELECT pid, gender FROM t LIMIT 2 200 200 unchanged

Every issue carries "code":"invalid" and "expression":["subject"].

fhnaumann added 2 commits August 11, 2026 12:28
A fault in a subject's own SQL - an unresolved column, an unknown function, a
missing GROUP BY, an ambiguous reference - is caught by Spark's analyser rather
than by SqlValidator. AnalysisException matches no clause in
ErrorHandlingInterceptor.convertError, so it fell through to the terminal
catch (Throwable) branch and reached the caller as an opaque 500 with the
message discarded, indistinguishable from a genuine server fault.

Translate it inside operations/sql/ instead: a 422 carrying issue.code = invalid,
expression = ["subject"] and Spark's own analyser message, which is what
sql-run.md already defines 422 to mean - a subject that is conformant but cannot
be processed. This also restores the behaviour SqlValidator.validateFunctionName
documents as intended, where an unknown function name is left for Spark's
analyser because its message is more helpful than a synthetic rejection.

The translation stays out of ErrorHandlingInterceptor. DeltaAnalysisException
extends AnalysisException, so a blanket catch there would have to sit after the
existing Delta carve-out and would widen behaviour across the import and Delta
paths, where an analysis failure genuinely is a server-side fault.

Runtime errors are unaffected and stay 500: they are raised inside the streaming
consumer, where the response may already be committed and the status can no
longer be rewritten.

Closes #2723
AnalysisException.getMessage appends the whole unresolved logical plan. It is
unbounded, it names the internal request-scoped views the dependency graph was
materialised under, and it tells the caller nothing they can act on. Return
getSimpleMessage instead, which is the same condition, position and "did you
mean" suggestions without the plan, and bound its length, since the suggestion
list is drawn from the subject's columns and a wide dependency makes it long.

Observed against a locally built image, `SELECT no_such_col FROM t`:

  before: [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... SQLSTATE: 42703; line 1 pos 7;
          'Project ['no_such_col]
          +- SubqueryAlias sqlquery_rko3ryqtta2xodqx_https___example_org_...
             +- View (`sqlquery_rKo3ryQTTa2xoDQx_https___example_org_...

  after:  [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function
          parameter with name `no_such_col` cannot be resolved. Did you mean one
          of the following? [`bd`, `gender`, `pid`]. SQLSTATE: 42703; line 1 pos 7
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant