fix: Report a Spark analysis failure as a 422 naming the subject - #2726
Open
fhnaumann wants to merge 2 commits into
Open
fix: Report a Spark analysis failure as a 422 naming the subject#2726fhnaumann wants to merge 2 commits into
fhnaumann wants to merge 2 commits into
Conversation
added 2 commits
August 11, 2026 12:28
A fault in a subject's own SQL - an unresolved column, an unknown function, a missing GROUP BY, an ambiguous reference - is caught by Spark's analyser rather than by SqlValidator. AnalysisException matches no clause in ErrorHandlingInterceptor.convertError, so it fell through to the terminal catch (Throwable) branch and reached the caller as an opaque 500 with the message discarded, indistinguishable from a genuine server fault. Translate it inside operations/sql/ instead: a 422 carrying issue.code = invalid, expression = ["subject"] and Spark's own analyser message, which is what sql-run.md already defines 422 to mean - a subject that is conformant but cannot be processed. This also restores the behaviour SqlValidator.validateFunctionName documents as intended, where an unknown function name is left for Spark's analyser because its message is more helpful than a synthetic rejection. The translation stays out of ErrorHandlingInterceptor. DeltaAnalysisException extends AnalysisException, so a blanket catch there would have to sit after the existing Delta carve-out and would widen behaviour across the import and Delta paths, where an analysis failure genuinely is a server-side fault. Runtime errors are unaffected and stay 500: they are raised inside the streaming consumer, where the response may already be committed and the status can no longer be rewritten. Closes #2723
AnalysisException.getMessage appends the whole unresolved logical plan. It is
unbounded, it names the internal request-scoped views the dependency graph was
materialised under, and it tells the caller nothing they can act on. Return
getSimpleMessage instead, which is the same condition, position and "did you
mean" suggestions without the plan, and bound its length, since the suggestion
list is drawn from the subject's columns and a wide dependency makes it long.
Observed against a locally built image, `SELECT no_such_col FROM t`:
before: [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... SQLSTATE: 42703; line 1 pos 7;
'Project ['no_such_col]
+- SubqueryAlias sqlquery_rko3ryqtta2xodqx_https___example_org_...
+- View (`sqlquery_rKo3ryQTTa2xoDQx_https___example_org_...
after: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function
parameter with name `no_such_col` cannot be resolved. Did you mean one
of the following? [`bd`, `gender`, `pid`]. SQLSTATE: 42703; line 1 pos 7
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes #2723.
Problem
Every Spark analysis failure raised by a
$sql-runor$sql-exportsubject reached the caller asan opaque
500:{"resourceType":"OperationOutcome","issue":[{"severity":"error","code":"processing","diagnostics":"Unexpected error occurred"}]}AnalysisExceptionextendsjava.lang.Exceptionand matches no clause inErrorHandlingInterceptor.convertError, so it fell through to the terminalcatch (Throwable) -> internalServerErrorbranch, which discards the cause. An unresolved column,an unknown function, a missing
GROUP BY, an ambiguous reference and a datatype mismatch were allbyte-identical to a genuine infrastructure fault.
SqlValidator.validateFunctionNamedeliberatelyleaves unknown function names for Spark's analyser on the grounds that its message "is more helpful
than a synthetic rejection" — but that message never reached anyone.
Change
An analysis failure is now translated inside
operations/sql/into a422carryingissue.code = invalid,expression = ["subject"]and Spark's own analyser message. That is whatsql-run.md's status table already defines422to mean: "The subject is of no admitted kind, oris conformant but cannot be processed." An unresolved column is a conformant SQLQuery
Librarythat cannot be processed, and
400stays reserved for the parameter-level faults that table lists.Three files, one new helper and two (well, three) call sites:
SqlOperationError.executionFailure(Exception)— returns a422via the existingunprocessable(...)helper for anAnalysisException, and returns anything else unaltered so itstill renders as a
500. The diagnostics come fromgetSimpleMessage()rather thangetMessage(): the latter appends the whole unresolved logical plan, which is unbounded, namesthe internal request-scoped views the dependency graph was materialised under, and tells the
caller nothing they can act on. The result is length-bounded as well, since the "did you mean"
list is drawn from the subject's columns and a wide dependency makes it long.
SqlRunProvider.runSqlLibrary— wrapspipeline.execute(...). The dataset is analysed insideexecutebefore the streaming consumer writes a byte, so the response is uncommitted and thestatus is still free to change.
SqlExportExecutor.runSqlandrunView— the same, beside theConstraintViolationExceptioncatch that already demonstrates the pattern.
AnalysisExceptionis a Scala-declared checked exception that no signature on the call pathdeclares, so it cannot be named in a
catchclause. It is matched by type instead, following theprecedent in
ExportFileWriter.writeCsv.Why not
ErrorHandlingInterceptorDeltaAnalysisException extends AnalysisException, so a blanket catch in the interceptor would haveto sit after the existing Delta carve-out and would silently widen behaviour across the import and
Delta paths, where an analysis failure genuinely is a server-side fault. The translation is
deliberately confined to
operations/sql/.Why
$sql-exportneeds noJobProviderchangeJobProviderrenders a failed job withErrorHandlingInterceptor.convertError(...), the sameconverter as a synchronous request, and its
catch (BaseServerResponseException)branch passes atyped exception through unaltered when its status is non-zero.
SqlOperationErrorproduces anUnprocessableEntityException, so a422thrown on the job thread survives the async round-tripwith its status and
OperationOutcomeintact.What deliberately does not change
500.1/0, a failingCASTand friends fire inside thestreaming consumer, after the response may already be committed, so the status cannot be
rewritten. Asserted negatively in the tests.
400— malformed SQL, an undeclared table, adisallowed operation. Asserted in
SqlRunProviderIT.$sql-exportis unchanged. An analysis error is still accepted with a202and surfaces as a deferred failure; this PR only makes that failure legible. Filedseparately as $sql-export does not honour its documented kick-off validation guarantee #2724.
Testing
SqlRunProviderTest— an analysis failure becomes a422naming the subject with the messageintact; a non-analysis failure propagates untranslated and is not a
BaseServerResponseException.SqlExportExecutorTest— the same pair, withpropagatesAFailingSubjectstrengthened to assert theuntranslated propagation.
SqlRunProviderIT— real Spark analysis failures end to end over the wire: an unresolved column(asserting the
UNRESOLVED_COLUMNcondition and the analyser's "did you mean" suggestions reachdiagnostics), an unresolved column nested in a subquery, an unknown function, and a regressionguard that the three statically-detected faults are still
400.Verified by hand
Against a locally built image,
POST /fhir/$sql-runwith an inline SQLQueryLibrarysubject overan inline
ViewDefinitioncontext projectingpid,gender,bd.SELECT no_such_col FROM t[UNRESOLVED_COLUMN.WITH_SUGGESTION] ... Did you mean one of the following? [\bd`, `gender`, `pid`]. SQLSTATE: 42703; line 1 pos 7`SELECT * FROM (SELECT no_such_col FROM t) x LIMIT 0line 1 pos 22SELECT no_such_fn(pid) FROM t[UNRESOLVED_ROUTINE] Cannot resolve routine \no_such_fn` ... SQLSTATE: 42883`SELECT pid FROM t GROUP BY nonexistent[UNRESOLVED_COLUMN.WITH_SUGGESTION] ...SELECT pid, count(*) FROM t[MISSING_GROUP_BY] The query does not include a GROUP BY clause ...SELECT pid FROM t a, t b[AMBIGUOUS_REFERENCE] Reference \pid` is ambiguous, could be: [`a`.`pid`, `b`.`pid`]`SELECT pid[1] FROM t[INVALID_EXTRACT_BASE_FIELD_TYPE] ...SELEKT pid FROM tPARSE_SYNTAX_ERRORSELECT x FROM nopeDROP TABLE tSELECT 1/0 FROM tSELECT CAST(pid AS INT) FROM tSELECT pid, gender FROM t LIMIT 2Every issue carries
"code":"invalid"and"expression":["subject"].