Repository navigation
Recover invalid collection elements and dictionary values - #180
Conversation
* Wrap every item read of the reflection collection converters in YamlReader.TryReadItem and every generated collection loop in one filtered catch, so that an element or dictionary value its converter rejects before consuming it is dropped and reported under FailAtEnd and Skip, as member values have been since xoofx#178 * Derive the report's Name from the path tracker, the document index of an element or the key of an entry, so the loops carry no counters, generated code formats no keys, and FailFast keeps costing nothing: allocations match main to the byte in every benchmark * Check duplicate dictionary keys before reading the value, in the typed-key reflection path and in generated code, and remember dropped typed keys, so a dropped entry still counts as seen, FirstWins skips a duplicate's value without parsing it, and Error is located at the value as for string keys * Leave dictionary keys, the root value and extension data fatal, and say so in the error-handling and converter docs along with the index shift a dropped element causes in the deserialized collection Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Remove merged fallbacks when their explicit replacement is rejected, while retaining earlier valid explicit and populated values. Track dropped typed keys using the populated dictionary's equality or ordering comparer. Cover reflection and generated root, member and Populate paths with regressions, and clarify dictionary recovery semantics in the docs. Co-authored-by: Codex <noreply@openai.com> Co-authored-by: CodeAlta <noreply@codealta.io>
|
Reviewed, fixed two edge cases, and squash merged as d24a464. Fixes pushed in 903003a:
Added regression coverage for reflection/generated root, member and Populate paths, object dictionaries, merge sequences, case-insensitivity, and dictionary/sorted/ordered comparers; clarified the docs. Validation: Release build with zero warnings/errors; all 1,078 tests passed; Windows native AOT publish and smoke execution passed; documentation build passed. Fable adaptive review approved the full PR and fixes with no findings, and the updated GitHub CI run passed before merging. The pre-existing generated JSON/Failsafe duplicate-key gap and the inconsistent treatment of prepopulated string versus non-string keys are tracked separately. This review was written by Codex through CodeAlta. |
Follow-up to #178.
Why
#178 recovers a member value or constructor argument that a converter rejects, but a value that is itself an element,
such as
abcinRetries: [1, abc, 3], or the value of a dictionary entry still fails the call in every mode, becausethe list and dictionary converters read their items in a loop with no catch around each one. A member of an object
inside a list is recovered, since that read goes through the object converter; the element next to it is not. This is
the first follow-up #178 named.
The cases are the same as before, one level down:
tags: [docs, 2026-13-01]with a date converter on the elementtype should lose the bad tag and warn, not fail the build.
ports: {http: 80, https: abc}should reporthttpstogether with every other invalid value, not stop there.
What
No new option. Under
FailAtEndandSkip, with "drop the item" as the rule:InvalidValueCallback,InvalidValuesFinalizerandYamlInvalidValuesExceptionas members, with the same
YamlInvalidValue:Pathis$.Retries[1],$.Ports.b, or$[1]for a root sequence.Nameis the zero-based document index of an element ("1") or the key of the entry.KeyStartof a dictionary value is the key; an element has no key, so itsKeyStartis its own start.DeclaringTypeis the declared collection type (List<int>,IList<int>,int[]),TargetTypethe element orvalue type, and
Instancethe collection the item would have been added to, ornullfor arrays and immutablecollections, which are built afterwards.
Every collection kind both paths know is covered:
List<T>, arrays,IEnumerable<T>,IList<T>,ICollection<T>,IReadOnlyList<T>,IReadOnlyCollection<T>,HashSet<T>,ISet<T>,ImmutableArray<T>,ImmutableList<T>,ImmutableHashSet<T>, classes implementingICollection<T>,Dictionary<string, TValue>,Dictionary<TKey, TValue>,IDictionary<,>,IReadOnlyDictionary<,>,OrderedDictionary<,>, and theList<object>,object[]andDictionary<string, object>containers; thePopulatepaths; root collections; nested collections; and dictionariesreached through merge keys, in the reflection path and in generated code.
Semantics
The recovery rule is unchanged: only a
YamlExceptionthat is not a parser error, thrown while the reader is still onthe token where the item read began. The mechanism is the one #178 uses for members. In the reflection path every loop
reads its item through one internal helper,
YamlReader.TryReadItem, which holds the filtered catch once; in generatedcode every collection loop gets one
tryper iteration, the same shape as the generatedReadMember.how
GetPathrenders[1]and.key, so the loops carry no counters, generated code formats no keys, andFailFastkeeps costing nothing.
resulting collection;
PathandNamekeep pointing into the document, where the error is.duplicate under
Error, keeps nothing underFirstWins, and a merge key (<<) does not fill it in. Dictionarieswith keys other than
stringdetected duplicates withContainsKey, which cannot see a dropped entry, so they nowremember dropped keys in a set allocated only on the recovery path, and only when duplicates are detected at all.
and 8210484 did it for members; the typed-key reflection path and both generated dictionary loops read the value
first. Moving the check is what keeps the catch shape to one
tryaround the value read. Two consequences arevisible outside recovery, both tested:
FirstWinsskips a duplicate's value without parsing it, so an invalidduplicate no longer throws, and
Erroris located at the value, as it already was for string keys.the item was consumed, other exception types, which the entry points still wrap, and parser errors. A
FormatExceptionfrom a dictionary value converter is still wrapped at the key's marks, as before.
ThrowNotSupportedbefore consuming anything, so underSkipandFailAtEndeach element is reported as invalidinstead of the call failing fast. And an element of a polymorphic type is buffered before the discriminator is
resolved, so an unknown discriminator surfaces after the reader moved and stays fatal, while member values inside
the buffered node are recovered as before.
Performance
Measured with the default
FailFastagainstmain(8210484) in a detached worktree, with the existing benchmarks.Allocations are identical to the byte. The branch run was faster in every benchmark, and the
mainrun, which wentfirst, had two to four times the standard deviation, so the differences are run-to-run noise.
SharpYaml_Deserialize_PocoSharpYaml_Deserialize_Poco_UnknownKeysSharpYaml_Deserialize_Poco_UnknownKeys_WithCallbackSharpYaml_SourceGenerated_DeserializeSharpYaml_SourceGenerated_Deserialize_MergeKeysThe reflection loops gain one non-inlinable call per item on top of the virtual converter call they already made; the
generated loops gain a
tryregion per iteration, which costs nothing until something throws. The AOT smoke projectgained a check that
[1, abc, null]intoint?[]gives[1, null]underSkip; it publishes without trimmingwarnings and exits with 0.
API surface
YamlInvalidValue's members; its XML docs,YamlInvalidValueHandlingandInvalidValueHandlingnow describe items as well as members.protected staticmember onYamlSerializerContext,RecoverInvalidItem(reader, exception, declaringType, instance, targetType), next to the two from Recover from invalid values while deserializing #178, because generated code lives in the user's assembly.YamlReader.TryReadItemandRecoverInvalidItem, andYamlPathTracker.GetCurrentName.error-handling.mddescribes dropped items, the key rule and the index shift;converters.mdsays therejection rule holds wherever the value sits.
plus a generated-source test for the emitted loops and two duplicate-key tests for the moved check.
Follow-ups, not in this PR
as the rejected value and skip both nodes, with "drop the entry" as the rule.
Failsafe and JSON schemas they never detect duplicates, while the reflection path keys the set on
DuplicateKeyHandling. Pre-existing and unrelated to recovery; worth its own issue.ThrowNotSupportedreturning a plainYamlException, and the lenient built-inDateOnlyparsing.Written with Claude Code (Claude Fable 5.1).