Skip to content

Recover invalid collection elements and dictionary values - #180

Merged
xoofx merged 2 commits into
xoofx:mainfrom
StefanGreve:feat/invalid-item-handling
Nov 14, 2026
Merged

xoofx merged 2 commits into
xoofx:mainfrom
StefanGreve:feat/invalid-item-handling

Conversation

@StefanGreve

Copy link
Copy Markdown
Contributor

Follow-up to #178.

Why

#178 recovers a member value or constructor argument that a converter rejects, but a value that is itself an element,
such as abc in Retries: [1, abc, 3], or the value of a dictionary entry still fails the call in every mode, because
the list and dictionary converters read their items in a loop with no catch around each one. A member of an object
inside a list is recovered, since that read goes through the object converter; the element next to it is not. This is
the first follow-up #178 named.

The cases are the same as before, one level down:

  • A site generator: a page whose front matter lists tags: [docs, 2026-13-01] with a date converter on the element
    type should lose the bad tag and warn, not fail the build.
  • Configuration loaders that report every error in one pass: ports: {http: 80, https: abc} should report https
    together with every other invalid value, not stop there.
  • Tolerant readers that take the valid items and keep going.

What

No new option. Under FailAtEnd and Skip, with "drop the item" as the rule:

  • A rejected sequence element is dropped, so the collection holds the valid elements in document order.
  • A rejected dictionary value drops the whole entry.
  • Both are reported through the same InvalidValueCallback, InvalidValuesFinalizer and YamlInvalidValuesException
    as members, with the same YamlInvalidValue:
    • Path is $.Retries[1], $.Ports.b, or $[1] for a root sequence.
    • Name is the zero-based document index of an element ("1") or the key of the entry.
    • KeyStart of a dictionary value is the key; an element has no key, so its KeyStart is its own start.
    • DeclaringType is the declared collection type (List<int>, IList<int>, int[]), TargetType the element or
      value type, and Instance the collection the item would have been added to, or null for arrays and immutable
      collections, which are built afterwards.
var options = new YamlSerializerOptions
{
    InvalidValueHandling = YamlInvalidValueHandling.Skip,
    InvalidValueCallback = v => log($"{v.Path}: {v.Message}"),
};

var root = YamlSerializer.Deserialize<Config>("Retries: [1, abc, 3]\nPorts: {http: 80, https: x}\n", options);
// root.Retries is [1, 3], root.Ports is {http: 80}
// $.Retries[1]: Invalid integer scalar 'abc'
// $.Ports.https: Invalid integer scalar 'x'

Every collection kind both paths know is covered: List<T>, arrays, IEnumerable<T>, IList<T>, ICollection<T>,
IReadOnlyList<T>, IReadOnlyCollection<T>, HashSet<T>, ISet<T>, ImmutableArray<T>, ImmutableList<T>,
ImmutableHashSet<T>, classes implementing ICollection<T>, Dictionary<string, TValue>, Dictionary<TKey, TValue>,
IDictionary<,>, IReadOnlyDictionary<,>, OrderedDictionary<,>, and the List<object>, object[] and
Dictionary<string, object> containers; the Populate paths; root collections; nested collections; and dictionaries
reached through merge keys, in the reflection path and in generated code.

Semantics

The recovery rule is unchanged: only a YamlException that is not a parser error, thrown while the reader is still on
the token where the item read began. The mechanism is the one #178 uses for members. In the reflection path every loop
reads its item through one internal helper, YamlReader.TryReadItem, which holds the filtered catch once; in generated
code every collection loop gets one try per iteration, the same shape as the generated ReadMember.

  • The name comes from the path tracker. It already knows the index or key of the item the reader is on, which is
    how GetPath renders [1] and .key, so the loops carry no counters, generated code formats no keys, and FailFast
    keeps costing nothing.
  • Indices are document positions. Once an element is dropped, the elements after it have a lower index in the
    resulting collection; Path and Name keep pointing into the document, where the error is.
  • A dropped entry's key still counts as seen, as a skipped member key does: a second occurrence of the key is a
    duplicate under Error, keeps nothing under FirstWins, and a merge key (<<) does not fill it in. Dictionaries
    with keys other than string detected duplicates with ContainsKey, which cannot see a dropped entry, so they now
    remember dropped keys in a set allocated only on the recovery path, and only when duplicates are detected at all.
  • Every dictionary loop checks duplicates before reading the value. The reflection string-key path already did,
    and 8210484 did it for members; the typed-key reflection path and both generated dictionary loops read the value
    first. Moving the check is what keeps the catch shape to one try around the value read. Two consequences are
    visible outside recovery, both tested: FirstWins skips a duplicate's value without parsing it, so an invalid
    duplicate no longer throws, and Error is located at the value, as it already was for string keys.
  • Still fatal, in every mode: dictionary keys, the root value, extension data, an exception thrown after part of
    the item was consumed, other exception types, which the entry points still wrap, and parser errors. A FormatException
    from a dictionary value converter is still wrapped at the key's marks, as before.
  • Two caveats from Recover from invalid values while deserializing #178 extend to items. A generated context that lacks a reader for an element type throws
    ThrowNotSupported before consuming anything, so under Skip and FailAtEnd each element is reported as invalid
    instead of the call failing fast. And an element of a polymorphic type is buffered before the discriminator is
    resolved, so an unknown discriminator surfaces after the reader moved and stays fatal, while member values inside
    the buffered node are recovered as before.

Performance

Measured with the default FailFast against main (8210484) in a detached worktree, with the existing benchmarks.
Allocations are identical to the byte. The branch run was faster in every benchmark, and the main run, which went
first, had two to four times the standard deviation, so the differences are run-to-run noise.

Benchmark main branch Allocated (both)
SharpYaml_Deserialize_Poco 2,388 us (StdDev 188) 2,178 us (StdDev 69) 1948.39 KB
SharpYaml_Deserialize_Poco_UnknownKeys 2,431 us (StdDev 214) 2,023 us (StdDev 39) 1950.6 KB
SharpYaml_Deserialize_Poco_UnknownKeys_WithCallback 2,391 us (StdDev 78) 2,237 us (StdDev 84) 1952.4 KB
SharpYaml_SourceGenerated_Deserialize 2,370 us (StdDev 240) 2,126 us (StdDev 63) 1872.89 KB
SharpYaml_SourceGenerated_Deserialize_MergeKeys 4.153 us (StdDev 0.14) 4.129 us (StdDev 0.19) 6.87 KB

The reflection loops gain one non-inlinable call per item on top of the virtual converter call they already made; the
generated loops gain a try region per iteration, which costs nothing until something throws. The AOT smoke project
gained a check that [1, abc, null] into int?[] gives [1, null] under Skip; it publishes without trimming
warnings and exits with 0.

API surface

  • No new option and no change to YamlInvalidValue's members; its XML docs, YamlInvalidValueHandling and
    InvalidValueHandling now describe items as well as members.
  • One protected static member on YamlSerializerContext, RecoverInvalidItem(reader, exception, declaringType, instance, targetType), next to the two from Recover from invalid values while deserializing #178, because generated code lives in the user's assembly.
  • Internally: YamlReader.TryReadItem and RecoverInvalidItem, and YamlPathTracker.GetCurrentName.
  • Docs: error-handling.md describes dropped items, the key rule and the index shift; converters.md says the
    rejection rule holds wherever the value sits.
  • Tests: 64 new, every behavior in reflection and generated mode, including a matrix over twelve collection kinds,
    plus a generated-source test for the emitted loops and two duplicate-key tests for the moved check.

Follow-ups, not in this PR

  • Dictionary keys. A key that fails to convert still ends the call in every mode. Recovering it would report the key
    as the rejected value and skip both nodes, with "drop the entry" as the rule.
  • Generated string-keyed dictionaries create their seen-key set only when merge keys are enabled, so under the
    Failsafe and JSON schemas they never detect duplicates, while the reflection path keys the set on
    DuplicateKeyHandling. Pre-existing and unrelated to recovery; worth its own issue.
  • The follow-ups Recover from invalid values while deserializing #178 named stay open: reporting without exceptions, ThrowNotSupported returning a plain
    YamlException, and the lenient built-in DateOnly parsing.

Written with Claude Code (Claude Fable 5.1).

StefanGreve and others added 2 commits November 11, 2026 23:01
* Wrap every item read of the reflection collection converters in
  YamlReader.TryReadItem and every generated collection loop in one filtered
  catch, so that an element or dictionary value its converter rejects before
  consuming it is dropped and reported under FailAtEnd and Skip, as member
  values have been since xoofx#178
* Derive the report's Name from the path tracker, the document index of an
  element or the key of an entry, so the loops carry no counters, generated
  code formats no keys, and FailFast keeps costing nothing: allocations match
  main to the byte in every benchmark
* Check duplicate dictionary keys before reading the value, in the typed-key
  reflection path and in generated code, and remember dropped typed keys, so a
  dropped entry still counts as seen, FirstWins skips a duplicate's value
  without parsing it, and Error is located at the value as for string keys
* Leave dictionary keys, the root value and extension data fatal, and say so
  in the error-handling and converter docs along with the index shift a
  dropped element causes in the deserialized collection

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Remove merged fallbacks when their explicit replacement is rejected,
while retaining earlier valid explicit and populated values. Track dropped
typed keys using the populated dictionary's equality or ordering comparer.
Cover reflection and generated root, member and Populate paths with
regressions, and clarify dictionary recovery semantics in the docs.

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: CodeAlta <noreply@codealta.io>
@xoofx
xoofx merged commit d24a464 into xoofx:main Nov 14, 2026
1 check passed
@xoofx

xoofx commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Reviewed, fixed two edge cases, and squash merged as d24a464.

Fixes pushed in 903003a:

  • An invalid explicit dictionary value now suppresses its merged fallback even when the merge appears first. Earlier valid explicit values and existing populated values remain intact.
  • Dropped typed dictionary keys use the populated dictionary's equality or ordering comparer, so Error and FirstWins still recognize equivalent keys.

Added regression coverage for reflection/generated root, member and Populate paths, object dictionaries, merge sequences, case-insensitivity, and dictionary/sorted/ordered comparers; clarified the docs.

Validation: Release build with zero warnings/errors; all 1,078 tests passed; Windows native AOT publish and smoke execution passed; documentation build passed. Fable adaptive review approved the full PR and fixes with no findings, and the updated GitHub CI run passed before merging.

The pre-existing generated JSON/Failsafe duplicate-key gap and the inconsistent treatment of prepopulated string versus non-string keys are tracked separately.

This review was written by Codex through CodeAlta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants