Skip to content

Split an xs:list on any XML whitespace character - #1034

Open
simpleqt wants to merge 2 commits into
tafia:masterfrom
simpleqt:fix/xs-list-whitespace-delimiters
Open

simpleqt wants to merge 2 commits into
tafia:masterfrom
simpleqt:fix/xs-list-whitespace-delimiters

Conversation

@simpleqt

@simpleqt simpleqt commented Nov 4, 2026

Copy link
Copy Markdown

Root cause

ListIter in src/de/simple_type.rs:363 searched for the space byte only:

const DELIMITER: u8 = b' ';
...
return match memchr(DELIMITER, string.as_bytes()) {

The XML Schema specification defines the items of an xs:list as separated by white space (#x20, #x9, #xD, #xA, collapsed), and the doc comment of SimpleTypeDeserializer says the items "should be delimited by a space ( , \t, \r, or \n)", so tab, carriage return and line feed have to be delimiters too. They were not, and the skip of the leading and trailing delimiters (src/de/simple_type.rs:387) had the same problem.

Repro

#[derive(Deserialize)]
struct List {
    item: [Vec<String>; 1],
}

from_str::<List>("<root><item>first\tsecond</item></root>").unwrap();

gives item: [vec!["first\tsecond"]] instead of [vec!["first", "second"]]. The same happens with \n, with \r, and through a character reference, because the reader reports the contents of &#9; as a tab:

from_str::<List>("<root><item a=\"first&#9;second\"/></root>")  // 1 item, 2 expected

An attribute value behaves the same way. The character reference form matters because it is the form the writer itself produces: escape_item escapes \t, \r and \n of a list item (src/se/simple_type.rs:100), so a list written by Serializer is read back as a single item containing that character.

Leading and trailing delimiters were affected as well: <item>\tfirst</item> produced the item "\tfirst" instead of "first", and <item>\t</item> produced one empty item instead of an empty list.

Fix

Use utils::is_whitespace - the definition of XML whitespace used in the rest of the crate - for both the search and the skip, instead of the space byte. AtomicDeserializer already used the same set of characters when trimming an item.

Tests

Added each_delimiter_is_a_separator, mixed_delimiters and only_delimiters to the list unit tests, and whitespace_delimiters to tests/serde-de-seq.rs.

RED on master (the fix stashed, the tests kept):

test de::simple_type::tests::list::each_delimiter_is_a_separator ... FAILED
test de::simple_type::tests::list::mixed_delimiters ... FAILED
test de::simple_type::tests::list::only_delimiters ... FAILED
assertion failed: `(left == right)`: "abc\tdef"

and the integration test:

---- fixed_name::fixed_size::xs_list::whitespace_delimiters stdout ----
assertion failed: `(left == right)`: "<root><item>first\tlist</item></root>"

GREEN with the fix: cargo test --all-features - 1473 lib tests and all 18 integration test binaries pass, cargo fmt -- --check is clean.

Impact

ListIter only ever receives decoded text, so no document that was deserialized correctly before is deserialized differently now, except that a tab, a carriage return or a line feed that is surrounded by item characters used to be kept inside the item and is now a delimiter, as the specification requires.

`ListIter`, which deserializes a sequence of atomic values from a text
content or an attribute value, searched for `b' '` only, although the
documentation of `SimpleTypeDeserializer` states that items "should be
delimited by a space (` `, `\t`, `\r`, or `\n`)", and the XML Schema
specification defines the items of an `xs:list` as separated by white
space (`#x20`, `#x9`, `#xD`, or `#xA`, collapsed).

As a result, only the space separated items. A tab, a line feed or a
carriage return stayed inside the item:

  <item>first\tsecond</item>          -> 1 item "first\tsecond"
  <item>first\nsecond</item>          -> 1 item "first\nsecond"
  <item a="first&tafia#9;second"/>         -> 1 item "first\tsecond"

with 2 items expected in each case. Note that the reader reports the
contents of a character reference for a delimiter as that character
(`&tafia#9;` becomes `\t`), so the character reference form was affected as
well, although it is the form the serializer itself produces.

The skip of leading and trailing delimiters had the same problem, so
`<item>\tfirst</item>` produced the item "\tfirst" instead of "first",
and `<item>\t</item>` produced one item instead of an empty list.

`AtomicDeserializer` already worked with the `XsList` delimiter set
through `str::trim_matches(char::is_whitespace)`, and `utils::is_whitespace`
is the definition used for XML whitespace in the rest of the crate, so
use it here too.

Tested with `cargo test --all-features`.

Signed-off-by: simpleqt <2015900818@qq.com>
Copilot AI balanced review requested due to automatic review settings November 4, 2026 16:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

The entry was written before the pull request number was known.

Signed-off-by: simpleqt <2015900818@qq.com>
@Mingun Mingun added serde Issues related to mapping from Rust types to XML arrays Issues related to mapping XML content onto arrays using serde labels Nov 4, 2026
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.88%. Comparing base (4a56631) to head (2256444).
⚠️ Report is 23 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1034      +/-   ##
==========================================
+ Coverage   53.71%   53.88%   +0.16%     
==========================================
  Files          51       51              
  Lines       19412    19541     +129     
==========================================
+ Hits        10428    10529     +101     
- Misses       8984     9012      +28     
Flag Coverage Δ
unittests 53.88% <100.00%> (+0.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrays Issues related to mapping XML content onto arrays using serde serde Issues related to mapping from Rust types to XML

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants