✨ list2need: build the needs directly, without generating reStructuredText - #1790
Merged
Conversation
This was referenced Aug 26, 2026
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1790 +/- ##
==========================================
+ Coverage 86.87% 91.03% +4.15%
==========================================
Files 56 77 +21
Lines 6532 11678 +5146
==========================================
+ Hits 5675 10631 +4956
- Misses 857 1047 +190
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…dText The directive rendered every item into a need directive through a template and handed the result back to the parser with ``state_machine.insert_input``. That one call is the reason a list in a MyST document produced nothing -- myst-parser's mock state machine does not implement it -- and the reason every need from the directive onwards was recorded at the wrong line: ``insert_input`` advances the parser's flat line counter by the length of the generated text, for the rest of the file (#1349). The items are now built with ``add_need``, and a nested item's nodes are placed inside its parent's. The syntax and its parsing are untouched: the same four regular expressions, the same level to type mapping, the same delimiter split, the same ``((option="value"))`` extraction and the same validation. Beyond losing those artefacts, one behaviour changes deliberately. Writing ``()`` used to suppress the ``:id:`` of the generated need, which sent it down a second id function -- one that honours ``needs_id_from_title``, where list2need's own does not -- so a list could carry two id schemes at once, selected by two characters of punctuation. The directive now derives every id itself, and ``()`` means exactly what no brackets at all means.
A continuation line is stripped of the indentation it was written with, so the options of a directive written in an item's content would land in the directive's own column -- which docutils reads as a directive followed by an unindented field list, reporting "Explicit markup ends without a blank line" and dropping the options. The three spaces the parser prepends to a line starting with ":" are what put them back underneath it, which is what makes the "rst-directives in lists" example in the documentation work. The indent looked like damage control for the generated text the directive no longer produces, and had no test. It has one now.
The "Markdown (MyST)" section said the directive works in reStructuredText documents only; it now shows the fence, says what the host parses and what list2need parses, and keeps the eval-rst block as the other way to write it. "Generated IDs" gains a versionchanged for the one behaviour change: a line whose bracketed group is empty now takes its ID from the formula documented there, like a line with no brackets at all. The known-limitations warning names what a content line starting with "*" or ":" actually does, and the "presentation" option no longer describes nesting as a property of the parent's content.
The changelog listed the three-space indent of a ":" content line among the things the template step cost, and the directive's known-limitations warning said such a line is rendered as a field list. Neither is right: the indent is kept, and what it is for is stated instead.
Building the needs directly means the directive decides four things the generated need directive used to decide, and each was decided wrongly: - A ``hide``\den item is taken out of the document once it has been read, so a child nested inside one was created, recorded in needs.json, and then rendered nowhere -- every ``:need:`` reference to it pointed at an anchor that never reached the page. A hidden item is no longer opened as a parent. - ``add_doc`` registered the document whenever the list parsed, even when every one of its items was refused and the document ended up with no needs. - ``title_from_content`` was not among the options an item may carry, so an item with no title of its own did not take the first sentence of its content the way a need directive does. The title is now decided by the need directives' own ``_get_title``, which also restores the "No title given" warning for an item whose title is only whitespace. - ``((id=""))`` was silently ignored and the title hash used instead. The empty value is now handed on, and refused with a diagnostic like any other invalid id. Each is pinned by a test that fails without its fix.
The migration text named two kinds of line that get a different generated ID and missed the widest of the three: one written "()" that also carries an ((option="value")) area. That one needs no configuration to reach, because the ID list2need derives is the hash of the title before the option area is removed from it, where the generator it used to fall through to hashed the title after. Named in the changelog and in the versionchanged, with the measured pair. The changelog also now lists the four inputs that used to be errors and are now defined -- an inline ((id=...)) among them -- and the source-mapping docstring says that its question is whether the content is source-mapped, not whether it is reStructuredText.
Three of the options in CORE_OPTIONS never reach the set it is matched against, because a need directive reads them as booleans and so does this directive. The set stays a mirror of what a need accepts; the comment now says which members are taken out earlier.
test_need_lineno pinned them at their offsets inside the block list2need used to generate -- 1 and 7 -- with a note saying that only building the needs directly would fix it. It does: the two items are written on lines 15 and 16 of that fixture, and that is what they now record, with an rst_prolog and an .. include:: above them. They are the strongest of the five assertions in that test, because they are the only ones not resolved back through get_source_info(): the directive gives each need the line of its own item, from the source mapping of its content.
A need carries two numbers that are not source lines: lineno_content, the offset its content is parsed at, and parser_lineno, the offset a rendered pre/post template is parsed at. Both count in the parser's own line space, and this directive was giving them as resolved source lines, so a document with an rst_prolog or an .. include:: above the list reported a warning inside an item's content, or inside its template, several lines away from where it is written -- by the amount of text inserted above. Measured on a list item carrying a pre_template and a bad role in its content: without a prolog both errors point at the right lines; with one they moved to 5 and 6. They now point at the same lines either way. The lineno a need records is unaffected: that is a source line, and stays the line of the item.
chrisjsewell
force-pushed
the
list2need-direct-construction
branch
from
August 26, 2026 09:43
061df40 to
e495ac9
Compare
A need carries two numbers that are not source lines -- lineno_content, the offset its content is parsed at, and parser_lineno, the offset a rendered pre/post template is parsed at. Both are offsets into the parser's own counter, and nothing pinned that list2need gives them in that space: with the directive handing over resolved source lines instead, every list2need test still passed. The case that catches it needs a document whose counter has been shifted, so this builds the same list twice, with and without a two-line rst_prolog, and asserts both warnings land on the same lines either way -- the content role on line 8, where it is written, and the template role on line 9. Without the prolog the two spaces coincide and the assertion holds regardless, which is why one project alone was never going to pin it.
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
list2needrenders every item into a need directive through a template and hands theresult back to the parser with
state_machine.insert_input. This replaces that step:the items are built with
add_need, and a nested item is placed inside its parent.The syntax and its parsing are untouched. The same four regular expressions, the
same level→type mapping, the same
(ID)capture, the same:delimiter:split, the same((option="value"))extraction, the same:tags:merge, the same:links-down:walk,the same validation and the same eleven inputs that end the build. Only the generation
mechanism changes.
This is offered as the answer to the question in #1427 — "can we save the syntax
(mostly) but have a better implementation (without jinja)?"
What the step cost
Markdown. myst-parser's mock state machine does not implement
insert_input, so a{list2need}fence produced no needs at all — only an error naming a myst-parserinternal. It now works:
MD-A,MD-BMD-B.parent_need == "MD-A"MockingError: MockStateMachine has not yet implemented attribute 'insert_input'Line numbers (#1349, the
list2needhalf).insert_inputadvances the parser's flatline counter by the length of the generated text, for the rest of the file, and the error
compounds with each list. For a document with two items on lines 10 and 11 and an
ordinary
.. req::on line 13:linenobeforeLN-BEFORELN-ALN-BLN-AFTER(an ordinary.. req::)In the repository's own
doc_list2needfixture,NEED-Zwas recorded at line 45 of a19-line file. The general case — a
linenoshifted byrst_prologor an.. include::— is a separate small PR, #1789.Template artefacts. An item written
()with options lost them: suppressing the:id:left a blank line where it would have been, which ended the generated need'soption block, so every option after it became body text. And a parent need's stored
contentwas the generated reStructuredText of its children rather than its own text,which
needs.jsonand any filter readingcontentsaw.The one deliberate change to an input that worked
Writing
()used to suppress the:id:of the generated need, which sent it down theneed directives' own id generator — a different function, which honours
needs_id_from_titlewherelist2need's does not, hashes the content when the titleis empty, and hashes the title after the
((option="value"))area has been removedfrom it. One list could carry two id schemes at once, chosen by two characters of
punctuation. The directive now derives every id itself, from the same input at the same
point, so
()means what no brackets at all means.Three kinds of item therefore get a different generated id:
* ()Alpha titlewithneeds_id_from_title = TrueR_ALPHAR_D1EC6, the plain hash* (). some content here(no title)R_EBC29, the hash of the contentR_DA39A, the hash of the empty title* ()Title with empty parens ((status="open"))R_6AFF7R_D7997— the id* Title with empty parens ((status="open"))already hadThe third needs no configuration to reach and is the one to watch for; the changelog
says so under Breaking changes, and the documentation carries a
versionchanged.Everything else keeps the id it had, which the untouched id tests prove.
Four inputs that were errors, and are now defined
None of these can appear in a document that builds cleanly today, so nothing that
works can regress; they are listed because each is a decision the template used to
delegate to docutils and the directive now has to make.
((id="MY-ID"))names the need. The template wrote the derived id and theinline one as two
:id:lines of one generated need, which docutils refused withduplicate option "id", dropping the item. The id is applied before:links-down:is built, so the other items' links agree with it. An empty
((id=""))is refusedwith a diagnostic rather than quietly replaced by the title hash.
:links-down:—((links="X"))under
:links-down: links— produced one corrupt link value (the template's{%- for %}glued the two field lines into:links: C-2:links: X) and anunknown outgoing linkwarning. The two sets are now merged.((title_from_content="true"))is read as a need directive reads it.((hide="true"))item is rendered at the level above it. A hiddenneed is taken out of the document once read, so nesting a child inside one would put
its target where no page can reach it.
Evidence
tests/doc_test/doc_list2need/is byte-identical before andafter, all four pages, apart from the per-build random
SNCB-…container ids. Itsneeds.jsonkeeps all 16 ids, every title, status, tag, link andparent_need; onlythe linenos and the two parent
contentstrings change.inputs and the
links-downlevel-skip test pass untouched — this PR takes neitherof those on. Four assertions are edited, each one pinning something the template
did: the drifted linenos, the MyST failure (which inverts), the parent's
generated-text content, and the second id function. A fifth prediction went the other
way (below).
Linenos inside an
{eval-rst}fence were wrong too — that block was the only way toreach the directive from a Markdown document, and its needs were recorded at 10 and
16 for items written on 7 and 8.
tests/__snapshots__/test_list2need.ambris rewritten: everylinenoin it was wrong.()-plus-options id, a directive written in an item'scontent, and one for each of the four inputs above.
One assertion moved the other way. Removing the three-space indent the parser prepends to
a
:-continuation line looked like removing a workaround for the template — it is not:it is the only thing standing in for the indentation
lstripremoves, and without it theoptions of a directive written in an item's content (the
rst-directives in listsexample in the documentation) land in the directive's own column and are dropped. It is
kept, and now has a test.
Refs #1427, Refs #1349.