🐛 A need records the line it is actually written on

[chrisjsewell]

The change

NeedDirective stored self.lineno, which counts the lines of what the parser was
handed rather than the lines of the file. Three perfectly ordinary things put text
into a document and shift that count for everything after: Sphinx prepends
rst_prolog to every document it parses, and both docutils' .. include:: and our
own .. list2need:: splice text in mid-parse with StateMachine.insert_input. The
shifts compound, so the recorded line routinely landed past the end of the file.

The line now comes from SphinxDirective.get_source_info(), which is exactly
state_machine.get_source_and_line(self.lineno) — the accessor get_location(), and
so every warning the directive itself emits, is already built on. The stored
lineno now agrees with the line our own warnings have been printing all along. On
a document with an rst_prolog and a list2need, for a need written on
index.rst:9:

	before	after
what the warning printed	index.rst:9	index.rst:9
what needs.json recorded	19	9

The accessor returns (None, None) when the state machine cannot map the line, in
which case the old value is kept.

No warning location regresses. The only warning locations that move are the ones
this corrects — the sites that report at a need's stored (docname, lineno) now name
the true line. Everything anchored through the directive's own location helper is
byte-identical to before. The one place that had to be taught the
difference is the rendered content of :pre_template:/:post_template:. That content
exists in no file, so it is anchored at the need's own directive, and nested_parse
resolves a nested parse through the enclosing state machine — meaning the anchor has
to be given in the parser's line space, not in the resolved one a need now records.
NeedItemSourceDirective therefore also keeps the unresolved number (parser_lineno,
internal — not in needs.json), and _create_need_node uses it for those two offsets.
tests/test_proper_warning.py pins these locations already; its values are unchanged,
and a second test now builds the same fixture with an rst_prolog in front of it
and asserts the same five locations, which without this would report 7 and 4 where
the file says 9 and 6.

What that looks like across the three causes

A 19-line document with all three, from the new test:

need	written on	before	after
after rst_prolog only	index.rst:4	7	4
after an .. include::	index.rst:9	24	9
after a .. list2need::	index.rst:18	47	18

Deliberately not changed

• docname stays self.env.docname. Better lineno in needs JSON #1692 also derived it from the resolved
  source path; measured, that turns docname into a filesystem path for any need
  inside an .. include::d file and silently breaks the :need: hyperlinks pointing
  at it. Changing docname changes link resolution, which is a separate question
  from getting the line right.
• lineno_content stays self.content_offset + 1. It is not read as a source
  line: _create_need_node hands it to nested_parse as an offset into the parser's
  own line space, which is what lets a need nested inside another need's content report
  sensible lines. Making the two agree wants a new field rather than a change to this
  one; noted as a follow-up.
• Needs that list2need generates still record their position inside the
  generated block rather than the line of the list item that produced them — that is
  where their warnings already point, so it is no worse than before, and it is only
  properly fixed by building those needs directly instead of round-tripping them
  through the parser.
• One honest trade. For a need written inside an .. include::d file, inside a
  list2need block, or in a Python docstring, the number now indexes a different
  file from the one docname names, and it looks like an ordinary line of that
  document — where the old value announced itself as wrong by sitting past the end of
  the file. Both are wrong; the new one is wrong plausibly. It is deliberate: the new
  number is the one that need's own warnings already print, and the real fix is for a
  need to carry a source path of its own rather than for docname to change meaning.
• Markdown is a no-op. myst-parser's MockStateMachine.get_source_and_line
  returns the line it is given, so .md needs are byte-identical before and after.
  There is a test pinning that.

Tests

• tests/test_need_lineno.py (new) — one project carrying all three shift sources,
  pinning the true line for every ordinary need directive, plus the Markdown no-op.
• tests/test_list2need_behaviour.py — the lineno characterization test added in
  🧪 Pin list2need behaviour and correct its documentation #1788 now records the correct line for the need after the list, and keeps the
  generated needs' offsets pinned as the remaining residue.
• tests/__snapshots__/test_list2need.ambr — 16 generated-need linenos, each checked
  against the generated text.
• tests/test_proper_warning.py — its existing pins are untouched; a new case rebuilds
  the same fixture behind an rst_prolog and asserts the same warning locations.

Closes #1349.

Credit where it is due: #1692 (sebastiansetzer) identified this fix and its
direction. This PR carries its need.py line-number piece on its own.

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.05%. Comparing base (4e10030) to head (b0216eb).
⚠️ Report is 333 commits behind head on master.

Files with missing lines	Patch %	Lines
sphinx_needs/api/need.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1789      +/-   ##
==========================================
+ Coverage   86.87%   91.05%   +4.17%     
==========================================
  Files          56       77      +21     
  Lines        6532    11593    +5061     
==========================================
+ Hits         5675    10556    +4881     
- Misses        857     1037     +180     

Flag	Coverage Δ
pytests	91.05% <88.88%> (+4.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ubmarco]

Nice catch