Skip to content

GH-145668: Add FOR_ITER specialization for virtual iterators. Specialize GET_ITER.#147967

Open
markshannon wants to merge 13 commits intopython:mainfrom
markshannon:specialize-iteration-with-jit
Open

GH-145668: Add FOR_ITER specialization for virtual iterators. Specialize GET_ITER.#147967
markshannon wants to merge 13 commits intopython:mainfrom
markshannon:specialize-iteration-with-jit

Conversation

@markshannon
Copy link
Copy Markdown
Member

@markshannon markshannon commented Apr 1, 2026

Add FOR_ITER specialization for virtual iterators. Specialize GET_ITER.

  • Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
  • Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
  • Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
  • Add tp_iteritem internal slot to PyTypeObject to support a wider range of classes as virtual iterators.

This PR adds strs as virtual iterators, with the potential to add bytes, bytearray, frozenset, frozendict and others in the future.


📚 Documentation preview 📚: https://cpython-previews--147967.org.readthedocs.build/

* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
@Fidget-Spinner
Copy link
Copy Markdown
Member

Doesn't this change the ABI guarantees of PyTypeObject? Or are there none of those? In any case, feels strange that we are adding a public field that is opaque for our own use.

@markshannon
Copy link
Copy Markdown
Member Author

tp_watchers is already documented as internal.
The header is public, so the fields have to go in a public header even if they aren't to be used.

@markshannon markshannon requested a review from iritkatriel as a code owner April 8, 2026 16:43
Copy link
Copy Markdown
Member

@Fidget-Spinner Fidget-Spinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments

+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
| :c:member:`~PyTypeObject.tp_vectorcall` | :c:type:`vectorcallfunc` | | | | | |
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
| [:c:member:`~PyTypeObject.tp_iteritem`] | opaque function pointer | | | | | |
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs are warning that

c:member reference target not found: PyTypeObject.tp_iteritem [ref.member]

Comment on lines +309 to +312
typedef struct _py_object_index_pair {
PyObject *object;
Py_ssize_t index;
} PyObjectIndexPair;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be underscore prefixed, otherwise it will be public as it's in object.h.

_GUARD_ITER_VIRTUAL +
_PUSH_TAGGED_ZERO;

op(_GET_ITER_TRAD, (iterable -- iter, index_or_null)) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does trad mean?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants