Skip to content

docs: document Top(order_by="_name") in hidden-attribute matrix#163

Open
dimitri-yatsenko wants to merge 1 commit intomainfrom
docs/hidden-top-order-by
Open

docs: document Top(order_by="_name") in hidden-attribute matrix#163
dimitri-yatsenko wants to merge 1 commit intomainfrom
docs/hidden-top-order-by

Conversation

@dimitri-yatsenko
Copy link
Copy Markdown
Member

@dimitri-yatsenko dimitri-yatsenko commented Apr 30, 2026

Summary

Adds one row to the §3.4 hidden-attribute behavior matrix to document that dj.Top(order_by="_name") flows through to SQL ORDER BY without heading validation.

Why this row is missing today

The current matrix (added in #162) covers reads (fetch/proj/to_dicts/...), writes (insert/update1), restrictions (dict and string), joins, describe(), and indexes — but not Top. Worth a row because Top.order_by behaves like the string-restriction case (passes the column name through to SQL untouched), so a hidden column can legitimately shape the result order even though its value can never return to Python.

Code path

  1. Top.__post_init__ (condition.py:142-148) — type-checks order_by is a string or list of strings; no name validation.
  2. restrict() (expression.py:241-254) — when the restriction is a Top, short-circuits with return result at line 254, before reaching the heading.names check at lines 260-262 that fires for non-Top conditions.
  3. sorting_clauses() (expression.py:133-145) → _flatten_attribute_list (:1580-1604, only special-cases "KEY") → _wrap_attributes (:1607-1617, regex \b((?!asc|desc)\w+)\b quotes the name as `_name`).
  4. The quoted hidden-column name lands in ORDER BY. The SELECT list is still built from heading.names at expression.py:163, so the value never returns to Python.

Worked example

# Five most recently populated rows
MyTable & dj.Top(5, order_by="_job_start_time DESC")

Generated SQL (visible columns only in SELECT; hidden column referenced only in ORDER BY):

SELECT <visible cols>
FROM <table>
ORDER BY `_job_start_time` DESC
LIMIT 5

Caveat

Because Top does no name check, a typoed column name passes DataJoint validation and fails as a SQL error from the DB. Not a hidden-attribute concern specifically, but worth knowing the failure mode if you mistype _job_start_tine.

Test plan

  • python scripts/gen_llms_full.py regenerates cleanly.
  • Build the docs locally and confirm the new row renders correctly in the matrix.

Top.order_by is not validated against heading.names — it flows from
condition.py:142-148 (type check only) through expression.py:241-254
(restrict() short-circuits on Top before the heading.names check)
into _flatten_attribute_list / _wrap_attributes, which quote the name
and concatenate it into ORDER BY. So a hidden column shapes the result
order even though its value never returns to Python.

This is the same shape as string restrictions: DataJoint-generated SQL
references the hidden column, but the visible-only filter on the SELECT
list keeps its value out of the result set. Worth one row in the
behavior matrix so users querying "5 most recently populated rows" via
Top(5, order_by="_job_start_time DESC") see it as an explicitly
supported pattern rather than an undocumented bypass.
@dimitri-yatsenko dimitri-yatsenko changed the base branch from docs/hidden-attributes-platform-framing to main April 30, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant