feat: add host range profiling by voltjia · Pull Request #817 · InfiniTensor/InfiniOps

voltjia · 2026-07-24T09:48:59Z

Summary

Add opt-in host-side range profiling behind INFINI_OPS_ENABLE_HOST_RANGE_PROFILING, disabled by default.
Instrument binding conversion, generated dispatch, cache/operator invocation, CUDA Add submission, and NVIDIA cuBLASLt GEMM submission.
Use named HostRangeScope RAII objects at call sites; no instrumentation macro is required.
Reduce profiling compile-time branches from eight to three deliberate boundaries: implementation selection, real/empty scope type, and the cache hot path.
Compile profiling-disabled scopes as an inline empty type so optimized builds eliminate them, and keep the profiler header out of installed public headers.
Extend the existing pytest benchmark flow with --host-range-profile, cold/warm JSONL reports, calibration/control tooling, and focused regression tests.

Motivation

The existing pytest benchmark path measures end-to-end operator latency but cannot attribute CPU-side overhead across Python binding, conversion, dispatch, cache, operator, and backend-submission layers.

This PR adds coarse host attribution without introducing a separate C++ benchmark framework or timing device execution. It is an opt-in diagnostic facility, not a kernel profiler or performance gate.

Related issue: N/A - follows the performance-testing design discussion.

Type of Change

feat - new feature / new operator / new platform
fix - bug fix
perf - performance improvement (no behavioral change)
refactor - code restructuring without behavior change
test - adding or fixing tests only
docs - documentation only
build / ci - build system or CI configuration
chore - tooling, formatting, or other non-code changes
Breaking change

Platforms Affected

Smoke Test Result

Final NVIDIA validation ran in accelerator-dev/nvidia:latest on ssh nvidia against commit 072e79680588a535dda9a1f50d0ca40d7436e0fc (tree 2cba932a0c92e06d08f1fca51e40b9ad1bacf74d). Physical GPU 4 was exposed as logical cuda:0.

NVIDIA profiling ON build/install (RelWithDebInfo): passed
NVIDIA profiling OFF build/install (Release):       passed
Compile-time guard count:                            3
Binary symbol check:
  ON  libinfiniops.so: HostRangeScope ctor/dtor present
  OFF libinfiniops.so: no HostRangeScope symbols

Focused profiling suite:
  NVIDIA profiling ON:  43 passed in 29.40s
  NVIDIA profiling OFF: 39 passed, 4 skipped in 0.46s

Exact final-SHA reports:
  Add profiling ON:  1 passed, 131 deselected in 1.69s; 39 JSONL rows
  GEMM profiling ON: 1 passed, 2999 deselected in 1.99s; 39 JSONL rows
  backend.submit rows: 4 in each report

Local generator/public-header tests: 20 passed in 0.45s
Ruff 0.15.22 check and format check:    passed
clang-format 21.1.8 --dry-run --Werror: passed
git diff --check:                       passed

Final-SHA GitHub checks:
  legacy unit: NVIDIA, Iluvatar, MetaX, Cambricon, Moore, Ascend passed
  CI v2 shadow: NVIDIA, Iluvatar, MetaX, Cambricon, Moore, Ascend passed
  clang-format, Ruff, documentation build passed; deploy skipped

Earlier CPU validation on pre-RAII commit c4ff40f passed its profiling build/install, focused suite (41 passed), and smoke suite (54 passed, 8 skipped). It was not rerun on the final SHA, so it is retained only as earlier evidence rather than a final-SHA platform claim.

Test Results on Supported Platforms

Platform	Affected	Build / Smoke Result	Full Result / Notes
CPU	Yes	Earlier pre-RAII build/smoke passed (`54 passed, 8 skipped`)	Final SHA not rerun on CPU; no final-SHA CPU claim
NVIDIA	Yes	Final-SHA legacy and shadow CI passed	Independent profiling ON/OFF builds, focused suites, symbol checks, and exact Add/GEMM reports passed
Iluvatar	Yes	Final-SHA legacy and shadow CI passed	No profiling-enabled device-specific report was run
MetaX	Yes	Final-SHA legacy and shadow CI passed	No profiling-enabled device-specific report was run
Cambricon	Yes	Final-SHA legacy and shadow CI passed	No profiling-enabled device-specific report was run
Moore	Yes	Final-SHA legacy and shadow CI passed	No profiling-enabled device-specific report was run
Ascend	Yes	Final-SHA legacy and shadow CI passed	No profiling-enabled device-specific report was run

Focused pytest output for final SHA

NVIDIA profiling ON:
...........................................                              [100%]
43 passed in 29.40s

NVIDIA profiling OFF:
ssss.......................................                              [100%]
39 passed, 4 skipped in 0.46s

Add report:
.                                                                        [100%]
1 passed, 131 deselected in 1.69s

GEMM report:
.                                                                        [100%]
1 passed, 2999 deselected in 1.99s

Benchmark / Performance Impact

Final-report measurements used one NVIDIA A100-SXM4-80GB (physical CUDA_VISIBLE_DEVICES=4, logical cuda:0) and the profiling-ON RelWithDebInfo build. Device synchronization occurs only outside collection windows.

Add: contiguous FP32 (13, 4), implementation 0.
GEMM: FP32 (4, 48, 64) x (4, 64, 6), cuBLASLt implementation 1.

Warm median	Add	GEMM
`end_to_end`	30.913 us	39.869 us
`binding.body` inclusive	18.887 us	25.543 us
`dispatch.call` inclusive	6.948 us	13.213 us
`operator.invoke` inclusive	5.486 us	11.538 us
`backend.submit` inclusive	4.594 us	10.890 us

The two final-SHA JSONL reports contain 39 rows each. add-raii-072e796.jsonl SHA256 is 682b9846e366bb674535beb6a4a5a28b753a005706f8f994201db922973c413f; gemm-raii-072e796.jsonl SHA256 is 2ebfae447276cbc4b1212cba36d50f3295ac03d8fb90f783aabb4ef85e423a10.

An earlier same-process alternating control on the same A100 characterized active-collection overhead at approximately +9.7% for Add and +6.6% to +7.0% for GEMM, with a conservative complete-scope calibration of 134.12 ns. Those values came from a pre-refactor experiment and are observer-effect guidance, not final-tree performance claims.

Complete `add-raii-072e796.jsonl` output

{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"binding.body","metric":"inclusive","count":1,"unit":"ns","mean":775383.0,"median":775383.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"binding.body","metric":"self","count":1,"unit":"ns","mean":2123.0,"median":2123.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"binding.tensor_conversion","metric":"inclusive","count":3,"unit":"ns","mean":10765.666666666666,"median":6305.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"binding.tensor_conversion","metric":"self","count":3,"unit":"ns","mean":10765.666666666666,"median":6305.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"binding.device_conversion","metric":"inclusive","count":1,"unit":"ns","mean":7986.0,"median":7986.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"binding.device_conversion","metric":"self","count":1,"unit":"ns","mean":7986.0,"median":7986.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"dispatch.call","metric":"inclusive","count":1,"unit":"ns","mean":732977.0,"median":732977.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"dispatch.call","metric":"self","count":1,"unit":"ns","mean":1485.0,"median":1485.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"operator.call","metric":"inclusive","count":1,"unit":"ns","mean":731492.0,"median":731492.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"operator.call","metric":"self","count":1,"unit":"ns","mean":229161.0,"median":229161.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"cache.key","metric":"inclusive","count":1,"unit":"ns","mean":3643.0,"median":3643.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"cache.key","metric":"self","count":1,"unit":"ns","mean":3643.0,"median":3643.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"cache.lookup","metric":"inclusive","count":1,"unit":"ns","mean":551.0,"median":551.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"cache.lookup","metric":"self","count":1,"unit":"ns","mean":551.0,"median":551.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"cache.construct","metric":"inclusive","count":1,"unit":"ns","mean":478299.0,"median":478299.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"cache.construct","metric":"self","count":1,"unit":"ns","mean":478299.0,"median":478299.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"operator.invoke","metric":"inclusive","count":1,"unit":"ns","mean":19838.0,"median":19838.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"operator.invoke","metric":"self","count":1,"unit":"ns","mean":2157.0,"median":2157.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"backend.submit","metric":"inclusive","count":1,"unit":"ns","mean":17681.0,"median":17681.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"cold","range":"backend.submit","metric":"self","count":1,"unit":"ns","mean":17681.0,"median":17681.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"binding.body","metric":"inclusive","count":6500,"unit":"ns","mean":19124.904615384614,"median":18887.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"binding.body","metric":"self","count":6500,"unit":"ns","mean":775.2773846153846,"median":674.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"binding.tensor_conversion","metric":"inclusive","count":19500,"unit":"ns","mean":3287.0886153846154,"median":3166.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"binding.tensor_conversion","metric":"self","count":19500,"unit":"ns","mean":3287.0886153846154,"median":3166.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"binding.device_conversion","metric":"inclusive","count":6500,"unit":"ns","mean":1399.406923076923,"median":1395.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"binding.device_conversion","metric":"self","count":6500,"unit":"ns","mean":1399.406923076923,"median":1395.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"dispatch.call","metric":"inclusive","count":6500,"unit":"ns","mean":7088.954461538461,"median":6947.5}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"dispatch.call","metric":"self","count":6500,"unit":"ns","mean":244.76738461538463,"median":229.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"operator.call","metric":"inclusive","count":6500,"unit":"ns","mean":6844.187076923077,"median":6714.5}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"operator.call","metric":"self","count":6500,"unit":"ns","mean":486.9393846153846,"median":449.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"cache.key","metric":"inclusive","count":6500,"unit":"ns","mean":522.2975384615385,"median":512.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"cache.key","metric":"self","count":6500,"unit":"ns","mean":522.2975384615385,"median":512.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"cache.lookup","metric":"inclusive","count":6500,"unit":"ns","mean":256.622,"median":257.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"cache.lookup","metric":"self","count":6500,"unit":"ns","mean":256.622,"median":257.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"operator.invoke","metric":"inclusive","count":6500,"unit":"ns","mean":5578.328153846154,"median":5486.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"operator.invoke","metric":"self","count":6500,"unit":"ns","mean":917.5878461538462,"median":886.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"backend.submit","metric":"inclusive","count":6500,"unit":"ns","mean":4660.740307692307,"median":4594.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"backend.submit","metric":"self","count":6500,"unit":"ns","mean":4660.740307692307,"median":4594.0}
{"nodeid":"tests/test_add.py::test_add[cuda-0-input_shape0-other_shape0-out_shape0-None-None-None-None-dtype0-1e-07-1e-07]","operator":"add","backend":"nvidia","phase":"warm","range":"end_to_end","metric":"inclusive","count":6500,"unit":"ns","mean":30906.074895308568,"median":30913.334339857105}

Complete `gemm-raii-072e796.jsonl` output

{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"binding.body","metric":"inclusive","count":1,"unit":"ns","mean":113921.0,"median":113921.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"binding.body","metric":"self","count":1,"unit":"ns","mean":1824.0,"median":1824.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"binding.tensor_conversion","metric":"inclusive","count":3,"unit":"ns","mean":12624.0,"median":7883.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"binding.tensor_conversion","metric":"self","count":3,"unit":"ns","mean":12624.0,"median":7883.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"binding.device_conversion","metric":"inclusive","count":1,"unit":"ns","mean":8177.0,"median":8177.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"binding.device_conversion","metric":"self","count":1,"unit":"ns","mean":8177.0,"median":8177.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"dispatch.call","metric":"inclusive","count":1,"unit":"ns","mean":66048.0,"median":66048.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"dispatch.call","metric":"self","count":1,"unit":"ns","mean":845.0,"median":845.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"operator.call","metric":"inclusive","count":1,"unit":"ns","mean":65203.0,"median":65203.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"operator.call","metric":"self","count":1,"unit":"ns","mean":3445.0,"median":3445.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"cache.key","metric":"inclusive","count":1,"unit":"ns","mean":3257.0,"median":3257.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"cache.key","metric":"self","count":1,"unit":"ns","mean":3257.0,"median":3257.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"cache.lookup","metric":"inclusive","count":1,"unit":"ns","mean":268.0,"median":268.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"cache.lookup","metric":"self","count":1,"unit":"ns","mean":268.0,"median":268.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"cache.construct","metric":"inclusive","count":1,"unit":"ns","mean":6145.0,"median":6145.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"cache.construct","metric":"self","count":1,"unit":"ns","mean":6145.0,"median":6145.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"operator.invoke","metric":"inclusive","count":1,"unit":"ns","mean":52088.0,"median":52088.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"operator.invoke","metric":"self","count":1,"unit":"ns","mean":799.0,"median":799.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"backend.submit","metric":"inclusive","count":1,"unit":"ns","mean":51289.0,"median":51289.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"cold","range":"backend.submit","metric":"self","count":1,"unit":"ns","mean":51289.0,"median":51289.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"binding.body","metric":"inclusive","count":5000,"unit":"ns","mean":27977.3664,"median":25542.5}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"binding.body","metric":"self","count":5000,"unit":"ns","mean":896.288,"median":734.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"binding.tensor_conversion","metric":"inclusive","count":15000,"unit":"ns","mean":3718.7997333333333,"median":3262.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"binding.tensor_conversion","metric":"self","count":15000,"unit":"ns","mean":3718.7997333333333,"median":3262.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"binding.device_conversion","metric":"inclusive","count":5000,"unit":"ns","mean":1573.5644,"median":1426.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"binding.device_conversion","metric":"self","count":5000,"unit":"ns","mean":1573.5644,"median":1426.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"dispatch.call","metric":"inclusive","count":5000,"unit":"ns","mean":14351.1148,"median":13213.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"dispatch.call","metric":"self","count":5000,"unit":"ns","mean":245.727,"median":209.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"operator.call","metric":"inclusive","count":5000,"unit":"ns","mean":14105.3878,"median":13003.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"operator.call","metric":"self","count":5000,"unit":"ns","mean":545.413,"median":463.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"cache.key","metric":"inclusive","count":5000,"unit":"ns","mean":844.6874,"median":771.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"cache.key","metric":"self","count":5000,"unit":"ns","mean":844.6874,"median":771.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"cache.lookup","metric":"inclusive","count":5000,"unit":"ns","mean":253.9402,"median":226.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"cache.lookup","metric":"self","count":5000,"unit":"ns","mean":253.9402,"median":226.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"operator.invoke","metric":"inclusive","count":5000,"unit":"ns","mean":12461.3472,"median":11538.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"operator.invoke","metric":"self","count":5000,"unit":"ns","mean":709.7614,"median":639.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"backend.submit","metric":"inclusive","count":5000,"unit":"ns","mean":11751.5858,"median":10890.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"backend.submit","metric":"self","count":5000,"unit":"ns","mean":11751.5858,"median":10890.0}
{"nodeid":"tests/test_gemm.py::test_gemm[cuda-1-dtype0-0.001-0.001-False-False-0-1-a_shape4-b_shape4-c_shape4-None-None-None]","operator":"gemm","backend":"nvidia","phase":"warm","range":"end_to_end","metric":"inclusive","count":5000,"unit":"ns","mean":40760.25132089853,"median":39868.587628006935}

Notes for Reviewers

backend.submit measures host API work through API return; it does not measure device execution or kernel duration. CUDA APIs may still block on queue/device progress.
Synchronization occurs only before and after cold/warm collection windows.
end_to_end and replayed C++ ranges use separate populations and must not be subtracted from one another.
Profiling is disabled by default; both compiled-ON and compiled-OFF builds were tested.
Call sites use named RAII objects rather than preprocessor macros.
The eight original source-level conditions are reduced to three: host_range_profiler.cc selects the implementation, host_range_profiler.h selects the real or empty scope, and operator.h preserves the exact profiling-OFF cache hot path.
In optimized OFF builds the empty scope is eliminated; the final OFF library contains no HostRangeScope symbols.
host_range_profiler.h is internal and is excluded from both installed headers and the generated public operator.h, avoiding an ON/OFF class-definition contract for downstream consumers.
cuBLASLt instrumentation is limited to one include and one function-entry scope, matching the Add backend path without restructuring the function body.
pytest-xdist is intentionally rejected when host-range output is enabled.
Backend-submission instrumentation currently covers the shared CUDA Add path and NVIDIA cuBLASLt GEMM. Other operators still receive common binding, dispatch, cache, and invocation ranges.
Raw reports are included above for review but intentionally are not committed as machine-specific repository artifacts.
All final-SHA legacy and shadow platform checks passed; no CI rerun was requested or triggered manually.
Process-only design and planning files under docs/superpowers/ were removed from the final branch.

voltjia force-pushed the feat/host-range-profiling branch from c4ff40f to f8d63bb Compare July 28, 2026 04:05

voltjia requested a review from Ziminli July 28, 2026 05:57

feat: add host range profiling

072e796

voltjia force-pushed the feat/host-range-profiling branch from f8d63bb to 072e796 Compare July 28, 2026 06:38

voltjia marked this pull request as ready for review July 28, 2026 07:07

voltjia requested a review from a team July 28, 2026 07:07

voltjia merged commit 5865a4c into master Jul 28, 2026
20 checks passed

voltjia deleted the feat/host-range-profiling branch July 28, 2026 07:08

baominghelly mentioned this pull request Jul 30, 2026

perf: avoid redundant operator cache lookups #858

Open

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add host range profiling - #817

feat: add host range profiling#817
voltjia merged 1 commit into
masterfrom
feat/host-range-profiling

voltjia commented Jul 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

voltjia commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Type of Change

Platforms Affected

Smoke Test Result

Test Results on Supported Platforms

Benchmark / Performance Impact

Notes for Reviewers

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

voltjia commented Jul 24, 2026 •

edited

Loading