fix(profiling): memalloc obj domain module finalizer

[gyuheon0h]

Description

While looking at crash reports, I saw a series of crash reports that look like such: https://app.datadoghq.com/error-tracking/issue/4fe0aa66-5607-11f1-b89e-da7ad0900002?query=version%3A4.15.2&et-issue__error-sample-viz=sample&refresh_mode=sliding&from_ts=1789393982142&to_ts=1790603582142&live=true

The crashes are in the memalloc profiler's allocator hook lifecycle caused SIGSEGV crashes in gunicorn/gevent workers (observed as SI_KERNEL at alloc.malloc(alloc.ctx, ...) in memalloc_alloc).

From looking at the code, I saw that there was no module finalizer so hooks outlive the interpreter

PyModuleDef had no m_free callback. When a process such as a gevent worker exits without the Python-level collector calling stop() explicitly, the OBJ and MEM hooks remain installed through Py_Finalize(). CPython then tears down its internal allocator state, and any allocation that reaches the hook calls alloc.malloc(alloc.ctx, ...) with a stale context which will fault.

I added g_saved_alloc_pub.store(nullptr) in stop() to match the MEM domain, and register memalloc_module_free as m_free in PyModuleDef to always uninstall hooks during module deallocation.

Testing

Its hard to write a deterministic test for this as waiting on GC to do what it does with module unload is not deterministic.I added a test-only helper _test_invoke_module_free that exposes the m_free path directly to Python and added this regression test: test_m_free_uninstalls_hooks_deterministic

Not the biggest fan of writing API just for testing but 🤷

Risks

Low IMO

PyMem_SetAllocator is a plain struct copy with no heap allocation, safe to call at any shutdown stage. memalloc_heap_tracker_deinit_no_cpython frees C++ objects without calling any CPython API.

Additional Notes

I used AI to write the test following my directions.

0x7f5d9485469c __pthread_kill_implementation (nptl/nptl/pthread_kill.c:44)
0x7f5d947ffd70 __restore_rt
0x7f5d94b58c1c
memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:117)
0x7f5d9030a535 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:130)
0x7f5d94b5a644 PyUnicode_New
0x7f5d94c44b2e
0x7f5d94401469
0x7f5d943ff160
0x7f5d943ff52e
0x7f5d943ff160
0x7f5d943fe442
0x7f5d94b6a793 _PyObject_MakeTpCall
0x7f5d94b7311b _PyEval_EvalFrameDefault
0x7f5d94b6f63f
0x7f5d5769a848 __pyx_f_7asyncpg_8protocol_8protocol_5Codec_decode_in_python (/project/asyncpg/protocol/protocol.c:14555)
__pyx_f_7asyncpg_8protocol_8protocol_5Codec_decode (/project/asyncpg/protocol/protocol.c:14615)
0x7f5d576a59df __pyx_f_7asyncpg_8protocol_8protocol_22PreparedStatementState__decode_row (/project/asyncpg/protocol/protocol.c:58903)
0x7f5d576a6690 __pyx_f_7asyncpg_8protocol_8protocol_12BaseProtocol__decode_row (/project/asyncpg/protocol/protocol.c:77580)
0x7f5d5767d95c __pyx_f_7asyncpg_8protocol_8protocol_12CoreProtocol__parse_data_msgs (/project/asyncpg/protocol/protocol.c:46130)
0x7f5d5767c5e2 __pyx_f_7asyncpg_8protocol_8protocol_12CoreProtocol__process__bind_execute (/project/asyncpg/protocol/protocol.c:43641)
0x7f5d576aef4f __pyx_f_7asyncpg_8protocol_8protocol_12CoreProtocol__read_server_messages (/project/asyncpg/protocol/protocol.c:41942)
__pyx_pf_7asyncpg_8protocol_8protocol_12BaseProtocol_64data_received (/project/asyncpg/protocol/protocol.c:79532)
0x7f5d5768910b __pyx_pw_7asyncpg_8protocol_8protocol_12BaseProtocol_65data_received (/project/asyncpg/protocol/protocol.c:79495)
0x7f5d94b6efde PyObject_VectorcallMethod
Py_DECREF (/opt/_internal/cpython-3.11.13/include/python3.11/object.h:537)
Py_XDECREF (/opt/_internal/cpython-3.11.13/include/python3.11/object.h:602)
0x7f5d6753c4b6 __pyx_f_6uvloop_4loop_11SSLProtocol__do_read__copied (/project/uvloop/loop.c:152450)
0x7f5d6751ee2e __pyx_f_6uvloop_4loop_11SSLProtocol__do_read (/project/uvloop/loop.c:151167)
__pyx_pf_6uvloop_4loop_11SSLProtocol_14buffer_updated (/project/uvloop/loop.c:145987)
0x7f5d674e1465 __pyx_pw_6uvloop_4loop_11SSLProtocol_15buffer_updated (/project/uvloop/loop.c:145873)
0x7f5d94ba7c0a
0x7f5d94b3230c
0x7f5d94b3238d
0x7f5d94ba312f
0x7f5d94b6efde PyObject_VectorcallMethod
Py_DECREF (/opt/_internal/cpython-3.11.13/include/python3.11/object.h:537)
Py_XDECREF (/opt/_internal/cpython-3.11.13/include/python3.11/object.h:602)
0x7f5d674c8f83 __pyx_f_6uvloop_4loop_run_in_context1 (/project/uvloop/loop.c:13464)
0x7f5d67528450 __pyx_f_6uvloop_4loop___uv_stream_buffered_on_read (/project/uvloop/loop.c:108025)
0x7f5d675912ea uv__read (/project/build/libuv-x86_64/src/unix/stream.c:1146)
0x7f5d67591e34 uv__stream_io (/project/build/libuv-x86_64/src/unix/stream.c:1205)
0x7f5d6759868b uv__io_poll (/project/build/libuv-x86_64/src/unix/linux.c:1528)
0x7f5d6758ab36 uv_run (/project/build/libuv-x86_64/src/unix/core.c:452)
0x7f5d674dd3a0 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18471)
0x7f5d6751b12d __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876)
__pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528)
0x7f5d67516e85 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331)
0x7f5d94b6efde PyObject_VectorcallMethod
Py_DECREF (/opt/_internal/cpython-3.11.13/include/python3.11/object.h:537)
Py_XDECREF (/opt/_internal/cpython-3.11.13/include/python3.11/object.h:602)
0x7f5d67519844 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33769)
0x7f5d6751a8b8 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33318)
0x7f5d94b7ea97 PyObject_Vectorcall
0x7f5d94b7311b _PyEval_EvalFrameDefault
0x7f5d94b6f63f
0x7f5d94b97dcb
0x7f5d94b76f97 _PyEval_EvalFrameDefault
0x7f5d94b6f63f
0x7f5d94bf6ccc PyEval_EvalCode
0x7f5d94c1319d
0x7f5d94c0f77a
0x7f5d94c0573d PyRun_StringFlags
0x7f5d94c05640 PyRun_SimpleStringFlags
0x7f5d94c1e90c Py_RunMain
0x7f5d94be6b5b Py_BytesMain
0x7f5d947e9ca8 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:74)
call_init (csu/libc-start.c:128)
0x7f5d947e9d65 __libc_start_main_alias_2 (csu/libc-start.c:347)
0x561fd04b0071 _start

[gyuheon0h]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix(profiling): memalloc obj domain module finalizer #20608 👈 (View in Graphite)
• fix(profiling): null g_saved_alloc_pub after stop() for OBJ domain #20621
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[datadog-datadog-prod-us1]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0e1dea7 | Docs | View more details | Give us feedback!}

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against gyuheon0h/PROF-16085-memalloc-obj-domain-saved-alloc-null using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/profiling/collector/_memalloc.cpp                               @DataDog/profiling-python
releasenotes/notes/mem-profiler-shutdown-crash-fix-4583f7c14333be0c.yaml  @DataDog/apm-python
tests/profiling/collector/test_memalloc.py                              @DataDog/profiling-python

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 201 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 201 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=132)
ddtrace.internal.ci_visibility.api._base -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=130)
ddtrace.internal.ci_visibility.filters -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=130)
ddtrace.debugging._exception.replay -×-> ddtrace.trace  (product:debugging -> product:tracing, score=130)
ddtrace.internal.test_visibility.api -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=130)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc754054f6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[taegyunkim]

Would it make sense to split this into two PRs, one per suspected gap?

On free-threaded Python (no GIL on PYMEM_DOMAIN_OBJ), any hook that already loaded the non-null pub after stop() would call through to the saved allocator context after the heap tracker was destroyed

Also, the module fails to compile on free-threaded Python, so not sure why this reasoning would be relevant.

dd-trace-py/ddtrace/profiling/collector/_memalloc_frame.h

Line 32 in 0bdf7a8

#error "_memalloc frame walking relies on the GIL-held allocator hook and is not yet supported on free-threaded CPython"

[KowalskiThomas]

Overall LGTM