[pyricau]

Takes over the referrers tool builderbot drafted in this Slack thread. The thread started with "how do I get backward refs?": ways_held walks to the GC roots and keeps only paths that share no object, so it doesn't say which objects point at an object. builderbot posted the patch in three parts. Only part 1 arrived, and it was cut off partway through. That part had the docs row, AgentJson.referrers(), the session verb and the tool itself, and all four are in this PR in revised form. The core half, referrersOf and its model, never arrived, so I wrote it here.

The last commits also change how the dominator tree reads a thread's local variables, which the referrers list made visible. See below.

Builds on #3015 and #3017, both merged. #3015 makes Shark read a thread's JNI_LOCAL and NATIVE_STACK roots as references from that thread, as it already read its JAVA_FRAME roots, and #3017 names those references <Native Local>. This PR used to read the native roots itself and now leaves that to Shark. The tree comes out the same.

What changed from the draft

The draft read the ReferrerIndex that paths walk. That index only holds the references the dominator tree follows. WeakeningAwareReferenceReader leaves out a weak reference to an object something holds strongly, and every reference to an owned object except the owner's. So for the objects people most often ask about, the draft would have reported one referrer as the complete list. Measured on large-dump.hprof:

object	pointing at it	in the tree's index
the running MainActivity	951	1 (ActivityThread.activities)
the running PaymentActivity	25	1
a MainActivity the thread doesn't run	922	920

To get every referrer, referrersOf reads a second index. That index is built over EveryReferenceReader, which returns every reference the tree's reader looks at, including the ones it drops. Each reference in the answer carries holds, which says whether the tree holds the object through it.

The answer

ObjectReferrers is the model and AgentJson.referrers the JSON.

• A page at a time. 30 referrers by default, limit up to 500, and offset to start further in. nextOffset is the offset of the next page, null on the last. An offset past the end is refused with the count.
• A referrer is an object or a GC root. An object is a row of find_objects (object, className, kind, the content fields, both sizes, strength) plus the references it points through. A GC root on the object is a GcRootReferrer: gcRootType and holds. The roots come from TreeGcRootProvider.rootsOn, which lists what Shark's MatchingGcRootProvider keeps, and holds from TreeGcRootProvider.holds, which decides whether the tree hangs the object off that root. Nothing on a referrer is about a path, so there's no verdict and no isSuspect, and no inspector runs.
• The order is fixed, so pages add up to the list. Holders first, a GC root ahead of an object, then the largest retained size. So the first page says what keeps the object in memory.
• The counts are of the whole list on every page. referrerCount and holdingReferrerCount include GC roots, so "is this the only thing keeping it in memory?" is a count.
• Where a reference comes from is in three fields. The referrer's kind is CLASS, INSTANCE, OBJECT_ARRAY or PRIMITIVE_ARRAY. A reference's locationType is STATIC_FIELD, INSTANCE_FIELD, ARRAY_ENTRY, LOCAL or NATIVE_LOCAL. And isVirtual is true when Shark reads the reference off a structure the heap dump doesn't record that way, such as an ArrayList holding its elements. See isVirtual.

The real first page from the window this PR leaves open, to --cli referrers heapDumpKey=large-dump.hprof object=0x2c07c580, trimmed to its first two referrers of 30:

{
    "referrerCount": 951,
    "holdingReferrerCount": 1,
    "nextOffset": 30,
    "referrers": [
        {
            "object": "0x2ac04100",
            "className": "android.app.ActivityThread",
            "kind": "INSTANCE",
            "shallowBytes": 190,
            "retainedBytes": 2154580,
            "strength": "STRONG",
            "references": [
                {
                    "name": "activities",
                    "ownerClassName": "ActivityThread",
                    "locationType": "INSTANCE_FIELD",
                    "isVirtual": true,
                    "holds": true
                }
            ]
        },
        {
            "object": "0x2af21dc0",
            "className": "java.util.ArrayList",
            "kind": "INSTANCE",
            "shallowBytes": 20,
            "retainedBytes": 3248428,
            "strength": "STRONG",
            "references": [
                {
                    "name": "3",
                    "ownerClassName": "ArrayList",
                    "locationType": "ARRAY_ENTRY",
                    "isVirtual": true,
                    "holds": false
                }
            ]
        }
    ]
}

And an interned string, object=0x71422ac8 limit=40, trimmed to its three holders and one of the 15 threads holding it in a local variable. Each thread is listed once, through its virtual LOCAL reference. The heap dump also has a JAVA_FRAME root for each of those locals, and the list leaves those out:

{
    "referrerCount": 18,
    "holdingReferrerCount": 3,
    "nextOffset": null,
    "referrers": [
        {
            "gcRootType": "VM_INTERNAL",
            "holds": true
        },
        {
            "gcRootType": "INTERNED_STRING",
            "holds": true
        },
        {
            "object": "0x7137b858",
            "className": "android.os.MessageQueue",
            "kind": "CLASS",
            "shallowBytes": 111,
            "retainedBytes": 236,
            "strength": "STRONG",
            "references": [
                {
                    "name": "TAG",
                    "ownerClassName": "MessageQueue",
                    "locationType": "STATIC_FIELD",
                    "isVirtual": false,
                    "holds": true
                }
            ]
        },
        {
            "object": "0x75ace698",
            "className": "java.lang.Thread",
            "kind": "INSTANCE",
            "threadName": "????",
            "shallowBytes": 132,
            "retainedBytes": 7927,
            "strength": "STRONG",
            "references": [
                {
                    "name": "",
                    "ownerClassName": "Thread",
                    "locationType": "LOCAL",
                    "isVirtual": true,
                    "holds": false
                }
            ]
        }
    ]
}

A thread's local variable no longer holds what a field holds

When the heap dump names a frame's thread, Shark reads each JAVA_FRAME root the dump records as a LOCAL reference from the Thread (JavaLocalReferenceReader). The tree followed that reference as a strong one, against what GcRootStrength.kt says a local is. So no object of any dump was ever LOCAL. And an object a field also held had two holders, the thread and the field's owner, so it was drawn under the whole heap dump. Before this commit the interned string above had 18 holders, 15 of them threads.

ReferenceStrengthReader now builds the readers ActualMatchingReferenceReaderFactory chains, less JavaLocalReferenceReader, and reads a thread's locals as weakening references at LOCAL (localVariableReferencesOf). An object only a stack holds is under its thread. An object a field also holds is under the field.

Native code a thread runs holds objects through JNI_LOCAL and NATIVE_STACK roots. These name their thread by serial number, as a JAVA_FRAME root does. Since #3015, Shark's JavaLocalReferenceReader reads all three kinds as references from the thread (LOCAL for a frame, NATIVE_LOCAL for the other two since #3017), and MatchingGcRootProvider drops all three roots for a thread the heap dump has. So each local is one way to its object. The referrers list takes its GC roots from that same provider, so each local is also one referrer, its thread. It used to read them off the heap dump, which listed every local twice and took a copy of Shark's rule to mark the root as holding nothing. On large-dump.hprof and on a 722 MB app dump, all 3,085 and 7,602 stack roots are on a thread the dump has.

Measured against main without #3015, every object on a JAVA_FRAME, JNI_LOCAL or NATIVE_STACK root:

dump	from the whole heap dump to an object	from the whole heap dump to its thread	now LOCAL, was STRONG
large-dump.hprof	160 objects, 45 KB	3 objects, 182 B	38 objects, 2.3 KB
compose_leak.hprof	30 objects, 8.1 KB	3 objects, 222 B	330 objects, 6.8 KB
leak_asynctask_o.hprof	21 objects, 302 KB	3 objects, 115 B	35 objects, 945 B

On large-dump.hprof the 160 are mostly what a parked thread waits on: 36 now under an AbstractQueuedSynchronizer$ConditionObject, 44 under a ScheduledThreadPoolExecutor or its DelayedWorkQueue, 16 under a LinkedBlockingQueue. The 3 are Strings only a JNI_LOCAL root on the main thread holds. The dominator-tree.md section has the table.

Reading the native roots is what keeps objects under their thread. With frames at LOCAL and the native roots left as roots, 7 objects of large-dump.hprof went from under their thread to under the whole heap dump. Six have a JNI_LOCAL or NATIVE_STACK root besides their frame, on the same thread, and the seventh is held by one of the six. A strong local had made them STRONG, so the strength rule dropped the native root. At LOCAL the root was kept, a second holder beside the thread. Reading the native roots puts all 7 back under their thread, and likewise 5 on compose_leak.hprof and 7 on leak_asynctask_o.hprof. Between reading them and not, no object of the eight dumps moved onto the whole heap dump. Every JNI_LOCAL and NATIVE_STACK root of them names a thread the dump has. Leaving that reading to Shark changed nothing: on the eight dumps and a production one, every node's dominator, strength and retained size are the same, and so are both referrer indexes, down to the order of each object's referrers.

One test fixture had to change. taskHoldingItsOwnThreadHeapDump built its loop of objects that hold each other through a stack frame, and frames now hold nothing a field holds. It now closes the loop the way the framework does: SerialExecutor.mActive holds the runnable, and the runnable holds the executor back through this$0. The isAbove KDoc uses that example now. A new HeapVerdictTest case pins that a stack frame closes no loop.

isVirtual

PathReference.isVirtual is Shark's LazyDetails.isVirtual, and referenceInto puts it on every reference an agent is answered with, paths included. Shark sets it on a reference it reads off a structure:

• an ArrayList holding its elements, read by InternalSharedArrayListReferenceReader;
• a thread holding its locals;
• ActivityThread.activities holding the running activities, read by RunningActivityReferenceReader.

What it goes through is still in the dump, with its own references, so an element of a list has both referrers you'd expect. The running MainActivity above has the list 0x2af21dc0 second on the first page, through a virtual [3]. Its elementData array 0x2b42c318 is a referrer too, at position 712 because it retains 40 bytes:

{
    "object": "0x2b42c318",
    "className": "java.lang.Object[]",
    "kind": "OBJECT_ARRAY",
    "arrayElementCount": 10,
    "shallowBytes": 40,
    "retainedBytes": 40,
    "strength": "STRONG",
    "references": [
        {
            "name": "3",
            "ownerClassName": "Object[]",
            "locationType": "ARRAY_ENTRY",
            "isVirtual": false,
            "holds": false
        }
    ]
}

I named it isVirtual, not "synthetic", to match Shark's field and the "virtual reference" that Shark Dive's code and notes already say. Happy to rename it.

Cost

Memory and time. The second index is built the first time anything calls referrersOf, and nothing the window draws calls it. So a session that never uses the tool pays nothing. Once built, it is up to 5% larger than the tree's index on seven of eight dumps, and 21% larger on unloaded_classes-stripped.hprof. Sorting a page reads whether each referrer holds the object off the tree's index, so only the page's objects are read. On large-dump.hprof, the first call took 573 ms in the window this PR leaves open, building the index included. After it, a page of 30 took 6 to 17 ms and a page of 500 took 38 to 44 ms. The new section of notes/referrer-index.md has the table, re-measured once the native roots were read. On large-dump.hprof the tree's index lost 2,877 references when a frame stopped holding what a field holds. Reading the native roots then added 178 references to the index of every reference, and 9 to the tree's. It also explains why I didn't add a bit per reference to the existing index: that would change what the hover-path walk reads.

Answer size. The running MainActivity's first page is 16,734 characters, about 560 per referrer. Its whole list in one answer was 505,394 characters. Claude Code saves a Bash result over 30,000 characters to a file and hands the model a 2 KB preview, which I confirmed by making that call from a Claude Code session. A page of 500 is 281,799 characters, so a large limit goes past that cap again. notes/agent-surface.md records this.

Worth a look

• The window has no equivalent. shark-dive-agent/AGENTS.md says the agent surface and the window should match in both directions, and this PR is agent-only. The docs row says so. A "pointed at by" list in the details panel would be the window's version. I left it out because it's UI work with its own measuring to do before it can go in the hover path.
• A weak reference is named after the instance's class, so it reads WeakReference.referent rather than Reference.referent. That comes from WeakeningReference.ofField, which paths already use, so this PR leaves it as it is. The tests pin the current behaviour.
• AgentTools was split. The referrers tool and main's investigation states put the class past detekt's LargeClass threshold of 750 lines. The config says the next tool to trip it should split the registry rather than raise the number again, so list_devices, list_processes and dump_heap moved to AgentDeviceTools, unchanged and still listed last.
• stepTo's reference-naming code is now referencesFrom, shared with referrersOf, with no change in behaviour. AgentJson.pathReference and the referrer JSON share referenceInto, so the keys come out in the same order as before, with isVirtual after locationType. find_objects and referrers share objectListEntryInto.

Tests

• ReferrersTest (new) covers:
  • two referrers under one parent being one way held but two referrers;
  • two pages adding up to the whole list, in its order, with the counts whole on both;
  • a weak reference to a strongly held object;
  • holders sorting ahead of a non-holder that retains more;
  • a GC root on an object nothing points at being its one referrer;
  • a local variable pointing at an object a field holds, a root that holds nothing, listed after the field;
  • a GC root that holds the object, listed ahead of the field holding it;
  • the virtual root;
  • a local of a thread the dump names, listed once as its thread's reference and holding nothing a field holds;
  • an object only that local holds, held through its thread at LOCAL;
  • an object a frame, a JNI_LOCAL and a NATIVE_STACK root of that thread hold, held through its thread and listed as one <local variable> and two <native local> references, with no root;
  • an element of a list, held by the list through a virtual entry and by the array through a real one.
• HeapVerdictTest reads its loop off the executor and its runnable, and adds a case pinning that a stack frame holding an object a field holds closes no loop.
• OwnerReferencesTest adds two cases:
  • the running activity: only the thread holds it, and the record, the leaker and five view contexts are listed with holds false;
  • a view: only its parent holds it, and the View[] entry and InputMethodManager.mServedView are listed with holds false.
• AgentToolsTest covers the JSON shape, including the absence of verdict and isSuspect, a GC root as a referrer, and the refusal of an offset past the end.

:shark:shark-dive:shark-dive-core:test, :shark:shark-dive:shark-dive-agent:test, :shark:shark-dive:shark-dive-app:test and detekt pass locally on JDK 17, rebased on main.

There's no changelog entry because Shark Dive has none, and no ABI dump because these modules are in modulesWithoutPublicApi.

🤖 Generated with Claude Code