Sled has a reservoir size of 0 after reboot

[augustuswm]

We have seen four sleds in rack3 that have come up after a recent restart with a reservoir size of 0 set in the database.

root@[fd00:1122:3344:116::3]:32221/omicron> select id, serial_number, reservoir_size, time_created, time_modified from sled where reservoir_size = 0;
                   id                  | serial_number | reservoir_size |         time_created          |         time_modified
---------------------------------------+---------------+----------------+-------------------------------+--------------------------------
  2c33e8ef-ae07-4191-804d-52a1840ce53c | BRM42220042   |              0 | 2023-09-02 13:07:22.168451+00 | 2026-08-12 15:39:45.228129+00
  857bd97b-8147-4d26-813b-d80ab8f36cd5 | BRM42220083   |              0 | 2023-09-02 13:07:20.885306+00 | 2026-08-12 17:45:59.820146+00
  eaad4483-ea93-48d1-99d3-bf43267b9eb7 | BRM42220015   |              0 | 2023-09-02 13:07:20.852863+00 | 2026-08-12 21:23:28.448887+00
  fac5180b-232a-489e-b7c5-313a0742d1cc | BRM42220054   |              0 | 2023-09-02 13:07:20.82203+00  | 2026-08-12 16:04:53.65634+00

All four of these sleds have sled-agent logs that look like:

{"msg":"Calculated eligible VMM reservoir size","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.231465519Z","hostname":"BRM42220015","pid":665,"component":"MemoryReservations","file":"sled-hardware/src/lib.rs:154","control_plane_earmark_bytes":"47244640256","max_page_t_bytes":"31834245120","physical_ram_bytes":"1086608900096","vmm_eligible":"1007530014720"}
{"msg":"91% of 1007530014720 VMM eligible ram = 916852313395 bytes)","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.23220472Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:258"}
{"msg":"Setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.232244165Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:266"}
{"msg":"Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: \"913-0000019\", revision: 6, serial: \"BRM42220015\" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(27), repo_depot_port: 12348, reservoir_size: ByteCount(916851785728), role: Gimlet, sa_address: \"[fd00:1122:3344:109::1]:12345\", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.256005206Z","hostname":"BRM42220015","pid":665,"component":"NexusNotifierTask","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/nexus.rs:500"}
{"msg":"Failed to setup VMM reservoir","v":0,"name":"SledAgent","level":50,"time":"2026-08-12T21:27:06.348569867Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:191","error":"Failed to create reservoir: Failed to resize reservoir: Not enough space (os error 12)"}

We do know that earlier in the day after a reboot we did successfully allocate the reservoir:

{"msg":"Calculated eligible VMM reservoir size","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.461675821Z","hostname":"BRM42220015","pid":666,"component":"MemoryReservations","file":"sled-hardware/src/lib.rs:154","control_plane_earmark_bytes":"47244640256","max_page_t_bytes":"31834245120","physical_ram_bytes":"1086608900096","vmm_eligible":"1007530014720"}
{"msg":"91% of 1007530014720 VMM eligible ram = 916852313395 bytes)","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.462365518Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:258"}
{"msg":"Setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.462378383Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:266"}
{"msg":"Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: \"913-0000019\", revision: 6, serial: \"BRM42220015\" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(25), repo_depot_port: 12348, reservoir_size: ByteCount(916851785728), role: Gimlet, sa_address: \"[fd00:1122:3344:109::1]:12345\", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.484793759Z","hostname":"BRM42220015","pid":666,"component":"NexusNotifierTask","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/nexus.rs:500"}
{"msg":"Finished setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"2026-08-12T14:37:53.242773137Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:270"}

After the failure we see the following:

BRM42220015 # mdb -ke '::memstat'
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   15050243             58790    6%
Boot pages                     11                 0    0%
ZFS File Data            23488784             91753    9%
VMM Memory              214695936            838656   81%
Anon                      6306679             24635    2%
Exec and libs               88068               344    0%
Page cache                   6815                26    0%
Free (cachelist)           313123              1223    0%
Free (freelist)           5333668             20834    2%

Total                   265283327           1036262
Physical                265283325           1036262

We do not currently see this on rack2, but we have a sled on Berlin that has reproduced this behavior multiple times as noted by @leftwo .

BRM44220007 # grep reservoir $(svcs -L sled-agent) | looker
00:00:52.446Z INFO SledAgent (MemoryReservations): Calculated eligible VMM reservoir size
    control_plane_earmark_bytes = 47244640256
    file = sled-hardware/src/lib.rs:154
    max_page_t_bytes = 31834245120
    physical_ram_bytes = 1086608900096
    vmm_eligible = 1007530014720
00:00:52.446Z INFO SledAgent: 91% of 1007530014720 VMM eligible ram = 916852313395 bytes)
    file = sled-agent/src/vmm_reservoir.rs:258
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
00:00:52.447Z INFO SledAgent: Setting reservoir size to 874378 MiB bytes
    file = sled-agent/src/vmm_reservoir.rs:266
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
00:00:52.491Z INFO SledAgent: Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: "913-0000019", revision: 6, serial: "BRM44220007" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(2), repo_depot_port: 12348, reservoir_size: ByteCount(0), role: Gimlet, sa_address: "[fd35:e7fa:ed75:103::1]:12345", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }
    file = sled-agent/src/nexus.rs:500
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
20:28:38.000Z ERRO SledAgent: Failed to setup VMM reservoir
    error = Failed to create reservoir: Failed to resize reservoir: Not enough space (os error 12)
    file = sled-agent/src/vmm_reservoir.rs:191
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255

Specifically we are failing at https://github.com/oxidecomputer/omicron/blob/main/illumos-utils/src/vmm_reservoir.rs#L50 from our current understanding.

[papertigers]

Just to add a little more context, we also saw this on a sled in berlin that @leftwo was using. It seems like we are unable to add all of the pages requested into the reservoir. We end up with a portion of the requested memory in the reservoir but because we get back ENOMEM from the ioctl call that sled-agent is making, we never report how much we did get to the rest of the control plane.

We were able to run /usr/lib/rsrvrctl -s SIZE on a sled (sometimes it took multiple executions) and then restart sled-agent and omdb reported the correct value in crdb. This allowed instances to be allocated on this sled again.

We should probably still report how much memory we were able to secure for the reservoir back to crdb so that the sled doesn't become completely useless. Additionally we probably want some sort of task, either driven by sled-agent or by reconfigurator, that would attempt to keep adding the rest of the memory over time. It's currently unclear why we are unable to get all of the memory up front - when we looked at mdb -ke ::memstat we did see ~20G on the freelist. Checking again revealed that the freelist grew after our first attempt which allowed /usr/lib/rsrvrctl -s SIZE to eventually succeed. We may want to look at how the reservoir adds pages in vmmr_alloc_pages via page_create_va.

[augustuswm]

One sled in particular is failing to manually resize to the value we expect (4 manual executions):

BRM42220042 # /usr/lib/rsrvrctl -s 874378
rsrvrctl: Could not set reservoir size to 916851785728 bytes: Not enough space
BRM42220042 # /usr/lib/rsrvrctl -q
Free MiB:       846848
Allocated MiB:  0
Transient Allocated MiB:        0
Size limit MiB: 988170

BRM42220042 # mdb -ke '::memstat'
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14832408             57939    6%
Boot pages                     11                 0    0%
ZFS File Data            13967143             54559    5%
VMM Memory              216793088            846848   82%
Anon                     12630391             49337    5%
Exec and libs               94919               370    0%
Page cache                   8748                34    0%
Free (cachelist)            89865               351    0%
Free (freelist)           6866754             26823    3%

Total                   265283327           1036262
Physical                265283325           1036262

BRM42220042 # zoneadm list
global
oxz_crucible_08cc7bd6-368e-4d16-a619-28b17eff35af
oxz_crucible_197695e1-d949-4982-b679-6e5c9ab4bcc7
oxz_crucible_390d1853-8be9-4987-b8b6-f022999bf4e7
oxz_crucible_42bb9833-5c39-4aba-b2c4-da2ca1287728
oxz_crucible_55fcfc62-8435-475f-a2aa-29373901b993
oxz_crucible_74b0613f-bce8-4922-93e0-b5bfccfc8443
oxz_crucible_76fe2161-90df-41b5-9c94-067de9c29db1
oxz_ntp_5b3333c3-075f-4926-9561-16557f49bfad
oxz_crucible_adba1a3b-5bac-44d5-aa5a-879dc6eadb5f
oxz_crucible_bf99d4f8-edf1-4de5-98d4-8e6a24965005
oxz_crucible_f49dc522-2b13-4055-964c-8315671096aa
oxz_nexus_89444824-c583-4c0e-95f7-c5ebc5466b79

[augustuswm]

We were able to allocate two additional chunks via /usr/lib/rsrvrctl -a 10000, but we are still short finding the last remaining 27530

[leftwo]

See also: #8257