further tune sled-agent memory budgeting

[iximeow]

the Omicron side of my RFD 413 refresh and an extension of #7448.

#7448 was resolved with an improvement to the VMM reservoir calculation but there are further steps for improvement here. in this thread Greg and I discussed the tradeoffs of being more precise with budget terms today. we settled on not being as prescriptive as we could, today, for a few reasons:

• we'd want to start the control_plane_memory_earmark_mb setting much higher than it is: in the range of 170 GiB for current systems
• we'd want to set vmm_reservoir_percentage much higher (90+%)
• we'd really want to be able to tune the control plane earmark down once sled-agent knows which control plane services are not on its sled
• we really want to better understand long-uptime and peak usage behaviors of all services

the first two points here conflict with our relatively poor characterization of the last point.

• we can do better about measuring propolis (propolis-server: want a better accounting of memory propolis#890) and crucible (would be nice to report memory use through Oximeter crucible#1692) over time. this will give us more confidence about stability with a higher vmm_reservoir_percentage
• tune control plane earmark down from an initial conservative guess. this is only half thought-through; if we size the reservoir too aggressively and need to shrink it to move a new CRDB instance onto a sled, there's clearly some overhead there. over long uptimes, presumably this would contribute to fragmenting the VMM reservoir too.
• we should collect memory use of other control plane services to inform these calculations, but as with the Propolis and Crucible issues, collecting address space metrics currently induces stalls in the measured processes, so we wouldn't want to do this in an unattended regular manner.

some additional work to refine RFD 413 numbers:

• "Network slush buffer" describes 18 GiB for buffers handling a 100gbps network link. this is probably an overestimate (it comes from some very boring math: 100 Gbps / 8 yields 12.5GiB per second of buffered data at line rate. so 1.5 seconds gets you 18.75GiB. but in reality on the tx path buffers are loaned from the guest, so assuming jumbo frames that's roughly 9000 that live in the guest plus some structures on the host describing the intended transmission. on the rx path we presumably have to allocate on the host, but there are probably settings for the driver that control how much allocation might be done.
• before I was hoping to measure process memory use, but as @leftwo has pointed out zonestat at least accounts everything in a zone. this is more correct for "how much memory do we use in service of a Propolis" anyway (and is what Josh and Patrick were trying to impress upon me a while ago..). it'd be good to pull this into oximeter or something to see what high watermarks are like across various load tests.

[leftwo]

Some stats to gather to look for memory pressure.
From the switch zone:

pilot host exec -O -c 'echo "::memstat" | mdb -k' 0-31
pilot host exec -O -c 'kstat -m memory_cap -p | egrep  "anaon_alloc_fail|anonpgin|execpgin"' 0-31 | awk '$2 != 0'
pilot host exec -O -c 'kstat -c kmem_cache -p :::reap' 0-31 | awk '$2 != 0'
pilot host exec -O -c "kstat -m zfs -n arcstats -j | /opt/ooce/bin/jq '.[0].data | .c, .c_max'" 0-31
pilot host exec -O -c "kstat -m cpu -n vm -p" 0-31
pilot host exec -O -c "zonestat -r virtual-memory,physical-memory -p 5 1" 0-31

[leftwo]

Testing on dogfood, Gathering the above info before starting I've put it at:
/staff/core/omicron-7911/dogfood-memory-usage-before.txt

To bump up the reservoir, I've run this on gimlets with regular memory which should result
in 874378 in the reservoir:

pilot host exec -O -c '/usr/lib/rsrvrctl -a 45160 -c 1024' 7 8 9 10 11 12 14 16 17 21

For sleds 2,3 (cosmo with less memory), we want 133322 in the reservoir:

/usr/lib/rsrvrctl -a 7546 -c 1024

For sled 23, we want 657102 in the reservoir, so:

/usr/lib/rsrvrctl -a 37194 -c 1024

For sled 25, we want 773492 in the reservoir, so:

/usr/lib/rsrvrctl -a 43782 -c 1024

[leftwo]

The zonestat command I discovered from @jmcarp's pr: #9439