Can attach 12 disks to instance (11 local, 1 distributed) but VM won't start

[paudmir]

Theoretically, we support attaching 12 disks to an instance.
However, in the latest run on Madrid (where there is plenty of space to allocate local disks), I saw that an instance with 12 disks (11 of those being local) won't boot up.
Checking the error message, we get:

"Insufficient capacity: Failed to place instance: Could not find any valid sled on which this instance can be placed", 

Switched to 1 GiB disks instead (surely, 1 GiB local disks should be doable), but I still got the same message.
@leftwo mentioned there's a condition for local disks to be placed in different physical disks, so even if there is enough storage (as a whole), if the number of physical disks is less than that then the request fails.

Maybe if we are already aware of this constraint, the error message could make a reference to that, since this only gets checked after we attempt booting the VM.

[david-crespo]

Discussed in 5/18 product sync.

• Maybe we can loosen this requirement that each local disk be on a different physical disk
• If not, maybe we can hard cap the number of local disks per instance to 10 since you'll never (for now) have a sled with more than 10 physical disks
• Otherwise we might not be able to solve this in a general way — there are always going to be constraints that only come up on instance startup
• Barring a real fix, the error message could at the very least tell you more about what constraint in failing. As it is, the user doesn't even know they can work around it by reducing the number of local disks.

[hawkw]

Discussed in 5/18 product sync.

• Maybe we can loosen this requirement that each local disk be on a different physical disk

• If not, maybe we can hard cap the number of local disks per instance to 10 since you'll never (for now) have a sled with more than 10 physical disks

We should do this.

• Otherwise we might not be able to solve this in a general way — there are always going to be constraints that only come up on instance startup

That is correct, even if we are able to prevent constructing instances with disks that are unstartable, we have other reasons instances will be unable to start, such as affinity/anti-affinity constraints that are unsatisfiable. And the way errors from any of these "unstartable instance due to unsatisfiable placement constraints" category of start failures are surfaced is...not good.

• Barring a real fix, the error message could at the very least tell you more about what constraint in failing. As it is, the user doesn't even know they can work around it by reducing the number of local disks.

This is the category of thing which RFD 631 (and the related issues #9185 and #9953) are complaining about and which that RFD started proposing a solution to (I have not had the time to finish it unfortunately). RFD 631 proposes solving this with a mechanism for reporting past instance events such as failures to start the instance, in addition to trying to get a better error message into a failure response to an instance start HTTP request. I think we should really build a mechanism for that into the product because, in local-storage-heavy workloads, it's likely that it actually takes so long to determine that the instance cannot start that, by the time we actually have an error to report, the HTTP request to start the instance has already timed out, and thus the error gets lost in the ether. Which is why I think it would be nice if there was a way to ask the control plane why an instance's past start attempts failed.

[hawkw]

Another low-hanging bit of fruit which might at least ease some pain here is that https://docs.oxide.computer/guides/managing-disks-and-snapshots#_local_disks doesn't actually say anything about how we will not place multiple local disks on the same U.2, implying a limit of 10 local disks per instance. So, the error is still a bummer even if we add something to the docs, but it's a worse bummer when we also don't tell you about this up front.