Label multiscales' reference to the `source.image` is broken; should label a coordinate system instead

[clbarnes]

Currently, label multiscales

• MUST be in a descendant of a labels group which MUST be a direct child of the multiscale image they label
• MUST be listed in the ome.labels key of the labels group
• SHOULD contain an ome.image-label (kebab 🙃) object which SHOULD contain an image key with a path to zarr node containing the data for the multiscale image they label

However, because ome.multiscales is an array, this relationship between a label multiscale and the multiscale image it labels is not meaningfully specified. This would be resolved by RFC-6 Flatten Multiscale Array but enthusiasm for that was low at the Zurich hackathon 2025.

There are two concerns here:

• rendering: if we don't know which multiscale a label refers to, we don't know how to overlay it. The spec requires that the label multiscale has the same number of scale levels as the multiscale image, but does not require that they downscale in the same way or starting from the same resolution.
• provenance: knowing how labels were generated would be nice (although this is currently not captured and defining an ontology of labeling methods would be a big task); labels could feasibly be generated based on multiple multiscale images or on only a subset of one (e.g. a single channel).

The rendering case should be at least partially covered by RFC-5 Coordinate Systems. It shouldn't be a multiscale image which is labelled, it should a a coordinate system. This would allow us to remove a number of existing restrictions

• the label group's location, as it no longer belongs strictly to a single multiscale image
• (implicitly) matching the multiscale image's shape, base resolution, and downscaling scheme, as it becomes just another object in the scene (so we could label only a specific ROI, or at a coarse resolution for e.g. labelling cell bodies in an EM volume for connectomics)

However, we still want to be able to distinguish between a label multiscale and a multiscale image. We could achieve this with a simple flag in the Multiscale object:

{
  "ome": {
    "multiscales": [
      {
        "datasets": [...],
        "coordinateTransformations": [...],
        ...
      },
      {
        "datasets": [...],
        "coordinateTransformations": [...],
        "isLabel": true,
        ...
      }
    ]
  }
}

This would provide a use case for multiscales being an array, and would allow us to additionally remove the restriction of having to list label multiscales in the ancestor labels Zarr group. Note that it moves the multiscale information about the label multiscale into a different metadata document.

Alternatively, we could rearrange the scene metadata/ discovery procedure to list multiscale images and labels but I haven't thought that through yet. In the majority of cases I imagine that multiscale images and their associated label multiscales probably want to be converted into a shared coordinate system, which could then be converted into other coordinate systems listed in the scene.

This does not solve the lack of provenance information, but we don't meaningfully store that today so no loss.

[clbarnes]

Pinging @bogovicj @jo-mueller @d-v-b @will-moore who were present on the call today when I raised this (Luca and Eric could also be brought in, don't have their handles to hand).

[jo-mueller]

Cc @LucaMarconato @perlman

[LucaMarconato]

Thanks for the thorough explanation!

• The label group's location, as it no longer belongs strictly to a single multiscale image
• (implicitly) matching the multiscale image's shape, base resolution, and downscaling scheme, as it becomes just another object in the scene (so we could label only a specific ROI, or at a coarse resolution, e.g., labeling cell bodies in an EM volume for connectomics)

Seeing these relaxed would be great! I also agree that RFC-5 will help with rendering, and I would be happy enough with that, and with waiting until collections are ready to be able to specify more complex labels <> image relationships.

[will-moore]

A couple of thoughts on this...

• If the multiscales list is considered deprecated (RFC-6 is merely rolled into RFC-8 and not "dead") then we might not want to start putting multiple items in the multiscales list.
• One advantage of nesting labels below images (instead of siblings under a scene) is that when you open the image, it's easier to check whether it contains any labels. Otherwise, if you want to check for labels when you view the image, you have to open the parent group to check whether it contains any images, and then either open all the images in the scene, or just check for images that are labels (with the same transforms that will overlay the image and the label-image?).
• It would also change how labels are handled in Plates, if we need to use a scene to link an image and it's label-image, as discussed at Allow nesting HCS and Scenes? #476

What needs to happen to fix or work around this issue for RFC5 (v0.6)?
In my draft implementation at napari-ome-zarr, will-moore/napari-ome-zarr#4 I was treating the label-image dataset arrays as being in the same coordinate system as the parent image's dataset arrays (and assuming that multiscales was a single-item list). That was informed by the spec saying that segmentation etc creates an "image that is in the same coordinate system as a corresponding multiscale image (usually having the same dimensions and coordinate transformations)".
It may be sufficient to update the wording of the spec to clarify this relationship and how to handle it when viewing the images, without changing the way that we store lmages/labels from v0.5 and earlier. Non-breaking changes are always nice!

[jo-mueller]

A possible way out that would work well within the confines of the current transformations. Considering the example for labels on the ngff-validator, we have roughly this structure to consider:

root
  zarr.json  # multiscales metadata lives here
  s0
  s1
  s2
  labels
    zarr.json  # "labels" metadata lives here
    0
      zarr.json  # this is actually just another multiscales with extra fields for labels
      s0
      s1
      s2

In the top-level multiscales group, the contents could look like this:

{
  "ome": {
  "version": "0.6" ,
  "multiscales": [
      {
        "coordinateSystems": [
          {
            "name": "physical",
            "axes": [
              {"name": "z", "type": "space"},
              {"name": "y", "type": "space"},
              {"name": "x", "type": "space"}
            ]
          }
        ],
        "datasets": [{...}, {...}],
        "coordinateTransformations":
        [
          {
            "type": "identity",
            "output": {"name": "physical", "path": "labels/0"},
            "input": {"name": "physical"},
            "name": "transform_to_label_image_0"
          }
        ]
      }
    ]
  }
}

This has several advantages:

• Usage of input/output fields becomes more homogeneous throughout the spec - see also RFC5 - de-duplicate allowed ways to specify named coordinate system #437
• By using other transformations (only scale, identity and translation should be allowed) it would be possible to explicitly and safely use label images obtained from different resolution levels without any implicit assumptions. Plus, this is a very common use case that people would probably like/want?
• Portability: The reference is strictly top-down instead of bottom-up. The label image metadata currently contains "source": "../.." which comes with some drawbacks.
• Labels metadata could stay as it is.

[will-moore]

The only fly in the ointment there is that within the same multiscales image, you have a mix of input and output values as "strings" and some as objects {"name": "physical", "path": "labels/0"}. So, maybe we change that to use input & output as objects everywhere? We decided not to do that before, but I forget why?

Also, I would expect that the input would be the labels and output would be the parent?
Like this (but I guess either is fine, depending on the graph you want):

          {
            "type": "identity",
            "output": {"name": "physical"},
            "input": {"name": "physical", "path": "labels/0"},
            "name": "transform_label_to_image_0"
          }

[jo-mueller]

Also, I would expect that the input would be the labels and output would be the parent?
[...] but I guess either is fine, depending on the graph you want

Jep, pretty much.

The only fly in the ointment there is that within the same multiscales image, you have a mix of input and output values as "strings" and some as objects {"name": "physical", "path": "labels/0"}.

I mean...currently we already have a mix of three conventions:

• Inside multiscales > datasets: input must be a string, which is actually a path to a zarr array. Output must be a string of a coordinate system in the same multiscale dataset
• Inside multiscales > coordinateTransformations: Currently input/output must be strings of coordinate systems in the same metadata. This would change to "input": {"name": "physical", "path": "labels/0"}, which could refer to either a coordinate system in the same metadata (no path), or to a coordinate system of a labels image (path and name)
• Inside scene: "input": {"name": "physical", "path": "labels/0"} - remains unchanged

The point being, that we already have a bit of mixture here, although input and outputs in the multiscales are always strings, they just mean something different depending on the location they're used at.

I would be fine with leaving multiscales transformations unchanged OR changing it. But for the sake of sticking to solving one issue at a time, I would prefer to make a single change at a time.

[jo-mueller]

...or, the more I think about it: Maybe we do change the spec for input/output everywhere to be an object of {"name": "physical", "path": "labels/0"} everywhere? For the multiscales transformations, this would essentially look like this:

{
  "type": "scale":
  "input": {"path": "s0"},
  "output": {"name": "intrinsic"},
  "scale": [1, 1, 1]
}

... and also close #437 in the process.

[will-moore]

Reading #437 again, I think this additional issue probably pushes that proposal over the line, to "let's do it".

[jo-mueller]

Agree. On it!

[jo-mueller]

0.6.dev4 here we go

[clbarnes]

So path, when given, always points at a Zarr array, and name, when given, always points at a coordinate system defined in this document?

Is there any chance of coordinate systems being defined in multiple places in one document, e.g. in scene and multiscales metadata?

Can path and name be used together so you can get a named coordinate system from any other metadata document?

This is partially what motivated my suggestion of always promoting coordinate systems into the top level ome object, so that if you have a path (I called it node) and a name (I called it coordinateSystem) you have one flat list to look into, rather than having to dig in to several places.

[jo-mueller]

So path, when given, always points at a Zarr array, and name, when given, always points at a coordinate system defined in this document?

I'd say more generally, to a Zarr group, not to zarr array? In the special case of the multiscales transforms, that group is a zarr array.

Can path and name be used together so you can get a named coordinate system from any other metadata document?

Yes. We do that in the scene. With the restriction that you can never point up, only down.