Skip to content

Reuse existing aggregation when resolving summary vector blob id - #2

Merged
jorgenherje merged 2 commits into
OPM:mainfrom
magnesj:reuse-existing-aggregation
Aug 12, 2026
Merged

Reuse existing aggregation when resolving summary vector blob id#2
jorgenherje merged 2 commits into
OPM:mainfrom
magnesj:reuse-existing-aggregation

Conversation

@magnesj

@magnesj magnesj commented Aug 11, 2026

Copy link
Copy Markdown
Member

Resolving the blob id for a summary vector triggered a new aggregation on Sumo on every call, costing around 5 seconds per vector even when the aggregation already existed. Loading four vectors for one well took 29 seconds, 21 of which was blob id resolution.

Cause

SearchContext.aggregation_async() looks for an existing aggregation by adding an aggregation filter to the context it is called on:

sc = self.filter(aggregation=operation, column=column)
numaggs = await sc.length_async()
if numaggs == 1:
    ...
    return agg
# ELSE
return await self.filter(realization=True).aggregate_async(...)

Filters accumulate rather than replace, and realization and aggregation map to different fields. The context passed in from _get_vector_agg_table carried realization=True, so the probe ended up requiring both:

must : [{"exists": {"field": "fmu.realization.id"}},
        {"term": {"fmu.aggregation.operation.keyword": "collection"}}]
hits : 0

No FMU object is both a realization member and an aggregation, so the probe matched nothing for every vector on every call, and the code always fell through to POST /aggregations plus a Retry-After poll. The fallback returns a correct result, so the only symptom was latency.

The last line of aggregation_async shows the implicit contract: it adds realization=True itself for the trigger path, so it expects the context not to be realization-filtered already.

Change

Probe for an existing aggregation on a separate context, without the realization filter:

must : [{"term": {"fmu.aggregation.operation.keyword": "collection"}}]
hits : 1

The aggregation is reused only when _is_aggregation_current confirms it still covers all realizations. That check mirrors what aggregation_async applies before reusing an aggregation, so an aggregation made before further realizations were uploaded is not handed out. Triggering aggregation is unchanged and remains the fallback.

Note that setting realization=False instead does not work: aggregate_async unconditionally re-adds realization=True, giving a query that both requires and forbids fmu.realization.id, and blob id resolution then returns nothing.

Measurements

Case e7f117b6-29fe-488f-989c-dbbc9bd03f09, ensemble iter-0, vectors WOPR:A1, WGOR:A1, WWCT:A1, WWPR:A1, driven from ResInsight through the service.

before after
blob id, per vector 4671-5947 ms 593-624 ms
four vectors end to end 28903 ms 9481 ms

Downloaded payloads are byte identical before and after (71468 / 69169 / 71616 / 71612 bytes), and the blob ids match.

The fallback was verified separately on a vector with no existing aggregation:

WBHP:A3    1st     907 ms (reused existing)        2nd  591 ms   same blob
WGPR:A4    1st    5753 ms (triggered aggregation)  2nd  607 ms   same blob
FPR        1st     617 ms (reused existing)        2nd  634 ms   same blob
WOPT:A2    1st     589 ms (reused existing)        2nd  617 ms   same blob

WGPR:A4 had no aggregation, triggered one, and the result was picked up by the fast path on the next call.

Upstream

This is a workaround for what looks like a defect in fmu-sumo: aggregation_async (and the sync aggregation) probe on whatever context they are called on, with no way to drop an inherited realization clause, and assert numaggs <= 1 cannot catch the case. A fix there would make this workaround unnecessary.

Resolving the blob id for a summary vector triggered a new aggregation on Sumo on every call, taking around 5 seconds per vector even when the aggregation already existed.

SearchContext.aggregation_async() looks for an existing aggregation by adding an aggregation filter to the context it is called on. Filters accumulate, so the per-realization context passed in here contributed a must clause on fmu.realization.id while the probe added one on fmu.aggregation.operation. No object carries both, so the probe matched nothing and the aggregation was always re-triggered.

Probe for an existing aggregation on a separate context without the realization filter, and reuse it when it still covers all realizations. Triggering aggregation is unchanged and remains the fallback. Measured against a case with four vectors, blob id resolution went from around 5000 ms to around 600 ms per vector.
@jorgenherje
jorgenherje self-requested a review August 12, 2026 05:54

@jorgenherje jorgenherje left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and looks good. Made minor adjustments for readability

@jorgenherje
jorgenherje merged commit 67826bf into OPM:main Aug 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants