[Bug] Operation state metrics may become inaccurate when state transition fails or races

[HanShuhang]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

KyuubiOperation#setState updates operation state metrics before the actual state transition is validated and applied by AbstractOperation#setState.

Current flow:

override def setState(newState: OperationState): Unit = {
  MetricsSystem.tracing { ms =>
    if (!OperationState.isTerminal(state)) {
      ms.markMeter(MetricRegistry.name(OPERATION_STATE, opType, state.toString.toLowerCase), -1)
    }
    ms.markMeter(MetricRegistry.name(OPERATION_STATE, opType, newState.toString.toLowerCase))
    ms.markMeter(MetricRegistry.name(OPERATION_STATE, newState.toString.toLowerCase))
  }
  super.setState(newState)
}

This can make operation state metrics inconsistent with the real operation state.

There are two related problems:

1. If AbstractOperation#setState rejects a stale or invalid transition, metrics may already have been updated.
2. Concurrent state transitions for the same operation are not protected as one atomic transition-and-metrics-update sequence, so multiple threads may update metrics based on the same stale old state.

Affects Version(s)

master

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @HanShuhang,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.