Cloud Action Costs Dashboard

[lainecsmith]

What was changed

Adds a new community Grafana dashboard for Temporal Cloud, sourced entirely from the OpenMetrics endpoint, plus a companion alert rules file:

temporal_cloud_action_costs.json — dashboard with two sections:

• Summary: 1-day / 7-day / month-to-date Action totals.
• Action Spike Detection: current Action rate vs. the same time 24h and 7d ago, percent-change indicators, a breakdown by Workflow Type and Action Type, and a ranked table of top Workflow Types by Action volume.

temporal_cloud_action_costs_alerts.yaml — optional Grafana unified-alerting provisioning file with 2 rules (action rate spike vs 24h ago, vs 7d ago).

Scope is intentionally limited to Action volume — Temporal Cloud doesn't expose price or credit-ledger balance via metrics, so this dashboard doesn't show cost in dollars or track spend against a commitment. That's called out explicitly in the dashboard's intro panel, with a pointer to Billing Center / the Billing API for anyone who needs authoritative $ figures.

Why?

Two of the most common cost-support conversations with customers are (1) someone ships a change or a new workflow that quietly drives a lot more Actions than expected, and nobody notices until the bill, and (2) there's no proactive signal — you only find out by going and looking. This gives customers a self-serve starting point for both: a dashboard to see it, and alert rules so it can page/notify instead of requiring someone to be staring at Grafana when it happens.

Checklist

How was this tested:

• Created a Temporal Cloud Service Account with the Metrics Read-Only role and generated an API key; verified the OpenMetrics endpoint directly with curl before touching Grafana.
• Ran Prometheus in a homelab Docker Compose stack, added a scrape job against metrics.temporal.io with Bearer auth, and confirmed the temporal-cloud job showed UP in Prometheus's own Targets page.
• Imported the dashboard into a homelab Grafana instance against that Prometheus data source; confirmed the temporal_namespace variable populates from label_values(), and confirmed (separately, via a direct endpoint query) that namespaces without recent Action traffic simply don't appear — expected behavior given OpenMetrics' sparse reporting, not a bug — and added a note about this to the dashboard's intro panel so it's not mistaken for one.
• Verified the account has 85 namespaces total but only 2 have recent traffic; confirmed the dashboard's namespace variable correctly surfaces those 2 and that switching to "All" sums correctly against real data.
• Sanity-checked the Actions MTD figure against the Cloud UI's own Usage Dashboard for the same namespace/timeframe — same order of magnitude (exact match not expected; different counting methodology, called out in the docs).

Any docs updates needed?
Updated cloud/README.md in this PR with a link to the new dashboard the alerts.
No docs.temporal.io changes made as part of this PR.

[prathyushpv]

does this file temporal_cloud_alert_rules.yaml exist?

[lainecsmith]

It does, except I renamed it and forgot to update here! Fixed, thanks!

[prathyushpv]

Should it be Estimated Actions?

[lainecsmith]

Sure, changed.

[Copilot]

Pull request overview

Adds a new community Grafana dashboard (plus optional unified-alerting provisioning) to help Temporal Cloud users monitor Action volume month-to-date and detect sudden Action-rate spikes using the OpenMetrics endpoint.

Changes:

• Added a new Grafana dashboard for Action MTD totals and spike-detection breakdowns.
• Added Grafana unified-alerting provisioning rules to alert on action-rate spikes vs 24h/7d baselines.
• Updated cloud README to link the new dashboard and sample alerts.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
cloud/temporal_cloud_action_costs.json	New Grafana dashboard for Action MTD totals, spike comparisons vs 24h/7d ago, and workflow/action-type breakdowns.
cloud/temporal_cloud_action_costs_alerts.yaml	New Grafana unified-alerting provisioning file with two spike-detection alert rules.
cloud/README.md	Adds links to the new dashboard and sample alert rules.

Suppressed comments (3)

cloud/temporal_cloud_action_costs.json:206

• This panel description says "billable" actions, but the query uses temporal_cloud_v1_total_action_count (not ..._billable_action_count). Either switch the query to the billable metric or adjust the description to avoid implying it is billable.

      "description": "Estimated total billable actions over the last 7 days",

cloud/temporal_cloud_action_costs.json:614

• This percent-change expression can divide by zero when the offset value is 0 (or missing), producing +Inf/NaN in the Stat panel and making the paired alerting behavior noisy. Clamp the denominator to a tiny positive value to keep the expression finite.

          "expr": "(sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"}) - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 7d)) / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 7d) * 100",

cloud/temporal_cloud_action_costs_alerts.yaml:115

• This alert expression can divide by zero when the offset value is 0 (or missing), producing +Inf/NaN and making the alert noisy. Clamp the denominator to a tiny positive value to keep the evaluation finite.

                (sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"})
                - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 7d))
                / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 7d) * 100

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.