Skip to content

Add PagerDuty services for bootstrap, content and route-origin - #42

Merged
TakGN merged 2 commits into
mainfrom
pagerduty-priority-services
Oct 20, 2026
Merged

TakGN merged 2 commits into
mainfrom
pagerduty-priority-services

Conversation

@TakGN

@TakGN TakGN commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

bootstrap, content and route-origin, the other priority services on the k3s boxes (sing1, lith1, chic1), are paged today through the needle service by the hub's Alertmanager (ipni/infra, lith1), so an incident doesn't say which service is down.

This gives each its own PagerDuty service on the IPNI escalation policy, with an Events API v2 integration (prevent_destroy, like the others) and its routing key as a sensitive output:

  • bootstrap_integration_key
  • content_integration_key
  • route_origin_integration_key

Box-level alerts stay on needle.

Every service is also named and described by what it does, so a page says it at a glance:

Resource PagerDuty name
swg Service Worker Gateway (inbrowser.link)
indexers IPNI indexers (cid.contact)
needle Delegated routing (needle)
route_origin Routing origin (route-origin)
bootstrap IPFS bootstrap peers (bootstrap)
content IPFS website hosting (content)

Renaming the three existing services is an in-place update; their integration keys (and the Cloudflare connection on the gateway) don't change. Nothing pages the new services until Alertmanager routes to them; that follows in ipni/infra once this is applied (keys in alertmanager-secrets, routes in kube-prometheus-stack.yaml, and the Paging docs).

Expected plan

  • pagerduty: 6 to add (3 pagerduty_service, 3 pagerduty_service_integration), 3 to change (name and description of swg, indexers, needle), 0 to destroy.
  • Cloudflare stacks: not affected.

Checklist

  • The plan comments show only the changes described above
  • Zone changes were tried on inbrowser.dev before inbrowser.link (no zone changes)
  • No private values in the code (origins, IPs, emails, tokens): they go in secrets
  • If this syncs a dashboard change: the dashboard and the code now match (not a sync)

🤖 Generated with Claude Code

These priority services on the k3s boxes are paged today through the
needle service, so an incident doesn't say which service is down. Give
each its own service and Events API v2 integration, with prevent_destroy
like the others, and expose their keys as sensitive outputs for the
hub's Alertmanager. Box-level alerts stay on needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TakGN
TakGN requested a review from nymd as a code owner October 19, 2026 20:07
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Terraform plan: pagerduty: Plan: 6 to add, 3 to change, 0 to destroy.

Applied on merge if the plan is still the same.

Show plan

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # pagerduty_service.bootstrap will be created
  + resource "pagerduty_service" "bootstrap" {
      + acknowledgement_timeout = "1800"
      + alert_grouping          = (known after apply)
      + alert_grouping_timeout  = (known after apply)
      + auto_resolve_timeout    = "14400"
      + created_at              = (known after apply)
      + description             = "kubo nodes that new IPFS peers dial to join the network (bootstrap.ipni.io and the legacy bootstrap.libp2p.io PeerIDs), with WSS for browsers. Paged by Alertmanager on lith1 (ipni/infra). Managed by Terraform: github.com/ipni/terraform-deployments"
      + escalation_policy       = "PYI77LX"
      + html_url                = (known after apply)
      + id                      = (known after apply)
      + last_incident_timestamp = (known after apply)
      + name                    = "IPFS bootstrap peers (bootstrap)"
      + response_play           = (known after apply)
      + status                  = (known after apply)
      + type                    = (known after apply)

      + auto_pause_notifications_parameters (known after apply)

      + incident_urgency_rule (known after apply)
    }

  # pagerduty_service.content will be created
  + resource "pagerduty_service" "content" {
      + acknowledgement_timeout = "1800"
      + alert_grouping          = (known after apply)
      + alert_grouping_timeout  = (known after apply)
      + auto_resolve_timeout    = "14400"
      + created_at              = (known after apply)
      + description             = "kubo and ipfs-cluster pinning the IPFS Project websites on every box, and the pin.ipni.io remote pinning API. Paged by Alertmanager on lith1 (ipni/infra). Managed by Terraform: github.com/ipni/terraform-deployments"
      + escalation_policy       = "PYI77LX"
      + html_url                = (known after apply)
      + id                      = (known after apply)
      + last_incident_timestamp = (known after apply)
      + name                    = "IPFS website hosting (content)"
      + response_play           = (known after apply)
      + status                  = (known after apply)
      + type                    = (known after apply)

      + auto_pause_notifications_parameters (known after apply)

      + incident_urgency_rule (known after apply)
    }

  # pagerduty_service.indexers will be updated in-place
  ~ resource "pagerduty_service" "indexers" {
      ~ description             = "Indexers on sf and sf2, paged by grafana.ipni.io. Managed by Terraform: github.com/ipni/terraform-deployments" -> "Map CIDs to the providers that advertise them: storetheindex on sf and sf2, behind cid.contact. Paged by grafana.ipni.io. Managed by Terraform: github.com/ipni/terraform-deployments"
        id                      = "P0EEBIT"
      ~ name                    = "IPNI indexers" -> "IPNI indexers (cid.contact)"
        # (8 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # pagerduty_service.needle will be updated in-place
  ~ resource "pagerduty_service" "needle" {
      ~ description             = "Routing service on the k3s boxes, paged by Alertmanager on lith1 (ipni/infra). Managed by Terraform: github.com/ipni/terraform-deployments" -> "Answers provider, peer and IPNS lookups for delegated-ipfs.dev from the Amino DHT and cid.contact, on sing1, lith1 and chic1. Also gets the boxes' own alerts (box down, ZFS pool). Paged by Alertmanager on lith1 (ipni/infra). Managed by Terraform: github.com/ipni/terraform-deployments"
        id                      = "PB38C92"
      ~ name                    = "needle" -> "Delegated routing (needle)"
        # (8 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # pagerduty_service.route_origin will be created
  + resource "pagerduty_service" "route_origin" {
      + acknowledgement_timeout = "1800"
      + alert_grouping          = (known after apply)
      + alert_grouping_timeout  = (known after apply)
      + auto_resolve_timeout    = "14400"
      + created_at              = (known after apply)
      + description             = "Envoy in front of needle: the origin of delegated-ipfs.dev and route-<box>.ipni.io, with TLS, the path allowlist and rate limits. Paged by Alertmanager on lith1 (ipni/infra). Managed by Terraform: github.com/ipni/terraform-deployments"
      + escalation_policy       = "PYI77LX"
      + html_url                = (known after apply)
      + id                      = (known after apply)
      + last_incident_timestamp = (known after apply)
      + name                    = "Routing origin (route-origin)"
      + response_play           = (known after apply)
      + status                  = (known after apply)
      + type                    = (known after apply)

      + auto_pause_notifications_parameters (known after apply)

      + incident_urgency_rule (known after apply)
    }

  # pagerduty_service.swg will be updated in-place
  ~ resource "pagerduty_service" "swg" {
      ~ description             = "inbrowser.link, paged by Cloudflare Notifications. Managed by Terraform: github.com/ipni/terraform-deployments" -> "Fetches and verifies IPFS content in the visitor's browser, at inbrowser.link. Paged by Cloudflare Notifications. Managed by Terraform: github.com/ipni/terraform-deployments"
        id                      = "P43ELBB"
      ~ name                    = "Service Worker Gateway" -> "Service Worker Gateway (inbrowser.link)"
        # (8 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # pagerduty_service_integration.bootstrap_events will be created
  + resource "pagerduty_service_integration" "bootstrap_events" {
      + email_filter_mode       = (known after apply)
      + email_incident_creation = (known after apply)
      + email_parsing_fallback  = (known after apply)
      + html_url                = (known after apply)
      + id                      = (known after apply)
      + integration_email       = (known after apply)
      + integration_key         = (known after apply)
      + name                    = "Events API v2"
      + service                 = (known after apply)
      + type                    = "events_api_v2_inbound_integration"
      + vendor                  = (known after apply)

      + email_filter (known after apply)
    }

  # pagerduty_service_integration.content_events will be created
  + resource "pagerduty_service_integration" "content_events" {
      + email_filter_mode       = (known after apply)
      + email_incident_creation = (known after apply)
      + email_parsing_fallback  = (known after apply)
      + html_url                = (known after apply)
      + id                      = (known after apply)
      + integration_email       = (known after apply)
      + integration_key         = (known after apply)
      + name                    = "Events API v2"
      + service                 = (known after apply)
      + type                    = "events_api_v2_inbound_integration"
      + vendor                  = (known after apply)

      + email_filter (known after apply)
    }

  # pagerduty_service_integration.route_origin_events will be created
  + resource "pagerduty_service_integration" "route_origin_events" {
      + email_filter_mode       = (known after apply)
      + email_incident_creation = (known after apply)
      + email_parsing_fallback  = (known after apply)
      + html_url                = (known after apply)
      + id                      = (known after apply)
      + integration_email       = (known after apply)
      + integration_key         = (known after apply)
      + name                    = "Events API v2"
      + service                 = (known after apply)
      + type                    = "events_api_v2_inbound_integration"
      + vendor                  = (known after apply)

      + email_filter (known after apply)
    }

Plan: 6 to add, 3 to change, 0 to destroy.

Changes to Outputs:
  + bootstrap_integration_key    = (sensitive value)
  + content_integration_key      = (sensitive value)
  + route_origin_integration_key = (sensitive value)

Run · commit 6da6b10

The names now say what each service is (Delegated routing (needle),
Routing origin (route-origin), IPFS bootstrap peers (bootstrap), IPFS
website hosting (content), IPNI indexers (cid.contact), Service Worker
Gateway (inbrowser.link)), and each description says what it does
before who pages it. Renaming the three existing services is an
in-place update; their integration keys don't change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TakGN
TakGN merged commit ebcc0de into main Oct 20, 2026
5 checks passed
@TakGN
TakGN deleted the pagerduty-priority-services branch October 20, 2026 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant