cardano-node: remove iohk-monitoring tracing backend by jutaro · Pull Request #6580 · IntersectMBO/cardano-node

jutaro · 2026-05-26T11:29:28Z

Summary

Removes the legacy iohk-monitoring-based tracing infrastructure from
cardano-node, leaving trace-dispatcher as the sole tracing backend.

trace-dispatcher has been the production default since node 8.x. The old
iohk-monitoring path was retained during a transition period but is no longer
used or tested in production deployments. Removing it eliminates ~11 000 lines
of dead code and 9 transitive build dependencies.

What changed

Deleted (~8 100 lines, 15 modules):

All of Cardano.Tracing.* — Config, HasIssuer, Metrics, Render,
Shutdown, Startup, Tracers, and all OrphanInstances.* modules
(Byron, Common, Consensus, HardFork, Network, Shelley)
Cardano.Node.Configuration.Logging — the iohk-monitoring init layer
Cardano.Node.Tracing.Compat — compatibility shim

Modified:

Cardano.Node.Run — simplified to use only initTraceDispatcher; all
legacy createLoggingLayer / setupTrace / Cardano.BM.* imports gone
Cardano.Node.TraceConstraints — removes HasIssuer, ToObject, and
related iohk-monitoring constraints
cardano-node.cabal — drops iohk-monitoring, lobemo-backend-*,
lobemo-scribe-systemd, ekg-wai, tracer-transformers, scientific
Nix service config — removes legacy logging-layer options
Benchmark profiles — "tracing_backend" updated to "trace-dispatcher"

Breaking change

Any node config still using "TracingBackend": "iohk-monitoring" or legacy
Logging config keys will need to be migrated to trace-dispatcher
configuration before upgrading. Refer to the trace-dispatcher migration guide.

Config options removed with iohk-monitoring

The following node configuration keys have no effect after the removal of the
iohk-monitoring tracing backend. All tracing is now handled exclusively by
trace-dispatcher.

Parsed by the node but silently ignored

These three keys are still accepted by the JSON parser to avoid hard failures
on existing configs, but have no effect:

Key	Previous effect
`TurnOnLogging`	Master on/off switch for the iohk-monitoring logging layer
`TurnOnLogMetrics`	Enabled/disabled node metrics collection via iohk-monitoring
`UseTraceDispatcher`	Chose between legacy (`false`) and trace-dispatcher (`true`) backend

Parsed by `Cardano.Tracing.Config` (module deleted)

Key	Previous effect
`TracingVerbosity`	Rendering verbosity: `MinimalVerbosity` / `NormalVerbosity` / `MaximalVerbosity`
`TraceAcceptPolicy`	Per-tracer on/off toggle (bool)
`TraceBackingStore`	"
`TraceBlockFetchClient`	"
`TraceBlockFetchDecisions`	"
`TraceBlockFetchProtocol`	"
`TraceBlockFetchProtocolSerialised`	"
`TraceBlockFetchServer`	"
`TraceBlockchainTime`	"
`TraceChainDB` / `TraceChainDb`	"
`TraceChainSyncClient`	"
`TraceChainSyncBlockServer`	"
`TraceChainSyncHeaderServer`	"
`TraceChainSyncProtocol`	"
`TraceChurnMode`	"
`TraceConnectionManager`	"
`TraceConnectionManagerCounters`	"
`TraceConnectionManagerTransitions`	"
`TraceCsj`	"
`TraceDevotedBlockFetch`	"
`TraceDiffusionInitialization`	"
`TraceDNS` / `TraceDnsResolver` / `TraceDnsSubscription`	"
`TraceErrorPolicy` / `TraceLocalErrorPolicy`	"
`TraceForge` / `TraceForgeStateInfo`	"
`TraceGDD`	"
`TraceGsm`	"
`TraceHandshake` / `TraceLocalHandshake`	"
`TraceInboundGovernor`	"
`TraceInboundGovernorCounters`	"
`TraceInboundGovernorTransitions`	"
`TraceIpSubscription`	"
`TraceKeepAliveClient` / `TraceKeepAliveProtocol`	"
`TraceKesAgent`	"
`TraceLedgerPeers`	"
`TraceLocalChainSyncProtocol`	"
`TraceLocalConnectionManager`	"
`TraceLocalMux` / `TraceLocalMuxBearer` / `TraceLocalMuxChannel`	"
`TraceLocalRootPeers`	"
`TraceLocalServer`	"
`TraceLocalStateQueryProtocol`	"
`TraceLocalTxMonitorProtocol`	"
`TraceLocalTxSubmissionProtocol` / `TraceLocalTxSubmissionServer`	"
`TraceMempool`	"
`TraceMux` / `TraceMuxBearer` / `TraceMuxChannel`	"
`TracePeerSelection` / `TracePeerSelectionActions` / `TracePeerSelectionCounters`	"
`TracePeerSharingProtocol`	"
`TracePublicRootPeers`	"
`TraceSanityCheckIssue`	"
`TraceServer`	"
`TraceTxCounters`	"
`TraceTxInbound` / `TraceTxOutbound`	"
`TraceTxLogic`	"
`TraceTxSubmissionProtocol` / `TraceTxSubmission2Protocol`	"

Read directly by the `iohk-monitoring` library

These keys were consumed by Cardano.BM.Configuration — the node never parsed
them itself. They are now entirely ignored:

Key	Previous effect
`minSeverity`	Global severity filter (`Debug` / `Info` / `Notice` / `Warning` / `Error` / `Critical`)
`rotation`	Log file rotation (`rpLogLimitBytes`, `rpKeepFilesNum`, `rpMaxAgeHours`)
`setupBackends`	Which backends to initialise (`KatipBK`, `EKGViewBK`, etc.)
`defaultBackends`	Default backend routing for namespaces
`setupScribes`	Log output targets (file, stdout, journald, trace-forwarder)
`defaultScribes`	Default scribe routing
`hasEKG`	Port for EKG web metrics interface
`hasPrometheus`	Host/port for Prometheus metrics endpoint
`options.mapBackends`	Per-namespace backend routing overrides
`options.mapSubtrace`	Per-namespace subtrace behaviour overrides
`options.mapSeverity`	Per-namespace severity filter overrides

Replacements in trace-dispatcher

The trace-dispatcher backend has its own equivalents for severity filtering
and output routing, configured under TraceOptions in the trace-dispatcher
config file. Refer to the trace-dispatcher documentation for the current config
format.

…iring contra-tracer < 0.2

jutaro · 2026-05-26T11:47:28Z

The change in [Byron.hs:122] is:

Before:
( "txid" .= txId tx )
which relied on instance ToJSON (TxId (GenTx ByronBlock)) from the now-deleted Cardano.Tracing.OrphanInstances.Byron, which was:

instance ToJSON (TxId (GenTx ByronBlock)) where
toJSON = String . Text.take 8 . renderTxId
After:

( "txid" .= (Text.take 8 . renderTxId $ txId tx) )
The output is identical — both produce an 8-character hex string. The change just inlines the behavior that used to live in the deleted orphan instance, because that instance no longer exists.

mgmeier · 2026-05-26T13:49:59Z

    configureTracers configReflection trConfig  [shutdownTr]
    shutdownTrDoc <- documentTracer (shutdownTr :: Logging.Trace IO ShutdownTrace)

-    chainDBTr <- mkCardanoTracer'


Why was this removed?

mgmeier · 2026-05-26T13:50:08Z

    consensusStartupErrorTrDoc <- documentTracer (consensusStartupErrorTr ::
      Logging.Trace IO ConsensusStartupException)

-    consensusGddTr <- mkCardanoTracer


Why was this removed?

mgmeier · 2026-05-26T14:00:26Z

+
+instance ToJSON NodeToClientVersionData where
+  toJSON = String . Text.pack . show
+


Isn't there a ToJSON for both of the above in cardano-diffusion/orphan-instances/Cardano/Network/OrphanInstances.hs?
Using Show here seems to lose structure.

mgmeier · 2026-05-26T14:01:52Z

                ["Consensus", "Startup"]
    configureTracers configReflection trConfig [consensusStartupErrorTr]

-    !consensusGddTr <- mkCardanoTracer


Every TraceGDDEvent is silently discarded. This makes Genesis Density Disconnection events completely invisible

mgmeier · 2026-05-26T14:03:04Z

-                             else return $ Last $ Just PartialTracingOff
+      _turnOnLogging :: Maybe Bool <- v .:? "TurnOnLogging"
+      _turnOnLogMetrics :: Maybe Bool <- v .:? "TurnOnLogMetrics"
+      _useTraceDispatcher :: Maybe Bool <- v .:? "UseTraceDispatcher"


These can go, right?

mgmeier · 2026-05-26T14:31:05Z

-            , "decisions" .= if dtal >= DMaximum
-                               then toJSON (Verbose <$> xs)
-                               else toJSON xs
+            , "decisions" .= List.foldl' (\acc x -> forMachine DDetailed x : acc) [] xs


Are you sure this produces the same output?
There's already a mismatch in detail levels.

mgmeier · 2026-05-26T14:37:05Z

                (case pt of
                  GenesisPoint -> "Genesis"
-                  BlockPoint _ h -> renderHeaderHash (Proxy @header) h)
+                  BlockPoint slot _ -> "slot-" <> showT slot)


This is completely wrong.
The output must look like

... "ns":"BlockFetch.Client.CompletedBlockFetch","data":{"block":"00c854f75889b72e14e59e7ed891a467f1b21e28378b092efc786c63b769f67f", ...}

mgmeier · 2026-05-26T14:37:28Z

              [ "kind" .= String "SendFetchRequest"
-              , "head" .= String (renderChainHash
-                                  (renderHeaderHash (Proxy @header))
-                                  (AF.headHash af))


The head key is missing after the change.

mgmeier · 2026-05-26T14:38:02Z

              , "length" .= toJSON (fragmentLength' af)]
              ++
-              [ "deltaq" .= toJSON gsv | dtal >= DDetailed ]
+              [ "deltaq" .= String (Text.pack $ show gsv) | dtal >= DDetailed ]


The field has a ToJSON in network, IIRC. It's an Object, not a String.

mgmeier · 2026-05-26T14:39:04Z

    mconcat
      [ "kind" .= String "TraceMempoolManuallyRemovedTxs"
-      , "txsRemoved" .= txs0
+      , "txsRemoved" .= map (String . Text.pack . show) (toList txs0)


This is suspicious. It might result in the same output, it may not.
This needs scrutiny.

mgmeier and others added 3 commits May 26, 2026 09:47

TRANSITIONAL: enable removal of iohk-monitoring with stack still requ…

487e2ff

…iring contra-tracer < 0.2

Remove old tracing

8a75b01

remove logging switch

50b09d5

jutaro added 2 commits May 26, 2026 13:48

Remove Katip constraint

ad334db

CI Fix

665b9b8

mgmeier requested changes May 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cardano-node: remove iohk-monitoring tracing backend#6580

cardano-node: remove iohk-monitoring tracing backend#6580
jutaro wants to merge 5 commits into
masterfrom
remove-iohk-monitoring

jutaro commented May 26, 2026 •

edited

Loading

Uh oh!

jutaro commented May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

mgmeier May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		instance ToJSON NodeToClientVersionData where
		toJSON = String . Text.pack . show

Conversation

jutaro commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Breaking change

Config options removed with iohk-monitoring

Parsed by the node but silently ignored

Parsed by Cardano.Tracing.Config (module deleted)

Read directly by the iohk-monitoring library

Replacements in trace-dispatcher

Uh oh!

jutaro commented May 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jutaro commented May 26, 2026 •

edited

Loading

Parsed by `Cardano.Tracing.Config` (module deleted)

Read directly by the `iohk-monitoring` library