From 6382cca615b870bf977357fcde6b168b24e911ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ois=C3=ADn=20Kyne?= Date: Fri, 3 Jul 2026 02:38:34 +0100 Subject: [PATCH 1/3] Fable pass on the site --- .gitignore | 1 + CLAUDE.md | 11 +- SUMMARY.md | 39 ++++ .../advanced/consensus-protocols.md | 88 ++++++++ .../advanced/feature-flags.md | 75 ++++++ .../advanced/self-relay.md | 2 +- .../security/risks.md | 2 +- .../troubleshooting/client_configurations.md | 6 +- .../troubleshooting/duty-failure-reasons.md | 213 ++++++++++++++++++ .../troubleshooting/errors.md | 2 + .../troubleshooting/test_command.md | 4 +- learn/charon/charon-cli-reference.md | 8 +- learn/charon/intro.md | 2 +- learn/further-reading/resources.md | 4 +- learn/intro/key-concepts.md | 4 +- learn/intro/obol-collective.md | 2 +- obol-stack/README.md | 2 +- obol-stack/build-a-profitable-stack.md | 2 +- obol-stack/faq.md | 2 +- obol-stack/quickstart.md | 2 +- run-a-dv/editing/recreate-private-keys.md | 2 +- run-a-dv/integrations/lido-csm.md | 2 +- .../lido-v3-stvault-for-capital-allocators.md | 2 +- .../lido-v3-stvault-for-node-operator.md | 2 +- run-a-dv/prepare/deployment-best-practices.md | 2 +- run-a-dv/prepare/test-a-cluster.md | 18 +- run-a-dv/running/claim-rewards.md | 2 +- run-a-dv/running/metrics.md | 149 ++++++++++++ run-a-dv/running/monitoring.md | 2 +- run-a-dv/start/create-a-dv-with-a-group.md | 4 +- sdk/index.md | 2 +- sdk/variables/README.md | 2 + 32 files changed, 611 insertions(+), 49 deletions(-) create mode 100644 advanced-and-troubleshooting/advanced/consensus-protocols.md create mode 100644 advanced-and-troubleshooting/advanced/feature-flags.md create mode 100644 advanced-and-troubleshooting/troubleshooting/duty-failure-reasons.md create mode 100644 run-a-dv/running/metrics.md create mode 100644 sdk/variables/README.md diff --git a/.gitignore b/.gitignore index e43b0f98..55045954 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ .DS_Store +*-validators.json \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 85a9ec68..f5600c86 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,14 +18,11 @@ This is the Obol Network GitBook documentation repository containing comprehensi - **`sdk/`** - Auto-generated SDK documentation - **`walkthrough-guides/`** - Step-by-step tutorials -### Duplicate Structure Pattern -The repository contains two parallel directory structures: -- **Long-form paths** (e.g., `run-a-dv/`, `advanced-and-troubleshooting/`) - Main content -- **Short-form paths** (`adv/`, `gov/`, `guides/`, `learn/`, `sdk/`) - Contains `_category_.json` files and `.mdx` versions +### Navigation +- `SUMMARY.md` is the single source of truth for navigation; it lists every page in a nested bullet structure. +- A handful of `_category_.json` files survive under `learn/` and `sdk/` from an old Docusaurus setup. They are inert leftovers, ignored by GitBook, and should not be used or extended. ### GitBook Configuration -- Uses `_category_.json` files for navigation structure with properties: `label`, `position`, `collapsed`, `collapsible`, `className` -- Supports both `.md` and `.mdx` file formats - Front matter with `description`, `sidebar_position`, and other metadata - Image assets stored in `.gitbook/assets/` directory @@ -88,7 +85,6 @@ Two distinct content types used throughout the documentation: ### File Organization - Keep related content together (e.g., all quickstart guides in `run-a-dv/start/`) - Use descriptive filenames that match content purpose -- Maintain parallel structure between long-form and short-form directories - Update `SUMMARY.md` table of contents when adding new content ### Cross-References @@ -101,7 +97,6 @@ Two distinct content types used throughout the documentation: ### Content Updates - Edit markdown files directly in appropriate directory structure -- Update both `.md` and `.mdx` versions if both exist - Refresh `SUMMARY.md` when adding new pages - Update version references (e.g., Charon version in CLI reference) diff --git a/SUMMARY.md b/SUMMARY.md index 4a14eef5..d234aa0c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -41,6 +41,7 @@ * [Claim Rewards](run-a-dv/running/claim-rewards.md) * [Update a DV](run-a-dv/running/update-a-dv.md) * [Monitoring Your Node](run-a-dv/running/monitoring.md) + * [Charon Metrics Reference](run-a-dv/running/metrics.md) * [Exit a DV](run-a-dv/running/exit-a-dv.md) * [Edit a Cluster](run-a-dv/editing/README.md) * [Adding Validators](run-a-dv/editing/add-validators.md) @@ -77,11 +78,14 @@ * [Advanced Docker Configs](advanced-and-troubleshooting/advanced/adv-docker-configs.md) * [Assign OVM Roles](advanced-and-troubleshooting/advanced/assign-ovm-roles.md) * [NAT Hole Punching](advanced-and-troubleshooting/advanced/hole-punching.md) + * [Consensus Protocols](advanced-and-troubleshooting/advanced/consensus-protocols.md) + * [Charon Feature Flags](advanced-and-troubleshooting/advanced/feature-flags.md) * [Troubleshooting](advanced-and-troubleshooting/troubleshooting/README.md) * [Errors & Resolutions](advanced-and-troubleshooting/troubleshooting/errors.md) * [Handling DKG Failure](advanced-and-troubleshooting/troubleshooting/dkg_failure.md) * [Client Configuration](advanced-and-troubleshooting/troubleshooting/client_configurations.md) * [Test Commands](advanced-and-troubleshooting/troubleshooting/test_command.md) + * [Duty Failure Reasons](advanced-and-troubleshooting/troubleshooting/duty-failure-reasons.md) * [Security](advanced-and-troubleshooting/security/README.md) * [Overview](advanced-and-troubleshooting/security/overview.md) * [Centralization Risks and Mitigation](advanced-and-troubleshooting/security/risks.md) @@ -129,12 +133,27 @@ * [FORK\_MAPPING](sdk/enumerations/FORK_MAPPING.md) * [Classes](sdk/classes/README.md) * [Client](sdk/classes/Client.md) + * [EOA](sdk/classes/EOA.md) + * [Exit](sdk/classes/Exit.md) + * [Incentives](sdk/classes/Incentives.md) + * [ObolSplits](sdk/classes/ObolSplits.md) * [Interfaces](sdk/interfaces/README.md) * [ClusterDefinition](sdk/interfaces/ClusterDefinition.md) + * [ExistingExitValidationBlobData](sdk/interfaces/ExistingExitValidationBlobData.md) + * [ExitClusterConfig](sdk/interfaces/ExitClusterConfig.md) + * [ExitClusterDefinition](sdk/interfaces/ExitClusterDefinition.md) + * [ExitDistributedValidator](sdk/interfaces/ExitDistributedValidator.md) + * [ExitOperator](sdk/interfaces/ExitOperator.md) + * [ExitValidationBlob](sdk/interfaces/ExitValidationBlob.md) + * [ExitValidationMessage](sdk/interfaces/ExitValidationMessage.md) + * [ExitValidationPayload](sdk/interfaces/ExitValidationPayload.md) * [RewardsSplitPayload](sdk/interfaces/RewardsSplitPayload.md) + * [SignedExitValidationMessage](sdk/interfaces/SignedExitValidationMessage.md) * [Type-Aliases](sdk/type-aliases/README.md) * [BuilderRegistration](sdk/type-aliases/BuilderRegistration.md) * [BuilderRegistrationMessage](sdk/type-aliases/BuilderRegistrationMessage.md) + * [ClaimableIncentives](sdk/type-aliases/ClaimableIncentives.md) + * [ClaimIncentivesResponse](sdk/type-aliases/ClaimIncentivesResponse.md) * [ClusterCreator](sdk/type-aliases/ClusterCreator.md) * [ClusterLock](sdk/type-aliases/ClusterLock.md) * [ClusterOperator](sdk/type-aliases/ClusterOperator.md) @@ -142,12 +161,32 @@ * [ClusterValidator](sdk/type-aliases/ClusterValidator.md) * [DepositData](sdk/type-aliases/DepositData.md) * [DistributedValidator](sdk/type-aliases/DistributedValidator.md) + * [EOADepositPayload](sdk/type-aliases/EOADepositPayload.md) + * [EOAWithdrawalPayload](sdk/type-aliases/EOAWithdrawalPayload.md) * [ETH\_ADDRESS](sdk/type-aliases/ETH_ADDRESS.md) + * [HttpRequestFunc](sdk/type-aliases/HttpRequestFunc.md) * [OperatorPayload](sdk/type-aliases/OperatorPayload.md) + * [OVMArgs](sdk/type-aliases/OVMArgs.md) + * [OVMBaseSplitPayload](sdk/type-aliases/OVMBaseSplitPayload.md) + * [OVMDepositPayload](sdk/type-aliases/OVMDepositPayload.md) + * [OVMRequestWithdrawalPayload](sdk/type-aliases/OVMRequestWithdrawalPayload.md) + * [OVMRewardsSplitPayload](sdk/type-aliases/OVMRewardsSplitPayload.md) + * [OVMSplitPayload](sdk/type-aliases/OVMSplitPayload.md) + * [OVMTotalSplitPayload](sdk/type-aliases/OVMTotalSplitPayload.md) + * [OWRTranches](sdk/type-aliases/OWRTranches.md) + * [ProviderType](sdk/type-aliases/ProviderType.md) + * [SafeRpcUrl](sdk/type-aliases/SafeRpcUrl.md) + * [SignerType](sdk/type-aliases/SignerType.md) * [SplitRecipient](sdk/type-aliases/SplitRecipient.md) + * [SplitV2Recipient](sdk/type-aliases/SplitV2Recipient.md) * [TotalSplitPayload](sdk/type-aliases/TotalSplitPayload.md) +* [Variables](sdk/variables/README.md) + * [CAPELLA\_FORK\_MAPPING](sdk/variables/CAPELLA_FORK_MAPPING.md) * [Functions](sdk/functions/README.md) + * [clusterConfigOrDefinitionHash](sdk/functions/clusterConfigOrDefinitionHash.md) + * [clusterLockHash](sdk/functions/clusterLockHash.md) * [validateClusterLock](sdk/functions/validateClusterLock.md) + * [verifyDepositData](sdk/functions/verifyDepositData.md) ## API diff --git a/advanced-and-troubleshooting/advanced/consensus-protocols.md b/advanced-and-troubleshooting/advanced/consensus-protocols.md new file mode 100644 index 00000000..363ac250 --- /dev/null +++ b/advanced-and-troubleshooting/advanced/consensus-protocols.md @@ -0,0 +1,88 @@ +--- +description: How Charon's pluggable consensus layer lets a cluster agree on duty data, and how operators can select a preferred protocol. +--- + +# Consensus Protocols + +Before a distributed validator cluster can sign anything, its nodes must agree on exactly what they are signing. Charon's consensus layer is the component responsible for that agreement, and it is designed to support more than one consensus protocol. + +## Overview + +Every duty a distributed validator performs (attesting, proposing a block, and so on) requires the operators in a cluster to independently arrive at the same view of the duty data before they produce their threshold signature shares. Without this step, different nodes could sign different data for the same duty, which would produce an invalid aggregate signature or, worse, contribute to a slashable offense. + +Charon uses a Byzantine fault-tolerant consensus protocol to solve this problem: as long as a threshold of nodes are online and honest, the cluster reaches agreement on duty data even if some nodes are offline, slow, or malicious. + +Historically, Charon has supported a single consensus protocol, QBFT v2.0. Charon's consensus layer now exposes a pluggable interface, which means a cluster can run different consensus protocols as long as they are available and accepted by a majority of the cluster. A cluster can also run multiple consensus protocols at the same time, for example, using one protocol for duty consensus and another for internal coordination. + +## QBFT: The Default Consensus Protocol + +QBFT is an implementation of the Istanbul Byzantine Fault Tolerant (BFT) consensus algorithm, and it has been Charon's consensus protocol since early releases. QBFT v2.0 remains present in every Charon version and cannot be deprecated, because it serves two purposes: + +- It runs the Priority protocol, described below, which the cluster uses to agree on which consensus protocol to use. +- It acts as the fallback protocol whenever no other protocol has been selected by the cluster. + +Because every node is guaranteed to support QBFT v2.0, it is effectively the cluster's default and safety-net protocol. + +## How Protocol Selection Works + +All nodes in a cluster must agree on the same consensus protocol, otherwise consensus fails entirely. Each node has its own list of preferred protocols, in order of precedence, based on its configuration and software version. Charon resolves these individual preferences into a single cluster-wide choice using a dedicated protocol called Priority. + +### The Priority Protocol + +The Priority protocol itself runs on top of QBFT v2.0. It takes each node's ordered list of preferred protocols as input, for example: + +```json +[ + "/charon/consensus/hotstuff/1.0.0", + "/charon/consensus/abft/2.0.0", + "/charon/consensus/abft/1.0.0", + "/charon/consensus/qbft/2.0.0" +] +``` + +The output is the common subset of protocols supported by a majority of nodes, respecting the original order of precedence, for example: + +```json +[ + "/charon/consensus/abft/1.0.0", + "/charon/consensus/qbft/2.0.0" +] +``` + +Because every node always supports QBFT v2.0, it is guaranteed to remain the fallback entry at the bottom of this list, so the Priority protocol can never produce an empty result. The Priority protocol runs once per epoch, during the epoch's last slot. If a different protocol rises to the top of the list, for example, because enough nodes have upgraded, Charon switches the whole cluster over to that protocol starting in the next epoch. + +### Protocol Mismatch and Fallback Behavior + +A node's preferred protocols come from two sources, applied in order: the cluster configuration and, with higher precedence, the node's own CLI flag. Until the Priority protocol reaches agreement on a shared protocol, the cluster falls back to QBFT v2.0 for all duties. This means a cluster is never blocked from performing duties while nodes negotiate or upgrade to a new consensus protocol. + +## Configuring a Preferred Consensus Protocol + +A cluster creator can set a preferred consensus protocol for the whole cluster with the `consensus_protocol` field in the cluster definition file. This field is optional. When it is not set, the cluster definition does not influence protocol selection. + +A node operator can also set a preferred protocol for their own node with the [`--consensus-protocol`](../../learn/charon/charon-cli-reference.md) flag on `charon run`, `charon create cluster`, and `charon create dkg`. This flag is also optional, and it takes precedence over the cluster definition file for that node. + +In both cases, specify the protocol family name only, for example, `abft`, rather than a fully qualified protocol ID. The exact version is determined by the Priority protocol, which always tries to select the latest available version. To list all consensus protocols available in a given Charon build, along with their versions, run `charon version --verbose`. + +{% hint style="info" %} +Setting a preferred protocol expresses a preference, not a guarantee. The cluster only switches to a protocol once a majority of nodes support it, as determined by the Priority protocol. +{% endhint %} + +## Observability + +The following consensus metrics are exposed by Charon: + +- `core_consensus_decided_rounds`. +- `core_consensus_decided_leader_index`. +- `core_consensus_duration_seconds`. +- `core_consensus_error_total`. +- `core_consensus_timeout_total`. + +Each of these metrics carries a `protocol` label, which lets operators distinguish consensus activity between different protocols running on the same cluster. A cluster may currently run at most two consensus protocols at the same time, for example, QBFT v2.0 for the Priority protocol and another protocol for duty consensus, so the `protocol` label may take multiple distinct values. Some protocols may also export their own protocol-specific metrics, prefixed with the protocol's name. + +For debugging, Charon exposes a `/debug/consensus` HTTP endpoint on the [debug address](../../learn/charon/charon-cli-reference.md), which returns a `consensus_messages.pb.gz` file containing the most recent consensus messages. Each message is tagged with the protocol it belongs to, which is useful when a cluster is running more than one protocol at once. + +## Related Topics + +- [Introduction to Charon](../../learn/charon/intro.md). +- [CLI Reference](../../learn/charon/charon-cli-reference.md). +- [Charon Networking](../../learn/charon/charon-networking.md). diff --git a/advanced-and-troubleshooting/advanced/feature-flags.md b/advanced-and-troubleshooting/advanced/feature-flags.md new file mode 100644 index 00000000..7f842ea1 --- /dev/null +++ b/advanced-and-troubleshooting/advanced/feature-flags.md @@ -0,0 +1,75 @@ +--- +description: How Charon's feature flags gate alpha and beta functionality, and how operators can enable, disable, or observe them. +--- + +# Charon Feature Flags + +Charon ships in-development functionality behind feature flags before it becomes the default behavior. This lets the Obol team roll out new consensus timers, client integrations, and other changes gradually, and lets operators opt in to (or out of) specific behavior on their own nodes. + +## Maturity Statuses + +Every feature flag has a maturity status: + +- **Alpha** - for internal devnet testing. Behavior may change or be removed without notice. +- **Beta** - for internal and external testnet testing. More stable than alpha, but not yet proven in production. +- **Stable** - ready for production. Stable features are enabled by default. + +Charon only enables features at or above a configured minimum status. By default, that minimum is `stable`, so only stable features run unless an operator explicitly changes the configuration. + +{% hint style="warning" %} +Enabling alpha or beta features on a mainnet cluster is done at the operator's own risk — these features are still being validated and may change or be withdrawn in a future release. Because Charon nodes reach agreement through consensus, features that affect consensus-relevant behavior (for example, round timers or attestation data fetching) generally need to be enabled consistently across all nodes in a cluster; mismatched feature sets between operators can cause the affected nodes to diverge from the rest of the cluster. +{% endhint %} + +## Enabling and Disabling Feature Flags + +Feature flags are controlled with three `charon run` flags (each with a matching `CHARON_` environment variable, following Charon's standard flag-to-env-var convention): + +| Flag | Environment Variable | Description | +|---|---|---| +| `--feature-set` | `CHARON_FEATURE_SET` | Minimum feature set to enable by default: `alpha`, `beta`, or `stable`. Defaults to `stable`. | +| `--feature-set-enable` | `CHARON_FEATURE_SET_ENABLE` | Comma-separated list of individual features to enable, overriding the default minimum feature set. | +| `--feature-set-disable` | `CHARON_FEATURE_SET_DISABLE` | Comma-separated list of individual features to disable, overriding the default minimum feature set. | + +`--feature-set-enable` and `--feature-set-disable` take precedence over `--feature-set` on a per-feature basis, so an operator can, for example, run with the default `stable` minimum while explicitly enabling a single alpha feature for testing. + +Example `docker-compose.yml` snippet enabling one alpha feature while leaving everything else at the stable default: + +```yaml +services: + charon: + environment: + - CHARON_FEATURE_SET=stable + - CHARON_FEATURE_SET_ENABLE=json_requests +``` + +## Current Feature Flags + +This table reflects Charon `v1.10.0-dev` (commit [`094953a`](https://github.com/ObolNetwork/charon/commit/094953a)). Feature flags are added and removed between releases as functionality graduates to stable and is eventually always-on, so check the [CLI reference](../../learn/charon/charon-cli-reference.md) or run `charon run --help` for the current, authoritative list. + +| Flag Name | Status | Default | Description | +|---|---|---|---| +| `eager_double_linear` | Stable | Enabled | Uses an eager double-linear round timer for consensus rounds. | +| `consensus_participate` | Stable | Enabled | Lets a node participate in an ongoing consensus round while it is still waiting for unsigned duty data from its beacon node. | +| `proposal_timeout` | Stable | Enabled | Uses a longer, 1.5-second first consensus round timeout for proposal duties. | +| `fetch_only_commidx_0` | Stable | Enabled | Queries the beacon node for attestation data only for committee index 0. | +| `linear` | Alpha | Disabled | Uses a linear round timer for consensus rounds; takes precedence over `eager_double_linear` when both are active. | +| `aggsigdb_v2` | Alpha | Disabled | Uses a newer, simpler implementation of the `aggsigdb` component. | +| `json_requests` | Alpha | Disabled | Uses JSON (instead of SSZ) requests when talking to the eth2 beacon node client. | +| `gnosis_block_hotfix` | Alpha | Disabled (auto-enabled on Gnosis/Chiado) | Applies an SSZ fix required by the Gnosis and Chiado networks. Charon automatically enables this feature when the configured network is Gnosis or Chiado, unless an operator explicitly disables it. | +| `sse_reorg_duties` | Alpha | Disabled | Lets the scheduler refresh duties when a chain reorg occurs. | +| `attestation_inclusion` | Alpha | Disabled | Tracks on-chain inclusion of attestations. This was previously always-on behavior, but tracking inclusion after the Electra upgrade adds enough load on the beacon node that it can throttle other duties, so it is now opt-in. | +| `quic` | Alpha | Disabled | Enables the QUIC transport protocol in libp2p networking. | +| `chain_split_halt` | Alpha | Disabled | Compares the locally fetched attestation's target and source against the leader's proposed attestation; if they differ, Charon does not sign the attestation. | +| `fetch_att_on_block` | Alpha | Disabled | Fetches attestation data as soon as a block-processing event is received from the beacon node over SSE, falling back to the standard one-third-of-slot timing if no block event arrives in time. | +| `fetch_att_on_block_with_delay` | Alpha | Disabled | Fetches attestation data with an added 300ms delay. Combined with `fetch_att_on_block`, uses one-third-of-slot-plus-300ms as the fallback timeout; used alone, it uses that same timeout directly. | +| `disable_duties_cache` | Alpha | Disabled | Safety switch to disable the internal duties cache. | +| `mock_alpha` | Alpha | Disabled | Internal/experimental placeholder feature used only for testing the feature flag system itself; it has no functional effect. | + +## Observing Enabled Feature Flags + +Charon exposes the `app_feature_flags` Prometheus metric, a constant gauge labeled with any non-default (custom-enabled) feature flags currently active on a node. See the [Charon Metrics Reference](../../run-a-dv/running/metrics.md) for the full metrics list and label details. Comparing this metric across nodes in a cluster is a quick way to confirm that operators have matching feature sets for consensus-relevant flags. + +## Related Reading + +- [CLI Reference](../../learn/charon/charon-cli-reference.md) - full list of `charon run` flags, including `--feature-set`, `--feature-set-enable`, and `--feature-set-disable`. +- [Consensus Protocols](consensus-protocols.md) - background on Charon's consensus layer, relevant to feature flags like `eager_double_linear`, `linear`, and `consensus_participate`. diff --git a/advanced-and-troubleshooting/advanced/self-relay.md b/advanced-and-troubleshooting/advanced/self-relay.md index aab63377..022884e5 100644 --- a/advanced-and-troubleshooting/advanced/self-relay.md +++ b/advanced-and-troubleshooting/advanced/self-relay.md @@ -31,7 +31,7 @@ Test whether the relay is publicly (or privately) accessible. This should return Ensure the ENR returned by the relay contains the correct public IP and port by decoding it with [ENR viewer](https://enr-viewer.com/). -Configure **ALL** charon nodes in your cluster to use this relay: +Configure **ALL** Charon nodes in your cluster to use this relay: * Either by adding a flag: `--p2p-relays=http://replace.with.public.ip.or.hostname:3640/enr` * Or by setting the environment variable: `CHARON_P2P_RELAYS=http://replace.with.public.ip.or.hostname:3640/enr` diff --git a/advanced-and-troubleshooting/security/risks.md b/advanced-and-troubleshooting/security/risks.md index 24c59c91..4a9ada48 100644 --- a/advanced-and-troubleshooting/security/risks.md +++ b/advanced-and-troubleshooting/security/risks.md @@ -9,7 +9,7 @@ description: Outlining potential centralization risks and their mitigations **Mitigation**: Self-host a relay. -One of the risks associated with Obol hosting the [LibP2P relays](../../learn/charon/networking.mdx) infrastructure allowing peer discovery is that if Obol-hosted relays go down, peers won't be able to discover each other and perform the DKG or reconnect after a restart. To mitigate this risk, external organizations and node operators can consider self-hosting a relay. This way, if Obol's relays go down, the clusters can still operate through other relays in the network. Ensure that all nodes in the cluster use the same relays, or they will not be able to find each other if they are connected to different relays. +One of the risks associated with Obol hosting the [LibP2P relays](../../learn/charon/charon-networking.md) infrastructure allowing peer discovery is that if Obol-hosted relays go down, peers won't be able to discover each other and perform the DKG or reconnect after a restart. To mitigate this risk, external organizations and node operators can consider self-hosting a relay. This way, if Obol's relays go down, the clusters can still operate through other relays in the network. Ensure that all nodes in the cluster use the same relays, or they will not be able to find each other if they are connected to different relays. The following non-Obol entities run relays that you can consider adding to your cluster (you can have more than one per cluster, see the `--p2p-relays` flag of [`charon run`](../../learn/charon/charon-cli-reference.md#the-run-command)): diff --git a/advanced-and-troubleshooting/troubleshooting/client_configurations.md b/advanced-and-troubleshooting/troubleshooting/client_configurations.md index c0e02320..73e35b84 100644 --- a/advanced-and-troubleshooting/troubleshooting/client_configurations.md +++ b/advanced-and-troubleshooting/troubleshooting/client_configurations.md @@ -18,7 +18,7 @@ Nethermind should be configured to not include blobs in locally-built blocks whi ### Consensus Client -Nothing specific for distributed validators is required. If you are configuring MEV-boost, consult the settings you need [here](../../adv/advanced/quickstart-builder-api.mdx#consensus-clients). +Nothing specific for distributed validators is required. If you are configuring MEV-boost, consult the settings you need [here](../advanced/enable-mev.md#consensus-clients). ### Validator Client @@ -32,7 +32,7 @@ Required flags: ### Consensus Client -Nothing specific for distributed validators is required. If you are configuring MEV-boost, consult the settings you need [here](../../adv/advanced/quickstart-builder-api.mdx#consensus-clients). +Nothing specific for distributed validators is required. If you are configuring MEV-boost, consult the settings you need [here](../advanced/enable-mev.md#consensus-clients). ### Validator Client @@ -56,7 +56,7 @@ Required flags: ### Consensus Client -Nothing specific for distributed validators is required. If you are configuring MEV-boost, consult the settings you need [here](../../adv/advanced/quickstart-builder-api.mdx#consensus-clients). +Nothing specific for distributed validators is required. If you are configuring MEV-boost, consult the settings you need [here](../advanced/enable-mev.md#consensus-clients). ### Validator Client diff --git a/advanced-and-troubleshooting/troubleshooting/duty-failure-reasons.md b/advanced-and-troubleshooting/troubleshooting/duty-failure-reasons.md new file mode 100644 index 00000000..6ccf93c7 --- /dev/null +++ b/advanced-and-troubleshooting/troubleshooting/duty-failure-reasons.md @@ -0,0 +1,213 @@ +--- +sidebar_position: 5 +description: Reference for every duty failure reason instrumented by Charon's tracker component, what each one means, and what to do about it. +--- + +# Duty Failure Reasons + +When a Charon node fails to complete a validator duty, the tracker component records a reason for the failure. Operators can see these reasons in their node's logs, and they are also exposed via the `core_tracker_failed_duty_reasons_total` Prometheus counter (see the [Charon Metrics Reference](../../run-a-dv/running/metrics.md)), so they can be graphed and alerted on in Grafana. + +This page explains each failure reason and what it means for your cluster. For guidance on troubleshooting specific error messages, see [Errors & Resolutions](errors.md). + +## Beacon Node Communication Failures + +These reasons indicate a problem communicating with the beacon node, either broadcasting a duty to it or fetching data from it. + +### `broadcast_bn_error` + +**Summary**: Failed to broadcast the duty to the beacon node. + +The beacon node returned an error while Charon was submitting the duty's aggregated signature to it. + +### `fetch_bn_error` + +**Summary**: Couldn't fetch duty data from the beacon node. + +The duty failed in the fetcher step because Charon couldn't fetch the required data from the beacon node API. This indicates a problem with the upstream beacon node. + +### `not_included_onchain` + +**Summary**: Duty not included on-chain. + +Charon broadcast the duty successfully, but it wasn't included in the beacon chain. This is expected for up to 20% of attestations, but it may also indicate problematic Charon broadcast delays or beacon node network problems. + +## Consensus and Peer Signature Failures + +These reasons relate to Charon's peer-to-peer consensus and partial signature exchange, and generally point to problems with peers, the p2p network, or the local validator client. + +### `no_consensus` + +**Summary**: Consensus algorithm didn't complete. + +The duty failed in the consensus step. This could indicate that insufficient honest peers participated in consensus, or that there are p2p network connection problems. + +### `no_peer_signatures` + +**Summary**: No partial signatures received from peers. + +No partial signature for the duty was received from any peer. This indicates that all peers are offline, or that there are p2p network connection problems. + +### `insufficient_peer_signatures` + +**Summary**: Insufficient partial signatures received, minimum required threshold not reached. + +Insufficient partial signatures for the duty were received from peers. This indicates problems with peers or p2p network connection problems. + +### `no_local_vc_signature` + +**Summary**: Signed duty not submitted by the local validator client. + +The partial signature was never submitted by the local validator client. This could indicate that the local validator client is offline, has connection problems with Charon, or has some other problem. Check the validator client logs for more details. + +### `par_sig_db_inconsistent_sync` + +**Summary**: Known limitation: inconsistent sync committee signatures received. + +The partial signed data for the sync committee duty was inconsistent. This is a known limitation in this version of Charon. + +## Prerequisite Duty Failures + +Several duties depend on a prerequisite duty completing first, such as an aggregation or block proposal duty depending on beacon committee selections or RANDAO reveals being aggregated across the cluster. These reasons indicate that the fetcher step couldn't proceed because the prerequisite duty failed. + +### `failed_aggregator_selection` + +**Summary**: Couldn't aggregate attestation due to failed prepare aggregator duty. + +The attestation aggregation duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated beacon committee selections. This indicates the associated prepare aggregation duty failed. + +### `insufficient_aggregator_selections` + +**Summary**: Couldn't aggregate attestation due to insufficient partial beacon committee selections. + +The attestation aggregation duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated beacon committee selections. The associated prepare aggregation duty failed due to insufficient partial beacon committee selections submitted by the cluster's validator clients. + +### `no_aggregator_selections` + +**Summary**: Couldn't aggregate attestation due to no partial beacon committee selections received from peers. + +The attestation aggregation duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated beacon committee selections. The associated prepare aggregation duty failed because no partial beacon committee selections were received from peers. + +### `zero_aggregator_prepares` + +**Summary**: Couldn't aggregate attestation due to zero partial beacon committee selections. + +The attestation aggregation duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated beacon committee selections. The associated prepare aggregation duty failed because no partial beacon committee selections were submitted by the cluster's validator clients. + +### `missing_aggregator_attestation` + +**Summary**: Couldn't aggregate attestation due to failed attester duty. + +The attestation aggregation duty failed in the fetcher step because it couldn't fetch the prerequisite attestation data. This indicates the associated attestation duty failed to obtain a cluster agreed-upon value. + +### `failed_proposer_randao` + +**Summary**: Couldn't propose block due to failed RANDAO duty. + +The block proposer duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated RANDAO. This indicates the associated RANDAO duty failed. + +### `proposer_insufficient_randaos` + +**Summary**: Couldn't propose block due to insufficient partial RANDAO signatures. + +The block proposer duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated RANDAO. The associated RANDAO duty failed due to insufficient partial RANDAO signatures submitted by the cluster's validator clients. + +### `proposer_no_external_randaos` + +**Summary**: Couldn't propose block due to no partial RANDAO signatures received from peers. + +The block proposer duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated RANDAO. The associated RANDAO duty failed because no partial RANDAO signatures were received from peers. + +### `proposer_zero_randaos` + +**Summary**: Couldn't propose block due to zero partial RANDAO signatures. + +The block proposer duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated RANDAO. The associated RANDAO duty failed because no partial RANDAO signatures were submitted by the cluster's validator clients. + +### `sync_contribution_failed_prepare` + +**Summary**: Couldn't fetch sync contribution due to failed prepare sync contribution duty. + +The sync contribution duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated sync contribution selections. This indicates the associated prepare sync contribution duty failed. + +### `sync_contribution_few_prepares` + +**Summary**: Couldn't fetch sync contribution due to insufficient partial sync contribution selections. + +The sync contribution duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated sync contribution selections. The associated prepare sync contribution duty failed due to insufficient partial sync contribution selections submitted by the cluster's validator clients. + +### `sync_contribution_no_external_prepares` + +**Summary**: Couldn't fetch sync contribution due to no partial sync contribution selections received from peers. + +The sync contribution duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated sync contribution selections. The associated prepare sync contribution duty failed because no partial sync contribution selections were received from peers. + +### `sync_contribution_zero_prepares` + +**Summary**: Couldn't fetch sync contribution due to zero partial sync contribution selections. + +The sync contribution duty failed in the fetcher step because it couldn't fetch the prerequisite aggregated sync contribution selections. The associated prepare sync contribution duty failed because no partial sync contribution selections were submitted by the cluster's validator clients. + +### `sync_contribution_no_sync_msg` + +**Summary**: Couldn't fetch sync contribution due to failed sync message duty. + +The sync contribution duty failed in the fetcher step because it couldn't fetch the prerequisite sync message. This indicates the associated sync message duty failed to obtain a cluster agreed-upon value. + +## Internal Bugs + +These reasons indicate an unexpected internal error in Charon rather than an operator-actionable problem. If you encounter one of these reasons, capture your node's debug logs and open an [issue](https://github.com/ObolNetwork/charon/issues) with the Obol team. + +### `bug_aggregation_error` + +**Summary**: Bug: failed to store aggregated signature in the aggregate signature database. + +This indicates a bug in the aggregate signature database, as it is unexpected. + +### `bug_duty_db_error` + +**Summary**: Bug: failed to store duty data in DutyDB. + +This indicates a bug in the DutyDB database, as it is unexpected. + +### `bug_fetch_error` + +**Summary**: Bug: couldn't fetch due to unexpected error. + +The duty failed in the fetcher step with an unexpected error. This indicates a problem in Charon, as it is unexpected. + +### `bug_par_sig_db_external` + +**Summary**: Bug: failed to store external partial signatures in the partial signature database. + +This indicates a bug in the partial signature database, as it is unexpected. + +### `bug_par_sig_db_inconsistent` + +**Summary**: Bug: inconsistent partial signatures received. + +The partial signed data for the duty was inconsistent. For non-sync-committee duties, this indicates a bug in Charon, as it is unexpected. + +### `bug_par_sig_db_internal` + +**Summary**: Bug: partial signature database didn't trigger partial signature exchange; this is unexpected. + +This indicates a bug in the partial signature database, as it is unexpected. Note that this may happen due to an expiry race. + +### `bug_sig_agg` + +**Summary**: Bug: threshold aggregation of partial signatures failed due to inconsistent signed data. + +BLS threshold aggregation of sufficient partial signatures failed, indicating inconsistent signed data. This indicates a bug in Charon, as it is unexpected. + +## Unknown + +### `unknown` + +**Summary**: Unknown error. + +An unknown error occurred. + +## Next Steps + +- For troubleshooting specific Charon, beacon node, and validator client error messages, see [Errors & Resolutions](errors.md). +- To validate the health of your cluster's beacon node, validator client, and peer connectivity, run the [test commands](test_command.md). diff --git a/advanced-and-troubleshooting/troubleshooting/errors.md b/advanced-and-troubleshooting/troubleshooting/errors.md index affdd358..bdd3249b 100644 --- a/advanced-and-troubleshooting/troubleshooting/errors.md +++ b/advanced-and-troubleshooting/troubleshooting/errors.md @@ -18,6 +18,8 @@ You can check your logs using docker compose logs ``` +If your logs show a failed duty with a specific reason code, see [Duty Failure Reasons](duty-failure-reasons.md) for an explanation of every reason Charon's tracker component can report. + ## ENRs & Keys ### How do I get my ENR if I want to generate it again? diff --git a/advanced-and-troubleshooting/troubleshooting/test_command.md b/advanced-and-troubleshooting/troubleshooting/test_command.md index 34b6f933..1159d2d2 100644 --- a/advanced-and-troubleshooting/troubleshooting/test_command.md +++ b/advanced-and-troubleshooting/troubleshooting/test_command.md @@ -81,7 +81,7 @@ charon alpha test \ * Peer might be too far away (geographically) from you. * If the connection to the peer is indirect, the route is from your node, to the relay, to the peer. Meaning you are measuring the travel time from you to the relay, and from the relay to the peer: (your node -> relay -> peer). This means, even if your peer's node is right next to yours, if the connection is being transmitted through a relay far away, the latency between your nodes might be too high to be effective. * Your general network latency to the public internet might be high. Verify with the [`charon test infra`](../../run-a-dv/prepare/test-a-cluster.md#test-machine-and-network-performance) tests. -* If the connection to the peer is indirect, there is a potential that the relay might be overloaded or under-resourced, consider adding [alternative relays](../security/risks.md#risk-obol-hosting-the-relay-infrastructure), or preferably [opening charon's p2p port](../../learn/charon/networking.mdx#libp2p-relays-and-peer-discovery) to the internet to establish direct peer to peer connections. +* If the connection to the peer is indirect, there is a potential that the relay might be overloaded or under-resourced, consider adding [alternative relays](../security/risks.md#risk-obol-hosting-the-relay-infrastructure), or preferably [opening charon's p2p port](../../learn/charon/charon-networking.md#libp2p-relays-and-peer-discovery) to the internet to establish direct peer to peer connections. #### PingLoad @@ -156,7 +156,7 @@ Same causes as PingMeasure test apply here and additionally: #### PingMeasure -* Validator client might be too far away (geographically) from the charon client. Generally a low latency between a validator client and its charon client is important for timely signing. +* Validator client might be too far away (geographically) from the Charon client. Generally a low latency between a validator client and its Charon client is important for timely signing. #### PingLoad diff --git a/learn/charon/charon-cli-reference.md b/learn/charon/charon-cli-reference.md index 44c97857..4bc9d38d 100644 --- a/learn/charon/charon-cli-reference.md +++ b/learn/charon/charon-cli-reference.md @@ -699,7 +699,7 @@ Usage: Flags: --fee-recipient string [REQUIRED] New fee recipient address to be applied to all specified validators. - --gas-limit uint Optional gas limit override for builder registrations. If not set, the most recent gas limit from the cluster lock, overrides file or remote API is used. + --gas-limit uint Optional gas limit override for builder registrations. If not set, the existing gas limit from the cluster lock or overrides file is used. -h, --help Help for sign --lock-file string Path to the cluster lock file defining the distributed validator cluster. (default ".charon/cluster-lock.json") --overrides-file string Path to the builder registrations overrides file. (default ".charon/builder_registrations_overrides.json") @@ -735,11 +735,11 @@ Flags: ### List current fee recipient details -The `charon feerecipient list` command displays the most recent builder registration for each validator, selecting the entry with the highest timestamp from the cluster lock file, the overrides file, or the remote API. +The `charon feerecipient list` command displays the most recent builder registration for each validator, selecting the entry with the highest timestamp from either the cluster lock file or the overrides file. ```markdown charon feerecipient list --help -Displays the most recent builder registration for each validator, selecting the entry with the highest timestamp from the cluster lock file, the overrides file, or the remote API. +Displays the most recent builder registration for each validator, selecting the entry with the highest timestamp from either the cluster lock file or the overrides file. Usage: charon feerecipient list [flags] @@ -748,8 +748,6 @@ Flags: -h, --help Help for list --lock-file string Path to the cluster lock file defining the distributed validator cluster. (default ".charon/cluster-lock.json") --overrides-file string Path to the builder registrations overrides file. (default ".charon/builder_registrations_overrides.json") - --publish-address string The URL of the remote API. (default "https://api.obol.tech/v1") - --publish-timeout duration Timeout for accessing the remote API. (default 5m0s) --validator-public-keys strings Optional comma-separated list of validator public keys to list builder registrations for. ``` diff --git a/learn/charon/intro.md b/learn/charon/intro.md index 031d1932..5d1ce57a 100644 --- a/learn/charon/intro.md +++ b/learn/charon/intro.md @@ -32,7 +32,7 @@ A DV cluster consists of multiple operators each provided with one of the M-of-N `Fetcher` fetches the unsigned duty data from the beacon node upon receiving an event from `Scheduler`. For attestations, this is the unsigned attestation, for block proposals, this is the unsigned block. -The `Consensus` component listens to events from Fetcher and starts a [QBFT](https://docs.goquorum.consensys.net/configure-and-manage/configure/consensus-protocols/qbft/) consensus game with the other Charon nodes in the cluster for that specific duty and slot. When consensus is reached, the resulting unsigned duty data is stored in the `DutyDB`. +The `Consensus` component listens to events from Fetcher and starts a [QBFT](https://docs.besu-eth.org/private-networks/how-to/configure/consensus/qbft) consensus game with the other Charon nodes in the cluster for that specific duty and slot. When consensus is reached, the resulting unsigned duty data is stored in the `DutyDB`. ### **Wait** for the VC to sign diff --git a/learn/further-reading/resources.md b/learn/further-reading/resources.md index 13647209..86fd5bf5 100644 --- a/learn/further-reading/resources.md +++ b/learn/further-reading/resources.md @@ -30,7 +30,7 @@ The following is a curated list of the best internal and external resources for * [Run a DV alone](../../run-a-dv/start/create-a-dv-alone.md) * [Run a DV as a group](../../run-a-dv/start/create-a-dv-with-a-group.md) -* [Run a DV using the SDK](../../adv/advanced/quickstart-sdk.mdx) +* [Run a DV using the SDK](../../advanced-and-troubleshooting/advanced/create-a-dv-using-the-sdk.md) ## Security and Best Practices @@ -44,7 +44,7 @@ The following is a curated list of the best internal and external resources for * A [review](../../advanced-and-troubleshooting/security/ev-assessment.md) of Obol Labs development processes by Ethereal Ventures * A [security assessment](https://github.com/ObolNetwork/obol-security/blob/f9d7b0ad0bb8897f74ccb34cd4bd83012ad1d2b5/audits/Sigma_Prime_Obol_Network_Charon_Security_Assessment_Report_v2_1.pdf) of Charon by [Sigma Prime](https://sigmaprime.io/). -* A [solidity audit](../../adv/security/smart_contract_audit.mdx) of the Obol Splits contracts by [Zach Obront](https://zachobront.com/). +* A [solidity audit](../../advanced-and-troubleshooting/security/smart-contract-audit.md) of the Obol Splits contracts by [Zach Obront](https://zachobront.com/). * [Charon Threat model](../../advanced-and-troubleshooting/security/threat_model.md) * [QuantStamp Charon audit Q4 2023](https://obol.tech/charon_quantstamp_assessment.pdf) * A [security assessment of Charon's editability features](https://github.com/ObolNetwork/charon/blob/main/docs/audit/2026%20-%20Charon%20V2%20Audit%20-%20TrailOfBits.pdf) by [Trail of Bits](https://www.trailofbits.com/). diff --git a/learn/intro/key-concepts.md b/learn/intro/key-concepts.md index 977973d2..be9ba0fe 100644 --- a/learn/intro/key-concepts.md +++ b/learn/intro/key-concepts.md @@ -42,7 +42,7 @@ A consensus client's duty is to run the proof-of-stake consensus layer of Ethere Examples of Consensus clients include: -* [Prysm](https://docs.prylabs.network/docs/how-prysm-works/beacon-node) +* [Prysm](https://prysm.offchainlabs.com/docs/learn/dev-concepts/beacon-node/) * [Teku](https://docs.teku.consensys.net/en/stable/) * [Lighthouse](https://lighthouse-book.sigmaprime.io/api-bn.html) * [Nimbus](https://nimbus.guide/) @@ -67,7 +67,7 @@ A validator client is a piece of code that operates one or more Ethereum validat Examples of validator clients include: -* [Prysm](https://docs.prylabs.network/docs/how-prysm-works/prysm-validator-client/) +* [Prysm](https://prysm.offchainlabs.com/docs/learn/dev-concepts/prysm-validator-client/) * [Lodestar](https://github.com/ChainSafe/lodestar) * [Teku](https://docs.teku.consensys.net/en/stable/) * [Lighthouse](https://lighthouse-book.sigmaprime.io/api-vc.html) diff --git a/learn/intro/obol-collective.md b/learn/intro/obol-collective.md index fbb5ed73..8f8ccc9c 100644 --- a/learn/intro/obol-collective.md +++ b/learn/intro/obol-collective.md @@ -22,7 +22,7 @@ DV Labs (originally “Obol Labs”) is one of the core research and software de ### The Obol Product Suite -The [Obol Product Suite](https://obol.org/product-suite) empowers any node operator to run fault-tolerant, slashing-resistant distributed validators. Choose from [a suite of tools](../further-reading/resources.md) to get distributed validators running on any type of hardware, with any combination of software clients. +The [Obol Product Suite](https://obol.org/product-suite-dvs) empowers any node operator to run fault-tolerant, slashing-resistant distributed validators. Choose from [a suite of tools](../further-reading/resources.md) to get distributed validators running on any type of hardware, with any combination of software clients. * Foundation: [Charon](../charon/intro.md), a middleware client that enables validators to run in a fault-tolerant, distributed manner; * Configuration: The [Distributed Validator Launchpad](launchpad.md), a user interface for configuring Distributed Validators. The Obol [SDK](../../sdk/index.md) & [API](../../api/what-is-this-api.md), allowing Distributed Validator clusters to be configured and run at scale, for example within staking protocols. diff --git a/obol-stack/README.md b/obol-stack/README.md index 7127f2f5..8f7db1ba 100644 --- a/obol-stack/README.md +++ b/obol-stack/README.md @@ -4,7 +4,7 @@ description: A framework for AI agents to run decentralized infrastructure local # Introduction to the Obol Stack -The Obol Stack is a local-first agent harness: a Kubernetes cluster on your laptop, a default AI agent ([Hermes](https://github.com/NousResearch/hermes)) with its own Ethereum wallet, dynamically-deployable blockchain networks, a Cloudflare tunnel for public exposure, and an [x402](https://www.x402.org/) payment gateway so agents can charge for what they serve. +The Obol Stack is a local-first agent harness: a Kubernetes cluster on your laptop, a default AI agent ([Hermes](https://github.com/NousResearch/hermes-agent)) with its own Ethereum wallet, dynamically-deployable blockchain networks, a Cloudflare tunnel for public exposure, and an [x402](https://www.x402.org/) payment gateway so agents can charge for what they serve. The thesis is simple: **agents should be able to run real infrastructure, build something valuable on top of it, and sell access to it for micropayments — without asking permission and without standing up cloud accounts.** diff --git a/obol-stack/build-a-profitable-stack.md b/obol-stack/build-a-profitable-stack.md index aea65166..cc5882d6 100644 --- a/obol-stack/build-a-profitable-stack.md +++ b/obol-stack/build-a-profitable-stack.md @@ -40,7 +40,7 @@ Pick `--since` based on what your agent needs to answer: - **Anything from the last year** → `--since=365d` (~600 GB). - **Recent execution-layer history for a quick indexer** → `--since=prague` (~0.4 TB). -See [Installing Networks](installing-networks.md#archive-nodes-and-bounded-history-since) for the full `--since` reference. +See [Installing Networks](installing-networks.md#archive-nodes-and-bounded-history---since) for the full `--since` reference. While the node syncs (hours to days depending on scope), keep moving. The other steps don't require the archive to be complete. diff --git a/obol-stack/faq.md b/obol-stack/faq.md index 84f5d52b..e9b325dd 100644 --- a/obol-stack/faq.md +++ b/obol-stack/faq.md @@ -80,7 +80,7 @@ echo "127.0.0.1 obol.stack" | sudo tee -a /etc/hosts ### What's the default agent? -[Hermes](https://github.com/NousResearch/hermes) is the default Obol Agent runtime as of v0.9.0. `obol stack up` provisions a default Hermes instance in the `hermes-obol-agent` namespace, with its own Ethereum signing wallet and a built-in skill library. +[Hermes](https://github.com/NousResearch/hermes-agent) is the default Obol Agent runtime as of v0.9.0. `obol stack up` provisions a default Hermes instance in the `hermes-obol-agent` namespace, with its own Ethereum signing wallet and a built-in skill library. OpenClaw remains supported as an optional alternate runtime — `obol agent new --runtime openclaw` if you want one. diff --git a/obol-stack/quickstart.md b/obol-stack/quickstart.md index b7bfe4de..8dbb38c3 100644 --- a/obol-stack/quickstart.md +++ b/obol-stack/quickstart.md @@ -101,7 +101,7 @@ obol hermes --help # full Hermes CLI surface ``` {% hint style="success" %} -Want the agent to message you on Telegram, Discord, or Slack? Run `obol hermes setup` and follow the prompts to wire up a chat-app integration. Hermes will then notify you when long-running work finishes. The full Telegram bot flow (which involves talking to `@BotFather` and `@userinfobot` first) is covered in [Build a profitable Obol Stack](build-a-profitable-stack.md#step-6-tell-your-agent-to-ping-you-on-telegram). +Want the agent to message you on Telegram, Discord, or Slack? Run `obol hermes setup` and follow the prompts to wire up a chat-app integration. Hermes will then notify you when long-running work finishes. The full Telegram bot flow (which involves talking to `@BotFather` and `@userinfobot` first) is covered in [Build a profitable Obol Stack](build-a-profitable-stack.md#step-7-tell-your-agent-to-ping-you-on-telegram). {% endhint %} The agent has its own Ethereum wallet — back it up before you put anything on it: diff --git a/run-a-dv/editing/recreate-private-keys.md b/run-a-dv/editing/recreate-private-keys.md index 619468ae..60ee4405 100644 --- a/run-a-dv/editing/recreate-private-keys.md +++ b/run-a-dv/editing/recreate-private-keys.md @@ -14,7 +14,7 @@ You might need to recreate private key shares in several scenarios: - **Security concerns**: If you suspect that private key shares may have been compromised - **Key rotation**: As part of regular security practices to rotate cryptographic material - **Recovery**: After a security incident where you want to refresh all key material -- **Compliance**: Meeting organisational policies that require periodic key rotation +- **Compliance**: Meeting organizational policies that require periodic key rotation {% hint style="info" %} This operation maintains the same validator public keys, so your validators remain registered on the beacon chain without any changes. Only the underlying private key shares held by operators are refreshed. diff --git a/run-a-dv/integrations/lido-csm.md b/run-a-dv/integrations/lido-csm.md index e7a0f242..2b782dbf 100644 --- a/run-a-dv/integrations/lido-csm.md +++ b/run-a-dv/integrations/lido-csm.md @@ -137,7 +137,7 @@ Lastly, share the cluster invite link with the other cluster members. ### Step 4: Distributed Key Generation (DKG) -All squad members need to open the cluster invitation link, connect their wallet, accept all necessary advisories, and to verify the cluster configuration is correct with a signature. Each squad member will also need to upload and sign an ENR to represent their charon client, so see [steps 1](lido-csm.md#step-1-clone-the-repo) and [2](lido-csm.md#step-2-create-enr-and-backup-your-private-key) above. +All squad members need to open the cluster invitation link, connect their wallet, accept all necessary advisories, and to verify the cluster configuration is correct with a signature. Each squad member will also need to upload and sign an ENR to represent their Charon client, so see [steps 1](lido-csm.md#step-1-clone-the-repo) and [2](lido-csm.md#step-2-create-enr-and-backup-your-private-key) above.
Screenshot: All squad members need to open the cluster invitation link, connect their wallet, accept all necessary advisories, and to verify the cluster configuration is correct with a…
diff --git a/run-a-dv/integrations/lido-v3-stvault-for-capital-allocators.md b/run-a-dv/integrations/lido-v3-stvault-for-capital-allocators.md index 0c8108fb..564a526c 100644 --- a/run-a-dv/integrations/lido-v3-stvault-for-capital-allocators.md +++ b/run-a-dv/integrations/lido-v3-stvault-for-capital-allocators.md @@ -311,7 +311,7 @@ Build a **DV cluster** that can qualify for Lido's **DVT category** and DVT tier - best practices on monitoring and upgrades. - Prepares technical input for the **Identified Node Operator** submission (DVT category). -For detailed steps on the identification process, see the [Node Operator Guide](lido-v3-stvault-for-node-operator.md#51-dvt-cluster-identification-process). +For detailed steps on the identification process, see the [Node Operator Guide](lido-v3-stvault-for-node-operator.md#51-dv-cluster-identification-process). --- diff --git a/run-a-dv/integrations/lido-v3-stvault-for-node-operator.md b/run-a-dv/integrations/lido-v3-stvault-for-node-operator.md index 39b2cc1e..d1e5ad21 100644 --- a/run-a-dv/integrations/lido-v3-stvault-for-node-operator.md +++ b/run-a-dv/integrations/lido-v3-stvault-for-node-operator.md @@ -235,7 +235,7 @@ After creating and publishing your Obol DV cluster, you must go through **Lido's - **Unidentified clusters** default to the **Default tier** with only a **50% Reserve Ratio** and limited stETH minting capacity. - **Identified DV clusters** can qualify for **DV tiers** with **Reserve Ratios as low as 2-4%** and significantly higher stETH minting limits. -For detailed tier breakdowns and capital efficiency benefits, see the [Capital Allocator Guide](lido-v3-stvault-for-capital-allocators.md#dvt-tiers). +For detailed tier breakdowns and capital efficiency benefits, see the [Capital Allocator Guide](lido-v3-stvault-for-capital-allocators.md#stvault-terminology). **The identification process:** diff --git a/run-a-dv/prepare/deployment-best-practices.md b/run-a-dv/prepare/deployment-best-practices.md index 7975b3e7..1c7f05d2 100644 --- a/run-a-dv/prepare/deployment-best-practices.md +++ b/run-a-dv/prepare/deployment-best-practices.md @@ -77,7 +77,7 @@ Cluster sizes that allow for Byzantine Fault Tolerance are recommended as they a ## MEV-Boost Relays -MEV relays are configured at the Consensus Layer or MEV-boost client level. Refer to our [guide](../../adv/advanced/quickstart-builder-api.mdx) to ensure all necessary configuration has been applied to your clients. As with all validators, low latency during proposal opportunities is extremely important. By default, MEV-Boost waits for all configured relays to return a bid, or will timeout if any have not returned a bid within 950ms. This default timeout is generally too slow for a distributed cluster (think of this time as additive to the time it takes the cluster to come to consensus, both of which need to happen within a 2 second window for optimal proposal broadcasting). It is likely better to only list relays that are located geographically near your node, so that once all relays respond (e.g. in < 50ms) your cluster will move forward with the proposal. +MEV relays are configured at the Consensus Layer or MEV-boost client level. Refer to our [guide](../../advanced-and-troubleshooting/advanced/enable-mev.md) to ensure all necessary configuration has been applied to your clients. As with all validators, low latency during proposal opportunities is extremely important. By default, MEV-Boost waits for all configured relays to return a bid, or will timeout if any have not returned a bid within 950ms. This default timeout is generally too slow for a distributed cluster (think of this time as additive to the time it takes the cluster to come to consensus, both of which need to happen within a 2 second window for optimal proposal broadcasting). It is likely better to only list relays that are located geographically near your node, so that once all relays respond (e.g. in < 50ms) your cluster will move forward with the proposal. Use Charon's [`test mev` command](./test-a-cluster.md#test-mev-relay) to test a number of your preferred relays, and select the two or three relays with the lowest latency to your node(s), you do not need to have the same relays on each node in a cluster. diff --git a/run-a-dv/prepare/test-a-cluster.md b/run-a-dv/prepare/test-a-cluster.md index 860dca29..9e0faacc 100644 --- a/run-a-dv/prepare/test-a-cluster.md +++ b/run-a-dv/prepare/test-a-cluster.md @@ -36,7 +36,7 @@ charon alpha test all \ --peers-enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -59,7 +59,7 @@ charon alpha test all \ --peers-definition-file="./.charon/cluster-definition.json" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -82,7 +82,7 @@ charon alpha test all \ --peers-lock-file="./.charon/cluster-lock.json" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -225,7 +225,7 @@ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/ch --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -250,7 +250,7 @@ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/ch --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -275,7 +275,7 @@ docker run -u $(id -u):$(id -g) --rm -v "$(pwd):/opt/charon/test" obolnetwork/ch --peers-private-key-file="/opt/charon/test/.charon/charon-enr-private-key" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -431,7 +431,7 @@ kubectl exec -it -n dv-pod my-dv-pod-0 -- charon alpha test all \ --peers-enrs="enr:-HW4QMno_MB_ID6GFVxoIQAHHVHZZZjzFctxtX2tm9D95tvaPbHathi8YUP8jh8v2YUAVu2fYWEOB_BT14pt8QgiGg2AgmlkgnY0iXNlY3AyNTZrMaECdpnK83s0dbBwCaEfDIkQ-3nJkkC93STvv6Vmi0bYlzg,enr:-HW4QO2vefLueTBEUGly5hkcpL7NWdMKWx7Nuy9f7z6XZInCbFAc0IZj6bsnmj-Wi4ElS6jNa0Mge5Rkc2WGTVemas2AgmlkgnY0iXNlY3AyNTZrMaECR9SmYQ_1HRgJmNxvh_ER2Sxx78HgKKgKaOkCROYwaDY" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -454,7 +454,7 @@ kubectl exec -it -n dv-pod my-dv-pod-0 -- charon alpha test all \ --peers-definition-file="./.charon/cluster-definition.json" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ @@ -477,7 +477,7 @@ kubectl exec -it -n dv-pod my-dv-pod-0 -- charon alpha test all \ --peers-lock-file="./.charon/cluster-lock.json" \ --beacon-endpoints="https://ethereum-hoodi-beacon-api.publicnode.com" \ --mev-endpoints="\ -https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi-helix.aestus.live,\ +https://0x98f0ef62f00780cf8eb06701a7d22725b9437d4768bb19b363e882ae87129945ec206ec2dc16933f31d983f8225772b6@hoodi.aestus.live,\ https://0x821f2a65afb70e7f2e820a925a9b4c80a159620582c1766b1b09729fec178b11ea22abb3a51f07b288be815a1a2ff516@bloxroute.hoodi.blxrbdn.com,\ https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net,\ https://0xb1559beef7b5ba3127485bbbb090362d9f497ba64e177ee2c8e7db74746306efad687f2cf8574e38d70067d40ef136dc@relay-hoodi.ultrasound.money,\ diff --git a/run-a-dv/running/claim-rewards.md b/run-a-dv/running/claim-rewards.md index 31d564c1..7f896f87 100644 --- a/run-a-dv/running/claim-rewards.md +++ b/run-a-dv/running/claim-rewards.md @@ -55,7 +55,7 @@ Once the funds are distributed, go to the [Operator Dashboard](https://launchpad This section documents the legacy OWR (Optimistic Withdrawal Recipient) flow for older `0x01` validators. For new clusters with `0x02` validators, please use the [New Claim Rewards Process](#new-claim-rewards-process-0x02-validators) above. {% endhint %} -The method for claiming rewards depends on the cluster's withdrawal configuration, whether it's an [**OWR**](../../learn/intro/obol-splits.md#optimistic-withdrawal-recipient) or an [**Exitable Withdrawal Configuration**](../../learn/intro/obol-splits.md#exitable-withdrawal-recipient). The table below outlines the latest details on how and where to claim rewards. +The method for claiming rewards depends on the cluster's withdrawal configuration, whether it's an [**OWR**](../../learn/intro/obol-splits.md#optimistic-withdrawal-recipient) or an [**Exitable Withdrawal Configuration**](../../learn/intro/obol-splits.md#optimistic-withdrawal-recipient). The table below outlines the latest details on how and where to claim rewards. ### Claim Status diff --git a/run-a-dv/running/metrics.md b/run-a-dv/running/metrics.md new file mode 100644 index 00000000..46161202 --- /dev/null +++ b/run-a-dv/running/metrics.md @@ -0,0 +1,149 @@ +--- +description: >- + Reference for the Prometheus metrics Charon exposes, including standard + labels, key metrics to watch, and the full metrics table for dashboards + and alerting. +--- + +# Charon Metrics Reference + +Charon exposes a Prometheus-compatible metrics endpoint on its monitoring API, configured with the `--monitoring-address` flag and defaulting to `127.0.0.1:3620`. Scraping this endpoint with Prometheus lets you build dashboards, health checks, and alerts on the distributed validator's real-time behavior, from peer connectivity to duty completion. + +If you are running the [**CDVN repository**](https://github.com/ObolNetwork/charon-distributed-validator-node), Prometheus and Grafana are already part of the docker compose setup and scrape this endpoint automatically, so no extra configuration is needed to get started. See [Monitoring Your Node](monitoring.md) for how to use the pre-built dashboards and alerts. + +## Standard Labels + +All metrics on this page include the following labels, so they are omitted from the tables below. + +- `cluster_hash`: The cluster lock hash uniquely identifying the cluster. +- `cluster_name`: The cluster lock name. +- `cluster_network`: The cluster network name; Goerli, Mainnet, etc. +- `cluster_peer`: The name of this node in the cluster. It is determined from the operator's Ethereum Node Record (ENR). + +The `cluster_*` labels uniquely identify a specific node's metrics, which is required when storing metrics from multiple nodes or clusters in one Prometheus instance. + +## Key Metrics to Watch + +The full reference table below covers every metric Charon exposes, but a smaller set is especially useful for day-to-day operations. These are a good starting point for dashboards and alerts. + +| Metric | Why It Matters | +|---|---| +| `app_monitoring_readyz` | Reports the node's readiness state: `1` if operational, `2` if the beacon node is down, `3` if the beacon node is syncing, or `4` if quorum peers are not connected (otherwise `0`). Drives the `ClusterInUnknownStatus` and `ClusterInsufficientPeers` alerts in [Monitoring Your Node](monitoring.md). | +| `app_eth2_using_fallback` | Indicates whether the client is using a fallback (`1`) or primary (`0`) beacon node, useful for spotting an unhealthy primary beacon node. | +| `app_beacon_node_peers` | Peer count of the upstream beacon node; low values can precede sync or attestation issues and relate to the `ClusterBeaconNodeZeroPeers` alert. | +| `app_log_error_total` | Total count of logged errors by topic, a quick signal for rising error rates before they cause duty failures. | +| `core_tracker_participation` | Set to `1` if a peer participated successfully for a given duty, or `0` otherwise; useful for spotting an underperforming peer. | +| `core_tracker_failed_duties_total` | Total number of failed duties by type, the underlying signal behind the `ClusterFailureRate` and `ClusterMissedAttestations` alerts. | +| `p2p_ping_latency_secs` | Ping latency in seconds per peer, the basis for the `PeerPingLatency` alert. | +| `p2p_ping_success` | Whether the last ping to a peer was successful (`1`) or not (`0`); can be used as a proxy for connected peers. | + +## Full Metrics Reference + +The table below is generated from the Charon source code and reflects the metrics defined in the codebase. For the latest, canonical version, see [`docs/metrics.md`](https://github.com/ObolNetwork/charon/blob/main/docs/metrics.md) in the Charon repository. + +| Name | Type | Help | Labels | +|---|---|---|---| +| `app_beacon_node_peers` | Gauge | Gauge set to the peer count of the upstream beacon node | | +| `app_beacon_node_sse_block` | Histogram | Block imported into fork choice delay, supplied by beacon node's SSE endpoint. Values between 0s and 4s for Ethereum mainnet are considered safe | `addr` | +| `app_beacon_node_sse_block_gossip` | Histogram | Block reception via gossip delay, supplied by beacon node's SSE endpoint. Values between 0s and 4s for Ethereum mainnet are considered safe | `addr` | +| `app_beacon_node_sse_block_processing_time` | Histogram | Time in seconds between block gossip and head events, indicating block processing time. Lower values indicate better CPU/disk/RAM performance. | `addr` | +| `app_beacon_node_sse_chain_reorg_depth` | Histogram | Chain reorg depth, supplied by beacon node's SSE endpoint | `addr` | +| `app_beacon_node_sse_head_delay` | Histogram | Delay in seconds between slot start and head update, supplied by beacon node's SSE endpoint. Values between 8s and 12s for Ethereum mainnet are considered safe. | `addr` | +| `app_beacon_node_sse_head_slot` | Gauge | Current beacon node head slot, supplied by beacon node's SSE endpoint | `addr` | +| `app_beacon_node_version` | Gauge | Constant gauge with labels set to the version and beacon_id of the upstream beacon node | `version, beacon_id` | +| `app_cache_hits_total` | Counter | Total number of times the cache was used | `endpoint` | +| `app_cache_invalidated_reorg_total` | Counter | Total number of times the cache was invalidated due to a chain reorg | `endpoint` | +| `app_cache_misses_total` | Counter | Total number of times the cache was missed | `endpoint` | +| `app_eth2_errors_total` | Counter | Total number of errors returned by eth2 beacon node requests | `endpoint` | +| `app_eth2_latency_seconds` | Histogram | Latency in seconds for eth2 beacon node requests | `endpoint` | +| `app_eth2_requests_total` | Counter | Total number of requests sent to eth2 beacon node | `endpoint` | +| `app_eth2_using_fallback` | Gauge | Indicates if client is using fallback (1) or primary (0) beacon node | | +| `app_execution_layer_version` | Gauge | Constant gauge with labels set to the version of the upstream execution layer | `version` | +| `app_feature_flags` | Gauge | Constant gauge with custom enabled feature flags | `feature_flags` | +| `app_git_commit` | Gauge | Constant gauge with label set to current git commit hash | `git_hash` | +| `app_health_checks` | Gauge | Application health checks by name and severity. Set to 1 for failing, 0 for ok. | `severity, name, description` | +| `app_health_checks_failed_total` | Counter | Total number of times each health check has been observed failing. Allows querying historical failures via increase(). | `severity, name, description` | +| `app_health_metrics_high_cardinality` | Gauge | Metrics with high cardinality by name. | `name` | +| `app_log_error_total` | Counter | Total count of logged errors by topic | `topic` | +| `app_log_loki_dropped_total` | Counter | Total count of dropped log lines due to full buffer | | +| `app_log_warn_total` | Counter | Total count of logged warnings by topic | `topic` | +| `app_monitoring_readyz` | Gauge | Set to 1 if the node is operational and monitoring api `/readyz` endpoint is returning 200s. Else `/readyz` is returning 500s and this metric is either set to 2 if the beacon node is down, 3 if the beacon node is syncing, or 4 if quorum peers are not connected. | | +| `app_peer_name` | Gauge | Constant gauge with label set to the name of the cluster peer | `peer_name` | +| `app_peerinfo_builder_api_enabled` | Gauge | Set to 1 if builder API is enabled on this peer, else 0 if disabled. | `peer` | +| `app_peerinfo_clock_offset_seconds` | Gauge | Peer clock offset in seconds | `peer` | +| `app_peerinfo_git_commit` | Gauge | Constant gauge with git_hash label set to peer's git commit hash. | `peer, git_hash` | +| `app_peerinfo_index` | Gauge | Constant gauge set to the peer index in the cluster definition | `peer` | +| `app_peerinfo_nickname` | Gauge | Constant gauge with nickname label set to peer's charon nickname. | `peer, peer_nickname` | +| `app_peerinfo_start_time_secs` | Gauge | Constant gauge set to the peer start time of the binary in unix seconds | `peer` | +| `app_peerinfo_version` | Gauge | Constant gauge with version label set to peer's charon version. | `peer, version` | +| `app_peerinfo_version_support` | Gauge | Set to 1 if the peer's version is supported by (compatible with) the current version, else 0 if unsupported. | `peer` | +| `app_start_time_secs` | Gauge | Gauge set to the app start time of the binary in unix seconds | | +| `app_validator_stack_params` | Gauge | Parameters for each component of the validator stack in which this Charon instance is deployed into | `component, cli_parameters` | +| `app_version` | Gauge | Constant gauge with label set to current app version | `version` | +| `cluster_network` | Gauge | Constant gauge with label set to the current network (chain) | `network` | +| `cluster_operators` | Gauge | Number of operators in the cluster lock | | +| `cluster_threshold` | Gauge | Aggregation threshold in the cluster lock | | +| `cluster_validators` | Gauge | Number of validators in the cluster lock | | +| `core_bcast_broadcast_delay_seconds` | Histogram | Duty broadcast delay since the expected duty submission in seconds by type | `duty` | +| `core_bcast_broadcast_total` | Counter | The total count of successfully broadcast duties by type | `duty` | +| `core_consensus_decided_leader_index` | Gauge | Index of the decided leader by protocol and duty | `protocol, duty` | +| `core_consensus_decided_rounds` | Gauge | Number of decided rounds by protocol, duty, and timer | `protocol, duty, timer` | +| `core_consensus_duration_seconds` | Histogram | Duration of the consensus process by protocol, duty, and timer | `protocol, duty, timer` | +| `core_consensus_error_total` | Counter | Total count of consensus errors by protocol | `protocol` | +| `core_consensus_timeout_total` | Counter | Total count of consensus timeouts by protocol, duty, and timer | `protocol, duty, timer` | +| `core_fetcher_proposal_blinded` | Gauge | Whether the fetched proposal was blinded (1) or local (2) | | +| `core_fetcher_proposal_local_mismatch_fee_recipient` | Gauge | Counts the number of times a local proposal has a mismatched fee recipient | | +| `core_parsigdb_exit_total` | Counter | Total number of partially signed voluntary exits per public key | `pubkey` | +| `core_parsigdb_store` | Histogram | Latency of partial signatures received since earliest expected time, per duty, per peer index | `duty, peer_idx` | +| `core_parsigex_set_verification_seconds` | Histogram | Duration to verify all partial signatures in a received set, in seconds | `duty` | +| `core_scheduler_current_epoch` | Gauge | The current epoch | | +| `core_scheduler_current_slot` | Gauge | The current slot | | +| `core_scheduler_duty_total` | Counter | The total count of duties scheduled by type | `duty` | +| `core_scheduler_skipped_slots_total` | Counter | Total number times slots were skipped | | +| `core_scheduler_submit_registration_errors_total` | Counter | The total count of failed submit registration requests | | +| `core_scheduler_submit_registration_total` | Counter | The total number of submit registration requests | | +| `core_scheduler_validator_balance_gwei` | Gauge | Total balance of a validator by public key | `pubkey_full, pubkey` | +| `core_scheduler_validator_status` | Gauge | Gauge with validator pubkey and status as labels, value=1 is current status, value=0 is previous. | `pubkey_full, pubkey, status` | +| `core_scheduler_validators_active` | Gauge | Number of active validators | | +| `core_sigagg_slot_aggregation_seconds` | Histogram | Total duration to aggregate all validators for a duty in a slot, in seconds | `duty` | +| `core_tracker_attestation_expect_total` | Counter | Total number of expected attestations for the slot (counts individual attestations, not duties) | | +| `core_tracker_attestation_success_total` | Counter | Total number of successful attestations for the slot (counts individual attestations, not duties) | | +| `core_tracker_expect_duties_total` | Counter | Total number of expected duties (failed + success) by type | `duty` | +| `core_tracker_failed_duties_total` | Counter | Total number of failed duties by type | `duty` | +| `core_tracker_failed_duty_reasons_total` | Counter | Total number of failed duties by type and reason code | `duty, reason` | +| `core_tracker_inclusion_delay` | Gauge | Cluster's average attestation inclusion delay in slots. Available only when attestation_inclusion feature flag is enabled. | | +| `core_tracker_inclusion_missed_total` | Counter | Total number of broadcast duties never included in any block by type | `duty` | +| `core_tracker_inconsistent_parsigs_total` | Counter | Total number of duties that contained inconsistent partial signed data by duty type | `duty` | +| `core_tracker_participation` | Gauge | Set to 1 if peer participated successfully for the given duty or else 0 | `duty, peer` | +| `core_tracker_participation_expected_total` | Counter | Total number of expected participations (fail + success) by peer and duty type | `duty, peer` | +| `core_tracker_participation_missed_total` | Counter | Total number of missed participations by peer and duty type | `duty, peer` | +| `core_tracker_participation_success_total` | Counter | Total number of successful participations by peer and duty type | `duty, peer` | +| `core_tracker_participation_total` | Counter | Total number of successful participations by peer and duty type | `duty, peer` | +| `core_tracker_success_duties_total` | Counter | Total number of successful duties by type | `duty` | +| `core_tracker_unexpected_events_total` | Counter | Total number of unexpected events by peer | `peer` | +| `core_validatorapi_proxy_request_latency_seconds` | Histogram | The validatorapi proxy request latencies in seconds by path | `path` | +| `core_validatorapi_request_error_total` | Counter | The total number of validatorapi request errors | `endpoint, status_code` | +| `core_validatorapi_request_latency_seconds` | Histogram | The validatorapi request latencies in seconds by endpoint | `endpoint` | +| `core_validatorapi_request_total` | Counter | The total number of requests per content-type and endpoint | `endpoint, content_type` | +| `core_validatorapi_vc_user_agent` | Gauge | Gauge with label set to user agent string of requests made by VC | `user_agent` | +| `p2p_peer_connection_total` | Counter | Total number of libp2p connections per peer. | `peer` | +| `p2p_peer_connection_types` | Gauge | Current number of libp2p connections by peer, type (`direct` or `relay`), and protocol (`tcp`, `quic`). Note that peers may have multiple connections. | `peer, type, protocol` | +| `p2p_peer_network_receive_bytes_total` | Counter | Total number of network bytes received from the peer by protocol and transport. Transport is based on first active connection (accurate in steady state). | `peer, protocol, transport` | +| `p2p_peer_network_sent_bytes_total` | Counter | Total number of network bytes sent to the peer by protocol and transport. Transport is based on first active connection (accurate in steady state). | `peer, protocol, transport` | +| `p2p_peer_streams` | Gauge | Current number of libp2p streams by peer, direction (`inbound` or `outbound` or `unknown`), protocol and transport. | `peer, direction, protocol, transport` | +| `p2p_ping_error_total` | Counter | Total number of ping errors per peer | `peer` | +| `p2p_ping_latency_secs` | Histogram | Ping latencies in seconds per peer | `peer` | +| `p2p_ping_success` | Gauge | Whether the last ping was successful (1) or not (0). Can be used as proxy for connected peers | `peer` | +| `p2p_reachability_status` | Gauge | Current libp2p reachability status of this node as detected by autonat: unknown(0), public(1) or private(2). | | +| `p2p_relay_connection_types` | Gauge | Current number of libp2p connections by relay, type (`direct` or `relay`), and protocol (`tcp`, `quic`). Note that peers may have multiple connections. | `peer, type, protocol` | +| `p2p_relay_connections` | Gauge | Connected relays by name | `peer` | +| `relay_p2p_active_connections` | Gauge | Current number of active connections by peer and cluster | `peer, peer_cluster` | +| `relay_p2p_connection_total` | Counter | Total number of new connections by peer and cluster | `peer, peer_cluster` | +| `relay_p2p_network_receive_bytes_total` | Counter | Total number of network bytes received from the peer and cluster | `peer, peer_cluster` | +| `relay_p2p_network_sent_bytes_total` | Counter | Total number of network bytes sent to the peer and cluster | `peer, peer_cluster` | +| `relay_p2p_ping_latency` | Histogram | Ping latency by peer and cluster | `peer, peer_cluster` | + +## Next Steps + +- See [Monitoring Your Node](monitoring.md) for pre-built Grafana dashboards, alert definitions, and best practices for running your own Prometheus and Grafana server. +- See [Push Metrics and Logs to Obol](../start/obol-monitoring.md) to send these metrics to Obol's hosted monitoring instead of running your own. diff --git a/run-a-dv/running/monitoring.md b/run-a-dv/running/monitoring.md index 628ca0d4..b1a4d4a6 100644 --- a/run-a-dv/running/monitoring.md +++ b/run-a-dv/running/monitoring.md @@ -9,7 +9,7 @@ description: >- This comprehensive guide will assist you in effectively monitoring your Charon clusters and setting up alerts by running your own Prometheus and Grafana server. If you want to use Obol’s [public dashboard](https://grafana.monitoring.gcp.obol.tech/d/d895e47a-3c2d-46b7-9b15-8f31202681af/clusters-aggregate-view?orgId=6) instead of running your servers, refer to [this section](../start/obol-monitoring.md) in Obol docs that teaches you how to push Prometheus metrics to Obol. -To explain quickly, Prometheus generates the metrics and Grafana visualizes them. To learn more about Prometheus and Grafana, visit [here](https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-prometheus/). If you are using [**CDVN repository**](https://github.com/ObolNetwork/charon-distributed-validator-node) or [**CDVC repository**](https://github.com/ObolNetwork/charon-distributed-validator-cluster), then Prometheus and Grafana are part of docker compose file and will be installed when you run `docker compose up`. +To explain quickly, Prometheus generates the metrics and Grafana visualizes them. To learn more about Prometheus and Grafana, visit [here](https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-prometheus/). If you are using [**CDVN repository**](https://github.com/ObolNetwork/charon-distributed-validator-node) or [**CDVC repository**](https://github.com/ObolNetwork/charon-distributed-validator-cluster), then Prometheus and Grafana are part of docker compose file and will be installed when you run `docker compose up`. For a complete list of the metrics Charon exposes, see the [Charon Metrics Reference](metrics.md). The local Grafana server will have a few pre-built dashboards: diff --git a/run-a-dv/start/create-a-dv-with-a-group.md b/run-a-dv/start/create-a-dv-with-a-group.md index d57767bd..f9d46a25 100644 --- a/run-a-dv/start/create-a-dv-with-a-group.md +++ b/run-a-dv/start/create-a-dv-with-a-group.md @@ -426,7 +426,7 @@ It is important to back up all artefacts generated by the DKG ceremony, and your 2. Click on the 'Backup now' button and it will open a new chrome window with a 'file save' option. Select the path where you want to save the Backup tar file.
Screenshot: Click on the 'Backup now' button and it will open a new chrome window with a 'file save' option. Select the path where you want to save the Backup tar file.
-3. Double click to extract the tar file. There will be folders for each charon node (max 5). Navigate to each node folder, and all artefacts related to each node will be present. +3. Double click to extract the tar file. There will be folders for each Charon node (max 5). Navigate to each node folder, and all artefacts related to each node will be present.
Screenshot: Double click to extract the tar file. There will be folders for each charon node (max 5). Navigate to each node folder, and all artefacts related to each node will be present.
@@ -542,7 +542,7 @@ docker compose up -d ``` {% hint style="danger" %} -Do not start this node until the DKG is complete, as the charon container will interfere with the charon instance attempting to take part in the DKG ceremony. +Do not start this node until the DKG is complete, as the Charon container will interfere with the Charon instance attempting to take part in the DKG ceremony. {% endhint %} If at any point you need to turn off your node, you can run: diff --git a/sdk/index.md b/sdk/index.md index 000056a4..975289f6 100644 --- a/sdk/index.md +++ b/sdk/index.md @@ -14,7 +14,7 @@ This repo contains the Obol Software Development Kit, for creating Distributed V ## Getting Started -Checkout our [docs](../advanced-and-troubleshooting/advanced/create-a-dv-using-the-sdk.md), [examples](https://github.com/ObolNetwork/obol-sdk-examples/), and SDK [reference](https://obolnetwork.github.io/obol-sdk). Further guides and walkthroughs coming soon. +Checkout our [docs](../advanced-and-troubleshooting/advanced/create-a-dv-using-the-sdk.md), [examples](https://github.com/ObolNetwork/obol-sdk-examples/), and SDK [reference](https://docs.obol.org/sdk). Further guides and walkthroughs coming soon. ### Terms and Conditions diff --git a/sdk/variables/README.md b/sdk/variables/README.md new file mode 100644 index 00000000..5df2cd1b --- /dev/null +++ b/sdk/variables/README.md @@ -0,0 +1,2 @@ +# Variables + From ab16f850284afade255e56424ba8e90319d5964a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ois=C3=ADn=20Kyne?= Date: Fri, 3 Jul 2026 02:43:10 +0100 Subject: [PATCH 2/3] Last updates --- run-a-dv/running/claim-rewards.md | 2 +- run-a-dv/running/metrics.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/run-a-dv/running/claim-rewards.md b/run-a-dv/running/claim-rewards.md index 7f896f87..f3709e58 100644 --- a/run-a-dv/running/claim-rewards.md +++ b/run-a-dv/running/claim-rewards.md @@ -55,7 +55,7 @@ Once the funds are distributed, go to the [Operator Dashboard](https://launchpad This section documents the legacy OWR (Optimistic Withdrawal Recipient) flow for older `0x01` validators. For new clusters with `0x02` validators, please use the [New Claim Rewards Process](#new-claim-rewards-process-0x02-validators) above. {% endhint %} -The method for claiming rewards depends on the cluster's withdrawal configuration, whether it's an [**OWR**](../../learn/intro/obol-splits.md#optimistic-withdrawal-recipient) or an [**Exitable Withdrawal Configuration**](../../learn/intro/obol-splits.md#optimistic-withdrawal-recipient). The table below outlines the latest details on how and where to claim rewards. +The method for claiming rewards depends on the cluster's withdrawal configuration, whether it's an [**OWR**](../../learn/intro/obol-splits.md#optimistic-withdrawal-recipient) or an [**Exitable Withdrawal Configuration**](../../learn/intro/obol-splits.md#obol-validator-managers). The table below outlines the latest details on how and where to claim rewards. ### Claim Status diff --git a/run-a-dv/running/metrics.md b/run-a-dv/running/metrics.md index 46161202..18c5849a 100644 --- a/run-a-dv/running/metrics.md +++ b/run-a-dv/running/metrics.md @@ -73,9 +73,9 @@ The table below is generated from the Charon source code and reflects the metric | `app_peerinfo_clock_offset_seconds` | Gauge | Peer clock offset in seconds | `peer` | | `app_peerinfo_git_commit` | Gauge | Constant gauge with git_hash label set to peer's git commit hash. | `peer, git_hash` | | `app_peerinfo_index` | Gauge | Constant gauge set to the peer index in the cluster definition | `peer` | -| `app_peerinfo_nickname` | Gauge | Constant gauge with nickname label set to peer's charon nickname. | `peer, peer_nickname` | +| `app_peerinfo_nickname` | Gauge | Constant gauge with nickname label set to peer's Charon nickname. | `peer, peer_nickname` | | `app_peerinfo_start_time_secs` | Gauge | Constant gauge set to the peer start time of the binary in unix seconds | `peer` | -| `app_peerinfo_version` | Gauge | Constant gauge with version label set to peer's charon version. | `peer, version` | +| `app_peerinfo_version` | Gauge | Constant gauge with version label set to peer's Charon version. | `peer, version` | | `app_peerinfo_version_support` | Gauge | Set to 1 if the peer's version is supported by (compatible with) the current version, else 0 if unsupported. | `peer` | | `app_start_time_secs` | Gauge | Gauge set to the app start time of the binary in unix seconds | | | `app_validator_stack_params` | Gauge | Parameters for each component of the validator stack in which this Charon instance is deployed into | `component, cli_parameters` | From 046c861b7206660dc01485f21ad7bf3234899028 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ois=C3=ADn=20Kyne?= Date: Tue, 7 Jul 2026 00:19:32 +0100 Subject: [PATCH 3/3] feat(stack): document agent businesses --- SUMMARY.md | 2 + obol-stack/README.md | 2 + obol-stack/agents-and-skills.md | 100 +++++++++++++++++++++++++ obol-stack/build-a-profitable-stack.md | 7 +- obol-stack/buying-services.md | 67 +++++++++++++++++ obol-stack/faq.md | 10 ++- obol-stack/quickstart.md | 15 ++-- obol-stack/selling-services.md | 33 ++++++-- 8 files changed, 220 insertions(+), 16 deletions(-) create mode 100644 obol-stack/agents-and-skills.md create mode 100644 obol-stack/buying-services.md diff --git a/SUMMARY.md b/SUMMARY.md index d234aa0c..4ef4d8b6 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -115,7 +115,9 @@ * [Introduction](obol-stack/README.md) * [Quickstart](obol-stack/quickstart.md) * [Build a Profitable Obol Stack](obol-stack/build-a-profitable-stack.md) +* [Agents & Skills](obol-stack/agents-and-skills.md) * [Selling Agent Services](obol-stack/selling-services.md) +* [Buying Services](obol-stack/buying-services.md) * [Set up a Permanent URL](obol-stack/permanent-url.md) * [Installing Networks](obol-stack/installing-networks.md) * [Installing Apps](obol-stack/installing-apps.md) diff --git a/obol-stack/README.md b/obol-stack/README.md index 8f7db1ba..6f812081 100644 --- a/obol-stack/README.md +++ b/obol-stack/README.md @@ -128,7 +128,9 @@ Running full Ethereum nodes requires significant disk space. Mainnet execution c - [Quickstart](quickstart.md) — install the stack, talk to your agent, and run `obol sell demo`. - [Build a profitable Obol Stack](build-a-profitable-stack.md) — end-to-end: bounded archive node → index → paid service → specialized agent → listed on marketplaces. +- [Agents & Skills](agents-and-skills.md) — create specialised sub-agents and see the embedded skill set they ship with. - [Selling agent services](selling-services.md) — full orientation on the three `sell` shapes, x402 economics, ERC-8004 registration, and Telegram notifications. +- [Buying services](buying-services.md) — rent remote models with `obol buy inference`, or pay any x402 endpoint from your agent. - [Installing Networks](installing-networks.md) — sync Ethereum / Aztec, including bounded archives via `--since`. ## Need assistance? diff --git a/obol-stack/agents-and-skills.md b/obol-stack/agents-and-skills.md new file mode 100644 index 00000000..926f870a --- /dev/null +++ b/obol-stack/agents-and-skills.md @@ -0,0 +1,100 @@ +--- +description: Create specialised sub-agents, manage their wallets, and understand the skills your agents ship with +--- + +# Agents & Skills + +## The default agent + +`obol stack up` creates a default **Hermes** agent in the `hermes-obol-agent` namespace, with its own Ethereum signing wallet (backed by an in-cluster remote signer — the agent never touches raw private keys) and a full embedded skill set. Talk to it with `obol hermes chat`, get its API bearer token with `obol agent auth`. + +{% hint style="warning" %} +Back up the agent's wallet before you put anything on it. Losing the keystore loses the agent's on-chain identity and funds. + +```shell +obol agent wallet backup -o ~/obol-wallet-backup.json --passphrase "..." +``` + +Store the backup **outside** `~/.config/obol/` — `obol stack purge -f` deletes that directory. +{% endhint %} + +## Creating sub-agents + +Additional agents are where the Stack gets interesting: each one is a separate, long-lived specialist with its own namespace, state, skills, and (optionally) its own wallet. + +```shell +obol agent new research \ + --model qwen3.5:9b \ + --skills ethereum-networks,buy-x402,indexing \ + --objective "Research onchain data and sell reports. Cite the block number your data came from." \ + --create-wallet +``` + +* `--objective` becomes the agent's identity — spend real effort on it: scope, output format, refusal policy, tone. One sharp paragraph beats a page of vibes. +* `--skills` should be **narrow**. A specialist selling on-chain analysis doesn't need generalist skills; a tight list keeps a smaller model focused and makes the agent's edge legible to buyers. +* `--create-wallet` gives the sub-agent its own wallet. Skip it if the agent doesn't need to hold or sign funds — payments for its services can go to your main agent's wallet instead. + +Manage the fleet: + +```shell +obol agent list # all agents +obol agent auth research # bearer token for a specific agent +obol agent sync # re-render agent deployments after config changes +obol agent delete research # remove one +``` + +An agent created this way can be put on sale in one command — `obol sell agent research --price 0.05 --token USDC --chain base` — turning it into an OpenAI-compatible paid endpoint. See [Selling Agent Services](selling-services.md). + +{% hint style="info" %} +OpenClaw remains supported as an alternate runtime: `obol agent new --runtime openclaw`, then `obol openclaw dashboard`. +{% endhint %} + +## Embedded skills + +Every agent ships with Obol's embedded skills — self-contained playbooks (instructions + scripts) the agent loads on demand. The default set gives your agent working knowledge of the commerce loop, Ethereum, and its own cluster: + +### Commerce & agents + +| Skill | What the agent can do with it | +| ----- | ----------------------------- | +| `sub-agent-business` | Design, evaluate, price, and sell specialised sub-agents people pay per turn — the business playbook | +| `agent-factory` | Spawn durable child agents with their own namespace, wallet, skills, and paid endpoint | +| `sell` | Create and manage payment-gated ServiceOffers, track reconciliation, register on ERC-8004 | +| `monetize-guide` | Guided end-to-end walkthrough for selling inference or an HTTP API | +| `buy-x402` | Buy from any x402 endpoint — probe pricing, pre-sign payments, auto-refill, check balances | +| `discovery` | Find agents registered on the ERC-8004 Identity Registry across chains | +| `swap` | Treasury moves — swap USDC/ETH/OBOL on Base and mainnet via Uniswap V3, with quotes and slippage guards | +| `autoresearch` / `autoresearch-coordinator` / `autoresearch-worker` | Run, coordinate, or sell GPU time for autonomous LLM optimization experiments | + +### Ethereum + +| Skill | What the agent can do with it | +| ----- | ----------------------------- | +| `ethereum-networks` | Read-only chain queries via the local RPC gateway — blocks, balances, contract reads, ERC-20, ENS | +| `ethereum-local-wallet` | Sign and send transactions via the per-agent remote signer | +| `addresses` | Verified contract addresses — payment rails first, then DeFi, tokens, bridges, registries | +| `building-blocks` | DeFi protocol composability — Uniswap, Aave, Aerodrome, Pendle | +| `concepts` | The onchain mental model — state machines, incentives, why nothing is automatic | +| `gas` | Real transaction costs today, mainnet vs L2 | +| `indexing` | Reading historical onchain data at scale — The Graph, Dune, Ponder, event-first design | +| `l2s` | The L2 landscape — which chain for which job | +| `standards` | ERC-8004, x402, EIP-3009, EIP-7702, ERC-4337 | +| `wallets` | Wallet architecture and key safety for AI agents | +| `why` | Why Ethereum for agents — the ERC-8004 + x402 loop | + +### Operations + +| Skill | What the agent can do with it | +| ----- | ----------------------------- | +| `obol-stack` | Kubernetes diagnostics from inside the cluster — pods, logs, events | +| `distributed-validators` | Obol DVT cluster monitoring via the Obol API | + +The list evolves — ask your agent, or run: + +```shell +obol hermes skills list # live catalogue (default agent) +obol hermes skills add # add a skill +obol hermes skills remove # remove one +``` + +Custom skills are just directories with a `SKILL.md` — drop your own curated data and playbooks into an agent's skills directory to give it an edge nobody else has. That curation is exactly what makes a sub-agent worth paying for: see [Build a Profitable Obol Stack](build-a-profitable-stack.md). diff --git a/obol-stack/build-a-profitable-stack.md b/obol-stack/build-a-profitable-stack.md index cc5882d6..b22a9b0c 100644 --- a/obol-stack/build-a-profitable-stack.md +++ b/obol-stack/build-a-profitable-stack.md @@ -67,7 +67,7 @@ obol sell http my-index \ --per-request 0.001 \ --chain base \ --token USDC \ - --wallet 0x...your-wallet... + --pay-to 0x...your-wallet... ``` Confirm it reconciles to `Ready`: @@ -124,10 +124,7 @@ Confirm it's live with `obol tunnel status`. For the full dashboard walkthrough Register your agent on [ERC-8004](https://eips.ethereum.org/EIPS/eip-8004) so buyers can find it: ```shell -obol sell register \ - --chain mainnet \ - --name my-quant \ - --private-key-file ~/.config/obol/agents/my-quant/wallet.json +obol sell register --chain mainnet --name my-quant ``` Registration writes to the on-chain Identity Registry. From there, multiple marketplaces and explorers index your offer: diff --git a/obol-stack/buying-services.md b/obol-stack/buying-services.md new file mode 100644 index 00000000..ac1f345c --- /dev/null +++ b/obol-stack/buying-services.md @@ -0,0 +1,67 @@ +--- +description: Rent a remote model with obol buy inference, or pay any x402 endpoint straight from your agent +--- + +# Buying Services + +The Obol Stack closes the commerce loop from both sides. Selling is covered in [Selling Agent Services](selling-services.md); this page is the buy side — how to spend from your agent's wallet on other people's services. + +There are two distinct buy paths. Don't conflate them: + +* **Buying inference is renting a brain.** You have no local Ollama, no provider API key, or you want a hosted model you don't otherwise have access to. What you get back is tokens. +* **Buying from an agent is renting specialised work.** You pay another agent per turn for a task — analysis, monitoring, drafting — not per token for completions. Drive this from `obol hermes chat`; your agent's embedded `buy-x402` skill handles the probe-pay-consume loop. + +## Rent a model: `obol buy inference` + +```shell +obol buy inference https://seller.example/ +``` + +Point it at any Obol Stack storefront URL (or omit the URL to use the default storefront). The command: + +1. Walks the seller's `/api/services.json` catalog and resolves the model and payment token from the offer. +2. Prompts you through the purchase on a TTY: auto-top-up yes/no → how many requests, with a cost preview → final confirmation. +3. Pre-signs payment authorizations via your agent's remote signer and creates a `PurchaseRequest`. +4. Publishes the model as `paid/` through your cluster's LiteLLM. + +After that, the remote model is just another model in your roster. Your agent (or any OpenAI-compatible client pointed at LiteLLM) can call it, and the in-cluster `x402-buyer` sidecar spends exactly one pre-signed authorization per request — **your maximum loss is bounded by what you pre-authorized**, and your agent never touches raw keys. + +### Useful flags + +```shell +obol buy inference https://seller.example/ --agent research # pay from agent `research`'s wallet + # and switch only that agent to the paid model +obol buy inference https://seller.example/ --set-default # promote the paid model globally, sync every agent +``` + +| Flag | What it does | +| ---- | ------------ | +| `--model ` | Pick a model when the seller offers several on one URL | +| `--count ` / `--budget ` | Size the purchase non-interactively (CI / scripts) | +| `--auto-refill` | Let the agent top up the budget when it runs low (`--refill-threshold`, `--refill-count`) | +| `--cost-cap ` | Refuse refills if the seller raises the per-request price above this ceiling | +| `--expected-agent-id ` | Opt-in ERC-8004 identity check of the seller (skipped by default) | + +{% hint style="info" %} +Call paid models with `"stream": true`. Responses buffered longer than ~100 seconds get dropped by Cloudflare quick tunnels on the seller side — streaming keeps the wire warm. +{% endhint %} + +## Pay any x402 endpoint from your agent + +Your agent ships with the `buy-x402` skill, which can probe and pay arbitrary x402-gated endpoints — one-shot HTTP calls, paid agent turns, or batch pre-authorizations. Just ask it in `obol hermes chat`: + +> "Probe https://seller.example/services/quant and tell me what it costs. If it's under 0.05 USDC per request, pay for one analysis of ETH gas trends." + +The skill checks pricing from the `402 Payment Required` response before spending, supports USDC (EIP-3009) and OBOL (Permit2), and tracks balances — ask your agent for `balance` any time. + +## Prefer a cloud provider instead? + +If you just want a good model and don't need the crypto rails, bring your own key: + +```shell +obol model setup # interactive picker (defaults to OpenRouter + free models) +obol model setup --provider anthropic --api-key sk-ant-... +obol model setup custom --endpoint http://192.168.1.20:8000/v1 --model my-model # any OpenAI-compatible endpoint +``` + +`obol buy inference` and `obol model setup` end in the same place — a model your agents can use — they just differ in who you pay and how. diff --git a/obol-stack/faq.md b/obol-stack/faq.md index e9b325dd..f4685c0d 100644 --- a/obol-stack/faq.md +++ b/obol-stack/faq.md @@ -161,13 +161,21 @@ USDC and other tokens settle on the rail their issuer supports (EIP-3009 for USD ### How do I list my service on a public agent registry? ```shell -obol sell register --chain mainnet --name my-service --private-key-file +obol sell register --chain mainnet --name my-service ``` This publishes the agent's wallet + service catalog to the [ERC-8004](https://eips.ethereum.org/EIPS/eip-8004) Identity Registry on the chain you specify. Note that this requires ETH on the registering wallet for gas. `obol sell demo` deliberately skips registration by default — run `obol sell register` later when you want on-chain discovery. +### How do I buy inference from another stack? + +```shell +obol buy inference https://seller.example/ +``` + +The command walks the seller's catalog, previews the cost, pre-signs payment authorizations from your agent's wallet, and publishes the remote model as `paid/` through your LiteLLM — your agents can then use it like any local model, with spend bounded by what you pre-authorized. See [Buying Services](buying-services.md). + ## Stack operations ### The cluster fails to start diff --git a/obol-stack/quickstart.md b/obol-stack/quickstart.md index 8dbb38c3..73c4d7a6 100644 --- a/obol-stack/quickstart.md +++ b/obol-stack/quickstart.md @@ -29,7 +29,7 @@ If you'd rather route through Anthropic or OpenAI, you can configure that with ` Run the bootstrap installer: ```shell -bash <(curl -s https://stack.obol.org) +bash <(curl -fsSL https://stack.obol.org) ``` The installer will: @@ -42,7 +42,7 @@ The installer will: {% tabs %} {% tab title="Default installation" %} ```shell -bash <(curl -s https://stack.obol.org) +bash <(curl -fsSL https://stack.obol.org) ``` Files are installed to: @@ -54,7 +54,7 @@ Files are installed to: {% tab title="Specific version" %} ```shell -OBOL_RELEASE=v0.9.0 bash <(curl -s https://stack.obol.org) +OBOL_RELEASE=v0.9.0 bash <(curl -fsSL https://stack.obol.org) ``` {% endtab %} @@ -137,9 +137,12 @@ Once you've watched a demo settle end-to-end, the same machinery lets you sell a ```shell obol sell inference my-model --model qwen3.5:9b --per-mtok 0.01 --token USDC --chain base obol sell http my-api --upstream my-svc --port 8080 --namespace my-ns \ - --per-request 0.001 --chain base --wallet + --per-request 0.001 --chain base --pay-to +obol sell agent my-analyst --price 0.05 --token USDC --chain base ``` +`sell agent` is the highest-margin shape — buyers pay for a whole specialised agent's replies (skills + memory + curated data), not just raw tokens. See [Agents & Skills](agents-and-skills.md) for building one worth paying for. + The mental model is: **anything in your cluster that exposes a Service can be wrapped in a `ServiceOffer` and gated behind x402**. The goal of v0.9 is to make that loop short enough that you can actually iterate on what's worth selling. See [Selling agent services](selling-services.md) for the full orientation on the three `sell` shapes (`http`, `inference`, `agent`), x402 economics, ERC-8004 registration, and marketplaces. @@ -155,7 +158,7 @@ Sellers receive `$OBOL` directly into their agent wallet. Read more about the [O `obol sell demo` skips on-chain registration by default (to avoid double-register reverts and the need for ETH on the agent wallet). When you're ready to be discoverable on a public agent registry: ```shell -obol sell register --chain mainnet --name my-service --private-key-file +obol sell register --chain mainnet --name my-service ``` This publishes the agent's wallet + service catalog to the [ERC-8004](https://eips.ethereum.org/EIPS/eip-8004) Identity Registry on the chain you specify. @@ -215,6 +218,8 @@ obol stack purge -f # remove everything, including data * [Build a profitable Obol Stack](build-a-profitable-stack.md) — the end-to-end narrative: sync a bounded archive node, build an index, wrap it as a paid service, and turn it into a specialized agent business. * [Selling agent services](selling-services.md) — depth on the three `sell` shapes, x402 economics, and getting listed on marketplaces. +* [Buying services](buying-services.md) — rent a remote model with `obol buy inference`, or pay any x402 endpoint from your agent. +* [Agents & Skills](agents-and-skills.md) — create specialised sub-agents and see everything your agent can already do. * [Installing Networks](installing-networks.md) — sync local Ethereum / Aztec nodes (including bounded archives via `--since`). * [Installing Apps](installing-apps.md) — deploy any Helm chart. * [FAQ](faq.md) — common questions and troubleshooting. diff --git a/obol-stack/selling-services.md b/obol-stack/selling-services.md index 4b850f5f..4f0d6a60 100644 --- a/obol-stack/selling-services.md +++ b/obol-stack/selling-services.md @@ -53,7 +53,7 @@ obol sell http my-index \ --per-request 0.001 \ --chain base \ --token USDC \ - --wallet 0x...your-wallet... + --pay-to 0x...your-wallet... ``` This is the path to take when **the value is in the data**: you've synced an archive node from a specific block, built an index on top, and you want to sell access to queries against it that no public RPC will serve. @@ -111,10 +111,7 @@ When quoting prices, name the unit explicitly — `0.01 OBOL / MTok`, `0.001 USD Buyers need to find you. The Obol Stack publishes an [ERC-8004](https://eips.ethereum.org/EIPS/eip-8004) agent registration document at `/.well-known/agent-registration.json` describing your services, supported payment methods, and endpoints. To list your agent on a public registry: ```shell -obol sell register \ - --chain mainnet \ - --name my-quant \ - --private-key-file ~/.config/obol/agents/my-quant/wallet.json +obol sell register --chain mainnet --name my-quant ``` This publishes your agent's wallet and service catalog to the ERC-8004 Identity Registry on the chain you specify. **Note that this requires ETH on the registering wallet for gas** — registration is not gas-sponsored. @@ -131,6 +128,19 @@ Once registered, several marketplaces and explorers index ERC-8004 registries an We do not recommend a single "official" marketplace — the agent-registry ecosystem is evolving fast, and the best strategy is to register on-chain and let multiple indexers pick you up. +### Brand your storefront + +Your tunnel hostname serves a public storefront landing page and a machine-readable catalog at `/api/services.json` (plus `/skill.md` for agent buyers). Make it look like a business, not a default install: + +```shell +obol sell info # buyer's-eye view: branding + every on-sale service +obol sell info my-quant # focus one service + how-to-buy +obol sell info set --display-name "Acme Labs" --tagline "Paid onchain analysis." --logo-url "https://…" +obol sell info reset # back to defaults +``` + +`obol sell info` shows exactly what buyers see — only operationally-ready offers appear. Operator-side health and conditions (including draining or not-ready offers) stay under `obol sell status`. After any change, run `sell info` and ask yourself: *would I buy from this storefront?* + ## Iterating on an agent business Selling the same agent forever at the same quality is leaving money on the table. The loop that compounds: @@ -156,6 +166,19 @@ obol sell delete my-quant --namespace my-ns Deletion removes the ServiceOffer CR, cascades the underlying Middleware and HTTPRoute via owner references, and deactivates the ERC-8004 registration (sets `active=false`). The agent's wallet and accumulated revenue are untouched. +### Surviving restarts + +Offers are persisted and replayed automatically: `obol stack up` re-publishes every offer after a reboot. To replay on demand — or to make it happen with nobody at the keyboard: + +```shell +obol sell resume # replay all persisted offers now +obol sell resume --install-boot-unit # Linux: install a systemd user unit that does it on boot +``` + +### Bonus shape: paid MCP tools + +`obol sell mcp [name]` runs a foreground x402-paid MCP server that forwards buyer JSON arguments to a backend HTTP service, injecting **your** API key server-side — the buyer pays per call and never sees the key. Useful for reselling metered access to an upstream API as an MCP tool. It's foreground-only: no ServiceOffer, and it is not replayed by `sell resume`. + ## Verifying your paths When you've shipped your first real (non-demo) offer, walk these checks: