Repository navigation
Work through five issues: cancel votes, rules observers, thresholds, reward IT, error logs - #560
Conversation
Every decman error line on devnet in the last day was this one: 630 lines from node 1, as health probes to two unreachable peers ran out of retries. A decman-wide error-rate alert would therefore fire all day and hide the errors a person must act on. The line is not where the error is handled. Each caller handles the returned error itself: the status probe marks the peer unreachable, the version check reports it, and workflow steps fail on their own terms. The ping loop also warns once when a peer becomes unreachable. Refs #362.
The devnet reward check could not do its job. The full suite onboards a fresh decparty, which earns no coupons, so the reward phase always skipped. Its devnet branch also voted in a delegation with member stand-ins, so it never checked the parties that are really deployed. devnet_reward_e2e runs only the reward phase, against the deployed cbtc-network decparty. It reads the active delegation from PQS instead of voting one in. The harness nodes then assign the live coupons. The test checks that a harness assigner's counter rose. It also checks each new coupon in PQS against the delegation's split, by value. The harness talks to local DecMan processes on both targets, so the HTTP layer needed no remote host. URLs now come from one helper keyed on the target. The Keycloak bearer for devnet was already in place. PQS is Postgres, so the tests enable sqlx's postgres feature as a dev-dependency. The lockfile already held sqlx-postgres. Every PQS session is read-only. The full suite no longer runs the reward phase on devnet. The new test is ignored, so CI never starts it. Closes #271.
Each workflow wrote one value to both the namespace threshold and the confirmation threshold. Operators could not ask for one owner signature but three confirming hosts. Onboarding and change-threshold now take an optional confirmation_threshold. When a caller omits it, both mappings get the same value. decman-cli and older clients therefore behave as before. The party's signing keys keep the namespace threshold. Kick and add-party keep matching thresholds matched. When a party's two thresholds differ, they keep the confirmation threshold. A kick caps it at the hosts that remain. The coordinator checks the chain before it signs. An out-of-date cache therefore cannot reset the threshold. Peers hold each threshold to the value in the accepted invitation. A peer built before this change refuses a proposal whose two thresholds differ. Closes #234.
GovernanceRules had no observers. A party saw the contract only by hosting the governance party. An additional proposer on another participant could not read the rules, so it could not confirm a self-action or fetch the contract to disclose it. #165 asked us to choose a fix. This takes option 4 from the issue: opt-in observers. governance-core 0.2.0 adds an optional disclosureObservers field to GovernanceRules. It is the last field, so Smart Contract Upgrade accepts it. Listed parties observe the contract. They gain no right to confirm or execute. Members change the list with a new SelfAction_SetDisclosureObservers vote. An empty set becomes None, as with additional proposers. A rules contract created under 0.1.0 reads as None, so its visibility stays the same. It gains observers only when that vote replaces it. The upgrade check passes. It warns that the observer expression changed, but for None that expression gives the empty set, as 0.1.0 did. decman reads the field, shows it on the rules view, and offers the vote in the self-management form. The deploy form gets the new field. It sends an empty list as None. A node with only 0.1.0 vetted drops a trailing None, but it refuses a Some. Closes #165.
GovernanceRules_ConfirmAction rejects every confirmation of a proposal whose proposer the live rules no longer list. Only that proposer could retract it. When nobody controls the proposer any more, the card never leaves the Approvals tab. #524 flagged such cards. No action could reach GovernableAction_Cancel, so they had no exit. GovernanceSelfAction gains SelfAction_CancelAction, which carries the proposal's contract id. Executing it cancels that proposal with the governance party's authority. It is a self-action, so only member votes count toward it. That answers the issue's first question on the ledger. This replaces the separate package the issue proposed. The cancellation leaves the proposal's confirmations in place. They point at an archived contract, so no execution can use them. A cancellation has no proposer, so a change in proposer rights cannot strand it. Its confirmations survive a membership change, and only current members count. Rules.daml states this beside the choice. On a flagged card a member can now propose the cancellation. The confirm step shows the proposal's label, proposer, description and contract id. The vote card names the proposal it cancels. The 0.2.0 DAR is rebuilt, since 0.2.0 has not been released. Closes #518.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved UI correctness issues can alter an existing confirmation policy and misclassify pending governance votes.
2 open issues
What changed in this PR
Extends DecMan’s governance and topology workflows, adds a standalone devnet reward test, and reduces retry-exhaustion log severity.
Changes:
- Adds cancellation votes and opt-in rules observers through governance-core 0.2.0.
- Separates namespace and confirmation thresholds across workflows, storage, APIs, and UI.
- Adds reward-only devnet testing and changes exhausted peer retries to warnings.
| File | Description |
|---|---|
| README.md | Documents thresholds and devnet reward testing. |
| integration-tests/run.sh | Validates reward-test prerequisites. |
| e2e/tests/governance-happy-path.spec.ts | Uses governance-core 0.2.0. |
| docs/CUSTOM_DAML_TEMPLATES.md | Documents observers and cancellation votes. |
| docs/coupon-reassignment/design.md | Updates devnet reward-test design. |
| docs/ARCHITECTURE.md | Explains new governance actions and thresholds. |
| docs/ADDING_A_MEMBER_TO_A_DECENTRALIZED_PARTY.md | Explains confirmation-threshold preservation. |
| development/remote/participant-3/.env.example | Documents reward-test configuration. |
| development/remote/participant-2/.env.example | Documents reward-test configuration. |
| development/remote/participant-1/.env.example | Documents reward-test configuration. |
| daml/governance-utility-onboarding-test/daml/Governance/UtilityOnboarding/TestUtils.daml | Updates rules fixtures. |
| daml/governance-utility-onboarding-test/daml/Governance/UtilityOnboarding/Test/GranularActionsTest.daml | Updates rules fixture. |
| daml/governance-utility-onboarding-test/daml/Governance/UtilityOnboarding/DualDecpartyTestUtils.daml | Updates dual-governance fixtures. |
| daml/governance-utility-onboarding-test/daml.yaml | Updates core dependency. |
| daml/governance-utility-credential-test/daml/Governance/UtilityCredential/TestUtils.daml | Updates rules fixtures. |
| daml/governance-utility-credential-test/daml.yaml | Updates core dependency. |
| daml/governance-token-custody-test/daml/Governance/TokenCustody/TestUtils.daml | Updates rules fixtures. |
| daml/governance-token-custody-test/daml.yaml | Updates core dependency. |
| daml/governance-rewards-test/daml.yaml | Updates core dependency. |
| daml/governance-rewards-assign-test/daml/Governance/Rewards/AssignTestUtils.daml | Updates rules fixtures. |
| daml/governance-rewards-assign-test/daml.yaml | Updates core dependency. |
| daml/governance-core/daml/Governance/Rules.daml | Implements observers and cancellation votes. |
| daml/governance-core/daml.yaml | Bumps core to 0.2.0. |
| daml/governance-core-test/daml/Governance/TestUtils.daml | Updates shared rules fixtures. |
| daml/governance-core-test/daml/Governance/Test/SelfActionTest.daml | Updates self-action fixtures. |
| daml/governance-core-test/daml/Governance/Test/RulesTest.daml | Updates rules-validation fixtures. |
| daml/governance-core-test/daml/Governance/Test/ExecutionTest.daml | Updates execution fixture. |
| daml/governance-core-test/daml/Governance/Test/DisclosureObserversTest.daml | Tests observer visibility and permissions. |
| daml/governance-core-test/daml/Governance/Test/ConfirmationTest.daml | Updates confirmation fixture. |
| daml/governance-core-test/daml/Governance/Test/CancelActionTest.daml | Tests cancellation authorization and cleanup. |
| daml/governance-core-test/daml/Governance/Test/AuthorizationTest.daml | Updates authorization fixtures. |
| daml/governance-core-test/daml.yaml | Updates core dependency. |
| crates/decman/tests/governance_workflows.rs | Adds reward-only devnet entrypoint. |
| crates/decman/tests/common/phases/mod.rs | Registers deployed reward phase. |
| crates/decman/tests/common/phases/distribute_dars.rs | Distributes updated core package. |
| crates/decman/tests/common/phases/deploy_gov_core.rs | Exposes credential configuration helper. |
| crates/decman/tests/common/mod.rs | Registers PQS helpers and service URLs. |
| crates/decman/tests/common/http.rs | Centralizes test service URLs. |
| crates/decman/src/workflow/validation.rs | Validates thresholds independently. |
| crates/decman/src/workflow/thresholds.rs | Centralizes threshold construction and preservation. |
| crates/decman/src/workflow/onboarding/steps/proposals/create.rs | Builds mappings with separate thresholds. |
| crates/decman/src/workflow/onboarding/config.rs | Stores optional confirmation threshold. |
| crates/decman/src/workflow/mod.rs | Exports threshold helpers. |
| crates/decman/src/workflow/kick/steps/proposals/create.rs | Preserves or caps confirmation threshold. |
| crates/decman/src/workflow/kick/steps/cache.rs | Updates cache fixture. |
| crates/decman/src/workflow/kick/config.rs | Carries confirmation threshold through kick. |
| crates/decman/src/workflow/contracts/steps/prepare.rs | Updates party fixture. |
| crates/decman/src/workflow/change_threshold/steps/proposals/submit.rs | Waits for both P2P thresholds. |
| crates/decman/src/workflow/change_threshold/steps/proposals/create.rs | Builds independent threshold changes. |
| crates/decman/src/workflow/change_threshold/steps/export_state.rs | Allows confirmation-only changes. |
| crates/decman/src/workflow/change_threshold/coordinator.rs | Passes thresholds to submission. |
| crates/decman/src/workflow/change_threshold/config.rs | Stores independent threshold configuration. |
| crates/decman/src/workflow/add_party/steps/proposals/create.rs | Preserves independent confirmation policy. |
| crates/decman/src/workflow/add_party/config.rs | Carries confirmation threshold through additions. |
| crates/decman/src/server/serde_snapshots.rs | Adds new action fixtures. |
| crates/decman/src/server/queries.rs | Returns disclosure observers. |
| crates/decman/src/server/mod.rs | Propagates invitation confirmation thresholds. |
| crates/decman/src/server/handlers/workflows.rs | Validates and forwards separate thresholds. |
| crates/decman/src/server/handlers/parties.rs | Reads and caches confirmation thresholds. |
| crates/decman/src/server/handlers/invitations.rs | Persists accepted confirmation thresholds. |
| crates/decman/src/server/handlers/governance.rs | Tests cancellation-action validation. |
| crates/decman/src/noise/mod.rs | Logs retry exhaustion at warning level. |
| crates/decman/src/db/sqlite.rs | Persists new threshold fields. |
| crates/decman/src/db/rows.rs | Extends party and invitation rows. |
| crates/decman/snapshots/dec_party_manager__server__serde_snapshots__action_types.snap | Records new action wire formats. |
| crates/decman/migrations/000024_confirmation_threshold.up.sql | Adds confirmation-threshold columns. |
| crates/decman/migrations/000024_confirmation_threshold.down.sql | Removes confirmation-threshold columns. |
| crates/decman/frontend/src/governanceMembership.test.ts | Updates governance fixture. |
| crates/decman/frontend/src/governanceFormat.ts | Formats observers and cancellation targets. |
| crates/decman/frontend/src/components/PartyDetail.tsx | Displays confirmation threshold. |
| crates/decman/frontend/src/components/PartyDetail.test.tsx | Tests threshold display. |
| crates/decman/frontend/src/components/OnboardingDialog.tsx | Adds confirmation-threshold input. |
| crates/decman/frontend/src/components/OnboardingDialog.test.tsx | Tests onboarding threshold requests. |
| crates/decman/frontend/src/components/NotificationsView.tsx | Adds cancellation voting and threshold details. |
| crates/decman/frontend/src/components/NotificationsView.threshold.test.tsx | Tests invitation threshold display. |
| crates/decman/frontend/src/components/NotificationsView.cancel.test.tsx | Tests cancellation UI. |
| crates/decman/frontend/src/components/GovernanceSection.tsx | Adds observer-management form. |
| crates/decman/frontend/src/components/GovernanceSection.disclosureObservers.test.tsx | Tests observer-management requests. |
| crates/decman/frontend/src/components/GovernanceRules.tsx | Displays disclosure observers. |
| crates/decman/frontend/src/components/GovernanceRules.test.tsx | Tests observer display. |
| crates/decman/frontend/src/components/ExecuteDialog.tsx | Labels new governance actions. |
| crates/decman/frontend/src/components/ContractsDialog.tsx | Adds deployment observer field. |
| crates/decman/frontend/src/components/ContractsDialog.test.tsx | Tests observer deployment encoding. |
| crates/decman/frontend/src/components/ChangeThresholdDialog.tsx | Adds independent confirmation input. |
| crates/decman/frontend/src/components/ChangeThresholdDialog.test.tsx | Tests separate threshold submissions. |
| crates/decman/Cargo.toml | Enables test-only Postgres support. |
| crates/decman-lib/src/framework/encode.rs | Adds party-set encoders. |
| crates/decman-lib/src/catalog/interpret.rs | Decodes rules observers. |
| crates/decman-lib/src/catalog/action.rs | Encodes and decodes new self-actions. |
| crates/decman-cli/src/ui.rs | Updates fixtures for threshold fields. |
| crates/decman-cli/src/app.rs | Updates party fixture. |
| crates/decman-cli/src/api.rs | Updates party fixture. |
| crates/common/src/types.rs | Extends party and invitation types. |
| crates/common/src/api.rs | Extends threshold and observer API fields. |
| Cargo.lock | Records Postgres feature dependency. |
Review effort: Balanced
The change-threshold dialog seeded the confirmation threshold from the namespace threshold. On a party whose two thresholds differ, opening the dialog therefore showed the wrong value, and a namespace-only change reset the confirmation threshold to match. The dialog now starts at the party's current confirmation threshold when the two differ. It follows the namespace threshold only when they already match. Refs #234.
The form posted the observers in the order the operator typed them, but the backend sorts a party set before it reaches the ledger. The audit entry and the ledger action then listed the parties in a different order. They no longer matched, so an open vote showed up under Completed. The form now sorts the list the same way before it posts. Refs #165.
Copilot review loop: disposition summary
By category
The two fixes:
Outcome: converged. Round 2 reported no open issues. Open items: none. The live devnet suite and |
scolear
left a comment
There was a problem hiding this comment.
Reviewed at cddf659, one commit per issue.
Verdict per commit
0c730b08(refs #362): sound. The exhaustion line moves towarnand keepspeer,attemptsanderroras fields. The panic hook, the fatal-exit line andspawn_supervisedstay aterror, as the issue requires. The new test fails if the line goes back toerror.02c08948(closes #271): sound. The entrypoint is#[ignore], and it needsDECPM_IT_TARGET=devnetplusDECPM_IT_REWARD=1.run.shchecks the inputs before the build. The repo holds no hostname or secret; the PQS URL stays in the operator's environment, and errors never echo it. The closes holds for the acceptance list in the issue.afabec59(closes #234): sound. See the threshold notes below.afd56d39(closes #165): sound. Option 4 from the issue, as the field the issue asked for. See the observer notes below.58ac006b(closes #518): sound. See the cancel notes below.
Thresholds (#234)
A note on terms first. The confirmation threshold in this PR is PartyToParticipant.threshold: how many hosting participants confirm a transaction. It is not the Daml GovernanceRules.threshold. No Daml template changes in this commit, and the rules threshold is untouched.
Before this change, onboarding, kick, add-party and change-threshold wrote one value to both mappings. Now each threshold has its own field in the mapping. check_confirmation_threshold bounds the confirmation threshold to 1..=hosts in both handlers, and both dialogs check the same range. ensure_carried_from compares the plan with the chain before the coordinator signs, so a stale cache cannot reset a differing confirmation threshold. The peer check pins each threshold to the accepted invitation value, and an older peer refuses a differing pair. A party with equal thresholds carries None everywhere, so its invites and configs are what every current node expects.
One combination deserves a word in the dialog: namespace threshold 1 with a confirmation threshold above 1. One owner key then signs a topology change, and that change can rewrite the mapping, including the confirmation threshold. The namespace threshold bounds what the confirmation threshold protects. This was already possible with a single threshold of 1, so it is not a regression. A one-line hint next to the confirmation field would help an operator see it. Not a blocker.
Observers (#165)
disclosureObservers is the last field and optional, so a 0.1.0 contract reads as None and its observer set stays empty. The upgrade-check warning is expected for that reason. Observers see only GovernanceRules: members, thresholds, proposers and timeout. They do not see self-confirmations or domain proposals, which have their own stakeholders, and the test shows an observer cannot confirm. A live party gains observers only through a SelfAction_SetDisclosureObservers vote, which needs 0.2.0 vetted on every member participant. The rollout risk section says so.
Cancel (#518)
GovernanceRules_ConfirmGovernanceAction accepts the cancel vote from members and additional proposers, and execution counts member votes only. Execution fetches the proposal, checks governanceParty, and exercises GovernableAction_Cancel with the governance party's authority. A cancel that lands after execution fails at the fetch, and its confirmations expire. The reverse race is covered by the test: old confirmations cannot execute a cancelled proposal. The audit decodes SelfAction_CancelAction through from_self_proto, so the entry carries the proposal cid. Note that members can cancel any proposal of their governance, not only a stranded one. That is the governance party's authority and it matches the interface comment.
Not a blocker: Noise is on its way out (#451), so the warn change is a short-lived fix. The UI does not gate the two new actions on 0.2.0 vetting, as the body says.

What
This PR works through five open issues. Each change is its own commit, so you can review one commit at a time.
0c730b08warn. The alert rule for #362 is in DLC-link/dlc-infra#312.02c08948cbtc-network.afabec59afd56d39GovernanceRulescan name opt-in disclosure observers (governance-core 0.2.0).58ac006bWhy
#362. On 2026-10-20, all 630 decman ERROR lines on devnet were
noise: peer unreachable after retry exhaustion, from node 1. Each caller already handles that error, and the ping loop warns once when a peer becomes unreachable. A decman-wide error alert can now fire on a single line.#271. The full suite onboards a fresh decparty, which earns no coupons, so the devnet reward phase always skipped.
devnet_reward_e2eruns only the reward phase, against the deployed delegation. It reads the delegation from PQS, waits for a harness assigner's counter to rise, then checks each new coupon in PQS against the split.#234. DA asked for the two thresholds to be set apart (#231).
POST /onboardingandPOST /change-thresholdtake an optionalconfirmation_threshold. Leaving it out keeps today's behaviour.#165. A proposer hosted on another participant cannot see
GovernanceRules, because the contract has no observers. The newdisclosureObserversfield is optional and last, so 0.1.0 contracts upgrade unchanged. A new self-action sets it.#518. A proposal whose proposer the rules no longer authorize could never be removed.
SelfAction_CancelActionlets members vote it away, through the self-action flow that members already use.Key decisions
PartyToParticipantfollows the namespace threshold. Add-party and kick keep a differing confirmation threshold. Kick lowers it to the number of remaining hosts when needed.postgresfeature as a dev-dependency only.sqlx-postgres0.8.6 was already inCargo.lock. Every PQS session is read-only.Rollout risks
new_threshold. On a party whose thresholds differ, a CLI threshold change sets them back to one value.None. A node with only 0.1.0 refusesSomeof an empty set. A local Canton 3.4.11 sandbox confirmed both cases.000024is free onmaintoday.How tested
cargo fmt --checkandcargo clippy --workspace --all-targets --all-features -- -D warningsare clean.cargo test --workspace: 1,298 passed, 0 failed.tsc -bis clean, andvitest runpasses 237 tests.dpm upgrade-checkagainstreleases/v1exits 0, with one expected warning: "observers expression changed". ForNonethe expression gives the empty set, as 0.1.0 did.error. The test fails if the line goes back toerror.config_json, both new columns, and both dialogs.devnet_reward_e2ehas never run against a real PQS or ledger. It also needs Fix five open issues: peer logs, devnet tests, signing docs, fingerprints, rewards #559's harness repair.Size
The diff is 99 files, +4,681 / −718. That is far over the usual 400. Tests and snapshots are about half. In #271, most of
coupon_reassignment.rsis de-indentation (git show -wgives +118 / −203).Left out