Skip to content

Work through five issues: cancel votes, rules observers, thresholds, reward IT, error logs - #560

Merged
schronck merged 7 commits into
mainfrom
fix/issue-sweep-3
Oct 20, 2026
Merged

schronck merged 7 commits into
mainfrom
fix/issue-sweep-3

Conversation

@schronck

@schronck schronck commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

What

This PR works through five open issues. Each change is its own commit, so you can review one commit at a time.

Commit Issue Change
0c730b08 Refs #362 Noise retry exhaustion logs at warn. The alert rule for #362 is in DLC-link/dlc-infra#312.
02c08948 Closes #271 A new ignored entrypoint runs the reward IT against devnet's cbtc-network.
afabec59 Closes #234 The namespace threshold and the confirmation threshold can differ.
afd56d39 Closes #165 GovernanceRules can name opt-in disclosure observers (governance-core 0.2.0).
58ac006b Closes #518 Members can vote to cancel a stranded proposal (governance-core 0.2.0).

Why

#362. On 2026-10-20, all 630 decman ERROR lines on devnet were noise: peer unreachable after retry exhaustion, from node 1. Each caller already handles that error, and the ping loop warns once when a peer becomes unreachable. A decman-wide error alert can now fire on a single line.

#271. The full suite onboards a fresh decparty, which earns no coupons, so the devnet reward phase always skipped. devnet_reward_e2e runs only the reward phase, against the deployed delegation. It reads the delegation from PQS, waits for a harness assigner's counter to rise, then checks each new coupon in PQS against the split.

#234. DA asked for the two thresholds to be set apart (#231). POST /onboarding and POST /change-threshold take an optional confirmation_threshold. Leaving it out keeps today's behaviour.

#165. A proposer hosted on another participant cannot see GovernanceRules, because the contract has no observers. The new disclosureObservers field is optional and last, so 0.1.0 contracts upgrade unchanged. A new self-action sets it.

#518. A proposal whose proposer the rules no longer authorize could never be removed. SelfAction_CancelAction lets members vote it away, through the self-action flow that members already use.

Key decisions

Rollout risks

  • governance-core 0.2.0 must be vetted on a node before it uses the two new actions. On a node without it, they fail with a ledger error. The UI does not check the version first.
  • An older peer refuses a proposal whose two thresholds differ. An onboarding run then fails at the P2P signing step, after the namespace is on-chain. Do not set the thresholds apart until every node runs this release. A version gate could block such runs up front. I left it out, because it needs the release number.
  • decman-cli sends only new_threshold. On a party whose thresholds differ, a CLI threshold change sets them back to one value.
  • The deploy form sends an empty observer list as None. A node with only 0.1.0 refuses Some of an empty set. A local Canton 3.4.11 sandbox confirmed both cases.
  • Migration 000024 is free on main today.

How tested

Size

The diff is 99 files, +4,681 / −718. That is far over the usual 400. Tests and snapshots are about half. In #271, most of coupon_reassignment.rs is de-indentation (git show -w gives +118 / −203).

Left out

Every decman error line on devnet in the last day was this one: 630
lines from node 1, as health probes to two unreachable peers ran out
of retries. A decman-wide error-rate alert would therefore fire all
day and hide the errors a person must act on.

The line is not where the error is handled. Each caller handles the
returned error itself: the status probe marks the peer unreachable,
the version check reports it, and workflow steps fail on their own
terms. The ping loop also warns once when a peer becomes unreachable.

Refs #362.
The devnet reward check could not do its job. The full suite onboards a
fresh decparty, which earns no coupons, so the reward phase always
skipped. Its devnet branch also voted in a delegation with member
stand-ins, so it never checked the parties that are really deployed.

devnet_reward_e2e runs only the reward phase, against the deployed
cbtc-network decparty. It reads the active delegation from PQS instead
of voting one in. The harness nodes then assign the live coupons. The
test checks that a harness assigner's counter rose. It also checks each
new coupon in PQS against the delegation's split, by value.

The harness talks to local DecMan processes on both targets, so the
HTTP layer needed no remote host. URLs now come from one helper keyed
on the target. The Keycloak bearer for devnet was already in place.

PQS is Postgres, so the tests enable sqlx's postgres feature as a
dev-dependency. The lockfile already held sqlx-postgres. Every PQS
session is read-only.

The full suite no longer runs the reward phase on devnet. The new test
is ignored, so CI never starts it.

Closes #271.
Each workflow wrote one value to both the namespace threshold and the
confirmation threshold. Operators could not ask for one owner signature
but three confirming hosts.

Onboarding and change-threshold now take an optional
confirmation_threshold. When a caller omits it, both mappings get the
same value. decman-cli and older clients therefore behave as before.
The party's signing keys keep the namespace threshold.

Kick and add-party keep matching thresholds matched. When a party's two
thresholds differ, they keep the confirmation threshold. A kick caps it
at the hosts that remain. The coordinator checks the chain before it
signs. An out-of-date cache therefore cannot reset the threshold.

Peers hold each threshold to the value in the accepted invitation. A
peer built before this change refuses a proposal whose two thresholds
differ.

Closes #234.
GovernanceRules had no observers. A party saw the contract only by
hosting the governance party. An additional proposer on another
participant could not read the rules, so it could not confirm a
self-action or fetch the contract to disclose it. #165 asked us to
choose a fix.

This takes option 4 from the issue: opt-in observers. governance-core
0.2.0 adds an optional disclosureObservers field to GovernanceRules.
It is the last field, so Smart Contract Upgrade accepts it. Listed
parties observe the contract. They gain no right to confirm or
execute. Members change the list with a new
SelfAction_SetDisclosureObservers vote. An empty set becomes None, as
with additional proposers.

A rules contract created under 0.1.0 reads as None, so its visibility
stays the same. It gains observers only when that vote replaces it.
The upgrade check passes. It warns that the observer expression
changed, but for None that expression gives the empty set, as 0.1.0
did.

decman reads the field, shows it on the rules view, and offers the
vote in the self-management form. The deploy form gets the new field.
It sends an empty list as None. A node with only 0.1.0 vetted drops a
trailing None, but it refuses a Some.

Closes #165.
GovernanceRules_ConfirmAction rejects every confirmation of a proposal
whose proposer the live rules no longer list. Only that proposer could
retract it. When nobody controls the proposer any more, the card never
leaves the Approvals tab. #524 flagged such cards. No action could
reach GovernableAction_Cancel, so they had no exit.

GovernanceSelfAction gains SelfAction_CancelAction, which carries the
proposal's contract id. Executing it cancels that proposal with the
governance party's authority. It is a self-action, so only member votes
count toward it. That answers the issue's first question on the ledger.
This replaces the separate package the issue proposed.

The cancellation leaves the proposal's confirmations in place. They
point at an archived contract, so no execution can use them. A
cancellation has no proposer, so a change in proposer rights cannot
strand it. Its confirmations survive a membership change, and only
current members count. Rules.daml states this beside the choice.

On a flagged card a member can now propose the cancellation. The
confirm step shows the proposal's label, proposer, description and
contract id. The vote card names the proposal it cancels. The 0.2.0
DAR is rebuilt, since 0.2.0 has not been released.

Closes #518.
@schronck
schronck requested review from a team and sosaucily October 20, 2026 12:25
@schronck schronck self-assigned this Oct 20, 2026
@schronck
schronck requested review from scolear and a balanced review from Copilot October 20, 2026 12:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved UI correctness issues can alter an existing confirmation policy and misclassify pending governance votes.

2 open issues
What changed in this PR

Extends DecMan’s governance and topology workflows, adds a standalone devnet reward test, and reduces retry-exhaustion log severity.

Changes:

  • Adds cancellation votes and opt-in rules observers through governance-core 0.2.0.
  • Separates namespace and confirmation thresholds across workflows, storage, APIs, and UI.
  • Adds reward-only devnet testing and changes exhausted peer retries to warnings.
File Description
README.md Documents thresholds and devnet reward testing.
integration-tests/​run.sh Validates reward-test prerequisites.
e2e/​tests/​governance-happy-path.spec.ts Uses governance-core 0.2.0.
docs/​CUSTOM_DAML_TEMPLATES.md Documents observers and cancellation votes.
docs/​coupon-reassignment/​design.md Updates devnet reward-test design.
docs/​ARCHITECTURE.md Explains new governance actions and thresholds.
docs/​ADDING_A_MEMBER_TO_A_DECENTRALIZED_PARTY.md Explains confirmation-threshold preservation.
development/​remote/​participant-3/​.env.example Documents reward-test configuration.
development/​remote/​participant-2/​.env.example Documents reward-test configuration.
development/​remote/​participant-1/​.env.example Documents reward-test configuration.
daml/​governance-utility-onboarding-test/​daml/​Governance/​UtilityOnboarding/​TestUtils.daml Updates rules fixtures.
daml/​governance-utility-onboarding-test/​daml/​Governance/​UtilityOnboarding/​Test/​GranularActionsTest.daml Updates rules fixture.
daml/​governance-utility-onboarding-test/​daml/​Governance/​UtilityOnboarding/​DualDecpartyTestUtils.daml Updates dual-governance fixtures.
daml/​governance-utility-onboarding-test/​daml.yaml Updates core dependency.
daml/​governance-utility-credential-test/​daml/​Governance/​UtilityCredential/​TestUtils.daml Updates rules fixtures.
daml/​governance-utility-credential-test/​daml.yaml Updates core dependency.
daml/​governance-token-custody-test/​daml/​Governance/​TokenCustody/​TestUtils.daml Updates rules fixtures.
daml/​governance-token-custody-test/​daml.yaml Updates core dependency.
daml/​governance-rewards-test/​daml.yaml Updates core dependency.
daml/​governance-rewards-assign-test/​daml/​Governance/​Rewards/​AssignTestUtils.daml Updates rules fixtures.
daml/​governance-rewards-assign-test/​daml.yaml Updates core dependency.
daml/​governance-core/​daml/​Governance/​Rules.daml Implements observers and cancellation votes.
daml/​governance-core/​daml.yaml Bumps core to 0.2.0.
daml/​governance-core-test/​daml/​Governance/​TestUtils.daml Updates shared rules fixtures.
daml/​governance-core-test/​daml/​Governance/​Test/​SelfActionTest.daml Updates self-action fixtures.
daml/​governance-core-test/​daml/​Governance/​Test/​RulesTest.daml Updates rules-validation fixtures.
daml/​governance-core-test/​daml/​Governance/​Test/​ExecutionTest.daml Updates execution fixture.
daml/​governance-core-test/​daml/​Governance/​Test/​DisclosureObserversTest.daml Tests observer visibility and permissions.
daml/​governance-core-test/​daml/​Governance/​Test/​ConfirmationTest.daml Updates confirmation fixture.
daml/​governance-core-test/​daml/​Governance/​Test/​CancelActionTest.daml Tests cancellation authorization and cleanup.
daml/​governance-core-test/​daml/​Governance/​Test/​AuthorizationTest.daml Updates authorization fixtures.
daml/​governance-core-test/​daml.yaml Updates core dependency.
crates/​decman/​tests/​governance_workflows.rs Adds reward-only devnet entrypoint.
crates/​decman/​tests/​common/​phases/​mod.rs Registers deployed reward phase.
crates/​decman/​tests/​common/​phases/​distribute_dars.rs Distributes updated core package.
crates/​decman/​tests/​common/​phases/​deploy_gov_core.rs Exposes credential configuration helper.
crates/​decman/​tests/​common/​mod.rs Registers PQS helpers and service URLs.
crates/​decman/​tests/​common/​http.rs Centralizes test service URLs.
crates/​decman/​src/​workflow/​validation.rs Validates thresholds independently.
crates/​decman/​src/​workflow/​thresholds.rs Centralizes threshold construction and preservation.
crates/​decman/​src/​workflow/​onboarding/​steps/​proposals/​create.rs Builds mappings with separate thresholds.
crates/​decman/​src/​workflow/​onboarding/​config.rs Stores optional confirmation threshold.
crates/​decman/​src/​workflow/​mod.rs Exports threshold helpers.
crates/​decman/​src/​workflow/​kick/​steps/​proposals/​create.rs Preserves or caps confirmation threshold.
crates/​decman/​src/​workflow/​kick/​steps/​cache.rs Updates cache fixture.
crates/​decman/​src/​workflow/​kick/​config.rs Carries confirmation threshold through kick.
crates/​decman/​src/​workflow/​contracts/​steps/​prepare.rs Updates party fixture.
crates/​decman/​src/​workflow/​change_threshold/​steps/​proposals/​submit.rs Waits for both P2P thresholds.
crates/​decman/​src/​workflow/​change_threshold/​steps/​proposals/​create.rs Builds independent threshold changes.
crates/​decman/​src/​workflow/​change_threshold/​steps/​export_state.rs Allows confirmation-only changes.
crates/​decman/​src/​workflow/​change_threshold/​coordinator.rs Passes thresholds to submission.
crates/​decman/​src/​workflow/​change_threshold/​config.rs Stores independent threshold configuration.
crates/​decman/​src/​workflow/​add_party/​steps/​proposals/​create.rs Preserves independent confirmation policy.
crates/​decman/​src/​workflow/​add_party/​config.rs Carries confirmation threshold through additions.
crates/​decman/​src/​server/​serde_snapshots.rs Adds new action fixtures.
crates/​decman/​src/​server/​queries.rs Returns disclosure observers.
crates/​decman/​src/​server/​mod.rs Propagates invitation confirmation thresholds.
crates/​decman/​src/​server/​handlers/​workflows.rs Validates and forwards separate thresholds.
crates/​decman/​src/​server/​handlers/​parties.rs Reads and caches confirmation thresholds.
crates/​decman/​src/​server/​handlers/​invitations.rs Persists accepted confirmation thresholds.
crates/​decman/​src/​server/​handlers/​governance.rs Tests cancellation-action validation.
crates/​decman/​src/​noise/​mod.rs Logs retry exhaustion at warning level.
crates/​decman/​src/​db/​sqlite.rs Persists new threshold fields.
crates/​decman/​src/​db/​rows.rs Extends party and invitation rows.
crates/​decman/​snapshots/​dec_party_manager__server__serde_snapshots__action_types.snap Records new action wire formats.
crates/​decman/​migrations/​000024_confirmation_threshold.up.sql Adds confirmation-threshold columns.
crates/​decman/​migrations/​000024_confirmation_threshold.down.sql Removes confirmation-threshold columns.
crates/​decman/​frontend/​src/​governanceMembership.test.ts Updates governance fixture.
crates/​decman/​frontend/​src/​governanceFormat.ts Formats observers and cancellation targets.
crates/​decman/​frontend/​src/​components/​PartyDetail.tsx Displays confirmation threshold.
crates/​decman/​frontend/​src/​components/​PartyDetail.test.tsx Tests threshold display.
crates/​decman/​frontend/​src/​components/​OnboardingDialog.tsx Adds confirmation-threshold input.
crates/​decman/​frontend/​src/​components/​OnboardingDialog.test.tsx Tests onboarding threshold requests.
crates/​decman/​frontend/​src/​components/​NotificationsView.tsx Adds cancellation voting and threshold details.
crates/​decman/​frontend/​src/​components/​NotificationsView.threshold.test.tsx Tests invitation threshold display.
crates/​decman/​frontend/​src/​components/​NotificationsView.cancel.test.tsx Tests cancellation UI.
crates/​decman/​frontend/​src/​components/​GovernanceSection.tsx Adds observer-management form.
crates/​decman/​frontend/​src/​components/​GovernanceSection.disclosureObservers.test.tsx Tests observer-management requests.
crates/​decman/​frontend/​src/​components/​GovernanceRules.tsx Displays disclosure observers.
crates/​decman/​frontend/​src/​components/​GovernanceRules.test.tsx Tests observer display.
crates/​decman/​frontend/​src/​components/​ExecuteDialog.tsx Labels new governance actions.
crates/​decman/​frontend/​src/​components/​ContractsDialog.tsx Adds deployment observer field.
crates/​decman/​frontend/​src/​components/​ContractsDialog.test.tsx Tests observer deployment encoding.
crates/​decman/​frontend/​src/​components/​ChangeThresholdDialog.tsx Adds independent confirmation input.
crates/​decman/​frontend/​src/​components/​ChangeThresholdDialog.test.tsx Tests separate threshold submissions.
crates/​decman/​Cargo.toml Enables test-only Postgres support.
crates/​decman-lib/​src/​framework/​encode.rs Adds party-set encoders.
crates/​decman-lib/​src/​catalog/​interpret.rs Decodes rules observers.
crates/​decman-lib/​src/​catalog/​action.rs Encodes and decodes new self-actions.
crates/​decman-cli/​src/​ui.rs Updates fixtures for threshold fields.
crates/​decman-cli/​src/​app.rs Updates party fixture.
crates/​decman-cli/​src/​api.rs Updates party fixture.
crates/​common/​src/​types.rs Extends party and invitation types.
crates/​common/​src/​api.rs Extends threshold and observer API fields.
Cargo.lock Records Postgres feature dependency.

Review effort: Balanced

The change-threshold dialog seeded the confirmation threshold from the
namespace threshold. On a party whose two thresholds differ, opening
the dialog therefore showed the wrong value, and a namespace-only
change reset the confirmation threshold to match.

The dialog now starts at the party's current confirmation threshold
when the two differ. It follows the namespace threshold only when they
already match.

Refs #234.
The form posted the observers in the order the operator typed them,
but the backend sorts a party set before it reaches the ledger. The
audit entry and the ledger action then listed the parties in a
different order. They no longer matched, so an open vote showed up
under Completed.

The form now sorts the list the same way before it posts.

Refs #165.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Governance upgrades and topology authorization changes need human review, with live devnet and browser integration validation still outstanding.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@schronck

Copy link
Copy Markdown
Collaborator Author

Copilot review loop: disposition summary

Round Commits Findings Fixed Fixed w/ mods Won't-fix Filed issue
1 58ac006b…cddf6592 2 2 0 0 0
2 (no new commits) 0 0 0 0 0
Total 2 2 0 0 0

By category

Category Findings Fixed Fixed w/ mods Won't-fix Filed
Security 0 0 0 0 0
Functionality 2 2 0 0 0
Cosmetic 0 0 0 0 0
Total 2 2 0 0 0

The two fixes:

  • The change-threshold dialog starts at the party's current confirmation threshold when the two thresholds differ.
  • The disclosure-observers form sorts the parties before it posts, so the audit entry matches the ledger action.

Outcome: converged. Round 2 reported no open issues. Open items: none. The live devnet suite and devnet_reward_e2e have not run yet.

@scolear scolear left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at cddf659, one commit per issue.

Verdict per commit

  • 0c730b08 (refs #362): sound. The exhaustion line moves to warn and keeps peer, attempts and error as fields. The panic hook, the fatal-exit line and spawn_supervised stay at error, as the issue requires. The new test fails if the line goes back to error.
  • 02c08948 (closes #271): sound. The entrypoint is #[ignore], and it needs DECPM_IT_TARGET=devnet plus DECPM_IT_REWARD=1. run.sh checks the inputs before the build. The repo holds no hostname or secret; the PQS URL stays in the operator's environment, and errors never echo it. The closes holds for the acceptance list in the issue.
  • afabec59 (closes #234): sound. See the threshold notes below.
  • afd56d39 (closes #165): sound. Option 4 from the issue, as the field the issue asked for. See the observer notes below.
  • 58ac006b (closes #518): sound. See the cancel notes below.

Thresholds (#234)

A note on terms first. The confirmation threshold in this PR is PartyToParticipant.threshold: how many hosting participants confirm a transaction. It is not the Daml GovernanceRules.threshold. No Daml template changes in this commit, and the rules threshold is untouched.

Before this change, onboarding, kick, add-party and change-threshold wrote one value to both mappings. Now each threshold has its own field in the mapping. check_confirmation_threshold bounds the confirmation threshold to 1..=hosts in both handlers, and both dialogs check the same range. ensure_carried_from compares the plan with the chain before the coordinator signs, so a stale cache cannot reset a differing confirmation threshold. The peer check pins each threshold to the accepted invitation value, and an older peer refuses a differing pair. A party with equal thresholds carries None everywhere, so its invites and configs are what every current node expects.

One combination deserves a word in the dialog: namespace threshold 1 with a confirmation threshold above 1. One owner key then signs a topology change, and that change can rewrite the mapping, including the confirmation threshold. The namespace threshold bounds what the confirmation threshold protects. This was already possible with a single threshold of 1, so it is not a regression. A one-line hint next to the confirmation field would help an operator see it. Not a blocker.

Observers (#165)

disclosureObservers is the last field and optional, so a 0.1.0 contract reads as None and its observer set stays empty. The upgrade-check warning is expected for that reason. Observers see only GovernanceRules: members, thresholds, proposers and timeout. They do not see self-confirmations or domain proposals, which have their own stakeholders, and the test shows an observer cannot confirm. A live party gains observers only through a SelfAction_SetDisclosureObservers vote, which needs 0.2.0 vetted on every member participant. The rollout risk section says so.

Cancel (#518)

GovernanceRules_ConfirmGovernanceAction accepts the cancel vote from members and additional proposers, and execution counts member votes only. Execution fetches the proposal, checks governanceParty, and exercises GovernableAction_Cancel with the governance party's authority. A cancel that lands after execution fails at the fetch, and its confirmations expire. The reverse race is covered by the test: old confirmations cannot execute a cancelled proposal. The audit decodes SelfAction_CancelAction through from_self_proto, so the entry carries the proposal cid. Note that members can cancel any proposal of their governance, not only a stranded one. That is the governance party's authority and it matches the interface comment.

Not a blocker: Noise is on its way out (#451), so the warn change is a short-lived fix. The UI does not gate the two new actions on 0.2.0 vetting, as the body says.

@schronck
schronck merged commit 79fe8b5 into main Oct 20, 2026
11 checks passed
@schronck
schronck deleted the fix/issue-sweep-3 branch October 20, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants