Skip to content

fix(portfolio): route account fetches around metadata-stripping RPC nodes - #66

Merged
feruzm merged 2 commits into
mainfrom
fix/account-metadata-node-failover
Aug 11, 2026
Merged

fix(portfolio): route account fetches around metadata-stripping RPC nodes#66
feruzm merged 2 commits into
mainfrom
fix/account-metadata-node-failover

Conversation

@feruzm

@feruzm feruzm commented Aug 11, 2026

Copy link
Copy Markdown
Member

Fixes #64.

Problem

/wallet-api/portfolio-v2 with onlyEnabled: true (what the web and mobile clients send) returned zero engine tokens and zero chain wallets for accounts that had explicitly enabled them — a normal 200, correct Hive balances, nothing logged.

Token visibility is derived from the account's posting_json_metadata. Two nodes in the pool serve accounts with that field empty while every other field is correct:

node posting_json_metadata
techcoderx.com empty for every account tested
hiveapi.actifit.io empty for every account tested
api.hive.blog, api.deathwing.me, rpc.mahdiyari.info, api.openhive.network full

That is a well-formed get_accounts array, so the shape validation passes and the node is recorded healthy. Both are among the fastest in the pool, so the latency EWMA kept them ranked first and the layers were empty for essentially every request. ExtractEnabledEngineTokenSymbols returned an empty set, BuildEngineLayer(onlyEnabled: true, …) dropped every token, and ExtractExternalWallets dropped every chain wallet. The Hive layer still rendered from account fields, which is why the failure looked like "my engine balances vanished" rather than an outage.

Same class as the malformed-200 case already described in CLAUDE.md: a node that answers successfully but uselessly scores as healthy and stays ranked first.

Change

  • Drop the two metadata-stripping nodes from HiveClients.Default, so correctness here does not depend on the runtime fallback firing.
  • Add an optional soft preferResult to HiveRpcClient.Call. Unlike validateResult it is not a health signal — the node is fine for other calls — so it is neither retried nor marked unhealthy; its answer is kept and we move to the next node. GetAccounts uses it to prefer a node that actually serves metadata.
  • Bound the probing to two nodes. About an eighth of active accounts genuinely carry no metadata, and for those no node can satisfy the preference; sweeping the pool every time would multiply RPC load on a common case in order to route around a rare one. If nothing satisfies the preference, the first well-formed answer is returned — never worse than before.

Verification

Built and run locally against the real endpoint, onlyEnabled: true:

account enabled engine tokens production today this build
A 17 0 17
B 6 0 6
C 1 engine + 3 chain 0 + 0 1 + 3

Production returned zero engine tokens on 30/30 consecutive requests for these accounts before the change.

Tests

dotnet test: 98 passed. Three new cases in HiveRpcFailoverTests:

  • a metadata-stripping node is skipped in favour of one that serves it,
  • when no node serves metadata the account is still returned (an account genuinely without metadata is indistinguishable from a stripped response),
  • probing stops after two nodes.

The stub node now serves a realistic account payload including posting_json_metadata, with a ServesMetadata toggle.

Summary by CodeRabbit

  • Bug Fixes
    • Account lookups now prefer responses that include complete account metadata.
    • Account retrieval can skip nodes that strip metadata while retaining valid fallback results.
    • Improved resilience when no available node provides metadata.
    • Limited metadata probing to avoid unnecessary network requests.
    • Updated the default node pool to improve connection reliability.

…odes

Some Hive nodes serve accounts with posting_json_metadata empty while every
other field is correct. That is a well-formed get_accounts array, so the shape
validation passes and the latency EWMA keeps such a node ranked first.

Portfolio token visibility is derived entirely from that field, so the engine
allowlist came back empty and BuildEngineLayer dropped every token; the chain
layer, reading the same metadata, dropped every wallet. Balances still rendered
from account fields, so the response was a normal 200 with no error and nothing
logged, and accounts with enabled tokens showed none of them.

Two of the pool's fastest nodes behave this way, so they won the ranking and the
layers were empty for essentially every request.

- Drop those two nodes from the pool.
- Add a soft result preference to Call: unlike validateResult it is not a health
  signal, so the node is neither retried nor marked unhealthy; we keep its answer
  and move on. GetAccounts uses it to prefer a node that serves metadata.
- Bound the probing to two nodes. Roughly an eighth of active accounts genuinely
  carry no metadata and no node can satisfy the preference for them, so sweeping
  the pool would multiply RPC load on a common case to route around a rare one.
  If no node satisfies it, the first well-formed answer is returned, never worse
  than before.
@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@feruzm, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 43 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: eb799949-403a-451c-9d40-d910eece5072

📥 Commits

Reviewing files that changed from the base of the PR and between d0711f5 and 0bd4dab.

📒 Files selected for processing (2)
  • dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs
  • dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs
📝 Walkthrough

Walkthrough

HiveRpcClient now prefers account responses with non-empty metadata while retaining valid fallback responses. Account metadata probing is limited to two alternative nodes. Tests cover metadata stripping, empty metadata, and bounded probing. The default Hive node pool removes techcoderx.com.

Changes

Hive metadata failover

Layer / File(s) Summary
Soft preference in HiveRpcClient.Call
dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs
Call accepts an optional preference predicate. It probes up to two alternative nodes, retains valid non-preferred responses, and returns the fallback when no preferred response is available.
Metadata-aware account selection
dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs
GetAccounts validates array results and prefers responses with non-empty posting_json_metadata. The default node pool removes techcoderx.com.
Failover behavior coverage
dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs
Tests cover metadata-serving nodes, all nodes returning empty metadata, and the two-node probe limit.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GetAccounts
  participant HiveRpcClient
  participant HiveNode
  GetAccounts->>HiveRpcClient: Request account data with metadata preference
  HiveRpcClient->>HiveNode: Call get_accounts
  HiveNode-->>HiveRpcClient: Valid account response
  HiveRpcClient->>HiveRpcClient: Check posting_json_metadata
  HiveRpcClient->>HiveNode: Probe another node when metadata is absent
  HiveNode-->>HiveRpcClient: Preferred response or valid fallback
  HiveRpcClient-->>GetAccounts: Return account response
Loading

Possibly related issues

Possibly related PRs

Poem

A rabbit watched the Hive nodes race,
Then skipped the ones with metadata erased.
Two probes hopped across the stream,
A useful account joined the beam.
“Fallback saved!” the rabbit said,
“No empty wallets lie ahead!”

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the primary change: routing account fetches around RPC nodes that strip metadata.
Linked Issues check ✅ Passed The PR addresses #64 with metadata-aware bounded failover, removal of known problematic nodes, and targeted tests for account-fetch behavior.
Out of Scope Changes check ✅ Passed The changes remain within #64 scope and directly support metadata-aware account fetching, RPC failover, node selection, and verification.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/account-metadata-node-failover

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs`:
- Around line 246-261: The NoNodeServesMetadata_StillReturnsTheAccount test does
not verify that failover returns the first valid fallback. Add an assertion that
accounts[0]["port"] matches a.Url, while preserving the existing non-null and
name assertions; keep the RPC failover behavior covered by this test.

In `@dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs`:
- Around line 107-122: Update the GetAccounts preference-failure path around
preferResult and the node-selection state to record a method-specific negative
capability or ordering penalty for the responding node, without changing generic
health success tracking used by other RPC methods. Apply that state during
subsequent GetAccounts node selection so the metadata-stripping node is skipped
or deprioritized on the next call, and add a two-call test verifying it receives
no second account request.
- Around line 264-292: The get_accounts result validation currently accepts
scalar array entries and can prefer malformed responses. Update validateResult
and the HasAnyAccountMetadata path to require every array element to be a
JsonObject or JSON null, while preserving all-null arrays as valid
unknown-account responses; reject arrays containing scalars so failover
proceeds. Add a test covering a scalar-array response and verifying failover.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0d8ae4dc-23ad-4048-b3cb-88c738747437

📥 Commits

Reviewing files that changed from the base of the PR and between 5c1aac5 and d0711f5.

📒 Files selected for processing (2)
  • dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs
  • dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs

Comment thread dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs
Comment thread dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs
Comment thread dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review


P2 Badge Preserve the cached result when a preference probe gets an RPC error

When the first node returns a valid but unpreferred account and the next node responds with a JSON-RPC error, this unconditional rethrow discards unpreferred and makes GetAccounts fail. Before this change the first response would have succeeded, contradicting the new soft preference's stated guarantee that callers are never worse off; once haveUnpreferred is true, a probe-specific RPC error should fall back to that cached valid response rather than turning the optional probe into a request failure.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown

Greptile Summary

This PR makes account retrieval prefer well-formed RPC responses containing posting metadata while retaining the first valid response as a fallback.

  • Adds bounded, soft result preference support to RPC failover.
  • Strengthens account-array validation and metadata detection.
  • Removes two metadata-stripping nodes from the default pool.
  • Adds coverage for stripped metadata, RPC errors, malformed account entries, fallback behavior, and bounded probing.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs Adds soft metadata-aware failover, stronger account-result validation, bounded preference probing, and removes known metadata-stripping nodes.
dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs Expands failover tests to cover metadata preference, fallback preservation, malformed entries, RPC probe errors, and probe bounds.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant Client as HiveRpcClient
    participant Node1 as Preferred RPC order: Node 1
    participant Node2 as Preferred RPC order: Node 2
    Caller->>Client: GetAccounts(names)
    Client->>Node1: condenser_api.get_accounts
    Node1-->>Client: Well-formed result
    alt Metadata is present
        Client-->>Caller: Return result
    else Metadata is absent
        Client->>Node2: Probe alternative node
        alt Node 2 returns metadata
            Node2-->>Client: Preferred result
            Client-->>Caller: Return Node 2 result
        else Preference remains unsatisfied or probe errors
            Node2-->>Client: Unpreferred result or RPC error
            Client-->>Caller: Return first well-formed result
        end
    end
Loading

Reviews (2): Last reviewed commit: "review: keep the cached answer when a pr..." | Re-trigger Greptile

…ount arrays

- An RPC-level error from a node consulted only to improve on an answer we
  already hold belongs to the optional probe, not to the caller's request.
  Rethrowing it failed a call that would have succeeded without the preference,
  contradicting the "never worse off" guarantee.
- get_accounts validated only "is an array", so entries like ["invalid"] passed
  and then read as empty downstream — silently blanking portfolio token
  visibility exactly like a metadata-stripping node. Require account objects or
  JSON null (unknown account) and fail over otherwise.
- Pin the first-well-formed-answer contract in the no-metadata test.
@feruzm

feruzm commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

Triaged all four findings. Three were real and are fixed in 0bd4dab; one is declined with reasoning.

Fixed — Codex, "preserve the cached result when a preference probe gets an RPC error" (P2). Correct, and it was a regression this PR introduced: catch (RpcException) rethrew unconditionally, so if the first node answered well-formed-but-unpreferred and the probe node returned a JSON-RPC error, the whole call failed where it would previously have succeeded — directly contradicting the "never worse off" guarantee. The probe's error belongs to the probe, not to the caller's request; we now return the cached answer.

Fixed — CodeRabbit, "reject scalar entries in get_accounts results" (Major). Worth doing, and more relevant than the data-integrity framing suggests: an array like ["invalid"] passed the bare is JsonArray check, and metadata reads off it come back empty, which silently blanks portfolio token visibility in exactly the way this PR exists to fix. Now requires each entry to be an account object or JSON null (unknown account), and fails over otherwise. Note this laxness predates the PR.

Fixed — CodeRabbit, "assert the first valid fallback is returned" (Minor). Fair; the test passed whether the first or last unpreferred answer came back. Now asserts the port of the first node.

Both new behaviours are covered by tests that were confirmed to fail without the fix and pass with it. Suite: 100 passed.

Declined — CodeRabbit, "deprioritize metadata-stripping nodes for later GetAccounts calls" (Major). The observation is accurate but the remedy does not fit the problem:

  • Empty metadata is ambiguous. Roughly an eighth of active accounts genuinely have empty posting_json_metadata, so "node returned no metadata" is not evidence the node is deficient. A capability penalty recorded from a single response would demote healthy nodes constantly, and could demote the entire pool.
  • It would not reduce the cost it targets. The recurring probe cost comes from those genuinely-empty accounts, where no node can satisfy the preference — a penalty helps only when a truly deficient node is in the pool, and both known ones are removed here.
  • A sound version needs same-call proof (node A returned empty while node B returned non-empty for the same request) plus per-method ordering state. That is a real design change for a case that is currently hypothetical, and the two-node probe bound already caps the damage.

Happy to revisit if a pool node starts stripping metadata.

@feruzm

feruzm commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review
@codex review

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown

@feruzm I will review the current changes in #66, including the reported fixes and the declined finding rationale.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@feruzm
feruzm merged commit 5ed4258 into main Aug 11, 2026
5 checks passed
@feruzm
feruzm deleted the fix/account-metadata-node-failover branch August 11, 2026 11:13
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 0bd4dab7a7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

portfolio-v2: engine and chain layers silently empty when the account fetch lands on a metadata-less RPC node

1 participant