fix: harden client telemetry control flow - #588
Conversation
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: xiaofan-luan The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #588 +/- ##
===========================================
+ Coverage 53.47% 87.44% +33.96%
===========================================
Files 52 379 +327
Lines 4432 15854 +11422
Branches 0 1729 +1729
===========================================
+ Hits 2370 13863 +11493
+ Misses 2062 1991 -71
🚀 New features to boost your workflow:
|
| const auto chunk = std::min(delay, kMaxWaitChunkMs); | ||
| if (condition.wait_for( | ||
| lock, std::chrono::milliseconds(static_cast<int64_t>(chunk)), | ||
| [this, observed_revision]() { return stopped || wait_revision != observed_revision; })) { |
There was a problem hiding this comment.
src/impl/ClientTelemetry.cpp line:664
Medium ---- A server-pushed heartbeat_interval_ms change cannot wake the new chunked wait: the wait predicate only reacts to stopped or wait_revision, but the push_config handler calls condition.notify_all() without bumping wait_revision. With this PR's INT64_MAX interval support chunked into 24h waits, a corrective push (e.g. back to 10s) is deferred until the current chunk elapses. Consider bumping wait_revision in the push_config handler so an interval re-push is applied on the next loop iteration.
| SaturatingWindowStart(int64_t end_time, uint64_t interval_ms) { | ||
| const auto signed_interval = static_cast<int64_t>(std::min(interval_ms, kMaxHeartbeatIntervalMs)); | ||
| if (end_time < std::numeric_limits<int64_t>::min() + signed_interval) { | ||
| return std::numeric_limits<int64_t>::min(); |
There was a problem hiding this comment.
src/impl/ClientTelemetry.cpp line:107
Low ---- SaturatingWindowStart saturates to INT64_MIN at the maximum heartbeat interval, and that value is later serialized verbatim as the snapshot "timestamp" in show_latency_history replies. A consumer expecting an epoch-millis timestamp receives -9223372036854775808. Consider capping the window start at the retained-history floor (e.g. end_time - kMaxHistoryRangeMs) so the reported timestamp stays interpretable.
| EXPECT_EQ(payload["ignored"], nlohmann::json({"ttl_seconds", "unknown_a", "unknown_b"})); | ||
| } | ||
|
|
||
| TEST(ClientTelemetryTest, ShowErrorsBoundsOversizedNonMessageFields) { |
There was a problem hiding this comment.
test/ut/TestClientTelemetry.cpp line:1309
Low ---- The show_errors truncation tests cover a single oversized field, but the selection loop that alternates across the four truncatable fields and the "still exceeds 1MB after all fields are emptied" throw path are only reachable when one retained error carries multiple oversized fields (e.g. both a huge collection and a huge error_msg). A test with two oversized fields would exercise the next-longest-field selection and the monotonic-progress fallback.
3 participants
Summary\n\n- make very large heartbeat intervals safe and cancellable across platforms\n- fence command batches across telemetry transport rebinding and canonicalize ACK IDs\n- wake unsupported backoff on channel replacement and bound show_errors truncation, including UTF-8 payloads\n\nFollow-up hardening for #585.\n\n## Validation\n\n- 854/854 unit tests passed\n- 291/291 mocked integration tests passed\n- final telemetry-focused suite: 33/33 passed\n- clang-format and git diff --check passed\n