diff --git a/CHANGELOG.md b/CHANGELOG.md index 0237a08..b2eb9a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,9 +11,23 @@ * Malformed bandit definitions or leaves degrade to the rule's aggregate weights instead of raising during evaluation; reported `variationWeights` always match the weights bucketing actually used, and a single, total validity rule governs every weight vector: `getBucketRanges` normalizes vectors with negative, non-finite, boolean, non-numeric, or float-overflowing entries (not just wrong length/sum) to equal weights — never raising, even on arbitrary-precision integers, so bucket ranges can never be inverted, and bandit leaf/aggregate/override propensities always describe the vector actually used. Bandit identifiers get the same treatment: a leaf with a non-integer `leafId` is malformed (aggregate-weights fallback), and a non-integer `banditVersion` is omitted, so exposure metadata never carries invalid attribution ids into bandit training. A rule that pairs `contextualBanditRef` with explicit `ranges` buckets on the ranges (unchanged, matching the JS SDK) but drops the bandit metadata, since leaf propensities cannot describe a ranges-governed assignment. * Contextual bandit exposure metadata survives remote evaluation: `rule.tracks` results from the proxy keep `leafId`, `variationWeights`, and `banditVersion` when replayed through the tracking callback, held to the same validity rules as locally evaluated exposures (invalid identifiers or weight vectors are dropped, not forwarded). * Async evaluations that perform I/O (remote eval or a sticky bucket service) freeze every mutable evaluation input — attributes, groups, overrides, forced variations and features, nested containers included — before their first await, so mutating a `UserContext` mid-flight cannot change leaf routing, remote payloads, or forced assignments. Plain CDN evaluations never yield and skip the copy entirely; deferred callbacks get fire-time snapshots in both clients. `preload_remote_eval` takes the same call-time snapshot, so mutating the context after preloading can no longer cache one attribute state's response under another's key via the stale-while-revalidate background refresh. +* Deferred tracking: buffer experiment exposures for forwarding to a client SDK (which fires them via `setDeferredTrackingCalls` + `fireDeferredTrackingCalls`). Opt-in and independent of `on_experiment_viewed` — the buffer records first, the callback still fires. Entries use the JS SDK's `TrackingData` shape, deduped per unique assignment and JSON round-tripped at exposure time, so buffered payloads are always `json.dumps`-ready — an exposure carrying a non-JSON value (e.g. a `datetime` attribute) is dropped and logged, never the batch. Firing the forwarded `user` context on the receiving side requires JS SDK 1.7.0+. + * Sync client: `GrowthBook(defer_tracking=True)` with `get_deferred_tracking_calls()` / `clear_deferred_tracking_calls()`. + * Async client: pass a per-request `TrackingBuffer` to `eval_feature` / `run` / `is_on` / `is_off` / `get_feature_value` and read it with `buffer.get_calls()`; the caller owns the buffer, so concurrent requests never mix exposures. + +### Performance + +* Telemetry plumbing overhead on evaluations with no callbacks configured, measured with `tests/scripts/benchmark_eval_overhead.py` (500k sequential in-memory evals, best of 5): the default-value path costs ~0.13 microseconds more per evaluation than 3.0 (~1.29 -> ~1.42 us/eval, ~10%); a realistic experiment-rule path is within ~1% (~6.3 us/eval). The residual is the `EvaluationContext` carrying the callback fields — the mechanism that fixes silently-dropped prerequisite exposures — and is accepted deliberately; usage bookkeeping and the deprecated-kwarg shim are already skipped entirely when unused. ### Bug Fixes +* Experiments that decide a prerequisite feature now report telemetry: previously `eval_prereqs` dropped every callback, so those exposures were silently lost. Fixed structurally by carrying `tracking_cb` / `feature_usage_cb` / `callback_subscription` on the `EvaluationContext` (like the JS SDK's global context) so nested evaluations inherit them. As a result: + * `on_experiment_viewed` fires for prerequisite experiment assignments (including when a gate ultimately fails, matching the JS SDK). + * `on_feature_usage` fires for every feature evaluated — prerequisites included — once per key per evaluation (previously it fired only for the top-level key, on every call, with no dedupe). + * The sync client's `subscribe()` callbacks see prerequisite experiments, and its `get_all_results()` includes them. The async client's subscriptions still fire only from `run()` — like the JS multi-user client, it has no per-user assignment change-detection, so per-eval firing would repeat every subscriber callback on every request. + * `core.eval_feature` / `core.run_experiment` now read callbacks from the `EvaluationContext`; the old `tracking_cb` / `callback_subscription` keyword arguments still work but are deprecated: they are installed on the context only for the duration of that call (prerequisites inherit them) and the previous context fields are restored afterwards, matching their pre-3.1 invocation-scoped behavior. +* The tracking dedupe key is now a field tuple instead of a concatenated string, so distinct exposures cannot collide on field boundaries regardless of what the values contain. +* `Experiment.to_dict()` no longer coerces an explicit `coverage` of `0` to `1`. * `savedGroups` from a feature refresh were applied to the evaluation context one refresh late in the synchronous client. * The built-in tracking plugin now sends the exposure-time user context attributes with experiment events (previously async client events had none). diff --git a/README.md b/README.md index 3e45cc2..48fe16a 100644 --- a/README.md +++ b/README.md @@ -498,6 +498,46 @@ is_enabled = gb.is_on("my-feature") # -> Tracked automatically The tracking plugin provides batching, error handling, and works alongside your existing tracking callbacks. See the [plugin documentation](https://docs.growthbook.io/lib/python#tracking-plugins) for more details. +#### Deferred Tracking + +When your server evaluates features but the analytics context lives on the client (server-side rendering, an API returning evaluated flags), buffer the exposures instead of tracking them server-side and forward them to a client SDK, which fires them through its own tracking callback. + +On the sync client, opt in with `defer_tracking=True`: + +```python +gb = GrowthBook(attributes={"id": "user-1"}, features=features, defer_tracking=True) +gb.is_on("my-feature") + +# JSON-ready entries in the JS SDK's TrackingData shape: +# [{"experiment": {...}, "result": {...}, "user": {"attributes": {...}, "url": ""}}] +calls = gb.get_deferred_tracking_calls() +# ...send json.dumps(calls) to the client... +gb.clear_deferred_tracking_calls() +``` + +The buffer lives as long as the instance: when one instance serves many users, read and clear it per request (or use one instance per request) so exposures are never forwarded to the wrong client. + +The receiving JavaScript SDK hydrates and fires them: + +```js +gb.setDeferredTrackingCalls(callsFromServer); +gb.fireDeferredTrackingCalls(); +``` + +Forwarding the exposure-time `user` context to the tracking callback requires JS SDK 1.7.0+; older SDKs fire the exposures without the user argument. + +On the async client, pass a per-request `TrackingBuffer` — the explicit buffer is the opt-in, and because the caller owns it, concurrent requests never mix exposures: + +```python +from growthbook import TrackingBuffer + +buffer = TrackingBuffer() +await client.eval_feature("my-feature", user_context, tracking_buffer=buffer) +calls = buffer.get_calls() +``` + +Buffering is independent of `on_experiment_viewed`: when both are configured, the buffer records first and the callback still fires. Entries are deduped per unique assignment and snapshotted at exposure time, and include prerequisite and passthrough exposures. + ## Using Features There are 3 main methods for interacting with features. diff --git a/growthbook/__init__.py b/growthbook/__init__.py index 3c27558..3c947a6 100644 --- a/growthbook/__init__.py +++ b/growthbook/__init__.py @@ -13,6 +13,7 @@ JSONValue, Options, Result, + TrackingBuffer, TrackingCallback, UserContext, ) @@ -62,6 +63,7 @@ "Options", "UserContext", "FeatureRefreshStrategy", + "TrackingBuffer", # Data model "Experiment", "Result", diff --git a/growthbook/codegen.py b/growthbook/codegen.py index 9bf357f..9ede292 100644 --- a/growthbook/codegen.py +++ b/growthbook/codegen.py @@ -32,7 +32,7 @@ from typing import {typing_imports} -from growthbook import FeatureResult, GrowthBook, GrowthBookClient, UserContext +from growthbook import FeatureResult, GrowthBook, GrowthBookClient, TrackingBuffer, UserContext ''' @@ -151,7 +151,13 @@ def _client_class(feature_types: Dict[str, str], is_async: bool) -> str: name = "TypedGrowthBookClient" if is_async else "TypedGrowthBook" base = "GrowthBookClient" if is_async else "GrowthBook" prefix = "async " if is_async else "" - ctx = ", user_context: UserContext" if is_async else "" + # The async overrides must stay signature-compatible with GrowthBookClient, + # whose eval methods take a keyword-only per-request tracking_buffer. + ctx = ( + ", user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None" + if is_async + else "" + ) lines = [ f"class {name}({base}):", f' """{base} with strictly-typed feature keys (checker-only, runtime no-op)."""', @@ -256,8 +262,8 @@ def generate(payload: Dict[str, Any], payload_format: str = "auto") -> str: if any("List[" in t for t in feature_types.values()): typing_imports.append("List") typing_imports.append("Literal") - if typed_count: - typing_imports.append("Optional") + # Always needed: the async overrides' tracking_buffer is Optional. + typing_imports.append("Optional") if any("Union[" in t for t in feature_types.values()): typing_imports.append("Union") # Overloads are emitted whenever get_feature_value or eval_feature has diff --git a/growthbook/common_types.py b/growthbook/common_types.py index 352659c..b3699f3 100644 --- a/growthbook/common_types.py +++ b/growthbook/common_types.py @@ -1,5 +1,9 @@ #!/usr/bin/env python +import json +import logging +import threading +from copy import deepcopy from dataclasses import dataclass, field, replace from typing import ( TYPE_CHECKING, @@ -31,6 +35,8 @@ # runtime, so this is cycle-free. from .plugins.base import PluginLike +logger = logging.getLogger("growthbook") + # Generic feature/experiment value type. Deliberately unbounded: a JSONValue # bound would reject TypedDict/dataclass-shaped fallbacks (see JS SDK issue #1729, # where the equivalent bound was shipped and then reverted). @@ -159,7 +165,9 @@ def to_dict(self) -> Dict[str, Any]: "variations": self.variations, "weights": self.weights, "active": self.active, - "coverage": self.coverage or 1, + # None means "no coverage set" and reads as full coverage, but an + # explicit 0 must survive serialization (`or` would coerce it to 1). + "coverage": self.coverage if self.coverage is not None else 1, "condition": self.condition, "namespace": self.namespace, "force": self.force, @@ -622,6 +630,23 @@ def snapshot_user_context(user: "UserContext") -> "UserContext": ) +# Identity of one exposure: (hashAttribute, hashValue, experiment key, +# variation id). Same fields as JS getExperimentDedupeKey / Go +# TrackingData.DedupeKey, but a tuple instead of a joined string so field +# values containing a would-be delimiter can never make two distinct +# exposures collide. +TrackingDedupeKey = Tuple[str, str, str, str] + + +def tracking_dedupe_key(experiment: "Experiment[Any]", result: "Result[Any]") -> TrackingDedupeKey: + return ( + result.hashAttribute, + str(result.hashValue), + experiment.key, + str(result.variationId), + ) + + def tracking_user_context(user: "UserContext") -> "UserContext": """Exposure-time snapshot of a user context for tracking and feature-usage callbacks (JS SDK: getTrackingUserContext). @@ -632,6 +657,66 @@ def tracking_user_context(user: "UserContext") -> "UserContext": return replace(user, attributes=snapshot_attributes(user.attributes)) +class TrackingBuffer: + """Collector for deferred tracking calls: experiment exposures buffered + during evaluation so a server can forward them to a client SDK (which + fires them through its own tracking callback — JS setDeferredTrackingCalls + / fireDeferredTrackingCalls). Buffering is independent of the tracking + callback: when both are wired, the buffer is written first and the + callback still fires (Go SDK semantics). + + Entries use the JS SDK's TrackingData shape — + ``{"experiment": {...}, "result": {...}, "user": {"attributes", "url"}}`` + — JSON round-tripped at record time, so nothing the caller mutates + afterwards can reach the buffer and every stored entry is guaranteed + json.dumps-ready (an exposure carrying a non-JSON value is dropped and + logged, never the batch). Deduped by tracking_dedupe_key, first exposure + wins, insertion-ordered. Thread-safe: the async client may share one + buffer across concurrent evaluations of the same request.""" + + def __init__(self) -> None: + self._lock = threading.Lock() + self._calls: Dict[TrackingDedupeKey, Dict[str, Any]] = {} + + def record(self, experiment: "Experiment[Any]", result: "Result[Any]", user: "UserContext") -> None: + key = tracking_dedupe_key(experiment, result) + if key in self._calls: + # Unlocked read is GIL-safe; the setdefault below closes the race + # (first exposure wins). This only skips building the entry twice. + return + try: + # JSON round-trip at record time (Go SDK: detachTrackingData). + # It snapshots — to_dict() aliases nested mutables (variations, + # condition, attribute values), so a shallow entry would let later + # caller mutations reach the buffer — AND it guarantees the + # json.dumps-ready contract per entry, so one exposure carrying a + # non-JSON value (datetime, NaN, custom object) is dropped here + # with a log instead of poisoning the whole forwarded batch at the + # caller's json.dumps. Telemetry must never break evaluation. + entry = json.loads(json.dumps({ + "experiment": experiment.to_dict(), + "result": result.to_dict(), + "user": {"attributes": user.attributes, "url": user.url}, + }, allow_nan=False)) + except Exception: + logger.exception("Dropped deferred tracking call that cannot be JSON-serialized") + return + with self._lock: + self._calls.setdefault(key, entry) + + def get_calls(self) -> List[Dict[str, Any]]: + """Buffered exposures as detached copies (mutating them cannot corrupt + the buffer, which get_calls leaves intact for a later read), ready for + json.dumps. Non-destructive; pair with clear().""" + with self._lock: + snapshot = list(self._calls.values()) + return deepcopy(snapshot) + + def clear(self) -> None: + with self._lock: + self._calls.clear() + + @dataclass class Options: url: Optional[str] = None @@ -690,6 +775,24 @@ class EvaluationContext: # directly, letting the async client schedule persistence off the event loop. # None (the default) preserves the sync client's direct-call behavior. save_sticky_bucket_doc: Optional[Callable[[Dict[str, Any]], None]] = None + # Client callbacks carried on the context rather than threaded through as + # function parameters, so nested evaluations — prerequisites especially — + # inherit them automatically (JS SDK: ctx.global.trackingCallback). A + # recursive call site that forgets a parameter silently drops telemetry; + # a shared context field cannot be forgotten. + tracking_cb: Optional[Callable[["Experiment[Any]", "Result[Any]", UserContext], None]] = None + callback_subscription: Optional[Callable[["Experiment[Any]", "Result[Any]"], None]] = None + feature_usage_cb: Optional[Callable[[str, "FeatureResult[Any]", UserContext], None]] = None + # Feature keys already reported through feature_usage_cb during this + # evaluation — one usage event per key per top-level eval call, however + # many times a prerequisite chain re-visits it. Allocated lazily by + # _report_feature_usage: contexts are built on every evaluation, and evals + # with no usage callback must not pay for the set. + reported_features: Optional[Set[str]] = None + # When set, every exposure produced by this evaluation (prerequisites and + # passthrough included) is recorded here, before and independent of + # tracking_cb. + tracking_buffer: Optional[TrackingBuffer] = None # --------------------------------------------------------------------------- diff --git a/growthbook/core.py b/growthbook/core.py index 5ec7431..01254fe 100644 --- a/growthbook/core.py +++ b/growthbook/core.py @@ -3,6 +3,7 @@ import math import re import json +import warnings from functools import lru_cache from urllib.parse import urlparse, parse_qs @@ -580,14 +581,16 @@ def getBucketRanges( def _fire_rule_tracks( rule_tracks: List[Dict[str, Any]], eval_context: EvaluationContext, - tracking_cb: Optional[Callable[[Experiment[Any], Result[Any], UserContext], None]], ) -> None: - """Fire tracking_cb for each deferred experiment-tracking entry attached to - a remote-eval force rule. The proxy server evaluates experiments server-side - and emits the resulting (experiment, result) pairs here so the SDK can still - drive its tracking pipeline. Mirrors the JS SDK behavior in + """Report each pre-evaluated experiment-tracking entry attached to a + remote-eval force rule through the context's tracking buffer and tracking + callback. The proxy server evaluates experiments server-side and emits the + resulting (experiment, result) pairs here so the SDK can still drive its + tracking pipeline. Mirrors the JS SDK behavior in packages/sdk-js/src/core.ts (`if (rule.tracks) ...`).""" - if not rule_tracks or not tracking_cb: + if not rule_tracks or ( + eval_context.tracking_cb is None and eval_context.tracking_buffer is None + ): return for entry in rule_tracks: exp_data = entry.get("experiment") or {} @@ -635,7 +638,7 @@ def _fire_rule_tracks( variationWeights=variation_weights, banditVersion=bandit_version, ) - tracking_cb(experiment, result, eval_context.user) + _report_exposure(experiment, result, eval_context) except Exception: logger.exception("Failed to fire rule.tracks tracking event") @@ -750,17 +753,99 @@ def _build_contextual_bandit_experiment( experiment.contextualBandit = cb +def _report_exposure( + experiment: Experiment[Any], + result: Result[Any], + evalContext: EvaluationContext, +) -> None: + """Report one experiment exposure: deferred tracking buffer first (it + snapshots for itself, so a failing callback can never lose it), then the + tracking callback. Buffering is independent of the callback — both fire.""" + if evalContext.tracking_buffer is not None: + evalContext.tracking_buffer.record(experiment, result, evalContext.user) + if evalContext.tracking_cb: + evalContext.tracking_cb(experiment, result, evalContext.user) + + +def _report_feature_usage( + key: str, + result: FeatureResult[Any], + evalContext: EvaluationContext, +) -> None: + """Fire the context's feature-usage callback, once per feature key per + evaluation, however many times a prerequisite chain re-visits it.""" + cb = evalContext.feature_usage_cb + if cb is None: + return + reported = evalContext.reported_features + if reported is None: + reported = evalContext.reported_features = set() + if key in reported: + return + reported.add(key) + cb(key, result, evalContext.user) + + +def _warn_legacy_callback(name: str) -> None: + warnings.warn( + f"the {name} argument is deprecated; set EvaluationContext.{name}", + DeprecationWarning, + stacklevel=4, + ) + + def eval_feature( key: str, evalContext: Optional[EvaluationContext] = None, callback_subscription: Optional[Callable[[Experiment[Any], Result[Any]], None]] = None, - tracking_cb: Optional[Callable[[Experiment[Any], Result[Any], UserContext], None]] = None + tracking_cb: Optional[Callable[[Experiment[Any], Result[Any], UserContext], None]] = None, ) -> FeatureResult[Any]: - """Core feature evaluation logic as a standalone function""" + """Core feature evaluation logic as a standalone function. + + Tracking, subscription, and feature-usage callbacks are read from the + EvaluationContext so recursive evaluations (prerequisites) report through + them too. The callback keyword arguments are a deprecated, + invocation-scoped compatibility shim: they are installed on the context + for the duration of this call (so prerequisites inherit them) and the + previous context fields are restored afterwards, matching their pre-3.1 + per-call behavior.""" if evalContext is None: raise ValueError("evalContext is required - eval_feature") - + + if callback_subscription is None and tracking_cb is None: + # Inlined _eval_feature_and_report: this is the hot path, and the + # extra call frame is measurable at millions of evals per second. + result = _eval_feature(key, evalContext) + if evalContext.feature_usage_cb is not None: + _report_feature_usage(key, result, evalContext) + return result + + previous = (evalContext.callback_subscription, evalContext.tracking_cb) + if callback_subscription is not None: + _warn_legacy_callback("callback_subscription") + evalContext.callback_subscription = callback_subscription + if tracking_cb is not None: + _warn_legacy_callback("tracking_cb") + evalContext.tracking_cb = tracking_cb + try: + return _eval_feature_and_report(key, evalContext) + finally: + evalContext.callback_subscription, evalContext.tracking_cb = previous + + +def _eval_feature_and_report(key: str, evalContext: EvaluationContext) -> FeatureResult[Any]: + result = _eval_feature(key, evalContext) + if evalContext.feature_usage_cb is not None: + _report_feature_usage(key, result, evalContext) + return result + + +def _eval_feature( + key: str, + evalContext: EvaluationContext, +) -> FeatureResult[Any]: + if key not in evalContext.global_ctx.features: logger.warning("Unknown feature %s", key) return FeatureResult(None, "unknownFeature") @@ -824,7 +909,7 @@ def eval_feature( # remote-eval proxy (no-op when the rule was not produced by remote # evaluation). if rule.tracks: - _fire_rule_tracks(rule.tracks, evalContext, tracking_cb) + _fire_rule_tracks(rule.tracks, evalContext) return FeatureResult(rule.force, "force", ruleId=rule.id) # Contextual bandit rules carry their variations under @@ -863,7 +948,7 @@ def eval_feature( if rule.contextualBanditRef: _build_contextual_bandit_experiment(exp, rule.contextualBanditRef, key, evalContext) - result = run_experiment(experiment=exp, featureId=key, evalContext=evalContext, tracking_cb=tracking_cb) + result = _run_experiment(exp, key, evalContext) # Bandit metadata is only meaningful for real hashed exposures; strip # it from the experiment for forced/QA/coverage-miss outcomes so it @@ -871,8 +956,8 @@ def eval_feature( if exp.contextualBandit is not None and not (result.hashUsed and result.inExperiment): exp.contextualBandit = None - if callback_subscription: - callback_subscription(exp, result) + if evalContext.callback_subscription: + evalContext.callback_subscription(exp, result) if not result.inExperiment: logger.debug( @@ -903,7 +988,7 @@ def eval_prereqs(parentConditions: List[Dict[str, Any]], evalContext: Evaluation if parent_id is None: continue # Skip if no valid ID - parentRes = eval_feature(key=parent_id, evalContext=evalContext) + parentRes = _eval_feature_and_report(parent_id, evalContext) if parentRes.source == "cyclicPrerequisite": return "cyclic" @@ -1000,10 +1085,26 @@ def _get_sticky_bucket_variation( def run_experiment(experiment: Experiment[Any], featureId: Optional[str] = None, evalContext: Optional[EvaluationContext] = None, - tracking_cb: Optional[Callable[[Experiment[Any], Result[Any], UserContext], None]] = None + tracking_cb: Optional[Callable[[Experiment[Any], Result[Any], UserContext], None]] = None, ) -> Result[Any]: if evalContext is None: raise ValueError("evalContext is required - run_experiment") + if tracking_cb is None: + return _run_experiment(experiment, featureId, evalContext) + # Deprecated, invocation-scoped compatibility shim (see eval_feature). + _warn_legacy_callback("tracking_cb") + previous = evalContext.tracking_cb + evalContext.tracking_cb = tracking_cb + try: + return _run_experiment(experiment, featureId, evalContext) + finally: + evalContext.tracking_cb = previous + + +def _run_experiment(experiment: Experiment[Any], + featureId: Optional[str], + evalContext: EvaluationContext, + ) -> Result[Any]: # 1. If experiment has less than 2 variations, return immediately if len(experiment.variations) < 2: logger.warning( @@ -1264,13 +1365,13 @@ def run_experiment(experiment: Experiment[Any], "assignment doc was not persisted" ) - # 14. Fire the tracking callback if set. The clients' _track wrappers - # snapshot the user context (tracking_user_context) before invoking the - # user's callback, so the logged attributes are exactly the ones used - # for bucketing; snapshotting there instead of here keeps evals - # allocation-free when no tracking callback is configured. - if tracking_cb: - tracking_cb(experiment, result, evalContext.user) + # 14. Report the exposure (see _report_exposure: buffer first, then the + # tracking callback). The clients' _track wrappers snapshot the user + # context (tracking_user_context) before invoking the user's callback, so + # the logged attributes are exactly the ones used for bucketing; + # snapshotting there instead of here keeps evals allocation-free when no + # tracking callback is configured. + _report_exposure(experiment, result, evalContext) # 15. Return the result logger.debug("Assigned variation %d in experiment %s", assigned, experiment.key) diff --git a/growthbook/growthbook.py b/growthbook/growthbook.py index ba8c9d5..ded32bc 100644 --- a/growthbook/growthbook.py +++ b/growthbook/growthbook.py @@ -36,8 +36,11 @@ AbstractStickyBucketService, AbstractAsyncStickyBucketService, FeatureRule, + TrackingBuffer, + TrackingDedupeKey, build_remote_eval_payload, features_from_dict, + tracking_dedupe_key, tracking_user_context, validate_remote_eval_options, ) @@ -56,7 +59,7 @@ # Only present in urllib3 2.x; the runtime dependency allows 1.x too. from urllib3.response import BaseHTTPResponse -from .core import _getHashValue, eval_feature as core_eval_feature, run_experiment +from .core import _eval_feature_and_report, _getHashValue, run_experiment logger = logging.getLogger("growthbook") @@ -933,6 +936,11 @@ def __init__( # included) so every existing positional call site keeps its meaning. contextual_bandits: Optional[Dict[str, Any]] = None, contextualBandits: Optional[Dict[str, Any]] = None, + # New in 3.1.0, after the deprecated block so every earlier positional + # index is preserved. Opt-in: buffer every exposure for forwarding to + # a client SDK (see get_deferred_tracking_calls). Independent of + # on_experiment_viewed. + defer_tracking: bool = False, ) -> None: remote_eval = remote_eval or remoteEval saved_groups = saved_groups if saved_groups is not None else savedGroups @@ -993,7 +1001,8 @@ def __init__( self._forcedVariations = forced_variations if forced_variations is not None else (forcedVariations if forcedVariations is not None else {}) self._forcedFeatures: Dict[str, Any] = forced_features or {} - self._tracked: Dict[str, Any] = {} + self._tracked: Dict[TrackingDedupeKey, Any] = {} + self._deferred_buffer: Optional[TrackingBuffer] = TrackingBuffer() if defer_tracking else None self._assigned: Dict[str, Any] = {} self._subscriptions: Set[Callable[[Experiment[Any], Result[Any]], None]] = set() self._is_updating_features = False @@ -1332,6 +1341,8 @@ def destroy(self, timeout: float = 10) -> None: try: self._subscriptions.clear() self._tracked.clear() + if self._deferred_buffer: + self._deferred_buffer.clear() self._assigned.clear() self._trackingCallback = None self._featureUsageCallback = None @@ -1455,7 +1466,14 @@ def _build_eval_context(self) -> EvaluationContext: return EvaluationContext( global_ctx = self._global_ctx, user = self._user_ctx, - stack = StackContext(evaluated_features=set()) + stack = StackContext(evaluated_features=set()), + # Wired only when a consumer exists (contexts are per-eval, so a + # callback installed later — e.g. by a plugin — is still picked + # up), letting core skip dead work like rule.tracks hydration. + tracking_cb = self._track if self._trackingCallback else None, + callback_subscription = self._fireSubscriptions, + feature_usage_cb = self._feature_usage if self._featureUsageCallback else None, + tracking_buffer = self._deferred_buffer, ) def _get_eval_context(self) -> EvaluationContext: @@ -1465,18 +1483,19 @@ def _get_eval_context(self) -> EvaluationContext: return self._build_eval_context() def eval_feature(self, key: str) -> FeatureResult[Any]: - result = core_eval_feature(key=key, - evalContext=self._get_eval_context(), - callback_subscription=self._fireSubscriptions, - tracking_cb=self._track - ) - # Call feature usage callback if provided - if self._featureUsageCallback: - try: - self._featureUsageCallback(key, result, tracking_user_context(self._user_ctx)) - except Exception: - pass - return result + # The internal entry point skips the public wrapper's deprecated-kwarg + # shim — the callbacks are already wired on the context. + return _eval_feature_and_report(key, self._get_eval_context()) + + def _feature_usage(self, key: str, result: FeatureResult[Any], user_context: UserContext) -> None: + if not self._featureUsageCallback: + return + try: + # Snapshot so the logged attributes are exactly the ones used + # for the evaluation, even if the caller mutates them afterwards. + self._featureUsageCallback(key, result, tracking_user_context(user_context)) + except Exception: + pass @deprecated("getAllResults is deprecated, use get_all_results instead") def getAllResults(self) -> Dict[str, Dict[str, Any]]: @@ -1504,11 +1523,8 @@ def _fireSubscriptions(self, experiment: Experiment[Any], result: Result[Any]) - pass def run(self, experiment: Experiment[T]) -> Result[T]: - # result = self._run(experiment) result = run_experiment(experiment=experiment, - evalContext=self._get_eval_context(), - tracking_cb=self._track - ) + evalContext=self._get_eval_context()) self._fireSubscriptions(experiment, result) return result @@ -1517,15 +1533,26 @@ def subscribe(self, callback: Callable[[Experiment[Any], Result[Any]], None]) -> self._subscriptions.add(callback) return lambda: self._subscriptions.remove(callback) + def get_deferred_tracking_calls(self) -> List[Dict[str, Any]]: + """Exposures buffered by defer_tracking=True, as JSON-ready dicts in + the JS SDK's TrackingData shape ({experiment, result, user}) — forward + them to a client SDK's setDeferredTrackingCalls. Non-destructive; + call clear_deferred_tracking_calls() once they have been handed off. + + When one instance serves many users (set_attributes per request), the + buffer spans all of them: read and clear it per request, or use one + instance per request, so exposures are never forwarded to the wrong + client.""" + return self._deferred_buffer.get_calls() if self._deferred_buffer else [] + + def clear_deferred_tracking_calls(self) -> None: + if self._deferred_buffer: + self._deferred_buffer.clear() + def _track(self, experiment: Experiment[Any], result: Result[Any], user_context: UserContext) -> None: if not self._trackingCallback: return None - key = ( - result.hashAttribute - + str(result.hashValue) - + experiment.key - + str(result.variationId) - ) + key = tracking_dedupe_key(experiment, result) if not self._tracked.get(key): try: # Snapshot so the logged attributes are exactly the ones used diff --git a/growthbook/growthbook_client.py b/growthbook/growthbook_client.py index 8fe4fa7..e19209b 100644 --- a/growthbook/growthbook_client.py +++ b/growthbook/growthbook_client.py @@ -19,7 +19,7 @@ from growthbook import FeatureRepository, feature_repo from contextlib import asynccontextmanager -from .core import eval_feature as core_eval_feature, run_experiment +from .core import _eval_feature_and_report, run_experiment from .common_types import ( T, AsyncEventLogger, @@ -37,6 +37,9 @@ build_remote_eval_payload, features_from_dict, snapshot_user_context, + TrackingBuffer, + TrackingDedupeKey, + tracking_dedupe_key, tracking_user_context, validate_remote_eval_options, ) @@ -645,7 +648,7 @@ def __init__( ) # Thread-safe tracking state - self._tracked: Dict[str, bool] = {} # Access only within async context + self._tracked: Dict[TrackingDedupeKey, bool] = {} # Access only within async context self._tracked_lock = threading.Lock() # Thread-safe subscription management @@ -759,12 +762,7 @@ def _track(self, experiment: Experiment[Any], result: Result[Any], user_context: return # Create unique key for this tracking event - key = ( - result.hashAttribute - + str(result.hashValue) - + experiment.key - + str(result.variationId) - ) + key = tracking_dedupe_key(experiment, result) with self._tracked_lock: if not self._tracked.get(key): @@ -794,7 +792,7 @@ def _track(self, experiment: Experiment[Any], result: Result[Any], user_context: except Exception: logger.exception("Error in tracking callback") - def _untrack(self, key: str) -> None: + def _untrack(self, key: TrackingDedupeKey) -> None: with self._tracked_lock: self._tracked.pop(key, None) @@ -1221,7 +1219,29 @@ async def __aexit__( ) -> None: await self.close() - async def create_evaluation_context(self, user_context: UserContext) -> EvaluationContext: + def _context_callbacks(self, tracking_buffer: Optional[TrackingBuffer]) -> Dict[str, Any]: + """Callback/buffer fields for a new EvaluationContext, shared by both + construction branches so neither can drift and silently drop telemetry. + + callback_subscription is intentionally NOT wired: subscriptions on the + multi-user client fire only from run() (like the JS multi-user client, + which has no eval-time subscriptions). Firing them per eval_feature + would spam subscribers, since unlike the single-user sync client there + is no per-user assignment change-detection here.""" + return { + # Wired only when a consumer exists (contexts are per-eval, so a + # callback installed later — e.g. by a plugin — is still picked + # up), letting core skip dead work like rule.tracks hydration. + "tracking_cb": self._track if self.options.on_experiment_viewed else None, + "feature_usage_cb": self._feature_usage if self.options.on_feature_usage else None, + "tracking_buffer": tracking_buffer, + } + + async def create_evaluation_context( + self, + user_context: UserContext, + tracking_buffer: Optional[TrackingBuffer] = None, + ) -> EvaluationContext: """Create evaluation context for feature evaluation""" # Capture the snapshot once; feature updates swap the reference, so # this evaluation runs against a consistent view without locking. @@ -1263,6 +1283,7 @@ async def create_evaluation_context(self, user_context: UserContext) -> Evaluati user=user_context, global_ctx=global_ctx, stack=StackContext(evaluated_features=set()), + **self._context_callbacks(tracking_buffer), ) # Get sticky bucket assignments if needed @@ -1285,50 +1306,79 @@ async def create_evaluation_context(self, user_context: UserContext) -> Evaluati self._schedule_sticky_bucket_save if self.options.sticky_bucket_service else None ), + **self._context_callbacks(tracking_buffer), ) - async def eval_feature(self, key: str, user_context: UserContext) -> FeatureResult[Any]: + async def eval_feature( + self, + key: str, + user_context: UserContext, + *, + tracking_buffer: Optional[TrackingBuffer] = None, + ) -> FeatureResult[Any]: """Evaluate a feature. Lock-free: the evaluation context captures an immutable feature snapshot, so concurrent evaluations never contend - with each other or with feature updates.""" - context = await self.create_evaluation_context(user_context) - result = core_eval_feature(key=key, evalContext=context, tracking_cb=self._track) - # Call feature usage callback if provided - if self.options.on_feature_usage: - try: - self._run_user_callback( - self.options.on_feature_usage, - # Fire-time snapshot: context.user may be the caller's own - # context (plain CDN evals skip the boundary copy), and - # this callback can run deferred on the event loop. - (key, result, tracking_user_context(context.user)), - "feature usage", - ) - except Exception: - logger.exception("Error in feature usage callback") - return result + with each other or with feature updates. + + Pass a per-request TrackingBuffer to collect the exposures this + evaluation produces (deferred tracking); the caller owns the buffer, + so requests never mix.""" + context = await self.create_evaluation_context(user_context, tracking_buffer) + # The internal entry point skips the public wrapper's deprecated-kwarg + # shim — the callbacks are already wired on the context. + return _eval_feature_and_report(key, context) + + def _feature_usage(self, key: str, result: FeatureResult[Any], user_context: UserContext) -> None: + if not self.options.on_feature_usage: + return + try: + self._run_user_callback( + self.options.on_feature_usage, + # Fire-time snapshot: core passes evalContext.user, which may + # be the caller's own context (plain CDN evals skip the + # boundary copy), and this callback can run deferred on the + # event loop. + (key, result, tracking_user_context(user_context)), + "feature usage", + ) + except Exception: + logger.exception("Error in feature usage callback") - async def is_on(self, key: str, user_context: UserContext) -> bool: + async def is_on( + self, key: str, user_context: UserContext, *, + tracking_buffer: Optional[TrackingBuffer] = None, + ) -> bool: """Check if a feature is enabled with proper async context management""" - result = await self.eval_feature(key, user_context) + result = await self.eval_feature(key, user_context, tracking_buffer=tracking_buffer) return result.on - async def is_off(self, key: str, user_context: UserContext) -> bool: + async def is_off( + self, key: str, user_context: UserContext, *, + tracking_buffer: Optional[TrackingBuffer] = None, + ) -> bool: """Check if a feature is set to off with proper async context management""" - result = await self.eval_feature(key, user_context) + result = await self.eval_feature(key, user_context, tracking_buffer=tracking_buffer) return result.off - async def get_feature_value(self, key: str, fallback: T, user_context: UserContext) -> T: - result = await self.eval_feature(key, user_context) + async def get_feature_value( + self, key: str, fallback: T, user_context: UserContext, *, + tracking_buffer: Optional[TrackingBuffer] = None, + ) -> T: + result = await self.eval_feature(key, user_context, tracking_buffer=tracking_buffer) return cast(T, result.value) if result.value is not None else fallback - async def run(self, experiment: Experiment[T], user_context: UserContext) -> Result[T]: + async def run( + self, + experiment: Experiment[T], + user_context: UserContext, + *, + tracking_buffer: Optional[TrackingBuffer] = None, + ) -> Result[T]: """Run experiment with tracking. Lock-free, same as eval_feature.""" - context = await self.create_evaluation_context(user_context) + context = await self.create_evaluation_context(user_context, tracking_buffer) result = run_experiment( experiment=experiment, evalContext=context, - tracking_cb=self._track ) # Fire subscriptions synchronously self._fire_subscriptions(experiment, result) diff --git a/tests/cases.json b/tests/cases.json index dcd1db1..5d4e57f 100644 --- a/tests/cases.json +++ b/tests/cases.json @@ -13831,5 +13831,268 @@ } ] ] + ], + + "trackingCalls": [ + [ + "experiment rule assignment fires the tracking callback", + { + "attributes": { + "id": "user-1" + }, + "features": { + "parent": { + "defaultValue": "off", + "rules": [ + { + "key": "parent-exp", + "coverage": 1, + "variations": [ + "on", + "off" + ], + "weights": [ + 1, + 0 + ] + } + ] + } + } + }, + "parent", + [ + [ + "parent-exp", + 0 + ] + ], + [ + "parent" + ] + ], + [ + "prerequisite experiment assignment is tracked", + { + "attributes": { + "id": "user-1" + }, + "features": { + "parent": { + "defaultValue": "off", + "rules": [ + { + "key": "parent-exp", + "coverage": 1, + "variations": [ + "on", + "off" + ], + "weights": [ + 1, + 0 + ] + } + ] + }, + "child": { + "defaultValue": "child-default", + "rules": [ + { + "parentConditions": [ + { + "id": "parent", + "condition": { + "value": "on" + } + } + ], + "force": "child-on" + } + ] + } + } + }, + "child", + [ + [ + "parent-exp", + 0 + ] + ], + [ + "parent", + "child" + ] + ], + [ + "prerequisite tracked even when the gate fails", + { + "attributes": { + "id": "user-1" + }, + "features": { + "parent": { + "defaultValue": "off", + "rules": [ + { + "key": "parent-exp", + "coverage": 1, + "variations": [ + "on", + "off" + ], + "weights": [ + 1, + 0 + ] + } + ] + }, + "child-gated": { + "defaultValue": "child-default", + "rules": [ + { + "parentConditions": [ + { + "id": "parent", + "condition": { + "value": "nope" + }, + "gate": true + } + ], + "force": "never" + } + ] + } + } + }, + "child-gated", + [ + [ + "parent-exp", + 0 + ] + ], + [ + "parent", + "child-gated" + ] + ], + [ + "prerequisite consulted by several rules is tracked once", + { + "attributes": { + "id": "user-1" + }, + "features": { + "parent": { + "defaultValue": "off", + "rules": [ + { + "key": "parent-exp", + "coverage": 1, + "variations": [ + "on", + "off" + ], + "weights": [ + 1, + 0 + ] + } + ] + }, + "child-twice": { + "defaultValue": "child-default", + "rules": [ + { + "parentConditions": [ + { + "id": "parent", + "condition": { + "value": "nope" + } + } + ], + "force": "r1" + }, + { + "parentConditions": [ + { + "id": "parent", + "condition": { + "value": "on" + } + } + ], + "force": "r2" + } + ] + } + } + }, + "child-twice", + [ + [ + "parent-exp", + 0 + ] + ], + [ + "parent", + "child-twice" + ] + ], + [ + "passthrough assignment is tracked before falling through", + { + "attributes": { + "id": "user-1" + }, + "features": { + "ramped": { + "defaultValue": "default", + "rules": [ + { + "key": "ramp", + "coverage": 1, + "variations": [ + "treatment", + "default" + ], + "weights": [ + 0, + 1 + ], + "meta": [ + { + "key": "0" + }, + { + "key": "1", + "passthrough": true + } + ] + }, + { + "force": "fallthrough" + } + ] + } + } + }, + "ramped", + [ + [ + "ramp", + 1 + ] + ], + [ + "ramped" + ] + ] ] -} \ No newline at end of file +} diff --git a/tests/codegen/expected_output.py b/tests/codegen/expected_output.py index 4c86480..8345b9f 100644 --- a/tests/codegen/expected_output.py +++ b/tests/codegen/expected_output.py @@ -5,7 +5,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Union, overload -from growthbook import FeatureResult, GrowthBook, GrowthBookClient, UserContext +from growthbook import FeatureResult, GrowthBook, GrowthBookClient, TrackingBuffer, UserContext FeatureKey = Literal['banner_text', 'dark_mode', 'donut_price', 'hero_layout', 'max_items', 'meal_overrides', 'promo_banner', 'recent_tabs'] @@ -81,61 +81,61 @@ class TypedGrowthBookClient(GrowthBookClient): if TYPE_CHECKING: @overload # type: ignore[override] - async def get_feature_value(self, key: Literal['banner_text'], fallback: None, user_context: UserContext) -> Optional[str]: ... + async def get_feature_value(self, key: Literal['banner_text'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[str]: ... @overload - async def get_feature_value(self, key: Literal['banner_text'], fallback: str, user_context: UserContext) -> str: ... + async def get_feature_value(self, key: Literal['banner_text'], fallback: str, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> str: ... @overload - async def get_feature_value(self, key: Literal['dark_mode'], fallback: None, user_context: UserContext) -> Optional[bool]: ... + async def get_feature_value(self, key: Literal['dark_mode'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[bool]: ... @overload - async def get_feature_value(self, key: Literal['dark_mode'], fallback: bool, user_context: UserContext) -> bool: ... + async def get_feature_value(self, key: Literal['dark_mode'], fallback: bool, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> bool: ... @overload - async def get_feature_value(self, key: Literal['donut_price'], fallback: None, user_context: UserContext) -> Optional[Union[int, float]]: ... + async def get_feature_value(self, key: Literal['donut_price'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[Union[int, float]]: ... @overload - async def get_feature_value(self, key: Literal['donut_price'], fallback: Union[int, float], user_context: UserContext) -> Union[int, float]: ... + async def get_feature_value(self, key: Literal['donut_price'], fallback: Union[int, float], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Union[int, float]: ... @overload - async def get_feature_value(self, key: Literal['hero_layout'], fallback: None, user_context: UserContext) -> Optional[str]: ... + async def get_feature_value(self, key: Literal['hero_layout'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[str]: ... @overload - async def get_feature_value(self, key: Literal['hero_layout'], fallback: str, user_context: UserContext) -> str: ... + async def get_feature_value(self, key: Literal['hero_layout'], fallback: str, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> str: ... @overload - async def get_feature_value(self, key: Literal['max_items'], fallback: None, user_context: UserContext) -> Optional[Union[int, float]]: ... + async def get_feature_value(self, key: Literal['max_items'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[Union[int, float]]: ... @overload - async def get_feature_value(self, key: Literal['max_items'], fallback: Union[int, float], user_context: UserContext) -> Union[int, float]: ... + async def get_feature_value(self, key: Literal['max_items'], fallback: Union[int, float], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Union[int, float]: ... @overload - async def get_feature_value(self, key: Literal['meal_overrides'], fallback: None, user_context: UserContext) -> Optional[Dict[str, Any]]: ... + async def get_feature_value(self, key: Literal['meal_overrides'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[Dict[str, Any]]: ... @overload - async def get_feature_value(self, key: Literal['meal_overrides'], fallback: Dict[str, Any], user_context: UserContext) -> Dict[str, Any]: ... + async def get_feature_value(self, key: Literal['meal_overrides'], fallback: Dict[str, Any], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Dict[str, Any]: ... @overload - async def get_feature_value(self, key: Literal['promo_banner'], fallback: None, user_context: UserContext) -> Optional[str]: ... + async def get_feature_value(self, key: Literal['promo_banner'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[str]: ... @overload - async def get_feature_value(self, key: Literal['promo_banner'], fallback: str, user_context: UserContext) -> str: ... + async def get_feature_value(self, key: Literal['promo_banner'], fallback: str, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> str: ... @overload - async def get_feature_value(self, key: Literal['recent_tabs'], fallback: None, user_context: UserContext) -> Optional[List[Any]]: ... + async def get_feature_value(self, key: Literal['recent_tabs'], fallback: None, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Optional[List[Any]]: ... @overload - async def get_feature_value(self, key: Literal['recent_tabs'], fallback: List[Any], user_context: UserContext) -> List[Any]: ... - async def get_feature_value(self, key: Any, fallback: Any, user_context: UserContext) -> Any: + async def get_feature_value(self, key: Literal['recent_tabs'], fallback: List[Any], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> List[Any]: ... + async def get_feature_value(self, key: Any, fallback: Any, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> Any: raise NotImplementedError @overload # type: ignore[override] - async def eval_feature(self, key: Literal['banner_text'], user_context: UserContext) -> "FeatureResult[str]": ... + async def eval_feature(self, key: Literal['banner_text'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[str]": ... @overload - async def eval_feature(self, key: Literal['dark_mode'], user_context: UserContext) -> "FeatureResult[bool]": ... + async def eval_feature(self, key: Literal['dark_mode'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[bool]": ... @overload - async def eval_feature(self, key: Literal['donut_price'], user_context: UserContext) -> "FeatureResult[Union[int, float]]": ... + async def eval_feature(self, key: Literal['donut_price'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[Union[int, float]]": ... @overload - async def eval_feature(self, key: Literal['hero_layout'], user_context: UserContext) -> "FeatureResult[str]": ... + async def eval_feature(self, key: Literal['hero_layout'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[str]": ... @overload - async def eval_feature(self, key: Literal['max_items'], user_context: UserContext) -> "FeatureResult[Union[int, float]]": ... + async def eval_feature(self, key: Literal['max_items'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[Union[int, float]]": ... @overload - async def eval_feature(self, key: Literal['meal_overrides'], user_context: UserContext) -> "FeatureResult[Dict[str, Any]]": ... + async def eval_feature(self, key: Literal['meal_overrides'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[Dict[str, Any]]": ... @overload - async def eval_feature(self, key: Literal['promo_banner'], user_context: UserContext) -> "FeatureResult[str]": ... + async def eval_feature(self, key: Literal['promo_banner'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[str]": ... @overload - async def eval_feature(self, key: Literal['recent_tabs'], user_context: UserContext) -> "FeatureResult[List[Any]]": ... - async def eval_feature(self, key: Any, user_context: UserContext) -> "FeatureResult[Any]": + async def eval_feature(self, key: Literal['recent_tabs'], user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[List[Any]]": ... + async def eval_feature(self, key: Any, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> "FeatureResult[Any]": raise NotImplementedError - async def is_on(self, key: FeatureKey, user_context: UserContext) -> bool: # type: ignore[override] + async def is_on(self, key: FeatureKey, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> bool: # type: ignore[override] raise NotImplementedError - async def is_off(self, key: FeatureKey, user_context: UserContext) -> bool: # type: ignore[override] + async def is_off(self, key: FeatureKey, user_context: UserContext, *, tracking_buffer: Optional[TrackingBuffer] = None) -> bool: # type: ignore[override] raise NotImplementedError diff --git a/tests/scripts/benchmark_eval_overhead.py b/tests/scripts/benchmark_eval_overhead.py new file mode 100644 index 0000000..986e89c --- /dev/null +++ b/tests/scripts/benchmark_eval_overhead.py @@ -0,0 +1,45 @@ +"""Micro-benchmark for per-evaluation overhead with telemetry disabled. + +Measures the two paths a hot server loop actually exercises, with no tracking +callback, no usage callback, and no deferred buffer configured — the +worst-case for the telemetry plumbing added in 3.1 (EvaluationContext carries +the callback fields; eval_feature reports usage through a wrapper): + + * default-value path: the cheapest possible evaluation + * experiment-rule path: a realistic hashed assignment + +Run against two checkouts to compare (best of 5 rounds each): + + python tests/scripts/benchmark_eval_overhead.py [N] + (cd /path/to/main-checkout && python tests/scripts/... ) +""" +import sys +import time + +sys.path.insert(0, ".") + +from growthbook import GrowthBook # noqa: E402 + +N = int(sys.argv[1]) if len(sys.argv) > 1 else 1_000_000 + +gb = GrowthBook(attributes={"id": "u1"}, features={ + "flag": {"defaultValue": True}, + "exp": {"defaultValue": 0, "rules": [{"key": "e", "variations": [0, 1], "coverage": 1}]}, +}) + + +def best_of(key, rounds=5): + best = float("inf") + for _ in range(rounds): + start = time.perf_counter() + for _ in range(N): + gb.eval_feature(key) + best = min(best, time.perf_counter() - start) + return best + + +for key, label in (("flag", "default-value"), ("exp", "experiment-rule")): + took = best_of(key) + print(f"{label:16s} {took:.3f}s / {N} evals ({took / N * 1e6:.3f} us/eval)") + +gb.destroy() diff --git a/tests/scripts/check_corpus_freshness.py b/tests/scripts/check_corpus_freshness.py index 816ca8f..ca6838f 100644 --- a/tests/scripts/check_corpus_freshness.py +++ b/tests/scripts/check_corpus_freshness.py @@ -71,6 +71,9 @@ "inNamespace", "getEqualWeights", "stickyBucket", + # Python-local extension today (reported as extras); once the JS SDK + # adopts the section, drift checking picks it up automatically. + "trackingCalls", ) diff --git a/tests/scripts/js_receiver.js b/tests/scripts/js_receiver.js new file mode 100644 index 0000000..6cbb566 --- /dev/null +++ b/tests/scripts/js_receiver.js @@ -0,0 +1,55 @@ +// Receiving half of the deferred-tracking simulation: the REAL GrowthBook JS +// SDK hydrates the Python-produced payloads and fires them, exactly as a +// browser would after an SSR render. +// +// stdin: [{"label": str, "calls": TrackingData[]}, ...] +// stdout: {label: [{experiment, variationId, hashAttribute, hashValue, +// userAttributes}, ...]} +// +// SDK resolution: $GB_JS_SDK or an installed @growthbook/growthbook package, +// >= 1.7.0 (older SDKs fire deferred calls without the user argument). +// Deliberately NO fallback to a repo checkout's dist/ — a stale local build +// artifact must fail loudly, not silently verify the wrong SDK. +function loadSdk() { + const candidates = [process.env.GB_JS_SDK, "@growthbook/growthbook"].filter(Boolean); + for (const candidate of candidates) { + try { + return { sdk: require(candidate), source: candidate }; + } catch (e) { + /* try the next candidate */ + } + } + throw new Error( + "No GrowthBook JS SDK found. `npm install @growthbook/growthbook` (>=1.7.0) " + + "or point GB_JS_SDK at one." + ); +} +const { sdk, source } = loadSdk(); +const { GrowthBook } = sdk; + +let input = ""; +process.stdin.on("data", (d) => (input += d)); +process.stdin.on("end", async () => { + const payloads = JSON.parse(input); + const out = {}; + for (const { label, calls } of payloads) { + const fired = []; + const gb = new GrowthBook({ + trackingCallback: (experiment, result, user) => { + fired.push({ + experiment: experiment.key, + variationId: result.variationId, + hashAttribute: result.hashAttribute, + hashValue: result.hashValue, + userAttributes: user ? user.attributes : null, + }); + }, + }); + gb.setDeferredTrackingCalls(calls); + await gb.fireDeferredTrackingCalls(); + out[label] = fired; + gb.destroy(); + } + out._receiver = source; + process.stdout.write(JSON.stringify(out)); +}); diff --git a/tests/scripts/simulate_deferred_tracking.py b/tests/scripts/simulate_deferred_tracking.py new file mode 100644 index 0000000..02dd3d1 --- /dev/null +++ b/tests/scripts/simulate_deferred_tracking.py @@ -0,0 +1,229 @@ +"""End-to-end simulation of deferred tracking and prerequisite telemetry. + +Two realistic server scenarios, verified against the REAL JavaScript SDK as +the receiving side (tests/scripts/js_receiver.js runs it in Node): + +1. Sync SSR: one GrowthBook instance per request (the documented pattern) + renders a page for each user, buffers exposures, embeds them as JSON, and + a simulated browser fires them through the JS SDK's + setDeferredTrackingCalls + fireDeferredTrackingCalls. + +2. Async API server: one long-lived GrowthBookClient serves concurrent + requests; each handler owns a per-request TrackingBuffer whose contents go + back in the API response. A server-side on_experiment_viewed callback runs + at the same time (buffering is independent of callbacks). + +Nothing is hard-coded to pass: 50/50 experiments are decided by real hashing, +expected exposures are derived from the SDK's own evaluation results, and the +fired events must match them user-by-user. Attributes are mutated after each +request to prove buffered snapshots are immune to caller aliasing. + +Run: python tests/scripts/simulate_deferred_tracking.py +Requires node + a built JS SDK (env GB_JS_SDK overrides the default path). +""" +import asyncio +import json +import os +import subprocess +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..")) + +from growthbook import GrowthBook, TrackingBuffer # noqa: E402 +from growthbook.common_types import Options, UserContext # noqa: E402 +from growthbook.growthbook_client import GrowthBookClient # noqa: E402 + +# Realistic payload: a prerequisite that is itself an experiment, a gated +# feature with its own experiment, a ramp with a passthrough holdback, and a +# plain flag. All splits decided by real hashing. +FEATURES = { + "pricing-engine": { + "defaultValue": "legacy", + "rules": [{ + "key": "pricing-engine-rollout", "coverage": 1, + "variations": ["legacy", "v2"], "weights": [0.5, 0.5], + "meta": [{"key": "control"}, {"key": "v2"}], + }], + }, + "checkout-redesign": { + "defaultValue": "classic", + "rules": [{ + "parentConditions": [{"id": "pricing-engine", "condition": {"value": "v2"}}], + "key": "checkout-redesign-exp", "coverage": 1, + "variations": ["classic", "one-page"], "weights": [0.5, 0.5], + }], + }, + "onboarding-tour": { + "defaultValue": "none", + "rules": [ + { + "key": "onboarding-tour-ramp", "coverage": 1, + "variations": ["candidate", "holdback"], "weights": [0.2, 0.8], + "meta": [{"key": "candidate"}, {"key": "holdback", "passthrough": True}], + }, + {"force": "checklist"}, + ], + }, + "cta-copy": {"defaultValue": "Start free trial"}, +} + +USERS = [{"id": f"user-{n}", "country": "US" if n % 3 else "CA"} for n in range(1, 31)] + +PAGE_FEATURES = ["checkout-redesign", "onboarding-tour", "cta-copy"] + + +def expected_exposures(attributes): + """Derive the exposures a page render must produce, from the SDK's own + assignments (no hard-coded variations).""" + gb = GrowthBook(attributes=dict(attributes), features=FEATURES) + pricing = gb.eval_feature("pricing-engine") + checkout = gb.eval_feature("checkout-redesign") + tour = gb.eval_feature("onboarding-tour") + gb.destroy() + + expected = {("pricing-engine-rollout", pricing.experimentResult.variationId)} + if pricing.value == "v2": + expected.add(("checkout-redesign-exp", checkout.experimentResult.variationId)) + # The ramp tracks either way: candidate directly, holdback via passthrough. + tour_variation = 0 if tour.value == "candidate" else 1 + expected.add(("onboarding-tour-ramp", tour_variation)) + return expected + + +def render_page_sync(attributes): + """One SSR request: per-request instance, several evals (one repeated, + as templates do), then harvest the buffer.""" + gb = GrowthBook(attributes=attributes, features=FEATURES, defer_tracking=True) + flags = {key: gb.get_feature_value(key, None) for key in PAGE_FEATURES} + gb.is_on("checkout-redesign") # template re-checks the same flag + calls = gb.get_deferred_tracking_calls() + gb.destroy() + return flags, calls + + +async def handle_api_request(client, attributes): + """One async API request: caller-owned buffer, several evals, JSON body.""" + buf = TrackingBuffer() + ctx = UserContext(attributes=attributes) + flags = {} + for key in PAGE_FEATURES: + flags[key] = (await client.eval_feature(key, ctx, tracking_buffer=buf)).value + await client.is_on("checkout-redesign", ctx, tracking_buffer=buf) # duplicate eval + return {"flags": flags, "trackingCalls": buf.get_calls()} + + +def fire_through_js_sdk(batches): + """Feed [{label, calls}] through the real JS SDK in Node; return + {label: fired events}.""" + receiver = os.path.join(os.path.dirname(__file__), "js_receiver.js") + proc = subprocess.run( + ["node", receiver], input=json.dumps(batches).encode(), + capture_output=True, timeout=60, + ) + if proc.returncode != 0: + raise RuntimeError(f"js_receiver failed: {proc.stderr.decode()}") + fired = json.loads(proc.stdout) + print(f" (receiver: JS SDK from {fired.pop('_receiver')})") + return fired + + +def check(condition, message): + if not condition: + raise AssertionError(message) + return 1 + + +def verify_fired(label, fired, expected, attributes): + checks = 0 + got = {(f["experiment"], f["variationId"]) for f in fired} + checks += check(got == expected, f"{label}: fired {got} != expected {expected}") + for f in fired: + checks += check(f["hashValue"] == attributes["id"], + f"{label}: exposure carries wrong hashValue {f['hashValue']}") + checks += check(f["userAttributes"] == attributes, + f"{label}: exposure user {f['userAttributes']} != {attributes}") + return checks + + +def simulate_sync(): + checks, gated_out, batches, expectations = 0, 0, [], {} + for user in USERS: + attrs = dict(user) + expected = expected_exposures(attrs) + flags, calls = render_page_sync(attrs) + + # Request teardown mutates the attributes dict the SDK saw (destroy() + # already cleared it in place — buffered snapshots must survive both). + attrs["id"] = "recycled" + attrs["country"] = "XX" + + page_json = json.dumps({"flags": flags, "trackingCalls": calls}) # the rendered page + batches.append({"label": user["id"], "calls": json.loads(page_json)["trackingCalls"]}) + expectations[user["id"]] = (expected, dict(user)) + + if flags["checkout-redesign"] == "classic" and ("checkout-redesign-exp", 0) not in expected: + gated_out += 1 # gated by prereq: pricing exposure must still exist + checks += check(("pricing-engine-rollout", 0) in expected, + f"{user['id']}: gated user lost the prerequisite exposure") + + fired_by_user = fire_through_js_sdk(batches) + for label, (expected, attrs) in expectations.items(): + checks += verify_fired(label, fired_by_user[label], expected, attrs) + + exposures = sum(len(v) for v in fired_by_user.values()) + print(f"sync SSR: {len(USERS)} requests, {exposures} exposures fired by the JS SDK, " + f"{gated_out} users gated out of checkout (prereq exposure still fired), " + f"{checks} assertions passed") + + +async def simulate_async(): + server_side = [] + client = GrowthBookClient(Options( + api_host="https://localhost.growthbook.io", client_key="sim", + on_experiment_viewed=lambda experiment, result, user_context: server_side.append( + (user_context.attributes["id"], experiment.key) + ), + )) + await client.set_features(FEATURES) + try: + responses = await asyncio.gather( + *(handle_api_request(client, dict(u)) for u in USERS) + ) + finally: + await client.close() + + checks, batches, expectations = 0, [], {} + for user, response in zip(USERS, responses): + expected = expected_exposures(user) # sync reference: same assignments + calls = response["trackingCalls"] + checks += check(all(c["user"]["attributes"] == user for c in calls), + f"{user['id']}: buffer mixed in another user's exposure") + got = {(c["experiment"]["key"], c["result"]["variationId"]) for c in calls} + checks += check(len(calls) == len(got), f"{user['id']}: duplicate eval not deduped") + checks += check(got == expected, f"{user['id']}: async {got} != sync reference {expected}") + batches.append({"label": user["id"], "calls": calls}) + expectations[user["id"]] = (expected, dict(user)) + + fired_by_user = fire_through_js_sdk(batches) + for label, (expected, attrs) in expectations.items(): + checks += verify_fired(label, fired_by_user[label], expected, attrs) + + # Buffering was independent of the server-side callback, which deduped + # per (user, experiment) exactly once. + for user in USERS: + expected_keys = {key for key, _ in expectations[user["id"]][0]} + got_keys = [k for uid, k in server_side if uid == user["id"]] + checks += check(sorted(got_keys) == sorted(expected_keys), + f"{user['id']}: server-side callback fired {got_keys}, expected {expected_keys}") + + exposures = sum(len(v) for v in fired_by_user.values()) + print(f"async API: {len(USERS)} concurrent requests, {exposures} exposures fired by the " + f"JS SDK, {len(server_side)} server-side callback events alongside, " + f"{checks} assertions passed") + + +if __name__ == "__main__": + simulate_sync() + asyncio.run(simulate_async()) + print("OK: every buffered exposure matched the SDK's own assignments and was " + "fired by the real JS SDK with the exposure-time user context.") diff --git a/tests/test_growthbook.py b/tests/test_growthbook.py index 97db6ab..4c161e5 100644 --- a/tests/test_growthbook.py +++ b/tests/test_growthbook.py @@ -181,6 +181,27 @@ def test_contextual_bandit(contextualBandit_data): gb.destroy() +def test_tracking_calls(trackingCalls_data): + # trackingCalls is a Python-local cases.json extension pending upstream + # adoption — no SDK's shared spec asserts tracking/usage callbacks today. + # The corpus freshness checker lists it in KEYS_TO_DIFF: its cases show + # up as never-failing extras now, and get drift-checked automatically + # once the JS corpus adopts the section. + _, ctx, key, expected_tracking, expected_usage = trackingCalls_data + tracked, usage = [], [] + gb = GrowthBook( + on_experiment_viewed=lambda experiment, result, user_context: tracked.append( + [experiment.key, result.variationId] + ), + on_feature_usage=lambda k, result, user_context: usage.append(k), + **ctx, + ) + gb.eval_feature(key) + assert tracked == expected_tracking + assert usage == expected_usage + gb.destroy() + + def test_run(run_data): _, ctx, exp, value, inExperiment, hashUsed = run_data gb = GrowthBook(**ctx) @@ -392,6 +413,14 @@ def test_handles_weird_experiment_values(): gb.destroy() +def test_experiment_to_dict_preserves_explicit_zero_coverage(): + # `or 1` would coerce a real coverage of 0 to 1; None still reads as + # full coverage. Serialized dicts are forwarded (deferred tracking), + # so the distinction is externally visible. + assert Experiment(key="e", variations=[0, 1], coverage=0).to_dict()["coverage"] == 0 + assert Experiment(key="e", variations=[0, 1]).to_dict()["coverage"] == 1 + + def test_custom_fields_parsed_from_api_dict(): # The API delivers experiment Custom Fields as a flat dict. exp = Experiment( diff --git a/tests/test_growthbook_client.py b/tests/test_growthbook_client.py index ec99861..5028341 100644 --- a/tests/test_growthbook_client.py +++ b/tests/test_growthbook_client.py @@ -580,7 +580,8 @@ def pytest_generate_tests(metafunc): test_data_map = { 'test_eval_feature': 'feature', 'test_experiment_run': 'run', - 'test_sticky_bucket': 'stickyBucket' + 'test_sticky_bucket': 'stickyBucket', + 'test_tracking_calls': 'trackingCalls' } for func, data_key in test_data_map.items(): @@ -646,7 +647,7 @@ async def test_experiment_run(test_experiment_run_data, base_client_setup): # Create and initialize client async with GrowthBookClient(Options(**client_opts)) as client: result = await client.run(Experiment(**exp), UserContext(**user_attrs)) - + # Verify experiment results assert result.value == value assert result.inExperiment == inExperiment @@ -658,6 +659,42 @@ async def test_experiment_run(test_experiment_run_data, base_client_setup): await client.close() await asyncio.sleep(0.1) +@pytest.mark.asyncio +async def test_tracking_calls(test_tracking_calls_data, base_client_setup): + """Run the shared trackingCalls cases (a Python-local cases.json + extension) against the async client — same expectations as the sync + runner in test_growthbook.py.""" + _, ctx, key, expected_tracking, expected_usage = test_tracking_calls_data + + user_attrs, client_opts, features_data = base_client_setup(ctx) + tracked, usage = [], [] + client_opts['on_experiment_viewed'] = ( + lambda experiment, result, user_context: tracked.append( + [experiment.key, result.variationId] + ) + ) + client_opts['on_feature_usage'] = ( + lambda k, result, user_context: usage.append(k) + ) + + EnhancedFeatureRepository._instances = {} + client = None + try: + with patch('growthbook.FeatureRepository.load_features_async', + new_callable=AsyncMock, return_value=features_data), \ + patch('growthbook.growthbook_client.EnhancedFeatureRepository.start_feature_refresh', + new_callable=AsyncMock), \ + patch('growthbook.growthbook_client.EnhancedFeatureRepository.stop_refresh', + new_callable=AsyncMock): + async with GrowthBookClient(Options(**client_opts)) as client: + await client.eval_feature(key, UserContext(**user_attrs)) + assert tracked == expected_tracking + assert usage == expected_usage + finally: + if client is not None: + await client.close() + await asyncio.sleep(0.1) + @pytest.mark.asyncio async def test_feature_methods(): """Test feature helper methods (isOn, isOff, getFeatureValue)""" diff --git a/tests/test_tracking_telemetry.py b/tests/test_tracking_telemetry.py new file mode 100644 index 0000000..ce34a22 --- /dev/null +++ b/tests/test_tracking_telemetry.py @@ -0,0 +1,335 @@ +"""Tracking behaviors the cases.json `trackingCalls` extension can't express: +run()-level prerequisites, subscriptions, and the deferred tracking buffer's +client API (opt-in, snapshot isolation, per-request async ownership). + +Pure evaluate-and-expect tracking/usage semantics live in cases.json +("trackingCalls", a Python-local extension run by both clients' suites). +""" +import asyncio +import datetime +import json +import threading + +import pytest + +from growthbook import GrowthBook, TrackingBuffer +from growthbook.common_types import Experiment, Options, UserContext +from growthbook.growthbook_client import GrowthBookClient + +# Weights of [1, 0] / [0, 1] make every assignment deterministic. +FEATURES = { + "parent": { + "defaultValue": "off", + "rules": [ + {"key": "parent-exp", "coverage": 1, "variations": ["on", "off"], "weights": [1, 0]} + ], + }, + "child": { + "defaultValue": "child-default", + "rules": [ + {"parentConditions": [{"id": "parent", "condition": {"value": "on"}}], "force": "child-on"} + ], + }, + "child-gated": { + "defaultValue": "child-default", + "rules": [ + { + "parentConditions": [{"id": "parent", "condition": {"value": "nope"}, "gate": True}], + "force": "never", + } + ], + }, + "child-twice": { + "defaultValue": "child-default", + "rules": [ + {"parentConditions": [{"id": "parent", "condition": {"value": "nope"}}], "force": "r1"}, + {"parentConditions": [{"id": "parent", "condition": {"value": "on"}}], "force": "r2"}, + ], + }, + "ramped": { + "defaultValue": "default", + "rules": [ + { + "key": "ramp", + "coverage": 1, + "variations": ["treatment", "default"], + "weights": [0, 1], + "meta": [{"key": "0"}, {"key": "1", "passthrough": True}], + }, + {"force": "fallthrough"}, + ], + }, + "unserializable": {"defaultValue": {1: "a", "b": 2}}, +} + + +def make_gb(**kwargs): + tracked, usage = [], [] + + def on_viewed(experiment, result, user_context): + tracked.append((experiment.key, result.variationId)) + + def on_usage(key, result, user_context): + usage.append(key) + + gb = GrowthBook( + attributes={"id": "user-1"}, + features=FEATURES, + on_experiment_viewed=on_viewed, + on_feature_usage=on_usage, + **kwargs, + ) + return gb, tracked, usage + + +def test_experiment_level_prerequisite_is_tracked(): + gb, tracked, usage = make_gb() + exp = Experiment( + key="direct", + variations=["x", "y"], + weights=[1, 0], + parentConditions=[{"id": "parent", "condition": {"value": "on"}}], + ) + seen = [] + gb.subscribe(lambda experiment, result: seen.append(experiment.key)) + res = gb.run(exp) + assert res.inExperiment and res.value == "x" + assert tracked == [("parent-exp", 0), ("direct", 0)] + assert usage == ["parent"] + assert seen == ["parent-exp", "direct"] + gb.destroy() + + +def test_unserializable_feature_values_do_not_break_evaluation(): + gb, _, usage = make_gb() + assert gb.eval_feature("unserializable").value == {1: "a", "b": 2} + assert usage == ["unserializable"] + gb.destroy() + + +def test_non_json_exposures_are_dropped_not_the_batch(): + # The buffer's record-time JSON round-trip drops exposures it cannot + # serialize (logged); evaluation and the tracking callback are unaffected, + # and every retained entry stays json.dumps-ready. Two failure modes: a + # non-JSON variation value, and a non-JSON user attribute (datetime). + lock = threading.Lock() + features = dict(FEATURES) + features["locked"] = { + "defaultValue": "off", + "rules": [{"key": "locked-exp", "coverage": 1, + "variations": [lock, "off"], "weights": [1, 0]}], + } + tracked = [] + gb = GrowthBook( + attributes={"id": "user-1"}, features=features, defer_tracking=True, + on_experiment_viewed=lambda experiment, result, user_context: tracked.append(experiment.key), + ) + assert gb.eval_feature("locked").value is lock + assert tracked == ["locked-exp"] + assert gb.get_deferred_tracking_calls() == [] + + gb.set_attributes({"id": "user-1", "signup": datetime.datetime(2026, 1, 1)}) + gb.eval_feature("parent") # user snapshot now unserializable -> dropped + gb.set_attributes({"id": "user-2"}) + gb.eval_feature("parent") # clean exposure still buffered + + calls = gb.get_deferred_tracking_calls() + assert [c["user"]["attributes"]["id"] for c in calls] == ["user-2"] + json.dumps(calls) # the batch is never poisoned + gb.destroy() + + +def test_dedupe_key_fields_cannot_collide_across_boundaries(): + # Exposure identity is a field tuple: a value containing a would-be + # delimiter can never merge two distinct exposures (hashValue "a\0b" + + # key "c" vs hashValue "a" + key "b\0c"). + tracked = [] + gb = GrowthBook( + attributes={"id": "a\x00b"}, defer_tracking=True, + on_experiment_viewed=lambda experiment, result, user_context: tracked.append(experiment.key), + ) + gb.run(Experiment(key="c", variations=[0, 1], weights=[1, 0])) + gb.set_attributes({"id": "a"}) + gb.run(Experiment(key="b\x00c", variations=[0, 1], weights=[1, 0])) + assert tracked == ["c", "b\x00c"] + assert len(gb.get_deferred_tracking_calls()) == 2 + gb.destroy() + + +def test_legacy_core_callback_kwargs_still_work(): + # Pre-3.1 direct consumers of growthbook.core passed callbacks as kwargs; + # they are deprecated and invocation-scoped: installed on the context for + # the duration of the call (prerequisites inherit them), then the previous + # context fields are restored — a later call on the same context must not + # fire them. + from growthbook.core import eval_feature as core_eval_feature + + gb = GrowthBook(attributes={"id": "user-1"}, features=FEATURES) + ctx = gb._get_eval_context() + tracked, seen = [], [] + with pytest.warns(DeprecationWarning): + res = core_eval_feature( + "child", + ctx, + callback_subscription=lambda experiment, result: seen.append(experiment.key), + tracking_cb=lambda experiment, result, user: tracked.append(experiment.key), + ) + assert res.value == "child-on" + assert tracked == ["parent-exp"] + assert seen == ["parent-exp"] + + assert core_eval_feature("ramped", ctx).value == "fallthrough" + assert tracked == ["parent-exp"] # the ramp exposure did not leak to the legacy callback + assert seen == ["parent-exp"] + gb.destroy() + + +def test_subscriptions_see_prerequisite_experiments(): + gb, _, _ = make_gb() + seen = [] + gb.subscribe(lambda experiment, result: seen.append(experiment.key)) + gb.eval_feature("child") + assert seen == ["parent-exp"] + assert "parent-exp" in gb.get_all_results() + gb.destroy() + + +def test_deferred_tracking_is_opt_in(): + gb = GrowthBook(attributes={"id": "user-1"}, features=FEATURES) + gb.eval_feature("child") + assert gb.get_deferred_tracking_calls() == [] + gb.destroy() + + +def test_deferred_tracking_buffers_without_a_callback(): + gb = GrowthBook(attributes={"id": "user-1"}, features=FEATURES, defer_tracking=True) + gb.eval_feature("child") + gb.eval_feature("child") # same assignment, deduped + + calls = gb.get_deferred_tracking_calls() + assert len(calls) == 1 + assert calls[0]["experiment"]["key"] == "parent-exp" + assert calls[0]["result"]["variationId"] == 0 + assert calls[0]["user"] == {"attributes": {"id": "user-1"}, "url": ""} + json.dumps(calls) # forwardable as-is + + gb.clear_deferred_tracking_calls() + assert gb.get_deferred_tracking_calls() == [] + gb.destroy() + + +def test_buffering_is_independent_of_the_callback(): + gb, tracked, _ = make_gb(defer_tracking=True) + gb.eval_feature("child") + assert tracked == [("parent-exp", 0)] + assert [c["experiment"]["key"] for c in gb.get_deferred_tracking_calls()] == ["parent-exp"] + gb.destroy() + + +def test_buffered_entries_keep_the_exposure_time_user(): + gb = GrowthBook(attributes={"id": "user-1"}, features=FEATURES, defer_tracking=True) + gb.eval_feature("child") + gb.set_attributes({"id": "user-2"}) + gb.eval_feature("child") + + calls = gb.get_deferred_tracking_calls() + assert [c["user"]["attributes"]["id"] for c in calls] == ["user-1", "user-2"] + gb.destroy() + + +def test_get_deferred_tracking_calls_returns_detached_copies(): + gb = GrowthBook(attributes={"id": "user-1"}, features=FEATURES, defer_tracking=True) + gb.eval_feature("child") + calls = gb.get_deferred_tracking_calls() + calls[0]["experiment"]["key"] = "mutated" + calls[0]["user"]["attributes"]["id"] = "mutated" + assert gb.get_deferred_tracking_calls()[0]["experiment"]["key"] == "parent-exp" + assert gb.get_deferred_tracking_calls()[0]["user"]["attributes"] == {"id": "user-1"} + gb.destroy() + + +def test_passthrough_and_gated_prerequisite_exposures_land_in_buffer(): + gb = GrowthBook(attributes={"id": "user-1"}, features=FEATURES, defer_tracking=True) + gb.eval_feature("ramped") + gb.eval_feature("child-gated") + keys = [c["experiment"]["key"] for c in gb.get_deferred_tracking_calls()] + assert keys == ["ramp", "parent-exp"] + gb.destroy() + + +def test_rule_tracks_exposures_land_in_buffer(): + # Pre-evaluated exposures attached by the remote-eval proxy buffer too, + # and contextual bandit metadata survives the hydrate -> record round trip. + features = { + "remote": { + "defaultValue": None, + "rules": [{ + "force": "server-value", + "tracks": [{ + "experiment": {"key": "proxy-exp", "variations": ["a", "b"]}, + "result": { + "variationId": 1, "inExperiment": True, "value": "b", + "hashUsed": True, "hashAttribute": "id", "hashValue": "user-1", + "leafId": 3, "variationWeights": [0.2, 0.8], "banditVersion": 7, + }, + }], + }], + }, + } + gb = GrowthBook(attributes={"id": "user-1"}, features=features, defer_tracking=True) + assert gb.eval_feature("remote").value == "server-value" + (call,) = gb.get_deferred_tracking_calls() + assert call["experiment"]["key"] == "proxy-exp" + assert call["result"]["leafId"] == 3 + assert call["result"]["variationWeights"] == [0.2, 0.8] + assert call["result"]["banditVersion"] == 7 + gb.destroy() + + +@pytest.mark.asyncio +async def test_async_client_buffers_per_request(): + client = GrowthBookClient(Options(api_host="https://localhost.growthbook.io", client_key="test")) + await client.set_features(FEATURES) + try: + buf1, buf2 = TrackingBuffer(), TrackingBuffer() + res1, res2 = await asyncio.gather( + client.eval_feature("child", UserContext(attributes={"id": "user-1"}), tracking_buffer=buf1), + client.eval_feature("child", UserContext(attributes={"id": "user-2"}), tracking_buffer=buf2), + ) + assert res1.value == "child-on" and res2.value == "child-on" + + calls1, calls2 = buf1.get_calls(), buf2.get_calls() + assert [c["experiment"]["key"] for c in calls1] == ["parent-exp"] + assert [c["experiment"]["key"] for c in calls2] == ["parent-exp"] + assert calls1[0]["user"]["attributes"] == {"id": "user-1"} + assert calls2[0]["user"]["attributes"] == {"id": "user-2"} + + # No buffer passed → nothing collected anywhere. + await client.eval_feature("child", UserContext(attributes={"id": "user-3"})) + assert len(buf1.get_calls()) == 1 and len(buf2.get_calls()) == 1 + finally: + await client.close() + + +@pytest.mark.asyncio +async def test_async_client_subscriptions_fire_only_from_run(): + # Deliberate asymmetry with the sync client: the multi-user client fires + # subscriptions only from run() (like the JS multi-user client, which has + # no eval-time subscriptions). It has no per-user assignment + # change-detection, so per-eval firing would repeat every subscriber + # callback on every request. + client = GrowthBookClient(Options( + api_host="https://localhost.growthbook.io", + client_key="test", + )) + await client.set_features(FEATURES) + seen = [] + try: + client.subscribe(lambda experiment, result: seen.append(experiment.key)) + await client.eval_feature("child", UserContext(attributes={"id": "user-1"})) + assert seen == [] + exp = Experiment(key="direct", variations=["x", "y"], weights=[1, 0]) + await client.run(exp, UserContext(attributes={"id": "user-1"})) + finally: + await client.close() # also drains scheduled subscription callbacks + assert seen == ["direct"]