Repository navigation
fix(sentinelone): add new endpoint SDL2 for saas, and fix logic for selfhosted drilldown (#490) - #588
Christophe Melchior (Kakudou) wants to merge 12 commits into
Conversation
Use the signature start_date as the fetch window start when present and not after end, otherwise keep the end - SENTINELONE_TIME_WINDOW fallback for both threat and Deep Visibility windows.
Probe on account listing (accountType discriminator), no config knob; SaaS-safe default on probe failure.
SaaS instances now run deep-search file events through SentinelOne SDL v2 (/sdl/v2/api/queries, S1QL 2.0): launch/poll with forward-tag echo, cursor pagination with inclusive-resume dedupe, 429 backoff retries, and rows mapped to the existing Deep-Visibility event shape. Self-hosted instances keep the legacy DV 1.0 path unchanged; the service selects the backend from the resolved is_self_hosted flag.
A transient transport failure (network drop, 429, 5xx) no longer resolves an expectation with a negative verdict: results are held pending in a time-based retry window (SENTINELONE_RETRY_WINDOW, default 10 min), re-fetched each cycle, and degrade to a verdict on the first attempt after it. Transport gets per-request 30 s timeouts with bounded GET-only retries, threat events paginate to completion, the SDL request carries the Bearer token, and the DV/SDL file-event window is decoupled from signature dates (SENTINELONE_DEEP_VISIBILITY_LOOKBACK, default 1 day).
|
Thank you for your contribution. This PR is but one step away from being ready for merging: all commits must be PGP-signed. To get started, please see https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits |
6736bbb to
8c4395e
Compare
… from earliest start date Two correctness fixes on top of the pending-on-transient-failure work: - _classify_error now maps requests RetryError to "transient". When the session-level urllib3 Retry exhausts its attempts on a retryable status (429/5xx), requests raises RetryError, which is neither HTTPError nor ConnectionError/Timeout and was therefore classified "permanent" — a 5xx outage that outlived the transport retries stamped an immediate negative verdict, the exact one-shot blip this feature is meant to avoid. - SignatureExtractor.extract_start_date now returns the earliest start_date across the batch instead of the first one encountered. A batch mixes expectations from several injects; anchoring the fetch window on a later start_date drops the older injects' alerts out of the window.
The collector read only the Threats API, which returns convicted threats (static-engine reputation hits and mitigated behavioral threats). SentinelOne's behavioral / STAR / AI detections land in Unified Alerts instead and never become Threats, so an inject such as "In-Memory Mimikatz-DumpCreds" that raises a "Potential Mimikatz Execution" alert was reported Not Detected even though the console showed the detection. This is the marquee case of issue #490. Add FetcherUnifiedAlerts, which lists Singularity alerts over the Unified Alerts GraphQL API (POST /web/api/v2.1/unifiedalerts/graphql, Bearer auth) in the threat fetch window, recovers the OpenAEV implant image name from each alert's OCSF rawData, and maps every alert to the existing Deep-Visibility-shaped SentinelOneThreat + event contract. The unchanged matching pipeline then correlates them to expectations by the parent_process_name signature; a MITIGATED alert marks the expectation prevented. The expectation service fetches alerts for the same window as threats and merges them before matching, in an isolated try/except so an alerts-API failure never breaks the Threats verdict. Behavior is gated by SENTINELONE_ENABLE_ALERTS (default on, SaaS only; skipped on self-hosted where the endpoint is absent). Verified live against a SaaS tenant: a "Potential Mimikatz Execution" alert with no backing Threat now resolves the detection expectation as Detected, while the same expectation stays Not Detected with the flag off.
Mariot Tsitoara (mariot)
left a comment
There was a problem hiding this comment.
Just some quick question. PR is solid
| f"{fetcher.client_api.base_url}/web/api/v2.1/threats/" | ||
| f"{threat.threat_id}/explore/events" | ||
| ) | ||
| expected_calls = [call(endpoint, params={"limit": 100}, timeout=30)] |
There was a problem hiding this comment.
We could use REQUEST_TIMEOUT_SECONDS ?
There was a problem hiding this comment.
Good point, done in bf9a966. REQUEST_TIMEOUT_SECONDS is now defined once in client_api.py and imported by every fetcher (fetcher_threat, fetcher_threat_events, fetcher_deep_visibility, fetcher_sdl, fetcher_unified_alerts), which also removes the 4 duplicated module-level constants and the hard-coded timeout=30 in the code. This test now asserts timeout=fetcher_threat_events_module.REQUEST_TIMEOUT_SECONDS.
| enable_alerts: bool = Field( | ||
| alias="SENTINELONE_ENABLE_ALERTS", | ||
| default=True, | ||
| description=( | ||
| "Also correlate SentinelOne Unified Alerts (Singularity alerts " | ||
| "GraphQL API), not only Threats. Behavioral / STAR / AI detections " | ||
| "(e.g. Potential Mimikatz Execution) surface as Unified Alerts and " | ||
| "never as Threats, so this is required to validate them. SaaS only; " | ||
| "ignored on self-hosted instances." | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Would defaulting this to True not bother the existing clients?
There was a problem hiding this comment.
I kept True on purpose: it is the actual fix for #490. Mimikatz-style behavioral / STAR / AI detections only surface as Unified Alerts and never as Threats, so with an opt-in default the issue would still be there out of the box.
For existing clients:
- self-hosted instances ignore it (no Unified Alerts endpoint there);
- an alerts failure is isolated: it is caught and never changes the Threats verdict;
- the remaining concern was a token without alerts access, which would log an error on every batch. As of bf9a966 a 401/403 from the Unified Alerts endpoint disables the correlation for the rest of the run with a single warning (
grant the alerts view permission or set SENTINELONE_ENABLE_ALERTS=false).SentinelOneAPIErrornow carriesstatus_code, and there are tests for 401/403 (disabled after one call) and 500 (stays enabled).
The flag is also documented in the README now, and passed through docker-compose.yml (main switched to an explicit env list in #646, so the new knobs were not reaching the container after the resync).
| else: | ||
| self.first_attempt_at.pop(result.expectation_id, None) | ||
|
|
||
| results = [result for result in results if not result in retry_results] |
There was a problem hiding this comment.
| results = [result for result in results if not result in retry_results] | |
| results = [result for result in results if result not in retry_results] |
| if status == 404: | ||
| raise SentinelOneAPIError(f"{context} failed with status 404: {detail}") | ||
| raise SentinelOneAPIError(f"{context} failed with status {status}: {detail}") |
There was a problem hiding this comment.
Why do we set apart error 404 if we're not doing anything special to it?
There was a problem hiding this comment.
You are right, it was dead code: both branches raised the same error. Removed in bf9a966. The 404 that actually matters (an expired query while polling) is handled in _poll_query_to_completion, which raises a dedicated "expired (HTTP 404)" error. That path is unchanged and still covered by test_poll_404_expired_raises_api_error.
There was a problem hiding this comment.
Is there a difference between the ExpectationResult here and the one in src/collector/models.py ?
There was a problem hiding this comment.
No difference: the two models were identical (same fields, only a docstring differed). The duplicate dates back to #174 and existed to avoid an import cycle (src.collector.__init__ → collector.py → expectation_service, so expectation_service cannot import src.collector.models).
Fixed in bf9a966: there is now a single definition in src/services/model_expectation_result.py (next to model_threat.py). expectation_service imports it from there, and src/collector/models.py re-exports it, so the collector modules, trace_service and the tests keep importing it from the same place and get the same class.
…nges - Define REQUEST_TIMEOUT_SECONDS once in client_api and reuse it in every fetcher and in the threat-events test instead of hard-coded 30s values. - Move ExpectationResult to services/model_expectation_result.py and re-export it from collector/models.py: the two copies were identical and only existed to avoid a src.collector <-> expectation_service import cycle. - Drop the redundant 404 branch in FetcherSDL._raise_for_error_response (expired-query 404s are handled where the poll happens). - Use `result not in retry_results` (review suggestion). - SENTINELONE_ENABLE_ALERTS stays on by default, but a 401/403 from the Unified Alerts endpoint (token without alerts access) now disables the correlation for the run with a single warning instead of logging an error on every batch. SentinelOneAPIError carries the HTTP status code. - Pass SENTINELONE_RETRY_WINDOW / ENABLE_ALERTS / DEEP_VISIBILITY_LOOKBACK through docker-compose.yml (explicit env list since #646) and document them in the README.
There was a problem hiding this comment.
🟡 Changes recommended
The account probe mishandles nested and malformed responses, disabling Unified Alerts on SaaS or aborting startup, while alert-scanning and time-window defects can miss detections.
5 open issues
What changed in this PR
This PR improves SentinelOne detection and prevention reporting in OpenAEV, addressing missed expectations described in #490.
Changes:
- Adds SaaS SDL v2 deep search and Unified Alerts correlation while retaining legacy Deep Visibility for self-hosted instances.
- Separates alert and file-event search windows, adds threat-event pagination, and introduces time-based retries for transient failures.
- Shares the expectation result model and adds configuration, documentation, and regression tests.
| File | Description |
|---|---|
sentinelone/tests/unit/services/test_expectation_service.py |
Tests search windows, retry timing, and missing signature data. |
sentinelone/tests/test_transient_threat_fetch.py |
Tests withholding verdicts after transient failures. |
sentinelone/tests/test_transient_threat_fetch_degrade.py |
Tests retry expiration across collection cadences. |
sentinelone/tests/services/test_self_hosted_detection.py |
Tests instance detection and probe fallback. |
sentinelone/tests/services/test_fetcher_unified_alerts.py |
Tests alert mapping, pagination, and errors. |
sentinelone/tests/services/test_fetcher_threat_events.py |
Tests cursor pagination and defensive limits. |
sentinelone/tests/services/test_fetcher_sdl.py |
Tests SDL queries, authentication, polling, and mapping. |
sentinelone/tests/services/test_expectation_service.py |
Tests window separation and alert-access handling. |
sentinelone/src/services/utils/signature_extractor.py |
Extracts the earliest valid start date. |
sentinelone/src/services/model_expectation_result.py |
Defines the shared result model with pending status. |
sentinelone/src/services/fetcher_unified_alerts.py |
Retrieves and maps Unified Alerts for correlation. |
sentinelone/src/services/fetcher_threat.py |
Adds an explicit request timeout. |
sentinelone/src/services/fetcher_threat_events.py |
Adds bounded cursor pagination and timeouts. |
sentinelone/src/services/fetcher_sdl.py |
Implements the SaaS SDL v2 search backend. |
sentinelone/src/services/fetcher_deep_visibility.py |
Uses the shared timeout constant. |
sentinelone/src/services/expectation_service.py |
Integrates backends, alerts, windows, and retry handling. |
sentinelone/src/services/exception.py |
Adds HTTP status metadata to API errors. |
sentinelone/src/services/client_api.py |
Adds transport retries and instance detection. |
sentinelone/src/models/configs/sentinelone_configs.py |
Defines alert, retry, and lookback settings. |
sentinelone/src/models/configs/config_loader.py |
Exposes new settings to daemon configuration. |
sentinelone/src/collector/models.py |
Re-exports the shared expectation result model. |
sentinelone/README.md |
Documents settings and Unified Alerts behavior. |
sentinelone/docker-compose.yml |
Passes new settings into the container. |
sentinelone/.dockerignore |
Excludes local virtual environments. |
Review effort: Balanced
| try: | ||
| response = self.session.get(endpoint, timeout=REQUEST_TIMEOUT_SECONDS) | ||
| response.raise_for_status() | ||
| accounts = response.json().get("data", []) |
There was a problem hiding this comment.
Checked against the SentinelOne v2.1 swagger: GET /web/api/v2.1/accounts returns data as an array of account objects (not data.accounts), so the iteration is correct. The malformed-body point is valid though: a non-list data (e.g. null) now raises inside the try and falls back to SaaS. Fixed in the latest commit.
| alert_start, alert_end = self._get_fetch_time_window(batch) | ||
| alert_threats, alert_events = ( | ||
| self.alerts_fetcher.fetch_alert_threats(alert_start, alert_end) |
There was a problem hiding this comment.
Agreed, this was the main cost issue. Fixed in 0ac1203:
- Unified Alerts are tenant-wide, so they are now listed once per collection cycle, over a window covering every expectation of the cycle, and shared by all batches. Matching is still per expectation on the implant name (inject id + agent id), so the wider window does not add false positives.
rawDatais fetched only when the list node does not already carry the implant name, and it is cached by alert id across cycles (bounded LRU, 5000 entries).- A non-auth failure is logged once and skipped for the rest of the cycle instead of being retried on every batch.
Tests: test_unified_alerts_fetched_once_per_cycle, test_detail_cached_across_calls, test_detail_not_fetched_when_implant_in_list_node.
| start_time = self._extract_start_date_from_batch(batch) | ||
| if start_time is None or start_time > end_time: | ||
| start_time = end_time - self.client_api.time_window |
There was a problem hiding this comment.
Agreed, fixed in the latest commit: each expectation resolves its own start (or the SENTINELONE_TIME_WINDOW fallback), and the window starts at the earliest of those.
| if index < MAX_DETAIL_FETCHES: | ||
| node["rawData"] = self._fetch_raw_data(node.get("id")) |
There was a problem hiding this comment.
Fixed in 0ac1203. MAX_DETAIL_FETCHES now bounds new detail fetches per call only. Details are cached by alert id, so the alerts left out by the cap are examined on the next cycle instead of never, and the run logs a warning with the number of alerts deferred. Together with the retry window, an older matching alert is reached within a few cycles even on a busy tenant. Covered by test_detail_cap_resumes_on_next_call.
| threat = SentinelOneThreat( | ||
| threat_id=str(alert_id), |
There was a problem hiding this comment.
Good catch, fixed in the latest commit. SentinelOneThreat now carries its source (threat / unified_alert) and the alert name, and TraceBuilder handles each one separately:
- Threats keep the
/incidents/threats/{id}/overviewlink. - Unified Alerts no longer get that broken link. SentinelOne does not document a console deep link for one alert (nothing in the v2.1 swagger or in their
ai-siemUAM references), so for now the trace links to the console root and its name carries the alert name, host and alert id (SentinelOne Unified Alert - Potential Mimikatz Execution - host (alert id …)). The deep-link path is a single constant,UNIFIED_ALERT_LINK_PATH, to fill once we confirm the URL from the console.
Covered by test_unified_alert_trace_does_not_use_threat_overview_link and test_threat_trace_keeps_threat_overview_link.
…probe, black 26 - Resolve each expectation's start date (or the SENTINELONE_TIME_WINDOW fallback) independently and take the earliest, so an expectation without a start date no longer narrows the window of the others. - Treat a non-list account listing body as a failed probe (defaults to SaaS) instead of raising outside the fallback handler. - Reformat with black 26 (what CI installs).
…t details - Unified Alerts are tenant-wide, so they are now listed once per collection cycle over a window covering every expectation, and shared by all batches (was: one listing + up to 200 detail POSTs per batch). Matching stays per expectation on the implant name. - rawData is fetched only when the list node does not already carry the implant name, and is cached by alert id across cycles (bounded LRU). - MAX_DETAIL_FETCHES now bounds new fetches only: alerts left out by the cap are examined on the next cycle instead of never. - A non-auth alerts failure is logged once and skipped for the cycle instead of being retried on every batch. - Tests: one fetch per cycle, cache reuse, skip when implant already known, cap resumes on next call, per-expectation window start.
A Unified Alert is not a Threat, so /incidents/threats/{id}/overview is a
broken link for it. SentinelOneThreat now carries its source ("threat" or
"unified_alert") and the alert name. TraceBuilder keeps the Threat overview
link for Threats; for Unified Alerts it links to the console root and puts
the alert name, host and alert id in the trace name. The alert deep-link
path is not documented by SentinelOne, so it is a single constant
(UNIFIED_ALERT_LINK_PATH) to fill once confirmed from the console.


Proposed changes
Closes #490.
In OAEV, with the OAEV agent as executor and the S1 collector, expectations for
injected payloads (for example "In-Memory Mimikatz-DumpCreds" in Atomic Testing)
never got their update in the console, even though S1's own log showed the
detection. The verdict came back as Not Detected or Not Prevented.
Scope: what this PR fixes, and what it does not
#490 mixed two different problems:
this PR. Injectors built on pyoaev's
SignatureManager(Nmap, Netexec,Nuclei, Email) wrote non-canonical signature types (
start_time,ports_discovered, ...). pyoaev built a collector's expectation list in onego, so a single invalid expectation made the whole fetch fail: 0
expectations processed per cycle, everything Expired. Fixed in pyoaev by
fix(signatures): send canonical signature types and skip invalid expectations (#352) client-python#364 (issue ci: realign branches and main and find a preproduction solution #352), shipped with collectors
3.261026.1+. On staging, the nightly ONCALL scenario (which also runs anNmap inject) went from Expired to Detected on the Linux EICAR inject on
2026-10-28, with no SentinelOne collector code change. The EICAR / bash case
from the issue comment therefore already works on
mainand is notevidence for this PR.
Behavioral / STAR / AI detections (Mimikatz, LSASS dump via
comsvcs.dll,...) only exist as Unified Alerts, which
mainnever reads. To validatethis PR, run a Mimikatz / LSASS-dump payload on a Windows endpoint that
has both the OpenAEV agent and a SentinelOne agent, after checking the
S1 console shows the alert. (Note: the "Dump LSASS.exe Memory using
comsvcs.dll" inject of the staging ONCALL scenario targets a CrowdStrike
host with no S1 agent, so its S1 Not Detected says nothing about this PR.)
On top of (2), this PR also fixes three gaps that can each produce a false
negative on their own: the deep-search backend on SaaS, the fetch windows, and
transient failure handling.
Deep-search backend (SaaS vs self-hosted). On SaaS, deep-search now runs on
the SDL v2 query API, a new
FetcherSDLthat launches and polls an async query,resumes via cursor, follows the forward-tag, and sends
Authorization: Bearerasthe SDL v2 contract requires. Self-hosted instances keep the existing Deep
Visibility 1.0 fetcher. The instance type is detected automatically: the client
probes the MGMT account listing (
GET /web/api/v2.1/accounts) once at init, anaccountTypefield means SaaS while its absence means self-hosted, and a failedprobe defaults to SaaS so a blip on a working account never flips it. The
service logs which backend it picked, once, at construction.
Fetch windows. The threat/alert window now ends at the current time and
starts from the
start_datesignature, or now minusSENTINELONE_TIME_WINDOWwhen there is no start date. The signature end date no longer anchors or
truncates that window, so a fresh alert can no longer land outside the range
that gets fetched. The DV/SDL file-event window is a separate, fixed lookback,
[now minus SENTINELONE_DEEP_VISIBILITY_LOOKBACK, now](new knob, defaultP1D), independent of the signature dates because a file can be dropped orexecuted long before the alert that references it. The threat-event fetcher
follows
nextCursorto the end of the result set (defensive 200-page cap)instead of reading only the first page, so a busy alert no longer has its file
events silently truncated.
Transient failure handling. A transient transport failure (a connection
drop, a timeout, a 408/429, a 5xx) no longer stamps a "Not Detected / Not
Prevented" verdict. The expectation is held pending: no verdict is emitted, it
stays unresolved in OAEV, and it is re-fetched on the next collector cycle, for
the length of a time-based
SENTINELONE_RETRY_WINDOW(new knob, defaultPT10M). The first attempt after that window has elapsed is the final one anddegrades to a negative verdict rather than staying open. That is the one-shot
blip from the issue, a single dropped connection being recorded as a verdict.
Permanent failures (validation, other 4xx) keep the immediate negative verdict.
On the transport side, every request the collector makes now carries an explicit
30 s timeout, and GET requests get a urllib3
Retry(429/5xx only, honorsRetry-After, backoff 0.5, at most 3 attempts).Unified Alerts. Behavioral / STAR / AI detections such as "Potential
Mimikatz Execution" are raised as SentinelOne Unified Alerts and never become
Threats, so a collector that reads only
/threatskeeps reporting NotDetected for them. A new
FetcherUnifiedAlertsqueries the Singularity alertsGraphQL API (
POST /web/api/v2.1/unifiedalerts/graphql, Bearer auth) over thethreat window, recovers the OpenAEV implant name from the alert's process
fields and OCSF
rawData, and maps each alert onto the existing threat + eventshape so the unchanged matcher correlates it. It is controlled by
SENTINELONE_ENABLE_ALERTS(default on, SaaS only, ignored on self-hosted).An alerts failure is isolated and never changes the Threats verdict. If the
token is refused (HTTP 401/403), the correlation is disabled for the run with a
single warning. Alerts are tenant-wide, so they are listed once per
collection cycle (window covering every expectation of the cycle) and shared
by all batches; per-alert
rawDatais only fetched when the listed fields donot already name the implant, and is cached by alert id across cycles.
Implementation notes
is_self_hostedis resolved on the client at construction. The expectationservice picks
FetcherDeepVisibility(DV 1.0) orFetcherSDL(SDL v2) onceat init and logs the choice.
failure_tracker/max_failure=5retry is replaced by atime-based lifecycle keyed on
first_attempt_at: the first failed attemptopens the window, attempts inside it are held, and the first attempt after it
is emitted. A valid result closes it too. The clock goes through a single
_now()seam._classify_error()mapsSentinelOneNetworkErrorto transient andSentinelOneValidationErrorto permanent, and readsrequestsHTTPErrorsby status: 408/429/5xx transient, other 4xx permanent, unknown transient. The
fetchers re-raise already-classified errors unchanged so the status code is
not buried in
__cause__and misclassified as permanent. Exhausted GETretries (
RetryError) are transient too.no daemon or manager contract change.
ExpectationResultgains anis_pendingflag and now has a single definition insrc/services/model_expectation_result.py, re-exported bysrc/collector/models.py. It used to be duplicated to avoid an import cycle.REQUEST_TIMEOUT_SECONDSconstant (client_api.py).SENTINELONE_RETRY_WINDOW(PT10M),SENTINELONE_DEEP_VISIBILITY_LOOKBACK(P1D) andSENTINELONE_ENABLE_ALERTS(
true). All three are optional with safe defaults, documented in the READMEconfig table, and passed through
docker-compose.yml. Since fix(collectors): example docker-compose files ignore the .env settings #646 that fileuses an explicit env list, so they would otherwise not reach the container.
main(merge, no force-push). The tests main addedunder
tests/unit/pass against this code.Testing
fetcher (launch/poll shape, cursor resume, forward-tag, Bearer auth), 5 BDD
tests for the transient lifecycle (no-verdict hold, plus degradation at 1-min,
2-min, and 1-hour cadences), plus tests for the alert-versus-DV window
decoupling, the pagination cap, and missing-signature-data not matching.
auto-disable (and a 500 that keeps it enabled), one fetch per cycle, detail
cache, skip when the implant is already known, cap resuming on the next
cycle, and the per-expectation window start.
python -m pytest(Python 3.14, pyoaev 3.261104.0): 121 passed.Checklist
I consider the submitted work as finished
I tested the code for its functionality
I added/updated tests
I updated the documentation (README config table, required permissions,
docker-compose env list)
Screenshot