Repository navigation
feat(expectation): add LOGGING expectation support for collectors (#8746) - #374
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified in the logging-expectation support and collector capability changes.
No open issues 🎉
What changed in this PR
This PR adds client-library support for LOGGING expectations, preparing collectors and injectors to use the platform’s new logging workflow while keeping existing collectors opted out by default.
Changes:
- Adds logging models, enums, signature extras, and an optional trace match field.
- Adds dedicated logging retrieval with a one-time warning on 404, plus explicit model mapping that skips unsupported types.
- Declares collector capabilities at registration and on pings, supports runtime updates, and adds regression tests.
| File | Description |
|---|---|
test/signatures/test_signature_manager_build_payload.py |
Tests logging signature groups and extra-data handling. |
test/daemons/test_collector_daemon.py |
Tests capability defaults, validation, registration, and runtime updates. |
test/contracts/test_contract_expectations.py |
Tests logging contract serialization. |
test/apis/inject_expectation/test_inject_expectation.py |
Tests logging retrieval, 404 fallback, and model filtering. |
test/apis/expectation/test_expectation.py |
Tests logging result labels and success updates. |
pyoaev/signatures/types.py |
Adds the logging signature expectation type. |
pyoaev/signatures/models.py |
Supports logging extras and returns empty extras for unknown types. |
pyoaev/daemons/collector_daemon.py |
Declares and updates supported expectation types. |
pyoaev/contracts/contract_config.py |
Adds logging to contract types and related documentation. |
pyoaev/apis/inject_expectation/model/expectation.py |
Defines the logging model and result labels. |
pyoaev/apis/inject_expectation/model/__init__.py |
Exports the logging model. |
pyoaev/apis/inject_expectation/inject_expectation.py |
Adds logging retrieval and shared response validation and model construction. |
pyoaev/apis/inject_expectation_trace.py |
Accepts the optional trace match type. |
pyoaev/apis/collector.py |
Allows supported expectation types in collector registration. |
Review effort: Balanced
Christophe Melchior (Kakudou)
left a comment
There was a problem hiding this comment.
A quality of life, and a little warning/question about the default _supported_expectation_types.
But globally a good contribution, thanks for the work done on that PR!
LGTM.
| self._supported_expectation_types = _validate_expectation_types( | ||
| DEFAULT_SUPPORTED_EXPECTATION_TYPES | ||
| if supported_expectation_types is None | ||
| else supported_expectation_types | ||
| ) |
There was a problem hiding this comment.
question: Is it intentional that every collector (including payload/vulnerability collectors like mitre-attack or nvd-nist-cve) always sends the legacy default?
gonna need to ensure that those collectors properly set supported expectation_types !
There was a problem hiding this comment.
I thought about it, but these collectors behave differently, if I remember correctly : the platform only seeds pending results for collectors that have a security platform (findAllByTenantIdAndSecurityPlatformIsNotNull), which the daemon only creates when collector_platform is set. And i f Iremember correctly, mitre-attack or nvd-nist-cve are never candidates.
And ["DETECTION","PREVENTION"] is exactly the platform's legacy default, so sending it or nothing behaves the same. Only collectors that answer LOGGING will need to set supported_expectation_types.
I'll merge as is, as you approved none the less, but we can definitely talk about it and create a followup issue if you see a bigger issue !
Proposed changes
LOGGINGexpectation type:LoggingExpectationmodel (labels Logged / Not Logged), andLOGGINGin the expectation, signature and contract enums, so injectors can offer it later (ADR-018 §4.4).GET /injects/expectations/logging/{source_id}(logging_expectations_for_source,logging_expectation_models_for_source). An older platform answers 404: it is treated as "no LOGGING expectation", with one warning (ADR-018 §4.2).expectations_models_for_sourcemaps DETECTION and PREVENTION explicitly and skips any other type with a warning, instead of building aPreventionExpectation. No change for existing collectors:/assets/{sourceId}only returns Prevention and Detection expectations.CollectorDaemondeclares the expectation types the collector supports (collector_supported_expectation_types) at registration and on every ping. Default: DETECTION + PREVENTION, so LOGGING is opt-in per collector.update_supported_expectation_typeschanges the list at runtime, e.g. to withdraw LOGGING when the log source becomes unusable.ExtraSignatureData.get_extrahandlesloggingand returns{}for unknown types. Before, nmap, nuclei and netexec would fail the whole signature payload, Detection signatures included, as soon as an inject carried a LOGGING expectation.inject_expectation_trace_match_type(marker,source_ip,target_ip).ai_expectations_for_sourcemoves to a shared helper (same messages, covered by the existing tests), the expectation models are built by one typed helper, andexpectations_assets_for_sourceis annotated as returning a list, which it always did.Testing Instructions
coverage run -m pytestthencoverage report -m --fail-under=80.isort --check-only .,black --check .,flake8 --ignore=E,W .Related issues
Checklist
Further comments
get_extrano longer raises, butExpectationSignatureGroupstill rejects an expectation type outside the signaturesExpectationTypeenum (e.g. MANUAL), so such a type would still failbuild_payload. Not an issue today: injector contracts only pass DETECTION, PREVENTION and now LOGGING. Filtering unknown types inbuild_payloadis out of scope.