Repository navigation
feat(expectation): add LOGGING expectation support for collectors (#8756) - #374
Yann (impolitepanda) wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified in the logging-expectation support and collector capability changes.
No open issues 🎉
What changed in this PR
This PR adds client-library support for LOGGING expectations, preparing collectors and injectors to use the platform’s new logging workflow while keeping existing collectors opted out by default.
Changes:
- Adds logging models, enums, signature extras, and an optional trace match field.
- Adds dedicated logging retrieval with a one-time warning on 404, plus explicit model mapping that skips unsupported types.
- Declares collector capabilities at registration and on pings, supports runtime updates, and adds regression tests.
| File | Description |
|---|---|
test/signatures/test_signature_manager_build_payload.py |
Tests logging signature groups and extra-data handling. |
test/daemons/test_collector_daemon.py |
Tests capability defaults, validation, registration, and runtime updates. |
test/contracts/test_contract_expectations.py |
Tests logging contract serialization. |
test/apis/inject_expectation/test_inject_expectation.py |
Tests logging retrieval, 404 fallback, and model filtering. |
test/apis/expectation/test_expectation.py |
Tests logging result labels and success updates. |
pyoaev/signatures/types.py |
Adds the logging signature expectation type. |
pyoaev/signatures/models.py |
Supports logging extras and returns empty extras for unknown types. |
pyoaev/daemons/collector_daemon.py |
Declares and updates supported expectation types. |
pyoaev/contracts/contract_config.py |
Adds logging to contract types and related documentation. |
pyoaev/apis/inject_expectation/model/expectation.py |
Defines the logging model and result labels. |
pyoaev/apis/inject_expectation/model/__init__.py |
Exports the logging model. |
pyoaev/apis/inject_expectation/inject_expectation.py |
Adds logging retrieval and shared response validation and model construction. |
pyoaev/apis/inject_expectation_trace.py |
Accepts the optional trace match type. |
pyoaev/apis/collector.py |
Allows supported expectation types in collector registration. |
Review effort: Balanced
| optional=( | ||
| # Signal that matched, for a LOGGING expectation: "marker", | ||
| # "source_ip" or "target_ip". Left out for the other types. | ||
| "inject_expectation_trace_match_type", |
There was a problem hiding this comment.
would a small enum (marker, source_ip, target_ip) help collectors here? The platform silently drops an unknown value, so a typo would go unnotice
Christophe Melchior (Kakudou)
left a comment
There was a problem hiding this comment.
A quality of life, and a little warning/question about the default _supported_expectation_types.
But globally a good contribution, thanks for the work done on that PR!
LGTM.
| def expectations_assets_for_source( | ||
| self, source_id: str, expiration_time: int = None, **kwargs: Any | ||
| ) -> Dict[str, Any]: | ||
| ) -> List[Dict[str, Any]]: |
There was a problem hiding this comment.
nitpick (non-blocking): Now that this is annotated List[Dict] and _as_list_of_dicts exists, could /assets use it too? A malformed body would then raise OpenAEVParsingError instead of an AttributeError in _models_from_dicts. Fine as a follow-up.
| self._supported_expectation_types = _validate_expectation_types( | ||
| DEFAULT_SUPPORTED_EXPECTATION_TYPES | ||
| if supported_expectation_types is None | ||
| else supported_expectation_types | ||
| ) |
There was a problem hiding this comment.
question: Is it intentional that every collector (including payload/vulnerability collectors like mitre-attack or nvd-nist-cve) always sends the legacy default?
gonna need to ensure that those collectors properly set supported expectation_types !
Proposed changes
LOGGINGexpectation type:LoggingExpectationmodel (labels Logged / Not Logged), andLOGGINGin the expectation, signature and contract enums, so injectors can offer it later (ADR-018 §4.4).GET /injects/expectations/logging/{source_id}(logging_expectations_for_source,logging_expectation_models_for_source). An older platform answers 404: it is treated as "no LOGGING expectation", with one warning (ADR-018 §4.2).expectations_models_for_sourcemaps DETECTION and PREVENTION explicitly and skips any other type with a warning, instead of building aPreventionExpectation. No change for existing collectors:/assets/{sourceId}only returns Prevention and Detection expectations.CollectorDaemondeclares the expectation types the collector supports (collector_supported_expectation_types) at registration and on every ping. Default: DETECTION + PREVENTION, so LOGGING is opt-in per collector.update_supported_expectation_typeschanges the list at runtime, e.g. to withdraw LOGGING when the log source becomes unusable.ExtraSignatureData.get_extrahandlesloggingand returns{}for unknown types. Before, nmap, nuclei and netexec would fail the whole signature payload, Detection signatures included, as soon as an inject carried a LOGGING expectation.inject_expectation_trace_match_type(marker,source_ip,target_ip).ai_expectations_for_sourcemoves to a shared helper (same messages, covered by the existing tests), the expectation models are built by one typed helper, andexpectations_assets_for_sourceis annotated as returning a list, which it always did.Testing Instructions
coverage run -m pytestthencoverage report -m --fail-under=80.isort --check-only .,black --check .,flake8 --ignore=E,W .Related issues
Checklist
Further comments
get_extrano longer raises, butExpectationSignatureGroupstill rejects an expectation type outside the signaturesExpectationTypeenum (e.g. MANUAL), so such a type would still failbuild_payload. Not an issue today: injector contracts only pass DETECTION, PREVENTION and now LOGGING. Filtering unknown types inbuild_payloadis out of scope.