feat(helper): have a better matching/correlation engine (a.k.a. replace the current helper)

[guzmud]

Backstory

Currently, pyoaev provides a helper named OpenAEVDetectionHelper, use through its main function match_alert_elements. This function takes as inputs on one-hand elements extracted from the expectations received from OpenAEV (called signatures in the codebase) and on the other hand elements extracted by the collector (called alert_data in the codebase).

Most of the time, the function will be piped into _match_alert_elements_original, where for each provided relevant signature, a match must be found in alert data of a similar type (whether through direct or fuzzy matching).

Case 0: 1-1 matching for a single signature type

If for a single type T, there is only one signature S of type T and only one alert data A of type T, the helper behaves as intuitively expected: True if A == S (or similar enough in case of fuzzy), False otherwise.

Case 1: 1-X matching for a single signature type

For a single type T, there is a single signature S1 and multiple alert data A1,...,Ax. In that case, according to the current codebase, the helper behaves as intuitively expected: True if one of the alert data Ak == S (or similar enough in case of fuzzy), False otherwise.
➡️ Currently, multiple alert data (of a single type) act as OR conditions.

Case 2: 1-1 matching for multiple signature types

For types T1 and T2, there are for each a single signature S1 and S2 respectively and a single alert data A1 and A2 respectively. According to the current codebase, the match will be done only if (S1 and A1 matches) and (S2 and A2 matches).
➡️ Currently, multiple signatures are treated as AND conditions.

Case 3: 1-X matching for multiple signature types

For multiple types T1 and T2, there are for each a single signature S1 and S2 respectively and multiple alert A11,...,A1x and A21,...,A2x respectively. The match is done following the or for alert data and the and between different types: ((S1 == A11) OR (S1 == A12) ... OR (S1 == A1x)) AND ((S2 == A21) OR (S2 == A22) ... OR (S2 == A2x))
➡️ TL;DR (match the single T1 signature with any T1 alert data) AND (match the single T2 signature with any T2 alert data)

Case 4: Y-X matching for a single signature type

For a single type T, there are multiple signature S1,...,Sy and multiple alert data A1,...,Ax. In that case, according to the current codebase, each signature is treated similarly to a different signature type: it must be matched too according to the alert data.
➡️ TL;DR (match each T signature with any T alert data)

So ... what is the issue?

• most, if not all, cases are X-Y matching with multiple values for multiple signature types
• control over whether alert data should be an AND or OR match has to be done outside the helper
• control over whether signatures should be an AND or OR match has to be done outside the helper
• there is no weight system to prioritize some values over others in a specific type
• there is no weight system to prioritize some signature types over others
• the fuzzy system is most of the time irrelevant and has been wrongly used in various collectors (e.g. fuzzy value on a properly formatted IP)
• while typing is missing from the codebase, data is implied to be a list otherwise the code may misbehave (e.g. the in having a different behavior for a in a list of [a,b,c] and a in a string this-is-not-a-please-do-not-match)

In the end, except from some very specific cases, most of the work has to be done outside of the helper and sometimes require to understand the helper internals in order to compensate for them. In some cases, the helper is entirely bypassed due to its mismatch with the needs / complexity to properly integrate into a real use case.

Issue examples

• all the internal and external IPs of an asset are provided as signature types: according to the codebase, by default, it means a collector should be able to match all those IPs to alert data in order to match those expectations
• fuzzy matching has been used on IDs and IPs properly parsed from the source data, leaving the door open to collision during analysis/matching (e.g. between two IPs from the same subnetwork)
• multiple collectors limit themselves to the parent process name (limiting the X-Y matching cases) and/or signatures that tend to not be provided by injectors
• multiple collectors implement a pre-helper matching using hardcoded oaev-implant- strings to filter out alert data (reinforcing the focus on the implant, the requirement for a parent process, and hardcoding a value that could change)

Use case

Matching data provided by the OAEV platform and gathered by the collectors to establish correlation between an offensive event and a defensive event.

Current workaround

Either:

• do not use the helper in the collectors
• broken collectors working only under very specific conditions

Proposed solution

?

Additional information