Aggregate functions in metrics

[mountaincode2]

I am reaching out in relation to the following metrics post:

https://www.googlecloudcommunity.com/gc/Community-Blog/New-to-Google-SecOps-Using-Metrics-in-YARA-L-...

I am a little confused here. In all the 4 parts in this series, why is the aggregate function `max` used.

For example, why is the `max` aggregate function used here instead of `min`:

 $min_byte_count_window = max(metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:min,
       principal.asset.ip: $ip
   ))

[jstoner]

To be candid, I started off writing an answer to your question, but as I did, I realize that the aggregation function at the front of the metric is superfluous. Between when I wrote the series and now, I'm not sure if something else changed, but I will provide some additional information here that hopefully clarifies things a bit.

For all rules that contain a match section, we need to aggregate each of the non-constant outcome variable statements, which is why we need to have the aggregation function of max (or min) on the front of the metric.

So, using that theory with metrics, I chose to use the max aggregation function. To be clear, because the metric itself has an aggregation occurring within it, we are getting a single value in the examples below and if we prepend either max or min to the metric, we would get the same results. Here are the aggregated max and min byte counts based on the metrics but with a different aggregation function prepended to them.

 outcome:
   $max_byte_count_window = max(metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:max,
       principal.asset.ip: $ip
   ))
   $min_byte_count_window = max(metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:min,
       principal.asset.ip: $ip
   ))
   $zmax_byte_count_window = min(metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:max,
       principal.asset.ip: $ip
   ))
   $zmin_byte_count_window = min(metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:min,
       principal.asset.ip: $ip
   ))

 Notice in the output, the results are identical because all the calculations occur within the metric; the max/min function at the beginning of the statement doesn't add anything and when I initially worked this up, I was thinking that I needed them to be syntactically correct.

This gets to the second part of this which is as I went back to revisit this today, I realized that in the case of metrics in this example, we don't need the aggregation function of max or min prepended in the outcome section, which makes sense since we have already reduced that value to a single value within the metric calculation. So my outcome section for the example in the blog could look like this, and be syntactically correct.

 outcome:
   $max_byte_count_window = metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:max,
       principal.asset.ip: $ip
   )
   $min_byte_count_window = metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:min,
       principal.asset.ip: $ip
   )
   $avg_byte_count = metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:avg,
       principal.asset.ip: $ip
   )
   $stddev_byte_count = metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:stddev,
       principal.asset.ip: $ip
   )
   $sum_byte_count_window = metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:sum,
       principal.asset.ip: $ip
   )
   $byte_count_days_seen = metrics.network_bytes_outbound(
       period:1d, window:30d,
       metric:value_sum,
       agg:num_metric_periods,
       principal.asset.ip: $ip
   )

And the output would look like this.

To be clear, leaving a min or max at the front of the metric in the outcome section will not break the rule nor will it change the result, but i can understand where confusion would be created when an aggregation in the metric is set for min but the aggregation function is set to max. I'm sorry for causing that.

Hope this helps!

[jstoner · Accepted solution]

One more thing, having gone back and looked at a few other examples in the other blogs in this series, the metrics that do not require the aggregation function prepended are the ones that contain metric: value_sum and do not contain an additional filter calculated within the metric. That isn't an exhaustive set of tests but hopefully some additional guidance that will help as you build rules using metrics.

[mountaincode2]

Hi @jstoner 

Thank you very much for the clarifications. This is very helpful. A few things i'd like to follow up and call out:

> For all rules that contain a match section, we need to aggregate each of the non-constant outcome variable statements, which is why we need to have the aggregation function of max (or min) on the front of the metric.

- Do you mean, for all the rules that contain an "outcome" section, we would need aggregate functions. Is that right? If not, what do you mean by: For all rules that contain a match section, we need to aggregate each of the non-constant outcome variable statements...

Could you break this part down.

> To be clear, because the metric itself has an aggregation occurring within it, we are getting a single value in the examples below and if we prepend either max or min to the metric, we would get the same results.

Got it. Yup, that makes sense now.

> To be clear, leaving a min or max at the front of the metric in the outcome section will not break the rule nor will it change the result, but i can understand where confusion would be created when an aggregation in the metric is set for min but the aggregation function is set to max.

- This is exactly where i was confused. Thank you for providing the clarification. 🙂

[jstoner]

> For all rules that contain a match section, we need to aggregate each of the non-constant outcome variable statements, which is why we need to have the aggregation function of max (or min) on the front of the metric.

- Do you mean, for all the rules that contain an "outcome" section, we would need aggregate functions. Is that right? If not, what do you mean by: For all rules that contain a match section, we need to aggregate each of the non-constant outcome variable statements...

Outcome sections have evolved over the past two years and when originally created, everything in them had to have an aggregation function associated with them, if I recall. That has evolved so that if you have an atomic detection, that is a single event rule with no aggregation, you can have outcome variables for a single event rule and because it is either passing a new value to a field or mapping a field like a userid to a field, you are essentially mapping a field, like this. Notice I have no match variable because I am not aggregating like events, each whoami is its own detection in this case. Here's the video covering this example of non-aggregated rules with outcomes. https://www.googlecloudcommunity.com/gc/Chronicle-Best-Practices/Getting-to-Know-Chronicle-Introduci...

rule whoami_execution {
  meta:
    author = "Google Cloud Security"

  events:
    $process.metadata.event_type = "PROCESS_LAUNCH"
    $process.target.process.command_line = "whoami"

  outcome:
    $risk_score = 10
    $mitre_attack_technique = "System Owner/User Discovery"
    $principal_user_userid = $process.principal.user.userid

  condition:
    $process
}

The exception to this rule in single event rules with no aggregation (match section) is when we have a repeating field like email address or IP address in the outcome section. Because I could have more than one IP address, the system needs to be told how to handle that repeated field. In most cases for a string, we can use something like this:

    $principal_ip = array_distinct($process.principal.ip)

For rules that contain a match section (aggregated), we can still use the outcome section, it's just that every outcome variable that contains a UDM field, repeated or otherwise must have an aggregation function prepended to it. So building on the example above, with a match section added, we add aggregation functions to the UDM fields but notice no on the risk score or technique since we are just passing constants through.

rule whoami_execution {
 
  meta:
    author = "Google Cloud Security"

  events:
    $process.metadata.event_type = "PROCESS_LAUNCH"
    $process.target.process.command_line = "whoami"
    $process.principal.hostname = $hostname

  match:
    $hostname over 5m

  outcome:
    $risk_score = 10
    $mitre_attack_technique = "System Owner/User Discovery"
    $target_process_file_sha256 = array_distinct($process.target.process.file.sha256)
    $principal_user_userid = array_distinct($process.principal.user.userid)
    $principal_ip = array_distinct($process.principal.ip)

  condition:
    $process
}

Hope this helps. For using outcome variables with aggregation in multiple event rules, there are a number of videos with different examples here for reference.

https://www.googlecloudcommunity.com/gc/Chronicle-Best-Practices/tkb-p/chronicle-best-practices

[mountaincode2]

This is very helpful. Thank you!