CX-161773; Taegis Magic Pipe: Pivot process->netflow

[chrislangreo]

Overview

Implementing a "pivot" function that takes in a DataFrame containing aggregate process event data and returns a DataFrame with non-aggregated netflow events that are based on the columns in the aggregate event DataFrame.

How the New Function Works

The signature of the new function, process_pivot_netflow, is below:


process_pivot_netflow expects an input DataFrame that contains aggregate process event data that looks like the following:


The function will parse the columns in the input DataFrame to see which column names are present in a static list called NETFLOW_PIVOT_COLUMNS which contains ["host_id", "sensor_id", "sensor_type", "sensor_tenant", "tenant_id"]. Afterwards, a query will be generated against the netflow table based on which columns from the input DataFrame are present in NETFLOW_PIVOT_COLUMNS and the values in those columns. A query from example input DataFrame above looks like the following:



The function then returns a new DataFrame uncorrelated to the input DataFrame as it only contains netflow data that looks like the following: