Filter not working

[D-o-c]

Hi All

i am trying to collect only logs from a specific Workload but the filter option is not working for me. To perform some tests i have merged the full config with the filter one without success.

Following my config:

log:  # Log settings. Debug will severely decrease performance
  path: 'collector.log'
  debug: True
collect:  # Settings determining which audit logs to collect and how to do it
  contentTypes:
    Audit.General: True
    Audit.AzureActiveDirectory: True
    Audit.SharePoint: True
    Audit.Exchange: False
    DLP.All: False
  rustEngine: True  # Use False to revert to the old Python engine. If running from python instead of executable, make sure to install the python wheel in the RustEngineWheels folder
#  schedule: 0 1 0  # How often to run in days/hours/minutes. Delete this line to just run once and exit.
  maxThreads: 50  # Maximum number of simultaneous threads retrieving logs
  retries: 3  # Times to retry retrieving a content blob if it fails
  retryCooldown: 3  # Seconds to wait before retrying retrieving a content blob
  autoSubscribe: True  # Automatically subscribe to collected content types. Never unsubscribes from anything.
  skipKnownLogs: True  # Remember retrieved log ID's, don't collect them twice
  resume: False  # Remember last run time, resume collecting from there next run
  hoursToCollect: 24  # Look back this many hours for audit logs (can be overwritten by resume)
filter:  # Only logs that match ALL filters for a content type are collected. Leave empty to collect all
  Audit.General:
    Policy: Spoof
  Audit.AzureActiveDirectory:
    Operation: UserLoginFailed
  Audit.SharePoint:
    Operation: FileDeleted
#  Audit.AzureActiveDirectory:
#  Audit.Exchange:
#  Audit.SharePoint:
#  DLP.All:
output:
  file:  # CSV output
    enabled: True
    separateByContentType: True  # Creates a separate CSV file for each content type, using file name from 'path' as a prefix
    path: 'output.csv'
    separator: ';'
    cacheSize: 500000  # Amount of logs to cache until each CSV commit, larger=faster but eats more memory
  azureLogAnalytics:
    enabled: False
    workspaceId:
    sharedKey:
    maxThreads: 50  # Maximum simultaneous threads sending logs to workspace
  azureTable:  # Provide connection string to executable at runtime with --table-string
    enabled: False
    tableName: AuditLogs  # Name of the table inside the storage account
    maxThreads: 10  # Maximum simultaneous threads sending logs to Table
  azureBlob:  # Write CSV to a blob container. Provide connection string to executable at runtime with --blob-string
    enabled: False
    containerName: AuditLogs  # Name of the container inside storage account
    blobName: AuditLog  # When separatedByContentType is true, this is used as file prefix and becomes e.g. AuditLog_AuditExchange.csv
    tempPath: './output'
    separateByContentType: True
    separator: ';'
    cacheSize: 500000  # Amount of logs to cache until each CSV commit, larger=faster but eats more memory
  sql:  # Provide connection string to executable at runtime with --sql-string
    enabled: False
    cacheSize: 500000  # Amount of logs to cache until each SQL commit, larger=faster but eats more memory
    chunkSize: 2000  # Amount of rows to write simultaneously to SQL, in most cases just set it as high as your DB allows. COUNT errors = too high
  graylog:
    enabled: False
    address:
    port:
  prtg:
    enabled: False
    channels:
  fluentd:
    enabled: False
    tenantName:
    address:
    port:

In the csv that I get i see all the entries of the Audit logs not only the filtered ones.

What am I missing?

Thx
Mattia

[ddbnl]

The config looks fine at first sight, might be a bug. I will try running with your config over the weekend and fix the bug if there's one. I'll let you know the result.

[ddbnl]

I've had time to look at this and noticed the issue. Filters are working, but the example configs contained an error. The 'filter' config is part of 'collect', so it should be indented. This was not the case in the example configs, so you could not have known this, apologies.

I will make a new commit to fix the examples. Below I will also copy paste the config you posted and fix the indentation. If you copy this config filtering should work for you. Could you let me know if this fixes your issue?

log:  # Log settings. Debug will severely decrease performance
  path: 'collector.log'
  debug: True
collect:  # Settings determining which audit logs to collect and how to do it
  contentTypes:
    Audit.General: True
    Audit.AzureActiveDirectory: True
    Audit.SharePoint: True
    Audit.Exchange: False
    DLP.All: False
  rustEngine: True  # Use False to revert to the old Python engine. If running from python instead of executable, make sure to install the python wheel in the RustEngineWheels folder
#  schedule: 0 1 0  # How often to run in days/hours/minutes. Delete this line to just run once and exit.
  maxThreads: 50  # Maximum number of simultaneous threads retrieving logs
  retries: 3  # Times to retry retrieving a content blob if it fails
  retryCooldown: 3  # Seconds to wait before retrying retrieving a content blob
  autoSubscribe: True  # Automatically subscribe to collected content types. Never unsubscribes from anything.
  skipKnownLogs: True  # Remember retrieved log ID's, don't collect them twice
  resume: False  # Remember last run time, resume collecting from there next run
  hoursToCollect: 24  # Look back this many hours for audit logs (can be overwritten by resume)
  filter:  # Only logs that match ALL filters for a content type are collected. Leave empty to collect all
    Audit.General:
      Policy: Spoof
    Audit.AzureActiveDirectory:
      Operation: UserLoginFailed
    Audit.SharePoint:
      Operation: FileDeleted
#  Audit.AzureActiveDirectory:
#  Audit.Exchange:
#  Audit.SharePoint:
#  DLP.All:
output:
  file:  # CSV output
    enabled: True
    separateByContentType: True  # Creates a separate CSV file for each content type, using file name from 'path' as a prefix
    path: 'output.csv'
    separator: ';'
    cacheSize: 500000  # Amount of logs to cache until each CSV commit, larger=faster but eats more memory
  azureLogAnalytics:
    enabled: False
    workspaceId:
    sharedKey:
    maxThreads: 50  # Maximum simultaneous threads sending logs to workspace
  azureTable:  # Provide connection string to executable at runtime with --table-string
    enabled: False
    tableName: AuditLogs  # Name of the table inside the storage account
    maxThreads: 10  # Maximum simultaneous threads sending logs to Table
  azureBlob:  # Write CSV to a blob container. Provide connection string to executable at runtime with --blob-string
    enabled: False
    containerName: AuditLogs  # Name of the container inside storage account
    blobName: AuditLog  # When separatedByContentType is true, this is used as file prefix and becomes e.g. AuditLog_AuditExchange.csv
    tempPath: './output'
    separateByContentType: True
    separator: ';'
    cacheSize: 500000  # Amount of logs to cache until each CSV commit, larger=faster but eats more memory
  sql:  # Provide connection string to executable at runtime with --sql-string
    enabled: False
    cacheSize: 500000  # Amount of logs to cache until each SQL commit, larger=faster but eats more memory
    chunkSize: 2000  # Amount of rows to write simultaneously to SQL, in most cases just set it as high as your DB allows. COUNT errors = too high
  graylog:
    enabled: False
    address:
    port:
  prtg:
    enabled: False
    channels:
  fluentd:
    enabled: False
    tenantName:
    address:
    port:

[D-o-c]

Sorry I tested it out just today :/
Anyway, many thanks, it works now! :)

[ddbnl]

No worries, happy its working now :)