nsg-parser

NOTICE: ALPHA

This project is currently in alpha. All functionality is subject to change. Major refactoring is still ahead.

Purpose

This tool is being developed in order to convert Azure NSG Flow logs into other formats.

Some background on NSG Flow Logs can be found here. https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview

Go (golang) was choosen for its performance, cross-platform capabilities, and the great Azure SDK for GO https://github.com/Azure/azure-sdk-for-go/tree/master/storage

Features

Basic

Convert NSG Flow Event logs to flat local JSON files.
Send NSG Flows Event logs to remote CEF Syslog.
Cross-Platform (Windows, OSX, Linux)
Can run as daemon
Can be installed as a service on Windows/Linux
Provides HTTP endpoint for status and metric information (WIP)

Implementation

Processing status is persisted to disk, keeping track of changes.
Blob contents are paged to reduce loading the same parts of the Flow JSON multiple times.
Processing can be interrupted and restarted at any time.

Config

Base Config is stored in nsg-parser.yml file. Most options can be overriden on the command line. Path to this config file is provided on the command-line with --config

Example:

# Base Azure settings.
storage_account_name: oivsjvoisjvoisjdvosdv
storage_account_key: secretSquirrelKey
container_name: insights-logs-networksecuritygroupflowevent
# Prefix is not required, but can be useful if sharing containers.
prefix: resourceId=/SUBSCRIPTIONS/SUBID/RESOURCEGROUPS/RGNAME/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/SOMENSG-NSG/y=2017/m=06/d=06
# User must have write privileges to this directory. Must be a full path.
data_path: L:\nsg-data\
# Serve HTTP Process/Metrics endpoint at /status. NOT Secure
serve_http: true
serve_http_bind: 127.0.0.1:3000
# How often do we poll? Less than 60 seconds is pointless since NSG Flows are paged by the minute.
poll_interval: 60
# Set begin_time here to ignore any Blobs stamped before this hour.
# Failing to set this sensibly could result in processing huge amounts of data.
begin_time: 2017-06-20-11
# file or syslog
destination: file
# syslog settings are required for syslog destination only
syslog_protocol: tcp
syslog_host: 127.0.0.1
syslog_port: 5514
# Equivalent to setting http_proxy and https_proxy variables. Useful for service config.
http_proxy: http:https://10.4.3.2:2222

Logging

Log Path: data_path

Logs are patterned and rotated every hour

nsg-parser-%Y%m%d%H%M.log

Limited logging to Stdout

Process to File

destination: file

Processing to file will convert the Azure NSG Flow logs into a flat format. Each file will be formatted with

nsgLog-NSGNAME-HOURTIME-STARTTIMESTAMP-ENDTIMESTAMP

Example:

nsgLog-NSGNAME-201706201400-1497967953-1497968013

See: NsgFlowLog in parser\types.go

Sample Object:

{
    "time": 1497967953,
    "systemId": null,
    "category": null,
    "resourceId": "/SUBSCRIPTIONS/SUBID/RESOURCEGROUPS/RGRP/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/MYNSG",
    "operationName": null,
    "rule": "UserRule_HTTP",
    "mac": "00:01:11:14:38:14",
    "sourceIp": "10.44.1.8",
    "destinationIp": "10.55.11.4",
    "sourcePort": "23653",
    "destinationPort": "80",
    "protocol": "T",
    "trafficFlow": "I",
    "traffic": "A"
  }

Process to Syslog:

destination: syslog

Send events to remote syslog. Events are sent once only.

All Syslog is formatted as CEF

See here for documentation:

https://community.saas.hpe.com/t5/ArcSight-Connectors/ArcSight-Common-Event-Format-CEF-Guide/ta-p/1589306

Format is

timestamp host CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]

Example

Jun 21 13:15:34|CEF:0|Microsoft|Azure NSG|1|nsg-flow|nsg-flow|0|cs1=UserRule_HTTP deviceDirection=0 dmac=00:0D:3A:F3:38:54 dpt=80 dst=10.193.160.4 outcome=Allow proto=TCP spt=18166 src=10.199.1.8 start=1498065334000

Sample Config

storage_account_name: oivsjvoisjvoisjdvosdv
storage_account_key: secretSquirrelKey
container_name: insights-logs-networksecuritygroupflowevent
data_path: L:\nsg-data-syslog\
serve_http: true
serve_http_bind: 127.0.0.1:3000
poll_interval: 60
destination: syslog
http_proxy: http:https://142.107.185.64:2222
begin_time: 2017-06-20-11
syslog_protocol: udp
syslog_host: 142.107.186.142
syslog_port: 514

Run:

λ nsg-parser.exe process --config l:\syslog-nsg.yml
INFO[0000] loaded config file                            config_file="l:\syslog-nsg.yml"
INFO[0000] started logging                               current_file="L:\nsg-data-syslog\nsg-parser-201706201100.log" logLevel=info path="L:\nsg-data-syslog\nsg-parser-%Y%m%d%H%M.log"

In the Log File:

time="2017-06-20T11:14:03-04:00" level=info msg="started logging" logLevel=info path="L:\nsg-data-syslog\nsg-parser-%Y%m%d%H%M.log" 
time="2017-06-20T11:14:03-04:00" level=info msg="using proxy" proxy="http:https://142.107.185.64:2222" 
time="2017-06-20T11:14:03-04:00" level=info msg="serving nsg-parser status  on HTTP" Host="127.0.0.1:3000" 
time="2017-06-20T11:14:04-04:00" level=info msg="processing new blob" LastModified=2017-06-20 13:01:16 +0000 GMT LastProcessedRecord=0001-01-01 00:00:00 +0000 UTC Nsg=NSG-NAME ShortName=NSG-NAME-2017-06-20-12 
time="2017-06-20T11:14:04-04:00" level=info msg="processing new blob" LastModified=2017-06-20 14:01:16 +0000 GMT LastProcessedRecord=0001-01-01 00:00:00 +0000 UTC Nsg=NSG-NAME ShortName=NSG-NAME-2017-06-20-13 
time="2017-06-20T11:14:04-04:00" level=info msg="processing new blob" LastModified=2017-06-20 15:01:19 +0000 GMT LastProcessedRecord=0001-01-01 00:00:00 +0000 UTC Nsg=NSG-NAME ShortName=NSG-NAME-2017-06-20-14 
time="2017-06-20T11:14:04-04:00" level=info msg="processing new blob" LastModified=2017-06-20 15:13:17 +0000 GMT LastProcessedRecord=0001-01-01 00:00:00 +0000 UTC Nsg=NSG-NAME ShortName=NSG-NAME-2017-06-20-15 
time="2017-06-20T11:14:04-04:00" level=info msg="LoadBlobRange()" end=378921 start=0 
time="2017-06-20T11:14:04-04:00" level=info msg="LoadBlobRange()" end=379077 start=0 
time="2017-06-20T11:14:05-04:00" level=info msg="LoadBlobRange()" end=379077 start=0 
time="2017-06-20T11:14:05-04:00" level=info msg="LoadBlobRange()" end=75641 start=0 
time="2017-06-20T11:14:05-04:00" level=info msg="processing completed" type=parser.SyslogClient

Running as a Service.

This is a WIP. There are some outstanding stability/restart tests to be done.

nsg-parser can be installed as a service. When installing as a service a config file MUST be used.

λ nsg-parser.exe --config l:\syslog-nsg.yml process service --help
Manage nsg-parser service

Usage:
  nsg-parser process service [flags]
  nsg-parser process service [command]

Available Commands:
  install     Install/Reinstall nsg-parser service
  run         run nsg-parser service
  uninstall   Uninstall nsg-parser service

Flags:
  -h, --help                         help for service
      --service_description string   Service Description (default "Parser for MS Azure NSG Flow Logs")
      --service_name string          Service Name (default "nsg-parser")

Example on Windows

λ nsg-parser.exe --config l:\syslog-nsg.yml process service install
INFO[0000] loaded config file                            config_file="l:\syslog-nsg.yml"
INFO[0000] started logging                               current_file="L:\nsg-data-syslog\nsg-parser-201706201100.log" logLevel=info path="L:\nsg-data-syslog\nsg-parser-%Y%m%d%H%M.log"
INFO[0000] installing service with config                config_file="l:\syslog-nsg.yml"
INFO[0000] installed service

λ net start nsg-parser
The nsg-parser service is starting.
The nsg-parser service was started successfully.

Get Status over HTTP

An optional feature. This is a WIP, will be split out into metrics and status.

Example Contents:

{
  "GoVersion": "go1.8.3",
  "Version": "0.0.3",
  "ProcessStatus": {
    "resourceId=/SUBSCRIPTIONS/SUBID/RESOURCEGROUPS/RGNAME/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-NAME/y=2017/m=06/d=20/h=14/m=00/PT1H.json": {
      "name": "resourceId=/SUBSCRIPTIONS/SUBID/RESOURCEGROUPS/RGNAME/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-NAME/y=2017/m=06/d=20/h=14/m=00/PT1H.json",
      "etag": "0x8D4B7ED358F55A9",
      "last_modified": "2017-06-20T15:01:19Z",
      "last_processed": "2017-06-20T11:14:05.7873103-04:00",
      "last_processed_record": "2017-06-20T14:59:35.479Z",
      "last_processed_time": 1497970773,
      "last_count": 60,
      "last_processed_range": {
        "Start": 0,
        "End": 379077
      },
      "log_time": "2017-06-20T14:00:00Z",
      "nsg_name": "NSG-NAME"
    },
    "resourceId=/SUBSCRIPTIONS/SUBID/RESOURCEGROUPS/RGNAME/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-NAME/y=2017/m=06/d=20/h=15/m=00/PT1H.json": {
      "name": "resourceId=/SUBSCRIPTIONS/SUBID/RESOURCEGROUPS/RGNAME/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-NAME/y=2017/m=06/d=20/h=15/m=00/PT1H.json",
      "etag": "0x8D4B7F08F2A695D",
      "last_modified": "2017-06-20T15:25:18Z",
      "last_processed": "2017-06-20T11:25:36.8141728-04:00",
      "last_processed_record": "2017-06-20T15:23:35.472Z",
      "last_processed_time": 1497972213,
      "last_count": 1,
      "last_processed_range": {
        "Start": 145186,
        "End": 151513
      },
      "log_time": "2017-06-20T15:00:00Z",
      "nsg_name": "NSG-NAME"
    }
  },
  "BuildDate": "20170620-15:13:46",
  "BuildUser": "bobthebuilder@itsdtojadim2022",
  "Revision": "acffa303b7c9587e678d60e1281d021ca93685b7",
  "ProcessedFlowCount": 973
}

Install from Binary

Only Windows binaries are being provided for the time being.

Windows Powershell

The following will download and extract the required files.

You must still edit/create the nsg-parser.yml file before using.

$url = "https://github.com/dimitertodorov/nsg-parser/releases/download/v0.0.4/nsg-parser-0.0.4.windows-amd64.zip"

$basePath = "l:\latest-nsg-parser\"
$dataPath = "$basePath\data"
$env:DATA_PATH=$dataPath

New-Item -ItemType Directory -Force -Path $basePath
New-Item -ItemType Directory -Force -Path $dataPath

$filePath = "$basePath\nsg-parser.zip"
Invoke-WebRequest -Uri $url -OutFile $filePath
Add-Type -assembly “system.io.compression.filesystem”
[io.compression.zipfile]::ExtractToDirectory($filePath, $basePath)

Building

To build on Windows see scripts\build_windows.ps1

To build on UNIX see Makefile

e.g.

make build

HPE Arcsight Integration

Primary driver behind developing this was to integrate Azure NSG into our Arcsight Logging environment.

The syslog destination uses CEF Syslog by default.

Currently, we are encountering an issue where the event timestamp is being overwritten somewhere along the way.

event.startTime is being preserved though.

TODO

Process NetworkSecurityGroupEvent (Not Flow)
Add other destination clients (LogStash, Encrypted Syslog)
More Tests (Mock Azure?)

Contributions

Any suggestions/contributions are welcome.

Name		Name	Last commit message	Last commit date
Latest commit History 44 Commits
cmd		cmd
parser		parser
pool		pool
scripts		scripts
testdata		testdata
vendor		vendor
.cobra.yaml		.cobra.yaml
.gitignore		.gitignore
.promu.yml		.promu.yml
LICENSE.txt		LICENSE.txt
Makefile		Makefile
README.md		README.md
VERSION		VERSION
glide.lock		glide.lock
glide.yaml		glide.yaml
main.go		main.go
nsg-parser.yml.sample		nsg-parser.yml.sample

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

nsg-parser

NOTICE: ALPHA

Purpose

Features

Basic

Implementation

Config

Logging

Process to File

Process to Syslog:

Sample Config

Running as a Service.

Example on Windows

Get Status over HTTP

Install from Binary

Windows Powershell

Building

HPE Arcsight Integration

TODO

Contributions

About

Releases

Packages

Languages

License

dimitertodorov/nsg-parser

Folders and files

Latest commit

History

Repository files navigation

nsg-parser

NOTICE: ALPHA

Purpose

Features

Basic

Implementation

Config

Logging

Process to File

Process to Syslog:

Sample Config

Running as a Service.

Example on Windows

Get Status over HTTP

Install from Binary

Windows Powershell

Building

HPE Arcsight Integration

TODO

Contributions

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages