Splunk Nova provides cloud APIs for logging and analyzing your app. Fluentd is an open source data collector that decouples data sources from backend systems by providing a unified logging layer in between. This layer allows developers and data analysts to utilize many types of logs, as they are generated, and sends them directly to Splunk Nova.

Splunk Nova Cloud APIs with the fluentd plugin help you quickly collect your logs and events to make sense of data points in your apps and infrastructure.

1. Use the Fluentd plugin with Splunk Nova

Use the Splunk Nova Fluentd plugin to send events and logs directly to Splunk Nova. In minutes, you can easily query these events using the Splunk Nova-CLI.

2. Use the Splunk Nova Fluentd plugin with Kubernetes (K8s) and Docker

The Fluentd Nova plugin is containerized in a Docker image. This Docker image is used in a Kubernetes orchestrated cluster as a login system daemonset. The Fluentd Nova plugin gathers all logs from all running containers in a Kubernetes cluster. These logs are redirected to Splunk Nova and can be queried instantaneously.

Works best on macOS and Linux.

• Bundler
• Fluentd
• Homebrew
• Ruby
• Splunk Nova API Keys

1. Clone or download the Splunk Nova Fluentd plugin.

   [email protected]:splunknova/fluentd.git

2. Use homebrew to install ruby and bundler. Once these dependencies are installed, from the command line run:

   brew install ruby

3. Fetch and update bundled gems by running the following Bundler command:

   gem install bundler

4. Sign up or Log in to Splunk Nova which generates your API credentials.

You're now ready to configure the Fluentd plugin with your Splunk Nova API credentials.

1. Configure your out_nova.rb file by navigating to the plugin directory: lib > fluent > plugin directory.
2. Login to Splunk Nova and grab your API Keys.
3. Open the out_nova.rb file.
4. Configure the fluentd plugin, by editing the following values using your Splunk Nova api-username and Base-64 encoded token. Save and close the file.

Sample

• splunk_url: The Splunk Nova url https://api.splunknova.com:443.
• splunk_token: The Splunk token is your Base-64 encoded Nova API Key
• splunk_format: Then Splunk format nova by default
• splunk_url_path: The Splunk entry point /services/collector/event by default (<--this OR v1/events?)

Example

config_param :splunk_url,       :string,   :default => 'api.splunknova.com:443'
config_param :splunk_token,     :string    :default => 'QmFzZS02NCBFbmNvZGVkIFNwbHVuayBOb3ZhIEFQSSBLZXk='
config_param :splunk_format,    :string,   :default => 'nova'
config_param :splunk_url_path,  :string,   :default => '/v1/events'

Verify that the splunknova/fluentd plugin is configured correctly to communicate with Splunk Nova:

Run verify command here

Expected output:

Huzzah,it's working!

Profit!

Kubernetes, is an open source framework that orchestrates and automates container deployments. The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s is an abbreviation derived by replacing the 8 letters “ubernete” with “8”.

A daemonset is a K8s concept that is automatically deployed on each node of a K8s cluster. The Fluentd Nova plugin is used as a daemonset which runs a docker container. The daemonset ingests system and application logs and sends the data to Splunk Nova. One Docker container instance runs on each node of the cluster.

Fluentd K8s Input: The K8s plugin is a Fluentd input component that pulls logs.

Splunk Nova Output: The Splunk Nova plugin is a Fluentd output component that sends ingested data to Splunk Nova.

K8s Add-on (optional): K8s add-on are responsible for managing/config the daemonset through K8s API knowledge objects such asfield extraction, monitor dashboard, etc.

Features:

• Collects events and stats to correlate logs across containers.
• Delivers host specific logs allows monitoring of components in a cluster.
• Log collection uses JSON logging driver.
• Enriches logs with kubernetes metadata (container, image, pod, daemon sets, jobs, cron jobs, etc).

• Splunk Nova API Keys
• Access to the docker hub repo.
• kubectl is required, see Install and Set Up kubectl.
• A running Kubernetes cluster. If new to Kubernetes, see Hello Minikube, a kubernetes tutorial with Minikube.

1. To configure the Nova fluentd plugin Kubernetes daemonset, download and open the file: [fluentd-daemonset-nova.yaml]"https://raw.githubusercontent.com/splunknova/fluentd/master/docker_images/fluentd-daemonset-nova.yaml

2. Within the file, edit the SPLUNK_URL and SPLUNK_TOKEN values using your Splunk Nova API Keys. The SPLUNK_URL is your Splunk Nova api-username. The SPLUNK_TOKEN is your Base-64 encoded token. Save your edits, and close the file.

   - name:  SPLUNK_URL
     value: 'https://api.splunknova.com:443'
   - name:  SPLUNK_TOKEN
     value: "<YOUR BASE64 ENCODED API KEY>=="

3. From the command line, create a daemonset by running:

   kubectl create -f fluentd-daemonset-nova.yaml

4. Start monitoring your Kubernetes cluster.

To pull a snapshot of your K8a fluentd container, you may choose to create a docker image. Docker images are created with the build command, and produce a container when started with the run command. Images are stored in a Docker registry: https://hub.docker.com/.

From within the terminal, change directories into the splunknova/fluentd repo, and run:

docker pull splunknova/fluentd

• K8s Logging architecture
• K8s Cluster debugging
• Fluentd K8s logging
• Fluentd K8s logging daemon code

See review the guidelines for contributing to this repository.