The Guardium universal connector enables Guardium Data Protection and Guardium Insights to get data from potentially any data source's native activity logs without using S-TAPs. It includes support for various plug-in packages, requiring minimal configuration. You can easily develop plug-ins for other data sources and install them in Guardium.

The captured events embed messages of any type that is supported by the configured data source. That includes: information and administrative system logs (e.g., login logs, various data lake platform native plug-in related data), DDLs and DMLs, errors of varying subtypes, etc. The incoming events received by the universal connector can be configured to arrive either encrypted or as plain text.

Figure 1. Guardium universal connector architecture

_{Data flow from input plugin to guardium sniffer in Guardium Data Protection}

The Guardium universal connector supports many platforms and connectivity options. It supports pull and push modes, multi-protocols, on-premises, and cloud platforms. For the data sources with pre-defined plug-ins, you configure Guardium to accept audit logs from the data source.

For data sources that do not have pre-defined plug-ins, you can customize the filtering and parsing components of audit trails and log formats. The open architecture enables reuse of prebuilt filters and parsers, and creation of shared library for the Guardium community.

The Guardium universal connector identifies and parses the received events, and converts them to a standard Guardium format. The output of the Guardium universal connector is forwarded to the Guardium sniffer on the collector, for policy and auditing enforcements. The Guardium policy, as usual, determines whether the activities are legitimate or not, when to alert, and the auditing level per activity.

The Guardium universal connector is scalable. It provides load-balancing and fail-over mechanisms among a deployment of universal connector instances, that either conform to Guardium Data Protection as a set of Guardium Collectors, or to Guardium Insights as a set of universal connector pods. The load-balancing mechanism distributes the events sent from the data source among a collection of universal connector instances installed on the Guardium endpoints (i.e., Guardium Data Protection collectors or Guardium Insights pods). For more information, see Enabling Load-Balancing and Fail-Over.

Connections to databases that are configured with the Guardium universal connector are handled the same as all other datasources in Guardium. You can apply policies, view reports, monitor connections, for example.

Under the hood, the universal connector is a Logstash pipeline comprised of a series of three plug-ins:

1. Input plug-in. This plug-in ingests events. Depending on the type of plug-in, there are settings to either pull events from APIs or receive a push of events.

2. Filter plug-in. This plug-in filters the events captured by the input plug-in. The filter plug-in parses, filters, and modifies event logs into a Guardium-digestible format.

3. Output plug-in. This plug-in receives the formatted event logs from the filter plug-in and transmits them to IBM Guardium (either Guardium Data Protection or Guardium Insights).

Note: the output plug-in is presented here as an internal component of the universal connector pipeline and is not to be accessed or modified by the user.

Universal Connector plug-ins are packaged and deployed in a Docker container environment.

There are a couple of flavors aimed at enabling audit log forwarding into Guardium for various data sources, comprised of either a cloud or on-premise data lake platform, of a database type that is supported by the Guardium sniffer¹:

1. Utilize the out-of-the-box, pre-installed plug-in packages² that require minimal configuration on the client's end by either plugging suited values into their respective template configuration files in the input and filter sections, or by adding a Ruby code subsection to the said filter section in case a more complex parsing method is necessary as a pre-processing stage to be executed prior to the execution of the respective filter plug-in. See each plug-in's user manual via Available Plug-ins.

2. For data sources that are not yet supported, you can either upload an IBM-approved filter plug-in or develop your own and add it to our plug-in repository. You can also clone and modify the existing plug-ins as a template for your convenience (either in Ruby or Java)³. You can optionally either let the parsing operations be executed by your filter plug-in, or assign this task to the Guardium Sniffer by transferring the event to the output plug-in in a designated structure as part of the filter plug-in development, as instructed in the links in the Developers Guide.

1. The pre-defined and pre-installed plug-ins do not require any manual uploads or other such prerequisites on the user's end, as opposed to user-made plug-ins or other available Logstash plug-ins. You can simply use a ready-made template for plugging in values to the input and filter sections of their respective configuration files, expand these sections by using online pre-installed Logstash plug-ins, or write your own Ruby code parser using the Ruby filter plug-in as a pre-processing stage prior to executing the filter plug-ins.
2. It is recommended to use one of the input plug-ins already in the repository and modify its config file input section. But if the input plug-ins already in the repository are insufficient for your needs, you can add a new one.
3. You can choose to configure either pull or push methods via the messaging middleware service installed on the data lake platform that is used by the input plug-in. Messages can be received with pull or push delivery. In pull mode, the universal connector instance initiates requests to the remote service to retrieve messages. In push mode, the remote service initiates requests to the universal connector instance to deliver messages.
4. The specific audit log types transmitted into the universal connector from the data source are configurable via the SQL instance settings installed on the data lake platform. This can vary depending on the installed data lake platform native plug-ins and the utilized messaging middleware service⁴.
5. For some data lake platforms, you can define inclusion and exclusion filters for the events routed to the universal connector to be ingested by the input plug-in. This can result in a more efficient filtering implemented either as part of the filter scope in the connector's configuration file, or in the developed filter plug-in.

When you use the built-in features of both Guardium Data Protection and Guardium Insights, you might inadvertently distribute the entire set of received events to each Guardium instance, which can result in duplicated and redundant event processing. To prevent this default behavior and ensure efficient operation, it is recommended to configure these mechanisms as part of the input scope in the configuration file of the installed connector. You can achieve this configuration through both pull (Pub/Sub⁵, JDBC⁶, SQS⁷) and push (Filebeat⁸) methods. It's important to note that when using the push method in Guardium Data Protection, configuring the entire set of collectors as part of the input scope is necessary. For more detailed information about each plug-in, please refer to the Available Plug-ins page.

In Guardium Data Protection, the overall workflow for deploying the universal connector is as follows:

1. Installing desired policies as instructed in Policies

2. Install and configure a plugin⁹.

3. Configuring native auditing¹⁰ on the data source

4. Sending native audit logs to the universal connector, using either a push or pull workflow.

5. Configuring the universal connector to read the native audit logs.

More detailed information about the workflow for Guardium Data Protection can be found here.

In Guardium Insights, the workflow for deploying the universal connector is slightly different, and can be found here

Note that the specific steps for each workflow may differ slightly per different data sources. See our list of available plugins to view detailed, step-by-step instructions for each supported data source/plug-in.

• You can optionally use a Guardium client installed on a database running on your local host for forwarding native audit logs into Universal Connector via Filebeat or Syslog¹¹. See Using GIM for more information.

• On how to configure Universal Connector for various data sources via AWS, see Using AWS

• On how to configure sample data sources and forward the generated audit log events into Universal Connector via Syslog or Filebeat, see Sample data sources Configurations via Filebeat and Syslog

• To see a sample configuration of MySQL with Filebeat, see Configuring SSL with Syslog

• To see suggested configurations for optimized database performance. see here

• To see a sample for changing connector protocol from TCP/UDp to SSL, see Changing the MongoDB Filebeat connector protocol from TCP or UDP to SSL

The Universal connector is monitored via tools that are already familiar to Guardium Data Protection and Guardium Insights users. As well as some unique tools that can be found in the following links.

Monitoring UC connections in Guardium Data Protection

Monitoring UC connections in Guardium Insights

With a few exceptions, using data from the universal connector is no different from using data from any other source in Guardium Data Protection or Guardium Insights. For using the universal connector in Guardium Data Protection, there are a few unique policies that can be found in this link:

Configuring Policies for the universal connector

For more general information about policies, refer to our Guardium Data Protection and Guardium Insights policy documentation.

Please note: limitations associated with specific datasources are described in the universal connector plug-in readme files for each datasource. See Available Plug-ins for more information.

• When configuring universal connectors, only use port numbers higher than 5000. Use a new port for each future connection.

• Use only the packages that are supplied by IBM. Do not use extra spaces in the title.

• IPV6 support

  • S3 SQS and S3 Cloudwatch plug-ins are not supported on IPV6 Guardium systems.
  • The DynamoDB plug-in does not support IPV6.

• Native MySQL plug-in¹²:

  • do not send the database name to Guardium if the database commands are performed by using MySQL native client.
  • When connected with this plug-in, queries for non-existent tables are not logged to GDM_CONSTRUCT.

• MongoDB plug-ins do not send the client source program to Guardium.

• Guardium recommends that you do not configure more than 10 universal connectors on a single Guardium collector.

Here is a list of frequently asked questions and troubleshooting sections for Guardium Data Protection.

Here is a list of frequently asked questions and troubleshooting sections for Guardium Insights.

Note: For further plug-in designated troubleshooting, see "troubleshooting" section in the plug-in's documentation linked at Available Plug-ins

Users can develop their own universal connector plugins, if needed, and contribute them back to the open source project, if desired.

(In order to overwrite old plug-ins, you can upload a new version from the official IBM GitHub page. Please make sure that the new plug-in has the exact same name as the old version.)

Here is a guide for developing new plug-ins for Guardium Data Protection.

Here is a guide for developing new plug-ins for Guardium Insights.

For adding a parser to the filter section of the configuration file as a pre-processing stage prior to executing the filter plug-in, use the Ruby filter plugin.

• For developing a Ruby filter plug-in, use How to write a Logstash filter plugin
• For developing a Java filter plug-in, use How to write a Java filter plugin

• For developing a Ruby input plug-in, use How to write a Logstash input plugin
• For developing a Java input plug-in, use How to write a Java filter plugin

Note: It is the developer's responsibility to maintain and update the database's supported versions

Useful links:

• Integrate Code Coverage tool into Universal Connector Plug-ins

To make your connector plug-in available to the community, submit your connector to this repository for IBM Certification. We also accept updates or bug fixes to existing plug-ins, to keep them current:

• Guidelines for contributing

• Benefits include:

  • Free, comprehensive testing and certification.

  • Expanding the reach of product APIs.

  • Driving usage of a product or solution.

If you find any problems or want to make suggestions for future features, please create issues and suggestions on GitHub.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.