WARNING: THIS IS WORK IN PROGRESS

The Elastic Common Schema (ECS) defines a common set of fields for ingesting data into Elasticsearch. A common schema helps you correlate data from sources like logs and metrics or IT operations analytics and security analytics.

ECS is still under development and backward compatibility is not guaranteed. Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the Contributing Guide.

The current version of ECS is 0.1.0.

• Fields
• Use cases
• Implementing ECS
• FAQ

ECS defines these fields.

• Base fields
• Agent fields
• Cloud fields
• Container fields
• Destination fields
• Device fields
• Error fields
• Event fields
• File fields
• Geo fields
• Host fields
• HTTP fields
• Log fields
• Network fields
• Organization fields
• Operating System fields
• Process fields
• Service fields
• Source fields
• URL fields
• User fields
• User agent fields

The base set contains all fields which are on the top level. These fields are common across all types of events.

The agent fields contain the data about the agent/client/shipper that created the event.

Examples: In the case of Beats for logs, the agent.name is filebeat. For APM, it is the agent running in the app/service. The agent information does not change if data is sent through queuing systems like Kafka, Redis, or processing systems such as Logstash or APM Server.

Fields related to the cloud or infrastructure the events are coming from.

Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.

Container fields are used for meta information about the specific container that is the source of information. These fields help correlate data based containers from any runtime.

Destination fields describe details about the destination of a packet/event.

Device fields are used to provide additional information about the device that is the source of the information. This could be a firewall, network device, etc.

These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.

The event fields are used for context information about the data itself.

File fields provide details about each file.

Geo fields can carry data about a specific location related to an event or geo information for an IP field.

Host fields provide information related to a host. A host can be a physical machine, a virtual machine, or a Docker container.

Normally the host information is related to the machine on which the event was generated/collected, but they can be used differently if needed.

Fields related to HTTP requests and responses.

Fields which are specific to log events.

Fields related to network data.

The organization fields enrich data with information about the company or entity the data is associated with. These fields help you arrange or filter data stored in an index by one or multiple organizations.

The OS fields contain information about the operating system. These fields are often used inside other prefixes, such as host.os.* or user_agent.os.*.

These fields contain information about a process. These fields can help you correlate metrics information with a process id/name from a log message. The process.pid often stays in the metric itself and is copied to the global field for correlation.

The service fields describe the service for or from which the data was collected. These fields help you find and correlate logs for a specific service and version.

Source fields describe details about the source of the event.

URL fields provide a complete URL, with scheme, host, and path. The URL object can be reused in other prefixes, such as host.url.* for example. Keep the structure consistent whenever you use URL fields.

The user fields describe information about the user that is relevant to the event. Fields can have one entry or multiple entries. If a user has more than one id, provide an array that includes all of them.

The user_agent fields normally come from a browser request. They often show up in web service logs coming from the parsed user agent string.

These are example on how ECS fields can be used in different use cases. Most use cases not only contain ECS fields but additional fields which are not in ECS to describe the full use case. The fields which are not in ECS are in italic.

Contributions of additional uses cases on top of ECS are welcome.

• APM
• Auditbeat
• Beats
• Filebeat Apache
• Kubernetes
• Logging
• Metricbeat
• TLS

• The document MUST have the @timestamp field.
• The data type defined for an ECS field MUST be used.
• It SHOULD have the field event.version to define which version of ECS it uses.
• As many fields as possible should be mapped to ECS.

Writing fields

• All fields must be lower case
• Combine words using underscore
• No special characters except _

Naming fields

• Present tense. Use present tense unless field describes historical information.
• Singular or plural. Use singular and plural names properly to reflect the field content. For example, use requests_per_sec rather than request_per_sec.
• General to specific. Organise the prefixes from general to specific to allow grouping fields into objects with a prefix like host.*.
• Avoid repetition. Avoid stuttering of words. If part of the field name is already in the prefix, do not repeat it. Example: host.host_ip should be host.ip.
• Use prefixes. Fields must be prefixed except for the base fields. For example all host fields are prefixed with host.. See dot notation in FAQ for more details.
• Do not use abbreviations. (A few exceptions like ip exist.)

Elasticsearch can index text multiple ways:

• text indexing allows for full text search, or searching arbitrary words that are part of the field.
• keyword indexing allows for much faster exact match filtering, prefix search, and allows for aggregations (what Kibana visualizations are built on).

By default, unless your index mapping or index template specifies otherwise (as the ECS index template does), Elasticsearch indexes text field as text at the canonical field name, and indexes a second time as keyword, nested in a multi-field.

Default Elasticsearch convention:

• Canonical field: myfield is text
• Multi-field: myfield.keyword is keyword

For monitoring use cases, keyword indexing is needed almost exclusively, with full text search on very few fields. Given this premise, ECS defaults all text indexing to keyword at the top level (with very few exceptions). Any use case that requires full text search indexing on additional fields can simply add a multi-field for full text search. Doing so does not conflict with ECS, as the canonical field name will remain keyword indexed.

ECS multi-field convention for text:

• Canonical field: myfield is keyword
• Multi-field: myfield.text is text

The only exceptions to this convention are fields message and error.message, which are indexed for full text search only, with no multi-field. These two fields don't follow the new convention because they are deemed too big of a breaking change with these two widely used fields in Beats.

Any future field that will be indexed for full text search in ECS will however follow the multi-field convention where text indexing is nested in the multi-field.

Despite the fact that IDs are often integers in various systems, this is not always the case. Since we want to make it possible to map as many data sources to ECS as possible, we default to using the keyword type for IDs.

The benefits to a user adopting these fields and names in their clusters are:

• Data correlation. Ability to easily correlate data from the same or different sources, including:
  • data from metrics, logs, and apm
  • data from the same machines/hosts
  • data from the same service
• Ease of recall. Improved ability to remember commonly used field names (because there is a single set, not a set per data source)
• Ease of deduction. Improved ability to deduce field names (because the field naming follows a small number of rules with few exceptions)
• Reuse. Ability to re-use analysis content (searches, visualizations, dashboards, alerts, reports, and ML jobs) across multiple data sources
• Future proofing. Ability to use any future Elastic-provided analysis content in your environment without modifications

The rename processor can help you resolve field conflicts. For example, imagine that you already have a field called "user," but ECS employs user as an object. You can use the rename processor on ingest time to rename your field to the matching ECS field. If your field does not match ECS, you can rename your field to user.value instead.

Events may contain fields in addition to ECS fields. These fields can follow the ECS naming and writing rules, but this is not a requirement.

There are two common key formats for ingesting data into Elasticsearch:

• Dot notation: user.firstname: Nicolas, user.lastname: Ruflin
• Underline notation: user_firstname: Nicolas, user_lastname: Ruflin

For ECS we decided to use the dot notation. Here's some background on this decision.

Ingesting user.firstname: Nicolas and user.lastname: Ruflin is identical to ingesting the following JSON:

"user": {
  "firstname": "Nicolas",
  "lastname": "Ruflin"
}

In Elasticsearch, user is represented as an object datatype. In the case of the underline notation, both are just string datatypes.

NOTE: ECS does not use nested datatypes, which are arrays of objects.

With dot notation, each prefix in Elasticsearch is an object. Each object can have parameters that control how fields inside the object are treated. In the context of ECS, for example, these parameters would allow you to disable dynamic property creation for certain prefixes.

Individual objects give you more flexibility on both the ingest and the event sides. In Elasticsearch, for example, you can use the remove processor to drop complete objects instead of selecting each key inside. You don't have to know ahead of time which keys will be in an object.

In Beats, you can simplify the creation of events. For example, you can treat each object as an object (or struct in Golang), which makes constructing and modifying each part of the final event easier.

In Elasticsearch, each key can only have one type. For example, if user is an object, you can't use it as a keyword type in the same index, like {"user": "nicolas ruflin"}. This restriction can be an issue in certain datasets. For the ECS data itself, this is not an issue because all fields are predefined.

Mixing the underline notation with the ECS dot notation is not a problem. As long as there are no conflicts, they can coexist in the same document.