Import fields.ecs.yml as generated by the ECS repo

[webmat]

This PR is based on elastic/ecs#379. Once this ECS issue is merged and backported to the 1.0 branch in ECS, a final export of the fields should be imported in this PR.

A few things of note:

• This new file doesn't include agent.hostname, which is not an ECS field. So this field definition has been moved to fields.common.yml.
• This generated file now includes user.* reusable fields under client, destination, server and source. They previously had been missed.
• This file now includes the geo fieldset at host.geo.* and observer.geo.*. They previously had been missed.
• This imports the various adjustments to definitions that came between ECS 1.0.0-beta2 and 1.0.0.
• The warning will mention ECS version 1.0.0 instead of 1.1.0-dev, once the file is generated from the ECS 1.0 branch.

Update

This is now based on the ECS v1.0 backport PR elastic/ecs#381.

This is ready for final review.

• Backport to 7.0 and 7.x

[webmat]

Here's a tip if you want to diff the previous file with this one. It doesn't help for the array sort order differences, but it helps for the YAML formatting differences and key order in YAML dictionaries.

Using Python 2.7 (so the dictionaries are sorted by key):

# on 7.0 branch
cat libbeat/_meta/fields.ecs.yml | python -c 'import sys; import yaml; print(yaml.dump(yaml.load(sys.stdin.read()),default_flow_style=False))' > ~/before.yml
# on this PR/branch
cat libbeat/_meta/fields.ecs.yml | python -c 'import sys; import yaml; print(yaml.dump(yaml.load(sys.stdin.read()),default_flow_style=False))' > ~/after.yml

[ycombinator]

Hey @webmat, this is the only PR (open or closed) in the beats repo that is using the needs backport label instead of the needs_backport label. I'm guessing that was a typo? Is it okay if we switch the label on this PR and then delete the needs backport label from the repo?

[webmat]

@ycombinator Nah, the space instead of the underscore is really important

[webmat]

I used "needs backport" in ECS, and I open my PRs with hub instead of the web interface.

Will adjust to needs_backport in my things :-) Thanks for pointing out

[webmat]

This is now based on the ECS v1.0 backport PR elastic/ecs#381.

This is ready for final review.

[ruflin]

LGTM. A follow up PR should be done to update the ecs version in each event.

One thing that keeps bugging me is that each Beat gets all ECS fields even if they are not used which leads to a much larger template then needed. This "should" not be a technical issue but I think it could be confusing for users to see fields in the template that are not used. The benefit of it is that users can't introduce fields which conflict with ECS in Beats with the rename processor for example. As you see, I'm on the fence. But this is for a later discussion, it just popped up because more fields are added here to each Beat.

[andrewkroh]

The vendored copy of the generated Go code should get updated too.

govendor fetch github.com/elastic/ecs/code/go/ecs@<tag>

[webmat]

@ruflin I'll open the follow-up PR shortly, to set the version on the events. Didn't want the massive amounts of test JSON files conflicting with all the other PRs to slow down the merge of this straightforward field update :-)

@andrewkroh The code generation hasn't been backported to 1.0 yet, but I should be able to get that done and open a PR here today.

[webmat]

Upon further inspection, it appears that libbeat/publisher/pipeline/module.go no longer sets the ECS version on the events.

Is this now solely based on the vendored Go ECS library's Version constant?

[ruflin]

@webmat Yes, looks like it depends on the go lib.