Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Elastic 7.0.1 #382

Merged
merged 3 commits into from
May 7, 2019
Merged

Conversation

kg-ops
Copy link
Contributor

@kg-ops kg-ops commented Apr 14, 2019

@sdellenb
Copy link

sdellenb commented Apr 14, 2019

Just FYI - I've been running the stack on 7.0.0 since Friday and every night when Logstash wants to roll over the index, it stops working with this message:

[2019-04-14T00:00:01,445][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.04.14", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5b565b59>], :response=>{"index"=>{"_index"=>"logstash-2019.04.14", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"The [default] mapping cannot be updated on index [logstash-2019.04.14]: defaults mappings are not useful anymore now that indices can have at most one type."}}}}

I searched my configuration up and down and through the containers but I cannot find a defaults mapping setting. Yesterday, I created the logstash-2019.04.13 index manually and it worked again until the roll over tonight.
The index looks like this:

GET logstash-2019.04.13

{
  "logstash-2019.04.13" : {
    "aliases" : { },
    "mappings" : {
      "dynamic_templates" : [
        {
          "message_field" : {
            "path_match" : "message",
            "match_mapping_type" : "string",
            "mapping" : {
              "norms" : false,
              "type" : "text"
            }
          }
        },
        {
          "string_fields" : {
            "match" : "*",
            "match_mapping_type" : "string",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              },
              "norms" : false,
              "type" : "text"
            }
          }
        }
      ],
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "facility" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "geoip" : {
          "dynamic" : "true",
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        },
        "host" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "norms" : false
        },
        "port" : {
          "type" : "long"
        },
        "procid" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "programname" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "severity" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "sysloghost" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "syslogtag" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "provided_name" : "logstash-2019.04.13",
        "creation_date" : "1555149990370",
        "number_of_replicas" : "1",
        "uuid" : "RAvoGCxPSHqu4H8zO5hrlw",
        "version" : {
          "created" : "7000099"
        }
      }
    }
  }
}

I'm quite new to Elastic Search and use it to mirror my rsyslogs to a searchable database (so nothing critical is lost), but I find it quite frustrating to debug such issues.

@antoineco
Copy link
Collaborator

@sdellenb did you start a new Stack or upgrade from 6.7? My gut feeling is that the Logstash index template may have changed between the 2 versions and needs to be replaced.

@sdellenb
Copy link

I upgraded from 6.7, with the same changes as this PR (except the port). I'll check the index template.

@antoineco
Copy link
Collaborator

antoineco commented Apr 14, 2019

@sdellenb the release notes mention the field reference parser is now more strict. Are you using a specific plugin to ship your logs to ship your logs to Logstash?

I also found this in the Elasticsearch 7.0.0 documentation: Removal of mapping types.

Sounds like we should wait for the first patch version before we merge this update to the repo :)

@sdellenb
Copy link

sdellenb commented Apr 14, 2019

No plugin, just an rsyslog template generating a JSON for logstash.
I've managed to get it working again by updating the index template with a simplified version of the mappings of the current index as posted above. Thanks for the hint! 👍

On a different note regarding the upgrade to 7.0.0:
discovery.zen.minimum_master_nodes has been deprecated.
I'd have no idea how to address it, but so far it doesn't break anything.

@antoineco
Copy link
Collaborator

Thanks for the feedback, much appreciated 🙏 We'll have to address this in the default configuration indeed.

@antoineco antoineco mentioned this pull request May 4, 2019
@antoineco
Copy link
Collaborator

@kg-ops 7.0.1 is out. Would you mind updating the PR so we can test the upgrade path from 6.7.x?

@antoineco antoineco changed the title Bump version to Elastic 7.0.0 Update to Elastic 7.0.1 May 6, 2019
@antoineco antoineco force-pushed the patch/bump-to-7.0 branch 2 times, most recently from 92f86c6 to eacb5e9 Compare May 6, 2019 22:12
@antoineco antoineco merged commit 05e527c into deviantony:master May 7, 2019
DanBrown47 pushed a commit to DanBrown47/docker-elk that referenced this pull request Jun 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7.x support?
3 participants