ELFK (elasticsearch, logstash, filebeat, kibana) Log analysis solution for nginx server log and application log

1. First Install Elasticsearch, Logstash, Kibana, Filebeat
2. Run Elasticsearch and Kibana
3. Setup Logstash
4. Run Logstash and Filebeat

6.3.5

Filebeat collects the following log files:

1. Nginx:

• access.log
• error.log

2. Application:

• appplication.log contains log level DEBUG, INFO, ERROR
• error.log only contains log level ERROR

Convert unstructured logs into structured log files.

• Use spring boot default log format(Unstructured log)
• Use JSON format application log(Structured log)

1. Start the elk stack, navigate to the elk-configs folder

./filebeat -e -c filebeat.yml -d "publish"

2. Start the logstash

./logstash -f ../config/logstash.conf

3. Start the kibana

./kibana --allow-root

4. Start the logging application

mvn spring-boot:run 

5. Login to kibana at https://localhost:5601
6. Import indexes and other ELK config using elk-configs/elkSavedObjects.json