Custom parser module for logster used to collect and process performance data.
Nginx needs to be set up with:
log_format perf_json '{"timestamp": "$time_iso8601", '
'"request": {'
'"remote_addr": "$remote_addr", '
'"request": "$request", '
'"status": "$status", '
'"request_time": "$request_time", '
'"upstream_response_time": "$upstream_response_time"}}';
# add following inside a specific server configuration block
access_log /var/log/$hostname-performance.log perf_json;
Logster needs to be installed on the server.
Logster needs to be called with our custom module:
./logster PerformanceLogster /var/log/app.com-performance.log --parser-options '--percentiles 25,75,90 --mapping=/path/to/my.map'
Parser supports following options:
percentilesComma-separated list of integer percentiles to track: (default: "90")mappingPath to a mapping file
The mapping file is a list of pairs composed of a label and a regular expression.
Example:
archive ^/news/archive/.*
news ^/news.*
profile ^/profile.*
Mind that the order of elements in the list is important as the first match will be used.
If the path can't be matched to any of the listed expressions a label unknown will be used. This can be renamed by adding my label .* to the end of the list.
Given that we use the following map:
news ^/news/.*
profile ^/profile/.*
And that logster picks up new log entries:
{"timestamp": "2015-05-21T06:30:32+00:00", "request": {"remote_addr": "192.168.56.1", "request": "GET /profile/me HTTP/1.1", "status": "200", "request_time": "0.036", "upstream_response_time": "0.046"}}
{"timestamp": "2015-05-21T06:30:34+00:00", "request": {"remote_addr": "192.168.56.1", "request": "GET /profile HTTP/1.1", "status": "200", "request_time": "0.041", "upstream_response_time": "0.041"}}
{"timestamp": "2015-05-21T06:30:36+00:00", "request": {"remote_addr": "192.168.56.1", "request": "GET /logout HTTP/1.1", "status": "200", "request_time": "0.023", "upstream_response_time": "0.024"}}
Following metrics would be generated:
1432129111 http_200.unknown.GET.upstreamTime.count 0.105263157895
1432129111 http_200.unknown.GET.upstreamTime.mean 32.0
1432129111 http_200.unknown.GET.upstreamTime.median 32.0
1432129111 http_200.unknown.GET.upstreamTime.90th_percentile 39.2
1432129111 http_200.unknown.GET.responseTime.count 0.105263157895
1432129111 http_200.unknown.GET.responseTime.mean 32.0
1432129111 http_200.unknown.GET.responseTime.median 32.0
1432129111 http_200.unknown.GET.responseTime.90th_percentile 39.2
1432129111 http_200.profile.GET.upstreamTime.count 0.0526315789474
1432129111 http_200.profile.GET.upstreamTime.mean 36.0
1432129111 http_200.profile.GET.upstreamTime.median 36.0
1432129111 http_200.profile.GET.upstreamTime.90th_percentile 36.0
1432129111 http_200.profile.GET.responseTime.count 0.0526315789474
1432129111 http_200.profile.GET.responseTime.mean 36.0
1432129111 http_200.profile.GET.responseTime.median 36.0
1432129111 http_200.profile.GET.responseTime.90th_percentile 36.0
git clone [email protected]:3fs/perflogster.git
cd perflogster
make dev
make qa