Ability to record performance data from servers/applications

[bryancall]

We have an internal tool that a simplified version of sar that we have been using for at least 10+ years. One the the great features it has is the ability to have application gather data for the tool. Other groups can write gather scripts to output comma separated stats that feed to the sar like tool and the sar like tool can report on this data.

For example we have gather scripts for Apache HTTPD, MySQL, and Apache Traffic Server. If you would like to see the output of the gather scripts or the sar like command I can paste it to this issue.

The reason we are interested in such a feature in sysstat is we would like to migrate away from this internal tool, but still have the ability to gather this information and report it on the command line.

[sysstat]

Yes, please show the output of your gather scripts and explain how your sar-like program use those data and report them.
Let's be clear: I'm not promising you that such a feature will ever be added to sysstat's sar (especially since I have so little time to work on sysstat nowdays), but this may give me some interesting ideas for a possible future version...

[bryancall]

Output of a gather script that is called every minute and the data is feed to our sar like application:

$ ats_gather
615869539,119877,461156724,508,2304230200,1702130337,33785056066538,14617616935708,273189708114,607897057,1749019732,109131873,4770336,1,0,0,0,7081908,0,1276248,1068770,3152122,246129189,529394,1617,0,1242196,0,334728,0,26550,616,0,0,9526552,1590179070,644332614,23463258,691546,61809349,0,2064417266,163086974,128406267,3278519,309551,682406690,96619139,7653340,4802117,546566513

Output of our sar like command (I change the name of the command) that displays information about Apache Traffic Server. "-int 1" means display at a granularity of 1 minute. "-ats" will display the detailed information about that application. There are other flags for other application data (see below).

$ our_sar_app -ats -int 1
            | ATS       Client traffic       avg resp         Origin traffic
Time           req/s    cons req/con    mbps      ms     req/s    cons req/con    mbps
03/20-16:01   2189.8  124385     3.9   258.9    95.2    1608.8     704     4.1   120.1
03/20-16:02   1772.2  121713     4.0   220.6    95.1    1291.6     660     4.3   104.3
03/20-16:03   2533.3  123103     4.1   313.1    92.5    1823.7     615     4.2   143.6
03/20-16:04   1952.2  122887     3.8   252.8   107.4    1407.3     736     4.1   117.2
03/20-16:05   2117.9  123302     4.1   242.5    93.6    1544.3     647     4.2   108.9
03/20-16:06   2204.1  123389     4.1   240.0    91.6    1625.6     631     4.1   102.1
03/20-16:07   2165.4  123128     4.1   242.4    92.7    1581.3     648     4.0   102.3
03/20-16:08   2183.4  122972     4.1   242.1    92.3    1595.2     575     4.0   102.5
03/20-16:09   2196.9  123634     4.0   249.5    93.1    1607.4     599     4.2   109.5
03/20-16:10   2124.9  123346     4.0   227.0    91.1    1548.8     664     4.3    94.2
03/20-16:11   2163.9  123368     4.0   240.9    88.7    1571.3     775     4.2   101.9
03/20-16:12   2151.7  123297     3.9   232.1    88.3    1542.8     658     4.2   102.0
03/20-16:13   2130.1  122479     4.0   237.7    87.9    1525.7     601     4.4   100.4
03/20-16:14   2096.3  121896     3.9   231.6    92.1    1510.1     582     4.4    98.9
03/20-16:15   2135.6  120673     4.0   238.7    91.5    1546.3     572     4.4   106.0
03/20-16:16   2277.8  120117     4.0   253.9    92.0    1633.7     572     4.3   106.8
03/20-16:17   2098.6  119954     4.0   246.4    97.1    1505.7     532     4.4   103.5
03/20-16:18   2114.9  119435     4.0   232.1    87.3    1496.3     554     4.6    94.6
03/20-16:19   2101.2  119642     4.0   239.0    90.1    1491.9     589     4.4    99.5

Min              0.0 26821.0     2.6     0.0    75.8       0.0   160.0     2.2     0.0
Mean          1444.3 78577.0     3.6   173.2   100.5    1053.8   465.6     3.6    75.7
Max           2533.3 124385.0     4.4   436.1   327.4    1823.7  3076.5     4.7   298.7

Output copied from "-h". There are other reports for Apache Traffic Sever, MySQL, and Apache HTTPD.

    -ats          ATS traffic performance (req/sec, mb/sec, connections)
    -atscache     ATS cache metrics (cache hit ratio, read/write errors)
    -atserr       ATS errors
    -atstls       ATS TLS errors
    -atscodes     ATS Request and Response types
    -atsall       All ATS data (see above)
    -mysql        MySQL metrics