A simple & high efficient Shell script for synchronizing log using NFS.
sync_log
tracks the log file(i.e. <src_file>
), save it as slice files under a temporary directory(i.e <tmp_dir>
) and move slice files to the destination directory(i.e. <dst_dir>
) at last every <check_interval_sec>
seconds(defaults to 20 seconds).
merge_log
merges slice files into <dst_file_name>
.
- CentOS 6.6
- macOS 10.12
- Other Linux(NOT TESTED)
- Simple: easy to deploy, only 1-2 script files, no need to run
yum install xxx
; - High efficient: script files use Linux native commands like
mv
,tail
,grep
,pgrep
,cat
,kill
, etc. only; - Re-transport on network failure: the slice files will be deleted only they are successfully tranported otherwise they be transported again and again;
- Lossless: logs will not be lost even under QPS 100k. (Please notice that logs will be repeated about 4% under QPS 100k while the number <= 2/1000 under QPS 14k);
# Mount points. Notice that `/mnt/nfs` is a remote directory while `/` is a local one:
df -h
# Filesystem Size Used Avail Use% Mounted on
# /dev/sda1 20G 4.1G 15G 22% /
# 192.168.3.20:/nfs/192.168.3.10 2.8T 1.6T 1.3T 56% /mnt/nfs
# make <tmp_dir> and <dst_dir>
mkdir -p /tmp/nginx/access.log.d /mnt/nfs/nginx/access.log.d
# start sync_log
sync_log start /var/log/nginx/access.log /tmp/nginx/access.log.d /mnt/nfs/nginx/access.log.d 20
# stop sync_log
# sync_log stop /tmp/nginx/access.log.d
# help
# sync_log -h
# <dst_dir> hirarchy:
tree /nfs/192.168.3.10
# 192.168.3.10
# └── nginx
# └── access.log.d
# ├── 0213T173610
# ├── 0213T173630
# ├── 0213T173650
# └── access.log
# start merge_log
merge_log /nfs nginx/access.log.d access.log --ips=192.168.3.10
# stop merge_log
# Ctrl + C
# help
# merge_log -h
lslxdx#163.com