补发丢失的贴子

简单的要求是本地和中心都要可以实时记录日志,方便安装、管理及扩展。
参考了下其它的方案,觉得syslog-ng是很方便高效的,并且已有现成的syslog-ng+splunk中心服务器.
Scribe是facebook开源的日志收集系统，c/c++实现,可用于搜索引擎中进行大规模日志分析处理。其通常与Hadoop结合使用,安装非常复杂.主要包括三部分，分别为scribe agent， scribe和存储系统
Apache的Chukwa,属于hadoop系列产品，java实现, 因而使用了很多hadoop的组件（用HDFS存储，用mapreduce处理数据）,主要有3种角色，分别为：adaptor，agent，collector
LinkedIn的Kafka 采用scala语言编写,Kafka是高吞吐量的分布式发布订阅消息系统,主要有三种角色，分别为producer，broker和consumer.支持Hadoop并行数据加载
Cloudera的Flume ,由三层组成，分别为agent，collector和storage.java实现,配置复杂
logstash Logstash 的过滤器是用 Ruby 写的，Kibana 是用纯 Javascript 写的，而 ElasticSearch 也有自己的 REST 接口和 JSON 模板。

graylog2比Kibana要弱

syslog-ng方案尝试了几种方式

nginx 1.7.1 起支持syslog

1.tnginx(nginx)通过syslog输出,本地用syslog接收并写入本地和发送至syslog-ng中心服务器.
A.每条访问日志前会有syslog的时间戳.
B.多个web日志文件分离不方便.
功能及便利就不如syslog-ng了,不推荐使用.
2.tnginx(nginx)不做修改还是输出至文件,syslog-ng监控本地文件并将变动发送至syslog-ng中心服务器.
syslog-ng中心服务器没有应用程序标记无法分离日志.
3.web访问日志通过tengine或nginx的syslog方式输出到本地syslog-ng服务,syslog-ng将日志保存在本地,并发送一份至syslog-ng中心服务器.
推荐使用

->local file tengine->local syslog-ng(syslog) ->remote syslog-ng

一.syslog-ng安装

先安装syslog-ng,本地及中心服务器都装上
最新版为syslog-ng_3.4.8,需python2.5以上支持,我用syslog-ng_3.3.9.

checking for a Python interpreter with version >= 2.5… none
configure: error: no suitable Python interpreter found
configure: error: ./configure.gnu failed for modules/afamqp/rabbitmq-c

wget https://my.balabit.com/downloads/libol/0.3/libol-0.3.18.tar.gz
wget http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/syslog-ng_3.3.9.tar.gz
wget http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/eventlog_0.2.12.tar.gz

wget http://www.balabit.com/downloads/files?path=/libol/0.3/libol-0.3.18.tar.gz http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/syslog-ng_3.3.9.tar.gz http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/eventlog_0.2.12.tar.gz

1.安装eventlog

tar -zxvf eventlog_0.2.12.tar.gz
cd eventlog-0.2.12
./configure –prefix=/usr/local/eventlog_0.2.12
make && make install
cd /usr/local/
ln -s eventlog_0.2.12 eventlog
cd –

2.安装libol
mv files?path=%2Flibol%2F0.3%2Flibol-0.3.18.tar.gz libol-0.3.18.tar.gz tar -zxvf libol-0.3.18.tar.gz cd libol-0.3.18 ./configure –prefix=/usr/local/libol-0.3.18 make && make install cd /usr/local/ ln -s libol-0.3.18 libol cd –

3.安装syslog-ng

yum install pcre

设置环境变量

export PKG_CONFIG_PATH=/usr/local/eventlog/lib/pkgconfig/:/usr/local/lib/pkgconfig/:$PKG_CONFIG_PATH

openssl如果升级过,编不过去可以关闭
–disable-ssl

tar -zxvf syslog-ng_3.3.9.tar.gz
cd syslog-ng-3.3.9
./configure –prefix=/usr/local/syslog-ng_3.3.9 –with-libol=/usr/local/libol/ –enable-pcre –disable-ipv6
make && make install
cd /usr/local/
ln -s syslog-ng_3.3.9 syslog-ng

/usr/local/syslog-ng_3.3.9/sbin/syslog-ng -V

二.方式1配置

vi /opt/nginx/conf/nginx.conf
access_log syslog:local5:notice:/dev/log:adminsyslog.c1gstudio.com access ;

查看是否是syslogd服务
ps aux|grep log
root 19199 0.0 0.0 6328 1108 ? Ss 2013 0:24 syslogd -m 0

vi /etc/init.d/syslog
SYSLOGD_OPTIONS=”-r -x -m 0″
-r: 打开接受外来日志消息的功能，其监控514 UDP端口；
-x: 关闭自动解析对方日志服务器的FQDN信息，这能避免DNS不完整所带来的麻烦；

vi /etc/syslog.conf

local5.* /opt/nginx/logs/weblog

local5.* @192.168.0.39

重启syslog服务
/etc/init.d/syslog restart

本地文件及splunk会记录日志

三.方式2配置

注释掉/etc/syslog.conf中的修改

vi /usr/local/syslog-ng/etc/syslog-ng.conf

source s_webfile { file(“/opt/nginx/logs/access.log”); };

destination r_centerlog { udp(“192.168.0.39” port(514) ); };

log { source(s_webfile); destination(r_centerlog); };

<13>Jul 22 11:30:18 192.168.0.38 192.168.0.40 – c1g_web [22/Jul/2014:11:30:18 +0800] “GET /5 HTTP/1.0” 404 219 “-” “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36” 116.210.54.88
本地文件及splunk会记录日志

四.方式3配置

vi /opt/nginx/conf/nginx.conf
access_log syslog:local5:notice:127.0.0.1:514:adminsyslog.c1gstudio.com access ;
php错误通过udp发送至syslog-ng中心服务器,web日志则通过tcp方式发送,并保存一份在/opt/nginx/logs/$PROGRAM.log

本地syslog-ng
vi /usr/local/syslog-ng/etc/syslog-ng.conf

@version: 3.3
@include “scl.conf”

options {
use_dns (no);
use_fqdn(no);
chain_hostnames(off);
keep_hostname(off);
flush_lines(0);
stats_freq(7200);
create_dirs(no);
};
source s_internal { internal(); };
destination d_syslognglog { file(“/var/log/syslog-ng.log”); };
log { source(s_internal); destination(d_syslognglog); };

source s_php { file(“/opt/php/logs/php_error.log”); };

source s_weblog { udp(ip(127.0.0.1) port(514) so_rcvbuf(8738000));};

filter f_weblog { facility(local5); };

destination r_remote { udp(“192.168.0.39” port(514) flush_lines(0) ); };
destination r_centerlog { tcp(“192.168.0.39” port(514) ); };

destination d_locallog { file(“/opt/nginx/logs/$PROGRAM.log” template(“$MSG\n”) owner(“www”) group(“website”) perm(0664) ) ); };

log { source(s_weblog); filter(f_weblog); destination(d_locallog); };
log { source(s_weblog); filter(f_weblog); destination(r_centerlog); };

log { source(s_php); destination(r_remote); };

要上没带上template(“$MSG\n”)会增加Jul 22 13:57:58 127.0.0.1 adminsyslog.c1gstudio.com[7150]:
Jul 22 13:57:58 127.0.0.1 adminsyslog.c1gstudio.com[7150]: 192.168.0.40 – c1g_web [22/Jul/2014:13:57:58 +0800] “GET /8 HTTP/1.0” 404 219 “-” “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36” 116.210.54.88

syslog-ng中心服务器配置
mkdir -p /opt/log/

vi /usr/local/syslog-ng/etc/syslog-ng.conf

source s_remotetcp {
tcp(ip(192.168.0.39) port(514) log_fetch_limit(100) log_iw_size(50000) max-connections(50) );
};
filter f_filter11 { facility(local5); };
destination r_local5 { file(“/opt/log/$YEAR.$MONTH/$DAY/$PROGRAM.log” template(“$MSG\n”) owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes) flush_lines(300) ); };

log { source(s_remotetcp); filter(f_filter11); destination(r_local5); };

完整的syslog-ng.conf
我这里还带着splunk

@version: 3.3
@include “scl.conf”
options {
use_dns (no);
use_fqdn(no);
chain_hostnames(off);
keep_hostname(off);
flush_lines(0);
stats_freq(1200);
create_dirs(yes);
log_fifo_size(1000000);
};
source s_internal { internal(); };
destination d_syslognglog { file(“/var/log/syslog-ng.log”); };
log { source(s_internal); destination(d_syslognglog); };

source s_sys { system();file(“/proc/kmsg”); unix-stream(“/dev/log”);};
destination d_cons { file(“/dev/console”); };
destination d_mesg { file(“/var/log/messages”); };
destination d_auth { file(“/var/log/secure”); };
destination d_mail { file(“/var/log/maillog”); };
destination d_spol { file(“/var/log/spooler”); };
destination d_boot { file(“/var/log/boot.log”); };
destination d_cron { file(“/var/log/cron”); };
destination d_rsync { file(“/var/log/rsync”); };
destination d_mlal { usertty(“*”); };
filter f_filter1 { facility(kern); };
filter f_filter2 { level(info) and
not (facility(mail)
or facility(authpriv) or facility(cron)); };
filter f_filter3 { facility(authpriv); };
filter f_filter4 { facility(mail); };
filter f_filter5 { level(emerg); };
filter f_filter6 { facility(uucp) or
(facility(news) and level(crit)); };
filter f_filter7 { facility(local7); };
filter f_filter8 { facility(cron); };
filter f_filter9 { facility(daemon); };
filter f_filter10 { facility(local6); };
filter f_filter11 { facility(local5); };

log { source(s_sys); filter(f_filter1); destination(d_cons); };

log { source(s_sys); filter(f_filter2); destination(d_mesg); };
log { source(s_sys); filter(f_filter3); destination(d_auth); };
log { source(s_sys); filter(f_filter4); destination(d_mail); };
log { source(s_sys); filter(f_filter5); destination(d_mlal); };
log { source(s_sys); filter(f_filter6); destination(d_spol); };
log { source(s_sys); filter(f_filter7); destination(d_boot); };
log { source(s_sys); filter(f_filter8); destination(d_cron); };

Remote logging

source s_remote {
udp(ip(192.168.0.39) port(514));
};
source s_remotetcp {
tcp(ip(192.168.0.39) port(514) log_fetch_limit(100) log_iw_size(50000) max-connections(50) );
};

template t_weblog { template(“$MSG\n”); template_escape(no) );

destination r_mesg { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/messages” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_auth { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/secure” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_mail { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/maillog” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_spol { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/spooler” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_boot { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/boot.log” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_cron { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/cron” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_daemon { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/daemon” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_local6 { file(“/var/log/syslog-ng/$YEAR.$MONTH/network/messages” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_local5 { file(“/opt/log/$YEAR.$MONTH/$DAY/$PROGRAM.log” template(“$MSG\n”) owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes) flush_lines(300) ); };

destination d_separatedbyhosts {

file(“/var/log/syslog-ng/$HOST/messages” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination d_tcp { tcp(“localhost” port(1999) localport(999) flush_lines(100) ); };

log { source(s_remote); destination(d_separatedbyhosts); };

log { source(s_remote); filter(f_filter2); destination(r_mesg); };
log { source(s_remote); filter(f_filter3); destination(r_auth); };
log { source(s_remote); filter(f_filter4); destination(r_mail); };
log { source(s_remote); filter(f_filter6); destination(r_spol); };
log { source(s_remote); filter(f_filter7); destination(r_boot); };
log { source(s_remote); filter(f_filter8); destination(r_cron); };
log { source(s_remote); filter(f_filter9); destination(r_daemon); };
log { source(s_remote); filter(f_filter10); destination(r_local6); };

log { source(s_remotetcp); filter(f_filter11); destination(r_local5); };

log { source(s_remote); destination(d_tcp); };

打开iptables接收端口,这里只开放给内网
iptables -A INPUT -p tcp -m tcp -s 192.168.0.0/16 –dport 514 -j ACCEPT iptables -A INPUT -p udp -m udp -s 192.168.0.0/16 –dport 514 -j ACCEPT

启动/
/usr/local/syslog-ng/sbin/syslog-ng syslogng参数优化
以下为4个关键参数默认值,调整不佳会丢消息
log_fetch_limit =10
max_connections =10
log_iw_size =1000
log_fifo_size =10000 输出队列的行数 出现以下错误说明参数值不合理
WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size=’10’, new_log_iw_size=’100′, min_log_fifo_size=’10000′ 参数公式
log_iw_size = max_connections * log_fetch_limit
log_fifo_size = log_iw_size * (10~20) syslog-ng中心增加接收能力
source s_remotetcp { tcp(ip(192.168.0.39) port(514) log_fetch_limit(100) log_iw_size(50000) max-connections(50) ); }; log_fifo_size为全局输出值
options { use_dns (no); use_fqdn(no); chain_hostnames(off); keep_hostname(off); flush_lines(0); stats_freq(1200); create_dirs(yes); log_fifo_size(1000000); }; 本地udp接收带8M缓存
source s_weblog { udp(ip(127.0.0.1) port(514) so_rcvbuf(8738000)); }; 内核优化
/etc/sysctl.conf
net.core.rmem_max = 8738000 net.core.wmem_max = 6553600 net.ipv4.tcp_rmem = 8192 873800 8738000 net.ipv4.tcp_wmem = 4096 655360 6553600
sysctl 查看统计信息
/usr/local/syslog-ng/sbin/syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number destination;d_mail;;a;processed;0 destination;d_spol;;a;processed;0 destination;r_boot;;a;processed;0 global;payload_reallocs;;a;processed;86 source;s_internal;;a;processed;4 source;s_sys;;a;processed;52 destination;d_mesg;;a;processed;41 source;s_remotetcp;;a;processed;32426 destination;r_local5;;a;processed;32426 日志切割后需发个信号
…. /bin/mv ${log_dir}/proxy.www.c1gstudio.com.log ${log_dir}/proxy.www.c1gstudio.com.${yesterday}.log /bin/kill -USR1 cat /usr/local/syslog-ng/var/syslog-ng.pid /bin/gzip ${log_dir}/proxy.www.c1gstudio.com.${yesterday}.log & …. 开机启动
vi /etc/rc.local
/usr/local/syslog-ng/sbin/syslog-ng 碎碎念 使用/dev/log
会把syslog接管,远程目录会偏移
nginx
access_log syslog:local5:notice:/dev/log:adminsyslog.c1gstudio.com access ; 本地syslogng
source s_sys { unix-dgram (“/dev/log”);}; Jul 22 14:45:43 Caserver caserver adminsyslog.c1gstudio.com[13318]: 192.168.0.40 – c1g_web [22/Jul/2014:14:45:43 +0800] “GET /9 HTTP/1.0” 404 219 “-” “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36” 116.210.54.88
unix-stream()主要用在Linux，使用SOCK_STREAM参数，是面向连接的，不会造成日志消息的丢失；unix-dgram()用在BSD系统上，使用SOCK_DGRAM参数，是无连接的，如果系统过载，会造成日志消息丢失。
测试
logger -p local5.info hello 垃圾收集状态
当满足一定的条件，syslog-ng即会进入垃圾收集状态，而暂时不再接受日志信息。这时，会造成非连接的传输协议的日志丢失（例如UDP）。通过设置下面两个选项可以控制：
引用
gc_idle_threshold(n) ：
意思是，一旦被分派的对象到达这个数字，并且当syslog-ng空闲时（100微秒内没有日志消息到达）。此时，syslog-ng就会启动垃圾信息收集状态。
已分配的对象可通过-v命令行参数指定其的最小值。而syslog-ng这个值应该比较小，但比已分配的对象要大即可。 例如，空闲状态，syslog-ng会显示：
引用
Nov 13 16:35:35 syslogng syslog-ng[4510]: STATS: dropped 0
Nov 13 16:45:35 syslogng syslog-ng[4510]: STATS: dropped 0 当忙时：
引用
gc_busy_threshold(n) ：当syslog-ng忙时，一旦分派的对象达到这个数字，syslog-ng就进入垃圾信息收集状态的时间。该值应该比较高，以保证正常情况下不会打断日志消息的收取。 ※TCP基于连接方式传输，不会造成日志丢失，而UDP则不同。但因为传统的syslog基于UDP的514端口，所以，UDP方式也经常会使用到。 另外，514也是rshell的默认端口，请注意冲突。 参考:
https://pzolee.blogs.balabit.com/2011/02/syslog-ng-performance-tuning/
http://www.balabit.com/support/documentation/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/pdf/syslog-ng-ose-v3.3-guide-admin-en.pdf
http://dongxicheng.org/search-engine/log-systems/