Skip to content


windows下VMware 虚拟机转换到 VirtualBox

在将 VMware 虚拟机迁移到 Vbox 之前,确保在 VMware 的状态为“关闭电源” —— 非挂起。如果是挂起,启动虚拟机然后关机。
接下来,找到 OVFTool 所在目录。如果你用的是 VMware Player,目录在类似 C:\Program Files (x86)\VMware\VMware Player\OVFTool。按住 Shift 键,右击选择在此处打开命令行窗口。

G:\Program Files\vmware\VMware Workstation\OVFTool

G:\vmos\CentOS 7_mini_vm5\CentOS 7_mini_vm5.vmx
ovftool “G:\vmos\CentOS 7_mini_vm5\CentOS 7_mini_vm5.vmx” G:\vmos\CentOS7_mini_vr\centos7.ovf

Opening VMX source: G:\vmos\CentOS 7_mini_vm5\CentOS 7_mini_vm5.vmx
Opening OVF target: G:\vmos\CentOS7_mini_vr\centos7.ovf
Writing OVF package: G:\vmos\CentOS7_mini_vr\centos7\centos7.ovf
Transfer Completed
Completed successfully

过程结束之后,可以使用管理菜单中的导入虚拟电脑选项,将 .ovf 文件导入 Vbox。

 

Posted in 技术.

Tagged with , .


安装一个自已的DNS服务器dnsmasq

补发丢失贴子

centos下超简单

yum install dnsmasq

vim /etc/dnsmasq.conf

# 让dnsmasq读取目录内所有配置文件
# conf-dir=/etc/dnsmasq.d
# 让dnsmasq读取设定的resolv-file
# no-resolv
no-poll
strict-order
resolv-file=/etc/dnsmasq.d/resolv.dnsmasq.conf

# 不读取系统hosts,读取你设定的
no-hosts
addn-hosts=/etc/dnsmasq.d/dnsmasq.hosts
# dnsmasq日志设置
log-queries
# dnsmasq缓存设置
cache-size=1024
# 单设置127只为本机使用,加入本机IP为内部全网使用
listen-address=127.0.0.1,192.168.0.56
echo 'no-poll' >> /etc/dnsmasq.conf
echo 'strict-order' >> /etc/dnsmasq.conf
echo 'resolv-file=/etc/dnsmasq.d/resolv.dnsmasq.conf' >> /etc/dnsmasq.conf
echo 'addn-hosts=/etc/dnsmasq.d/dnsmasq.hosts' >> /etc/dnsmasq.conf
echo 'log-queries' >> /etc/dnsmasq.conf
echo 'cache-size=1024' >> /etc/dnsmasq.conf
echo 'listen-address=127.0.0.1,10.100.131.1' >> /etc/dnsmasq.conf

cp /etc/hosts /etc/dnsmasq.d/dnsmasq.hosts

vi /etc/resolv.conf
nameserver 127.0.0.1

vi /etc/dnsmasq.d/resolv.dnsmasq.conf
nameserver 114.114.114.114
nameserver 223.5.5.5

dnsmasq --test
dnsmasq: syntax check OK.

/etc/init.d/dnsmasq start

chkconfig dnsmasq on

iptables -A INPUT -p tcp -m state -s 192.168.0.0/24 --state NEW --dport 53 -j ACCEPT
iptables -A INPUT -p udp -m state -s 192.168.0.0/24 --state NEW --dport 53 -j ACCEPT

/etc/init.d/iptables save

客户端
vi /etc/resolv.conf
nameserver 192.168.0.56

dnsmasq -v
Dnsmasq version 2.48 Copyright (C) 2000-2009 Simon Kelley
Compile time options IPv6 GNU-getopt DBus no-I18N DHCP TFTP “–bind-interfaces with SO_BINDTODEVICE”

This software comes with ABSOLUTELY NO WARRANTY.
Dnsmasq is free software, and you are welcome to redistribute it
under the terms of the GNU General Public License, version 2 or 3.

http://pub-shbt.s3.360.cn/cert-public-file/%E3%80%90360CERT%E3%80%91Dnsmasq__%E5%A4%9A%E4%B8%AA%E9%AB%98%E5%8D%B1%E6%BC%8F%E6%B4%9E%E9%A3%8E%E9%99%A9%E9%80%9A%E5%91%8A.pdf
修复安全漏洞,启动时加上 –dns-forward-max=100

vi /etc/init.d/dnsmasq

  start)
        if [ $UID -ne 0 ] ; then
            echo "User has insufficient privilege."
            exit 4
        fi
        echo -n "Starting dnsmasq: "
        daemon $dnsmasq --dns-forward-max=100 $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/dnsmasq

重启
/etc/init.d/dnsmasq restart
验证
ps aux|grep dnsmasq

nobody 28688 0.0 0.0 12888 712 ? S 11:42 0:00 /usr/sbin/dnsmasq –dns-forward-max=100

开启日志

log-facility=/var/log/dnsmasq.log

vi /etc/logrotate.d/dnsmasq
输入:
/var/log/dnsmasq.log {
notifempty
weekly
dateext
rotate 58
sharedscripts
postrotate
[ ! -f /var/run/dnsmasq.pid ] || kill -USR2 cat /var/run/dnsmasq.pid
endscript
}

/usr/sbin/logrotate -vf /etc/logrotate.conf

统计dnsmasq的解析

cat /var/log/dnsmasq.log |grep ‘query[A]’|gawk ‘{print $6}’|sort|uniq -c|sort -nr

Posted in dns/dnsmasq.

Tagged with , .


nginx+syslog-ng实现web日志集中管理

补发丢失的贴子

简单的要求是本地和中心都要可以实时记录日志,方便安装、管理及扩展。
参考了下其它的方案,觉得syslog-ng是很方便高效的,并且已有现成的syslog-ng+splunk中心服务器.
Scribe是facebook开源的日志收集系统,c/c++实现,可用于搜索引擎中进行大规模日志分析处理。其通常与Hadoop结合使用,安装非常复杂.主要包括三部分,分别为scribe agent, scribe和存储系统
Apache的Chukwa,属于hadoop系列产品,java实现, 因而使用了很多hadoop的组件(用HDFS存储,用mapreduce处理数据),主要有3种角色,分别为:adaptor,agent,collector
LinkedIn的Kafka 采用scala语言编写,Kafka是高吞吐量的分布式发布订阅消息系统,主要有三种角色,分别为producer,broker和consumer.支持Hadoop并行数据加载
Cloudera的Flume ,由三层组成,分别为agent,collector和storage.java实现,配置复杂
logstash Logstash 的过滤器是用 Ruby 写的,Kibana 是用纯 Javascript 写的,而 ElasticSearch 也有自己的 REST 接口和 JSON 模板。

graylog2比Kibana要弱

syslog-ng方案尝试了几种方式

nginx 1.7.1 起支持syslog

1.tnginx(nginx)通过syslog输出,本地用syslog接收并写入本地和发送至syslog-ng中心服务器.
A.每条访问日志前会有syslog的时间戳.
B.多个web日志文件分离不方便.
功能及便利就不如syslog-ng了,不推荐使用.
2.tnginx(nginx)不做修改还是输出至文件,syslog-ng监控本地文件并将变动发送至syslog-ng中心服务器.
syslog-ng中心服务器没有应用程序标记无法分离日志.
3.web访问日志通过tengine或nginx的syslog方式输出到本地syslog-ng服务,syslog-ng将日志保存在本地,并发送一份至syslog-ng中心服务器.
推荐使用

->local file tengine->local syslog-ng(syslog) ->remote syslog-ng

一.syslog-ng安装

先安装syslog-ng,本地及中心服务器都装上
最新版为syslog-ng_3.4.8,需python2.5以上支持,我用syslog-ng_3.3.9.

checking for a Python interpreter with version >= 2.5… none
configure: error: no suitable Python interpreter found
configure: error: ./configure.gnu failed for modules/afamqp/rabbitmq-c


wget https://my.balabit.com/downloads/libol/0.3/libol-0.3.18.tar.gz
wget http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/syslog-ng_3.3.9.tar.gz
wget http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/eventlog_0.2.12.tar.gz

wget http://www.balabit.com/downloads/files?path=/libol/0.3/libol-0.3.18.tar.gz http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/syslog-ng_3.3.9.tar.gz http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.9/source/eventlog_0.2.12.tar.gz

1.安装eventlog

tar -zxvf eventlog_0.2.12.tar.gz
cd eventlog-0.2.12
./configure –prefix=/usr/local/eventlog_0.2.12
make && make install
cd /usr/local/
ln -s eventlog_0.2.12 eventlog
cd –

2.安装libol
mv files?path=%2Flibol%2F0.3%2Flibol-0.3.18.tar.gz libol-0.3.18.tar.gz tar -zxvf libol-0.3.18.tar.gz cd libol-0.3.18 ./configure –prefix=/usr/local/libol-0.3.18 make && make install cd /usr/local/ ln -s libol-0.3.18 libol cd –

3.安装syslog-ng

yum install pcre

设置环境变量

export PKG_CONFIG_PATH=/usr/local/eventlog/lib/pkgconfig/:/usr/local/lib/pkgconfig/:$PKG_CONFIG_PATH

openssl如果升级过,编不过去可以关闭
–disable-ssl

tar -zxvf syslog-ng_3.3.9.tar.gz
cd syslog-ng-3.3.9
./configure –prefix=/usr/local/syslog-ng_3.3.9 –with-libol=/usr/local/libol/ –enable-pcre –disable-ipv6
make && make install
cd /usr/local/
ln -s syslog-ng_3.3.9 syslog-ng

/usr/local/syslog-ng_3.3.9/sbin/syslog-ng -V

二.方式1配置

vi /opt/nginx/conf/nginx.conf
access_log syslog:local5:notice:/dev/log:adminsyslog.c1gstudio.com access ;

查看是否是syslogd服务
ps aux|grep log
root 19199 0.0 0.0 6328 1108 ? Ss 2013 0:24 syslogd -m 0

vi /etc/init.d/syslog
SYSLOGD_OPTIONS=”-r -x -m 0″
-r: 打开接受外来日志消息的功能,其监控514 UDP端口;
-x: 关闭自动解析对方日志服务器的FQDN信息,这能避免DNS不完整所带来的麻烦;

vi /etc/syslog.conf

local5.* /opt/nginx/logs/weblog

local5.* @192.168.0.39

重启syslog服务
/etc/init.d/syslog restart

本地文件及splunk会记录日志

三.方式2配置

注释掉/etc/syslog.conf中的修改

vi /usr/local/syslog-ng/etc/syslog-ng.conf

source s_webfile { file(“/opt/nginx/logs/access.log”); };

destination r_centerlog { udp(“192.168.0.39” port(514) ); };

log { source(s_webfile); destination(r_centerlog); };

Jul 22 11:30:18 192.168.0.38 192.168.0.40 – c1g_web [22/Jul/2014:11:30:18 +0800] “GET /5 HTTP/1.0” 404 219 “-” “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36” 116.210.54.88
本地文件及splunk会记录日志

四.方式3配置

vi /opt/nginx/conf/nginx.conf
access_log syslog:local5:notice:127.0.0.1:514:adminsyslog.c1gstudio.com access ;
php错误通过udp发送至syslog-ng中心服务器,web日志则通过tcp方式发送,并保存一份在/opt/nginx/logs/$PROGRAM.log

本地syslog-ng
vi /usr/local/syslog-ng/etc/syslog-ng.conf

@version: 3.3
@include “scl.conf”

options {
use_dns (no);
use_fqdn(no);
chain_hostnames(off);
keep_hostname(off);
flush_lines(0);
stats_freq(7200);
create_dirs(no);
};
source s_internal { internal(); };
destination d_syslognglog { file(“/var/log/syslog-ng.log”); };
log { source(s_internal); destination(d_syslognglog); };

source s_php { file(“/opt/php/logs/php_error.log”); };

source s_weblog { udp(ip(127.0.0.1) port(514) so_rcvbuf(8738000));};

filter f_weblog { facility(local5); };

destination r_remote { udp(“192.168.0.39” port(514) flush_lines(0) ); };
destination r_centerlog { tcp(“192.168.0.39” port(514) ); };

destination d_locallog { file(“/opt/nginx/logs/$PROGRAM.log” template(“$MSG\n”) owner(“www”) group(“website”) perm(0664) ) ); };

log { source(s_weblog); filter(f_weblog); destination(d_locallog); };
log { source(s_weblog); filter(f_weblog); destination(r_centerlog); };

log { source(s_php); destination(r_remote); };

要上没带上template(“$MSG\n”)会增加Jul 22 13:57:58 127.0.0.1 adminsyslog.c1gstudio.com[7150]:
Jul 22 13:57:58 127.0.0.1 adminsyslog.c1gstudio.com[7150]: 192.168.0.40 – c1g_web [22/Jul/2014:13:57:58 +0800] “GET /8 HTTP/1.0” 404 219 “-” “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36” 116.210.54.88

syslog-ng中心服务器配置
mkdir -p /opt/log/

vi /usr/local/syslog-ng/etc/syslog-ng.conf

source s_remotetcp {
tcp(ip(192.168.0.39) port(514) log_fetch_limit(100) log_iw_size(50000) max-connections(50) );
};
filter f_filter11 { facility(local5); };
destination r_local5 { file(“/opt/log/$YEAR.$MONTH/$DAY/$PROGRAM.log” template(“$MSG\n”) owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes) flush_lines(300) ); };

log { source(s_remotetcp); filter(f_filter11); destination(r_local5); };

完整的syslog-ng.conf
我这里还带着splunk

@version: 3.3
@include “scl.conf”
options {
use_dns (no);
use_fqdn(no);
chain_hostnames(off);
keep_hostname(off);
flush_lines(0);
stats_freq(1200);
create_dirs(yes);
log_fifo_size(1000000);
};
source s_internal { internal(); };
destination d_syslognglog { file(“/var/log/syslog-ng.log”); };
log { source(s_internal); destination(d_syslognglog); };

source s_sys { system();file(“/proc/kmsg”); unix-stream(“/dev/log”);};
destination d_cons { file(“/dev/console”); };
destination d_mesg { file(“/var/log/messages”); };
destination d_auth { file(“/var/log/secure”); };
destination d_mail { file(“/var/log/maillog”); };
destination d_spol { file(“/var/log/spooler”); };
destination d_boot { file(“/var/log/boot.log”); };
destination d_cron { file(“/var/log/cron”); };
destination d_rsync { file(“/var/log/rsync”); };
destination d_mlal { usertty(“*”); };
filter f_filter1 { facility(kern); };
filter f_filter2 { level(info) and
not (facility(mail)
or facility(authpriv) or facility(cron)); };
filter f_filter3 { facility(authpriv); };
filter f_filter4 { facility(mail); };
filter f_filter5 { level(emerg); };
filter f_filter6 { facility(uucp) or
(facility(news) and level(crit)); };
filter f_filter7 { facility(local7); };
filter f_filter8 { facility(cron); };
filter f_filter9 { facility(daemon); };
filter f_filter10 { facility(local6); };
filter f_filter11 { facility(local5); };

log { source(s_sys); filter(f_filter1); destination(d_cons); };

log { source(s_sys); filter(f_filter2); destination(d_mesg); };
log { source(s_sys); filter(f_filter3); destination(d_auth); };
log { source(s_sys); filter(f_filter4); destination(d_mail); };
log { source(s_sys); filter(f_filter5); destination(d_mlal); };
log { source(s_sys); filter(f_filter6); destination(d_spol); };
log { source(s_sys); filter(f_filter7); destination(d_boot); };
log { source(s_sys); filter(f_filter8); destination(d_cron); };

Remote logging

source s_remote {
udp(ip(192.168.0.39) port(514));
};
source s_remotetcp {
tcp(ip(192.168.0.39) port(514) log_fetch_limit(100) log_iw_size(50000) max-connections(50) );
};

template t_weblog { template(“$MSG\n”); template_escape(no) );

destination r_mesg { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/messages” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_auth { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/secure” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_mail { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/maillog” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_spol { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/spooler” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_boot { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/boot.log” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_cron { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/cron” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_daemon { file(“/var/log/syslog-ng/$YEAR.$MONTH/$HOST/daemon” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_local6 { file(“/var/log/syslog-ng/$YEAR.$MONTH/network/messages” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes)); };
destination r_local5 { file(“/opt/log/$YEAR.$MONTH/$DAY/$PROGRAM.log” template(“$MSG\n”) owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes) flush_lines(300) ); };

destination d_separatedbyhosts {

file(“/var/log/syslog-ng/$HOST/messages” owner(“root”) group(“root”) perm(0640) dir_perm(0750) create_dirs(yes));

};

destination d_tcp { tcp(“localhost” port(1999) localport(999) flush_lines(100) ); };

log { source(s_remote); destination(d_separatedbyhosts); };

log { source(s_remote); filter(f_filter2); destination(r_mesg); };
log { source(s_remote); filter(f_filter3); destination(r_auth); };
log { source(s_remote); filter(f_filter4); destination(r_mail); };
log { source(s_remote); filter(f_filter6); destination(r_spol); };
log { source(s_remote); filter(f_filter7); destination(r_boot); };
log { source(s_remote); filter(f_filter8); destination(r_cron); };
log { source(s_remote); filter(f_filter9); destination(r_daemon); };
log { source(s_remote); filter(f_filter10); destination(r_local6); };

log { source(s_remotetcp); filter(f_filter11); destination(r_local5); };

log { source(s_remote); destination(d_tcp); };

打开iptables接收端口,这里只开放给内网
iptables -A INPUT -p tcp -m tcp -s 192.168.0.0/16 –dport 514 -j ACCEPT iptables -A INPUT -p udp -m udp -s 192.168.0.0/16 –dport 514 -j ACCEPT

启动/
/usr/local/syslog-ng/sbin/syslog-ng
syslogng参数优化
以下为4个关键参数默认值,调整不佳会丢消息
log_fetch_limit =10
max_connections =10
log_iw_size =1000
log_fifo_size =10000 输出队列的行数 出现以下错误说明参数值不合理
WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size=’10’, new_log_iw_size=’100′, min_log_fifo_size=’10000′ 参数公式
log_iw_size = max_connections * log_fetch_limit
log_fifo_size = log_iw_size * (10~20) syslog-ng中心增加接收能力
source s_remotetcp { tcp(ip(192.168.0.39) port(514) log_fetch_limit(100) log_iw_size(50000) max-connections(50) ); }; log_fifo_size为全局输出值
options { use_dns (no); use_fqdn(no); chain_hostnames(off); keep_hostname(off); flush_lines(0); stats_freq(1200); create_dirs(yes); log_fifo_size(1000000); }; 本地udp接收带8M缓存
source s_weblog { udp(ip(127.0.0.1) port(514) so_rcvbuf(8738000)); }; 内核优化
/etc/sysctl.conf
net.core.rmem_max = 8738000 net.core.wmem_max = 6553600 net.ipv4.tcp_rmem = 8192 873800 8738000 net.ipv4.tcp_wmem = 4096 655360 6553600
sysctl 查看统计信息
/usr/local/syslog-ng/sbin/syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number destination;d_mail;;a;processed;0 destination;d_spol;;a;processed;0 destination;r_boot;;a;processed;0 global;payload_reallocs;;a;processed;86 source;s_internal;;a;processed;4 source;s_sys;;a;processed;52 destination;d_mesg;;a;processed;41 source;s_remotetcp;;a;processed;32426 destination;r_local5;;a;processed;32426 日志切割后需发个信号
…. /bin/mv ${log_dir}/proxy.www.c1gstudio.com.log ${log_dir}/proxy.www.c1gstudio.com.${yesterday}.log /bin/kill -USR1 cat /usr/local/syslog-ng/var/syslog-ng.pid /bin/gzip ${log_dir}/proxy.www.c1gstudio.com.${yesterday}.log & …. 开机启动
vi /etc/rc.local
/usr/local/syslog-ng/sbin/syslog-ng 碎碎念 使用/dev/log
会把syslog接管,远程目录会偏移
nginx
access_log syslog:local5:notice:/dev/log:adminsyslog.c1gstudio.com access ; 本地syslogng
source s_sys { unix-dgram (“/dev/log”);}; Jul 22 14:45:43 Caserver caserver adminsyslog.c1gstudio.com[13318]: 192.168.0.40 – c1g_web [22/Jul/2014:14:45:43 +0800] “GET /9 HTTP/1.0” 404 219 “-” “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36” 116.210.54.88
unix-stream()主要用在Linux,使用SOCK_STREAM参数,是面向连接的,不会造成日志消息的丢失;unix-dgram()用在BSD系统上,使用SOCK_DGRAM参数,是无连接的,如果系统过载,会造成日志消息丢失。
测试
logger -p local5.info hello 垃圾收集状态
当满足一定的条件,syslog-ng即会进入垃圾收集状态,而暂时不再接受日志信息。这时,会造成非连接的传输协议的日志丢失(例如UDP)。通过设置下面两个选项可以控制:
引用
gc_idle_threshold(n) :
意思是,一旦被分派的对象到达这个数字,并且当syslog-ng空闲时(100微秒内没有日志消息到达)。此时,syslog-ng就会启动垃圾信息收集状态。
已分配的对象可通过-v命令行参数指定其的最小值。而syslog-ng这个值应该比较小,但比已分配的对象要大即可。 例如,空闲状态,syslog-ng会显示:
引用
Nov 13 16:35:35 syslogng syslog-ng[4510]: STATS: dropped 0
Nov 13 16:45:35 syslogng syslog-ng[4510]: STATS: dropped 0 当忙时:
引用
gc_busy_threshold(n) :当syslog-ng忙时,一旦分派的对象达到这个数字,syslog-ng就进入垃圾信息收集状态的时间。该值应该比较高,以保证正常情况下不会打断日志消息的收取。 ※TCP基于连接方式传输,不会造成日志丢失,而UDP则不同。但因为传统的syslog基于UDP的514端口,所以,UDP方式也经常会使用到。 另外,514也是rshell的默认端口,请注意冲突。 参考:
https://pzolee.blogs.balabit.com/2011/02/syslog-ng-performance-tuning/
http://www.balabit.com/support/documentation/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/pdf/syslog-ng-ose-v3.3-guide-admin-en.pdf
http://dongxicheng.org/search-engine/log-systems/

Posted in Nginx, 日志.

Tagged with , , .


linux下安装rabbitmq及php-amqplib

补发丢失的贴子。

linux下安装rabbitmq及php-amqplib

一.linux 安装 git

yum 自带git 版本太低了
git version 1.7.1

yum remove git
yum -y install curl-devel expat-devel gettext-devel openssl-devel zlib-devel gcc perl-ExtUtils-MakeMaker
https://github.com/git/git/archive/v2.21.0.tar.gz
tar zxvf v2.21.0.tar.gz
cd git-2.21.0/
make configure
./configure –prefix=/usr/local/git –with-iconv=/usr/local/
make
make install

/usr/local/git/bin/git –version
git version 2.21.0

ln -s /usr/local/git/bin/git /usr/bin/git

git clone –no-checkout ‘https://github.com/php-amqplib/php-amqplib.git’ ‘/opt/php-5.4.45_phar/lib/composer/vendor/php-amqplib/php-amqplib’

SSL connect error fatal: unable to access ‘https://github.com/php-amqplib/php-amqplib.git/’: SSL connect error
git config –global http.sslversion tlsv1 vi /etc/hosts 192.30.255.112 github.com

二.安装composer

Composer 需要 PHP 5.3.2+ openssl 1.0.1+才能运行。

openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
php编译时不能带–with-curlwrappers 和–disable-phar

php查看编译参数
php -i |grep configure
Configure Command => ‘./configure’ ‘–prefix=/opt/php-5.4.45’ ‘–with-config-file-path=/opt/php-5.4.45/etc’ ‘–with-mysql=/opt/mysql’ ‘–with-mysqli=/opt/mysql/bin/mysql_config’ ‘–with-iconv-dir=/usr/local’ ‘–with-freetype-dir’ ‘–with-jpeg-dir’ ‘–with-png-dir’ ‘–with-zlib’ ‘–with-libxml-dir=/usr’ ‘–disable-rpath’ ‘–enable-bcmath’ ‘–enable-shmop’ ‘–enable-sysvsem’ ‘–enable-inline-optimization’ ‘–with-curl’ ‘–with-curlwrappers’ ‘–enable-mbregex’ ‘–enable-cgi’ ‘–enable-fpm’ ‘–enable-mbstring’ ‘–with-mcrypt’ ‘–with-gd’ ‘–enable-gd-native-ttf’ ‘–with-openssl’ ‘–with-mhash’ ‘–enable-pcntl’ ‘–enable-sockets’ ‘–with-xmlrpc’ ‘–enable-zip’ ‘–enable-soap’ ‘–enable-xml’ ‘–disable-debug’ ‘–disable-ipv6’ ‘–without-pear’ ‘–disable-phar’ ‘–enable-ftp’ ‘–with-pdo-mysql=/opt/mysql’

–disable-phar 错误

Some settings on your machine make Composer unable to work properly.
Make sure that you fix the issues listed below and run this script again:

The phar extension is missing.
Install it or recompile php without –disable-phar

–with-curlwrappers 错误
PHP was compiled with –with-curlwrappers which will cause issues with HTTP authentication and GitHub. Recompile it without this flag if possible

git 没安装错误
sh: git: command not found

wget https://getcomposer.org/composer.phar ln -s /opt/php/bin/php /usr/bin/php php composer.phar
这将返回给你一个可执行的命令列表。

或者安装成全局方式
mv composer.phar /usr/bin/composer chmod +x /usr/bin/composer composer
创建一个composer.json
{ “require”: { “monolog/monolog”: “1.0.*”, “php-amqplib/php-amqplib”: “>=2.6.1” } }

创建国内镜像
composer clearcache composer config -g repo.packagist composer https://packagist.phpcomposer.com

建议不要用root运行,
运行时compose会在当前用户下创建一个可写的cache目录/home//.compose
sudo -uandychu composer config -g repo.packagist composer https://packagist.phpcomposer.com
会composer.json中添加
“repositories”: { “packagist”: { “type”: “composer”, “url”: “https://packagist.phpcomposer.com” } }

命令行下
php composer.phar install
或者全局方式
composer up

Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals

  • Installing php-amqplib/php-amqplib (v2.8.1): Downloading (failed)
    Downloading (failed)
    Downloading (failed) Failed to download php-amqplib/php-amqplib from dist: Th
    e “https://api.github.com/repos/php-amqplib/php-amqplib/zipball/84449ffd3f5a7466
    bbee3946facb3746ff11f075″ file could not be downloaded: SSL operation failed wit
    h code 1. OpenSSL Error messages:
    error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
    Failed to enable crypto
    failed to open stream: operation failed
    Now trying to download from source
  • Installing php-amqplib/php-amqplib (v2.8.1): Cloning 84449ffd3f from cache
    Writing lock file
    Generating autoload files

vi php.ini
将/opt/php-5.4.45_phar/lib/composer 放入include_path和 open_basedir

reload php

三.linux 安装 Erlang


yum install ncurses ncurses-base ncurses-devel ncurses-libs ncurses-static ncurses-term ocaml-curses ocaml-curses-devel -y
wget http://erlang.org/download/otp_src_21.2.tar.gz
tar zxvf otp_src_21.2.tar.gz
cd otp_src_21.2
./configure –with-ssl -enable-threads -enable-smmp-support -enable-kernel-poll –enable-hipe –without-javac
make
make install
/usr/local/bin/erl
Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:40:40] [ds:40:40:10] [async-threads:1] [hipe]

Eshell V10.2 (abort with ^G)
1> halt().

四.linux 安装 rabbitmq

http://www.rabbitmq.com/download.html
wget https://github.com/rabbitmq/rabbitmq-server/releases/download/v3.7.11/rabbitmq-server-generic-unix-3.7.11.tar.xz tar xvf rabbitmq-server-generic-unix-3.7.11.tar.xz mv rabbitmq_server-3.7.11 /opt/ cd /opt ln -s rabbitmq_server-3.7.11 rabbitmq cd /opt/rabbitmq/sbin ./rabbitmq-server &


## ##
## ## RabbitMQ 3.7.11. Copyright (C) 2007-2019 Pivotal Software, Inc.
########## Licensed under the MPL. See http://www.rabbitmq.com/
###### ##
########## Logs: /opt/rabbitmq/var/log/rabbitmq/[email protected]
/opt/rabbitmq/var/log/rabbitmq/rabbit@c1gstudio_upgrade.log

          Starting broker...

completed with 0 plugins.

[bin]# ./rabbitmqctl status
Status of node rabbit@c1gstudio …
[{pid,28884},
{running_applications,
[{rabbit,”RabbitMQ”,”3.7.11″},
{mnesia,”MNESIA CXC 138 12″,”4.15.5″},
{os_mon,”CPO CXC 138 46″,”2.4.7″},
{sysmon_handler,”Rate-limiting system_monitor event handler”,”1.1.0″},
{rabbit_common,
“Modules shared by rabbitmq-server and rabbitmq-erlang-client”,
“3.7.11”},
{ranch,”Socket acceptor pool for TCP protocols.”,”1.7.1″},
{ssl,”Erlang/OTP SSL application”,”9.1″},
{public_key,”Public key infrastructure”,”1.6.4″},
{asn1,”The Erlang ASN1 compiler version 5.0.8″,”5.0.8″},
{inets,”INETS CXC 138 49″,”7.0.3″},
{recon,”Diagnostic tools for production use”,”2.3.6″},
{xmerl,”XML parser”,”1.3.18″},
{jsx,”a streaming, evented json parsing toolkit”,”2.9.0″},
{crypto,”CRYPTO”,”4.4″},
{lager,”Erlang logging framework”,”3.6.5″},
{goldrush,”Erlang event stream processor”,”0.1.9″},
{compiler,”ERTS CXC 138 10″,”7.3″},
{syntax_tools,”Syntax tools”,”2.1.6″},
{sasl,”SASL CXC 138 11″,”3.3″},
{stdlib,”ERTS CXC 138 10″,”3.7″},
{kernel,”ERTS CXC 138 10″,”6.2″}]},
{os,{unix,linux}},
{erlang_version,
“Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:40:40] [ds:40:40:10] [async-threads:640] [hipe]\n”},
{memory,
[{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,0},
{queue_procs,0},
{queue_slave_procs,0},
{plugins,10044},
{other_proc,33404336},
{metrics,197308},
{mgmt_db,0},
{mnesia,73360},
{other_ets,2314472},
{binary,150240},
{msg_index,29488},
{code,20364814},
{atom,1082561},
{other_system,30248961},
{allocated_unused,50253824},
{reserved_unallocated,0},
{strategy,rss},
{total,[{erlang,87875584},{rss,113528832},{allocated,138129408}]}]},
{alarms,[]},
{listeners,[{clustering,25672,”::”},{amqp,5672,”0.0.0.0″}]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,13413631590},
{disk_free_limit,50000000},
{disk_free,322518425600},
{file_descriptors,
[{total_limit,51100},
{total_used,2},
{sockets_limit,45988},
{sockets_used,0}]},
{processes,[{limit,1048576},{used,253}]},
{run_queue,1},
{uptime,128},
{kernel,{net_ticktime,60}}]

启动服务: ./rabbitmq-server &

查看服务状态:./rabbitmqctl status

关闭服务:./rabbitmqctl stop

启动管理 ./rabbitmq-plugins enable rabbitmq_management

查看mq用户:./rabbitmqctl list_users

查看用户权限:./rabbitmqctl list_user_permissions guest

新增用户: ./rabbitmqctl add_user admin 123456
./rabbitmqctl add_user admin tEneYxJm
./rabbitmqctl add_user yjsworker 7G2hMuXf

设为管理员
./rabbitmqctl set_user_tags admin administrator

编辑配置文件

cd /opt/rabbitmq/etc/rabbitmq
wget https://raw.githubusercontent.com/rabbitmq/rabbitmq-server/master/docs/rabbitmq.conf.example
cp rabbitmq.conf.example rabbitmq.conf

vi rabbitmq.conf
listeners.tcp.local = 192.168.0.37:5672
management.tcp.port = 15672
management.tcp.ip = 192.168.0.37

management.http_log_dir = /var/log/nginx/access.log

management.tcp.port = 15672
management.tcp.ip = 192.168.0.37

五.开机启动

vi /etc/rc.local /opt/rabbitmq/sbin/rabbitmq-server &

六.iptables


iptables -A INPUT -s 192.168.0.0/24 -p tcp -m tcp –dport 5672 -j ACCEPT
iptables -A INPUT -s 192.168.0.40/32 -p tcp -m tcp –dport 15672 -j ACCEPT
/etc/init.d/iptables save

七.nginx配置

vi /opt/nginx/conf/nginx.conf

server
{
listen 80;
server_name admin.c1gstudio.com;
index index.html index.htm index.php;
root /opt/htdocs/www;

    include manageip.conf;
    deny    all;

     location /rabbit/ {
      proxy_pass        http://192.168.0.37:15672/;
      proxy_set_header  Host        $host;
      proxy_set_header  X-Real-IP   $remote_addr;
      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
     }            

     location / {
      proxy_pass        http://192.168.0.37:80/;
      proxy_set_header  Host        $host;
      proxy_set_header  X-Real-IP   $remote_addr;
      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
      #proxy_set_header X-Forwarded-For $remote_addr;
      proxy_redirect    off;
     }            

         access_log  /opt/nginx/logs/access.log  access;
 }

八.测试

打开send.php文件:

<?php

require_once DIR . ‘/vendor/autoload.php’;

use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;

// RabbitMQ 默认端口5672, 默认账号密码: guest / guest
// $host = ‘localhost’;
$host = ‘192.168.99.100’;
$port = ‘5672’;
$username = ‘guest’;
$password = ‘guest’;

// $connection = new AMQPStreamConnection(‘localhost’, 5672, ‘guest’, ‘guest’);
$connection = new AMQPStreamConnection($host, $port, $username, $password);
$channel = $connection->channel();
$channel->queue_declare(‘hello’, false, false, false, false);

$msg = new AMQPMessage(‘Hello World!’);
$channel->basic_publish($msg, ”, ‘hello’);
echo ” [x] Sent ‘Hello World!’\n”;

$channel->close();
$connection->close();

打开 receive.php 文件:

<?php
// file: receive.php
require_once DIR . ‘/vendor/autoload.php’;
use PhpAmqpLib\Connection\AMQPStreamConnection;

// RabbitMQ 默认端口5672, 默认账号密码: guest / guest
// $host = ‘localhost’;
// $host = ‘5672’;
$host = ‘192.168.99.100’;
$port = ‘5672’;
$username = ‘guest’;
$password = ‘guest’;

$connection = new AMQPStreamConnection($host, $port, $username, $password);
$channel = $connection->channel();

$queueName = ‘hello-test’;
$channel->queue_declare($queueName, false, false, false, false);

echo ‘ [*] Waiting for messages. To exit press CTRL+C’, “\n”;

$callback = function($msg) {
echo ” [x] Received: {“, $msg->body, “}\n”;
};

$channel->basic_consume($queueName, ”, false, true, false, false, $callback);

while(count($channel->callbacks)) {
$channel->wait();
}

发送和结束队列消息
先执行receive.php.
再执行send.php文件,来发送消息:

$ php send.php
[x] Sent: {18-02-10 02:35:15: Hello World!}
$ php send.php
[x] Sent: {18-02-10 02:35:34: Hello World!}

https://docs.phpcomposer.com/01-basic-usage.htmlhttp://www.rabbitmq.com/tutorials/tutorial-one-php.html

php-amqplib amqp-ext 性能对比

https://blog.forma-pro.com/php-amqp-clients-benchmark-them-all-8a4e6adb1a6b

RabbitMQ的用户角色分类:
none、management、policymaker、monitoring、administrator

RabbitMQ各类角色描述:
none
不能访问 management plugin

management
用户可以通过AMQP做的任何事外加:
列出自己可以通过AMQP登入的virtual hosts
查看自己的virtual hosts中的queues, exchanges 和 bindings
查看和关闭自己的channels 和 connections
查看有关自己的virtual hosts的“全局”的统计信息,包含其他用户在这些virtual hosts中的活动。

policymaker
management可以做的任何事外加:
查看、创建和删除自己的virtual hosts所属的policies和parameters

monitoring
management可以做的任何事外加:
列出所有virtual hosts,包括他们不能登录的virtual hosts
查看其他用户的connections和channels
查看节点级别的数据如clustering和memory使用情况
查看真正的关于所有virtual hosts的全局的统计信息

administrator
policymaker和monitoring可以做的任何事外加:
创建和删除virtual hosts
查看、创建和删除users
查看创建和删除permissions
关闭其他用户的connections

创建用户并设置角色:
可以创建管理员用户,负责整个MQ的运维,例如:
$sudo rabbitmqctl add_user user_admin passwd_admin
赋予其administrator角色:
$sudo rabbitmqctl set_user_tags user_admin administrator

可以创建RabbitMQ监控用户,负责整个MQ的监控,例如:
$sudo rabbitmqctl add_user user_monitoring passwd_monitor
赋予其monitoring角色:
$sudo rabbitmqctl set_user_tags user_monitoring monitoring

可以创建某个项目的专用用户,只能访问项目自己的virtual hosts
$sudo rabbitmqctl add_user user_proj passwd_proj
赋予其monitoring角色:
$sudo rabbitmqctl set_user_tags user_proj management

创建和赋角色完成后查看并确认:
$sudo rabbitmqctl list_users

RabbitMQ 权限控制
默认virtual host:”/”
默认用户:guest
guest具有”/”上的全部权限,仅能有localhost访问RabbitMQ包括Plugin,建议删除或更改密码。可通过将配置文件中loopback_users置孔来取消其本地访问的限制:
[{rabbit, [{loopback_users, []}]}]

用户仅能对其所能访问的virtual hosts中的资源进行操作。这里的资源指的是virtual hosts中的exchanges、queues等,操作包括对资源进行配置、写、读。配置权限可创建、删除、资源并修改资源的行为,写权限可向资源发送消息,读权限从资源获取消息。比如:
exchange和queue的declare与delete分别需要exchange和queue上的配置权限
exchange的bind与unbind需要exchange的读写权限
queue的bind与unbind需要queue写权限exchange的读权限
发消息(publish)需exchange的写权限
获取或清除(get、consume、purge)消息需queue的读权限

对何种资源具有配置、写、读的权限通过正则表达式来匹配,具体命令如下:
set_permissions [-p ]
其中, 的位置分别用正则表达式来匹配特定的资源,如’^(amq.gen.*|amq.default)$’可以匹配server生成的和默认的exchange,’^$’不匹配任何资源

需要注意的是RabbitMQ会缓存每个connection或channel的权限验证结果、因此权限发生变化后需要重连才能生效。

为用户赋权:
$sudo rabbitmqctl set_permissions -p /vhost1 user_admin ‘.’ ‘.’ ‘.*’
该命令使用户user_admin具有/vhost1这个virtual host中所有资源的配置、写、读权限以便管理其中的资源

按用户查看权限:

$sudo rabbitmqctl list_user_permissions user_admin

按服务器查看权限

$sudo rabbitmqctl list_permissions -p /vhost1

Fatal error: Uncaught exception ‘PhpAmqpLib\Exception\AMQPProtocolConnectionException’ with message ‘UNEXPECTED_FRAME – expected content header for class 60, got non content header frame instead’ in /opt/php-5.4.45/lib/composer/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php:715 Stack trace: #0 [internal function]: PhpAmqpLib\Connection\AbstractConnection->connection_close(Object(PhpAmqpLib\Wire\AMQPReader)) #1 /opt/php-5.4.45/lib/composer/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php(211): call_user_func(Array, Object(PhpAmqpLib\Wire\AMQPReader)) #2 /opt/php-5.4.45/lib/composer/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php(369): PhpAmqpLib\Channel\AbstractChannel->dispatch(‘10,50’, ‘\x01\xF9]UNEXPECTED_F…’, NULL) #3 /opt/php-5.4.45/lib/composer/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(644): PhpAmqpLib\Channel\AbstractChannel->wait() #4 /opt/php-5.4.45/lib/composer/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Chan in /opt/php-5.4.45/lib/composer/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php on line 715

后续安装参考
https://help.aliyun.com/document_detail/53100.html?spm=a2c4g.11186623.0.0.503b522f7G2rRn

Posted in 消息rabbitmq.

Tagged with , .


nginx自适应https的反向代理

方式一,写死地址,用变量

最简洁

map $scheme $online_proxy_www {
default 39.156.66.10;
}

proxy_pass $scheme://$online_proxy_www:$server_port;

方式二,写两个upstream,再用proxy_pass覆盖法

缺点需要维护2个upsteam

upstream online_proxy_www {
server 39.156.66.10:80;
}
upstream online_proxy_www_https {
server 39.156.66.10:443;
}

proxy_pass $scheme://online_proxy_www;

自适应https

if ( $scheme = https) {
proxy_pass $scheme://online_proxy_www_https;
}

方式三,upstream backup法

最简单,缺点会多一次请求,多个错误日志

upstream online_proxy_www {
server 39.156.66.10:80;39.156.66.10
server 39.156.66.10:443 backup;
}
proxy_pass $scheme://online_proxy_www;

==========================

nginx.conf示例

upstream online_proxy_www {
    server   39.156.66.10:80;
    #server   39.156.66.10:443 backup;
}
upstream online_proxy_www_https {
    server   39.156.66.10:443;
}

server
{
    listen       80;
    listen       443 ssl;
    server_name  blog.c1gstudio.com;
    index index.html index.htm index.php;
    root  /opt/htdocs/www;
    access_log  /var/log/nginx/blog.c1gstudio.com.log  access ;

    include ssl.conf;

    location /
    {
        proxy_set_header Host  $host;
        proxy_set_header X-Forwarded-For $proxypass_forwarded_for;
        proxy_pass $scheme://$online_proxy_www:$server_port;

        add_header      X-Cache   C1GPROXY1;
    }

}

Posted in Nginx.

Tagged with , .


部分贴子丢失

服务商自动升级了DB,导致乱码而丢失了部分贴子.

Posted in 技术.


openresty+lua实现WAF应用防火墙

1.#基础包安装

yum -y install readline-devel pcre-devel zlib zlib-devel gcc

2.升级openssl

#yum -y openssl-devel

openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

wget –no-check-certificate https://www.openssl.org/source/openssl-1.1.1l.tar.gz
tar -zxvf openssl-1.1.1l.tar.gz
cd openssl-1.1.1l
./config shared zlib
make && make install

#ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key

备份当前Openssl
mv /usr/bin/openssl /usr/bin/openssl.old
mv /usr/lib64/openssl /usr/lib64/openssl.old

使用新版Openssl
ln -s /usr/local/bin/openssl /usr/bin/openssl
ln -s /usr/local/include/openssl/ /usr/include/openssl
ln -s /usr/local/lib64/libssl.so.1.1 /usr/lib64/libssl.so.1.1
ln -s /usr/local/lib64/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1

更新动态链接库数据
echo “/usr/local/lib/” >> /etc/ld.so.conf
ldconfig

openssl version
openssl: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
ln -s /usr/local/lib64/libssl.so.1.1 /usr/lib64/libssl.so.1.1
ln -s /usr/local/lib64/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1

openssl version
OpenSSL 1.1.1l 24 Aug 2021

3.安装pcre

pcre没找到,编辑时加上–with-pcre=../pcre-8.30 \
0.10/src/ngx_stream_lua_regex.c:205: undefined reference to `pcre_jit_stack_alloc’
collect2: ld returned 1 exit status

#wget -nv http://downloads.sourceforge.net/project/pcre/pcre/8.30/pcre-8.30.tar.gz -O pcre-8.30.tar.gz

wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.30.tar.gz
tar xzvf pcre-8.30.tar.gz
cd pcre-8.30
./configure –enable-utf8 –enable-unicode-properties
make
make install
ln -s /lib64/libpcre.so.0.0.1 /lib64/libpcre.so.1

4.下载ngx_cache_purge清缓组件

wget http://labs.frickle.com/files/ngx_cache_purge-2.3.tar.gz
tar zxvf ngx_cache_purge-2.3.tar.gz

5.安装openresty

wget https://openresty.org/download/openresty-1.19.9.1.tar.gz
tar zxvf openresty-1.19.9.1.tar.gz
cd openresty-1.19.9.1

伪装openresty为xcdn
sed -i ‘s/”openresty/”xcdn/g’ bundle/nginx-1.19.9/src/core/nginx.h
sed -i “s#Server: openresty#Server: xcdn#” bundle/nginx-1.19.9/src/http/ngx_http_header_filter_module.c
sed -i “s#\”


openresty\”#\”


xcdn\”#” bundle/nginx-1.19.9/src//http/ngx_http_special_response.c

./configure –user=www –group=website –prefix=/opt/openresty-1.19.9.1 \
–with-http_ssl_module –with-http_v2_module –with-http_realip_module –with-http_addition_module \
–with-http_geoip_module \
–with-http_gzip_static_module \
–with-http_auth_request_module \
–with-http_secure_link_module \
–with-http_degradation_module \
–with-http_stub_status_module \
–add-module=../ngx_cache_purge-2.3 \
–with-pcre=../pcre-8.30 \
–with-cc-opt=”-I /usr/local/include/openssl/ ” \
–with-ld-opt=”-L/usr/local/lib64″

gmake
gmake install

复制原配置文件
cd /opt/nginx/conf/
cp -ar ssl webip.conf geo.*.conf GeoIP.dat manageip.conf fcgi.conf htpasswd nginx.conf /opt/openresty-1.19.9.1/nginx/conf/

4.下载和配置 ngx_lua_waf

nginx下常见的开源 waf 有 mod_security、naxsi、ngx_lua_waf 这三个,ngx_lua_waf 性能高和易用性强,基本上零配置,而且常见的攻击类型都能防御,是比较省心的选择。

其git 地址为 https://github.com/loveshell/ngx_lua_waf
wget --no-check-certificate https://github.com/loveshell/ngx_lua_waf/archive/master.zip<br>unzip master.zip<br>mv ngx_lua_waf-master /opt/openresty-1.19.9.1/nginx/conf/waf<br>chown -R www:website /opt/openresty-1.19.9.1/nginx/conf<br>chown -R www:website /opt/openresty-1.19.9.1/nginx/logs<br>chmod 775 /opt/openresty-1.19.9.1/nginx/conf<br>chmod 775 /opt/openresty-1.19.9.1/nginx/conf/waf<br>chmod 775 /opt/openresty-1.19.9.1/nginx/conf/waf/wafconf<br>chmod 664 /opt/openresty-1.19.9.1/nginx/conf/<em>.</em>

mkdir -p /opt/openresty-1.19.9.1/nginx/logs/hack
chown www:website /opt/openresty-1.19.9.1/nginx/logs/hack<br>chmod -R 775 /opt/openresty-1.19.9.1/nginx/logs/hack

测试配置文件
/opt/openresty-1.19.9.1/nginx/sbin/nginx -t

注意和tengine的不兼容配置
注释掉server_info和#server_tag
[emerg] unknown directive “server_info” in /opt/openresty-1.19.9.1/nginx/conf/nginx.conf:49

#server_info off;

#server_tag off;

在http段增加清除server头
more_clear_headers "Server";

去除dso段

limit_req_zone 不支持多个key
ginx: [emerg] invalid number of arguments in “limit_req_zone” directive in /opt/openresty-1.19.9.1/nginx/conf/nginx.conf:90

#limit_req_zone $binary_remote_addr $uri zone=two:30m rate=20r/s;

5.配置ngx_lua_waf

https://github.com/loveshell/ngx_lua_waf在nginx.conf的http段添加

lua_package_path "/opt/openresty-1.19.9.1/lualib/?.lua;/opt/nginx/conf/waf/?.lua";
lua_shared_dict limit 10m;
init_by_lua_file  /opt/openresty-1.19.9.1/nginx/conf/waf/init.lua; 
access_by_lua_file /opt/openresty-1.19.9.1/nginx/conf/waf/waf.lua;

配置config.lua里的waf规则目录(一般在waf/conf/目录下)

RulePath = "/opt/nginx/conf/waf/wafconf/"

然后重启nginx即可
部署完毕可以尝试如下命令:

curl http://xxxx/test.php?id=../etc/passwd
日志文件名称格式如下:虚拟主机名_sec.log

配置时一些错误
nginx: [alert] failed to load the ‘resty.core’ module (https://github.com/openresty/lua-resty-core); ensure you are using an OpenResty release from https://openresty.org/en/download.html (reason: module ‘resty.core’ not found:
no field package.preload[‘resty.core’]
no file ‘/opt/nginx/conf/waf/resty/core.lua’
no file ‘/opt/openresty-1.19.9.1/lualib/resty/resty/core.lua’
no file ‘/opt/openresty-1.19.9.1/site/lualib/resty/core.so’
no file ‘/opt/openresty-1.19.9.1/lualib/resty/core.so’
no file ‘./resty/core.so’
no file ‘/usr/local/lib/lua/5.1/resty/core.so’
no file ‘/opt/openresty-1.19.9.1/luajit/lib/lua/5.1/resty/core.so’
no file ‘/usr/local/lib/lua/5.1/loadall.so’
no file ‘/opt/openresty-1.19.9.1/site/lualib/resty.so’
no file ‘/opt/openresty-1.19.9.1/lualib/resty.so’
no file ‘./resty.so’
no file ‘/usr/local/lib/lua/5.1/resty.so’
no file ‘/opt/openresty-1.19.9.1/luajit/lib/lua/5.1/resty.so’
no file ‘/usr/local/lib/lua/5.1/loadall.so’) in /opt/openresty-1.19.9.1/nginx/conf/nginx.conf:214

将/opt/openresty-1.19.9.1/lualib/?.lua; 加进lua_package_path
lua_package_path “/opt/openresty-1.19.9.1/lualib/?.lua;/opt/nginx/conf/waf/?.lua”;

user-agent中去除|bench,允许NetworkBench访问
“Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.77.34.5 Safari/537.36; NetworkBench/79.0.3917.0-8710402-0” “(HTTrack|harvest|audit|dirbuster|pangolin|nmap|sqln|-scan|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|PycURL|zmeu|BabyKrokodil|netsparker|httperf|bench| SF/)”

user-agent中去除|PycURL,允许监控访问

openresty编辑说明
–help this message

–prefix=PATH set the installation prefix (default to /usr/local/openresty)

–with-debug enable debug logging

–with-no-pool-patch enable the no-pool patch for debugging memory issues

-jN pass -jN option to make while building LuaJIT 2.1

–without-http_echo_module disable ngx_http_echo_module
–without-http_xss_module disable ngx_http_xss_module
–without-http_coolkit_module disable ngx_http_coolkit_module
–without-http_set_misc_module disable ngx_http_set_misc_module
–without-http_form_input_module disable ngx_http_form_input_module
–without-http_encrypted_session_module
disable ngx_http_encrypted_session_module
–without-http_srcache_module disable ngx_http_srcache_module
–without-http_lua_module disable ngx_http_lua_module
–without-http_lua_upstream_module disable ngx_http_lua_upstream_module
–without-http_headers_more_module disable ngx_http_headers_more_module
–without-http_array_var_module disable ngx_http_array_var_module
–without-http_memc_module disable ngx_http_memc_module
–without-http_redis2_module disable ngx_http_redis2_module
–without-http_redis_module disable ngx_http_redis_module
–without-http_rds_json_module disable ngx_http_rds_json_module
–without-http_rds_csv_module disable ngx_http_rds_csv_module
–without-stream_lua_module disable ngx_stream_lua_module
–without-ngx_devel_kit_module disable ngx_devel_kit_module
–without-stream disable TCP/UDP proxy module
–without-http_ssl_module disable ngx_http_ssl_module
–without-stream_ssl_module disable ngx_stream_ssl_module

–with-http_iconv_module enable ngx_http_iconv_module
–with-http_drizzle_module enable ngx_http_drizzle_module
–with-http_postgres_module enable ngx_http_postgres_module

–without-lua_cjson disable the lua-cjson library
–without-lua_tablepool disable the lua-tablepool library (and by consequence, the
lua-resty-shell library)
–without-lua_redis_parser disable the lua-redis-parser library
–without-lua_rds_parser disable the lua-rds-parser library
–without-lua_resty_dns disable the lua-resty-dns library
–without-lua_resty_memcached disable the lua-resty-memcached library
–without-lua_resty_redis disable the lua-resty-redis library
–without-lua_resty_mysql disable the lua-resty-mysql library
–without-lua_resty_upload disable the lua-resty-upload library
–without-lua_resty_upstream_healthcheck
disable the lua-resty-upstream-healthcheck library
–without-lua_resty_string disable the lua-resty-string library
–without-lua_resty_websocket disable the lua-resty-websocket library
–without-lua_resty_limit_traffic disable the lua-resty-limit-traffic library
–without-lua_resty_lock disable the lua-resty-lock library
–without-lua_resty_lrucache disable the lua-resty-lrucache library
–without-lua_resty_signal disable the lua-resty-signal library (and by consequence,
the lua-resty-shell library)
–without-lua_resty_shell disable the lua-resty-shell library
–without-lua_resty_core disable the lua-resty-core library

–with-luajit=DIR use the external LuaJIT 2.1 installation specified by DIR
–with-luajit-xcflags=FLAGS Specify extra C compiler flags for LuaJIT 2.1
–with-luajit-ldflags=FLAGS Specify extra C linker flags for LuaJIT 2.1
–without-luajit-lua52 Turns off the LuaJIT extensions from Lua 5.2 that may break
backward compatibility
–without-luajit-gc64 Turns off the LuaJIT GC64 mode (which is enabled by default
on x86_64)

–with-libdrizzle=DIR specify the libdrizzle 1.0 (or drizzle) installation prefix
–with-libpq=DIR specify the libpq (or postgresql) installation prefix
–with-pg_config=PATH specify the path of the pg_config utility

Options directly inherited from nginx

–sbin-path=PATH set nginx binary pathname
–modules-path=PATH set modules path
–conf-path=PATH set nginx.conf pathname
–error-log-path=PATH set error log pathname
–pid-path=PATH set nginx.pid pathname
–lock-path=PATH set nginx.lock pathname

–user=USER set non-privileged user for
worker processes
–group=GROUP set non-privileged group for
worker processes

–build=NAME set build name
–builddir=DIR set build directory

–with-select_module enable select module
–without-select_module disable select module
–with-poll_module enable poll module
–without-poll_module disable poll module

–with-threads enable thread pool support

–with-file-aio enable file AIO support

–with-http_ssl_module enable ngx_http_ssl_module (default on)
–with-http_v2_module enable ngx_http_v2_module
–with-http_realip_module enable ngx_http_realip_module
–with-http_addition_module enable ngx_http_addition_module
–with-http_xslt_module enable ngx_http_xslt_module
–with-http_xslt_module=dynamic enable dynamic ngx_http_xslt_module
–with-http_image_filter_module enable ngx_http_image_filter_module
–with-http_image_filter_module=dynamic
enable dynamic ngx_http_image_filter_module
–with-http_geoip_module enable ngx_http_geoip_module
–with-http_geoip_module=dynamic enable dynamic ngx_http_geoip_module
–with-http_sub_module enable ngx_http_sub_module
–with-http_dav_module enable ngx_http_dav_module
–with-http_flv_module enable ngx_http_flv_module
–with-http_mp4_module enable ngx_http_mp4_module
–with-http_gunzip_module enable ngx_http_gunzip_module
–with-http_gzip_static_module enable ngx_http_gzip_static_module
–with-http_auth_request_module enable ngx_http_auth_request_module
–with-http_random_index_module enable ngx_http_random_index_module
–with-http_secure_link_module enable ngx_http_secure_link_module
–with-http_degradation_module enable ngx_http_degradation_module
–with-http_slice_module enable ngx_http_slice_module
–with-http_stub_status_module enable ngx_http_stub_status_module

–without-http_charset_module disable ngx_http_charset_module
–without-http_gzip_module disable ngx_http_gzip_module
–without-http_ssi_module disable ngx_http_ssi_module
–without-http_userid_module disable ngx_http_userid_module
–without-http_access_module disable ngx_http_access_module
–without-http_auth_basic_module disable ngx_http_auth_basic_module
–without-http_mirror_module disable ngx_http_mirror_module
–without-http_autoindex_module disable ngx_http_autoindex_module
–without-http_geo_module disable ngx_http_geo_module
–without-http_map_module disable ngx_http_map_module
–without-http_split_clients_module disable ngx_http_split_clients_module
–without-http_referer_module disable ngx_http_referer_module
–without-http_rewrite_module disable ngx_http_rewrite_module
–without-http_proxy_module disable ngx_http_proxy_module
–without-http_fastcgi_module disable ngx_http_fastcgi_module
–without-http_uwsgi_module disable ngx_http_uwsgi_module
–without-http_scgi_module disable ngx_http_scgi_module
–without-http_grpc_module disable ngx_http_grpc_module
–without-http_memcached_module disable ngx_http_memcached_module
–without-http_limit_conn_module disable ngx_http_limit_conn_module
–without-http_limit_req_module disable ngx_http_limit_req_module
–without-http_empty_gif_module disable ngx_http_empty_gif_module
–without-http_browser_module disable ngx_http_browser_module
–without-http_upstream_hash_module
disable ngx_http_upstream_hash_module
–without-http_upstream_ip_hash_module
disable ngx_http_upstream_ip_hash_module
–without-http_upstream_least_conn_module
disable ngx_http_upstream_least_conn_module
–without-http_upstream_random_module
disable ngx_http_upstream_random_module
–without-http_upstream_keepalive_module
disable ngx_http_upstream_keepalive_module
–without-http_upstream_zone_module
disable ngx_http_upstream_zone_module

–with-http_perl_module enable ngx_http_perl_module
–with-http_perl_module=dynamic enable dynamic ngx_http_perl_module
–with-perl_modules_path=PATH set Perl modules path
–with-perl=PATH set perl binary pathname

–http-log-path=PATH set http access log pathname
–http-client-body-temp-path=PATH set path to store
http client request body temporary files
–http-proxy-temp-path=PATH set path to store
http proxy temporary files
–http-fastcgi-temp-path=PATH set path to store
http fastcgi temporary files
–http-uwsgi-temp-path=PATH set path to store
http uwsgi temporary files
–http-scgi-temp-path=PATH set path to store
http scgi temporary files

–without-http disable HTTP server
–without-http-cache disable HTTP cache

–with-mail enable POP3/IMAP4/SMTP proxy module
–with-mail=dynamic enable dynamic POP3/IMAP4/SMTP proxy module
–with-mail_ssl_module enable ngx_mail_ssl_module
–without-mail_pop3_module disable ngx_mail_pop3_module
–without-mail_imap_module disable ngx_mail_imap_module
–without-mail_smtp_module disable ngx_mail_smtp_module

–with-stream enable TCP/UDP proxy module (default on)
–with-stream=dynamic enable dynamic TCP/UDP proxy module
–with-stream_ssl_module enable ngx_stream_ssl_module (default on)
–with-stream_realip_module enable ngx_stream_realip_module
–with-stream_geoip_module enable ngx_stream_geoip_module
–with-stream_geoip_module=dynamic enable dynamic ngx_stream_geoip_module
–with-stream_ssl_preread_module enable ngx_stream_ssl_preread_module
–without-stream_limit_conn_module disable ngx_stream_limit_conn_module
–without-stream_access_module disable ngx_stream_access_module
–without-stream_geo_module disable ngx_stream_geo_module
–without-stream_map_module disable ngx_stream_map_module
–without-stream_split_clients_module
disable ngx_stream_split_clients_module
–without-stream_return_module disable ngx_stream_return_module
–without-stream_upstream_hash_module
disable ngx_stream_upstream_hash_module
–without-stream_upstream_least_conn_module
disable ngx_stream_upstream_least_conn_module
–without-stream_upstream_random_module
disable ngx_stream_upstream_random_module
–without-stream_upstream_zone_module
disable ngx_stream_upstream_zone_module

–with-google_perftools_module enable ngx_google_perftools_module
–with-cpp_test_module enable ngx_cpp_test_module

–add-module=PATH enable external module
–add-dynamic-module=PATH enable dynamic external module

–with-compat dynamic modules compatibility

–with-cc=PATH set C compiler pathname
–with-cpp=PATH set C preprocessor pathname
–with-cc-opt=OPTIONS set additional C compiler options
–with-ld-opt=OPTIONS set additional linker options
–with-cpu-opt=CPU build for the specified CPU, valid values:
pentium, pentiumpro, pentium3, pentium4,
athlon, opteron, sparc32, sparc64, ppc64

–without-pcre disable PCRE library usage
–with-pcre force PCRE library usage
–with-pcre=DIR set path to PCRE library sources
–with-pcre-opt=OPTIONS set additional build options for PCRE
–with-pcre-jit build PCRE with JIT compilation support

–with-zlib=DIR set path to zlib library sources
–with-zlib-opt=OPTIONS set additional build options for zlib
–with-zlib-asm=CPU use zlib assembler sources optimized
for the specified CPU, valid values:
pentium, pentiumpro

–with-libatomic force libatomic_ops library usage
–with-libatomic=DIR set path to libatomic_ops library sources

–with-openssl=DIR set path to OpenSSL library sources
–with-openssl-opt=OPTIONS set additional build options for OpenSSL

–dry-run dry running the configure, for testing only
–platform=PLATFORM forcibly specify a platform name, for testing only

参考:
接入层限流之OpenResty提供的Lua限流模块lua-resty-limit-tra
https://www.jianshu.com/p/687e63118d84

又拍云张聪:OpenResty 动态流控的几种姿势
https://www.cnblogs.com/upyun/p/10307741.html

Posted in Nginx.

Tagged with , .


升级安装Tenginer2.3.3+lua+ngx_lua支持WAF防护功能

1、安装或升级autoconf

安装jemalloc是可能出现autoconf版本不够高
configure.ac:2: error: Autoconf version 2.68 or higher is required
configure.ac:2: the top level
autom4te: /usr/bin/m4 failed with exit status: 63
Error 0 in autoconf

升级autoconf
yum -y install autogen autoconf
Package autoconf-2.63-5.1.el6.noarch already installed and latest version
Nothing to do

查询当前autoconf版本
rpm -qf /usr/bin/autoconf
autoconf-2.63-5.1.el6.noarch

下载并编译
wget ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.68.tar.gz
tar zxvf autoconf-2.68.tar.gz
cd autoconf-2.68
./configure –prefix=/usr/

卸载当前版本
rpm -e –nodeps autoconf-2.63

安装
make && make install

查看版本
autoconf -V
autoconf (GNU Autoconf) 2.68
cd ..

2、jemalloc安装

github 地址 :https://github.com/jemalloc/jemalloc/tree/master
编译安装jemalloc

wget –no-check-certificate https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2

wget -O jemalloc-5.2.1.tar.gz https://github.com/jemalloc/jemalloc/archive/refs/tags/5.2.1.tar.gz
tar -xf jemalloc-5.2.1.tar.gz && cd jemalloc-5.2.1
./autogen.sh

make -j8
make install

如果/usr/local/lib 没在ldconfig中
echo ‘/usr/local/lib’ > /etc/ld.so.conf.d/local.conf
ldconfig
cd ..

3、lua安装

curl -R -O http://www.lua.org/ftp/lua-5.4.3.tar.gz
tar zxf lua-5.4.3.tar.gz
cd lua-5.4.3
make linux
make install
cd ..

4、安装luajit

然后从https://github.com/openresty/luajit2下载安装
https://github.com/openresty/luajit2/archive/refs/heads/v2.1-agentzh.zip
unzip luajit2-2.1-agentzh.zip
cd luajit2-2.1-agentzh
make && make install
cd ..

5、安装tengine

tengine 编译说明
–prefix= 指向安装目录
–sbin-path 指向(执行)程序文件(nginx)
–conf-path= 指向配置文件(nginx.conf)
–error-log-path= 指向错误日志目录
–pid-path= 指向pid文件(nginx.pid)
–lock-path= 指向lock文件(nginx.lock)(安装文件锁定,防止安装文件被别人利用,或自己误操作。)
–user= 指定程序运行时的非特权用户
–group= 指定程序运行时的非特权用户组
–builddir= 指向编译目录
–with-rtsig_module 启用rtsig模块支持(实时信号)
–with-select_module 启用select模块支持(一种轮询模式,不推荐在高载环境下使用)禁用:–without-select_module
–with-poll_module 启用poll模块支持(功能与select相同,与select特性相同,为一种轮询模式,不推荐在高载环境下使用)
–with-file-aio 启用file aio支持(一种APL文件传输格式)
–with-ipv6 启用ipv6支持
–with-http_ssl_module 启用ngx_http_ssl_module支持(使支持https请求,需已安装openssl)
–with-http_realip_module 启用ngx_http_realip_module支持(这个模块允许从请求标头更改客户端的IP地址值,默认为关)
–with-http_addition_module 启用ngx_http_addition_module支持(作为一个输出过滤器,支持不完全缓冲,分部分响应请求)
–with-http_xslt_module 启用ngx_http_xslt_module支持(过滤转换XML请求)
–with-http_image_filter_module 启用ngx_http_image_filter_module支持(传输JPEG/GIF/PNG 图片的一个过滤器)(默认为不启用。gd库要用到)
–with-http_geoip_module 启用ngx_http_geoip_module支持(该模块创建基于与MaxMind GeoIP二进制文件相配的客户端IP地址的ngx_http_geoip_module变量)
–with-http_sub_module 启用ngx_http_sub_module支持(允许用一些其他文本替换nginx响应中的一些文本)
–with-http_dav_module 启用ngx_http_dav_module支持(增加PUT,DELETE,MKCOL:创建集合,COPY和MOVE方法)默认情况下为关闭,需编译开启
–with-http_flv_module 启用ngx_http_flv_module支持(提供寻求内存使用基于时间的偏移量文件)
–with-http_gzip_static_module 启用ngx_http_gzip_static_module支持(在线实时压缩输出数据流)
–with-http_random_index_module 启用ngx_http_random_index_module支持(从目录中随机挑选一个目录索引)
–with-http_secure_link_module 启用ngx_http_secure_link_module支持(计算和检查要求所需的安全链接网址)
–with-http_degradation_module 启用ngx_http_degradation_module支持(允许在内存不足的情况下返回204或444码)
–with-http_stub_status_module 启用ngx_http_stub_status_module支持(获取nginx自上次启动以来的工作状态)
–without-http_charset_module 禁用ngx_http_charset_module支持(重新编码web页面,但只能是一个方向–服务器端到客户端,并且只有一个字节的编码可以被重新编码)
–without-http_gzip_module 禁用ngx_http_gzip_module支持(该模块同-with-http_gzip_static_module功能一样)
–without-http_ssi_module 禁用ngx_http_ssi_module支持(该模块提供了一个在输入端处理处理服务器包含文件(SSI)的过滤器,目前支持SSI命令的列表是不完整的)
–without-http_userid_module 禁用ngx_http_userid_module支持(该模块用来处理用来确定客户端后续请求的cookies)
–without-http_access_module 禁用ngx_http_access_module支持(该模块提供了一个简单的基于主机的访问控制。允许/拒绝基于ip地址)
–without-http_auth_basic_module禁用ngx_http_auth_basic_module(该模块是可以使用用户名和密码基于http基本认证方法来保护你的站点或其部分内容)
–without-http_autoindex_module 禁用disable ngx_http_autoindex_module支持(该模块用于自动生成目录列表,只在ngx_http_index_module模块未找到索引文件时发出请求。)
–without-http_geo_module 禁用ngx_http_geo_module支持(创建一些变量,其值依赖于客户端的IP地址)
–without-http_map_module 禁用ngx_http_map_module支持(使用任意的键/值对设置配置变量)
–without-http_split_clients_module 禁用ngx_http_split_clients_module支持(该模块用来基于某些条件划分用户。条件如:ip地址、报头、cookies等等)
–without-http_referer_module 禁用disable ngx_http_referer_module支持(该模块用来过滤请求,拒绝报头中Referer值不正确的请求)
–without-http_rewrite_module 禁用ngx_http_rewrite_module支持(该模块允许使用正则表达式改变URI,并且根据变量来转向以及选择配置。如果在server级 别设置该选项,那么他们将在 location之前生效。如果在location还有更进一步的重写规则,location部分的规则依然会被执行。如果这个URI重写是因为 location部分的规则造成的,那么 location部分会再次被执行作为新的URI。 这个循环会执行10次,然后Nginx会返回一个500错误。)
–without-http_proxy_module 禁用ngx_http_proxy_module支持(有关代理服务器)
–without-http_fastcgi_module 禁用ngx_http_fastcgi_module支持(该模块允许Nginx 与FastCGI 进程交互,并通过传递参数来控制FastCGI 进程工作。 )FastCGI一个常驻型的公共网关接口。
–without-http_uwsgi_module 禁用ngx_http_uwsgi_module支持(该模块用来医用uwsgi协议,uWSGI服务器相关)
–without-http_scgi_module 禁用ngx_http_scgi_module支持(该模块用来启用SCGI协议支持,SCGI协议是CGI协议的替代。它是一种应用程序与HTTP服务接口标准。它有些像FastCGI但他的设计 更容易实现。)
–without-http_memcached_module 禁用ngx_http_memcached_module支持(该模块用来提供简单的缓存,以提高系统效率)
-without-http_limit_zone_module 禁用ngx_http_limit_zone_module支持(该模块可以针对条件,进行会话的并发连接数控制)
–without-http_limit_req_module 禁用ngx_http_limit_req_module支持(该模块允许你对于一个地址进行请求数量的限制用一个给定的session或一个特定的事件)
–without-http_empty_gif_module 禁用ngx_http_empty_gif_module支持(该模块在内存中常驻了一个1*1的透明GIF图像,可以被非常快速的调用)
–without-http_browser_module 禁用ngx_http_browser_module支持(该模块用来创建依赖于请求报头的值。如果浏览器为modern ,则$modern_browser等于modern_browser_value指令分配的值;如 果浏览器为old,则$ancient_browser等于 ancient_browser_value指令分配的值;如果浏览器为 MSIE中的任意版本,则 $msie等于1)
–without-http_upstream_ip_hash_module 禁用ngx_http_upstream_ip_hash_module支持(该模块用于简单的负载均衡)
–with-http_perl_module 启用ngx_http_perl_module支持(该模块使nginx可以直接使用perl或通过ssi调用perl)
–with-perl_modules_path= 设定perl模块路径
–with-perl= 设定perl库文件路径
–http-log-path= 设定access log路径
–http-client-body-temp-path= 设定http客户端请求临时文件路径
–http-proxy-temp-path= 设定http代理临时文件路径
–http-fastcgi-temp-path= 设定http fastcgi临时文件路径
–http-uwsgi-temp-path= 设定http uwsgi临时文件路径
–http-scgi-temp-path= 设定http scgi临时文件路径
-without-http 禁用http server功能
–without-http-cache 禁用http cache功能
–with-mail 启用POP3/IMAP4/SMTP代理模块支持
–with-mail_ssl_module 启用ngx_mail_ssl_module支持
–without-mail_pop3_module 禁用pop3协议(POP3即邮局协议的第3个版本,它是规定个人计算机如何连接到互联网上的邮件服务器进行收发邮件的协议。是因特网电子邮件的第一个离 线协议标 准,POP3协议允许用户从服务器上把邮件存储到本地主机上,同时根据客户端的操作删除或保存在邮件服务器上的邮件。POP3协议是TCP/IP协议族中 的一员,主要用于 支持使用客户端远程管理在服务器上的电子邮件)
–without-mail_imap_module 禁用imap协议(一种邮件获取协议。它的主要作用是邮件客户端可以通过这种协议从邮件服务器上获取邮件的信息,下载邮件等。IMAP协议运行在 TCP/IP协议之上, 使用的端口是143。它与POP3协议的主要区别是用户可以不用把所有的邮件全部下载,可以通过客户端直接对服务器上的邮件进行操作。)
–without-mail_smtp_module 禁用smtp协议(SMTP即简单邮件传输协议,它是一组用于由源地址到目的地址传送邮件的规则,由它来控制信件的中转方式。SMTP协议属于 TCP/IP协议族,它帮助每台计算机在发送或中转信件时找到下一个目的地。)
–with-google_perftools_module 启用ngx_google_perftools_module支持(调试用,剖析程序性能瓶颈)
–with-cpp_test_module 启用ngx_cpp_test_module支持
–add-module= 启用外部模块支持
–with-cc= 指向C编译器路径
–with-cpp= 指向C预处理路径
–with-cc-opt= 设置C编译器参数(PCRE库,需要指定–with-cc-opt=”-I /usr/local/include”,如果使用select()函数则需要同时增加文件描述符数量,可以通过–with-cc- opt=”-D FD_SETSIZE=2048”指定。)
–with-ld-opt= 设置连接文件参数。(PCRE库,需要指定–with-ld-opt=”-L /usr/local/lib”。)
–with-cpu-opt= 指定编译的CPU,可用的值为: pentium, pentiumpro, pentium3, pentium4, athlon, opteron, amd64, sparc32, sparc64, ppc64
–without-pcre 禁用pcre库
–with-pcre 启用pcre库
–with-pcre= 指向pcre库文件目录
–with-pcre-opt= 在编译时为pcre库设置附加参数
–with-md5= 指向md5库文件目录(消息摘要算法第五版,用以提供消息的完整性保护)
–with-md5-opt= 在编译时为md5库设置附加参数
–with-md5-asm 使用md5汇编源
–with-sha1= 指向sha1库目录(数字签名算法,主要用于数字签名)
–with-sha1-opt= 在编译时为sha1库设置附加参数
–with-sha1-asm 使用sha1汇编源
–with-zlib= 指向zlib库目录
–with-zlib-opt= 在编译时为zlib设置附加参数
–with-zlib-asm= 为指定的CPU使用zlib汇编源进行优化,CPU类型为pentium, pentiumpro
–with-libatomic 为原子内存的更新操作的实现提供一个架构
–with-libatomic= 指向libatomic_ops安装目录
–with-openssl= 指向openssl安装目录
–with-openssl-opt 在编译时为openssl设置附加参数
–with-debug 启用debug日志

wget http://tengine.taobao.org/download/tengine-2.3.3.tar.gz
tar zxvf tengine-2.3.3.tar.gz
cd tengine-2.3.3

./configure –user=www –group=website –prefix=/opt/tengine-2.3.3 \
–with-http_ssl_module –with-http_v2_module –with-http_realip_module –with-http_addition_module \
–with-http_geoip_module \
–with-http_gzip_static_module \
–with-http_auth_request_module \
–with-http_secure_link_module \
–with-http_degradation_module \
–with-http_stub_status_module \
–add-module=modules/ngx_http_concat_module \
–add-module=modules/ngx_http_footer_filter_module \
–add-module=modules/ngx_http_proxy_connect_module \
–add-module=modules/ngx_http_reqstat_module \
–add-module=modules/ngx_http_sysguard_module \
–add-module=modules/ngx_http_trim_filter_module \
–add-module=modules/ngx_http_upstream_check_module \
–add-module=modules/ngx_http_upstream_consistent_hash_module \
–add-module=modules/ngx_http_upstream_dynamic_module \
–add-module=modules/ngx_http_upstream_dyups_module \
–add-module=modules/ngx_http_upstream_session_sticky_module \
–add-module=modules/ngx_http_upstream_vnswrr_module \
–add-module=modules/ngx_http_user_agent_module \
–add-module=modules/ngx_multi_upstream_module \
–with-jemalloc \
–with-http_lua_module \
–with-luajit-lib=/usr/local/lib/ –with-luajit-inc=/usr/local/include/luajit-2.1/ –with-ld-opt=-Wl,-rpath,/usr/local/lib

make -j4
make install

cd /opt/tengine-2.3.3/
sbin/nginx -V
Tengine version: Tengine/2.3.3
nginx version: nginx/1.18.0
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC)
built with OpenSSL 1.0.2u 20 Dec 2019
TLS SNI support enabled
configure arguments: –user=www –group=website –prefix=/opt/tengine-2.3.3 –with-http_ssl_module –with-http_v2_module –with-http_realip_module –with-http_addition_module –with-http_geoip_module –with-http_gzip_static_module –with-http_auth_request_module –with-http_secure_link_module –with-http_degradation_module –with-http_stub_status_module –add-module=modules/ngx_http_concat_module –add-module=modules/ngx_http_footer_filter_module –add-module=modules/ngx_http_proxy_connect_module –add-module=modules/ngx_http_reqstat_module –add-module=modules/ngx_http_sysguard_module –add-module=modules/ngx_http_trim_filter_module –add-module=modules/ngx_http_upstream_check_module –add-module=modules/ngx_http_upstream_consistent_hash_module –add-module=modules/ngx_http_upstream_dynamic_module –add-module=modules/ngx_http_upstream_dyups_module –add-module=modules/ngx_http_upstream_session_sticky_module –add-module=modules/ngx_http_upstream_vnswrr_module –add-module=modules/ngx_http_user_agent_module –add-module=modules/ngx_multi_upstream_module –with-jemalloc –with-http_lua_module –with-luajit-lib=/usr/local/lib/ –with-luajit-inc=/usr/local/include/luajit-2.1/ –with-ld-opt=-Wl,-rpath,/usr/local/lib

查看安装的模块

sbin/nginx -m
Tengine version: Tengine/2.3.3
nginx version: nginx/1.18.0
nginx: loaded modules:
nginx: ngx_core_module (static)
nginx: ngx_errlog_module (static)
nginx: ngx_conf_module (static)
nginx: ngx_openssl_module (static)
nginx: ngx_regex_module (static)
nginx: ngx_events_module (static)
nginx: ngx_event_core_module (static)
nginx: ngx_epoll_module (static)
nginx: ngx_procs_module (static)
nginx: ngx_proc_core_module (static)
nginx: ngx_http_module (static)
nginx: ngx_http_core_module (static)
nginx: ngx_http_log_module (static)
nginx: ngx_http_upstream_module (static)
nginx: ngx_http_v2_module (static)
nginx: ngx_http_static_module (static)
nginx: ngx_http_gzip_static_module (static)
nginx: ngx_http_autoindex_module (static)
nginx: ngx_http_index_module (static)
nginx: ngx_http_mirror_module (static)
nginx: ngx_http_try_files_module (static)
nginx: ngx_http_auth_request_module (static)
nginx: ngx_http_auth_basic_module (static)
nginx: ngx_http_access_module (static)
nginx: ngx_http_limit_conn_module (static)
nginx: ngx_http_limit_req_module (static)
nginx: ngx_http_realip_module (static)
nginx: ngx_http_geo_module (static)
nginx: ngx_http_geoip_module (static)
nginx: ngx_http_map_module (static)
nginx: ngx_http_split_clients_module (static)
nginx: ngx_http_referer_module (static)
nginx: ngx_http_rewrite_module (static)
nginx: ngx_http_ssl_module (static)
nginx: ngx_http_proxy_module (static)
nginx: ngx_http_fastcgi_module (static)
nginx: ngx_http_uwsgi_module (static)
nginx: ngx_http_scgi_module (static)
nginx: ngx_http_grpc_module (static)
nginx: ngx_http_memcached_module (static)
nginx: ngx_http_empty_gif_module (static)
nginx: ngx_http_browser_module (static)
nginx: ngx_http_secure_link_module (static)
nginx: ngx_http_degradation_module (static)
nginx: ngx_http_upstream_hash_module (static)
nginx: ngx_http_upstream_ip_hash_module (static)
nginx: ngx_http_upstream_least_conn_module (static)
nginx: ngx_http_upstream_random_module (static)
nginx: ngx_http_upstream_keepalive_module (static)
nginx: ngx_http_upstream_zone_module (static)
nginx: ngx_http_stub_status_module (static)
nginx: ngx_http_proxy_connect_module (static)
nginx: ngx_http_reqstat_module (static)
nginx: ngx_http_sysguard_module (static)
nginx: ngx_http_upstream_check_module (static)
nginx: ngx_http_upstream_consistent_hash_module (static)
nginx: ngx_http_upstream_dynamic_module (static)
nginx: ngx_http_dyups_module (static)
nginx: ngx_http_upstream_vnswrr_module (static)
nginx: ngx_http_user_agent_module (static)
nginx: ngx_http_multi_upstream_module (static)
nginx: ngx_http_write_filter_module (static)
nginx: ngx_http_header_filter_module (static)
nginx: ngx_http_chunked_filter_module (static)
nginx: ngx_http_v2_filter_module (static)
nginx: ngx_http_range_header_filter_module (static)
nginx: ngx_http_gzip_filter_module (static)
nginx: ngx_http_postpone_filter_module (static)
nginx: ngx_http_ssi_filter_module (static)
nginx: ngx_http_charset_filter_module (static)
nginx: ngx_http_addition_filter_module (static)
nginx: ngx_http_userid_filter_module (static)
nginx: ngx_http_headers_filter_module (static)
nginx: ngx_http_concat_module (static)
nginx: ngx_http_footer_filter_module (static)
nginx: ngx_http_trim_filter_module (static)
nginx: ngx_http_upstream_session_sticky_module (static)
nginx: ngx_http_lua_module (static)
nginx: ngx_http_copy_filter_module (static)
nginx: ngx_http_range_body_filter_module (static)
nginx: ngx_http_not_modified_filter_module (static)
nginx: the configuration file /opt/tengine-2.3.3/conf/nginx.conf syntax is ok
nginx: configuration file /opt/tengine-2.3.3/conf/nginx.conf test is successful

cd /opt/nginx/conf
cp -ar ssl ssl.conf webip.conf geo.*.conf GeoIP.dat manageip.conf fcgi.conf htpasswd nginx.conf /opt/tengine-2.3.3/conf/
/opt/tengine-2.3.3/sbin/nginx -t -c /opt/tengine-2.3.3/conf/nginx.conf

chown www:website /opt/tengine-2.3.3/logs/
chown www:website /opt/nginx/conf/

删除旧的nginx软链接,给tengine增加软链接
rm /opt/nginx
ln -s /opt/tengine-2.3.3 /opt/nginx

关闭tengine
/opt/nginx/sbin/nginx -s quit
启动tengine
/opt/nginx/sbin/nginx
重启tengine
/opt/nginx/sbin/nginx -s reload

6、下载和配置 ngx_lua_waf

nginx下常见的开源 waf 有 mod_security、naxsi、ngx_lua_waf 这三个,ngx_lua_waf 性能高和易用性强,基本上零配置,而且常见的攻击类型都能防御,是比较省心的选择。

其git 地址为 https://github.com/loveshell/ngx_lua_waf
wget –no-check-certificate https://github.com/loveshell/ngx_lua_waf/archive/master.zip
unzip master.zip
mv ngx_lua_waf-master /opt/tengine-2.3.3/conf/waf
chgrp -R website waf
chmod -R 0775 waf

mkdir -p /opt/nginx/logs/hack
chown www:website /opt/nginx/logs/hack
chmod -R 775 /opt/nginx/logs/hack

在nginx.conf的http段添加
lua_package_path “/opt/nginx/conf/waf/?.lua”;
lua_shared_dict limit 10m;
init_by_lua_file /opt/nginx/conf/waf/init.lua;
access_by_lua_file /opt/nginx/conf/waf/waf.lua;
配置config.lua里的waf规则目录(一般在waf/conf/目录下)

RulePath = "/opt/nginx/conf/waf/wafconf/"

然后重启nginx即可
部署完毕可以尝试如下命令:

curl http://xxxx/test.php?id=../etc/passwd
日志文件名称格式如下:虚拟主机名_sec.log

Posted in Nginx.

Tagged with , , .


ngx_cache_puage更新缓存404错误

在清除缓存时部分成功部分失败,有时通过浏览器可以但用程序就失败.

location ~ /purge(/.*)
{
    #设置只允许指定的IP或IP段才可以清除URL缓存。
    allow            127.0.0.1;
    allow            192.168.0.0/16;
    include manageip.conf;
    deny            all;
    proxy_cache_purge    cache_www   $host$1$is_args$args;
    error_page 405 =200 /purge$1; #处理squidclient purge的时候出现的405错误
}    
if ( $request_method = "PURGE" ) {
    rewrite ^(.*)$ /purge$1 last;
}
模拟测试
curl -H "Host:blog.c1gstudio.com" -H "User-Agent: c1gtest"  -X PURGE https://blog.c1gstudio.com/static/image/common/qrcode.png

404 Not Found

404 Not Found


openresty

去掉权限,并输出url,发现没有$1的输出
location ~ /purge(/.*)
{
#设置只允许指定的IP或IP段才可以清除URL缓存。
allow 127.0.0.1;
allow 192.168.0.0/16;
echo cache_bbs $host$1$is_args$args;
error_page 405 =200 /purge$1; #处理squidclient purge的时候出现的405错误
}

curl -H “Host:blog.c1gstudio.com” -H “User-Agent: c1gtest” -X PURGE https://blog.c1gstudio.com/static/image/common/qrcode.png
cache_bbs: blog.c1gstudio.com

可能是$1被其它脚本占用,重新定义一个 $purgeurl变量
location ~ /purge(?/.*)
{
    #设置只允许指定的IP或IP段才可以清除URL缓存。
    allow        127.0.0.1;
    allow        192.168.0.0/16;
    echo    cache_bbs   $host$purgeurl$is_args$args;
    error_page 405 =200 /purge$1; #处理squidclient purge的时候出现的405错误
}    

测试可以输出
curl -H "Host:blog.c1gstudio.com" -H "User-Agent: c1gtest"  -X PURGE https://blog.c1gstudio.com/static/image/common/qrcode.png
cache_bbs: blog.c1gstudio.com/static/image/common/qrcode.png
最后修改并测试成功
location ~ /purge(?/.*)
{
    #设置只允许指定的IP或IP段才可以清除URL缓存。
    allow        127.0.0.1;
    allow        192.168.0.0/16;
    include manageip.conf;
    deny            all;
    proxy_cache_purge    cache_bbs   $host$purgeurl$is_args$args;
    error_page 405 =200 /purge$purgeurl; #处理squidclient purge的时候出现的405错误
}    
if ( $request_method = "PURGE" ) {
    rewrite ^(.*)$ /purge$1 last;
}      

curl -H "Host:blog.c1gstudio.com" -H "User-Agent: c1gtest"  -X PURGE https://blog.c1gstudio.com/static/image/common/qrcode.png

Successful purge

Successful purge


Key : blog.c1gstudio.com/static/image/common/qrcode.png
Path: /dev/shm/nginx/proxy_cache_bbs/1/92/9a7ee4d7167bc0ead33f4ccdb4439921
openresty/1.19.9.1

Posted in Nginx.

Tagged with .


nginx用map获取X-Forwarded-For中的客户端ipv4/ipv6地址

网上搜搜连个像样的取ipv6代码也没有,这里写一个.如果有合法的http_x_forwarder_for就取第一或最后一个,没有则取直连IP.


$ForwardedFirstIp 容易伪造,但可以做流量控制用
$ForwardedLastIp 在前端代理可信情况下,可以做IP限制用

http
{

#获取左边第一个
map $http_x_forwarded_for $ForwardedFirstIp {
    default $remote_addr;
    ~^(?P
[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+|[0-9a-fA-F]+:[0-9a-fA-F:]+:[0-9a-fA-F\.]+),?.*$ $firstAddr;
}

#获取右边第一个
map $http_x_forwarded_for $ForwardedLastIp {
    default $remote_addr;
    ~,?(?P
[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+|[0-9a-fA-F]+:[0-9a-fA-F:]+:[0-9a-fA-F\.]+)$ $lastAddr;
}

}

支持ipv4;

支持ipv6缩写如 1080::8:800:200C:417A
支持正常ipv6如 2409:8907:a125:2e4d:c5d:9d59:c2d5:a13a
支技ipv4映射地址如 0:0:0:0:0:ffff:192.168.56.10
但不支持::简写回环等如 ::1 , ::192.168.56.10

2022/1/4更新
更严谨的正则语法
支持::1回环,支持 ::192.168.56.10
格式不正确返回空
没有forwared时取remote_addr

map $http_x_forwarded_for $ForwardedFirstIp {
    "" $remote_addr;
    "~(?([0-9]{1,3}\.){3}[0-9]{1,3}|[0-9a-fA-F]{0,4}:[0-9a-fA-F:]{0,30}:[0-9a-fA-F\.]{1,15}),?.*$" $firstAddr;
}
map $http_x_forwarded_for $ForwardedLastIp {
    "" $remote_addr;
    "~,?\s?(?([0-9]{1,3}\.){3}[0-9]{1,3}|[0-9a-fA-F]{0,4}:[0-9a-fA-F:]{0,30}:[0-9a-fA-F\.]{1,15})$" $lastAddr;
}

2022/1/13更新

支持[]包含的ipv6块



    map $http_x_forwarded_for $ForwardedFirstIp {
        default $remote_addr;
        "~\[?(?<firstAddr>([0-9]{1,3}\.){3}[0-9]{1,3}|[0-9a-fA-F]{0,4}:[0-9a-fA-F:]{0,30}:[0-9a-fA-F\.]{1,15})\]?,?.*$" $firstAddr;
    }
    map $http_x_forwarded_for $ForwardedLastIp {
        default $remote_addr;
        "~,?\s?\[?(?<lastAddr>([0-9]{1,3}\.){3}[0-9]{1,3}|[0-9a-fA-F]{0,4}:[0-9a-fA-F:]{0,30}:[0-9a-fA-F\.]{1,15})\]?$" $lastAddr;
    }

Posted in Nginx.