一.概述

服务器 web1 R410 E55202 16G SAS300G2 centos5.5 ip:192.168.1.21 web2(new) R410 E56062 16G SAS300G2 centos5.5 ip:192.168.1.23 db R410 E55042 16G SAS146G2 centos5.3

discuz论坛现有500多万贴子,近2万人在线,400万pv/日, 新增web2服务器分担论坛php处理. 尝试过在web2用nfs mount web1的程序来跑,但是访问速度太慢,无法接受 改用inotify+rsync效果不错,原来web1负载8左右,现两台各自3.5

web2安装

web2安装好php后配置php-fpm

192.168.1.23:9002 #监听地址及端口 rsyncuser#后面rsync同步的用户 website#同步用户的用户组 128#目前开128个 大概占用物理内存4G左右 192.168.1.21,192.168.1.23 #充许请求的服务器(web1,web2自已)

web2打开iptables端口

iptables -A INPUT -p tcp -m tcp -s 192.168.1.21 –dport 9002 -j ACCEPT /etc/init.d/iptables save

db服务器加权限 给web2加上和web1一样的用户访问权限(discuz,ucenter…)

web1安装设置

web1安装inotify https://github.com/rvoicilas/inotify-tools/wiki/ inotify-tools 3.14 is the latest version, released on the 7th of March 2010.

wget –no-check-certificate http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz tar zxvf inotify-tools-3.14.tar.gz cd inotify-tools-3.14 make make install 完成后，注意查看manpage，man inotify、man inotifywait

查看是否支持inotify 从kernel 2.6.13开始正式并入内核，RHEL5已经支持。 看看是否有 /proc/sys/fs/inotify/目录，以确定内核是否支持inotify ll /proc/sys/fs/inotify

total 0 -rw-r–r– 1 root root 0 Sep 14 14:01 max_queued_events -rw-r–r– 1 root root 0 Sep 14 14:01 max_user_instances -rw-r–r– 1 root root 0 Sep 14 14:01 max_user_watches

测试inotify /usr/local/bin/inotifywait -mrq –timefmt ‘%d/%m/%y %H:%M’ –format ‘%T %w%f’ -e modify,delete,create,attrib /opt/lampp/htdocs/bbs

16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log 16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log 16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log ….

rsync同步 方法一:使用ssh用户免登录 方法二:使用rsync认证免登录

我这里使用方法一

web2增加同步用户 useradd -g website rsyncuser passwd rsyncuser

web1创建公钥并传到web2 ssh-keygen -t rsa

Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is:

复制到web2

scp -P 22 ~/.ssh/id_rsa.pub [email protected]:.

web2设置免登录 将id_rsa.pub导入到.ssh/authorized_keys

cd /home/rsyncuser mkdir .ssh cat id_rsa.pub >> .ssh/authorized_keys chown -R rsyncuser:website .ssh rm id_rsa.pub

可以在web1测试下登录 ssh -p 22 [email protected]

同步脚本 inofity 的exclude支持posix正则,但只能写一个. 注意两边放bbs的路径要一样 可以先用rsync同步一下

vi inotify.sh

#!/bin/sh yesterday=`date +%Y%m%d` src=/opt/lampp/htdocs/bbs [email protected]:/opt/lampp/htdocs/ /usr/local/bin/inotifywait -mrq –exclude “data/(threadcache|log|template|sendmail\.lock|cache)” –timefmt ‘%d/%m/%y %H:%M’ –format ‘%T %w%f’ \ -e modify,delete,create,attrib ${src} | while read file do echo -e $(date +%Y-%m-%d_%H:%M:%S)”\r” >>rsynclog/inotify.${yesterday}.log rsync -av –delete –exclude “data/threadcache/” –exclude “data/log/” –exclude “data/template/” –exclude “data/sendmail.lock” ${src} -e “/usr/bin/ssh -p 22” ${des} >>rsynclog/inotify.${yesterday}.log done

在web2上创建不同步的目录和文件 cd /opt/lampp/htdocs/bbs mkdir data/{threadcache,log,template} chown -R rsyncuser:website data/{threadcache,log,template} chmod -R 0775 data/{threadcache,log,template} touch data/sendmail.lock chown -R rsyncuser:website data/sendmail.lock chmod -R 0775 data/sendmail.lock

设置权限并运行 chmod 700 ./inofity.sh ./inofity.sh &

加到开机运行

echo ‘cd /opt/shell && ./inofity.sh & ‘>>/etc/rc.local

cache目录的更新 如果使用memcache可以跳过此步 使用文件缓存时 /opt/lampp/htdocs/bbs/data/cache 目录有新注册会员及在线人数等cache,更新很频繁不适于放入inotify中 vi rsync.sh

#!/bin/sh #* * * * * cd /opt/shell && /bin/sh ./rsync.sh > /dev/null 2>&1 yesterday=`date +%Y%m%d` src=/opt/lampp/htdocs/bbs/data/cache [email protected]:/opt/lampp/htdocs/bbs/data/ echo -e $(date +%Y-%m-%d_%H:%M:%S)”\r” >>rsynclog/rsync.${yesterday}.log rsync -av –delete ${src} -e “/usr/bin/ssh -p 22” ${des2} >>rsynclog/rsync.${yesterday}.log

设置权限 chmod 775 ./rsync.sh

crontab -e 加入crontab中 每分钟运行

* * * * * cd /opt/shell && /bin/sh ./rsync.sh > /dev/null 2>&1

第一次同步完后,开启php /opt/php/sbin/php-fpm start

web1开启负载均衡

web1上调整nginx 先创建个test.php来测试,可以在phpinfo里看到不同的机器名或通firebug查看文件header来确定是否在web2上

echo ‘‘ > test.php

vi /opt/nginx/conf/nginx.conf

location ~ /test\.php?$ { #fastcgi_pass 127.0.0.1:9000; fastcgi_pass 192.168.1.23:9002; add_header App-Server php2; fastcgi_index index.php; include fcgi.conf; break; }

正式上线只需针对forum.php进行负载均衡,90%的负载都在这文件上(只开启论坛情况下)

upstream backend { ip_hash; #确保分流内网能访问公网,或无对外访问需求 server 192.168.1.23:9002 max_fails=3 fail_timeout=60s; server 127.0.0.1:9000; } location ~ /forum\.php?$ { #fastcgi_pass unix:/tmp/php-cgi.sock; #fastcgi_pass 127.0.0.1:9000; #fastcgi_pass 192.168.1.23:9002; #add_header App-Server php2; fastcgi_pass backend; fastcgi_index index.php; include fcgi.conf; break; }

fcgi.conf

fastcgi_param GATEWAY_INTERFACE CGI/1.1; fastcgi_param SERVER_SOFTWARE nginx; fastcgi_param QUERY_STRING $query_string; fastcgi_param REQUEST_METHOD $request_method; fastcgi_param CONTENT_TYPE $content_type; fastcgi_param CONTENT_LENGTH $content_length; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param REQUEST_URI $request_uri; fastcgi_param DOCUMENT_URI $document_uri; fastcgi_param DOCUMENT_ROOT $document_root; fastcgi_param SERVER_PROTOCOL $server_protocol; fastcgi_param REMOTE_ADDR $remote_addr; fastcgi_param REMOTE_PORT $remote_port; fastcgi_param SERVER_ADDR $server_addr; fastcgi_param SERVER_PORT $server_port; fastcgi_param SERVER_NAME $server_name; # PHP only, required if PHP was built with –enable-force-cgi-redirect fastcgi_param REDIRECT_STATUS 200;

重启nginx reload nginx /bin/kill -HUP cat /dev/shm/nginx.pid

discuz上传附件,后台固定在web1,读贴等操作会依据用户ip分布在两台机器上.

参考: http://www.infoq.com/cn/articles/inotify-linux-file-system-event-monitoring http://hi.baidu.com/tonyty163/blog/item/3c14ca2698672a0a918f9daf.html