Skip to content


inotify+rsync+nginx+fastcgi双机负载均衡discuzx2

一.概述

服务器
web1 R410 E5520*2 16G SAS300G*2 centos5.5 ip:192.168.1.21
web2(new) R410 E5606*2 16G SAS300G*2 centos5.5 ip:192.168.1.23
db R410 E5504*2 16G SAS146G*2 centos5.3

discuz论坛现有500多万贴子,近2万人在线,400万pv/日,
新增web2服务器分担论坛php处理.
尝试过在web2用nfs mount web1的程序来跑,但是访问速度太慢,无法接受
改用inotify+rsync效果不错,原来web1负载8左右,现两台各自3.5

web2安装

web2安装好php后配置php-fpm


192.168.1.23:9002 #监听地址及端口
rsyncuser#后面rsync同步的用户
website#同步用户的用户组
128#目前开128个 大概占用物理内存4G左右
192.168.1.21,192.168.1.23 #充许请求的服务器(web1,web2自已)

web2打开iptables端口

iptables -A INPUT -p tcp -m tcp -s 192.168.1.21 –dport 9002 -j ACCEPT
/etc/init.d/iptables save

db服务器加权限
给web2加上和web1一样的用户访问权限(discuz,ucenter…)

web1安装设置

web1安装inotify
https://github.com/rvoicilas/inotify-tools/wiki/
inotify-tools 3.14 is the latest version, released on the 7th of March 2010.

wget –no-check-certificate http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
tar zxvf inotify-tools-3.14.tar.gz
cd inotify-tools-3.14
make
make install
完成后,注意查看manpage,man inotify、man inotifywait

查看是否支持inotify
从kernel 2.6.13开始正式并入内核,RHEL5已经支持。
看看是否有 /proc/sys/fs/inotify/目录,以确定内核是否支持inotify
ll /proc/sys/fs/inotify

total 0
-rw-r–r– 1 root root 0 Sep 14 14:01 max_queued_events
-rw-r–r– 1 root root 0 Sep 14 14:01 max_user_instances
-rw-r–r– 1 root root 0 Sep 14 14:01 max_user_watches

测试inotify
/usr/local/bin/inotifywait -mrq –timefmt ‘%d/%m/%y %H:%M’ –format ‘%T %w%f’ -e modify,delete,create,attrib /opt/lampp/htdocs/bbs

16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
….

rsync同步
方法一:使用ssh用户免登录
方法二:使用rsync认证免登录

我这里使用方法一
web2增加同步用户
useradd -g website rsyncuser
passwd rsyncuser

web1创建公钥并传到web2
ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:

复制到web2

scp -P 22 ~/.ssh/id_rsa.pub rsyncuser@192.168.1.23:.

web2设置免登录
将id_rsa.pub导入到.ssh/authorized_keys

cd /home/rsyncuser
mkdir .ssh
cat id_rsa.pub >> .ssh/authorized_keys
chown -R rsyncuser:website .ssh
rm id_rsa.pub

可以在web1测试下登录
ssh -p 22 rsyncuser@192.168.1.23

同步脚本
inofity 的exclude支持posix正则,但只能写一个.
注意两边放bbs的路径要一样
可以先用rsync同步一下

vi inotify.sh

#!/bin/sh
yesterday=`date +%Y%m%d`
src=/opt/lampp/htdocs/bbs
des=rsyncuser@192.168.1.23:/opt/lampp/htdocs/

/usr/local/bin/inotifywait -mrq –exclude “data/(threadcache|log|template|sendmail\.lock|cache)” –timefmt ‘%d/%m/%y %H:%M’ –format ‘%T %w%f’ \
-e modify,delete,create,attrib ${src} | while read file
do
echo -e $(date +%Y-%m-%d_%H:%M:%S)”\r” >>rsynclog/inotify.${yesterday}.log
rsync -av –delete –exclude “data/threadcache/” –exclude “data/log/” –exclude “data/template/” –exclude “data/sendmail.lock” ${src} -e “/usr/bin/ssh -p 22” ${des} >>rsynclog/inotify.${yesterday}.log
done

在web2上创建不同步的目录和文件
cd /opt/lampp/htdocs/bbs
mkdir data/{threadcache,log,template}
chown -R rsyncuser:website data/{threadcache,log,template}
chmod -R 0775 data/{threadcache,log,template}
touch data/sendmail.lock
chown -R rsyncuser:website data/sendmail.lock
chmod -R 0775 data/sendmail.lock

设置权限并运行
chmod 700 ./inofity.sh
./inofity.sh &

加到开机运行

echo ‘cd /opt/shell && ./inofity.sh & ‘>>/etc/rc.local

cache目录的更新
如果使用memcache可以跳过此步
使用文件缓存时 /opt/lampp/htdocs/bbs/data/cache 目录有新注册会员及在线人数等cache,更新很频繁不适于放入inotify中
vi rsync.sh

#!/bin/sh
#* * * * * cd /opt/shell && /bin/sh ./rsync.sh > /dev/null 2>&1
yesterday=`date +%Y%m%d`
src=/opt/lampp/htdocs/bbs/data/cache
des=rsyncuser@192.168.1.23:/opt/lampp/htdocs/bbs/data/
echo -e $(date +%Y-%m-%d_%H:%M:%S)”\r” >>rsynclog/rsync.${yesterday}.log
rsync -av –delete ${src} -e “/usr/bin/ssh -p 22” ${des2} >>rsynclog/rsync.${yesterday}.log

设置权限
chmod 775 ./rsync.sh

crontab -e 加入crontab中
每分钟运行

* * * * * cd /opt/shell && /bin/sh ./rsync.sh > /dev/null 2>&1

第一次同步完后,开启php
/opt/php/sbin/php-fpm start

web1开启负载均衡

web1上调整nginx
先创建个test.php来测试,可以在phpinfo里看到不同的机器名或通firebug查看文件header来确定是否在web2上


echo ‘‘ > test.php

vi /opt/nginx/conf/nginx.conf

location ~ /test\.php?$
{
#fastcgi_pass 127.0.0.1:9000;
fastcgi_pass 192.168.1.23:9002;
add_header App-Server php2;
fastcgi_index index.php;
include fcgi.conf;
break;
}

正式上线只需针对forum.php进行负载均衡,90%的负载都在这文件上(只开启论坛情况下)

upstream backend {
ip_hash;
#确保分流内网能访问公网,或无对外访问需求
server 192.168.1.23:9002 max_fails=3 fail_timeout=60s;
server 127.0.0.1:9000;
}

location ~ /forum\.php?$
{
#fastcgi_pass unix:/tmp/php-cgi.sock;
#fastcgi_pass 127.0.0.1:9000;
#fastcgi_pass 192.168.1.23:9002;
#add_header App-Server php2;
fastcgi_pass backend;
fastcgi_index index.php;
include fcgi.conf;
break;
}

fcgi.conf

fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;

fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;

fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;

fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;

# PHP only, required if PHP was built with –enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;

重启nginx
reload nginx
/bin/kill -HUP `cat /dev/shm/nginx.pid`

discuz上传附件,后台固定在web1,读贴等操作会依据用户ip分布在两台机器上.

参考:
http://www.infoq.com/cn/articles/inotify-linux-file-system-event-monitoring
http://hi.baidu.com/tonyty163/blog/item/3c14ca2698672a0a918f9daf.html

Posted in 网站架构, Discuz/Uchome/Ucenter.

Tagged with , , , , , .


No Responses (yet)

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.



Some HTML is OK

or, reply to this post via trackback.