一.概述
服务器
web1 R410 E55202 16G SAS300G2 centos5.5 ip:192.168.1.21
web2(new) R410 E56062 16G SAS300G2 centos5.5 ip:192.168.1.23
db R410 E55042 16G SAS146G2 centos5.3
discuz论坛现有500多万贴子,近2万人在线,400万pv/日,
新增web2服务器分担论坛php处理.
尝试过在web2用nfs mount web1的程序来跑,但是访问速度太慢,无法接受
改用inotify+rsync效果不错,原来web1负载8左右,现两台各自3.5
web2安装
web2安装好php后配置php-fpm
192.168.1.23:9002 #监听地址及端口
rsyncuser#后面rsync同步的用户
website#同步用户的用户组
128#目前开128个 大概占用物理内存4G左右
192.168.1.21,192.168.1.23 #充许请求的服务器(web1,web2自已)
web2打开iptables端口
iptables -A INPUT -p tcp -m tcp -s 192.168.1.21 –dport 9002 -j ACCEPT
/etc/init.d/iptables save
db服务器加权限
给web2加上和web1一样的用户访问权限(discuz,ucenter…)
web1安装设置
web1安装inotify
https://github.com/rvoicilas/inotify-tools/wiki/
inotify-tools 3.14 is the latest version, released on the 7th of March 2010.
wget –no-check-certificate http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
tar zxvf inotify-tools-3.14.tar.gz
cd inotify-tools-3.14
make
make install
完成后，注意查看manpage，man inotify、man inotifywait
查看是否支持inotify
从kernel 2.6.13开始正式并入内核，RHEL5已经支持。
看看是否有 /proc/sys/fs/inotify/目录，以确定内核是否支持inotify
ll /proc/sys/fs/inotify
total 0
-rw-r–r– 1 root root 0 Sep 14 14:01 max_queued_events
-rw-r–r– 1 root root 0 Sep 14 14:01 max_user_instances
-rw-r–r– 1 root root 0 Sep 14 14:01 max_user_watches
测试inotify
/usr/local/bin/inotifywait -mrq –timefmt ‘%d/%m/%y %H:%M’ –format ‘%T %w%f’ -e modify,delete,create,attrib /opt/lampp/htdocs/bbs
16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
16/09/11 15:59 /opt/lampp/htdocs/bbs/data/cache/forum_threadviews_1.log
….
rsync同步
方法一:使用ssh用户免登录
方法二:使用rsync认证免登录
我这里使用方法一
web2增加同步用户
useradd -g website rsyncuser
passwd rsyncuser
web1创建公钥并传到web2
ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
复制到web2
scp -P 22 ~/.ssh/id_rsa.pub
[email protected]:.
web2设置免登录
将id_rsa.pub导入到.ssh/authorized_keys
cd /home/rsyncuser
mkdir .ssh
cat id_rsa.pub >> .ssh/authorized_keys
chown -R rsyncuser:website .ssh
rm id_rsa.pub
可以在web1测试下登录
ssh -p 22 [email protected]
同步脚本
inofity 的exclude支持posix正则,但只能写一个.
注意两边放bbs的路径要一样
可以先用rsync同步一下
vi inotify.sh
#!/bin/sh
yesterday=`date +%Y%m%d`
src=/opt/lampp/htdocs/bbs
[email protected]:/opt/lampp/htdocs/
/usr/local/bin/inotifywait -mrq –exclude “data/(threadcache|log|template|sendmail\.lock|cache)” –timefmt ‘%d/%m/%y %H:%M’ –format ‘%T %w%f’ \
-e modify,delete,create,attrib ${src} | while read file
do
echo -e $(date +%Y-%m-%d_%H:%M:%S)”\r” >>rsynclog/inotify.${yesterday}.log
rsync -av –delete –exclude “data/threadcache/” –exclude “data/log/” –exclude “data/template/” –exclude “data/sendmail.lock” ${src} -e “/usr/bin/ssh -p 22” ${des} >>rsynclog/inotify.${yesterday}.log
done
在web2上创建不同步的目录和文件
cd /opt/lampp/htdocs/bbs
mkdir data/{threadcache,log,template}
chown -R rsyncuser:website data/{threadcache,log,template}
chmod -R 0775 data/{threadcache,log,template}
touch data/sendmail.lock
chown -R rsyncuser:website data/sendmail.lock
chmod -R 0775 data/sendmail.lock
设置权限并运行
chmod 700 ./inofity.sh
./inofity.sh &
加到开机运行
echo ‘cd /opt/shell && ./inofity.sh & ‘>>/etc/rc.local
cache目录的更新
如果使用memcache可以跳过此步
使用文件缓存时 /opt/lampp/htdocs/bbs/data/cache 目录有新注册会员及在线人数等cache,更新很频繁不适于放入inotify中
vi rsync.sh
#!/bin/sh
#* * * * * cd /opt/shell && /bin/sh ./rsync.sh > /dev/null 2>&1
yesterday=`date +%Y%m%d`
src=/opt/lampp/htdocs/bbs/data/cache
[email protected]:/opt/lampp/htdocs/bbs/data/
echo -e $(date +%Y-%m-%d_%H:%M:%S)”\r” >>rsynclog/rsync.${yesterday}.log
rsync -av –delete ${src} -e “/usr/bin/ssh -p 22” ${des2} >>rsynclog/rsync.${yesterday}.log
设置权限
chmod 775 ./rsync.sh
crontab -e 加入crontab中
每分钟运行
* * * * * cd /opt/shell && /bin/sh ./rsync.sh > /dev/null 2>&1
第一次同步完后,开启php
/opt/php/sbin/php-fpm start
web1开启负载均衡
web1上调整nginx
先创建个test.php来测试,可以在phpinfo里看到不同的机器名或通firebug查看文件header来确定是否在web2上
echo ” > test.php
vi /opt/nginx/conf/nginx.conf
location ~ /test\.php?$
{
#fastcgi_pass 127.0.0.1:9000;
fastcgi_pass 192.168.1.23:9002;
add_header App-Server php2;
fastcgi_index index.php;
include fcgi.conf;
break;
}
正式上线只需针对forum.php进行负载均衡,90%的负载都在这文件上(只开启论坛情况下)
upstream backend {
ip_hash;
#确保分流内网能访问公网,或无对外访问需求
server 192.168.1.23:9002 max_fails=3 fail_timeout=60s;
server 127.0.0.1:9000;
}
location ~ /forum\.php?$
{
#fastcgi_pass unix:/tmp/php-cgi.sock;
#fastcgi_pass 127.0.0.1:9000;
#fastcgi_pass 192.168.1.23:9002;
#add_header App-Server php2;
fastcgi_pass backend;
fastcgi_index index.php;
include fcgi.conf;
break;
}
fcgi.conf
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
# PHP only, required if PHP was built with –enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;
重启nginx
reload nginx
/bin/kill -HUP cat /dev/shm/nginx.pid
discuz上传附件,后台固定在web1,读贴等操作会依据用户ip分布在两台机器上.
参考:
http://www.infoq.com/cn/articles/inotify-linux-file-system-event-monitoring
http://hi.baidu.com/tonyty163/blog/item/3c14ca2698672a0a918f9daf.html
近期评论