Skip to content

linux ulimit max open files [WARNING] fpm_stdio_child_said(), line 167: child 29588 (pool default) said into stderr

tail -n70 /opt/php/logs/php-fpm.logs

Aug 21 10:47:28.134817 [NOTICE] fpm_children_make(), line 352: child 20649 (pool default) started
Aug 21 10:47:28.135757 [NOTICE] fpm_children_make(), line 352: child 20650 (pool default) started
Aug 21 10:47:28.136035 [NOTICE] fpm_children_make(), line 352: child 20651 (pool default) started
Aug 21 10:47:28.136971 [NOTICE] fpm_children_make(), line 352: child 20652 (pool default) started
Aug 21 10:47:28.138028 [NOTICE] fpm_children_make(), line 352: child 20653 (pool default) started
Aug 21 10:47:28.138058 [NOTICE] fpm_event_loop(), line 107: libevent: entering main loop
Aug 21 10:47:28.138119 [WARNING] fpm_stdio_child_said(), line 167: child 20653 (pool default) said into stderr: “Aug 21 10:47:28.137371 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138139 [WARNING] fpm_stdio_child_said(), line 167: child 20652 (pool default) said into stderr: “Aug 21 10:47:28.136442 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138151 [WARNING] fpm_stdio_child_said(), line 167: child 20651 (pool default) said into stderr: “Aug 21 10:47:28.136243 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138161 [WARNING] fpm_stdio_child_said(), line 167: child 20650 (pool default) said into stderr: “Aug 21 10:47:28.135276 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138171 [WARNING] fpm_stdio_child_said(), line 167: child 20649 (pool default) said into stderr: “Aug 21 10:47:28.134357 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138181 [WARNING] fpm_stdio_child_said(), line 167: child 20648 (pool default) said into stderr: “Aug 21 10:47:28.133984 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138193 [WARNING] fpm_stdio_child_said(), line 167: child 20647 (pool default) said into stderr: “Aug 21 10:47:28.132846 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138203 [WARNING] fpm_stdio_child_said(), line 167: child 20646 (pool default) said into stderr: “Aug 21 10:47:28.132496 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138213 [WARNING] fpm_stdio_child_said(), line 167: child 20645 (pool default) said into stderr: “Aug 21 10:47:28.126485 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138257 [WARNING] fpm_stdio_child_said(), line 167: child 20644 (pool default) said into stderr: “Aug 21 10:47:28.123230 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138270 [WARNING] fpm_stdio_child_said(), line 167: child 20643 (pool default) said into stderr: “Aug 21 10:47:28.116077 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138282 [WARNING] fpm_stdio_child_said(), line 167: child 20642 (pool default) said into stderr: “Aug 21 10:47:28.112005 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138292 [WARNING] fpm_stdio_child_said(), line 167: child 20641 (pool default) said into stderr: “Aug 21 10:47:28.114069 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138303 [WARNING] fpm_stdio_child_said(), line 167: child 20640 (pool default) said into stderr: “Aug 21 10:47:28.109341 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138314 [WARNING] fpm_stdio_child_said(), line 167: child 20639 (pool default) said into stderr: “Aug 21 10:47:28.109003 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138325 [WARNING] fpm_stdio_child_said(), line 167: child 20638 (pool default) said into stderr: “Aug 21 10:47:28.108659 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138334 [WARNING] fpm_stdio_child_said(), line 167: child 20637 (pool default) said into stderr: “Aug 21 10:47:28.107447 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138344 [WARNING] fpm_stdio_child_said(), line 167: child 20636 (pool default) said into stderr: “Aug 21 10:47:28.105309 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138354 [WARNING] fpm_stdio_child_said(), line 167: child 20635 (pool default) said into stderr: “Aug 21 10:47:28.104641 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138364 [WARNING] fpm_stdio_child_said(), line 167: child 20634 (pool default) said into stderr: “Aug 21 10:47:28.103690 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138375 [WARNING] fpm_stdio_child_said(), line 167: child 20633 (pool default) said into stderr: “Aug 21 10:47:28.101346 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138386 [WARNING] fpm_stdio_child_said(), line 167: child 20632 (pool default) said into stderr: “Aug 21 10:47:28.100429 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138396 [WARNING] fpm_stdio_child_said(), line 167: child 20631 (pool default) said into stderr: “Aug 21 10:47:28.098602 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138407 [WARNING] fpm_stdio_child_said(), line 167: child 20630 (pool default) said into stderr: “Aug 21 10:47:28.096524 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138418 [WARNING] fpm_stdio_child_said(), line 167: child 20629 (pool default) said into stderr: “Aug 21 10:47:28.094494 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138428 [WARNING] fpm_stdio_child_said(), line 167: child 20628 (pool default) said into stderr: “Aug 21 10:47:28.093321 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138446 [WARNING] fpm_stdio_child_said(), line 167: child 20627 (pool default) said into stderr: “Aug 21 10:47:28.092317 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138457 [WARNING] fpm_stdio_child_said(), line 167: child 20626 (pool default) said into stderr: “Aug 21 10:47:28.091560 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138467 [WARNING] fpm_stdio_child_said(), line 167: child 20625 (pool default) said into stderr: “Aug 21 10:47:28.088570 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138476 [WARNING] fpm_stdio_child_said(), line 167: child 20624 (pool default) said into stderr: “Aug 21 10:47:28.084022 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138486 [WARNING] fpm_stdio_child_said(), line 167: child 20623 (pool default) said into stderr: “Aug 21 10:47:28.081269 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138495 [WARNING] fpm_stdio_child_said(), line 167: child 20622 (pool default) said into stderr: “Aug 21 10:47:28.080984 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138505 [WARNING] fpm_stdio_child_said(), line 167: child 20621 (pool default) said into stderr: “Aug 21 10:47:28.076449 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138514 [WARNING] fpm_stdio_child_said(), line 167: child 20620 (pool default) said into stderr: “Aug 21 10:47:28.077009 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138523 [WARNING] fpm_stdio_child_said(), line 167: child 20619 (pool default) said into stderr: “Aug 21 10:47:28.069281 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138533 [WARNING] fpm_stdio_child_said(), line 167: child 20618 (pool default) said into stderr: “Aug 21 10:47:28.068479 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138542 [WARNING] fpm_stdio_child_said(), line 167: child 20617 (pool default) said into stderr: “Aug 21 10:47:28.066524 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138552 [WARNING] fpm_stdio_child_said(), line 167: child 20616 (pool default) said into stderr: “Aug 21 10:47:28.065905 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138562 [WARNING] fpm_stdio_child_said(), line 167: child 20615 (pool default) said into stderr: “Aug 21 10:47:28.065567 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138571 [WARNING] fpm_stdio_child_said(), line 167: child 20614 (pool default) said into stderr: “Aug 21 10:47:28.062630 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138583 [WARNING] fpm_stdio_child_said(), line 167: child 20613 (pool default) said into stderr: “Aug 21 10:47:28.059793 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138593 [WARNING] fpm_stdio_child_said(), line 167: child 20612 (pool default) said into stderr: “Aug 21 10:47:28.056453 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138604 [WARNING] fpm_stdio_child_said(), line 167: child 20611 (pool default) said into stderr: “Aug 21 10:47:28.053536 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138614 [WARNING] fpm_stdio_child_said(), line 167: child 20610 (pool default) said into stderr: “Aug 21 10:47:28.052367 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138632 [WARNING] fpm_stdio_child_said(), line 167: child 20609 (pool default) said into stderr: “Aug 21 10:47:28.051691 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138642 [WARNING] fpm_stdio_child_said(), line 167: child 20608 (pool default) said into stderr: “Aug 21 10:47:28.051512 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138653 [WARNING] fpm_stdio_child_said(), line 167: child 20607 (pool default) said into stderr: “Aug 21 10:47:28.048297 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138664 [WARNING] fpm_stdio_child_said(), line 167: child 20606 (pool default) said into stderr: “Aug 21 10:47:28.048226 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138674 [WARNING] fpm_stdio_child_said(), line 167: child 20605 (pool default) said into stderr: “Aug 21 10:47:28.047565 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138684 [WARNING] fpm_stdio_child_said(), line 167: child 20604 (pool default) said into stderr: “Aug 21 10:47:28.044563 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138694 [WARNING] fpm_stdio_child_said(), line 167: child 20603 (pool default) said into stderr: “Aug 21 10:47:28.042441 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138704 [WARNING] fpm_stdio_child_said(), line 167: child 20602 (pool default) said into stderr: “Aug 21 10:47:28.040655 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138714 [WARNING] fpm_stdio_child_said(), line 167: child 20601 (pool default) said into stderr: “Aug 21 10:47:28.030726 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138726 [WARNING] fpm_stdio_child_said(), line 167: child 20600 (pool default) said into stderr: “Aug 21 10:47:28.024933 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138736 [WARNING] fpm_stdio_child_said(), line 167: child 20599 (pool default) said into stderr: “Aug 21 10:47:28.021018 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138746 [WARNING] fpm_stdio_child_said(), line 167: child 20598 (pool default) said into stderr: “Aug 21 10:47:28.017561 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138757 [WARNING] fpm_stdio_child_said(), line 167: child 20597 (pool default) said into stderr: “Aug 21 10:47:28.016513 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138767 [WARNING] fpm_stdio_child_said(), line 167: child 20596 (pool default) said into stderr: “Aug 21 10:47:28.012491 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138778 [WARNING] fpm_stdio_child_said(), line 167: child 20595 (pool default) said into stderr: “Aug 21 10:47:28.011405 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138789 [WARNING] fpm_stdio_child_said(), line 167: child 20594 (pool default) said into stderr: “Aug 21 10:47:28.010464 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138799 [WARNING] fpm_stdio_child_said(), line 167: child 20593 (pool default) said into stderr: “Aug 21 10:47:28.008462 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138820 [WARNING] fpm_stdio_child_said(), line 167: child 20592 (pool default) said into stderr: “Aug 21 10:47:28.007855 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138832 [WARNING] fpm_stdio_child_said(), line 167: child 20591 (pool default) said into stderr: “Aug 21 10:47:28.007511 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”
Aug 21 10:47:28.138842 [WARNING] fpm_stdio_child_said(), line 167: child 20590 (pool default) said into stderr: “Aug 21 10:47:28.004508 [ERROR] fpm_unix_init_child(), line 168: setrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”

网上搜了下说是系统最大文件数和php-fpm.conf中rlimit_files不匹配的原因

一、我的配置
centos5.2 64bit
nginx0.7.61+php5.2.10+php-5.2.10-fpm-0.5.13

/etc/rc.local

ulimit -Shn 51200

/opt/nginx/etc/nginx.conf

worker_rlimit_nofile 51200;
events
{
use epoll;

worker_connections 51200;
}

/opt/php/etc/php-fpm.conf

51200


ulimit -n

1024


cat /proc/sys/fs/file-max

765985


cat /proc/sys/fs/file-nr

3060 0 765985

二、修改
1.
/etc/security/limits.conf
文件尾部增加

* soft nofile 51200
* hard nofile 51200

设置为星号代表全局
这个当中的硬限制是实际的限制，而软限制，是warnning限制，只会做出warning。

2.
退出控制台重新登录

ulimit -n

51200

3.
php-fpm reload不会生效，所以用 php-fpm restart

4.
检查一下
tail /opt/php/logs/php-fpm.log
ok,收工

Posted in PHP, 技术.

Tagged with .

rev="post-833" No comments

By C1G – 2009/08/21

discuz7 右上角出现xml解析错误


http://bbs.c1gstudio.com/pm.php?checknewpm=0.5297697346042771&inajax=1&ajaxtarget=pm_ntc 行：1，列：1：

用户也不能登录。

解决:
发现uc_client/model 目录有被更新过，但里面的文件大小和时间没有异常.
覆盖里面的pm.php后解决问题，原因不详不知道和服务器换ip有没关系。

Posted in Discuz/Uchome/Ucenter.

Tagged with .

rev="post-831" No comments

By C1G – 2009/08/21

BUG: soft lockup – CPU#3 stuck for 10s!

故障现象
刚上的机器,网站不能访问.ssh也没反应

查看/var/log/message有一堆错误
Aug 19 21:35:23 bora kernel: BUG: soft lockup – CPU#3 stuck for 10s! [php-cgi:23997]
Aug 19 21:35:23 bora kernel: CPU 3:
Aug 19 21:35:23 bora kernel: Modules linked in: ip_conntrack_netbios_ns xt_state ip_conntrack nfnetlink iptable_filter ip_tables deflate zlib_deflate ccm serpent blowfish twofish ecb xcbc crypto_hash cbc md5 sha256 sha512 des aes_generic testmgr_cipher testmgr crypto_blkcipher aes_x86_64 ipcomp6 ipcomp ah6 ah4 esp6 xfrm6_esp esp4 xfrm4_esp aead crypto_algapi xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_tunnel xfrm6_tunnel tunnel6 ipv6 xfrm_nalgo crypto_api af_key autofs4 hidp rfcomm l2cap bluetooth sunrpc ipt_REJECT xt_limit xt_tcpudp x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom serio_raw sg bnx2 pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache ata_piix libata shpchp mptsas mptscsih mptbase sc
Aug 19 21:35:23 bora kernel: i_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 19 21:35:23 bora kernel: Pid: 23997, comm: php-cgi Not tainted 2.6.18-128.el5 #1
Aug 19 21:35:23 bora kernel: RIP: 0010:[] [] .text.lock.spinlock+0x2/0x30
Aug 19 21:35:23 bora kernel: RSP: 0000:ffff81013399bc68 EFLAGS: 00000282
Aug 19 21:35:23 bora kernel: RAX: ffff810224273c80 RBX: ffff810223ca7980 RCX: ffff810224273c80
Aug 19 21:35:23 bora kernel: RDX: 0000000000000000 RSI: 00000000000001f4 RDI: ffff810224273cc0
Aug 19 21:35:23 bora kernel: RBP: ffff81013399bbe0 R08: 0000000000000002 R09: 0000000000000000
Aug 19 21:35:23 bora kernel: R10: ffff81022b310680 R11: 0000000000000000 R12: ffffffff8005dc8e
Aug 19 21:35:23 bora kernel: R13: ffff810224273c80 R14: ffffffff800774da R15: ffff81013399bbe0
Aug 19 21:35:23 bora kernel: FS: 00002b5378ee4c20(0000) GS:ffff81012fc4e6c0(0000) knlGS:0000000000000000
Aug 19 21:35:23 bora kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 21:35:23 bora kernel: CR2: 00002b5380d78000 CR3: 0000000215b15000 CR4: 00000000000006e0
Aug 19 21:35:23 bora kernel:
Aug 19 21:35:23 bora kernel: Call Trace:
Aug 19 21:35:23 bora kernel: [] udp_rcv+0x431/0x5d1
Aug 19 21:35:23 bora kernel: [] ip_local_deliver+0x19d/0x263
Aug 19 21:35:23 bora kernel: [] ip_rcv+0x53a/0x57d
Aug 19 21:35:23 bora kernel: [] netif_receive_skb+0x370/0x39c
Aug 19 21:35:23 bora kernel: [] :bnx2:bnx2_poll_work+0xf7d/0x10b5
Aug 19 21:35:23 bora kernel: [] sk_free+0xc3/0x105
Aug 19 21:35:23 bora kernel: [] sched_balance_self+0x154/0x2f0
Aug 19 21:35:23 bora kernel: [] ip_local_deliver_finish+0x0/0x1e9
Aug 19 21:35:23 bora kernel: [] nf_hook_slow+0x58/0xbc
Aug 19 21:35:23 bora kernel: [] ip_local_deliver+0x19d/0x263
Aug 19 21:35:23 bora kernel: [] ip_rcv+0x53a/0x57d
Aug 19 21:35:23 bora kernel: [] :bnx2:bnx2_poll_msix+0x2e/0xc5
Aug 19 21:35:24 bora kernel: [] net_rx_action+0xa4/0x1a4
Aug 19 21:35:24 bora kernel: [] __do_softirq+0x89/0x133
Aug 19 21:35:24 bora kernel: [] call_softirq+0x1c/0x28
Aug 19 21:35:24 bora kernel: [] do_softirq+0x2c/0x85
Aug 19 21:35:24 bora kernel: [] do_IRQ+0xec/0xf5
Aug 19 21:35:24 bora kernel: [] ret_from_intr+0x0/0xa
Aug 19 21:35:24 bora kernel:
Aug 19 21:35:26 bora kernel: BUG: soft lockup – CPU#6 stuck for 10s! [pluto:3927]
Aug 19 21:35:26 bora kernel: CPU 6:
Aug 19 21:35:26 bora kernel: Modules linked in: ip_conntrack_netbios_ns xt_state ip_conntrack nfnetlink iptable_filter ip_tables deflate zlib_deflate ccm serpent blowfish twofish ecb xcbc crypto_hash cbc md5 sha256 sha512 des aes_generic testmgr_cipher testmgr crypto_blkcipher aes_x86_64 ipcomp6 ipcomp ah6 ah4 esp6 xfrm6_esp esp4 xfrm4_esp aead crypto_algapi xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_tunnel xfrm6_tunnel tunnel6 ipv6 xfrm_nalgo crypto_api af_key autofs4 hidp rfcomm l2cap bluetooth sunrpc ipt_REJECT xt_limit xt_tcpudp x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom serio_raw sg bnx2 pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache ata_piix libata shpchp mptsas mptscsih mptbase sc
Aug 19 21:35:26 bora kernel: i_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 19 21:35:26 bora kernel: Pid: 3927, comm: pluto Not tainted 2.6.18-128.el5 #1
Aug 19 21:35:26 bora kernel: RIP: 0010:[] [] .text.lock.spinlock+0x2/0x30
Aug 19 21:35:26 bora kernel: RSP: 0018:ffff81012e5f5bb0 EFLAGS: 00000282
Aug 19 21:35:26 bora kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff810224273c80
Aug 19 21:35:26 bora kernel: RDX: 0000000000000000 RSI: 00000000000001e0 RDI: ffff810224273cc0
Aug 19 21:35:26 bora kernel: RBP: 0000000000000206 R08: ffff81012e5f5a38 R09: 0000000000000000
Aug 19 21:35:26 bora kernel: R10: ffff81012e5f5ab8 R11: 0000000000000048 R12: ffff810224273c80
Aug 19 21:35:26 bora kernel: R13: 0000100000000011 R14: 0000000400000000 R15: 0000000000000000
Aug 19 21:35:26 bora kernel: FS: 00002b4ce020bdb0(0000) GS:ffff81013397ce40(0000) knlGS:0000000000000000
Aug 19 21:35:26 bora kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 21:35:26 bora kernel: CR2: 00002b5380d78000 CR3: 00000002269a2000 CR4: 00000000000006e0
Aug 19 21:35:26 bora kernel:
Aug 19 21:35:26 bora kernel: Call Trace:
Aug 19 21:35:26 bora kernel: [] release_sock+0x6b/0xaa
Aug 19 21:35:26 bora kernel: [] udp_sendmsg+0x4de/0x5ce
Aug 19 21:35:26 bora kernel: [] sock_sendmsg+0xf3/0x110
Aug 19 21:35:26 bora kernel: [] inode_has_perm+0x56/0x63
Aug 19 21:35:26 bora kernel: [] autoremove_wake_function+0x0/0x2e
Aug 19 21:35:26 bora kernel: [] selinux_inode_getattr+0x50/0x5e
Aug 19 21:35:26 bora kernel: [] _atomic_dec_and_lock+0x39/0x57
Aug 19 21:35:26 bora kernel: [] sys_sendto+0x11c/0x14f
Aug 19 21:35:26 bora kernel: [] tracesys+0xd5/0xe0
Aug 19 21:35:26 bora kernel:
Aug 19 21:35:33 bora kernel: BUG: soft lockup – CPU#3 stuck for 10s! [php-cgi:23997]
Aug 19 21:35:33 bora kernel: CPU 3:
Aug 19 21:35:33 bora kernel: Modules linked in: ip_conntrack_netbios_ns xt_state ip_conntrack nfnetlink iptable_filter ip_tables deflate zlib_deflate ccm serpent blowfish twofish ecb xcbc crypto_hash cbc md5 sha256 sha512 des aes_generic testmgr_cipher testmgr crypto_blkcipher aes_x86_64 ipcomp6 ipcomp ah6 ah4 esp6 xfrm6_esp esp4 xfrm4_esp aead crypto_algapi xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_tunnel xfrm6_tunnel tunnel6 ipv6 xfrm_nalgo crypto_api af_key autofs4 hidp rfcomm l2cap bluetooth sunrpc ipt_REJECT xt_limit xt_tcpudp x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom serio_raw sg bnx2 pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache ata_piix libata shpchp mptsas mptscsih mptbase sc

硬件:
dell R410
E5504(四核) x2
4G x2
SAS 15k 146g x2

系统:
CentOS 5.2 64bit
nginx-0.7.61
php-5.2.10
mysql-5.1.37
eaccelerator-0.9.5.3

#top
top – 14:03:42 up 16:11, 1 user, load average: 0.30, 0.40, 0.43
Tasks: 260 total, 1 running, 259 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.4%us, 1.0%sy, 0.0%ni, 94.8%id, 0.5%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 8168412k total, 5068160k used, 3100252k free, 510276k buffers
Swap: 4096532k total, 0k used, 4096532k free, 3251992k cached

#cat /proc/version
Linux version 2.6.18-128.el5 ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Jan 21 10:41:14 EST 2009

# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
stepping : 5
cpu MHz : 1596.000
cache size : 4096 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips : 3993.25
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual

#iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
PING icmp — anywhere anywhere icmp echo-request state NEW
ACCEPT all — anywhere anywhere
DROP all — 127.0.0.0/8 anywhere
DROP all — anywhere anywhere state INVALID
DROP tcp — anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,PSH,URG
DROP tcp — anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,PSH,ACK,URG
DROP tcp — anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,ACK,URG
DROP tcp — anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP tcp — anywhere anywhere tcp flags:SYN,RST/SYN,RST
DROP tcp — anywhere anywhere tcp flags:FIN,SYN/FIN,SYN
ACCEPT tcp — anywhere anywhere tcp dpt:smtp
ACCEPT tcp — anywhere anywhere tcp dpt:http
ACCEPT tcp — anywhere anywhere tcp dpt:mysql
ACCEPT tcp — anywhere anywhere tcp dpt:webcache
ACCEPT tcp — anywhere anywhere tcp dpt:15666
ACCEPT tcp — anywhere anywhere tcp dpt:ssh
DROP tcp — anywhere anywhere tcp flags:FIN,SYN,RST,ACK/SYN

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all — anywhere anywhere
ACCEPT all — anywhere anywhere

Chain PING (1 references)
target prot opt source destination
RETURN icmp — anywhere anywhere icmp echo-request limit: avg 1/sec burst 5
REJECT icmp — anywhere anywhere reject-with icmp-port-unreachable

Chain SYNFLOOD (0 references)
target prot opt source destination

===========================
故障原因好像是kernel-2.6.18-128有冲突，继续观察中

相关资料
http://bugs.centos.org/view.php?id=3582
https://bugzilla.redhat.com/show_bug.cgi?id=484590

Posted in LINUX, 技术.

Tagged with .

rev="post-829" No comments

By C1G – 2009/08/20

安装Tripwire检查文件完整性

当服务器遭到黑客攻击时，在多数情况下，黑客可能对系统文件等等一些重要的文件进行修改。对此，我们用Tripwire建立数据完整性监测系统。虽然 它不能抵御黑客攻击以及黑客对一些重要文件的修改，但是可以监测文件是否被修改过以及哪些文件被修改过，从而在被攻击后有的放矢的策划出解决办法。

Tripwire的原理是Tripwire被安装、配置后，将当前的系统数据状态建立成数据库，随着文件的添加、删除和修改等等变化，通过系统数据现 状与不断更新的数据库进行比较，来判定哪些文件被添加、删除和修改过。正因为初始的数据库是在Tripwire本体被安装、配置后建立的原因，我们务必应 该在服务器开放前，或者说操作系统刚被安装后用Tripwire构建数据完整性监测系统。

和tripwire差不多的还有AIDE

一.工作原理

二.下载tripwire

tripwire 在sf上的地址

wget http://sourceforge.net/projects/tripwire/files/tripwire-src/tripwire-2.4.1.2-src/tripwire-2.4.1.2-src.tar.bz2/download
tar jxvf tripwire-2.4.1.2-src.tar.bz2
cd tripwire-2.4.1.2-src

三.安装tripwire

./configure –prefix=/usr/local/tripwire
make
make install

license agreement. [do not accept] accept
Continue with installation? [y/n] y
Enter the site keyfile passphrase:c1gstudio
Verify the site keyfile passphrase:c1gstudio
Enter the local keyfile passphrase:abcdefgh
Verify the local keyfile passphrase:abcdefgh
Please enter your site passphrase: c1gstudio
Please enter your site passphrase: c1gstudio

四.设置tripwire
编辑twpol.txt来控制对哪些目录进行检查,我这里省略了很多目录
vi /usr/local/tripwire/etc/twpol.txt

#Global Configuration Files
#注释以下目录
#/etc/mail/statistics -> $(Growing) ;

#OS Boot Files and Mount Points
#注释以下目录
#/cdrom -> $(Dynamic) ;
#/floppy -> $(Dynamic) ;
#/mnt -> $(Dynamic) ;

#OS Devices and Misc Directories
#禁止检查以下目录
#/opt -> $(Dynamic) ;
#/lost+found -> $(Dynamic) ;
#/var/lost+found -> $(Dynamic) ;
#/home/lost+found -> $(Dynamic) ;

#OS Binaries and Libraries
#禁止检查以下目录
#/lib -> $(ReadOnly) ;
#/usr/lib -> $(ReadOnly) ;
#/usr/libexec -> $(ReadOnly) ;
#/usr/X11R6/lib -> $(ReadOnly) ;

#User Binaries and Libraries
#只保留以下三个
/usr/local/bin -> $(ReadOnly) ;
/usr/local/etc -> $(ReadOnly) ;
/usr/local/sbin -> $(ReadOnly) ;

#Temporary Directories
#禁止全部目录
#/usr/tmp -> $(Temporary) ;
#/var/tmp -> $(Temporary) ;
#/tmp -> $(Temporary) ;

#Monitor Filesystems
#禁止全部目录
#/ -> $(ReadOnly) ;
#/home -> $(ReadOnly) ; # Modify as needed
#/usr -> $(ReadOnly) ;
#/var -> $(ReadOnly) ;

五.初始化数据库
/usr/local/tripwire/sbin/tripwire –init

六.更新数据库
当你更新了twpol.txt后需用此命令更新数据库
cd /usr/local/tripwire
./sbin/tripwire –update-policy –secure-mode low ./etc/twpol.txt

Please enter your local passphrase: abcdefgh
Please enter your site passphrase: c1gstudio
======== Policy Update: Processing section Unix File System.
======== Step 1: Gathering information for the new policy.
The object: “/etc/rhgb/temp” is on a different file system…ignoring.
### Warning: Policy Update Changed Object.
### An object has been changed since the database was last updated.
### Object name: Conflicting properties for object
### /usr/local/tripwire/etc/tw.pol
### > Modify Time
### > CRC32
### > MD5
### Continuing…
### Warning: Policy Update Changed Object.
### An object has been changed since the database was last updated.
### Object name: Conflicting properties for object /etc/cups/certs
### > Modify Time
### > Change Time
### Continuing…
### Warning: Policy Update Changed Object.
### An object has been changed since the database was last updated.
### Object name: Conflicting properties for object /etc/cups/certs/0
### > Modify Time
### > Change Time
### > CRC32
### > MD5
### Continuing…
======== Step 2: Updating the database with new objects.
======== Step 3: Pruning unneeded objects from the database.
Wrote policy file: /usr/local/tripwire/etc/tw.pol
Wrote database file: /usr/local/tripwire/lib/tripwire/local.c1gstudio.com.twd

七.检查文件异动
安装完tripwire后你可以定期检查文件是否存在异动
加上interactive在当前显示结果
./sbin/tripwire –check –interactive

Parsing policy file: /usr/local/tripwire/etc/tw.pol
*** Processing Unix File System ***
Performing integrity check…
The object: “/etc/rhgb/temp” is on a different file system…ignoring.
Wrote report file: /usr/local/tripwire/lib/tripwire/report/local.c1gstudio.com-20090807-112337.twr

Open Source Tripwire(R) 2.4.1 Integrity Check Report

Report generated by: root
Report created on: 2009年08月07日 星期五 11时23分37秒
Database last updated on: 2009年08月07日 星期五 11时09分27秒

===============================================================================
Report Summary:
===============================================================================

Host name: local.c1gstudio.com
Host IP address: 127.0.0.1
Host ID: None
Policy file used: /usr/local/tripwire/etc/tw.pol
Configuration file used: /usr/local/tripwire/etc/tw.cfg
Database file used: /usr/local/tripwire/lib/tripwire/local.c1gstudio.com.twd
Command line used: ./sbin/tripwire –check –interactive

===============================================================================
Rule Summary:
===============================================================================

——————————————————————————-
Section: Unix File System
——————————————————————————-

Rule Name Severity Level Added Removed Modified
——— ————– —– ——- ——–
* Tripwire Data Files 0 0 0 1
Tripwire Binaries 0 0 0 0
User Binaries and Libraries 0 0 0 0
OS Binaries and Libraries 0 0 0 0
* Global Configuration Files 0 0 0 2
System Boot Changes 0 0 0 0 RPM Checksum Files 0 0 0 0 OS Boot Files and Mount Points 0 0 0 0
(/boot) OS Devices and Misc Directories 0 0 0 0 Root Directory and Files 0 0 0 0
Total objects scanned: 64406
Total violations found: 3
===============================================================================
Object Summary:===============================================================================
——————————————————————————-
# Section: Unix File System——————————————————————————-
——————————————————————————-
Rule Name: Tripwire Data Files (/usr/local/tripwire/etc/tw.pol)
Severity Level: 0——————————————————————————-

Remove the “x” from the adjacent box to prevent updating the database
with the new values for this object.

Modified:
[x] “/usr/local/tripwire/etc/tw.pol”
——————————————————————————-Rule Name: Global Configuration Files (/etc)
Severity Level: 0——————————————————————————-

Remove the “x” from the adjacent box to prevent updating the databasewith the new values for this object.

Modified:[x] “/etc/cups/certs”
[x] “/etc/cups/certs/0”
===============================================================================
Object Detail:===============================================================================
——————————————————————————-
Section: Unix File System——————————————————————————-
——————————————————————————-Rule Name: Tripwire Data Files (/usr/local/tripwire/etc/tw.pol)
Severity Level: 0——————————————————————————-
—————————————-
Modified Objects: 1
—————————————-

Modified object name: /usr/local/tripwire/etc/tw.pol
Property: Expected Observed ————- ———– ———– Object Type Regular File Regular File Device Number 64768 64768 Mode -rw-r—– -rw-r—– Num Links 1 1 UID root (0) root (0) GID root (0) root (0) Size 4159 4159
* Modify Time 2009年08月07日 星期五 11时05分06秒 2009年08月07日 星期五 11时16分18秒 Blocks 24 24 * CRC32 BbMp+k CasvDM * MD5 AedDw/7U0K3jGZeAQ+TluE BqtFj3lGlb5i44+KkjyB9u

——————————————————————————-Rule Name: Global Configuration Files (/etc)
Severity Level: 0——————————————————————————- —————————————-
Modified Objects: 2 —————————————-
Modified object name: /etc/cups/certs
Property: Expected Observed ————- ———– ———– Object Type Directory Directory Device Number 64768 64768 File Device Number 0 0 Inode Number 1557621 1557621 Mode drwx–x–x drwx–x–x Num Links 2 2 UID root (0) root (0) GID sys (3) sys (3) Size 4096 4096
* Modify Time 2009年08月07日 星期五 11时07分09秒 2009年08月07日 星期五 11时22分12秒
* Change Time 2009年08月07日 星期五 11时07分09秒 2009年08月07日 星期五 11时22分12秒 Blocks 16 16

Modified object name: /etc/cups/certs/0
Property: Expected Observed ————- ———– ———– Object Type Regular File Regular File Device Number 64768 64768 File Device Number 0 0 Inode Number 1556488 1556488 Mode -r–r—– -r–r—– Num Links 1 1 UID root (0) root (0) GID sys (3) sys (3) Size 32 32
* Modify Time 2009年08月07日 星期五 11时07分09秒 2009年08月07日 星期五 11时22分12秒
* Change Time 2009年08月07日 星期五 11时07分09秒 2009年08月07日 星期五 11时22分12秒 Blocks 16 16 * CRC32 Bh604c DClI5t * MD5 CYQG5hqBS+c69bcyXaK6Wl DDovWtxN44ScT7sn/IJiZa

===============================================================================
Error Report:===============================================================================

No Errors
——————————————————————————-*** End of report ***
Open Source Tripwire 2.4 Portions copyright 2000 Tripwire, Inc. Tripwire is a registeredtrademark of Tripwire, Inc. This software comes with ABSOLUTELY NO WARRANTY;for details use –version. This is free software which may be redistributed
or modified only under certain conditions; see COPYING for details.All rights reserved.

八.查看报告
所有tripwire的报告以.twr后缀保存在lib/tripwire目录下，需要使用twprint命令来转化成文本格式
./sbin/twprint –print-report –twrfile ./lib/tripwire/report/local.c1gstudio.com-20090807-112337.twr >/tmp/tripwire_readable.txt
cat /tmp/tripwire_readable.txt

九.定期检查
每天4点定期检查

00 4 * * * /usr/local/tripwire/sbin/tripwire –check

十.查看当前配置
./sbin/twadmin –print-polfile

@@section GLOBAL
TWDOCS=”/usr/local/tripwire/doc/tripwire”;
TWBIN=”/usr/local/tripwire/sbin”;
TWPOL=”/usr/local/tripwire/etc”;
TWDB=”/usr/local/tripwire/lib/tripwire”;
TWSKEY=”/usr/local/tripwire/etc”;
TWLKEY=”/usr/local/tripwire/etc”;
TWREPORT=”/usr/local/tripwire/lib/tripwire/report”;
HOSTNAME=local.c1gstudio.com;

./sbin/twadmin –print-cfgfile

ROOT =/usr/local/tripwire/sbin
POLFILE =/usr/local/tripwire/etc/tw.pol
DBFILE =/usr/local/tripwire/lib/tripwire/$(HOSTNAME).twd
REPORTFILE =/usr/local/tripwire/lib/tripwire/report/$(HOSTNAME)-$(DATE).twr
SITEKEYFILE =/usr/local/tripwire/etc/site.key
LOCALKEYFILE =/usr/local/tripwire/etc/local.c1gstudio.com-local.key
EDITOR =/bin/vi
LATEPROMPTING =false
LOOSEDIRECTORYCHECKING =false
MAILNOVIOLATIONS =true
EMAILREPORTLEVEL =3
REPORTLEVEL =3
MAILMETHOD =SENDMAIL
SYSLOGREPORTING =false
MAILPROGRAM =/usr/sbin/sendmail -oi -t

参考:
Tripwire Tutorial: Linux Host Based Intrusion Detection System
Tripwire-2.4.1.2 tutorial

Posted in 安全, 技术.

Tagged with .

rev="post-809" 1 comment

By C1G – 2009/08/07

当前几个主要的Lucene中文分词器的比较[转]

 

1. 基本介绍：

paoding ：Lucene中文分词“庖丁解牛” Paoding Analysis
imdict ：imdict智能词典所采用的智能中文分词程序
mmseg4j ： 用 Chih-Hao Tsai 的 MMSeg 算法 实现的中文分词器
ik ：采用了特有的“正向迭代最细粒度切分算法“，多子处理器分析模式

2. 开发者及开发活跃度：

paoding ：qieqie.wang， google code 上最后一次代码提交：2008-06-12，svn 版本号 132
imdict ：XiaoPingGao， 进入了 lucene contribute，lucene trunk 中 contrib/analyzers/smartcn/ 最后一次提交：2009-07-24，
mmseg4j ：chenlb2008，google code 中 2009-08-03 （昨天），版本号 57，log为：mmseg4j-1.7 创建分支
ik ：linliangyi2005，google code 中 2009-07-31，版本号 41

3. 用户自定义词库：

paoding ：支持不限制个数的用户自定义词库，纯文本格式，一行一词，使用后台线程检测词库的更新，自动编译更新过的词库到二进制版本，并加载
imdict ：暂时不支持用户自定义词库。但 原版 ICTCLAS 支持。支持用户自定义 stop words
mmseg4j ：自带sogou词库，支持名为 wordsxxx.dic， utf8文本格式的用户自定义词库，一行一词。不支持自动检测。 -Dmmseg.dic.path
ik ： 支持api级的用户词库加载，和配置级的词库文件指定，无 BOM 的 UTF-8 编码，
\n 分割。不支持自动检测。

4. 速度（基于官方介绍，非自己测试）

paoding ：在PIII 1G内存个人机器上，1秒 可准确分词 100万 汉字
imdict ：483.64 (字节/秒)，259517(汉字/秒)
mmseg4j ： complex 1200kb/s左右, simple 1900kb/s左右
ik ：具有50万字/秒的高速处理能力

5. 算法和代码复杂度

paoding ：svn src 目录一共1.3M，6个properties文件，48个java文件，6895 行。使用不用的 Knife 切不同类型的流，不算很复杂。
imdict ：词库 6.7M（这个词库是必须的），src 目录 152k，20个java文件，2399行。使用 ICTCLAS HHMM隐马尔科夫模型，“利用大量语料库的训练来统计汉语词汇的词频和跳转概率，从而根据这些统计结果对整个汉语句子计算最似然(likelihood)的切分”
mmseg4j ： svn src 目录一共 132k，23个java文件，2089行。MMSeg 算法 ，有点复杂。
ik ： svn src 目录一共6.6M(词典文件也在里面)，22个java文件，4217行。多子处理器分析，跟paoding类似，歧义分析算法还没有弄明白。

6. 文档

paoding ：几乎无。代码里有一些注释，但因为实现比较复杂，读代码还是有一些难度的。
imdict ： 几乎无。 ICTCLAS 也没有详细的文档，HHMM隐马尔科夫模型的数学性太强，不太好理解。
mmseg4j ： MMSeg 算法 是英文的，但原理比较简单。实现也比较清晰。
ik ： 有一个pdf使用手册，里面有使用示例和配置说明。

7. 其它

paoding ：引入隐喻，设计比较合理。search 1.0 版本就用的这个。主要优势在于原生支持词库更新检测。主要劣势为作者已经不更新甚至不维护了。
imdict ：进入了 lucene trunk，原版 ictclas 在各种评测中都有不错的表现，有坚实的理论基础，不是个人山寨。缺点为暂时不支持用户词库。
mmseg4j ： 在complex基础上实现了最多分词(max-word)，但是还不成熟，还有很多需要改进的地方。
ik ：  针对Lucene全文检索优化的查询分析器IKQueryParser

8. 结论

个人觉得，可以在 mmseg4j 和 paoding 中选一个。关于这两个分词效果的对比，可以参考：

http://blog.chenlb.com/2009/04/mmseg4j-max-word-segment-compare-with-paoding-in-effect.html

或者自己再包装一下，将 paoding 的词库更新检测做一个单独的模块实现，然后就可以在所有基于词库的分词算法之间无缝切换了。

ps，对不同的 field 使用不同的分词器是一个可以考虑的方法。比如 tag 字段，就应该使用一个最简单的分词器，按空格分词就可以了。

原贴:http://blog.fulin.org/2009/08/lucene_chinese_analyzer_compare.html

——————————–
目前使用的是paoding

Posted in Lucene, 技术.

Tagged with , .

rev="post-806" No comments

By C1G – 2009/08/06

使用 LambdaProbe监控Tomcat

Lambda Probe
　Lambda Probe 是基于 Web + AJAX 的强大的免费开源工具,拥有几乎所有Tomcat Manager的功能，可以说是一个增强版本的 Tomcat Manager。除此之外，Tomcat Probe 还拥有很多让开发者和系统管理者更方便的性能。从而使得Tomcat对开发者和管理者更加透明。包括应用程序、数据源、发布、日志、线程、集群、系统信 息、状态、连接器状态这些功能。如配合 JDK 1.5 甚至可以实时的画出 Server 的详细内存占用状态。

　　Lambda Probe 的官方地址：http://www.lambdaprobe.org，在此可以下载Lambda Probe的最新版本。

安装Probe
下载 ZIP 文件(LambdaProbe 1.7b, BINARIES ~7Mb), 解压后只需要发布probe.war文件到 Tomcat 服务器的 webapps 目录下. 支持的Tomcat 服务器版本: 5.0, 5.5, 6.0. 还有一个地方需要设置, 在 conf/tomcat-users.xml 中添加 manager 账户, probe 需要这个账户才能正确登录使用. 如果不知道怎么做, 你可以把文件内容改成这样即可拥有一个用户名和密码都为 c1g 的管理员账户:







充许查看内存使用情况
编辑tomcat/bin/catalina.sh，在-Xmx1200m 后增加

-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=29001 -Dcom.sun.management.jmxremote.ssl=false

如下

JAVA_OPTS=’-Xms1000m -Xmx1200m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=29001 -Dcom.sun.management.jmxremote.ssl=false’

官方截图
界面本人很喜欢哈

null

在线demo
用户:demo
口令:demo
http://demo.lambdaprobe.org/

Posted in Tomcat, 技术.

Tagged with , , .

rev="post-800" No comments

By C1G – 2009/08/06

优化 tomcat session和thread

Server Information
Tomcat Version :Apache Tomcat/6.0.16
JVM Version :1.6.0_10-beta-b24
JVM Vendor :Sun Microsystems Inc.
OS Name :Linux
OS Version :2.6.9-5.ELsmp
OS Architecture :i386

问题
应用为lucene全文搜索;
当php的缓存失效时会带来大量不经过squid的请求，导致半天搜索结果不出来，php占用大量cpu，系统一直满负载

解决方法
1.增加php缓存时间同时加上随机时间让请求分散
2.增加tomcat的并发能力

首先打开tomcat的管理功能方便查看状态
编辑conf/tomcat-users.xml增加用户






重启tomcat
请求http://localhost:8080/manager/html
可以看到当前各应用的session数，内存占用，线程数…

优化1:减少session生命周期
编缉conf/web.xml将默认的30分钟改成5分钟


5


也可以编辑各应用下的web.xml针对应用进行调整

优化2:增加线程数
编辑conf/server.xml,修改默认40到100


优化3:增加内存
优化tomcat 内存

优化后server status
JVM
Free memory: 589.87 MB Total memory: 1162.43 MB Max memory: 1162.43 MB
http-8080

Max threads: 100 Current thread count: 26 Current thread busy: 6
Max processing time: 7545 ms Processing time: 4180.072 s Request count: 14471 Error count: 142 Bytes received: 0.00 MB Bytes sent: 297.62 MB

Posted in Tomcat, 技术.

Tagged with , , .

rev="post-796" No comments

By C1G – 2009/08/04

linux 下安装 subversion(svn) 客户端

svn server 为只支持http://协议的windows;
test web server 为as4,现需安装svn客户端方便同步代码

网上找了下都是讲如何安装svn server的，我只需要一个支持http协议的客户端哈，不想装apache。

安装所需软件
apr,apr-util,sqlite,neon,subversion

1.下载软件

wget http://labs.xiaonei.com/apache-mirror/apr/apr-1.3.7.tar.gz
wget http://labs.xiaonei.com/apache-mirror/apr/apr-util-1.3.8.tar.gz
wget http://www.sqlite.org/sqlite-amalgamation-3.6.16.tar.gz
wget http://www.webdav.org/neon/neon-0.28.4.tar.gz
wget http://subversion.tigris.org/downloads/subversion-1.6.3.tar.bz2

2.安装apr

tar zxvf apr-1.3.7.tar.gz
cd apr-1.3.7
./configure -prefix=/usr/local/apr
make
make install
cat /etc/ld.so.conf
echo /usr/local/apr/lib >> /etc/ld.so.conf

3.安装apr-util

tar zxvf apr-util-1.3.8.tar.gz
cd apr-util-.1.3.8
./configure –prefix=/usr/local/apr-util –with-apr=/usr/local/apr/
make
make install
echo /usr/local/apr-util/lib >> /etc/ld.so.conf
ldconfig -v

4.安装sqlite

tar zxvf sqlite-amalgamation-3.6.16.tar.gz
cd sqlite-3.6.16/
configure –prefix=/usr/local/sqlite
make
make install

5.安装neon
不需要支持http协议可以略掉安装

tar zxvf neon-0.28.4.tar.gz
cd neon-0.28.4
./configure –prefix=/usr/local/neon –enable-shared
make
make install

方式二:解压后重命名为neon,移动至subversion编译目录
但subversion编译时好像找不到neon
报错如下

configure: checking neon library

An appropriate version of neon could not be found, so libsvn_ra_neon
will not be built. If you want to build libsvn_ra_neon, please either
install neon 0.28.4 on this system

or

get neon 0.28.4 from:
http://www.webdav.org/neon/neon-0.28.4.tar.gz
unpack the archive using tar/gunzip and rename the resulting
directory from ./neon-0.28.4/ to ./neon/

no suitable neon found

6.安装subversion

tar -jxvf subversion-1.6.3.tar.bz2
cd subversion-1.6.3
./configure –prefix=/usr/local/svn –with-apr=/usr/local/apr –with-apr-util=/usr/local/apr-util –with-sqlite=/usr/local/sqlite –with-neon=/usr/local/neon
make
make install

7.检查测试
安装后应该有三个模块

/usr/local/svn/bin/svn –version
svn，版本 1.6.3 (r38063)
编译于 Jul 30 2009，14:31:41

版权所有 (C) 2000-2009 CollabNet。
Subversion 是开放源代码软件，请参阅 http://subversion.tigris.org/ 站点。
此产品包含由 CollabNet(http://www.Collab.Net/) 开发的软件。

可使用以下的版本库访问模块:

* ra_neon : 通过 WebDAV 协议使用 neon 访问版本库的模块。
– 处理“http”方案
* ra_svn : 使用 svn 网络协议访问版本库的模块。 – 使用 Cyrus SASL 认证
– 处理“svn”方案
* ra_local : 访问本地磁盘的版本库模块。
– 处理“file”方案

导出项目

cd /opt/srv/
/usr/local/svn/bin/svn export –username c1g –password 123456 http://192.168.1.9/pub37

参考：
http://www.9say.com/2009/04/subversion-compile-with-ra-dav/

Posted in Subversion.

Tagged with , , .

rev="post-793" No comments

By C1G – 2009/07/30

redhat 系统内存为8G时，SWAP如何划分？

目前Red Hat推荐交换分区的大小应当与系统物理内存的大小保持线性比例关系。不过在小于2GB物理内存的系统中，交换分区大小应该设置为内存大小的两倍，如果内存大小多于2GB，交换分区大小应该是物理内存大小加上2GB。其原因在于，系统中的物理内存越大， 对于内存的负荷可能也越大。

但是，如果物理内存大小扩展到数百GB，这样做就没什么意义了。

实际上，系统中交换分区的大小并不取决于物理内存的量，而是取决于系统中内存的负荷。Red Hat Enterprise Linux 5可以在这样的情况下工作：完全没有交换分区，而且系统中匿名内存页和共享内存页小于3/4的物理内存量。在这种情况下，系统会将匿名内存页和共享内存页锁定在物理内存中，而使用剩余的物理内存来缓冲文件系统数据(pagecache)，当内存耗尽时， 系统内核只会回收利用这些pagecache内存。

考虑到以下情况：

1）安装系统时难以确定内存的负荷，如何设置交换分区大小

2）系统中物理内存越大，所需交换分区就会越少

因此，在Red Hat Enterprise Linux 5中，以下是设置合适的交换分区大小的规则：

小于等于4G物理内存的系统，至少设置2GB的交换分区

4G～16G物理内存的系统，至少设置4GB的交换分区

16G～64G物理内存的系统，至少设置8GB的交换分区

16G～256G物理内存的系统，至少设置16GB的交换分区

参考:
http://kbase.redhat.com/faq/docs/DOC-17162

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Installation_Guide/s2-diskpartrecommend-x86.html

Posted in LINUX, 技术.

Tagged with , .

rev="post-791" No comments

By C1G – 2009/07/27

解决dreamhost上传速度缓慢问题

dreamhost的下载速度还行，但是上传实在太慢了，
今天要传些文件到服务器，速度据然只有2～4K。。。
传了半天，中间又断了次。。。哭死

解决方法是先把文件上传到国外的中传服务器再wget回来
FileUrls提供上传一个文件，设置到期的期限（最长7天），给你一个下载链接，如果你需要的话可以为这个文件加上密码。

具体方法：
1.先把本地文件打包成xxx.zip
2.把xxx.zip上传至http://fileurls.com 上传大概有几十K
3.得到一个可以直接下载的url,在dh上wget http://fileurls.com/download.ashx?id=kprgdd

HTTP request sent, awaiting response… 200 OK
Length: 5,306,368 [application/octet-stream]

100%[==========================>] 5,306,368 227.49K/s ETA 00:00

02:54:39 (287.06 KB/s) – `download.ashx?id=kprgdd’ saved [5306368/5306368]

4.unzip xxx.zip
完成

Posted in Tools, 其它.

Tagged with , .

rev="post-787" No comments

By C1G – 2009/07/11