Analog是一款基于C语言功能强大的开源的网站访问日志分析软件,支持多语言(含中文),可以运行在linux,windows下,支持apache、ngix、iis等主流WEB日志.速度飞快，10分钟内可以处理2千万条日志,数据统计以PV为主,相比Awstats和Webalizer 的报告页面简单了点，更漂亮的图表可用Report Magic 2.21.

目前最新版为analog-6.0,作者自19-Dec-04后就没更新过.演示地址
安装很简单,到:http://www.analog.cx/download.html 下载相应的版本,这里以源码版为例:将下载回来的源码包解压到安装目录,再进入该目录执行make命令即可.

wget http://www.analog.cx/analog-6.0.tar.gz
tar zxvf analog-6.0.tar.gz
cp -ar analog-6.0 /usr/local/
cd /usr/local/analog-6.0
make
ln -s analog-6.0 analog
mkdir /opt/htdocs/www/analog
chown www:website /opt/htdocs/www/analog
cp images /opt/htdocs/www/analog/
mkdir conf
cp analog.cfg conf/c1g.cfg

配置

vi conf/c1g.cfg

#定义为中文
LANGUAGE SIMP-CHINESE
#nginx日志格式
LOGFORMAT (%s – %j [%d/%M/%Y:%h:%n:%j %j] “%j %r %j” %c %b “%f” “%B”\n)
#日志文件
LOGFILE /opt/log/Y.%M/*/*c1gstudio.com.log.gz
#输出文件
OUTFILE /opt/htdocs/www/analog/c1gstudiolY.%M/index.html
#主机名
HOSTNAME “c1gstudio.com”
#主机URL
HOSTURL http://www.c1gstudio.com/
#web图片目录
IMAGEDIR ../images/
#只列出访问最高的200个页面URL
REQFLOOR 1000p
#forum.php文件算一个文件
FILEALIAS /forum.php* /forum.php
#统计子目录
SUBDIR */*

LOGFORMAT 说明

%S
host (the client hostname, or address of the computer making the request)
%s
numerical IP address of client (if recorded in a separate field; used when %S is empty)
%r
file requested
%q
query string (part of filename after ?, if recorded in a separate field)
%B
browser
%A
browser with +’s instead of spaces
%f
referrer
%u
user (tip: a cookie or session id can usefully be defined as %u too)
%v
virtual host (the server hostname, also called the virtual domain)
%d
day of the month
%m
month in digits
%M
month, three letter English abbreviation
%y
year, last two digits
%Y
year, four digits
%Z
year, two or four digits (less efficient)
%h
hour of the day
%n
minute of the hour
%a
a or A for am, or p or P for pm, if %h is in the 12-hour clock. (So to match “am” you need %am and to match “AM” you need %aM)
%U
“Unix time” (seconds since beginning of 1970, GMT). If it includes decimals, use %U.%j
%b
number of bytes transferred
%t
processing time in seconds
%T
processing time in milliseconds
%D
processing time in microseconds
%c
HTTP status code
%C
code words used instead of HTTP status code in some servers — only used internally
%j
junk: ignore this field (field can be empty too)
%w
white space: spaces or tabs
%W
optional white space
%%
% sign
\n
new line
\t
tab stop
\\
single backslash

我的nginx日志格式

‘$remote_addr – $remote_user [$time_local] “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
‘”$http_user_agent” $http_x_forwarded_for’;


183.62.5.13 – – [06/Aug/2014:17:16:44 +0800] “GET /aboutc1g.html HTTP/1.1” 200 6642 “http://www.c1gstudio.com/web/hello.html” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36” 183.62.5.13

我这多了个$http_x_forwarded_for’,后面也要加个%j表示丢弃,它不会处理”-”

LOGFORMAT (%s – %j [%d/%M/%Y:%h:%n:%j %j] “%j %r %j” %c %b “%f” “%B” %j\n)

更多参考

LOGFILE 和OUTFILE说明

LOGFILE new1.log,old*.log
LOGFILE /opt/log/%Y.%M/%D/*.c1gstudio.com.log.gz
支持通配符,日期变量及gz压缩,OUTFILE不会自动创建目录

%D date of month
%m month name, in English
%M month number
%y two-digit year
%Y four-digit year
%H hour
%n minute
%w day of week, in English

但是日期不支持运算有点麻烦,需要外部用shell来解决了
更多参考
==================================
2014-8-26更新

The arguments to LOGFILE and CACHEFILE commands are checked for containing only certain allowed characters (specifically, letters, digits, /\.:_*? space, and – between two {letter, digit, underscore}’s). This is because they could match an UNCOMPRESS command and thus be passed to the shell when the uncompress command is popen()’ed.

可以将一个月份分成3部分来减轻压力
LOGFILE /opt/log/%Y.%M/[2-3]?/*.c1gstudio.com.log.gz
Analog运行时会将日志读到内存中,想要运行快最好准备比日志大的内存,CACHEOUTFILE和CACHEFILE会占用大量空间,感觉没什么用.
==================================

配置文件内统计开关变量

MONTHLY ON # one line for each month
WEEKLY ON # one line for each week
DAILYREP ON # one line for each day
DAILYSUM ON # one line for each day of the week
HOURLYREP ON # one line for each hour of the day
GENERAL ON # the General Summary at the top
REQUEST ON # which files were requested
FAILURE ON # which files were not found
DIRECTORY ON # Directory Report
HOST ON # which computers requested files
ORGANISATION ON # which organisations they were from
DOMAIN ON # which countries they were in
REFERRER ON # where people followed links from
FAILREF ON # where people followed broken links from
SEARCHQUERY ON # the phrases and words they used…
SEARCHWORD ON # …to find you from search engines
BROWSERSUM ON # which browser types people were using
OSREP ON # and which operating systems
FILETYPE ON # types of file requested
SIZE ON # sizes of files requested
STATUS ON # number of each type of success and failure

命令行参数

x GENERAL General Summary
1 YEARLY Yearly Report
Q QUARTERLY Quarterly Report
m MONTHLY Monthly Report
W WEEKLY Weekly Report
D DAILYREP Daily Report
d DAILYSUM Daily Summary
H HOURLYREP Hourly Report
h HOURLYSUM Hourly Summary
w WEEKHOUR Hour of the Week Summary
4 QUARTERREP Quarter-Hour Report
6 QUARTERSUM Quarter-Hour Summary
5 FIVEREP Five-Minute Report
7 FIVESUM Five-Minute Summary
S HOST Host Report
l REDIRHOST Host Redirection Report
L FAILHOST Host Failure Report
Z ORGANISATION Organisation Report
o DOMAIN Domain Report
r REQUEST Request Report
i DIRECTORY Directory Report
t FILETYPE File Type Report
z SIZE File Size Report
P PROCTIME Processing Time Report
E REDIR Redirection Report
I FAILURE Failure Report
f REFERRER Referrer Report
s REFSITE Referring Site Report
N SEARCHQUERY Search Query Report
n SEARCHWORD Search Word Report
Y INTSEARCHQUERY Internal Search Query Report
y INTSEARCHWORD Internal Search Word Report
k REDIRREF Redirected Referrer Report
K FAILREF Failed Referrer Report
B BROWSERREP Browser Report
b BROWSERSUM Browser Summary
p OSREP Operating System Report
v VHOST Virtual Host Report
R REDIRVHOST Virtual Host Redirection Report
M FAILVHOST Virtual Host Failure Report
u USER User Report
j REDIRUSER User Redirection Report
J FAILUSER User Failure Report
c STATUS Status Code Report

#+a可以带上全部统计
更多参考

#输出当前配置
analog -settings > file

#使用命令行配置LOGFILE和OUTFILE
./analog +O/opt/htdocs/www/analog/c1gstudio2014.html /opt/log/2014.08/02/*.c1gstudio.com.log.gz
我使用时一直会报日志格式错误,无法出报告

#我使用的参数
/usr/local/analog -G +g/usr/local/analog/conf/c1g.cfg +b +s +S -n -o -Z -r
+b 浏览器概要报告
-n 检索字报告
+s 来源网站报告
-o 网域报告
-Z 来源组织单位报告
+S 主机报告
-r 请求报告

-G 不读analog.cfg
+g读取自定义配置文件

我这每日报告用awstats统计,每月报告用analog统计,每个域名汇总一个月报告.
日志按天存放在/opt/log/2014.08/07/目录下
www.c1gstudio.com.log.gz
blog.c1gstudio.com.log.gz
www.c1g.com.log.gz

每日运行完awstats后运行analog
crontab

10 5 * * * /bin/sh /opt/shell/analog.sh > /dev/null 2>&1

vi /opt/shell/analog.sh

#!/bin/sh
ana_dir=/usr/local/analog/
web_dir=/opt/htdocs/www/analog/
conf_dir=”${ana_dir}/conf/”

today=`date +%d`
yesterday=`date +%Y%m%d`
lastday_month=`date +%Y.%m -d ‘1 day ago’`
lastday_day=`date +%d -d ‘1 day ago’`

c1g_LOGFILE=/opt/log/${lastday_month}/*/*c1gstudio.com.log.gz
c1g_OUTFILE=${web_dir}c1gstudio${lastday_month}/index.html

POST_LOGFILE=/opt/log/${lastday_month}/*/c1g.com.log.gz
POST_OUTFILE=${web_dir}c1g${lastday_month}/index.html

#if [ $today == “02” ]; then
if [ ! -d $(dirname “${c1g_OUTFILE}”) ]; then
mkdir -p $(dirname “${c1g_OUTFILE}”)
chown www:website $(dirname “${c1g_OUTFILE}”)
fi
if [ ! -d $(dirname “${POST_OUTFILE}”) ]; then
mkdir -p $(dirname “${POST_OUTFILE}”)
chown www:website $(dirname “${POST_OUTFILE}”)
fi
sed -i “s;LOGFILE.*;LOGFILE ${c1g_LOGFILE};” ${conf_dir}c1gstudio.cfg
sed -i “s;OUTFILE.*;OUTFILE ${c1g_OUTFILE};” ${conf_dir}c1gstudio.cfg
sed -i “s;LOGFILE.*;LOGFILE ${POST_LOGFILE};” ${conf_dir}c1g.cfg
sed -i “s;OUTFILE.*;OUTFILE ${POST_OUTFILE};” ${conf_dir}c1g.cfg
#fi

${ana_dir}analog -G +g${conf_dir}c1gstudio.cfg +b +D -d +s +S -n -o -Z -r
${ana_dir}analog -G +g${conf_dir}c1g.cfg +b +D -d +s +S -n -o -Z +r