kickleech反爬虫

介绍

shell反爬虫脚本手动版，分析web日志找出访问量异常高的IP。

• 手动版反爬虫，输出ip及相关信息供反爬虫判断;
• 运行前请确保日志临时目录V_TMPFILE存在并可读写,调用IP所在地公网接口需要curl支持
• 支持单独白名单文件，nginx白名单文件，支持ipv4,ipv6 的/24斜线写法，不支持255.255.255.0 写法
• 支持取日志中remote_addr或http_x_forwarded_for左一(没有则取remote_addr)为判断IP
• 可以指定提取全部或多少条日志进行分析，排序显示单IP或IP C段大于指定阈值的IP
• 运行结果可以输出到指定日志文件
• 命令行支持指定WEB日志文件/日志条数/单IP阈值/IP C段阈值/显示IP数量/显示当前IP白名单
• 可以指定是否排除蜘蛛等特定UserAgent
• 分析后显示单IP请求数/页面分布/HTTP状态分布/抽样/IP所在地/是否(白|黑)名单中/是否正常请求

优势

• 支持http_x_forwarded_for取IP
• 支持IPV6地址
• 支持IPC段统计
• 支持IP黑白名单

项目地址

• Gitee(国内)： https://gitee.com/c1g/kickleech
• Blog： https://blog.c1gstudio.com/

安装部署

kickleech手动版

wget -O kickleech_manual.sh https://gitee.com/c1g/kickleech/raw/master/kickleech/kickleech_manual.sh
chmod +x ./kickleech_manual.sh
#脚本会自动创建临时目录和日志目录，也可以手动创建
mkdir -p /opt/nginx/logs/
mkdir ./log

./kickleech_manual.sh

使用说明

Example: ./kickleech_manual.sh -f /var/log/nginx/access.log -n 500000 -t 1000 -c 3000 -m 100 -s -i

OPTION:
-f apache log file,default=/var/log/nginx/access.log
-n output the last K lines {full|500000},default=full
-t threshold for the number of requests from same IP address,default=1000
-c threshold for the number of requests from same class C IP address,default=3000
-m show number of IP,default=100
-s show IP allow list
-i show IP location
-v show this information
RESULT:
IP Count: ./log/kickleech_manual_ip_count.log
Class C IP Count: ./log/kickleech_manual_ipc_count.log
IP Detail: ./log/kickleech_manual_ip_info.log
IP Result: ./log/kickleech_manual_ip_bad.log

也可以直接编辑脚本将常用参数写在变量里。  

示例

./kickleech_manual.sh -f /var/log/nginx/bbs.c1gstudio.com.log -n 50000 -t 100 -c 300 -m 100 -s -i

=====start:20230706 18:04:46=====
LOAD nginx safe ip
SAFE IP:
127.0.0.1/32 192.168.0.0/24
LINE number:50000
364 64.62.252.163
241 173.208.200.2
139 182.148.222.121
115 110.188.45.251
101 183.146.253.6
KICKLEECH BAD 64.62.252.163
KICKLEECH BAD 173.208.200.2
KICKLEECH BAD 182.148.222.121
KICKLEECH BAD 110.188.45.251
KICKLEECH BAD 183.146.253.6
skip reload nginx!
=====end:20230706 18:04:50=====

结果查看

cat log/kickleech_manual_ip_info.log

get head line:

220.185.30.0 – – [27/Jun/2023:13:08:06 +0800] “GET /forum.php?mod=forumdisplay&fid=56&filter=digest&digest=1&typeid=25&specialtype=trade&orderby=views HTTP/1.1” 200 7925 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.873.3 Safari/537.36”

364 64.62.252.163
get status layout:5
364 200
get page layout:7
199 /space
125 /thread
40 /forum.php?mod=forumdisplay
get sample:3
64.62.252.163 – – [27/Jun/2023:13:08:16 +0800] “GET /space-uid-735.html HTTP/1.1” 200 4182 “-” “The Knowledge AI” 64.62.252.163 – – [27/Jun/2023:13:08:16 +0800] “GET /space-username-a.html HTTP/1.1” 200 4203 “-” “The Knowledge AI” 64.62.252.163 – – [27/Jun/2023:13:08:17 +0800] “GET /thread-935-1-1.html HTTP/1.1” 200 4959 “-” “The Knowledge AI” get host:
Host 163.252.62.64.in-addr.arpa. not found: 3(NXDOMAIN)
get ip location:
{“ip”:”64.62.252.163″,”beginip”:”64.62.244.152″,”endip”:”64.62.255.255″,”country”:”美国”,”area”:”加利福尼亚州弗里蒙特市Hurricane Electric公司”}

241 173.208.200.2
get status layout:5
241 200
get page layout:7
91 /forum.php?mod=forumdisplay
38 /connect.php?mod=login
6 /forum.php?mod=viewthread
3 /forum.php?mod=post
1 /forum.php?mod=redirect
get sample:3
173.208.200.2 – – [27/Jun/2023:13:09:32 +0800] “GET /forum.php?mod=forumdisplay&fid=94&orderby=lastpost HTTP/1.1” 200 7989 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2545.102 Safari/537.36”
173.208.200.2 – – [27/Jun/2023:13:09:34 +0800] “GET /forum.php?mod=forumdisplay&fid=34&filter=author HTTP/1.1” 200 8210 “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.1717.77 Safari/537.36”
173.208.200.2 – – [27/Jun/2023:13:09:35 +0800] “GET /forum.php?mod=forumdisplay&fid=39&filter=author HTTP/1.1” 200 8250 “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.1349.50 Safari/537.36”
get host:
2.200.208.173.in-addr.arpa domain name pointer smtp101.florybeauty.com.
get ip location:
{“ip”:”173.208.200.2″,”beginip”:”173.208.128.0″,”endip”:”173.208.255.255″,”country”:”美国”,”area”:”密苏里州堪萨斯城WholeSale互联网股份有限公司”}

目录结构

.
├── kickleech_manual.sh
├── kickleechsafeip.conf
└── log

├── kickleech_manual_ip_bad.log
├── kickleech_manual_ipc_count.log
├── kickleech_manual_ip_count.log
├── kickleech_manual_ip_info.log
└── kickleech_manual_log_2023.log