手撕包菜SSBC(DHT磁力链源码) 2016.8月版 安装图文教程

环境:
Vultr日本    2G内存  Centos7 x64   有公网IP
DO    1G内存      Centos7 x64     有公网IP
Linode    1G内存      Centos7 x64     有公网IP
[Vultr日本1 G内存在索引时报错:FATAL: out of memory (unable to allocate 1278738432 bytes)],可能是Vultr的VPS没有swap的原因
经测试,采集到的英文资源居多。

本文环境:

Linode 2048

24GB DISK1 CPU Core2TB XFER.015/hr to $10/mo

Linode全面升级 推出月付5刀的套餐 机房可选东京2

Vultr 充5美元可以用5个月VPS 充值最高送100美元

一.确认python版本,获取ssbc源代码,关闭防火墙

1.环境检测(Ssbc当前版本是基于django1.8.1开发,所需python环境为python2.7.5以上。)

 [[email protected] ~]# python -V    //执行python -V即可获取当前版本
 Python 2.7.5
 [[email protected] ~]# systemctl stop firewalld.service  //关闭firewall防火墙
 [[email protected] ~]# systemctl disable firewalld.service    //禁止firewall防火墙开机启动
 [[email protected] ~]# systemctl stop iptables.service   //关闭iptables防火墙
 [[email protected] ~]# systemctl disable iptables.service    //禁止iptables防火墙开机启动

2.获取ssbc安装包并解压

 [[email protected] ~]# wget https://github.com/78/ssbc/archive/master.zip
 [[email protected] ~]# yum -y install unzip
 [[email protected] ~]# unzip master.zip

解压后你会发现在/root目录下有个文件夹ssbc-master

二.安装MariaDB,创建数据库

1.安装数据库及所需环境

 [[email protected] ~]# yum -y install gcc gcc-c++ python-devel mariadb mariadb-devel mariadb-server
 [[email protected] ~]# cd ssbc-master
 [[email protected] ssbc-master]# wget https://bootstrap.pypa.io/get-pip.py
 [[email protected] ssbc-master]# python get-pip.py
 [[email protected] ssbc-master]# pip install -r requirements.txt

2.创建ssbc数据库

 [[email protected] ssbc-master]# systemctl start mariadb.service    //启动数据库
 [[email protected] ssbc-master]# mysql -uroot -p
 Enter password: (回车即可)
 MariaDB [(none)]> create database ssbc default character set utf8;
 MariaDB [(none)]> \q    //创建成功后退出

三.安装Sphinx

[[email protected] ssbc-master]# yum -y install unixODBC unixODBC-devel postgresql-libs
[[email protected] ssbc-master]# wget http://sphinxsearch.com/files/sphinx-2.2.9-1.rhel7.x86_64.rpm
[[email protected] ssbc-master]# rpm -ivh sphinx-2.2.9-1.rhel7.x86_64.rpm
[[email protected] ssbc-master]# systemctl restart sphinx.service

四. 建立文件夹,创建以下文件夹并赋予755权限

[[email protected] ssbc-master]# mkdir  -p  /data/bt/index/db /data/bt/index/binlog  /tem/downloads
[[email protected] ssbc-master]# chmod  755 -R /data
[[email protected] ssbc-master]# chmod  755 -R /tem

五. 生成索引

[[email protected] ssbc-master]# systemctl restart mariadb.service  //重新启动Mariadb
[[email protected] ssbc-master]# systemctl enable mariadb.service  //设置mariadb开启自启动
[[email protected] ssbc-master]# indexer -c sphinx.conf --all     //(all 前面是空格减号减号)
[[email protected] ssbc-master]# searchd --config ./sphinx.conf   //(config前是空格减号减号)

确定没有报错,继续下一步

六.Web服务器设置并启动

[[email protected] ssbc-master]# python manage.py makemigrations
[[email protected] ssbc-master]# python manage.py migrate
[[email protected] ssbc-master]# nohup python manage.py runserver 0.0.0.0:80 >/dev/zero &    //启动网站并在后台运行

按回车键继续,这里可以使用其他端口然后前端再使用nginx做反向代理。手撕包菜SSBC(DHT磁力链源码) 前端使用 Nginx反向代理 80端口 图文教程
浏览器输入http://IP,网站能打开

七. 开启爬虫(workers目录下)

 [[email protected] ssbc-master]# cd workers
 [[email protected] workers]# python simdht_worker.py  #爬虫运行,等2分钟出现数据之后CTRL+C停止
 [[email protected] workers]# nohup python simdht_worker.py > /dev/zero  #让爬虫在后台运行

按回车键继

 [[email protected] workers]# python index_worker.py  #入库索引,等待10分钟出现数据后CTRL+C停止
 [[email protected] workers]# nohup python index_worker.py >/dev/zero #让索引在后台运行

按回车键继

增加后台管理员

 [[email protected] workers]#cd ..
 [[email protected] ssbc-master]# python manage.py createsuperuser

输入管理员用户名
输入管理员邮箱
输入管理员密码
确认密码,完成
管理员登录地址:http://IP/admin

测试效果:
搜索中文英文关键字均正常
1

2


去除搜索页 右下角广告

[[email protected] ssbc-master]# cd web/static/js
[[email protected] js]# vi ssbc.js   找到如下3行,在前面添加//进行注释,保存
//        document.write(‘<script src=”https://www.jiloc.com/wp-content/uploads/201608074015uMCYx.jpg”><\/script>’);
//            document.writeln(“<script language=\”JavaScript\” type=\”text/javascript\” src=\”http://js.6dad.com/js/xiaoxia.js\”></script>”);
//           document.writeln(“<script language=\”JavaScript\” type=\”text/javascript\” src=\”http://js.ta80.com/js/12115.js\”></script>”);

1.搜索中文报错

‘ascii’ codec can’t encode characters in position 42-43: ordinal not in range(128)
2
解决办法:
如果是centos7系统,修改/usr/lib64/python2.7/site.py
vi  /usr/lib64/python2.7/site.py
在import sys下添加2行:

reload(sys)
sys.setdefaultencoding('utf8')

2.爬虫运行时 可能会遇到如下问题:

Python and Django OperationalError (2006, ‘MySQL server has gone away’)
解决方法:
http://stackoverflow.com/questions/14163429/python-and-django-operationalerror-2006-mysql-server-has-gone-away
/etc/my.cnf 在最后一行的下面添加

 wait_timeout=2880000
 interactive_timeout = 2880000
 max_allowed_packet = 512M

常见问题:

1.必须centos7吗?

非常建议使用centos7,centos6可能会有意想不到的错误

2.怎么查看入库的文件?

登录管理员后台,点击 Hashs

3.怎么查看每天入库了多少文件,以便清楚入库效率?

登录管理员后台,点击 Status reports

4.如何确认web服务器、采集、入库正在运行?

ps -ef|grep python

结果里面有
python manage.py runserver 0.0.0.0:80
python simdht_worker.py
python index_worker.py
即表示正在运行。

5.如出现没有模块pygeoip,请使用命令

pip install pygeoip

 

进行安装

[[email protected] ssbc-master]# python workers/simdht_worker.py
Traceback (most recent call last):
File “workers/simdht_worker.py”, line 28, in <module>
import pygeoip
ImportError: No module named pygeoip

6.IOError: [Errno 2] No such file or directory: ‘GeoIP.dat’提示错误,解决办法:

从http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz下载GeoIP包,上传到/root/ssbc-master文件夹即可。

[[email protected] ssbc-master]# python workers/simdht_worker.py
exceptions must be old-style classes or derived from BaseException, not NoneType
exceptions must be old-style classes or derived from BaseException, not NoneType
Traceback (most recent call last):
File “workers/simdht_worker.py”, line 39, in <module>
import simMetadata
File “/root/ssbc-master/workers/simMetadata.py”, line 11, in <module>
from simdht_worker import entropy
File “/root/ssbc-master/workers/simdht_worker.py”, line 41, in <module>
from metadata import save_metadata
File “/root/ssbc-master/workers/metadata.py”, line 14, in <module>
geoip = pygeoip.GeoIP(‘GeoIP.dat’)
File “/usr/lib/python2.7/site-packages/pygeoip/__init__.py”, line 75, in __call__
cls._instances[filename] = super(_GeoIPMetaclass, cls).__call__(*args, **kwargs)
File “/usr/lib/python2.7/site-packages/pygeoip/__init__.py”, line 118, in __init__
self._fp = codecs.open(filename, ‘rb’, ENCODING)
File “/usr/lib64/python2.7/codecs.py”, line 881, in open
file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: ‘GeoIP.dat’

7.”Can’t connect to MySQL server on ’12 7.0.0.1′ (111)”,无法连接到数据库。

在运行脚本之前需要修改2个文件,因为从git上下载下来的源码里写的mariadb的端口是9306,我们这里做一下修改改成3306,否则无法启动。workers/index_worker.py:24行,search/models.py:13行,把9306替换成3306。否则会出现无法连接数据库。

可能是因为sphinx的原因造成的,由于先前的版本是先运行脚本然后再运行sphinx,所以会报错,这里我们已经将过程进行对调。

[[email protected] ssbc-master]# python manage.py makemigrations
Traceback (most recent call last):
File “manage.py”, line 10, in <module>
execute_from_command_line(sys.argv)
File “/usr/lib/python2.7/site-packages/django/core/management/__init__.py”, li ne 338, in execute_from_command_line
utility.execute()
File “/usr/lib/python2.7/site-packages/django/core/management/__init__.py”, li ne 312, in execute
django.setup()
File “/usr/lib/python2.7/site-packages/django/__init__.py”, line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File “/usr/lib/python2.7/site-packages/django/apps/registry.py”, line 108, in populate
app_config.import_models(all_models)
File “/usr/lib/python2.7/site-packages/django/apps/config.py”, line 198, in im port_models
self.models_module = import_module(models_module_name)
File “/usr/lib64/python2.7/importlib/__init__.py”, line 37, in import_module
__import__(name)
File “/root/ssbc-master/search/models.py”, line 13, in <module>
search_conn = mdb.connect(‘127.0.0.1’, ‘root’, ”, ”, port=9306, charset=’u tf8′, cursorclass=MySQLdb.cursors.DictCursor)
File “/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py”, line 81, in Con nect
return Connection(*args, **kwargs)
File “/usr/lib64/python2.7/site-packages/MySQLdb/connections.py”, line 193, in __init__
super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2003, “Can’t connect to MySQL server on ’12 7.0.0.1′ (111)”)

[[email protected] ssbc-master]# grep -rn 9306 /root/ssbc-master    #查找一下文件夹中包含9306的文件
workers/index_worker.py:24:dst_conn = mdb.connect(DST_HOST, DST_USER, DST_PASS, ‘rt_main’, port=9306, charset=’utf8′)
search/models.py:13:search_conn = mdb.connect(‘127.0.0.1’, ‘root’, ”, ”, port 9306, charset=’utf8′, cursorclass=MySQLdb.cursors.DictCursor)

 

16 Replies to “手撕包菜SSBC(DHT磁力链源码) 2016.8月版 安装图文教程”

  1. 怎么关闭调试模式呢?
    You’re seeing this error because you have DEBUG = True in your Django settings file. Change that to False, and Django will display a standard 404 page.

  2. 博主 你好我重启linux主机之后就无法启动索引入库了

    [[email protected] workers]# python index_worker.py
    fetched 586
    Traceback (most recent call last):
    File “index_worker.py”, line 44, in
    if work() == 10000:
    File “index_worker.py”, line 35, in work
    (one[‘id’], one[‘name’], one[‘category’], one[‘length’], one[‘create_time’], one[‘last_seen’]))
    File “/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py”, line 205, in execute
    self.errorhandler(self, exc, value)
    File “/usr/lib64/python2.7/site-packages/MySQLdb/connections.py”, line 36, in defaulterrorhandler
    raise errorclass, errorvalue
    _mysql_exceptions.ProgrammingError: (1064, “no such index ‘rt_main'”)

  3. 数据表没有啊,爬行的时候显示ssbc.search_hash not exist
    只创建了数据库没有任何表!!!

      1. 执行python manage.py makemigrations的时候,提示连接数据库密码不对,而且using password:NO,应该改哪里?

  4. 执行python manage.py makemigrations的时候,提示连接数据库1045,而且using password:NO,应该改哪里?

      1. manage.py没有用户名什么的啊,就几句代码,你qq多少,我加上你吧

          1. 你可以给我弄一下么?或者在群里回复我一下,我给你远程账号密码,您帮我弄一下可以么?linux真不会

发表评论

电子邮件地址不会被公开。 必填项已用*标注