A spider which crawls and analyzes ads in the web page. Our main objective is generating ABP filters automatic.

A lot of ads are presented with the model of below.

<a href ="ads Target URL">
  <img src = "ads content URL" />
</a>

The host URL is the URL of page which hosts the ads.

0. Based on Ads present model, crawl the web and record the ads profile item into database.
1. Analyze the profiles and find the items which are probably ads.
2. Generate ABP filters from ads profile item.

0. python 2.7
1. tld
2. lxml

0. MySQL-python
1. python-gflags
2. google mysql-tools

1. Scrapy
2. Frontera

0. Python 2.7
1. pip and setuptools Python packages. Nowadays pip requires and installs setuptools if not installed.
2. Install tld through pip

$ pip install tld

3. Install lxml for python

$ pip install lxml

4. Install MySQL-python through yum

$ yum install MySQL-python

5. Install python-gflags

$ pip install python-gflags

6. Intsall Scrapy

$ pip install scrapy

7. Install Frontera

$ pip install frontera[distributed,zeromq,sql]

Thanks for ScarpyHub, AdSpider integrate Scrapy with Frontera to achieve a broad distributed Spdier.