Python Spider.crawl_robots Examples

Example #1

File: main.py Project: jorsae/Crawler

def test2():
    sp = Spider('reddit.com')
    sp.crawl_robots()
    sp.test()
    sp.crawl()