Python Spider.crawl_async_slots Examples

Example #1

File: main.py Project: NikolaiMagnussen/ercin

def start_sp(queue, parent_queue):
    sp = Spider(queue, parent_queue, verbose=True, batch_size=10)
    sp.crawl_async_slots()
    print("[INFO] I am done crawling - handing over to someone else")
    gc.collect()