Python Crawler.extract_links Examples

Example #1

File: crawler_test.py Project: phanindra-anchuri/crawlsearch

 def robots_test(self):
     crawler = Crawler(self.test_url, 1)
     self.assertTrue(self.robot_path and self.robot_url
                     not in crawler.extract_links(self.test_url))

Example #2

File: crawler_test.py Project: phanindra-anchuri/crawlsearch

 def extract_test(self):
     crawler = Crawler(self.test_url, 1)
     extracted_urls = crawler.extract_links(self.test_url)
     self.assertTrue(len(extracted_urls) > 0)