Skip to content

renxiawang/tncrawler

Repository files navigation

tncrawler

A crawler for crawling user profiles, followings and followers on Twitter.

Requires

  1. Tweepy
  2. Pymongo
  3. MongoDB

Usages

To initial the crawling task, execute:

python main.py user_list

where user_list is the file name of a text file that contains user ids. One user id per line.

To continue the task after program crash, execute:

python main_load.py

Todos

  1. Complete the commond-line arguments processing function
  2. Meet PEP8 Standard
  3. Rewrite three thread classes; Consider a solution using thread pool
  4. Logging function
  5. Combine main.py and main_load.py

About

Twitter network crawler

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages