Skip to content

anuragkumar/python-multiprocessing-bfs-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 

Repository files navigation

python-multiprocessing-bfs-crawler

A simple multiprocessing example in python

This is a multiprocessing example in python.

Multi-threading is not used as it has a major drawback. Python interpreter doesn't actually implement multthreading to use multiple cores of the cpu. So, implementing multi-threading won't do any merit in terms of faster execution. It will be same as single thread. In multi-processing, the os initializes a separate python interpreter instance to each sub process and so, it can be used to speed up your code.

Multi-threading and multi-processing has different advantages depending on the usage.

Without multiprocessing, this script took ~11 hours to complete. With multiprocessing, it was completed in ~3 to 3.5 hours

About

A simple multiprocessing example in python

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages