Skip to content

brilliang/scrapy_selenium_middleware

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 

Repository files navigation

This is a simplest Scrapy project. current at its 1.05 version.

Inside there is a download middleware working with a selenium grid, which is a web browser programming interface and is very suitable for javascript rendering job for web data grabbing.

In a scrapy Spider , you only need set up a flag of SELENIUM, and optional some action inside the browser, such as click a button, or waiting for some elements inside the webpage. Spiders without the SELENIUM flag, will go the original way.

Also, the selenium grid setting is easy, just follow this instruction.

About

the download middleware

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages