A scrapy selector that supports javascript querying.

Ghost Selector understands Selenium's PhantomJS webdriver and takes advantage of the API it offers to behave just like the built-in Scrapy selectors.

Under the hood, selenium uses GhostDriver to talk to PhantomJS via selenium's WebDriver Wire Protocol. All Ghost Selector does is wrap whatever selenium returns after running javascript.

>>> from selenium.webdriver import phantomjs
>>> from ghostsel import WebdriverXPathSelector
>>> driver = phantomjs.webdriver.WebDriver()
>>> driver.get("http://en.wikipedia.org/wiki/Main_Page")
>>> selector = WebdriverXPathSelector(driver)
>>> list_items = selector.select_script("return $('#p-navigation li')")
>>> list_items[0].extract()
u'Main page'
>>> 

See test for usage.

Thanks goes to my good friend Nicolas for sponsoring its development.