Zester is a library that makes it easier to develop Python clients for websites without APIs.
No lxml, no XPath, just javascript.
Let's make a client library for Hacker News by saving the following code in a file named hnclient.py:
from zester import MultipleClient, Attribute
class HNClient(MultipleClient):
url = "http://news.ycombinator.com/"
title = Attribute(selector="$('.title a')", modifier="$(el).html()")
link = Attribute(selector="$('.title a')"), modifier="$(el).attr('href')")
points = Attribute(selector="$('.subtext span')", modifier="$(el).html().replace(' points', '')")Now, let's use the client we just made. Open a python shell:
>>> from hnclient import HNClient
>>> client = HNClient()
>>> stories = client.process()
>>> stories[0]
HNClientResponse(points=u'200', link=u'http://daltoncaldwell.com/what-twitter-could-have-been', title=u'What Twitter could have been')
>>> print stories[0].title
What Twitter could have been
>>> print stories[0].link
http://daltoncaldwell.com/what-twitter-could-have-been
>>> print stories[0].points
56We subclassed MultipleClient there because we were planning on returning multiple results. If we wanted to make a client for something like Weather.gov that returned a single result, we could do something like this:
from zester import SingleClient, Attribute
class WeatherClient(SingleClient):
url = "http://forecast.weather.gov/MapClick.php?lat={lat}&lon={lng}"
temperature = Attribute(selector="$('.myforecast-current-lrg').html()")
humidity = Attribute(selector="$('.current-conditions-detail li').contents()[1]")
heat_index = Attribute(selector="$('.current-conditions-detail li').contents()[11]")
def __init__(self, lat, lng, *args, **kwargs):
super(WeatherClient, self).__init__(*args, **kwargs)
self.url = self.url.format(lat=lat, lng=lng)This also demonstrates how you can allow arguments to be taken:
>>> from weather_client import WeatherClient
>>> client = WeatherClient(lat=40.7143528, lng=-74.0059731)
>>> curr_weather = client.process()
>>> curr_weather
WeatherClientResponse(heat_index=u'82\xb0F (28\xb0C)', temperature=u'80\xb0F', humidity=u'58%')
>>> print curr_weather.temperature
80°F
>>> print curr_weather.humidity
58%
>>> print curr_weather.heat_index
82°F (28°C)Zester is dependant upon Ghost.py. You must install it before installing Zester. Ghost.py will also require the installation of either PyQt or PySide.
After Ghost.py is installed, to install zester: :
$ pip install zester