This is a streaming capture client designed to work asynchronously with a web server for starting/stopping captures. The project for the website front-end Twitter Capture Django. The website enables a centralized location for researchers to manage their collections remotely. In the current version, this client uses a REST API to communicate with the server and report statistics back to it. This design allows many collections to be managed across many different machines. We're currently engineering pubsub changes, to allow the clients to send and receive updates in real-time to both the server and to the website.
The software is based on Tweepy. More about the main projet can be found on the Twitter Capture Client github site, including some setup instructions. For the client, you will need the following from Twitter:
- API Key
- API Secret
- Access Token
- Access Token Secret
This can be retrieved by creating an App in the Twitter Developer App Console. After creating an app you will need to generate the access tokens. As the website mentions, be sure to keep these private.
From the Twitter Capture Client server, you will need the following:
- REST API url
- Auth token
The auth token can be generated by creating a new client. THere is a link on the front page.
Below is an example settings file.
{
"server": {
"base_url": "<insert server API address>",
"client_id": 1,
"auth_token": "<auth token generated from server>",
"ping_interval": 10.0,
"update_interval": 30.0
},
"twitter_auth": {
"api_key": "<<<twitter api key>>>",
"api_secret": "<<<twitter api secret>>>",
"access_token": "<<<twitter access token>>>",
"access_token_secret": "<<<access token secret>>>"
},
"source_addr": "<<<ip address to bind to>>>",
"output": {
"base_dir": "./data/",
"extension": ".json",
"temporary_extension": ".tmp",
"minute_interval": 10,
"filename_timefmt": "%Y%m%d_%H%M"
},
"logging": {
"version": 1,
"disable_existing_loggers": false,
"formatters" : {
"default" : {
"format": "%(asctime)s %(levelname)-8s %(name)-15s %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S"
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"level": "DEBUG",
"formatter": "default",
"stream": "ext://sys.stdout"
},
"error_file_handler": {
"class": "logging.handlers.RotatingFileHandler",
"level": "ERROR",
"formatter": "default",
"filename": "logs/54/errors.log",
"maxBytes": 10485760,
"backupCount": 20,
"encoding": "utf8"
}
},
"loggers": {
"twitter_monitor.listener": {
"level": "WARN",
"handlers": ["console"],
"propagate": "no"
},
"requests.packages.urllib3.connectionpool": {
"level": "WARN",
"handlers": ["console"],
"propagate": "no"
}
},
"root": {
"level": "DEBUG",
"handlers": [
"console",
"error_file_handler"
]
}
}
}