NASTY is a tool/library for retrieving Tweets via the Twitter Web UI. Instead of using the Twitter Developer API it works by acting like a normal web browser accessing Twitter. That is, it sends AJAX requests and parses Twitter's JSON responses. This approach makes it substantially different from the other popular crawlers and allows for the following features:

• Search for Tweets by keyword (and filter by latest/top/photos/videos, date of authorship, and language).
• Retrieve all direct replies to a Tweet.
• Retrieve all Tweets threaded under a Tweet.
• Return fully-hydrated JSON-objects of Tweets that exactly match the extended mode of the developer API
• Operate in batch mode to execute a large set of requests, abort at any time, and rerun both uncompleted and failed requests.
• Transform collected Tweets into sets of Tweet-IDs for publishing datasets. Automatically download full Tweet information from sets of Tweet-IDs.
• Written in tested, linted, and fully type-checked Python code.

Python 3.6, 3.7, 3.8 and PyPy are currently supported. Install via:

$ pip install nasty

To get help for the command line interface use the --help option:

$ nasty --help
usage: nasty [-h] [-v] [search|replies|thread|batch|idify|unidify] ...

NASTY Advanced Search Tweet Yielder.

Commands:
  The following commands (and abbreviations) are available, each supporting
  the help option. For example, try out `nasty search --help`.

  <COMMAND>
    search (s)         Retrieve Tweets using the Twitter advanced search.
    replies (r)        Retrieve all directly replying Tweets to a Tweet.
    thread (t)         Retrieve all Tweets threaded under a Tweet.
    batch (b)          Execute previously created batch of requests.
    idify (i, id)      Reduce Tweet-collection to Tweet-IDs (for publishing).
    unidify (u, unid)  Collect full Tweet information from Tweet-IDs (via
                       official Twitter API).

General Arguments:
  -h, --help           Show this help message and exit.
  -v, --version        Show program's version number and exit.
  --log-level <LEVEL>  Logging level (DEBUG, INFO, WARN, ERROR.)

You can also get help for the individual sub commands. For example, try out nasty search --help.

You can search for Tweets about "climate change":

$ nasty search --query "climate change"

NASTY's output are lines of JSON objects, one per retrieved Tweet. Each Tweet-JSON has the following format (pretty-printed and abbreviated for clarity, many other interesting features are also available, such as referenced entities, etc.):

{
    "created_at": "Wed Jan 11 04:52:08 +0000 2017",
    "id_str": "8190441963...",
    "full_text": Thank you for everything..."
    "retweet_count": 795...,
    "favorite_count": 1744...,
    "reply_count": 22...,
    "lang": "en",
    "user": {
        "id_str": "15367...",
        "name": "Presi...",
        "screen_name": "POTUS...",
        "location": "Washing...",
        "description": "This is an archive...",
        ...
    },
    ...
}

By default this returns TOP Tweets according to Twitter's own ranking rules. Alternatively you can also request the very LATEST Tweets via:

$ nasty search --query "climate change" --filter LATEST

Other possible values for --filter are PHOTOS and VIDEOS.

By default only English Tweets are found. For example, to instead search for German Tweets:

$ nasty search --query "climate change" --lang "de"

Additionally, you can specifically search for Tweets created after and/or before specific dates:

$ nasty search --query "climate change" --since 2019-01-01 --until 2019-01-31

You can fetch all direct replies to the Tweet with ID 332308211321425920:

$ nasty replies --tweet-id 332308211321425920

You can fetch all Tweets threaded under the Tweet with ID 332308211321425920:

$ nasty thread --tweet-id 332308211321425920

NASTY supports appending requests to a batch file instead of executing them immediately, so that they can executed in batch mode later. The benefits of this include being able to track the progress of a large set of requests, aborting at any time, and rerunning both completed and failed requests.

To append a request to a batch file, use the --to-batch argument on any of the above requests, for example:

$ nasty search --query "climate change" --to-batch batch.jsonl

To run all files stored in a jobs file and write the output to directory out/:

$ nasty batch --batch-file batch.jsonl --results-dir out/

When this command finished a tally of successful, skippend, and failed requests is printed. If any request failed, you may retry execution with the same command. Requests that succeeded will automatically be skipped.

The Twitter Developer Policy states that for sharing collected Tweets with others, only Tweet-IDs may be (publicly) distributed (see Legal and Moral Considerations for more information).

To transform lines of Tweet-JSON-objects into lines of Tweet-IDs, use nasty idify. For example:

$ nasty search --query "climate change" | nasty idify > climate-change-tweet-ids.txt

To perform the reverse, that is getting full Tweet information from just Tweet-IDs, use nasty unidify:

$ cat climate-change-tweet-ids.txt | nasty unidify

Note that unidify is implemented using the Twitter Developer API, since for this specific case, the available free API covers all needed functionality and rate-limits are not to limiting. Additionally, this means, that this specific functionality is officially supported by Twitter, meaning the API should be stable over time (thus making it ideal for reproducing shared datasets of Tweets).

The downside is that you need to apply for API keys from Twitter (see Twitter Developers: Getting Started). After you have obtained your keys, provide them to NASTY via the environment variables NASTY_CONSUMER_KEY and NASTY_CONSUMER_SECRET. For convenience, you may use the config.example.sh shell script to do this:

$ cp config.example.sh config.sh
$ # Edit config.sh to contain your consumer key and secret
$ source config.sh

Idify/unidify also support operating on batch results (and keep meta information, that is which Tweets were the results of which requests). To idify batch results in directory out/:

$ nasty idify --in-dir out/ --out-dir out-idified/

To do the reverse:

$ nasty unidify --in-dir out-idified/ --out-dir out/

To fetch all Tweets about "climate change" written after 14 January 2019 in German:

import nasty
from datetime import datetime

tweet_stream = nasty.Search("climate change",
                            until=datetime(2019, 1, 14),
                            lang="de").request()
for tweet in tweet_stream:
    print(tweet.created_at, tweet.text)

Similar functionality is available in the nasty.Replies and nasty.Thread classes. The returned tweet_stream is an Iterable of nasty.Tweets.

The batch functionality is available in the nasty.Batch class. To read the output of a batch execution (for example, from nasty batch) written to directory out/:

import nasty
from pathlib import Path

results = nasty.BatchResults(Path("out/"))
for entry in results:
    print("Tweets that matched query '{}' (completed at {}):"
          .format(entry.request.query, entry.completed_at))
    for tweet in results.tweets(entry):
        print("-", tweet)

A comprehensive Python API documentation is coming in the future. For now, the existing code should be relatively easy to understand.

At the time of writing, the Twitter Terms of Service (TOS) specify the following of relevance to this project:

You may not do any of the following while accessing or using the Services: [...] access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces that are provided by Twitter (and only pursuant to the applicable terms and conditions), unless you have been specifically allowed to do so in a separate agreement with Twitter (NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited)

The text does not detail what separates crawling from scraping but states that obeying the robots.txt is a necessity. These are, for the subdomains we access:

• https://mobile.twitter.com/robots.txt
• https://api.twitter.com/robots.txt

For mobile.twitter.com the URLs NASTY accesses are allowed for any user-agent but require waiting a delay of one second between successive requests. For api.twitter.com accessing any URL is forbidden for any user-agent, except the Googlebot, who may access everything. No crawl delay is specified here. NASTY implements a one second delay between any URL requests (even those to api.twitter.com), but because it does automatically request URLs from the latter subdomain and because it is not the Googlebot, NASTY does technically violate the robots.txt. Therefore, NASTY does violate the Twitter TOS.

This of course begs the question of whether it is morally justified to allow one of the world's most wealthy companies (here, Google) to automatically retrieve all of your web site's user-generated content while simultaneously disallowing anyone else from doing the same thing. Keep in mind, that Twitter is not any web site, but among other things hosts much of the world's political discussion (example) to which, naturally, every citizen should have free and unfiltered access.

Luckily, using NASTY is still perfectly legal in many cases:

• It is unclear (and dependent on jurisdiction) to whom the TOS apply. Since using NASTY does not require signing into Twitter or opening it manually in a web browser, a court may decide that the user never agreed to the TOS and is therefore not bound to its conditions.

• A jurisdiction may guarantee certain rights that can not be overruled by TOS. Especially common are laws that allow to for web scraping in academic and personal contexts.

  For example, in Germany up to 75% of any publicly accessible database (here, Twitter) may copied for academic research. For more details, see Klawonn, T. (2019). "Urheberrechtliche Grenzen des Web Scrapings (Web Scraping under German Copyright Law)". Available at SSRN 3491192.

  Also in the United States, some courts have affirmed the right to scrape publicly available information.

Note, that the above does not imply that it is legal or moral to publicly share a dataset that you created using NASTY. Specifically, the Twitter Developer Policy state:

If you provide Twitter Content to third parties, including downloadable datasets of Twitter Content or an API that returns Twitter Content, you will only distribute or allow download of Tweet IDs, Direct Message IDs, and/or User IDs.

Use the nasty idify command on retrieved Tweets, before sharing them publicly.

Last, it should be mentioned that NASTY is a tool specifically created for personal and academic contexts, where the funds to pay for enterprise access to the Twitter API are usually not available. If you operate in a commercial context, you should pay for the services where possible.

For more discussion on the topic, see Perry Stephenson (2018). "Is it okay to scrape Twitter?"

Please feel free to submit bug reports and pull requests!

There are the Makefile-helpers to run the plethora of auxiliary development tools. See make help for detailed descriptions. The most important commands are:

usage: make <target>

Targets:
  help        Show this help message.
  devinstall  Install the project in editable mode with all test and dev dependencies (in a virtual environment).
  test        Run all tests and report test coverage.
  check       Run linters and perform static type-checking.
  format      Auto format all code.
  publish     Build and check source and binary distributions.
  clean       Remove all created cache/build files, test/coverage reports, and virtual environments.

• Raphael Menges designed the NASTY-bird logo.
• Steffen Jünger and Matthias Wellstein wrote the initial still HTML-based crawler prototype.

Copyright 2019-2020 Lukas Schmelzeisen. Licensed under the Apache License, Version 2.0.

NASTY is a tool/library for retrieving Tweets via the Twitter Web UI. Instead of using the Twitter Developer API it works by acting like a normal web browser accessing Twitter. That is, it sends AJAX requests and parses Twitter's JSON responses. This approach makes it substantially different from the other popular crawlers and allows for the following features:

• Search for Tweets by keyword (and filter by latest/top/photos/videos, date of authorship, and language).
• Retrieve all direct replies to a Tweet.
• Retrieve all Tweets threaded under a Tweet.
• Return fully-hydrated JSON-objects of Tweets that exactly match the extended mode of the developer API
• Operate in batch mode to execute a large set of requests, abort at any time, and rerun both uncompleted and failed requests.
• Transform collected Tweets into sets of Tweet-IDs for publishing datasets. Automatically download full Tweet information from sets of Tweet-IDs.
• Written in tested, linted, and fully type-checked Python code.

Python 3.6, 3.7, 3.8 and PyPy are currently supported. Install via:

$ pip install nasty

Next, you need to place the configuration file in a location where NASTY searches for it. For example:

$ mkdir -p .config
$ curl -o .config/nasty.toml https://raw.githubusercontent.com/lschmelzeisen/nasty/master/config-example.nasty.toml

The places where NASTY looks for the nasty.toml file are in order:

• In a .config sub-directory of the current directory and all subdirectories: ./.config/nasty.toml, ../.config/nasty.toml, etc.
• If the respective environment variables exists in ${XDG_CONFIG_HOME}/nasty.toml and ${XDG_CONFIG_DIRS}/nasty.toml. If not, it defaults to ~/.config/nasty.toml and /etc/xdg/nasty.toml.

That's it. For most operations you won't need to modify the default settings at all.

To get help for the command line interface use the --help option:

$ nasty --help
usage: nasty [-h] [-v] [search|replies|thread|batch|idify|unidify] ...

NASTY Advanced Search Tweet Yielder.

Commands:
  The following commands (and abbreviations) are available, each supporting
  the help option. For example, try out `nasty search --help`.

  <COMMAND>
    search (s)         Retrieve Tweets using the Twitter advanced search.
    replies (r)        Retrieve all directly replying Tweets to a Tweet.
    thread (t)         Retrieve all Tweets threaded under a Tweet.
    batch (b)          Execute previously created batch of requests.
    idify (i, id)      Reduce Tweet-collection to Tweet-IDs (for publishing).
    unidify (u, unid)  Collect full Tweet information from Tweet-IDs (via
                       official Twitter API).

General Arguments:
  -h, --help           Show this help message and exit.
  -v, --version        Show program's version number and exit.
  --log-level <LEVEL>  Logging level (DEBUG, INFO, WARN, ERROR.)

You can also get help for the individual sub commands. For example, try out nasty search --help.

You can search for Tweets about "climate change":

$ nasty search --query "climate change"

NASTY's output are lines of JSON objects, one per retrieved Tweet. Each Tweet-JSON has the following format (pretty-printed and abbreviated for clarity, many other interesting features are also available, such as referenced entities, etc.):

{
    "created_at": "Wed Jan 11 04:52:08 +0000 2017",
    "id_str": "8190441963...",
    "full_text": "Thank you for everything..."
    "retweet_count": 795...,
    "favorite_count": 1744...,
    "reply_count": 22...,
    "lang": "en",
    "user": {
        "id_str": "15367...",
        "name": "Presi...",
        "screen_name": "POTUS...",
        "location": "Washing...",
        "description": "This is an archive...",
        ...
    },
    ...
}

By default this returns TOP Tweets according to Twitter's own ranking rules. Alternatively you can also request the very LATEST Tweets via:

$ nasty search --query "climate change" --filter LATEST

Other possible values for --filter are PHOTOS and VIDEOS.

By default only English Tweets are found. For example, to instead search for German Tweets:

$ nasty search --query "climate change" --lang "de"

Additionally, you can specifically search for Tweets created after and/or before specific dates:

$ nasty search --query "climate change" --since 2019-01-01 --until 2019-01-31

The query parameter supports many more complex operations. For example, exact phrase matches or only returning Tweets from/to specific accounts. To explore what is possible, you can use the Twitter Advanced Search Interface and copy the q-URL parameter from the result page.

For example, if in the advanced search interface we specify climate change for This exact phrase and realDonaldTrump for From these accounts, after clicking Search, Twitter will redirect to the following URL:

https://twitter.com/search?q=%22climate%20change%22%20(from%3ArealDonaldTrump)&src=typed_query

Here, %22climate%20change%22%20(from%3ArealDonaldTrump) is the query string we are interested in. After URL-decoding this, we end up with the following:

$ nasty search --query '"climate change" (from:realDonaldTrump)'

You can fetch all direct replies to the Tweet with ID 332308211321425920:

$ nasty replies --tweet-id 332308211321425920

You can fetch all Tweets threaded under the Tweet with ID 332308211321425920:

$ nasty thread --tweet-id 332308211321425920

NASTY supports appending requests to a batch file instead of executing them immediately, so that they can executed in batch mode later. The benefits of this include being able to track the progress of a large set of requests, aborting at any time, and rerunning both completed and failed requests.

To append a request to a batch file, use the --to-batch argument on any of the above requests, for example:

$ nasty search --query "climate change" --to-batch batch.jsonl

To run all files stored in a jobs file and write the output to directory out/:

$ nasty batch --batch-file batch.jsonl --results-dir out/

When this command finished a tally of successful, skippend, and failed requests is printed. If any request failed, you may retry execution with the same command. Requests that succeeded will automatically be skipped.

The Twitter Developer Policy states that for sharing collected Tweets with others, only Tweet-IDs may be (publicly) distributed (see Legal and Moral Considerations for more information).

To transform lines of Tweet-JSON-objects into lines of Tweet-IDs, use nasty idify. For example:

$ nasty search --query "climate change" | nasty idify > climate-change-tweet-ids.txt

To perform the reverse, that is getting full Tweet information from just Tweet-IDs, use nasty unidify:

$ cat climate-change-tweet-ids.txt | nasty unidify

Note that unidify is implemented using the Twitter Developer API, since for this specific case, the available free API covers all needed functionality and rate-limits are not to limiting. Additionally, this means, that this specific functionality is officially supported by Twitter, meaning the API should be stable over time (thus making it ideal for reproducing shared datasets of Tweets).

The downside is that you need to apply for API keys from Twitter (see Twitter Developers: Getting Started). After you have obtained your keys, provide them to NASTY in the [twitter_api] section of the nasty.toml configuration file.

Idify/unidify also support operating on batch results (and keep meta information, that is which Tweets were the results of which requests). To idify batch results in directory out/:

$ nasty idify --in-dir out/ --out-dir out-idified/

To do the reverse:

$ nasty unidify --in-dir out-idified/ --out-dir out/

To fetch all Tweets about "climate change" written after 14 January 2019 in German:

import nasty
from datetime import datetime

tweet_stream = nasty.Search("climate change",
                            until=datetime(2019, 1, 14),
                            lang="de").request()
for tweet in tweet_stream:
    print(tweet.created_at, tweet.text)
    # Use tweet.to_json() to access the full JSON result dictionary.

Similar functionality is available in the nasty.Replies and nasty.Thread classes. The returned tweet_stream is an Iterable of nasty.Tweets.

The batch functionality is available in the nasty.Batch class. To read the output of a batch execution (for example, from nasty batch) written to directory out/:

import nasty
from pathlib import Path

results = nasty.BatchResults(Path("out/"))
for entry in results:
    print("Tweets that matched query '{}' (completed at {}):"
          .format(entry.request.query, entry.completed_at))
    for tweet in results.tweets(entry):
        print("-", tweet)

A comprehensive Python API documentation is coming in the future. For now, the existing code should be relatively easy to understand.

At the time of writing, the Twitter Terms of Service (TOS) specify the following of relevance to this project:

You may not do any of the following while accessing or using the Services: [...] access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces that are provided by Twitter (and only pursuant to the applicable terms and conditions), unless you have been specifically allowed to do so in a separate agreement with Twitter (NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited)

The text does not detail what separates crawling from scraping but states that obeying the robots.txt is a necessity. These are, for the subdomains we access:

• https://mobile.twitter.com/robots.txt
• https://api.twitter.com/robots.txt

For mobile.twitter.com the URLs NASTY accesses are allowed for any user-agent but require waiting a delay of one second between successive requests. For api.twitter.com accessing any URL is forbidden for any user-agent, except the Googlebot, who may access everything. No crawl delay is specified here. NASTY implements a one second delay between any URL requests (even those to api.twitter.com), but because it does automatically request URLs from the latter subdomain and because it is not the Googlebot, NASTY does technically violate the robots.txt. Therefore, NASTY does violate the Twitter TOS.

This of course begs the question of whether it is morally justified to allow one of the world's most wealthy companies (here, Google) to automatically retrieve all of your web site's user-generated content while simultaneously disallowing anyone else from doing the same thing. Keep in mind, that Twitter is not any web site, but among other things hosts much of the world's political discussion (example) to which, naturally, every citizen should have free and unfiltered access.

Luckily, using NASTY is still perfectly legal in many cases:

• It is unclear (and dependent on jurisdiction) to whom the TOS apply. Since using NASTY does not require signing into Twitter or opening it manually in a web browser, a court may decide that the user never agreed to the TOS and is therefore not bound to its conditions.

• A jurisdiction may guarantee certain rights that can not be overruled by TOS. Especially common are laws that allow to for web scraping in academic and personal contexts.

  For example, in Germany up to 75% of any publicly accessible database (here, Twitter) may copied for academic research. For more details, see Klawonn, T. (2019). "Urheberrechtliche Grenzen des Web Scrapings (Web Scraping under German Copyright Law)". Available at SSRN 3491192.

  Also in the United States, some courts have affirmed the right to scrape publicly available information.

Note, that the above does not imply that it is legal or moral to publicly share a dataset that you created using NASTY. Specifically, the Twitter Developer Policy state:

If you provide Twitter Content to third parties, including downloadable datasets of Twitter Content or an API that returns Twitter Content, you will only distribute or allow download of Tweet IDs, Direct Message IDs, and/or User IDs.

Use the nasty idify command on retrieved Tweets, before sharing them publicly.

Last, it should be mentioned that NASTY is a tool specifically created for personal and academic contexts, where the funds to pay for enterprise access to the Twitter API are usually not available. If you operate in a commercial context, you should pay for the services where possible.

For more discussion on the topic, see Perry Stephenson (2018). "Is it okay to scrape Twitter?"

Please feel free to submit bug reports and pull requests!

There are the Makefile-helpers to run the plethora of auxiliary development tools. See make help for detailed descriptions. The most important commands are:

usage: make <target>

Targets:
  help        Show this help message.
  devinstall  Install the project in editable mode with all test and dev dependencies (in a virtual environment).
  test        Run all tests and report test coverage.
  check       Run linters and perform static type-checking.
  format      Auto format all code.
  publish     Build and check source and binary distributions.
  clean       Remove all created cache/build files, test/coverage reports, and virtual environments.

• Raphael Menges designed the NASTY-bird logo.
• Steffen Jünger and Matthias Wellstein wrote the initial still HTML-based crawler prototype.

Copyright 2019-2020 Lukas Schmelzeisen. Licensed under the Apache License, Version 2.0.