A URL shortener service demo using Python 2.7 and Django 1.9 on Google App Engine with a Google Cloud SQL (1st Generation) and Google Datastore backend. The install loads Django 1.9, overriding GAE's Django 1.5.11 supported (as of 3/2016) install.

This is intended as a learning exercise, so it does not use the Google URL Shortener API.

v1.0.1 Supports Django 1.9 on Google App Engine.

v1.0.2 Adds optional database event logging, basic bot/crawler blocking, and optional blacklisting at the city, country, IP, http host, and/or http useragent levels.

v1.0.3 Corrected target page title extraction; other bug fixes.

v1.0.4 More bug fixes.

v1.0.5 Rolled back due to GAE bug; not deployed.

v1.0.6 Added support for writing the event stream to either Google Cloud SQL (MySQL) and/or Google Datastore (NoSQL). Cloud SQL supports a traditional star schema while the Datastore uses a versioned, schemaless structure.

v1.0.7 Bug fixes to doc title extraction for certain complex URLs or URL targets containing certain Unicode values.

v1.0.8 Fix to make A 302 an INFORMATION message in GC Logging, rather than a WARNING.

v1.0.9 Separate botdetection and blacklisting into two separately configurable features; renamed module trafficfilters.py.

v1.0.10 Treat Twitterbot as a special case of allowable bot, and add a setting to enable/disable this special case.

v1.0.11 Add better notification when a long url target website blacklists Google Cloud traffic and/or snakr as a possible bad bot when attempting to get the page title of the long url.

v1.0.12 Add better notification when a long url target website blacklists Google Cloud traffic and/or snakr as a possible bad bot when attempting to get the page title of the long url.

v1.0.13 No thirteenth floor? Then no thirteenth version either.

v1.0.14 Added third party blacklisting service protection. Loads several Palo Alto Networks free daily IP blacklists and scans for requests from IPs on these and if found 403s them. Current PAN lists uploaded are listed below.

v1.0.15 Added cron.yaml support to automatically refresh third party IP blacklists on a preset schedule.

v1.0.16 Whitelist appengine-google http_user_agent

v1.0.17 Whitelisted other Google, Bing, Yahoo, Twitter, Facebook, Apple, DuckDuckGo search crawlers; updated blacklist; removed ALLOW_TWITTERBOT setting (can be controlled via standard whitelisting/blacklisting settings); renamed related settings.

URL shorteners are used to generate a smaller “abbreviated” version of a URL so that the smaller URL can be used as an alias in place of the longer URL. Subsequent calls to the smaller URL will redirect to the same resource originally identified by the longer URL, including all querystring parameters and other valid URL components. This is useful for several reasons:

There are downsides to URL shorteners:

Snakr can create and return a short url in under 2s on smallest-possible GC and GAE settings. I haven't load tested this at volume but assume GC can handle growth.

NEW IN 1.0.6 Events can be optionally logged to Google Cloud SQL (when settings.DATABASE_LOGGING = True and settings.PERSIST_EVENTSTREAM_TO_CLOUDSQL = True) and/or Google Datastore ((when settings.DATABASE_LOGGING = True and settings.PERSIST_EVENTSTREAM_TO_DATASTORE = True). Events are always logged to Google App Engine logging.

Existing Features

1. A basic HTTP POST/GET interface is provided.

2. POST is used to turn a long URL into a shortened URL.

3. A subsequent GET is used to redirect the shortened URL to its long URL target.

4. URLs are cleansed and validate before shortening.

5. Only valid URLs are recongnized (see “Non-goals” #2 below for the definition of “valid”).

6. SSL/TLS is honored on both ends. HTTP long URLs create HTTP short URLs, while HTTPS long URLs create HTTPS short URLs. Ditto FTP and SFTP.

7. Per RFC 3986, schemas and domains/hosts/netlocs are treated as case-insensitive, while the rest of the URL is treated as case-sensitive. For example, three URLs identical in all aspects other than the schemas“http”, “hTtP”, and “HTTP” are all treated as identical and generate identical short URLs for the same .

8. The design and code should support global character sets. Additional testing would be needed here prior to production-ready.

9. Duplicate long URLs submitted to the shortening service will generate only one short URL on the first call. Subsequent calls with the same long URL return the existing short URL.

10. Short URL paths are generated from a random selection of DNS-safe characters (A-Z, a-z-, and 0-9, although the exact set of characters used can be modified via a metadata change).

11. The number of characters used defaults to 6; this is also modifiable up to 12 via a metadata change.

12. Easily-confused character combinations are excluded. No short URLs are generated which contain both:

13. A “0” (zero) and an “O” (uppercase letter ‘oh’), and/or;

14. A “1” (one) and a “l” (lowercase letter ‘ell’).

15. If a collision is detected (a 2nd long url hash to the same short URL, an improbable occurance whose probablity defaults to 1 in 1.886 billion), the short URL is regenerated. This repeats until a unique short URL is generated. The number of max retries is set in the metadata as well, and defaults to 3. Increasing the size of the short URL generated by one in the MAXSIZE parameter in the metadata greatly increases the number of possible combinations at the expense of a short URL one character longer. If MAXSIZE is increased, old URLs using the smaller size still work, and you can even flip back to the old size later and both sizes will still resolve.

16. Basic bot and crawler blocking is available. A blacklist of known bot user agent strings is provided. Any request coming from a user agent containing one of the strings in the blacklist will be 403d.

1. Django 1.5.11's SuspiciousOperation returns HTTP 500, not HTTP 400 (see http://stackoverflow.com/questions/35439621/django-suspiciousoperation-returns-as-http-500-on-google-app-engine-not-http-40). This means among other things that a badly formatted URL crashes WSGI, which is NOT good. This is my top priority to fix by applying the workaround mentioned in the StackOverflow page to run under Django 1.9.2 or latest possible.

   UPDATE: v1.0.0 I worked around this by implementing a custom /django/core/handlers/base.py that includes Django 1.9's SuspiciousOperation exception block with some minor changes. Unknown short URLs etc. now return HTPP 400 with a meaningful error message.

   UPDATE: v1.0.1 Fixed by implementing Django 1.9

2. Request filtering is not yet robust. Some non-supported requests will return HTTP 500 due to the previous issue.

   UPDATE: v1.0.0 Much improved. Not fully tested yet. Haven't gone through all of the use cases.

   UPDATE: v1.0.1 Fixed by implementing Django 1.9 and a few bug fixes

3. There is no admin page or reporting. Will add that.

4. Geolocation detection is not working (see http://stackoverflow.com/questions/35492617/x-appengine-citylatlong-not-populated-on-google-app-engine-django-1-5-11-when-us). Will workaround or fix.

   UPDATE: v1.0.0 Fixed

5. Other diagnostic/tracking info to be added to snakr_log.

   UPDATE: v1.0.0 Added storage for some of these. Looks like I'm hinting that I'm adding a robust demo fraud and abuse detection system later.

   UPDATE: v1.0.1 Logging now removed from the db and placed into json logging files.

6. snakr_log collects IPs with no encryption. This may not be legit for your applicable privacy and/or compliance needs. No other PII is collected. Check with your legal department or ex-spouse for all bad news.

   UPDATE: v1.0.0 I encode these now into binary(128). This is for minimizing storage, NOT for encryption. They are NOT encrypted and can be reversed into the original readable IPs.

   UPDATE: v1.0.1 Logging now removed from the db and placed into json logging files.

7. Local testing with a local MySQL db was not tested or used with this.

   UPDATE: v1.0.1 Local dev and QA now supported using local dev_appserver.py against either local MySQL or remote GCS.

8. The GAE Development Environ was not tested or used with this.

   UPDATE: v1.0.1 Local dev and QA now supported using local dev_appserver.py against either local MySQL or remote GCS.

9. No test units are yet defined. It's a pretty simple app functionally; I'm not so sure this is really warranted.

10. No GAE endpoints for mobile are coded or supported. I may or may not add that. Feel free to do so.

11. Submission of a long url via a POST with a query string (e.g. "http://snakr.appspot.com/?u=http://www.shortenthisurlplease.com") may be added. Feel free to do so.

UPDATE: v1.0.0 I'm going to add this shortly, no pun intended. Some code already implemented for this.

UPDATE: v1.0.1 CANCELLED: this makes the app vulnerable to certain attacks when the GET is redirected into a PUT. Not implementing.

12. No aging or cleanup of URLs based on usage age-off or other rules is not provided. I did add a is_active status to support this in the future.

13. No other security or access features such as OAuth access control are provided. Access is controlled solely via the Google App Engine Authentication features.

14. Server 500 and other HTTP errors generated by the service are logged in the available Google Cloud logs which can be accessed via GAE Dashboard and if you are using it the Pycharm IDE. This catches a lot of bugs. No other syslogs or other logging access or monitoring is provided.

15. You can buy a short domain name and have it CNAME to yourdomain.appspot.com and then change the settings.py to prepend your short URLs with the short domain. To support secure domain redirection, your short domain will need an SSL certificate and secure redirection enabled on your domain provider. Alternatively, you can use your short domain for short HTTP urls and yorudomain.appspot.com for short HTTPS urls. The relevant settings in settings.py are:

    # host (netloc) of the short URL to use for HTTP urls
    SHORTURL_HOST           = "bret.guru"  #change to your short domain
    
    # host (netloc) of the short URL to use for HTTPS urls
    SECURE_SHORTURL_HOST    = "snakr.appspot.com"
    

    This produces short URLs:

    http://bret.guru/xxxxxx
    https://snakr.appspot.com/xxxxxx
    

    Once you have your SSL certficate and secure redirection turned on, you can use the same short domain on both:

    SHORTURL_HOST           = "bret.guru"   
    SECURE_SHORTURL_HOST    = "bret.guru"
    

    producing:

    http://bret.guru/xxxxxx
    https://bret.guru/xxxxxx
    

    16. A mobile app/endpoints would be neat.

Why do you need blacklists, you ask?

I use this app on Twitter and other social media, and well over half my hits were coming from non-human traffic. Sure, it drives up your impression counts, but also your costs for using this URL shortener. Personally, I'd rather impress people (pun intended) than software.

If enabled, Snakr can also filter traffic out based on third party blacklists from Palo Alto Networks. These blacklists are reloaded at app start and if enabled in cron on a regularly scheduled update (daily is recommended).

Currently, this includes (from https://panwdbl.appspot.com/):

These files can be set to automatically reload via a GC cron job specified in your cron.yaml (see GC cron help for Python for more info). Pick a long, pseudorandom URL string and append it to your root domain, e.g.:

http://yoursnakrdomain.appspot.com/this_should_be_a_really_long_and_fairly_random_url_to_avoid_malicious_DoS_attacks_from_hitting_it

Then in your settings.py, add the THIRDPARTY_IP_BLACKLIST_CRONJOB_URL setting and point to it as follows:

# note the leading slash
THIRDPARTY_IP_BLACKLIST_CRONJOB_URL = "/this_should_be_a_really_long_and_fairly_random_url_to_avoid_malicious_DoS_attacks_from_hitting_it"

The upload a cron.yaml like the one below to run once a day using the instructions on the Google Cloud cron job reference page:

cron:
- description: reload 3rd party blacklists daily
  url: /this_should_be_a_really_long_and_fairly_random_url_to_avoid_malicious_DoS_attacks_from_hitting_it
  schedule: every day 06:00

Now your 3rd party lists will be reloaded automatically every day at 6:00AM in your selected GC timezone. Be sure you select a refresh time by when the third party has actually written new files to their remote URL locations for you to pick up.

Again, it's imperative that you use a really long randomized URL to avoid bots and other bad actors from guessing it and hitting it over and over.

1. Install curl or another REST client testing tool like Advanced Rest Client Application for Chrome

2. Using your REST testing tool, test the POST action:

   URL: http://bret.guru
   POST:
       Content-Type: application/json
       Payload:      {"u":"<the url you want to shorten goes here without the angle brackets>"}
   

3. Using your browser, curl or the tool, test the GET action:

   URL: <the short url generated by the POST step above>
   

1. Install the App Engine Python SDK. See the README file for directions. You'll need python 2.7 and pip 1.4 or later installed too.

2. Clone this repo with

   git clone https://github.com/bretlowery/snakr.git
   

3. Install dependencies in the project's lib directory. Note: App Engine can only import libraries from inside your project directory.

   cd snakr
   pip install -r requirements-local.txt -t lib
   pip install -r requirements-vendor.txt -t lib
   

4. Create your GAE and Cloud SQL instances.

5. Upload from your Python IDE to GAE. I use Pycharm as it has Google Cloud and GAE integration.

6. Create an instance in Cloud SQL to hold the database. You do not need to pre-create anything in Google Datastore.

7. Create a user account in SQL to install the database such as "dbo" with a password. Do NOT use the sql "root" login and do NOT change its existing password. GAE will use root@localhost to connect to Cloud SQL on the back end.

8. In your browser, visit WhatIsMyIP.com. Get your local IP address, and over in the Cloud SQL admin page, grant your IP access to your SQL instance. This allows you to connect from your local machine. If your local IP changes, you will not be able to connect from your local MySQL Workbench client until you grant your new IP acesss on the Cloud SQL admin page.

9. Install MySQL Workbench and connect to Cloud SQL using: *Host = the Coud SQL IP (not your local one, the one of the Cloud SQL instance *Port = the default 3306 MySQL port *Username = "dbo", or the username you created *Password = the password you entered *Database/Scheme = the db/scheme name you entered

10. In MySQL Workbench, run install.sql to create the db objects.

11. In Settings.py, add your personal GAE config info, db connection info for your GAE Cloud SQL instance, etc. Adjust these parameters accordingly:

Snakr persists its data in Google Cloud SQL in this version; later versions will use other storage options on Google Cloud such as Datastore or Memcached. This initial design uses three Cloud SQL tables.

The original design had both long and short urls in a single table. I split it in two thinking about future features such as multiple short URLs for a single long URL (useful in certain business-oriented use cases).

####Table snakr_longurl One row per long URL successfully submitted for shortening to Snakr.

####Table snakr_shorturl One row per short URL successfully generated by Snakr.

Snakr uses standard Django logging which appears in the Google Cloud Logging console and can be downoaded via the Google Logging API.

In addition to standard request and state logging, Snakr also logs the following application-specific data elements to the log in JSON format, and to the snakrdb database if settings.DATABASE_LOGGING = True AND settings.PERSIST_EVENTSTREAM_TO_CLOUDSQL = True:

If database logging is enabled, this information is stored in table snakr_eventlog and a set of associated dimension tables. The values above are stored in the tables below.

One row per Snakr-submitted event.

One row per city of traffic origination in any request to Snakr.

One row per country of traffic origination in any request to Snakr.

One row per unique HTTP_HOST in any request to Snakr.

One row per unique absolute IPv4 or IPv6 address in any request to Snakr.

One row per unique HTTP_USER_AGENT in any request to Snakr.

One row per unique third party IP blacklist file ingested by Snakr.

See the Third party libraries page for libraries that are already included in the SDK. To include SDK libraries, add them in your app.yaml file. Other than libraries included in the SDK, only pure python libraries may be added to an App Engine project.

Bret Lowery