Algorithms for URL Classification. Will implement LICA, LWCA, DFR etc

For LICA:

>>> import classifier_LICA
>>> classifier = classifier_LICA.LICA()
>>> classifier.classify("http://www.coinweek.com/us-coins/the-marvelous-pogue-family-coin-collection-part-2-the-oliver-jung-1833-half-dime/")
[u'hobbies & interests', u'coins']

For DFR

>>> import classifier_DFR
>>> classifier = classifier_DFR.DFR()
>>> classifier.classify("http://www.coinweek.com/us-coins/the-marvelous-pogue-family-coin-collection-part-2-the-oliver-jung-1833-half-dime/")
(u'hobbies & interests', u'coins')

Options:

• To use the title as well, use a named argument like: classifier.classify("http://domain.com", title="Interesting Coin Website")
• To use DFR rules only (i.e. no overall LICA-style matching for unknown domains), add a rules_only=True argument

• Total documents (i.e. url/title pairs) tested: 1,000,001
• Level refers to returning MozCat level 1 or both 1+2
• The functionality here (in test.py) requires access to our document collection, but can be modified for your dataset without much editing.
• For the dataset column, u = url, t = title, c = content