Python tokenize Examples

Example #1

File: tests.py Project: jasonbot/urlmeme

 def test_tokenize(self):
     self.assertEqual('helloworld', tokenize('hello world'))

Example #2

File: tests.py Project: seanlane/urlmeme

 def test_tokenize(self):
     self.assertEqual('helloworld', tokenize('hello world'))
     self.assertEqual('helloworld', tokenize('HELLOWORLD'))