Python split Examples

Example #1

File: test_tokenizer.py Project: 1stvamp/pubbot

 def test_non_alphanum(self):
     self.assertEqual(
         list(tokenizer.split("foo, bar: baz")),
         ["foo,", "bar:", "baz"],
     )

Example #2

File: test_tokenizer.py Project: 1stvamp/pubbot

 def test_simple_sentence(self):
     self.assertEqual(
         list(tokenizer.split("a b c")),
         ["a", "b", "c"]
     )

Example #3

File: test_tokenizer.py Project: 1stvamp/pubbot

 def test_simple_numbers(self):
     self.assertEqual(
         list(tokenizer.split("1.11 1 0.1")),
         ["1.11", "1", "0.1"]
     )