Python tokenize Examples

Example #1

File: test_tokenizing.py Project: jernsthausen/datesplitter

    def test_spaces(self) :

        assert tokenize('foo bar') == ['foo', 'bar']
        assert tokenize('foo  bar') == ['foo', 'bar']
        assert tokenize('foo bar ') == ['foo', 'bar']
        assert tokenize(' foo bar') == ['foo', 'bar']

Example #2

File: test_tokenizing.py Project: jernsthausen/datesplitter

    def test_split_on_punc(self) :

        assert tokenize('foo,bar') == ['foo,', 'bar']