Python tokenize示例

示例#1

文件： test_tokenizing.py 项目： jernsthausen/datesplitter

    def test_spaces(self) :

        assert tokenize('foo bar') == ['foo', 'bar']
        assert tokenize('foo  bar') == ['foo', 'bar']
        assert tokenize('foo bar ') == ['foo', 'bar']
        assert tokenize(' foo bar') == ['foo', 'bar']

示例#2

文件： test_tokenizing.py 项目： jernsthausen/datesplitter

    def test_split_on_punc(self) :

        assert tokenize('foo,bar') == ['foo,', 'bar']