Python tokenize示例

示例#1

文件： indent.py 项目： bwhmather/codetalker

    def multi_dedent(self):
        text = IndentText('''
one
    level
        another

down
    more''')
        tokens = tuple(tokenize([ID, WHITE, NEWLINE], text))
        self.assertEqual(len(tokens), 20)
        self.assertEqual(tokens[12].__class__, DEDENT)
        self.assertEqual(tokens[13].__class__, DEDENT)

示例#2

文件： indent.py 项目： bwhmather/codetalker

 def tokens(self):
     text = IndentText("hi\n and\n\n more\nstuff")
     tokens = tuple(tokenize([ID, WHITE, NEWLINE], text))
     self.assertEqual(len(tokens), 13)
     self.assertEqual(tokens[2].__class__, INDENT)
     self.assertEqual(tokens[-3].__class__, DEDENT)