Python tokenizeの例

コード例 #1

    def multi_dedent(self):
        text = IndentText('''
one
    level
        another

down
    more''')
        tokens = tuple(tokenize([ID, WHITE, NEWLINE], text))
        self.assertEqual(len(tokens), 20)
        self.assertEqual(tokens[12].__class__, DEDENT)
        self.assertEqual(tokens[13].__class__, DEDENT)

コード例 #2

ファイル: indent.py プロジェクト: DeadWisdom/codetalker

    def multi_dedent(self):
        text = IndentText('''
one
    level
        another

down
    more''')
        tokens = tuple(tokenize([ID, WHITE, NEWLINE], text))
        self.assertEqual(len(tokens), 20)
        self.assertEqual(tokens[12].__class__, DEDENT)
        self.assertEqual(tokens[13].__class__, DEDENT)

コード例 #3

 def tokens(self):
     text = IndentText("hi\n and\n\n more\nstuff")
     tokens = tuple(tokenize([ID, WHITE, NEWLINE], text))
     self.assertEqual(len(tokens), 13)
     self.assertEqual(tokens[2].__class__, INDENT)
     self.assertEqual(tokens[-3].__class__, DEDENT)

コード例 #4

ファイル: indent.py プロジェクト: DeadWisdom/codetalker

 def tokens(self):
     text = IndentText("hi\n and\n\n more\nstuff")
     tokens = tuple(tokenize([ID, WHITE, NEWLINE], text))
     self.assertEqual(len(tokens), 13)
     self.assertEqual(tokens[2].__class__, INDENT)
     self.assertEqual(tokens[-3].__class__, DEDENT)