Python ReplacementTokenizer Exemples

Exemple #1

def test_tokenizer_errors(content, expected):
    tokenizer = ReplacementTokenizer()
    tokens = tokenizer.tokenize(content)
    assert expected == tokens

Exemple #2

Fichier : test_replacement_tokenizer.py Projet : Coder206/servo

def test_tokenizer_errors(content, expected):
    tokenizer = ReplacementTokenizer()
    tokens = tokenizer.tokenize(content)
    assert expected == tokens