Python SacreMosesTokenizer Examples

Example #1

def test_moses_tokenizer():
    tokenizer = t.SacreMosesTokenizer()
    text = u"Introducing Gluon: An Easy-to-Use Programming Interface for Flexible Deep Learning."
    try:
        ret = tokenizer(text)
    except ImportError:
        warnings.warn("NLTK not installed, skip test_moses_tokenizer().")
        return
    assert isinstance(ret, list)
    assert len(ret) > 0

Example #2

File: test_transforms.py Project: zhengtong0807/gluon-nlp

def test_moses_tokenizer():
    tokenizer = t.SacreMosesTokenizer()
    text = u"Introducing Gluon: An Easy-to-Use Programming Interface for Flexible Deep Learning."
    ret = tokenizer(text)
    assert isinstance(ret, list)
    assert len(ret) > 0