Python _unescape_token Examples

Programming Language: Python

Namespace/Package Name: official.transformer.utils.tokenizer

Method/Function: _unescape_token

Examples at hotexamples.com: 2

Python _unescape_token - 2 examples found. These are the top rated real world Python examples of official.transformer.utils.tokenizer._unescape_token extracted from open source projects. You can rate examples to help us improve the quality of examples.

Example #1

Show file

  def test_unescape_token(self):
    escaped_token = u"Underline: \\u, Backslash: \\\\, Unicode: \\52;"

    unescaped_token = tokenizer._unescape_token(escaped_token)
    self.assertEqual(
        "Underline: _, Backslash: \\, Unicode: 4", unescaped_token)

Example #2

Show file

File: tokenizer_test.py Project: 812864539/models

  def test_unescape_token(self):
    escaped_token = u"Underline: \\u, Backslash: \\\\, Unicode: \\52;"

    unescaped_token = tokenizer._unescape_token(escaped_token)
    self.assertEqual(
        "Underline: _, Backslash: \\, Unicode: 4", unescaped_token)