Python _join_tokens_to_string Examples

Example #1

  def test_join_tokens_to_string(self):
    tokens = ["test", "? ", "testing", "123", "."]

    s = tokenizer._join_tokens_to_string(tokens)
    self.assertEqual("test? testing 123.", s)

Example #2

File: tokenizer_test.py Project: 812864539/models

  def test_join_tokens_to_string(self):
    tokens = ["test", "? ", "testing", "123", "."]

    s = tokenizer._join_tokens_to_string(tokens)
    self.assertEqual("test? testing 123.", s)