Python convert_by_vocab Examples

Example #1

File: run_classifier.py Project: tallamjr/google-research

 def convert_tokens_to_ids(self, tokens):
     return tokenization.convert_by_vocab(
         self.subwork_tokenizer.
         _subtoken_string_to_id,  # pylint: disable = protected-access
         tokens)

Example #2

 def convert_ids_to_tokens(self, ids):
     return convert_by_vocab(self.inv_vocab, ids)

Example #3

File: utils.py Project: julian-pani/bert-text

 def convert_ids_to_tokens(self, ids):
     return tokenization.convert_by_vocab(self.inv_vocab, ids)

Example #4

 def convert_tokens_to_ids(self, tokens):
     return convert_by_vocab(self.vocab, tokens)

Example #5

File: utils.py Project: julian-pani/bert-text

 def convert_tokens_to_ids(self, tokens):
     return tokenization.convert_by_vocab(self.vocab, tokens)