Python BertTokenizer.from_pretrained示例

示例#1

 def get_tokenizer(*args, **kwargs):
     """
     Gets the tokenizer that is customized for this module.
     """
     return BertTokenizer.from_pretrained(
         pretrained_model_name_or_path='bert-wwm-ext-chinese',
         *args,
         **kwargs)

示例#2

文件： module.py 项目： xyxie87/PaddleHub

 def get_tokenizer(*args, **kwargs):
     """
     Gets the tokenizer that is customized for this module.
     """
     return BertTokenizer.from_pretrained(
         pretrained_model_name_or_path='bert-base-multilingual-cased',
         *args,
         **kwargs)