Tencent Document

THUOCL dataset

icwb dataset

news and novel dataset

我爱自然语言处理 reference

HMM Framework reference

Before test the project, use the following command to install the required python packages. Note that we are using python 3.7.0 and above.

pip3 install requirement.txt

• Part1: segmentation, e.g.

  'woaibeijingtiananmen' --> ['wo', 'ai', 'bei', 'jing', 'tian', 'an','men']
  

• Part2: tokenization, e.g.

  ['wo', 'ai', 'bei', 'jing', 'tian', 'an','men'] --> ['wo', 'ai', 'beijing', 'tiananmen']
  

• Part3: translation, e.g.

  ['wo', 'ai', 'beijing', 'tiananmen'] --> ['我', '爱', '北京', '天安门']
  
  

• test: in folder part2

  bash train.sh              % train the 2nd part
  
  

  in folder part3

  bash train.sh              % train the 3rd part
  
  

  in folder combination

  bash singlesentence.sh             
  
  % test a single sentence; alternatively, use
  
  python ./test/interpret.py
  
  

  in folder combination

  bash accuracytest.sh                
  
  % test a batch of sentence, to get the accuracy on words / sentences; note that this may cost long time, and some relative path problem may occur on different PCs, so if you need to run the accuracy test, please contact yuzy@shanghaitech.edu.cn. Alternatively, use the following code
  
  python ./test/test.py 
  

  for accuracy test of part1 and part3, please refer to branch 'zheng_dev' and 'main' respectively

• test_dataset

  data used to test

• train_dataset

  data used to train

Through out the experiments, we have the following notations:

Seg: Segmentation, i.e. part 1

Tok: Tokenaization, i.e. part 2

Trs: Translation, i.e. part 3

W: accuracy per word, i.e. # of correct words / # of total words

S: accuracy per sentence, i.e. # of correct sentences / # of total sentences

3-5: a sentence with 3-5 tokens

6-8: a sentence with 3-5 tokens

>=9: a sentence with >=9 tokens

top1: accuracy of the highest scored sentence

top3: accuracy of top 3 highest scored sentences

top5: accuracy of top 5 highest scored sentences