4th place solution for Goodsification task of Data Fusion 2021 Contest.

Multiclass classification. Predict category of item from receipt based on its name and some additional data. There is a lot of unlabeled data (~3m items) and small part of labeled data (~48k items). Item names are very dirty.

• Use only text data (names of items).
• Train tokenizer from scratch on all data.
• Pretrain small custom distilbert from scratch on all data as masked language model.
• Train this distilbert on labeled data.
• Make ensemble (simple average) of 3 such models with different tokenizers (wordpiece, BPE and unigram).

There was 500 mb solution size limit. So training small custom models helps.

File with data data_fusion_train.parquet should be added to data folder.

run_all.sh contains all steps to fully reproduce solution:

• python src/prepare_data.py - prepare data for training language model and training on labeled data.
• python src/train_tokenizers.py - train 3 different tokenizers.
• python src/train_lm.py --config_path=src/configs/train_lm{1,2,3}.yaml - pretrain 3 language models with this tokenizers.
• python src/train.py --config_path=src/configs/train{1,2,3}.yaml - train this models on labeled data.
• python src/compress_models.py - save models without optimizer state, makes them much smaller. Saving without optimizer state during training didn't work as expected.

submit folder contains final submission. copy_to_submit.sh copies all required generated files to it.

Pretraining language models takes a lot of time. Smaller data file item_name_100k.txt can be used for testing purposes.