Our paper is available here.

We propose the task: Text Classification with Extremely Weak Supervision, which is to classify documents to classes, with the plain class name as the only guidance.
Our method X-Class, breaks up this task into three modules

• Class-oriented Document Representation
  • We estimate both the class representation (based on the given class names), and the document representation (guidede by the class representations).
• Document-Class Alignment
  • We apply Gaussian Mixture Models to align the document representations into clusters. The GMM is initialized with a prior of every document assigned to its nearest class, and therefore, we know which cluster represents which class.
• Text Classifier Training
  • We further select the confident document-class pairs from the previous step, and train a supervised text classifier (e.g. BERT) above it. This pipeline also illustrates our method.

Since many previous weakly supervised text classification problem evaluated on different datasets, we try to collect a unified benchmark for this problem. We try our best to cover different text source (news & reviews), and different types of classes (topics, sentiment, and locations). We present some statistics of them in the table below (Imbalance is the ratio between the largest class size and the smalles class size).

It is worth mentioning that NYT-Topic and NYT-Location is the same underlying corpus, but annotated with two different type of class. We defer some special handling of each dataset (e.g. mapping a class name phrase to a single word) to each specific dataset's folder.

We used python=3.6, torch-1.4.0, cudatoolkit=10.0. Other packages can be installed via pip install -r requirements.txt.

1. Follow the guidance in data/datasets and download the specific dataset.
2. Create an environment, and make sure there is a gpu available. For the final BERT text classifer training step, around 16GB GPU memory is required. Reducing the batch size and sequence length can effectively reduce memory requirement.
3. In the scripts folder, run ./run.sh GPU_INDEX DATASET_NAME, where GPU_INDEX is the index of the free gpu (e.g. 0), and DATASET_NAME is the name of the dataset in data/datasets folder (e.g. agnews).

Our method can be easily applied to new datasets, to do that:

1. Place the new dataset in data/datasets folder, following the instructions.
2. Follow the reproduce steps.

All our intermediate data are stored in data/intermediate_data/DATASET_NAME/, it contains information such as document representations, clusters, etc.

Please cite the following paper if you found our dataset or framework useful. Thanks!

Zihan Wang, Dheeraj Mekala, and Jingbo Shang. "X-Class: Text Classification with Extremely Weak Supervision." arXiv preprint arXiv:2010.12794 (2020).

@misc{wang2020xclass,
      title={X-Class: Text Classification with Extremely Weak Supervision}, 
      author={Zihan Wang and Dheeraj Mekala and Jingbo Shang},
      year={2020},
      eprint={2010.12794},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}