The image2label package implements SVM based image concept recognition, with
- a high-level python interface for loading labels, feature vectors, and SVM models, training and applying models
- linear SVMs and fast intersection kernel SVMs
- model compression that allows new models to be added into an existing model
- cross-platform support (linux, mac, windows)
It has been used in multiple tasks including visual categorization [1], tag relevance learning [2], and object localization [3].
##Prerequisites
- The package does not include any visual feature extractors. Features of training and test data need to be pre-computed, and converted to required binary format using txt2bin.py.
- To minimize one's coding effort, the package requires training data and test data to be organized in a fixed structure, see the sample data.
##Getting started
####Setup
- Add simpleknn and
image2label
toPYTHONPATH
- Change
ROOT_PATH
in basic/constant.py to local folder where training and test data are stored - Download sample data (from PASCAL VOC 2008), using the script download_voc_data.sh or via google drive
- Check if everything can be properly loaded by running predict_and_eval.py
The following code samples show how to learn a linear classifier for a given label, say dog, from voc2008train
, and evaluate the classifier on voc2008val
, with Average Precision (AP) as the performance metric.
####Step 1. Load annotations
from basic.constant import ROOT_PATH
from basic.annotationtable import readAnnotationsFrom
rootpath = ROOT_PATH
trainCollection = 'voc2008train'
trainAnnotationName = 'conceptsvoc2008train.txt'
concept = 'dog'
names,labels = readAnnotationsFrom(trainCollection, trainAnnotationName, concept, skip_0=True, rootpath=rootpath)
name2label = dict(zip(names,labels))
####Step 2. Load feature vectors
from simpleknn.bigfile import BigFile
feature = 'dsift'
train_feat_file = BigFile(os.path.join(rootpath,trainCollection,'FeatureData',feature))
renamed,vectors = train_feat_file.read(names)
Ys = [name2label[x] for x in renamed]
####Step 3. Train model
from fastlinear.liblinear193.python.liblinearutil import train as train_model
svm_params += ' -s 2 -B -1 '
model = train_model(Ys, vectors, svm_params + ' -q')
####Step 4. Compress trained model for fast prediction
from fastlinear.fastlinear import liblinear_to_fastlinear as compress_model
feat_dim = train_feat_file.ndims
new_model = compress_model([model], [1.0], feat_dim, params={})
####Step 5. Prediction
testCollection = 'voc2008val'
testAnnotationName = 'conceptsvoc2008val.txt'
from basic.util import readImageSet
test_imset = readImageSet(testCollection, testCollection, rootpath=rootpath)
test_feat_file = BigFile(os.path.join(rootpath,testCollection,'FeatureData',feature))
test_renamed, test_vectors = test_feat_file.read(test_imset)
ranklist = [(test_renamed[i], new_model.predict(test_vectors[i])) for i in range(len(test_renamed))]
ranklist.sort(key=lambda v:v[1], reverse=True)
####Step 6. Evaluation
names,labels = readAnnotationsFrom(testCollection, testAnnotationName, concept, skip_0=True, rootpath=rootpath)
test_name2label = dict(zip(names,labels))
sorted_labels = [test_name2label[x[0]] for x in ranklist]
from basic.metric import getScorer
scorer = getScorer('AP')
perf = scorer.score(sorted_labels)
See predict_and_eval.py for a complete code to compute AP scores for an array of concept models.
####Save model to file # linear SVMs from fastlinear.fastlinear import fastlinear_save_model as save_model
# fik SVMs
from fiksvm.fiksvm import fiksvm_save_model as save_model
save_model(model_file_name, new_model)
####Load model from file
# linear SVMs
from fastlinear.fastlinear import fastlinear_load_model as load_model
# fik SVMs
from fiksvm.fiksvm import fiksvm_load_model as load_model
model = load_model(model_file_name)
####Add a new (compressed) model to an existing model
# model_1 <- model_1 * w1 + model_2 * w2
model_1.add_fastsvm(model_2, w1, w2)
[1] Xirong Li, Cees G. M. Snoek, Marcel Worring, Dennis Koelma, Arnold W. M. Smeulders, Bootstrapping visual categorization with relevant negatives, IEEE Transactions on Multimedia, 15(4):933-945, 2013
[2] Xirong Li, Cees G. M. Snoek, Classifying tag relevance with relevant positive and negative examples, ACM Multimedia, 2013
[3] Xirong Li, Qin Jin, Shuai Liao, Junwei Liang, Xixi He, Yujia Huo, Weiyu Lan, Bin Xiao, Yanxiong Lu, Jieping Xu: RUC-Tencent at ImageCLEF 2015: Concept Detection, Localization and Sentence Generation, CLEF Working Notes 2015