The goal of this project was to implement a simple Content-Based Image Retrieval system from scratch. The system is based on two feature extractors, one using 2D color histograms and the other based on the VGG16 convolutional neural network. The system is backed by ElasticSearch. The final report is available here.
This project was carried out as part of the DD2476 "Search Engines & Information Retrieval" course at KTH Royal Institute of Technology.
- Antoine Broyelle (github: antoinebrl)
- Andreas Drangel (github: iwut)
- Wojciech Kryściński (github: muggin)
- Safir Najafi (github: SafirN)
src
directory - holds most of the code
feature_extractors
- packages holding all feature extractorsbase_ext.BaseExtractor
- base class for all extractorscnn_ext.CNNExtractor
- convnet histogram extractorhist_ext.HistogramExtractor
- 2D histogram feature extractor
indexes
- package holding all indexesbase_index.BaseIndex
- base class for all indexesmem_index.MemoryIndex
- very simple in-memory index only for small test caseses_index.ESIndex
- elasticsearch based index
parsers
- package holding all image parsersbase_parser.BaseParser
- base class for all parserssimple_parser.SimpleParser
- very simple parser using histograms as features
similarity_mesaures
- packages holding similarity measure functionscosine_similarity
- cosine similarity of two vectorseuclidean_similarity
- euclidean distance between to vectorschisq_similarity
- chi-squared similarity of two distributionskl_similarity
- kullback-leibler divergence of two distributionsbhattacharyya_similarity
- bhattacharyya similarity of two distributionsintersection_similarity
- intersections of two distributions
data
directory - holds small datasets for local testing
small
- small dataset for quick local testsxsmall
- super small dataset, only 4 imgaes for prototypingflickr_25k
- not in repo but should be placed here (locally)flickr_25k/meta/annotations
- not in the repo. Should contains the annotations files provided.
scripts
directory - holds all scripts related to the image retrieval system
index_data.py
- script that runs indexing, requires config in appropriate format (look below for details).
configs
directory - holds index_data.py
configurations
simple_test.cfg
- simple indexing using SimpleParser and MemoryIndex
frontend
directory - holds the React-based frontend for the project. Uses webpack to build JavaScript and package it to a bundle. See the README in the frontend directory for more instructions.
server.py
file - contains a light-weight flask application for receiving HTTP-requests from the frontend.
The index_data.py
script is responsible for indexing images. the script takes as arguments a path to the directory holding the files and a path to the config file.
Example call: python scripts/index_data.py -d data/xsmall -c configs/simple_test.cfg
The config file should be in the following format
[General]
parsers=ParserClassName # this is mandatory, names as found in the `parsers` package (without package prefix)
index=IndexClassName # this is mandatory, names as found in the `indexes` package (without package prefix)
[ParserParams]
key=value # these params will be passed to the class on init, this section can be empty
[IndexParams]
key=value # these params will be passed to the class on init this section can be empty
There are two docker containers necessary in order to run this application.
Elasticsearch is run by first increasing the vm limit, then building and running the docker container
sudo sysctl -w vm.max_map_count=262144
docker-compose build elasticsearch
docker-compose up elasticsearch
The backend server is built and run with the following commands:
docker-compose build server
docker-compose up server
In order to index or debug within the server container you can run bash inside the server docker container:
docker-compose run server bash