An image search tool that utilizes an LSTM (Long Short-Term Memory) neural network to caption images on a file system and then matches an input search phrase with the captions via word vector similarity. Results are ordered by relevance in returned in HTML format.

Jack was my family's cat for the last thirteen years. After a trip to the emergency room on new years eve 2016, Jack was diagnosed with heart failure and had to be put down. I had lots of images of Jack from over the years stored on several different external hard drives, and I wanted a way to search for them. I thought this was a good opportunity to work on a project in his memory.

The app is built on top of TensorFlow and uses its command line interface functionality. The image caption generator that JackSearch uses is from the im2txt model implementation in TensorFlow. A modified version of im2txt is included in this repository which modifies the code to work with the linked model below and Python 3+. The natural language processing is accomplished via spaCy and the GloVe word vectors.

These instructions explain how to install the repo.

• Bazel (Install).

JackSearch has been tested on Python 3.5.2. This installation assumes you already have pip installed and I recommend you install the following in a virtual environment. Using pip, you may install the necessary packages via the requirements file:

pip install -r requirements.txt

Then, you have to install supplemental data for several packages:

• Natural Language Toolkit (NLTK): Download 'all' available data (details).
• SpaCy: Install the parser and GloVE:

python -m spacy.en.download parser
python -m spacy.en.download glove

python main.py --base_dir=/path/to/your/directory \
 --search_phrase="Cats sitting on a bed" \
 --model_file=/path/to/your/model/checkpoint \
 --vocab_file=/path/to/your/vocab/file