Chinese Menu Reader is a mobile application that takes a picture of a dish name on a Chinese menu and gives you an English translation of the dish name, a description, reviews, photos and tags.

This repository contains the code for the Python server. For the iOS application, check out this repository.

The iOS application takes the image, crops it, and uploads it to the Python server. The Python server does some basic image processing (smoothing, converting to grayscale, binarizing/thresholding) and then uses Google Tesseract OCR to get a string of Chinese characters from the image. If Tesseract fails to recognize any characters in the image, the server thins the image (using Stentiford's preprocessing steps and Scikit-Image's skeletonize() function) and runs it through Tesseract again.

If Tesseract returns characters, the server looks them up first in a dishes table. If a dish is found, it returns a JSON object representing that dish. If not, it attempts to translate the characters relying first on a list of common food words and finally on CEDICT, an open source Chinese dictionary maintained by MDBG.

1. Clone the repository:

   $ git clone https://github.com/ejanzer/menureader.git

2. Install Google Tesseract OCR.

3. Add language training data to your tessdata directory (usually at /usr/local/share/tessdata - if not, just set your TESSDATA_PREFIX environment variable to the parent directory of the tessdata directory):

   • Simplified characters
   • English – for testing purposes
   • Traditional characters (Note: to use traditional characters, you'll need to also set LANG to 'chi_tra' in config.py.)

4. Make sure Tesseract is installed properly by running pytesser on the test images:

   $ python tesseract/pytesser.py

5. Install Python packages:

   $ pip install -r requirements.txt

6. Run the app:

   $ python app.py

7. Deploy the app or use ngrok to forward your port:

   $ ./ngrok 5000

8. In the iOS Xcode project, change the server URL to the server's address (see instructions on iOS repo for details).

For best results, take a picture with your camera about 6 inches away from the menu, then zoom and crop around the name of the dish you want to translate. Try to hold the menu straight and avoid blur if possible.

Tesseract has trouble recognizing some Chinese characters that have vertical space between two halves or radicals. I haven't yet figured out how to address this problem, so the app will have trouble recognizing dish names with these characters, especially if the font accentuates these spaces.