GitHub - Glench/nutrimatic: copy over nutrimatic from google code with some changes for x64

Glench / nutrimatic Public

forked from ebenolson/nutrimatic

Notifications You must be signed in to change notification settings
Fork 0
Star 0

copy over nutrimatic from google code with some changes for x64

GPL-3.0, Unknown licenses found

Licenses found

0 stars 1 fork Branches Tags Activity

Star

Notifications

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
nutrimatic-grammar		nutrimatic-grammar
openfst-1.0		openfst-1.0
COPYING		COPYING
LICENSE		LICENSE
README		README
build.py		build.py
cgi-search.py		cgi-search.py
dump-index.cpp		dump-index.cpp
explore-index.cpp		explore-index.cpp
expr-anagram.cpp		expr-anagram.cpp
expr-filter.cpp		expr-filter.cpp
expr-intersect.cpp		expr-intersect.cpp
expr-optimize.cpp		expr-optimize.cpp
expr-parse.cpp		expr-parse.cpp
expr.h		expr.h
find-anagrams.cpp		find-anagrams.cpp
find-expr.cpp		find-expr.cpp
find-phone-words.cpp		find-phone-words.cpp
index-reader.cpp		index-reader.cpp
index-walker.cpp		index-walker.cpp
index-writer.cpp		index-writer.cpp
index.h		index.h
make-index.cpp		make-index.cpp
memoize.py		memoize.py
merge-indexes.cpp		merge-indexes.cpp
remove-markup.cpp		remove-markup.cpp
search-driver.cpp		search-driver.cpp
search-printer.cpp		search-printer.cpp
search.h		search.h
test-expr.cpp		test-expr.cpp

Repository files navigation

This is Nutrimatic (http://nutrimatic.org/usage.html).

To build the source, run "./build.py".  You will need the following installed:
   * Python
   * g++
   * libxml2 (ubuntu: apt-get install libxml2-dev)
   * libtre (ubuntu: apt-get install libtre-dev)

To do anything useful, you will need to build an index from Wikipedia.

1. Download the latest Wikipedia database dump (this is a 13GB file!):
https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2

2. Convert it to a text file (this generates a 9GB file, and takes ~12 hours!):

     bzcat enwiki-latest-pages-articles.xml.bz2 |
     bin/remove-markup > wikipedia.txt

   The remove-markup tool could probably be optimized to run faster...

3. Index the text (this will generate many wikipedia.#####.index files):

     bin/make-index wikipedia < wikipedia.txt

   This also takes a while, but is less likely to have room for optimization.
   This will generate 40GB of data!

4. Merge the indexes; I normally do this in two stages:

     for x in 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016
     do bin/merge-indexes 2 wikipedia.$x*.index wiki-merged.$x.index
     done

     bin/merge-indexes 5 wiki-merged.*.index wiki-merged.index

   Adjust the 016 depending on the number of files make-index output for you
   (or come up with your own way to construct subsets of files for merging).
   The 2 and 5 are minimum phrase frequency cutoffs (how many times a word or
   phrase must occur to be included in a subindex or final mreged index).

5. Enjoy your new index:

     bin/find-expr wiki-merged.index '<aciimnrttu>'

If you want to set up the web interface, write a short shell wrapper that runs
cgi-search.py with arguments pointing it at your binaries and data files, e.g.:

     #!/bin/sh

     exec $HOME/svn/nutrimatic/cgi-search.py \
         $HOME/svn/nutrimatic/bin/find-expr \
         $HOME/svn/nutrimatic/wiki-merged.index

Then arrange for your web server to invoke that shell wrapper as a CGI script.

Have fun,

-- egnor@ofb.net