This repository contains code to automatically create a tagged sense corpus from OpenSubtitles2018. It also contains a lot of corpora wrangling code, most notably code to convert (the CC-NC licensed) EuroSense into a format usable by finn-wsd-eval.

You will need HFST and OMorFi installed globally before beginning. The reason for this is neither are currently PyPI installable. You will also need poetry. You can then run

$ ./install.sh

There is a Makefile. Reading the source is a recommended next step after this README. It has variables for most file paths. These are over ridable with default. You can override them to help make it convenient when supplying intermediate steps/upstream corpora yourself, wanting outputs in a particular place, and when running it with Docker when you may want to use bind mounts to make the aforementioned appear on the host.

You can make the data needed for finn-wsd-eval by running::

make wsd-eval

which will make the STIFF and EuroSense WSD evaluation corpora, including trying to fetch all dependencies. However,

1. It will take a long time. The longest step is building STIFF from scratch which can take around two weeks. To speed things up you can supply a premade stiff.raw.xml.zst downloaded from here (TODO).
2. It will not fetch one dependency with restrictions upon it: BABELWNMAP.

You will next need to set the environment variable BABELWNMAP as the path to a TSV mapping from BabelNet synsets to WordNet synsets. You can either:

1. Obtain the BabelNet indices by following these instructions and dump out the TSV by following the instructions at https://github.com/frankier/babelnet-lookup
2. If you are affiliated with a research institution, I have permission to send you the TSV file, but you must send me a direct communication from your institutional email address. (Please shortly state your position/affiliation and non-commercial research use in the email so there is a record.)
3. Alternatively (subject to the same conditions) if you prefer, I can just send you eurosense.unified.sample.xml eurosense.unified.sample.key

Run::

make corpus-eval

Both the following pipelines first create a corpus tagged in the unified format, which consists of an xml and key file, and then create a directory consisting of the files needed by finn-wsd-eval.

poetry run python scripts/fetch_opensubtitles2018.py cmn-fin

poetry run python scripts/pipeline.py mk-stiff cmn-fin stiff.raw.xml.zst

poetry run python scripts/variants.py proc bilingual-precision-4 stiff.raw.xml.zst stiff.bp4.xml.zst
./stiff2unified.sh stiff.bp4.xml.zst stiff.unified.bp4.xml stiff.unified.bp4.key

You will first need to obtain EuroSense. Since there are some language tagging issues with the original, I currently recommend you use a version I have attempted to fix.

You will next need to set the environment variable BABEL2WN_MAP as the path to a TSV mapping from BabelNet synsets to WordNet synsets. You can either:

1. Obtain the BabelNet indices by following these instructions and dump out the TSV by following the instructions at https://github.com/frankier/babelnet-lookup
2. If you are affiliated with a research institution, I have permission to send you the TSV file, but you must send me a direct communication from your institutional email address. (Please shortly state your position/affiliation and non-commercial research use in the email so there is a record.)
3. Alternatively (subject to the same conditions) if you prefer, I can just send you eurosense.unified.sample.xml eurosense.unified.sample.key

Then run:

poetry run python scripts/pipeline.py eurosense2unified \
  /path/to/eurosense.v1.0.high-precision.xml eurosense.unified.sample.xml \
  eurosense.unified.sample.key

First obtain finn-man-ann.

Then run:

poetry run python scripts/munge.py man-ann-select --source=europarl \
  ../finn-man-ann/ann.xml - \
  | poetry run python scripts/munge.py lemma-to-synset - man-ann-europarl.xml
poetry run python scripts/munge.py man-ann-select --source=OpenSubtitles2018 \
  ../finn-man-ann/ann.xml man-ann-opensubs18.xml

This makes a directory usable by finn-wsd-eval.

poetry run python scripts/pipeline.py unified-to-eval \
  /path/to/stiff-or-eurosense.unified.xml /path/to/stiff-or-eurosense.unified.key \
  stiff-or-eurosense.eval/

TODO: STIFF

poetry run python scripts/filter.py tok-span-dom man-ann-europarl.xml \
  man-ann-europarl.filtered.xml
poetry run python scripts/pipeline.py stiff2unified --eurosense \
  man-ann-europarl.filtered.xml man-ann-europarl.uni.xml man-ann-europarl.uni.key
poetry run python scripts/pipeline.py stiff2unified man-ann-opensubs18.xml \
  man-ann-opensubs18.uni.xml man-ann-opensubs18.key.xml
poetry run python scripts/pipeline.py unified-auto-man-to-evals \
  eurosense.unified.sample.xml man-ann-europarl.uni.xml \
  eurosense.unified.sample.key man-ann-europarl.uni.key eurosense.eval

First process finn-man-ann.

poetry run python scripts/variants.py eval /path/to/stiff.raw.zst stiff-eval-out
poetry run python scripts/eval.py pr-eval --score=tok <(poetry run python scripts/munge.py man-ann-select --source=OpenSubtitles2018 /path/to/finn-man-ann/ann.xml -) stiff-eval-out stiff-eval.csv

poetry run python scripts/munge.py man-ann-select --source=europarl /path/to/finn-man-ann/ann.xml - | poetry run python scripts/munge.py lemma-to-synset - man-ann-europarl.xml
mkdir eurosense-pr
mv /path/to/eurosense/high-precision.xml eurosense-pr/EP.xml
mv /path/to/eurosense/high-coverage.xml eurosense-pr/EC.xml
poetry run python scripts/eval.py pr-eval --score=tok man-ann-europarl.xml eurosense-pr europarl.csv

Warning, plot may be misleading...

poetry run python scripts/eval.py pr-plot stiff-eval.csv europarl.csv

For help using the tools, try running with --help. The main entry points are in scripts.

• scripts/tag.py: Produce an unfiltered STIFF
• scripts/filter.py: Filter STIFF according to various criteria
• scripts/munge.py: Convert between different corpus/stream formats

• scripts/stiff2unified.sh: Convert from STIFF format to the unified format
• scripts/pipeline.py: Various pipelines composing multiple layers of filtering/conversion

The Makefile and Makefile.manann.