This project goal is to have a smart plant that help our fictive target, an old man named Lei, learn to take care of a plant over several weeks.
We opted for an ambient solution that take form of a plant pot. It can listen to the user questions and help him, using visual indication projected on a flowerpot.

We gave a special consideration on the ethic part of this project, especially by making sure this object is not intruding in Lei's life, by reducing request to third parties (and in particular Google's servers) to the strict minimum

A demo (in french) is available at the end of this README

• An hexagonal flowerpot of any size
• A video projector that's has a resolution as low as 360p
• A webcam (to detect movement near the flowerpot)
• A microphone (if not include in the webcam)
• [Optional] Lei

Note The speech recognition work only on Linux, Mac distribution. Because we use pvporcupine

If you are on windows you can only use the video mapping application py_video_mapping

For this project you will need a python3.7 version.

You will need the following package on Ubuntu:

sudo apt install python3
sudo apt install python3-dev python3-tk python-imaging-tk
sudo apt install cmake
sudo apt install python-pyaudio python3-pyaudio
sudo apt install ffmpeg

[Optional] Opencv optimisation lib (in particular to improve performance on embarked hardware)

sudo apt install libxmu-dev libxi-dev libglu1-mesa libglu1-mesa-dev
sudo apt install libjpeg-dev libpng-dev libtiff-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev
sudo apt install libopenblas-dev libatlas-base-dev liblapack-dev gfortran
sudo apt install libhdf5-serial-dev libgtk-3-dev

Install pip and virtualenv:

wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
pip3 install virtualenv virtualenvwrapper

Prepare your virtualenv:

python3 -m virtualenv venv
. venv/bin/activate

If you want to exit your virtualenv:

deactivate

From now on, all commands should be run in the virtualenv

Then install requirements

pip install --upgrade --force-reinstall -r requirements.txt

A speech to text to control an IoT object

On Ubuntu, you need to run:

sudo apt update && sudo apt install portaudio19-dev swig libpulse-dev libasound2-dev

Install pvporcupine for hot word recognition

pip install pvporcupine

You will need pyaudio. If you use a python version > 3.6 install with apt (pip doesn't work for 3.7 python)

sudo apt install python-pyaudio python3-pyaudio

or this if fail

sudo apt install portaudio19-dev
sudo apt install python-all-dev python3-all-dev

To install rasa just run

pip install rasa

First you need to map your video projector with the flowerpot. To do this you need to run this command

python configure_video_mapping.py

When the mapping his done you can test by running the test script.

python test_video_projection.py

We do not use the full capacities of rasa but only the NLU part

To detect witch action do we use Rasa intent recognition. First you need to train the Rasa NLU model

cd speech_to_text/plant_intent_recognizer && rasa train nlu

Then start the server

rasa run --enable-api

To test if all work

curl localhost:5005/model/parse -d '{"text":"hello"}'

Before to run the main script, copy and rename the config.ini.example file to config.ini

Note and remember to replace all the parameters with the text <To fill>

Default microphone values are unlikely to be adapted to your system, please change them.

To help find the right index for input_device_index, you can run detect_mic_to_use
As each microphone is different, noise_level can be found by running detect_mic_sensibility, it will also asset the mic index is working.

Once you are all set, you can run the program:

python main.py config.ini