The hotword.py and the color.json should go in the google assistant sdk, which is under ~/assistant-sdk/google-assistant-sdk/googlesamples/assistant/library on the pi.

For supervisor, to make this run correctly make sure your paths to your virtualenv are matching what you are running as well as the directory to hotword.py.