Chatterbox is a multimodal networked chat interface with three core features:
- Emoji suggestions based on emotion classification from webcam data
- Hands-free scrolling using gaze tracking from webcam data
- Voice-to-text functionality
| Filename | Description |
|---|---|
chat.js |
Modifies index.html depending on the state of the application. Combines speech.js, scroll.js, and logging.js into the interface. |
chatServer.js |
Starts and runs a Node.js server using Express. |
emojiBank.js |
Contains a dictionary used to suggest emojis based on the emotion classification. |
index.html |
Initial layout of the interface. Modified using chat.js |
logging.js |
Logs desired mouse and keyboard events to a Google spreadsheet. The data was then processed after user testing. |
package.json |
Used by npm to keep track of dependencies. |
scroll.js |
Triggers scrolling up or down when gaze tracking is turned on and a cutoff is reached at the top or bottom of the screen. |
speech.js |
Processes speech using webkitSpeechRecognition API |
style.css |
Provides styling to the interface. |
WebGazer.js |
Sourced from https://webgazer.cs.brown.edu/ and used to implement gaze tracking. |
python_chat/ |
Python 3 code from initial implementation. Filenames are similar to those described above. |
sounds/ |
Sounds used to provide auditory feedback |
- Install Node.js
- Navigate to directory
6835-project - Run
npm install
- Run
npm start - Open a browser and go to
localhost:3000
Or go to https://agile-citadel-17828.herokuapp.com/.
Runs on Chrome, Firefox, and Edge. Tested on Windows, Linux, and Mac. Works best in a well-lit area, with the laptop on a desk or table so that the webcam is closer to eye level than it would be from a user's lap.
From initial implementation. Not improved past Prototype Studio.
- Navigate to directory
python_chat - Run the server:
python3 chat_server.py - Open a GUI:
python3 chat_gui.py