A collaborative computational cooking robot using computer vision built with Raspberry Pi

Python API integrating sensors, actuators and interfaces

See it in action on YouTube | Read more about the project on DesignSpark

How can we apply robotics to home cooking?

Automation tech in the food industry is well known to reduce physical and cognitive demands on production-line operators. Perhaps the same technology could also reduce errors and help decision-making for home cooking? For example, how might robots augment the cooking skills of busy parents and professionals?

The problem: kitchens pose very different design engineering challenges to production lines, because home cooking requires multi-purpose tools, not specialised machines. Robot arms can mimic human-kitchen interaction, but these are currently far too large and expensive to be feasible for the home. For multi-purpose sensing, cameras can detect a wide variety of cooking information, but there are currently no datasets for training cooking image classification algorithms.

Prototyping integration of industry automation techniques and machine vision into a simple robot that fits on a countertop.

API.py Access the OnionBot portal over the local network

camera.py Control the camera using the Picamera module (threaded)

classification.py Classify images with TensorFlow and the Coral Edge TPU (threaded)

cloud.py Upload images to Google Cloud storage buckets (threaded)

config.json Configure settings, labels and models

config.py Interface with the config.json file

control.py Wrapper for PID module, manage control data structures for main.py

data.py Manage data structures and metadata for API

knob.py Wrapper for servo module to control hob temperature setting (threaded)

launcher.py Launch OnionBot software from the big red button

lib_para_360_servo.py Parallax 360 servo drivers

main.py Main script (threaded)

pid.py Proportional Integral Derivative hob temperature controller (threaded)

runlauncher Launch big red button listener script

runonion Launch OnionBot software

thermal_camera.py Wrapper for the Adafruit MLX90640 thermal camera module (threaded)

1. pip3 install pillow
2. pip3 install adafruit-circuitpython-mlx90640
3. pip install Adafruit-Blinka
4. Tensorflow install guide
5. Servo driver install

Check out the paper for technical details in much more depth!

OnionBot: A System for Collaborative Computational Cooking - ArXiv

Get in touch!