Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots
These source code can be used in both iCub simulator and real iCub.
Figure 1: This figure shows an overview of the cross-situational learning scenario as the focus of this study.
The robot obtains multimodal information from multiple sensory-channels in a situation and estimates the relationships between words and sensory-channels.
Figure 2: This figure shows the procedure for obtaining and processing data.
- Ubuntu 14.04
- iCub software and YARP
- ODE 0.13.1 and SDL
- Python 2.7.6 (numpy,scipy,scikit-learn)
- OpenCV 3.1.0
- CNN feature extracter: Caffe (Reference model:Alex-net)
Plase see these sites.
- http://wiki.icub.org/wiki/ICub_Software_Installation
- http://wiki.icub.org/wiki/Linux:Installation_from_binaries
Abstract:
Human infants can acquire word meanings by estimating the relationships among multimodal information and words. In this paper, we propose a novel Bayesian generative model that can form multiple categories based on each sensory-channel and can associate words with any of four sensory-channels (action, position, object, and color). This paper focuses on cross-situational learning using the co-occurrence between words and information of sensory-channels in complex situations. We conducted a learning experiment using a simulator and a real humanoid iCub robot. In the experiments, a human tutor provided a sentence that describes an object of visual attention and an accompanying action to the robot. The experimental results showed that the proposed method was able to estimate the multiple categorizations and to learn the relationships between multiple sensory-channels and words accurately. In addition, we conducted an action generation task and an action description task based on word meanings learned in the cross-situational learning experiment. The experimental results showed the robot could successfully use the word meanings learned by using the proposed method.
Keywords: Bayesian model, cross-situational learning, lexical acquisition, multimodal categorization, symbol grounding, word meaning
Citation: Taniguchi A, Taniguchi T and Cangelosi A (2017) Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots. Front. Neurorobot. 11:66. doi: 10.3389/fnbot.2017.00066
Paper:
https://www.frontiersin.org/articles/10.3389/fnbot.2017.00066/full
Video:
https://youtu.be/SzyoWaj47Xc