Towards life long learning: Multimodal learning of mnist handwritten digits.

Robots need to be able to continually learn from natural interactions, which are inherently multimodal or multi- sensory (e.g. hearing, vision). Here, we report the development of a deep multimodal autoencoder, capable of unsupervised learning. We created a joint vector space combining labels and images, using a multimodal autoencoder, which maps from images to labels and vice versa. We achieved a label prediction accuracy of 96.9% in the image-only testing condition, where the state-of-the-art is 99.79% and uses a committee of 5 neural networks. In robotics sensory data, is often cheap whilst computation is expensive, thus multimodal systems which require less computation to achieve comparable recognition rates are highly desirable.