Life-long learning for visual scene understanding

Christoph Lampert’s project “Life-long learning for visual scene understanding”, funded by the European Commission through an ERC Starting Independent Researchers Grant, has now begun. This project at the interface of computer vision and machine learning aims to give computers the ability to understand visual scenes, i.e. natural images, and to continuously improve their understanding through learning.

Visual scene understanding means that the computer sees an image and understands it at the same level as humans do, recognizing objects in the image as well as understanding what is happening in the scene. Currently, computers require separate algorithms for solving each task in an image. Recognizing cats requires learning a new model from scratch, even if the computer had learned about dogs earlier. At the same time, programs for visual scene understanding are typically not able to improve their performance once their learning phase is completed.If a new task or complexity level is added, a new program has to be developed. The idea behind Christoph Lampert’s project is that a computer can learn continuously. In this way, the computer – over time – builds up a common sense background knowledge for understanding images, just as we humans do when growing up. We know, for example, that a face is highly unlikely to float in the air, not connected to any body. A computer, however, might recognize all the elements of a face, incorrectly flagging up a floating face. With background knowledge developed through continuous learning, such mistakes are avoided. The computer comes to ‘know’ that a floating face hardly makes sense.

Christoph Lampert’s ERC project is in equal parts divided into theory – machine learning – and practice – computer vision. Using methods of machine learning, the theory of what continuous learning actually means will be developed, leading to algorithms implemented in computer vision. Computer vision, in turn, represents the acquired knowledge to the program in such a way that it is usable in future, e.g. allowing the computer to know how big objects usually are, which objects are found inside or outside, which behaviors are common in a certain situation. The ability to continuously and automatically learn from natural scenes without a ‘teacher’ will not be limited to one tasks, but apply to several interconnected tasks within computer vision, doing away with having separate programs for separate objects- cats, dogs, chairs, etc. – or separate activities. As learning occurs in many situation, the idea of continuous, automatic learning by computers is, in the long-run, interdisciplinary, touching on areas of natural learning and cognitive science. Christoph Lampert’s project is supported by an ERC Starting Independent Researchers Grant of 1.5 Mio Euro for the duration of five years, and started in January 2013.