University of Tokyo Makes Possible Interaction With Virtual Characters In Real World

Researchers from Naemura Lab at the University of Tokyo have developed a new interface that brings mixed reality in the picture. This new kind of interface makes it possible for the user to interact with the virtual characters on screen. Imagine you moving your hand to make the character jump on screen. The mixed reality interface brings the virtual characters to a new reality. This novel kind of interactive experience has made it possible to guide the characters on screen. The researchers shared that, "Recently, devices have been developed that can form images in mid-air. We've utilized one of those, and combined it with sensors and a projector, to provide an intuitive display experience where a picture in the air is skillfully merged with the real world."


Using Kinect, data such as the height of the user's hand and the shape of the block and the system is programmed to automatically move the character to the highest place. This data is used to display the character and its shadow correctly in mid-air, both geometrically and optically, creating a very realistic looking experience. They have used an optical device to determine the image position using the distance between the display and the device. Therefore, changing the image position by changing the display position is possible. In the video shared by DigInfo, we see that they've utilized this feature to make the character in mid-air look as if it's moving backwards and forwards, by connecting the display to an actuator to enable physical movement.

With just a picture in mid-air, it's hard for the human eye to recognize where the character is. So, they have projected a shadow separately. Projecting a shadow makes it easier to sense the picture's depth. Because this system can skillfully merge real objects with mid-air images, they think it could be applied to advertising showcases.

The next time you visit Tokyo, you should be prepared to see advertisement that combing real objects with virtual ones. Check out the following video to see it in action -