IIT Madras US University develops AI-powered algorithm for 3D

3D effects in phone video
Post Date:

Researchers at the Indian Institute of Technology (IIT), Madras and US-based North-western University have developed deep learning algorithms that can significantly improve depth perception and 3D effects in videos shot using smartphone cameras. A significant advantage of the advanced algorithm is that it eliminates the need for fancy equipment or arrays of lenses for capturing intense video. This type of algorithm will prevent mobile phone images from being “flat” and provide a realistic 3D experience.

The intensity and direction of light in a scene, known as the field of light

Mr. Mitra explained that advanced professional cameras capture information about both the intensity and direction of light in a scene known as the “field of light” (LF) to provide a perception of depth.
Although some mid- and high-end smartphone cameras are now programmed to capture such effects in still photographs, especially in portrait mode, it is still not possible to render them in video captured using smartphones, they said.

Bokeh and other such aesthetic 3D effects can be achieved by

Bokeh and other such 3D effects can be achieved through a smartphone equipped with a dual-camera system. In addition to providing depth, our algorithm enables us to watch the same video not just from a single perspective, but from any 7×7 (grid) perspective, – he said.
One of the great benefits of our team’s algorithm is that it eliminates the need for fancy equipment or lens arrays to capture intense video.

Some 3D features like Bokeh effect

In addition to the flat look, there are some 3D features such as “Bokeh Effect” – aesthetic blur of the background – which is easy with DSLR cameras, challenging on smartphone cameras, – K. Mitra, Assistant Professor, Department (Electrical Engineering), IIT Madras told Media.

Artificial Intelligence and Machine Learning Strategies

Multiple micro lenses cannot be installed on mobile phones due to space constraints. Instead, an algorithm is being developed to post-process the image captured by the existing mobile camera. Artificial intelligence and machine learning techniques are used for this type of image manipulation.

LF capture is achieved by using an array of micro lenses inserted between the camera’s main lens and the camera sensor.
Our team has seen this problem and developed an in-depth learning algorithm that converts stereo images captured using a smartphone into LF images, he said.