Comparing it to a sort of Photoshop for 3D objects, researchers at Carnegie Mellon and the University of California have created a suite of 3D manipulation software tools aimed at letting users select an object within a 2D image and transform it into a 3D model which can then be manipulated.
Taking advantage of libraries of stock photographs and existing 3D models, a complex algorithm compares those images to the two dimensional object to create a 3D model which can be viewed from any angle.
"Our goal in this paper is to allow users to seamlessly perform 3D manipulation of objects in a single consumer photograph with the realism and convenience of Photoshop," says Natasha Kholgade, a co-author of the research. "Instead of simply editing 'what we see' in the photograph, our goal is to manipulate 'what we know' about the scene behind the photograph."
The researchers say that a new era of 'Big Visual Data' means there are huge quantities of images and videos uploaded to the internet daily, and it's "increasingly likely that for most objects in an average user photograph, a stock 3D model will soon be available, if it is not already."
The researchers say their method enables users to perform the full range of 3D manipulations like scaling, rotation, translation, and nonrigid deformations to an object in a photograph by using publicly available 3D models to guide the completion of the geometry hidden in a photograph, and therefore 'fill-in' the appearance of the revealed areas of the object with realistic 3D data.
The paper calls the 3D manipulation of a 2D object sprite "highly underconstrained," and suggests that previously unobserved areas of the object can be created to produce new, 'scene-dependent' shading and shadows to achieve a seamless break from the original photograph and to recreate the scene in 3D using their software's internal representation.
The work by the Carnegie Mellon and University of California team is aimed at recreating the 3D geometry, illumination, and appearance of a given object from a 2D photograph by dealing with several types of mismatch between the photographed object and the stock 3D model.
They say that to perform realistic manipulations in 3D, their software is capable of generating plausible lighting effects like shadows on an object and on contact surfaces by using pixel information in visible parts of the object to correct three sources of mismatch.
As a user "semiautomatically aligns" a stock 3D model to a photograph using a real-time geometry correction interface that preserves symmetries in the object, once the aligned model and photograph are in place, their approach automatically estimates environment illumination and appearance information in hidden parts of the object. They say that while a given photograph and 3D model may not contain all the information needed to precisely recreate the scene, their approach to the problem sufficiently approximates the illumination, geometry, and appearance of the underlying object and scene to produce plausible completion of uncovered areas.
They say it's this ability to manipulate objects in 3D – while maintaining a degree of realism – that "greatly expands the repertoire of creative manipulations that can be performed on a photograph."
"Users are able to quickly perform object-level motions that would be time-consuming or simply impossible in 2D," says the research team. "We tie our approach to standard modeling and animation software to animate objects from a single photograph. In this way, we re-imagine typical Photoshop edits – such as object rotation, translation, rescaling, deformation, and copy-paste – as object manipulations in 3D, and enable users to more directly translate what they envision into what they can create."
More information is available from Carnegie Mellon's site.