Multimedia Information Processing Group


Plenoptic Cameras - Analysis and Simulation

The main aim of the project is to create and understand data from so-called light field or plenoptic cameras with a focus on the different types of plenoptic cameras: Plenoptic 1.0 (Lytro Cameras), Plenoptic 2.0 (Raytrix Cameras as shown below) and Plenoptic Microscopy (FiMic).

Raytrix Cameras







This research project includes several topics (Click on images to enlarge):

Depth estimation: One goal is the development of efficient and high-quality methods to estimate the depth map from plenoptic images as shown in the image below. More information regarding this aspect and source code with images are available at
  plenoptic depth image
Dataset Aquisition: A new dataset was created capturing the same scene under the same conditions with both plenoptic 1.0 (Lytro Illum) and plenoptic 2.0 (Raytrix R29) cameras. More information and data available at:
Simulation: We also aim to simulate the different plenoptic cameras in order to produce ground truth data for the evaluation of algorithms concerning e.g. the calibration or depth estimation. Additional information and Blender models are available at
Image of a scene rendered via our Blender plenoptic camera model.


Project Partners and Funding

European Training Network on Full Parallax Imaging
supported by the Marie Skłodowska-Curie actions under the EU Research and Innovation programme Horizon 2020
Project website:

eu logo

Raytrix GmbH

raytrix logo

Light Field Capturing

In our multimedia lab we are maintaining a large movable multi-camera array for room-scale light field capturing, featuring commodity color cameras and Time-of-Flight cameras for depth acquisition.

The capturing system currently consists of 24 IDS uEye RGB cameras and 2 Kinect v2 RGB-D cameras assembled on a beam to span a horizontal range of approx. 2.5 meters. The beam can be moved by two linear axes within a horizontal range of 25 cm and 2 meters within the vertical direction, using two isel iMC-S8 microstep controllers with very precise positioning. Image capturing with the uEye cameras is synchronized by an external hardware trigger.


Research Topics

The captured RGB-D data can be used as a basis for several applications, e.g., full-parallax imaging, free-viewpoint video and content creation for 3D displays, augmented reality, analysis of lighting and materials, and digital video post-processing.

Current research topics based on the light field capturing system involve amongst others:

  • Calibration of multi-camera capturing systems (intrinsic/extrinsic calibration, hand-eye calibration, color correction, calibration of depth cameras)
  • Multi-modal sensor data fusion (fusion of depth maps, color and depth fusion)
  • Light field representations for dynamic scenes
  • Efficient representations for large-scale light fields (adaptive sparse grids, kd-trees, tensors)
  • Coding, compression, and storage of dynamic light field data
  • Novel view synthesis from dense/sparse light field samples of dynamic scenes
  • Real-time acquisition, processing, and rendering of light field data


Project Partners and Funding

European Training Network on Full Parallax Imaging
supported by the Marie Skłodowska-Curie actions under the EU Research and Innovation programme Horizon 2020
Project website:
Effiziente Rekonstruktion und Darstellung großflächiger dynamischer Lichtfelder
(Efficient reconstruction and representation of large-scale dynamic light fields)
supported by DFG – German Research Foundation (Deutsche Forschungsgemeinschaft)
Parallel On-Line Methods for High Quality Lightfield Acquisition and Reconstruction supported by Intel Labs, Computational Imaging Lab Intel-Logo

Deformation Tracking

Deformation Tracking Header









The deformation tracking project has its focus on the reconstruction of flexible objects from depth and color videos. The Analysis by Synthesis (AbS) method developed in this project is able to meet the different requirements of various applications by its modular approach.










The applications range from real-time tracking of partly occluded objects and Human Computer Interaction (HCI) to grasp evaluation for robots and material parameter estimation based on the visible deflection. An exemplary application of this project is shown in the following video.

Flexpad: Highly Flexible Bending Interactions for Projected Handheld Displays

Project Partners

Rekonstruktion komplexer Deformationen in 3D Szenen aus Bild- und Tiefendaten
(Reconstruction of complex deformations in 3D scenes from image and depth data)
supported by DFG – German Research Foundation (Deutsche Forschungsgemeinschaft)


Seafloor Reconstruction

3D-Modeling of Seafloor Structures from ROV-based Video Data

The goal of this research project is to investigate and develop the necessary adaptations to classic 3D reconstruction methods from the area of computer vision in order to apply them to underwater images. Applications can be found in the areas of Geology and Archaeology. Thus, the project is in collaboration with the Geomar Helmholtz Centre for Ocean Research Kiel, the scientific divers group of Kiel University and the group for maritime and limnic archaeology of Kiel University. In addition to the DFG financing, parts of the project have been financed by the Future Ocean Excellence Cluster. The objective of the cluster is to gain knowledge about a whole range of topics concerning the so far largely unknown deep ocean.

Some the image data to be examined coming for the area of Geology has been captured in great water depths, for example using the ROV Kiel 6000 (Remotely Operated Vehicle) that can reach water depths of 6000 m.
ROV image scaled to 800x600
Equipped with several cameras, one of them being a HDTV camera, it is used to examine black smokers, a type of hydrothermal vent, found for example at the bottom of the Atlantic Ocean (Wikipedia link).
Because of limited diving time during which scientists need to complete a variety of examinations, the task of computer vision is to compute 3D reconstructions of the black smokers. In order to examine and measure the vents after the dive, a 3D model including the absolute scale needs to be determined. The 3D reconstructions are computed with a state-of-the-art structure from motion approach, that has been adapted to the special conditions of the underwater environment.

Special characteristics of the underwater imaging environment in general and the black smokers specifically, that need to be considered include:

  • optical path/refraction causes errors in geometry estimation
  • scattering and absorption of light cause green or blue hue in images and low contrast and therefore impede feature matching, and
  • floating particles, moving animals and smoke violate the rigid scene constraint.

Ozeangrund und Schwebeteilchen. Quelle: IFM-Geomar 

Color Correction

While still traveling through the water, light is attenuated and scattered depending on the distance traveled, causing the typical green or blue hue and low contrast and visibility in underwater images. The Jaffe-McGlamery model can be used to model these effects. Simplifications of the model equations are applied in many color correction algorithms in the literature. Usually, the distance the light traveled through the water needs to be known. After running the SfM algorithm, and after computing the final 3D model, those distances are known. This allows to apply a physics-based, simplified model equation for color correction to the texture image:



Refraction at the underwater housing causes the light rays to change their direction when entering the air within the housing. To be exact, light rays are refracted twice, once when entering the glass and again, when entering the air. In the literature, the perspective pinhole camera model including distortion is used for computing the reconstruction. A calibration below water causes focal length, principal point, and radial distortion parameters to absorb part of the error, hence the perspective calibration can approximate the effects. However, a systematic model error caused by refraction remains due to the single view point model being invalid.
In the image, this can be observed by tracing the rays in water while ignoring refraction (dashed lines) - they do not intersect in the center of projection. It can be easily shown that this model error leads to an accumulating error in pose estimation, when using the perspective model for pose computation.

Therefore, refraction has been modeled explicitly in the whole reconstruction pipeline:

  • calibration of underwater housing glass port, assuming the camera's intrinsics are known
  • Structure-from-Motion algorithm that explicitly models refraction, and
  • dense depth computation using a refractive Plane Sweep method.

The corresponding publications for all three components can be found here.

The complete pipeline allowed for the first time to reconstruct 3D models from multiple images captured by monocular or stereo cameras with explicitly modeled refraction at the underwater housing and the major conclusion was that the systematic model error caused by using the perspective camera model can be eliminated completely by using the proposed refractive reconstruction.

The following figure shows results on real data captured in a tank in a lab. From left to right: exemplary input image, segmented input image, and results for two different camera-glass configurations. Note that the red camera trajectory and point cloud are the result of perspective reconstruction and the blue camera trajectory and point cloud were computed using the proposed refractive method. The result in the right image shows that perspective reconstruction failed, while the refractive method did not.

Input image and resulting camera path and 3D point cloud from an underwater volcano near the Cape Verdes:

In an underwater cave system in Yucatan, Mexico, archaeologists found a skull, which resulted in the following reconstruction:

Archaeology - Automatic 3D Modelling of Excavation Sites

In this collaboration with the archaeology department of Kiel university, a scene reconstruction has been performed based upon two photographs. The images are automatically matched and robustly calibrated providing a sparse set of verified correspondences. Finally a depth map can be produced containing for each pixel the distance of the imaged object from the camera center. This can then be "backprojected" to yield a 3D surface model.

real photooutput




3D Picture


This project is funded by the BMWi within the InnoNet program.


The goal of this project is to manufacture large-scale premium class 3D pictures which can be viewed without visual aids, like e. g. polarized eyeglasses. Therefore it is necessary to establish and refine the individual production steps that are needed for industrial production.


The 3D picture can be thought of as an array of very small slide projectors. In front of the light source is a developed film, exposed with the picture, similar to a reversal film. This picture has a size of 256x256 pixels adding up to a total of 65,536 pixels. On top is a lens system with a diameter of 2 mm to reduce aberration.
These "small projectors" are packed very tight resulting in about 250,000 lens systems per square meter. Depending on the vantage point every lens shows a pixel from the underlying film yielding the overall picture that can be seen. A slightly different vantage point will therefore result in a different picture. This dense light field grants stereo vision in a natural way.


Main tasks of the CAU, among other duties, are:

  • simulation of the 3D picture for one person
  • reconstruction of the light field, sampled by calibrated cameras, for rigid scenes
  • reconstruction of the light field, sampled by uncalibrated video cameras, for rigid scenes
  • development of a renderer to deliver the pictures needed for the film exposition

Project partners

Research partners:

  • Fraunhofer IPM
  • Fraunhofer IPT

Industry partners:

  • RealEyes GmbH
  • AutoPan GmbH & Co
  • Euromediahouse GmbH
  • Meuser Optik GmbH
  • Kleinhempel Ink-Jet-Center GmbH

Associate partners:

  • Viaoptic GmbH
  • Soul Pix

Indoor Mapping

Today there are no simple and cost-effective systems available to map interiors. The purpose of the research is the development of a low-cost solution for automatic mapping of empty or slightly furnished interiors.


3d reconstruction of a slightly furnished room