HDR-ARtiSt

The HDR-ARtiSt platform

This page describes a complete FPGA-based smart camera architecture named HDR-ARtiSt (High Dynamic Range Adaptive Real-time Smart camera) which produces a real-time high dynamic range (HDR) live video stream from multiple captures. A specific memory management unit has been defined to adjust the number of acquisitions to improve HDR quality. This smart camera is built around a standard CMOS image sensor and a Xilinx FPGA. It embeds multiple captures, HDR processing, data display and transfer. The proposed architecture enables a real-time HDR video flow for a full sensor resolution (1.3 Mega pixels) at 60 frames per second.

Two releases of HDR-ARtiSt are currently available. The first release (V1.0) is built around a Xilinx ML507 board, equipped with a Xilinx Virtex-5 XC5VFX70T FPGA and a black and white Ev76c560 image sensor, a 1280 x 1024-pixel CMOS sensor from e2v.

The second release (V2.0) is built around a Xilinx ML605 board, equipped with a Xilinx Virtex-6 XC6VLX240T FPGA and a colored Ev76c560 image sensor from e2v.

Video gallery

1. Overview of the HDR-ARtiSt platform in action

Description: Simple overview video of the HDR-ARtiSt camera capturing a scene and displaying the HDR video on a LCD monitor.

The experimental scene is a poorly illuminated desk on which we can find some elements (coffee boxes, a toy car in a coffee box, a R2-D2 robot, ...). A bright lamp has been placed behind this scene to significantly enhance the dynamic range.

The HDR live video displayed on the LCD monitor shows that the toy car can be detected even the area inside the cup is particularly dark. Similarly, the word ”HDR” written in the lampshade can be easily read.

When the light is switched off (respectively on), the Multiple Exposure Control automatically evaluates the best exposure times for the 3 captured frames in order to maximize the captured dynamic range and then provide the best HDR live video.

Download video 1: 11.5 MB - MP4 format

2. HDR live video

Description: HDR live video obtained from 3 different exposures.

The experimental scene is identical to the one used in Video 1.

The HDR live video is built from the 3 video streams captured with three different exposures (respectively 0.44 ms, 1.08 ms, and 4.81 ms). We can see a lot of details of the scene in the HDR live video: as examples, the word "HDR" and the car in the coffee box in the foreground, the word "HDR" in the lamp, and the European map in the background.

Download HDR video 2: 3 MB - MP4 format

From left to right, low exposure video (0.44 ms), middle exposure video (1.08 ms), and high exposure video (4.81 ms).
Below, HDR live video obtained from the multiple exposures.

3. Multiple Exposure Control (MEC)

Description: HDR live video with fast change of the light conditions .

The set of exposure times is continuously updated from frame to frame in order to instantaneously handle any change of the light conditions.

The MEC algorithm is the following: we require that fewer than 10% of the pixels are saturated at high-level for the short exposure frame. If too many pixels are bright, the exposure time is decreased for the subsequent short exposure captures. Similarly, we require that fewer than 10% pixels are saturated at low-level for the long exposure. If too many pixels are dark, the exposure time is increased.

The first video shows the HDR live video obtained from the 3 streams captured with 3 automatic exposure times. There is always a lot of details clearly visible in the scene whatever the light conditions. You can also note that the MEC algorithm is able to converge rapidly when the light switches on (respectively off).

Download video 3: 8.3 MB - MP4 format

The second video shows simultaneously the 3 streams captured by the sensor: low, middle, and high exposure times from left to right.

When the light is switched off, the different exposure times are close (with similar colors of the background in the 3 streams). On the contrary, when the light is switched on, the different exposure times are clearly separated in order to capture the wide dynamic range of the scene.

Only details in the lampshade are clearly seen in the low exposure video when moving the lamp.

4. Moving hands

Description: HDR live video with moving objects in the scene.

HDR based on multiple captures can have some limitations especially in the context of video. Indeed, it requires that each frame captures exactly the same scene at a pixel level accuracy. Motion in the scene leads to artifacts such as ghosting. One solution is then to implement image registration techniques to compensate for camera and object motion.

In our case, we have not worked on the implementation of such compensation techniques because the artifacts due to motion are limited by the acquisition process of our hardware platform. Indeed, using a sensor operating at 60 fps, the 2-frame and even the 3-frame acquisition process are fast enough to limit the artifacts for moving objects. Moreover, the visual perception of the scene is not significantly disturbed by these artifacts because the HDR-ARtiSt platform is able to produce a real-time 60 fps video flow.

Download video 4: 3.8 MB - MP4 format

5. Moving lamp

Description: HDR color live video with moving objects in the scene.

This is the first HDR color video captured from the release 2.0 of the HDR-ARtiSt platform.

Download video 5: 5.1 MB - MP4 format