The following video clips demonstrate image processing operations performed completely on the pixel-level within the KOVA1 chip. No image processing operations were performed on the on-board FPGA or on a computer, only the processed video data was read out of the sensor array.
Real-time input dynamic range compression
With the KOVA1 pixel-level analog processing circuitry, photodiode sensor signals can be processed and evaluated even during the image integration time. This enables the implementation of image dynamic range compression in real-time, without any additional delay from e.g. multi-frame analysis or adaptation.
In the dynamic range adjustment process, the pixel integration times are limited locally if the area seen by given pixel includes very bright objects which could saturate the sensor output. In this way, all of the relevant detail in both bright and darker regions is retained. The compressive imaging mode can be applied in many situations and applications, where the illumination cannot be controlled or there input image dynamics are inherently very high.
The top video shows a CD reflecting bright light, captured with a normal, global integration time mode (left) and with the real-time dynamic compression applied (right). Both video streams were read out at the same time from the KOVA1 chip, with a 1 frame difference between the normal and compressed images. The middle clip shows a scene looking both inside and out of a window, which was illuminated with direct bright sunlight.
The bottom clip demonstrates the application of the dynamic compression mode for the imaging of an electrode welding process, with only regular ambient lighting, i.e. no additional illumination. As can be seen, the surface formations of the metal are also clearly visible, while the sensor is not saturared by the bright arc discharge and molten metal light emission. The image on the right side shows the result of a pixel-level edge detection operation.
Object segmentation and tracking with pixel-level processing
A fundamental part of the KOVA1 sensor-level processing is the segmentation of image data, which means the extraction or separation of image objects based on their spatial features or morphology. While the purpose of the grayscale front-end processing is to remove as much redundant image information as possible while retaining and emphasizing salient features, shape-based object analysis can be most efficiently performed in the 1-bit (black-and-white) image domain.
The programmable extraction and tracking of objects with 1-bit processing, based on their morphological properties, can be used to trigger the chip readout only to the presence of the desired visual objects. The image based triggering could also be used to capture a higher resolution image (or high-speed video) of the desired object with another camera controlled by the KOVA1 chip.
A disc with different drawn features and shapes was spinning at 3-4 revolutions per second (top-left clip). In the top-right video, chip readout was enabled only when the letter "A" was present in the view of the camera. In the middle clip, the holed square object was extracted and its path was tracked as it moved across the view. The image data was captured and processed at approximately 1000 fr/s.
The bottom clip illustrates a tracking operation, where the marker in the middle of the screen should stay within the black path. If the marker moves out of the path, the vertical direction where the marker should be moved for correcting the location is indicated with black. Very simple output data, which can be obtained with extremely low lag, enables very efficient actuator control. In this unoptimized example, the processing delay was approximately 120 μs.
Spatiotemporal tracking of visual data
Multiple image frames can be stored on the KOVA1 sensor plane, either in grayscale or binary (1-bit segmented data) form. This means that time-domain (multi-frame) analysis can also be performed on the pixel-level. Temporal events at any time-scales permitted by illumination and sensor sensitivity can be processed without any off-chip image readout and storage of frame data outside of the sensor array.
Both the spatial and temporal properties of visual data, as well as the results of previous analysis steps, can be used to control the execution of subsequent analysis steps. This can mean e.g. limiting the application of processing or data readout only to some parts of the image, indicated by some other image operation. The programs can also contain conditional branches and jumps, based on previous results obtained from the processed input data.
The top video demonstrates simple multiframe tracking of object movement (tip of the pen). The diminishing "tail" trailing the pen movement shows the position of the tip in the two previous frames. The middle clips demonstrate a combination of spatial and temporal analysis, again performed completely on the sensor plane. The card with the letters "L" and "R" was waved rapidly and either L or R was highlighted, depending on the movement direction of the card. In the middle-right clip, image readout was triggered only by detected movement to the left.
In the bottom-left video the hole in the paper sheet was extracted and the an analog-domain smoothing operation was performed only within the selected area. In the bottom-right clip, hand movement was tracked and used to "paint" areas where a result of an edge-detection operation was read out of the chip instead of grayscale image data.
Operation at all time scales
The KOVA1 chip can operate over a very large scale of temporal imaging rates. The on-chip analysis and only-on-trigger readout allow high-speed continuous tracking of visual data which, unlike conventional high-speed cameras, is not limited by the high data storage requirements. However, also extremely slow temporal events can be tracked, while still retaining all processing and image storage on the sensor level. This is made possible by static storage of frame data on the KOVA1 pixel-cell level. Even full grayscale images can be indefinitely (untill poweroff) stored on the sensor plane, thanks to a sensor-level A/D/A conversion loop.
The top-left video shows a scene recorded at 1kfr/s. Continuous full image data readout limits the frame-rate of the KOVA1, compared to e.g. a trigger-mode operation or only limited-data readout. However, relatively fast full-frame video recording is still possible for visualization or algorithm development purposes. The chip I/O limits the readout of grayscale image data to approximately 2000 fr/s, while 1-bit images (e.g. segmented images) can be recorded at > 10 kfr/s.
The top-right clip illustrates the readout flexibility enabled by on-chip processing. The scene was continuouly tracked for any movement, i.e. a new frame was captured always when a sufficient change was detected in the scene, even if that change occurred over a very long time. This allows the same algorithm to track both the minute hand of the clock and the falling tennis ball. The bottom clip illustrates this with the 1-bit image showing the perceived change that was used to trigger a new frame readout.