Fast VMZ : code enhancements for video mosaicing and summarization

Video Mosaicing and Summarization (VMZ) is a novel image processing pipeline that summarizes the content of a long sequence of geospatial or biomedical videos using a few coverage maps or mini mosaics. The existing VMZ algorithm uses Normalized Cross-Correlation (NCC), Structure Tensor (ST), Affine-Invariant SIFT (ASIFT), Speeded up robust features for its feature matching and homography estimation pipeline, which are the most computationally expensive modules in the VMZ pipeline. Due to these long-running compute-intensive modules, the VMZ pipeline is not suitable for real-time mosaic formation in drones or UAVs. For instance, VMZ takes around 4 hours to generate mini-mosaics from an image sequence containing 9291 image frames. The blending algorithms used for mini-mosaic generation suffer from illumination variation due to the illumination difference in image frames. Such illumination inconsistency causes severe problems for biomedical scene understanding where curvilinear or tiny biological structures are present. VMZ pipeline is also dependent on 3rd party libraries not aligned with the flow of VMZ, which introduces redundant computation. One of the main reasons for the slow processing of the VMZ pipeline is not leveraging any parallel processing techniques and available graphics processing hardware. Therefore, the objective of this thesis is mainly three-fold: (i) speeding up the computeintensive and long-running modules in the VMZ pipeline, (ii) modifying the existing libraries and interfaces for better alignment with VMZ workflow, and (iii) resolving the illumination difference problem of the blending algorithms. Selected longrunning modules with the most impact on the overall run-time have been improved using CPU-based Multi-Threading, GPU-based Parallelization, and better integration with the existing VMZ pipeline. An illumination-matched blending algorithm has been proposed to improve the illumination problem. Besides, to evaluate the performance of different blending algorithms, a novel metric named Maximum Overall Illumination Difference (MOID) has been proposed. The improvement of VMZ modules has resulted in more than 100x speed-up in certain modules, with a 4.4x speed-up for the total VMZ run-time. The novel illumination matched blending resulted in a better MOID value for image sequences not having illumination variance in a single frame.