Building video processing engine, to identify the objects in videos and process it in real time.
iOS Development, OpenCV, C++
When Jacob Henderson, CEO and founder of WindowLikr (Boston based startup), approached us he had a very ambitious project to be executed in as little as 2 month’s time. He wanted to get a video processing engine built which would identify the objects in a video and process it in real time to deliver information, statistics and suggestions to the user. The user could click on the objects on the video to see the information.
One of the biggest challenges in building an engine with these requirements, other than the fact that this technology didn't exist, was to identify the technology which would handle the huge amount of data to be processed in real time. Also, the application required for compatibility of different video players across platforms to deliver seamless and rich interactive experience.
The flow which we devised to tackle the core problem of image and video processing involved an iteration based approach. We came up with multiple possible combinations and algorithms for achieving the desired result. We then set about building a POC (Proof of Concept) for the most convincing approach. We then added a lot of constraints to the POC and concentrated in building around most common cases while ignoring edge cases. This speeded up the process and once the POC was satisfactory we included all the edge cases to implement it on a more serious and generic scale. Tools used in the process: OpenCV, C++, Protobuf.
We started with an iPad client application. This application took processed meta-data from the video and made the video interactive and rich. This was achieved using standardised data modelling tools which were highly optimized to perform intensive routines that handled a large amount of data and video synchronizations in real time. Then we built user interface needs, which were built keeping in mind the processing needs of the application along with plug and play based architecture to keep it flexible for different brands, videos and architecture. Technologies used: iOS SDK, Objective-C, Protobuf.