Query-Aware Adaptive Windowing for Spatiotemporal Complex Video Event Processing for Internet of Multimedia Things

With the evolution of the Internet of Things (IoT), there is an exponential rise in sensor devices that are deployed ubiquitously. Due to the extensive usage of IoT applications in smart cities, smart homes, self-driving cars, and social media, there is enormous growth in multimedia data streams like videos and images. We are now transitioning to an era of the Internet of Multimedia Things (IoMT), where unstructured data like videos are continuously streamed from visual sensors like CCTV cameras and smartphones. Video data is highly expressive and has traditionally been very difficult for a machine to interpret. Middleware systems such as Complex Event Processing (CEP) mine patterns from data streams and send notifications to users in a timely fashion. Current CEP systems have inherent limitations to mine event patterns from video streams due to their unstructured data model, lack of expressive query language, and resource intensiveness. This work introduces VidCEP, a data-driven, distributed, on the fly, near real-time complex event matching framework for video streams with five key contributions:

  • Expressive Video Event Query Language- Current event query languages are highly focused on temporal reasoning. A SQL-like declarative Video Event Query Language (VEQL) is proposed which enables state-based spatiotemporal video event matching without needing to focus on low-level video features. The VEQL enables the creation of robust spatiotemporal operators using a hybrid approach.
  • Structured Video Stream Representation- The work introduces Video Event Knowledge Graph (VEKG), a knowledge graph driven model of video data streams using an ensemble of deep learning models and VEQL operators. VEKG creates a semantic knowledge representation of video data by modeling video objects as nodes and their relationship interaction as edges over time and space.
  • Query and State-aware stream summarization- Objects coexist across multiple frames, leading to the creation of redundant nodes and edges at different time instances that result in high memory usage and increased matching time. The work introduces two novel state and multi-query based spatiotemporal graph summarizations of VEKG streams- Time Aggregated Graph (VEKG-TAG) and Event Aggregated Graph (VEKG-EAG) for an efficient state-based event matching.
  • Resource constraint distributed content-driven windowing- An adaptive and content-driven windowing technique VID-WIN is proposed to improve Quality of Service (QoS). VID-WIN instances are deployed over edge and cloud to perform efficient state-based video event matching. The VID-WIN adopts resource and query-aware runtime optimization strategies to improve the CEP matching performance under limited available resources and application requirements.
  • VidCEP Complex Event Processing Framework: The above techniques are integrated into Video Complex Event Processing framework (VidCEP). A prototype of the framework is developed which performs query-based video event pattern matching in a distributed setting. The VidCEP capability is demonstrated using a real-work traffic estimation service for OpenStreetMap.

A total of 18 event rules are defined from the traffic management and activity recognition domain. Extensive experiments have been performed across 20 datasets consisting of more than 3900 video clips to evaluate the performance and efficacy of proposed techniques. Results of this study show that VidCEP achieves a throughput of approximately 70 frames per second (fps) for five parallel streams (at 17fps) with sub-second matching latency. The system successfully detects different spatiotemporal video event patterns with good F-scores (0.51-0.90). The optimization techniques proposed in the thesis improves ~5X search time, ~2.3X throughput, and ~99% bandwidth savings.


Piyush Yadav

Dr Edward Curry

Institution: NUI Galway


Relevant Publications

[12] Piyush Yadav, Dhaval Salwala, Edward Curry, "VID-WIN: Fast Video Event Matching With Query-Aware Windowing at the Edge for the Internet of Multimedia Things", In IEEE Internet of Things Journal, vol. 8, no. 13, pp. 10367-10389, 2021. [bib] [pdf] [doi]
[11] Piyush Yadav, Dipto Sarkar, Shailesh Deshpande, Edward Curry, "Human Assisted Artificial Intelligence Based Technique to Create Natural Features for OpenStreetMap (Poster)", In Free and Open Source Software for Geospatial (FOSS4G), 2021. [bib]
[10] P. Yadav, D. Sarkar, D. Salwala, E. Curry, "Traffic Prediction Framework for OpenStreetMap Using Deep Learning Based Complex Event Processing and Open Traffic Cameras", In 11th International Conference on Geographic Information Science (GIScience 2021) - Part I, vol. 177, pp. 1-17, 2021. [bib] [pdf]
[9] Piyush Yadav, Dhaval Salwala, Dibya Prakash Das, Edward Curry, "Knowledge Graph Driven Approach to Represent Video Streams for Spatiotemporal Event Pattern Matching in Complex Event Processing", In International Journal of Semantic Computing, vol. 14, no. 03, pp. 423-455, 2020. [bib] [pdf] [doi]
[8] Piyush Yadav, Dibya Prakash Das, Edward Curry, "State Summarization of Video Streams for Spatiotemporal Query Matching in Complex Event Processing", In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), IEEE, Boca Raton, Florida, pp. 81-88, 2019. [bib] [pdf] [doi]
[7] Piyush Yadav, Edward Curry, "VidCEP: Complex Event Processing Framework to Detect Spatiotemporal Patterns in Video Streams", In 2019 IEEE International Conference on Big Data (Big Data), IEEE, Los Angeles, pp. 2513-2522, 2019. [bib] [pdf] [doi]
[6] Piyush Yadav, Shamsuddin Ladha, Shailesh Deshpande, Edward Curry, "Feature Set Consolidation for Object Representation by Parts", In 2019 IEEE International Symposium on Multimedia (ISM), IEEE, San Diego, California, USA,, pp. 124-1243, 2019. [bib] [pdf] [doi]
[5] Piyush Yadav, Edward Curry, "VEKG: Video Event Knowledge Graph to Represent Video Streams for Complex Event Pattern Matching", In 2019 First International Conference on Graph Computing (GC), IEEE, pp. 13-20, 2019. [bib] [pdf] [doi]
[4] Piyush Yadav, Dibya Prakash Das, Edward Curry, "Data-Driven Windows to Accelerate Video Stream Content Extraction in Complex Event Processing", In Proceedings of the 20th International Middleware Conference Demos and Posters - Middleware '19, ACM Press, New York, New York, USA, pp. 15-16, 2019. [bib] [pdf] [doi]
[3] Piyush Yadav, Shamsuddin Ladha, Shailesh Deshpande, Edward Curry, "Computational Model for Urban Growth Using Socioeconomic Latent Parameters", Chapter in The First International Workshop on Urban Reasoning, pp. 65-78, 2019. [bib] [pdf] [doi]
[2] Piyush Yadav, Umair ul Hassan, Souleiman Hasan, Edward Curry, "The Event Crowd", In Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, ACM, New York, NY, USA, pp. 44-53, 2017. [bib] [pdf] [doi]
[1] Piyush Yadav, Souleiman Hasan, Adegboyega Ojo, Edward Curry, "The Role of Open Data in Driving Sustainable Mobility in Nine Smart Cities", In 25th European Conference on Information Systems (ECIS 2017), Guimarães, Portugal, pp. 1248-1263, 2017. [bib] [pdf]
Powered by bibtexbrowser