Azure Data-Ingestion Automated Notification

Team

Mangesh Bhattacharya, Samveg Thapa, Vinnistan Manoharan

Overview

One of the sponsor's clients uses Enterprise's modern cloud-based data warehouse system, which daily generates terabytes of data stored in various servers at multiple locations in Canada. This data is streamed to a data lake (Azure Cloud) in batches through data ingestion pipelines. The data is extracted daily for Management Information purposes (MIS) and Executive Report creation. A glitch in the data ingestion pipeline, either from servers to Cloud OR Cloud to Management Information Systems (MIS), can lead to inadequate information. This can further result in incorrect Management reporting and improper Management decision-making. Currently, the client runs 24 pipelines which are monitored manually around the clock and resolved if any issues occur. Since the pipelines are monitored manually, the shift Engineer has to review each pipeline in order to verify that the pipelines are in proper health. There are chances of delays in data delivery due to human error. The pipelines are manually triggered and inter-dependent. The project goal was to create Auto-alerts/notifications as soon as a glitch/failure in the ingestion pipelines, during data streaming, arose. These alerts/notifications will trigger different levels of IT Support to dispatch qualified personnel to attend & resolve the issues, which will shorten the resolution timelines. Achievements: In this project, all involved Cloud services are created and implemented as infrastructure-as-a-code (so that they can be reused for any number of clients/environments). Successfully implemented and tested email notifications, pinpointing the exact pipeline failures. Pipeline automation has generated huge savings due to less manpower involvement required to regularly monitor all data pipelines. Also, it has helped to improve the timely data delivery resulting in accurate Management reporting.