Data Eng2025

AMFI ETL Pipeline

An end-to-end ETL pipeline ingesting, transforming and loading AMFI regulatory data into an analytics warehouse, orchestrated with Apache Airflow and surfaced through Power BI dashboards. Built during my Data Engineering internship at Intellect Design Arena.

Regulatory reporting cycle
Automated
Pipeline stages with quality gates
4
Stakeholder-facing dashboards
Power BI

The problem

Regulatory data from the Association of Mutual Funds in India is public, structured, and almost unusable in practice — inconsistent formatting, siloed sources, and no guarantee that today's file looks like yesterday's. Analysts were spending their time on manual preparation rather than analysis, and manual preparation is exactly where silent errors enter a dataset nobody later questions.

The approach

Automate the whole path and make the pipeline assert its own correctness. Airflow orchestrates ingest, transform and load on a schedule; automated data quality checks sit between the stages so a malformed upstream file fails loudly instead of quietly poisoning the warehouse. Power BI dashboards sit on the clean end for stakeholders.

Financial regulators produce vast amounts of structured data that remains trapped in siloed, poorly formatted sources. I wanted to build a pipeline that turns raw AMFI data into a reliable, analysis-ready dataset that financial engineers can actually trust.

Why I built it

Architecture

  1. 01

    Ingest

    Scheduled extraction of AMFI regulatory data from its published sources.

  2. 02

    Quality gates

    Automated checks between stages — a malformed upstream file fails the DAG rather than reaching the warehouse.

  3. 03

    Transform & load

    Normalisation into a structured analytics warehouse in PostgreSQL.

  4. 04

    Reporting

    Power BI dashboards over the clean dataset for stakeholder consumption.

Stack

Orchestration
Apache Airflow
Processing
Python · Pandas · NumPy
Warehouse
PostgreSQL · SQL
  • Python
  • Apache Airflow
  • ETL
  • SQL
  • PostgreSQL
  • Power BI
  • Pandas
  • NumPy

Outcome

Regulatory reporting that runs itself, with a significant reduction in manual data preparation time and quality gates that surface upstream problems instead of hiding them. Shipped as an internship deliverable and used by stakeholders.

NextIoT

IoT system for real-time residential energy monitoring.