Kafka Realtime Ops Monitor

A real-time order event processing and monitoring platform built using FastAPI, Apache Kafka, PySpark Structured Streaming, MySQL, and Power BI.

The system receives order events through a REST API, publishes them to Kafka, processes them using PySpark Structured Streaming, stores processed data in MySQL, and visualizes operational insights through a Power BI dashboard.

---

Architecture

Client / Application
          |
          v
     REST API (FastAPI)
          |
          v
    Kafka Producer
          |
          v
 Kafka Topic (orders-events)
          |
          v
 PySpark Structured Streaming
          |
    +-----+-----+
    |           |
 Validate    Transform
    |           |
    +-----+-----+
          |
          v
 Aggregate Processed Data
          |
          v
       MySQL Database
          |
          v
    Power BI Dashboard

---

Features

Real-Time Event Processing

• Accepts order events through FastAPI REST endpoints
• Publishes events into Kafka topics
• Processes streaming data using PySpark Structured Streaming

Data Processing

PySpark performs:

• Data validation
• Data transformation
• Aggregation
• Streaming analytics

Data Storage

Processed order events are stored in MySQL for:

• Historical analysis
• Reporting
• Business intelligence

Visualization

Power BI dashboard provides:

• Total orders monitoring
• Order status distribution
• Revenue analysis
• Order trends over time

---

Technology Stack

Component	Technology
API Layer	FastAPI
Messaging	Apache Kafka
Stream Processing	PySpark Structured Streaming
Database	MySQL
Visualization	Power BI
Containerization	Docker

---

Project Flow

1. Order Ingestion

Applications send order events:

POST /orders

Example event:

{
  "order_id": "ORD1001",
  "customer_id": "C101",
  "amount": 250,
  "status": "COMPLETED"
}

---

2. Kafka Processing

The FastAPI service publishes events to:

Topic:
orders-events

---

3. Spark Streaming

PySpark consumes Kafka events and performs:

• Validation
• Transformation
• Aggregation

---

4. MySQL Storage

Processed results are stored in MySQL tables.

Example:

orders
order_summary

---

5. Power BI Dashboard

Power BI connects to MySQL and displays:

• Order volume
• Order status
• Revenue metrics
• Streaming insights

---

Running the Project

Start Infrastructure

Start Kafka and MySQL containers:

docker compose up -d

---

Start FastAPI Service

uvicorn main:app --reload

API runs on:

http://localhost:8000

---

Start PySpark Streaming

spark-submit streaming_job.py

---

Database Configuration

MySQL runs using Docker.

Connection:

Host: localhost
Port: 3307
Database: <database_name>

---

Power BI Dashboard

Power BI connects to MySQL:

MySQL Database
        |
        |
Power BI Desktop

Dashboard contains:

Total Orders

Displays total processed orders.

Order Status Distribution

Donut chart showing:

• Completed
• Pending
• Failed

Order Trends

Line chart showing order activity over time.

Revenue Monitoring

Displays revenue generated from processed events.

---

Repository Structure

kafka_realtime_ops_monitor/

├── fastapi/
│
├── kafka/
│
├── pyspark/
│
├── mysql/
│
├── powerbi/
│   ├── kafka_orders_dashboard.pbix
│   ├── screenshots/
│   └── README.md
│
└── README.md

---

Future Improvements

• Add Kafka consumer lag monitoring
• Add Prometheus + Grafana metrics
• Add real-time Power BI streaming dataset
• Deploy using Kubernetes
• Add alerting for failed events

---

Author

Vignesh Saravanan