Kafka, Spark Streaming, Spark SQL, Javascript project

Qubole Delta Lake Spark Streaming ingestion end to end Demo

A demo is about data pipeline for Customer Review Project.

Propose a reliable data pipeline solution to capture high-velocity stream of patient vitals such as body temperature, heartbeat, blood pressure (BP) coming from IoT devices and send an instant email notification incase of abnormal vitals.

real-time data pipeline on generated access log data

In this project I stream data and do crime classification using Spark. This dataset contains incidents derived from the SFPD Crime Incident Reporting system. The data ranges from 1/1/2003 to 5/13/2015. I do some data analysis of crime scenes in different areas and with respect to other parameters.

Implemented the following framework using Apache Spark Streaming, Kafka, Elastic, and Kibana. The framework performs SENTIMENT analysis of hash tags in twitter data in real-time. For example, we want to do the sentiment analysis for all the tweets for #trump, #coronavirus.

Basic sentiment analysis of real-time tweets with the help of Spark’s streaming APIs

A Real-Time Tweet Streaming Pipeline (Using Flume, Kafka & Spark-Streaming) with Deep Learning Sentiment Analysis Model for instant scoring.

Spark Streaming using Flume (pushed based Approach)

Apache Spark Streaming application which reads web log data and monitors for 404 errors.

Analyzing real-time data using Apache Kafka & Spark Streaming

Engineered a data pipeline on GCP for a mock game development company, to track player activity in guilds and in-game purchases, using Docker and streaming events from a Flask app through Kafka, PySpark filtering, Cloudera storage, and Presto queries.

Health Information Application

Innovative movie recommendation project using MovieLens data. This project aims to integrate Big Data analysis and machine learning to improve movie suggestions, using Apache Spark, Elasticsearch and a Flask API, to deliver a personalized and dynamic user experience.

Smart City Realtime Data Engineering Project