What is DStream in Spark

In this tutorial, we shall learn what is spark streaming and what is a discretized stream or DStream in Spark.

What are DStreams in Spark?

In this tutorial, we shall learn what is spark streaming and what is discretized stream or DStream in Spark. Spark Streaming is a feature of the core Spark API that allows for scalable, high-throughput, and fault-tolerant live data stream processing. Data can be ingested from a variety of sources, including Kafka, Kinesis, and TCP connections, and processed with complicated algorithms described using high-level functions like map, reduce, join, and window. Finally, data can be written to filesystems, databases, and live dashboards. Spark's machine learning and graph processing methods can even be used on data streams.

Access Snowflake Real Time Data Warehousing Project with Source Code

   


A discretized stream, or DStream, is a high-level abstraction provided by Spark Streaming that describes a continuous stream of data. DStreams can be produced by performing high-level operations on existing DStreams or by using input data streams from sources like Kafka and Kinesis. A DStream is internally represented as a succession of RDDs. A DStream's RDDs each hold data from a certain interval.



Any operation on a DStream corresponds to operations on the RDDs beneath it. The flatMap operation is executed to each RDD in the lines DStream to construct the RDDs of the words DStream in the previous example of converting a stream of lines to words.

 

The Spark engine calculates the underlying RDD transforms. The DStream operations mask the majority of these complexities and provide a higher-level API for developer convenience.

What Users are saying..

profile image

Ed Godalle

Director Data Analytics at EY / EY Tech
linkedin profile url

I am the Director of Data Analytics with over 10+ years of IT experience. I have a background in SQL, Python, and Big Data working with Accenture, IBM, and Infosys. I am looking to enhance my skills... Read More

Relevant Projects

Build a Real-Time Spark Streaming Pipeline on AWS using Scala
In this Spark Streaming project, you will build a real-time spark streaming pipeline on AWS using Scala and Python.

Yelp Data Processing Using Spark And Hive Part 1
In this big data project, you will learn how to process data using Spark and Hive as well as perform queries on Hive tables.

SQL Project for Data Analysis using Oracle Database-Part 5
In this SQL Project for Data Analysis, you will learn to analyse data using various SQL functions like ROW_NUMBER, RANK, DENSE_RANK, SUBSTR, INSTR, COALESCE and NVL.

Hive Mini Project to Build a Data Warehouse for e-Commerce
In this hive project, you will design a data warehouse for e-commerce application to perform Hive analytics on Sales and Customer Demographics data using big data tools such as Sqoop, Spark, and HDFS.

Project-Driven Approach to PySpark Partitioning Best Practices
In this Big Data Project, you will learn to implement PySpark Partitioning Best Practices.

Build a Scalable Event Based GCP Data Pipeline using DataFlow
In this GCP project, you will learn to build and deploy a fully-managed(serverless) event-driven data pipeline on GCP using services like Cloud Composer, Google Cloud Storage (GCS), Pub-Sub, Cloud Functions, BigQuery, BigTable

Build a Real-Time Dashboard with Spark, Grafana, and InfluxDB
Use Spark , Grafana, and InfluxDB to build a real-time e-commerce users analytics dashboard by consuming different events such as user clicks, orders, demographics

Data Processing and Transformation in Hive using Azure VM
Hive Practice Example - Explore hive usage efficiently for data transformation and processing in this big data project using Azure VM.

Snowflake Real Time Data Warehouse Project for Beginners-1
In this Snowflake Data Warehousing Project, you will learn to implement the Snowflake architecture and build a data warehouse in the cloud to deliver business value.

Yelp Data Processing using Spark and Hive Part 2
In this spark project, we will continue building the data warehouse from the previous project Yelp Data Processing Using Spark And Hive Part 1 and will do further data processing to develop diverse data products.