Intermediate

Apache Spark Streaming with Scala

Unlock the power of Apache Spark Streaming with Scala. Learn to process massive data in real time, integrate with Kafka, JDBC, Cassandra, and other tools, and manage live data streams with ease. Ideal for mastering real-time data processing and streaming applications.

Duration: 11h of 4K content
Lessons: 31 lessons

By Daniel Ciocîrlan

Join 100,000+ students enrolled in Rock the JVM courses

Money-back guarantee · Unlimited access · Free updates

Skills You'll Learn

Use Spark Structured Streaming APIs as fluently as batch Spark
Perform projections, joins, aggregations, and groupings on streams
Process streaming data with DStreams at a low level
Work with time columns and sliding and tumbling window functions
Use reduce-by-window and reduce-by-window-and-key operations
Integrate Spark with Kafka, JDBC, NoSQL, and Akka sources
Manually manage stateful data processing with mapGroupsWithState
Process per-RDD individually with arbitrary transformations

Goal

Nothing static, all in motion.

You probably know by now: Spark is the most popular computing engine for big data, the most maintained, and with a proven track record of performance. It’s 100 times faster than the old MapReduce paradigm, and can easily be extended with machine learning and streaming capabilities, and much more.

In this Spark Streaming course, we’ll take the natural step forward: process big data as it arrives.

What’s in for you:

You’ll learn how Spark Structured Streaming and “normal” Spark batch operations are similar and different
You’ll work with new streaming abstractions (DStreams) for low-level, high-control processing
You’ll integrate Kafka, JDBC, Cassandra and Akka Streams (!) so that you can later integrate anything you like
You’ll work with powerful stateful APIs that only a few know how to properly use

And some extra perks: - You’ll have access to the entire code I write on camera (2200+ LOC) - You’ll be invited to our private Slack room where I’ll share latest updates, discounts, talks, conferences, and recruitment opportunities - (soon) You’ll have access to the takeaway slides - (soon) You’ll be able to download the videos for your offline view

What Our Students Say

Trusted by 100,000+ students worldwide

My team is expanding the use of Akka in our products so I needed a quick introduction on this topic. I have tried a couple of courses but the introduction to Akka was always too abrupt, too hard to comprehend. I blamed Akka for this as being too hard to explain. This was until I was exposed to the Rock The JVM courses which were an absolute delight when it comes to presenting such complex topics in such an easy to understand way. And Daniel has not stopped at Akka but has added to his portfolio amazing courses on Scala and Spark too. It seems like he is quite enjoying taking such challenges like complex technologies and making them so simple for everyone. I have instantly recommended Daniel’s work to my team, which helped them immensely with taking their skills to a new level, and I do recommend these courses to anyone who wants to have the fastest ramp-up in these tough but popular technologies.

Mihai Fecioru
Adobe · California
From Scala, to Akka, to Spark, Daniel delivers exceptional material in each and every one of these technologies. I’ve been using them for a long time and there is always something new I will discover from him. The level of detail he gets into as well as the way he delivers material is mindblowing. I personally find his latest course Spark Optimization pure gold and one of a kind. I’ve been using Spark for a year now and I haven’t even thought how much you can leverage query plans to make such optimizations. I can’t stop thinking every time, how he manages to go so deep - because using a technology is one thing, but knowing its internals so well and how everything works behind the scenes is another story when it comes to distributed systems. Long story short Daniel is definitely the best instructor I’ve come across and each one of his courses is the best resource you can find online. Kudos for all your work and knowledge sharing.

Giannis Polyzos
Ververica · Greece
Daniel’s courses on Scala and Big Data are the best in class. I’ve been in touch with Daniel’s teaching and courses since early 2018. The first course that I took from him was Scala & Functional Programming; I was skeptical about it because over the internet there are many courses you can find, but few really worthy. I remember the very first day when Daniel started to speak and shared his examples - I started to love Scala, and then more as we went on. I am with Scala for the last 5 years now, but never ever has anyone explained to me or gave me comparable resources to Rock the JVM. Daniel gave me a shift in life and helped me crack top tech company interviews. His courses on big data are a must for any aspiring big data developer or data enthusiast. I highly recommend Daniel as an educator both online and on campus.

Anirban Goswami
Apple · California

What's Included

9 sections

31 lessons

11h 17m total length

Prologue

Welcome

18:04 Preview

Introduction

Scala Recap

25:05 Preview

Spark Recap

27:10 Preview

Spark Streaming First Principles

08:28 Preview

Structured Streaming

Streaming DataFrames

27:16 Preview

Streaming Aggregations

15:57 Preview

Streaming Joins

25:23 Preview

Streaming Datasets

26:45 Preview

Low-Level Streaming with DStreams

Discretized Streams (DStreams)

31:57 Preview

DStreams Transformations

28:39 Preview

DStreams Window Functions

32:10 Preview

Spark Streaming Integrations

Kafka & Structured Streaming

25:07 Preview

Kafka & DStreams

29:56 Preview

JDBC with Postgres

11:50 Preview

Akka and Akka Streams

28:57 Preview

Cassandra

21:54 Preview

Twitter Sentiment Analysis

Setting up a Twitter App

08:15 Preview

Our First Custom Receiver

15:44 Preview

Reading Tweets

20:04 Preview

Reading Tweets: Exercises

15:21 Preview

Sentiment Analysis on Tweets using NLP

20:49 Preview

Advanced Spark Streaming

Event Time Windows

31:19 Preview

Event Time Windows: Exercises

14:53 Preview

Processing Time Windows

11:15 Preview

Watermarks

25:15 Preview

Watermarks, Part 2

25:21 Preview

Arbitrary Stateful Computation

25:39 Preview

Arbitrary Stateful Computation

16:35 Preview

The Science Project

Setting up the REST Server and the Kafka Broker

27:50 Preview

Integrating Spark Structured Streaming, Test, Run!

33:53 Preview

Epilogue

You Rock!

00:30 Preview

Meet Rock the JVM

Daniel Ciocîrlan

Founder, Rock the JVM

I'm a software engineer and the founder of Rock the JVM.

I started Rock the JVM out of love for Scala and the technologies it powers. They are amazing tools, and I want to share as much of my experience with them as I can.

I've taught Java, Scala, Kotlin and related technologies such as Cats, ZIO and Spark to 100,000+ students at various levels. I've held live training sessions for companies including Adobe and Apple, taught university students who now work at Google and Facebook, run Hour of Code for 7-year-olds, and taught more than 50,000+ kids to code.

I have a Master's Degree in Computer Science and I wrote my Bachelor and Master thesis on Quantum Computation. Before learning programming, I won medals at international Physics competitions.

Enroll now!

Best Value

MonthlyYearly

All-Access Membership

Full (and growing) catalog

$16.25/month

$195 billed yearly —Save 54%

Unlimited access to every Rock the JVM course

348 hours of 4K content
All Scala courses
All Kotlin courses
All Typelevel courses
All ZIO courses
All Apache Spark courses
All Apache Flink courses
All Akka/Pekko courses
Access to the private Rock the JVM community
New courses included automatically

Start Membership

The Apache Spark Bundle with Scala

4 courses, one price

$180

Save 42%vs buying separately

All courses in this bundle with a one-time payment

4 courses included
38 hours of 4K content
All PDF slides
Free updates
Lifetime access
Access to the private Rock the JVM community

Buy Bundle