Apache Spark Course Content

Day 1 – Spark core, spark internals, performance tuning, building spark application

Introduction to Spark

• What is Apache Spark- the story of the evolution from Hadoop
• Advantages of Spark over Hadoop Map Reduce
• Lambda architecture for enterprise data and analytics services
• Deployment modes – YARN, Standalone, Mesos
• Developing on Spark using REPL, Jupyter Notebook, IDE
• Data sources for Spark application
• What is pyspark

Lab Exercise

• Install and get started with VM
• Launching spark REPL and Jupyter Notebook

Resilient Distributed Dataset (RDD)

• RDD – operations – read from the file, transforming and saving persistent
• Leveraging in memory processing
• Pair RDD – operations
• Working with semi structured data formats using regex, json, xml libraries

Lab Exercise

• Explore data from data.sfgov.org using RDD
• Join and aggregate data from grouplens.org

Spark Internals

• Anatomy of Spark jobs on YARN, Standalone and Mesos
• RDD partitions
• Spark literature: Narrow, wide operations, shuffle, DAG, Shuffle, Stages, and Tasks
• Job metrics
• Fault Tolerance

Performance Tuning

• Factors that affect performance of spark application
• Configuring memory and CPU for Spark drivers and executors in standalone and YARN mode
• Controlling logging of Spark daemons and spark applications
• Capturing job metrics using Spark History Server
• Benefits of shared variables – accumulator, broadcast var
• Types of spark caching and their use cases
• Role of checking pointing of Spark RDD

Lab Exercises

• Examine spark metrics and logs
• Evaluate impact of different caching types on memory and processing time
• Use shared variables

Setup Spark Cluster

• Set up Spark on YARN
• Set up Spark Standalone [Optional]

Building Application

• Building application using IDE
• Submitting spark Jobs
• Building Spark application using Jupyter Notebook

Lab Exercise

• Build an application and submit spark job on cluster

Day 2: Spark Dataframe, SQL and Spark Streaming

Dataframe Basics

• Introduction to Dataframe
• Difference between RDD and Dataframe
• Dataframe internals that makes it fast – Catalyst Optimizer and Tungsten
• Loading and processing data into dataframe
• Saving dataframe to file systems

Lab Exercise

• Process data.sfgov.org data using dataframe

Dataframe Advanced

• Hive Context vs Spark SQL Context
• Working with Hive Tables
• Working with JDBC data source
• Data formats – text format such csv, json, xml, binary formats such as parquet, orc
• UDF in Spark Dataframe
• Spark SQL as JDBC service and its benefits and limitations
• Analytical queries in Spark – windows functions, pivot, rollup and cubes
• Working with data from jdbc soruces

Hands On

• Persist spark tables in Hive
• Processing Mysql data using Dataframe
• Working with UDF
• Working with file formats – text formats (CSV, json, xml) and binary formats (ORC, Parquet, avro)

Introduction to Spark Streaming

• Architecture of streaming application
• Streaming Context – initialization, configuration, characteristics
• Dstream – operations
• Receiver characteristics
• Window operation – batch internal, window length, sliding interval
• Fault Tolerance using checkpointing and replication
• Partition behavior of Dstream

Lab Exercises

• Process socket streaming source
• Processing file streaming source
• Saving processed streams into HDFS
• Saving processes streams into HBase using Spark SQL (optional)

Day 3: Kafka Streaming and Structured Streaming

Introduction to Apache Kafka

• What is apache kafka and it its use cases
• Overview Kafka Architecture
• Kafka components – brokers, messages, consumer groups, consumer, producer
• Kafka logs, log access patters
• Configure Kafka cluster using multi nodes
• Fault tolerance, consistency, schema validation
• Kafka connectors for JDBC, HDFS

Lab Exercise

• Setting up single node Kafka cluster

Streaming Advanced Receiver

• Introduction to KAFKA receiver for Spark
• Lambda architecture

Lab Exercise

• Create a spark streaming application using KAFKA as source
• Setup Kafka JDBC connector

Structured Streaming

• What is structured streaming
• Understanding source, sink patterns
• Streaming SQL
• Source types
• Sink types
• Foreach sink

Lab Exercise

• Processing live stream using structured streaming

 

FAQ’S

What if I miss one (or) more class?

No need to worry about the classes you missed. We will definitely guide you by having optional classes or by having classes with other batches with the same topic you missed previous classes.

Who is my instructor?

IT professionals who have strong knowledge in technical know how to convey things with the real-time example. Even a layman could understand the concepts which given by our experts.

What are the modes of training offered for this course?

We offer this course in “Live Instructor-Led Online Training” mode. Through this way you won’t mess anything in your real-life schedule. You will be shared with live meeting access while your session starts.

What are the system requirements to work?

Minimum 2GB RAM and i3 processor is required

Can I attend a demo session?

You can get a sample class recording to ensure you are in right place. We ensure you will be getting complete worth of your money by assigning a best instructor in that technology.

How about group discounts (or) corporate training for our team?

We are absolutely loved to talk in-person about group training (or) corporate training. So, please get in touch with our team through “Quick Enquiry”, “Live Chat” or “Request Call-back” channels.

Where do Our Online learners and Trainer’s come from

We are providing online training, One-to-One training with the help of experts. Our learners and trainers are frequently coming from different countries like USA, India, UK, Australia, New Zealand, Canada and UAE. To specify in cities London, Bangalore, California, New York, Pune, Mumbai, Chennai, New Delhi, San Francisco, New Jersey, Texas, Florida, Kolkata, Gurgaon, Berlin and Hyderabad among many.

I have more queries?

If you want to know More Details about Online Training Please Contact us. Or you can share your quires through info@monstercourses.com. Estimated turnaround time will be 24 hours for mails.

Contact us