Over View

What is Apache Spark?

Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, Spark ML for machine learning, GraphX for graph data processing, and Spark Streaming for live data stream processing. With Spark running on Apache Hadoop YARN, developers can create applications to derive actionable insights within a single, shared dataset in Hadoop.

Course Overview

This training course will teach you how to solve Big Data problems using Apache Spark framework. The training will cover a wide range of Big Data use cases such as ETL, DWH, data virtualization, streaming, graph data structure, machine learning. It will also demonstrate how Spark integrates with other well established Hadoop ecosystem products. You will learn the course curriculum through theory lectures, live demonstrations and lab exercises. This course will be taught in primarily Scala programming language.

Duration: 3 Days or 24 Hours

Mode of Delivery: Virtual Training

Prerequisites

Following are the pre-requisites for the course.

  • Basic Knowledge of big data use-cases
  • Basic knowedge of Hadoop, HDFS and Hive
  • Basic knowledge of databases, OLAP/OTLP use cases, SQL
  • Programming knowledhe in python

Apache Spark Course Content

Day 1 – Spark core, spark internals, performance tuning, building spark application

Introduction to Spark

  • What is Apache Spark- the story of the evolution from Hadoop
  • Advantages of Spark over Hadoop Map Reduce
  • Lambda architecture for enterprise data and analytics services
  • Deployment modes – YARN, Standalone, Mesos
  • Developing on Spark using REPL, Jupyter Notebook, IDE
  • Data sources for Spark application
  • What is pyspark

Lab Exercise

  • Install and get started with VM
  • Launching spark REPL and Jupyter Notebook

Resilient Distributed Dataset (RDD)

  • RDD – operations – read from the file, transforming and saving persistent
  • Leveraging in memory processing
  • Pair RDD – operations
  • Working with semi structured data formats using regex, json, xml libraries

Lab Exercise

  • Explore data from data.sfgov.org using RDD
  • Join and aggregate data from grouplens.org

Spark Internals

  • Anatomy of Spark jobs on YARN, Standalone and Mesos
  • RDD partitions
  • Spark literature: Narrow, wide operations, shuffle, DAG, Shuffle, Stages, and Tasks
  • Job metrics
  • Fault Tolerance

Performance Tuning

  • Factors that affect performance of spark application
  • Configuring memory and CPU for Spark drivers and executors in standalone and YARN mode
  • Controlling logging of Spark daemons and spark applications
  • Capturing job metrics using Spark History Server
  • Benefits of shared variables – accumulator, broadcast var
  • Types of spark caching and their use cases
  • Role of checking pointing of Spark RDD

Lab Exercises

  • Examine spark metrics and logs
  • Evaluate impact of different caching types on memory and processing time
  • Use shared variables

Setup Spark Cluster

  • Set up Spark on YARN
  • Set up Spark Standalone [Optional]

Building Application

  • Building application using IDE
  • Submitting spark Jobs
  • Building Spark application using Jupyter Notebook

Lab Exercise

  • Build an application and submit spark job on cluster

Day 2: Spark Dataframe, SQL and Spark Streaming

Dataframe Basics

  • Introduction to Dataframe
  • Difference between RDD and Dataframe
  • Dataframe internals that makes it fast – Catalyst Optimizer and Tungsten
  • Loading and processing data into dataframe
  • Saving dataframe to file systems

Lab Exercise

  • Process data.sfgov.org data using dataframe

Dataframe Advanced

  • Hive Context vs Spark SQL Context
  • Working with Hive Tables
  • Working with JDBC data source
  • Data formats – text format such csv, json, xml, binary formats such as parquet, orc
  • UDF in Spark Dataframe
  • Spark SQL as JDBC service and its benefits and limitations
  • Analytical queries in Spark – windows functions, pivot, rollup and cubes
  • Working with data from jdbc soruces

Hands On

  • Persist spark tables in Hive
  • Processing Mysql data using Dataframe
  • Working with UDF
  • Working with file formats – text formats (CSV, json, xml) and binary formats (ORC, Parquet, avro)

Introduction to Spark Streaming

  • Architecture of streaming application
  • Streaming Context – initialization, configuration, characteristics
  • Dstream – operations
  • Receiver characteristics
  • Window operation – batch internal, window length, sliding interval
  • Fault Tolerance using checkpointing and replication
  • Partition behavior of Dstream

Lab Exercises

  • Process socket streaming source
  • Processing file streaming source
  • Saving processed streams into HDFS
  • Saving processes streams into HBase using Spark SQL (optional)

Day 3: Kafka Streaming and Structured Streaming

Introduction to Apache Kafka

  • What is apache kafka and it its use cases
  • Overview Kafka Architecture
  • Kafka components – brokers, messages, consumer groups, consumer, producer
  • Kafka logs, log access patters
  • Configure Kafka cluster using multi nodes
  • Fault tolerance, consistency, schema validation
  • Kafka connectors for JDBC, HDFS

Lab Exercise

  • Setting up single node Kafka cluster

Streaming Advanced Receiver

  • Introduction to KAFKA receiver for Spark
  • Lambda architecture

Lab Exercise

  • Create a spark streaming application using KAFKA as source
  • Setup Kafka JDBC connector

Structured Streaming

  • What is structured streaming
  • Understanding source, sink patterns
  • Streaming SQL
  • Source types
  • Sink types
  • Foreach sink

Lab Exercise

  • Processing live stream using structured streaming

 

FAQ’S

What if I miss one (or) more class?

No need to worry about the classes you missed. We will definitely guide you by having optional classes or by having classes with other batches with the same topic you missed previous classes.

Who is my instructor?

IT professionals who have strong knowledge in technical know how to convey things with the real-time example. Even a layman could understand the concepts which given by our experts.

What are the modes of training offered for this course?

We offer this course in “Live Instructor-Led Online Training” mode. Through this way you won’t mess anything in your real-life schedule. You will be shared with live meeting access while your session starts.

What are the system requirements to work?

Minimum 2GB RAM and i3 processor is required

Can I attend a demo session?

You can get a sample class recording to ensure you are in right place. We ensure you will be getting complete worth of your money by assigning a best instructor in that technology.

How about group discounts (or) corporate training for our team?

We are absolutely loved to talk in-person about group training (or) corporate training. So, please get in touch with our team through “Quick Enquiry”, “Live Chat” or “Request Call-back” channels.

Where do Our Online learners and Trainer’s come from

We are providing online training, One-to-One training with the help of experts. Our learners and trainers are frequently coming from different countries like USA, India, UK, Australia, New Zealand, Canada and UAE. To specify in cities London, Bangalore, California, New York, Pune, Mumbai, Chennai, New Delhi, San Francisco, New Jersey, Texas, Florida, Kolkata, Gurgaon, Berlin and Hyderabad among many.

I have more queries?

If you want to know More Details about Online Training Please Contact us. Or you can share your quires through info@monstercourses.com. Estimated turnaround time will be 24 hours for mails.

Contact us

Enquiry Now..!!

Contact Details :

    Address: # 4110 Rainy Creek Ln, Cedar Park, TX, 78613, USA.

    Contact us: +1(772)777-1557

    Email ID: info@monstercourses.com

     

    Popular Courses We Offered :

     

    4110 Rainy Creek Ln, Cedar Park,

    TX USA, 78613.

    Phone: +1(772)777-1557

    Email : info@monstercourses.com

     

    We are very much benefited from Business objects online training program. Handouts are very useful and very nicely given so that at-least by seeing the handouts we can practice and get knowledge. We have been taught many topics from this Training, nice to getting trained from “MonsterCourses

    Govardhan Bhaskar….

    Vijay Naresh, Monster Courses

    The Training Classes what i have attended from MonsterCourses was quite beneficial.Good way of diving modules and covering important topics in each module. Overall the training from MonsterCourses was very good and useful to me.

    Vijay Naresh…

    Raju K, Monster Courses

    I felt OBIEE online training was very informative as compared to other online training providers. The faculty of MonsterCourses are very efficient in their subject and gave quality training. They covered each and every topic clearly elaborated them with examples.

    Vinay K…

    Govardhan Bhaskar, Monster Courses