E2E ML Apache Spark

Rodrigo Hernández Mota



Data Engineer / Scientist at LeadGenius


  1. ML Project Overview
  2. Operationalizing
  3. Spark-Based Projects
  4. Code Example!


ML Project Cycle

Machine Learning Cycle
Machine Learning Cycle

ML Pipelines

Machine Learning Pipeline
Machine Learning Pipeline


Traditional Approach

Impedance Mismatch
Impedance Mismatch

A simple solution

Predictive Markdown Model Language is:

"an XML-based language that provides a way for applications to define statistical and data-mining models as well as to share models between PMML-compliant applications."

PMML Interop

Integration with the most popular ML frameworks via JPMML:

Simple Solution using PMML

Simple Scoring
Simple Scoring

Best Practice

We can perform model scoring either with a stream-processing engine or a stream-processing library.

Suggested Architecture
Suggested Architecture

Best Practice - Akka Aproach

We can use Akka Streams - based on Akka Actors (see syntax example).

Simple Akka Implementation
Simple Akka Implementation

Akka Cluster

Akka Cluster Implementation
Akka Cluster Implementation

The Big Picture

Data Architecture
Data Architecture

Spark-Based Projects

Why Apache Spark?

According to their website,

"Apache Spark is a unified analytics engine for large-scale data processing."

Intro to Spark ML

Spark ML is a practical and scalable machine learning library based on a [Dataset].

  • Dataset[A].map(fn: A => B): Dataset[B]
  • Dataset[A].flatMap(fn: A => Dataset[B]): Dataset[B]
  • Dataset[A].filter(fn: A => Boolean): Dataset[A]

Relevant Concepts

  • Dataset[Row]
  • Transformer
  • Estimator
  • Pipeline

Intro to JPMML

val pmmlBuilder = new PMMLBuilder(schema, pipelineModel)

See the official jpmml-sparkml github repo for a complete list of supported PipelineStages types.

Intro to Openscoring

We can use Openscoring, a java-based REST web-service, as our scoring-engine of the resulting PMML model.

  • Simple but powerful API
  • Allows for single predictions and for batch predictions.
  • Acceptable performance (usually sub-milliseconds respond time)

Demo Time!