In a world of interconnected devices, the amount of data being generated is skyrocketing. In order to analyze and process this data, engineers often employ Machine Learning (ML) techniques that allow them to gather valuable insights and actionable information. However, as the volume of data continues to grow, the need to run big data processing pipelines arises, a demand your laptop can’t cope with.

Therefore, the use of distributed frameworks in fleets of instances is becoming increasingly popular. In order to run a distributed framework in a cluster platform like AWS EMR effectively, it is important to have a good understanding of the underlying architecture and infrastructure. In the second part of our two-blog series, we present a hands-on tutorial where we will dive into the details of setting up and configuring the necessary infrastructure to launch an EMR cluster using Lambda functions to run a sample pySpark script, the one we introduced in part one. You will need an AWS account to follow this tutorial as we’ll be using their services.

EMR

EMR (Elastic MapReduce) is an AWS solution for running big data frameworks. It provides a managed cluster platform that simplifies running tools such as Apache Hadoop, Apache Spark and Presto in order to process and analyze large volumes of data. Additionally, it provides several options for storing data, including Amazon S3, Hadoop Distributed File System (HDFS), and HBase.

EMR offers flexibility and scalability against changing processing demands by allowing you to easily adjust the number and type of node-instances in the cluster. The costs will depend on the quantity and type of instances launched. You have the opportunity to further minimize the cost by buying Reserved Instances or Spot Instances instead of on-demand. Spot Instances can provide considerable cost savings, sometimes being as low as one-tenth of on-demand pricing.

It can integrate with other AWS services to provide different capabilities and functionalities. IAM integration can be used to manage fine-grained access, also provides data encryption in transit and at rest. Instances can be launched in a virtual network of your choosing. Furthermore, it provides monitoring and logging capabilities that enable you to track cluster performance and diagnose issues.

Overall, AWS EMR is a great platform for processing and analyzing data since it provides the ability to handle complex workloads, scale as needed, lots of customization and other services integrations. We chose it for today’s tutorial and for our Spark pipeline. The parallel aspect of the framework allows us to process data and train ML models much faster and cost-effective.

There are many ways in which one could trigger an EMR Cluster. In this tutorial we are going to be launching the cluster through a Lambda, another very popular tool that we will be describing next. We opted for this method to launch the cluster because it simplifies the process of updating cluster configurations as well as the various parameters required for the script that we will be running. 

Networking

Prior to creating the EMR cluster, you must create a new VPC or select an existing one that will include the subnets in which the cluster will be launched. It is recommended that you use private subnets, as having a direct route to the internet would make the cluster vulnerable to external threats.

It’s also important to ensure that the networking configuration is compatible with the requirements of the EMR cluster. This includes ensuring that the subnet has sufficient IP addresses to accommodate the number of nodes in the cluster, and that the networking is configured to allow communication between the nodes.

Furthermore, you may want to consider implementing additional network infrastructure such as a VPN or direct connect to allow secure access to the cluster from on-premises data centers or other cloud environments.

Storage

Another must is having a reliable storage system to store the scripts and data we will be using on the cluster. In this blog we will use S3. Amazon S3 is a scalable and durable object storage service that is designed to store and retrieve any amount of data from anywhere on the web. 

It enables easy data transfer and sharing between different applications and services. The EMR cluster can read data from the S3 bucket and write back the results to it. Using an S3 bucket to store your EMR scripts and data provides a secure, scalable, and cost-effective solution that we can’t stop recommending.

IAM

Lastly, we will need 3 different roles, one for implementing the Lambda and two different roles for running the EMR Cluster. While it’s possible to use a single role with permissions for both services, it is not a recommended practice.
Lambda functions require an execution role. It is a permission set that defines the level of access an AWS Lambda function has to AWS services and resources. When you create a Lambda function, you need to specify an execution role that grants permission for the function to interact with other AWS services, such as S3 and EMR. As a minimum it must include the following AWS managed permissions:
AWSLambdaBasicExecutionRole, AmazonEMRFullAccessPolicy_v2 and AmazonElasticMapReduceFullAccess.
EMR will require a job flow role. This role is used to manage clusters and execute jobs. The role allows EMR to access resources that are necessary for cluster creation and operation, such as launching and terminating Amazon EC2 instances, reading and writing data to Amazon S3, and accessing other AWS services. We will select the EMR Role for EC2 option as trusted entity for this role and include AmazonElasticMapReduceforEC2Role in permitted policies.
EMR will also require a service role. This role will be responsible for granting permissions to the EMR service to perform tasks that are necessary for EMR’s operation, such as creating, modifying, or deleting resources used by EMR clusters. This includes creating temporary security groups, network interfaces, and instance profiles for the EMR cluster’s nodes. We can use the AWS pre-defined EMR_DefaultRole.

lambda

AWS Lambda is another AWS Service that allows developers to run code without the need to provision or manage servers since it manages all aspects of the infrastructure and resources, including server maintenance, operating system updates, automatic scaling, logging and more. Instead of deploying an entire server or container, you simply write code and have it executed in response to events.
It is highly available and implements a pay-as-you-go pricing model, which means you only pay for the compute time that your function uses, making it cost-effective. Use cases include file and stream processing, web applications, IoT and mobile backends, among others.

Creating a Lambda function

Log in to your AWS account and type Lambda on the search bar. Access the service and select the Create function button. Fill in basic information for your Lambda function. On this occasion we will create a script from scratch and will be using Python as our runtime language.

Regarding the execution role, you could create a new role or use an existing one.

Creating a handler

AWS Lambda handlers are the entry point for the code that runs in an AWS Lambda function. They are the functions that AWS Lambda invokes when it executes your function code. Since we specified a Python runtime, the file has a .py extension and we will be writing Python code. Under the Code source section you will find a Runtime settings section. There you will find the Handler property that defines a method inside a file that is our entry point, initially being the lambda_handler method inside the lambda_function file. 

The AWS Lambda function handler has two parameters, event and context. Event represents the input data, it contains information about the triggering event that caused the invocation of the function, for example an HTTP request, a scheduled event or a message from another AWS service. Context parameter provides information about the current invocation and execution environment as well as the Lambda function’s runtime, such as the function name, version, and ARN, the AWS request ID, the CloudWatch Logs stream name, and the function’s remaining execution time.

As per this blog, we will be including some arguments in our Event parameter while configuring our test event, it could include any sort of information in JSON format. 

Lambda_function.py

The code we include on the handler file, will begin by importing some libraries, then defining the EMR Cluster and lastly implementing a return value.

1 
2  

import  boto3
connection = boto3.client('emr')

1 import  boto3
2 connection = boto3.client('emr')

By making use of boto3, a Python library that provides an interface to easily create, manage and configure AWS resources, we will be defining and creating our EMR cluster. The policies attached to the execution role we previously associated, grant the Lambda function permissions to access other AWS services, so in order to enable integration with EMR we previously attached the AmazonEMRFullAccessPolicy_v2 policy.

The next step is to create our handler

 4
 5
 6
 7

def lambda_handler(event, context):
    max_iter = event.get('epochs', "10")
    x1 = event.get('x1', "10")
    x2 = event.get('x2', "10")

 4 def lambda_handler(event, context):
 5      max_iter = event.get('epochs', "10")
 6     x1 = event.get('x1', "10")
 7     x2 = event.get('x2', "10")

The first thing we will do is extract the parameters from the event that will be used in the script. These are max_iter, which refers to the maximum number of iterations the model can run over the data to learn. X1 and X2, which are meant to be the coefficients in our polynomial function. All parameters default to 10 in case they can’t be obtained from the event. 

Now is time to configure the EMR cluster. We will begin by defining the basics.

 8
 9
10
11
12

   cluster_id = connection.run_job_flow(
        Name="Guess-the-function",
        LogUri="s3:/guess-the-function-emr/logs/"
        ReleaseLabel='emr-6.6.0',
        Applications=[{'Name': 'Spark'}],

 8     cluster_id = connection.run_job_flow(
 9        Name="Guess-the-function",
10        LogUri="s3:/guess-the-function-emr/logs/"
11        ReleaseLabel='emr-6.6.0',
12        Applications=[{'Name': 'Spark'}],

Name is the name that will be adopted by the cluster. LogUri will point to an S3 folder where to store the logs, although this is not required, we highly recommend you to have EMR logs available to ease debugging. ReleaseLabel attribute is used to specify the version of software applications and Hadoop components used by the EMR cluster. Finally, Applications refers to the packages we will use and need installed on the cluster.

Now we will be defining the group of instances used on the cluster. The amount of computing resources that will be used is an absolute overkill for the task in hand. It’s done that way to show how to set up multiple instances in a cluster. 

13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

  Instances={
'InstanceGroups': [
{
'InstanceRole': 'MASTER',
'Name': "Master node",
'InstanceType': 'm4.large',
'InstanceCount': 1,
},
{
'InstanceRole': 'CORE',
'Name': "Slave node",
'InstanceType': 'm4.large',
'InstanceCount': 2,
}
],
'KeepJobFlowAliveWhenNoSteps': False,
'Ec2SubnetId': 'subnet-jnp2fed5',
},

13      Instances={
14         'InstanceGroups': [
15             {
16                 'InstanceRole': 'MASTER',
17                 'Name': "Master node",
18                 'InstanceType': 'm4.large',
19                 'InstanceCount': 1,
20            },
21
22                 'InstanceRole': 'CORE',
23                 'Name': "Slave node",
24                 'InstanceType': 'm4.large',
25                 'InstanceCount': 2,
26
27          ],
28          'KeepJobFlowAliveWhenNoSteps': False,
29          'Ec2SubnetId': 'subnet-jnp2fed5',
30      },        

EMR always requires a master node to coordinate the overall cluster and distribute tasks to worker nodes, there are typically one or more worker nodes. These are responsible for executing data processing tasks and storing data in the HDFS. We will allocate m4.large instances for both master and slave nodes. There can only be a single master node at a time but a large number of worker nodes.

KeepJobFlowAliveWhenNoSteps determines whether the cluster should terminate automatically when there are no more running steps. Having the cluster alive can save you some time as long as you are submitting steps but it will continue to incur costs, even if there are no running steps.

Ec2SubnetId should be a subnet created within the VPC we previously mentioned.

31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

 Steps=[
        {
            'Name': 'Guess the coefficients',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': [
                    'spark-submit',
                    "s3:/guess-the-function-emr/scripts/guess.py",
                    "--max_iter", max_iter, 
                    "--x1", x1,
                    "--x2", x2
                ]
            }
        }
    ],

31  Steps=[
32        {
33            'Name': 'Guess the coefficients',
34            'ActionOnFailure': 'CONTINUE',
35            'HadoopJarStep': {
36                'Jar': 'command-runner.jar',
37                'Args': [
38                    'spark-submit',
39                    "s3:/guess-the-function-emr/scripts/guess.py",
40                    "--max_iter", max_iter, 
41                    "--x1", x1,
42                    "--x2", x2
43                ]
44            }
45        }
46    ],

Here we are defining a step for the EMR cluster, a step is a unit of work that you can add to a cluster to perform a specific task or processing on your data. It consists of an executable program or script, along with any necessary configuration and input/output data. There are several types of steps. The code defines one that involves submitting a Spark job to execute a script stored in S3 with some previously defined parameters.

47
48
49
50
51
52
53
54

VisibleToAllUsers=True,
    JobFlowRole='blog-emr-job-role',
    ServiceRole='EMR_DefaultRole',
    Tags=[{
        "Key":"demo",
        "Value":"guess-the-function"
    }],
)

47     VisibleToAllUsers=True,
48    JobFlowRole='blog-emr-job-role',
49    ServiceRole='EMR_DefaultRole',
50    Tags=[{
51        "Key":"demo",
52        "Value":"guess-the-function"
53    }],
54  )

On the last definition block we will set the VisibleToAllUsers flag to true, that way allowing us to see the cluster that was created by the lambda without the need to deal with IAM policies. We will also define JobFlowRole and ServiceRole, both previously described. You can also see how to include a tag although this is not necessary but definitely recommended if you want to use tools such as the AWS Cost Explorer more efficiently.

56
57
58
59

return {
    'statusCode': 200,
    'body': f"Submitted job with Id {cluster_id['JobFlowId']}"
}

56 return {
57    'statusCode': 200,
58    'body': f"Submitted job with Id {cluster_id['JobFlowId']}"
59   }

Let’s finish by returning a statusCode and a body with the EMR Cluster Id. 

Putting it all together, the handler would look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

import boto3
connection = boto3.client('emr')
def lambda_handler(event, context):
    max_iter = event.get('epochs', "10")
    x1 = event.get('x1', "10")
    x2 = event.get('x2', "10")
    cluster_id = connection.run_job_flow(
        Name="Guess-the-function",
        LogUri="s3:/guess-the-function-emr/logs/"
        ReleaseLabel='emr-6.6.0',
        Applications=[{'Name': 'Spark'}],
        Instances={
            'InstanceGroups': [
                {
                    'InstanceRole': 'MASTER',
                    'Name': "Master node",
                    'InstanceType': 'm4.large',
                    'InstanceCount': 1,
                },
                {
                    'InstanceRole': 'CORE',
                    'Name': "Slave node",
                    'InstanceType': 'm4.large',
                    'InstanceCount': 2,
                }
            ],
            'KeepJobFlowAliveWhenNoSteps': False,
            'Ec2SubnetId': 'subnet-jnp2fed5',
        },
    Steps=[
        {
            'Name': 'Guess the coefficients',
            'ActionOnFailure': 'CONTINUE',
            'HadoopJarStep': {
                'Jar': 'command-runner.jar',
                'Args': [
                    'spark-submit',
                    "s3:/guess-the-function-emr/scripts/guess.py",
                    "--max_iter", max_iter, 
                    "--x1", x1,
                    "--x2", x2
                ]
            }
        }
    ],
    VisibleToAllUsers=True,
    JobFlowRole='blog-emr-job-role',
    ServiceRole='EMR_DefaultRole',
    Tags=[{
        "Key":"demo",
        "Value":"guess-the-function"
    }],
)
return {
    'statusCode': 200,
    'body': f"Submitted job with Id {cluster_id['JobFlowId']}"
}

 1   import boto3
 2   connection = boto3.client('emr')
 3
 4 def lambda_handler(event, context):
 5    max_iter = event.get('epochs', "10")
 6    x1 = event.get('x1', "10")
 7   x2 = event.get('x2', "10")
 8    cluster_id = connection.run_job_flow(
 9        Name="Guess-the-function",
10        LogUri="s3:/guess-the-function-emr/logs/"
11        ReleaseLabel='emr-6.6.0',
12        Applications=[{'Name': 'Spark'}],
13        Instances={
14            'InstanceGroups': [
15                {
16                    'InstanceRole': 'MASTER',
17                    'Name': "Master node",
18                    'InstanceType': 'm4.large',
19                    'InstanceCount': 1,
20                },
21                {
22                    'InstanceRole': 'CORE',
23                    'Name': "Slave node",
24                    'InstanceType': 'm4.large',
25                    'InstanceCount': 2,
26                }
27           ],
28            'KeepJobFlowAliveWhenNoSteps': False,
29            'Ec2SubnetId': 'subnet-jnp2fed5',
30        },
31    Steps=[
32        {
33            'Name': 'Guess the coefficients',
34            'ActionOnFailure': 'CONTINUE',
35            'HadoopJarStep': {
36                'Jar': 'command-runner.jar',
37                'Args': [
38                    'spark-submit',
39                    "s3:/guess-the-function-emr/scripts/guess.py",
40                    "--max_iter", max_iter, 
41                    "--x1", x1,
42                    "--x2", x2
43                ]
44            }
45        }
46    ],
47    VisibleToAllUsers=True,
48    JobFlowRole='blog-emr-job-role',
49    ServiceRole='EMR_DefaultRole',
50    Tags=[{
51        "Key":"demo",
52        "Value":"guess-the-function"
53    }],
54 )
55
56 return {
57    'statusCode': 200,
58    'body': f"Submitted job with Id {cluster_id['JobFlowId']}"
59 }
60

Here is a small snippet for how the Lambda’s test event could look like: 

{
  “epochs”: 10,
“x1”: 5,  
“x2”: 3
}

You just need to test it after deploying the changes by pressing the Test button and poof! You got yourself an EMR running cluster that will train a regression model to try to approximate a polynomial function. 

In order to see the output of the model just open the cluster logs after it has finished. On S3 go to steps > s-ID_OF_NODE > stdout.gz and you will see the desired output.

number of iterations: 4
RMSE: 0.2895831584955561
r2: 0.9952144735311422
model coefficients: [10.004679311797036,10.017003111930043]

By: Maximiliano Palay and Tiziana Romani

Stay ahead of the curve on the latest trends and insights in big data, machine learning and artificial intelligence. Don’t miss out and subscribe to our newsletter!

In a world of interconnected devices, data is being generated at an unprecedented rate. To take advantage of this data, engineers often employ Machine Learning (ML) techniques that allow them to gather valuable insights and actionable information. As the volume of data increases, there is the need to run big data processing pipelines at scale – your laptop just won’t cut it.

To help remedy this, we can use distributed frameworks such as Spark, that allow us to use distributed computing to analyze our data. In this two-part series, you’ll take a small step towards running your data workloads at scale. We present a hands-on tutorial where you’ll learn how to use AWS services to run your processing on the cloud. To do this, we’ll launch an EMR cluster using Lambda functions and run a sample PySpark script. You will need an AWS account to follow this series, as we’ll be using their services.

In this first part, we’ll take a high-level peek at the technologies and architecture, and take a look at a sample problem. In part 2, we’ll dive into the details including architecture and set up of necessary infrastructure.

High-level architecture

Before diving into our sample problem, let’s briefly review the architecture and AWS services involved. We’ll be using Spark, an open-source distributed computing framework for processing large data volumes. Spark can be run on EMR, and we’ll be using its Python API, PySpark. AWS Elastic MapReduce (EMR) is a managed service that simplifies the deployment and management of Hadoop and Spark clusters.  Simple Storage Service (S3) is Amazon’s object storage service, which we’ll use to store our PySpark script.

To create the EMR cluster, we’ll use AWS Lambda, a serverless compute service which allows us to run code without the need to provision or manage servers. Using Lambda to create the cluster allows us to define infrastructure as code, so we can define the cluster specifications and configurations once and execute multiple times. We can then launch multiple clusters easily using the same function. Lambda also enables us to define automatic triggers, which we can use to provision clusters automatically based on a set of actions.

In our sample use case, we’ll create a Python script using AWS’s Python library boto3. This script will be loaded on AWS Lambda and it will trigger the EMR cluster to run our sample PySpark script. The latter will be stored in S3, from where the EMR cluster will retrieve it at runtime.

The problem

We have created a very simple Python script to train a machine learning model using Spark. We’ll generate random data, calculate a linear combination of it, add noise to the results and train a model so that the model can learn how we combined the data. The model should be able to filter out the noise and pretty accurately figure out the coefficients we used to calculate the linear combination of the input data.

Let’s get coding

In this section we’ll walk you through the code, explaining code blocks and commenting on the used functions. At the end you will have the full script available.

1 
2  
3 
4 
5 
6 
7 

import argparse
from pyspark.sql import SparkSession
import pyspark.sql.types as T
import pyspark.sql.functions as F
from pyspark.ml.regression import LinearRegression
from pyspark.ml.feature import VectorAssembler
from pyspark.mllib.evaluation import RegressionMetrics

1 import argparse
2 from pyspark.sql import SparkSession
3 import pyspark.sql.types as T
4 import pyspark.sql.functions as F
5 from pyspark.ml.regression import LinearRegression
6 from pyspark.ml.feature import VectorAssembler
7 from pyspark.mllib.evaluation import RegressionMetrics

We’ll be using arguments for our script so we need to import argparse. We’re importing SparkSession to get a hold of the running Spark session. We’ll be using user-defined functions so we need pyspark.sql.types and pyspark.sql.functions. In this example we’ll fit a linear regression model from Spark’s MLlib. The features this model needs are formatted using VectorAssembler. Finally, the model will be evaluated with the built in RegressionMetrics. 

 9 NUM_SAMPLES = 1000  # number of samples we'll generate
10
11 if __name__ == "__main__":
12
13  # arguments parsing
14  parser = argparse.ArgumentParser(
15      prog='LinearRegression',
16      description='Randomly generates data and fits a linear regression model using Spark MLlib.'
17  )
18
19    parser.add_argument('--x1', required=True, type=float)
20    parser.add_argument('--x2', required=True, type=float)
21    parser.add_argument('--max_iter', required=True, type=int)
22    args = parser.parse_args()

 9
10
11
12
13
14
15
16
17
18
19
20
21
22

NUM_SAMPLES = 1000  # number of samples we'll generate
if __name__ == "__main__":
    # arguments parsing
    parser = argparse.ArgumentParser(
        prog='LinearRegression',
        description='Randomly generates data and fits a linear regression model using Spark MLlib.'
    )
    parser.add_argument('--x1', required=True, type=float)
    parser.add_argument('--x2', required=True, type=float)
    parser.add_argument('--max_iter', required=True, type=int)
    args = parser.parse_args()

Next, define the number of rows of synthetic data we’ll generate. On lines 14-22 we set up the arguments parser so we can pass parameters to the script. This is a great feature we can use at our advantage when performing tests, and increases the modularity of our code. We’re telling the parser the names of our arguments, if they are required to run the program and their data type. Note the arguments are passed as strings on the Lambda function, and the parser casts them to the specified type.

24  # We're obtaining the spark session.
25  spark = SparkSession
26    .builder
27    .appName("SparkMLExample")
28    .getOrCreate()

24
25
26
27
28

# We're obtaining the spark session.
spark = SparkSession
    .builder
    .appName("SparkMLExample")
    .getOrCreate()

In lines 25-28 we’re obtaining the Spark Session.

30  # create a dataframe
31  num_sample = [[i] for i in range(NUM_SAMPLES)]
32  df = spark.createDataFrame(num_sample)
33 df = df.repartition(4)

30
31
32
33

 # create a dataframe
 num_sample = [[i] for i in range(NUM_SAMPLES)]
 df = spark.createDataFrame(num_sample)
 df = df.repartition(4)

Between lines 31 and 32, we’re initializing a new dataframe with numbers ranging from zero to NUM_SAMPLES – 1. Although this is not directly required, it is an easy way of initializing the dataframe for our purpose. Due to how we’re generating the dataframe, it has a single partition. If that’s the case, all the processing will be done by one machine belonging to the cluster. We’re repartitioning the dataframe into four partitions, this way, the processing will be distributed across the cluster. Note this is correct for our current example, but performing this operation on large amounts of data can result in an expensive shuffle. 

34  # generate two columns of random values
35  df = df.withColumn("rand", F.rand())
36  df = df.withColumn("rand2", F.rand())
37  print("displaying generated dataframe...")
38  df.show(10)

34
35
36
37
38

# generate two columns of random values
df = df.withColumn("rand", F.rand())
df = df.withColumn("rand2", F.rand())
print("displaying generated dataframe...")
df.show(10)

On lines 35-36 we’re generating two columns with random values using Spark’s rand function. This generates uniformly distributed random numbers in the range [0,1). We’ll use these as features for our model.

41   # calculate the linear combination of the two generated columns, using the 
42  # args x1, x2    
43  df = df.withColumn(
44   "raw_result",
45   F.udf(lambda x: args.x1 * x[0] + args.x2 * x[1], T.FloatType())
46  (F.array(F.col('rand'), F.col('rand2'))))
47
48   # generate some random noise to add to the linear combination
49  df = df.withColumn("noise", F.rand())
50 
51  # shift the noise from [0,1) to [-0.5,0.5)
52  df = df.withColumn("noise", F.udf(
53   lambda x: x - 0.5, T.FloatType())(F.col('noise')))
54
55  # add the noise to the result
56  df = df.withColumn(
57  "noisy_result",
58   F.udf(lambda x, y: x + y, T.FloatType())(F.col('raw_result'), F.col('noise')))
59
60  print("displaying generated dataframe with labels...")
61  df.show(10)

41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

 # calculate the linear combination of the two generated columns, using the 
# args x1, x2    
df = df.withColumn(
 "raw_result",
 F.udf(lambda x: args.x1 * x[0] + args.x2 * x[1], T.FloatType())
 (F.array(F.col('rand'), F.col('rand2'))))

 # generate some random noise to add to the linear combination
 df = df.withColumn("noise", F.rand())
 
# shift the noise from [0,1) to [-0.5,0.5)
 df = df.withColumn("noise", F.udf(
 lambda x: x - 0.5, T.FloatType())(F.col('noise')))

# add the noise to the result
df = df.withColumn(
"noisy_result",
 F.udf(lambda x, y: x + y, T.FloatType())(F.col('raw_result'), F.col('noise')))
print("displaying generated dataframe with labels...")
df.show(10)

On lines 43-46 we’re calculating a linear combination of the two generated columns, using the multipliers passed by arguments. We’re then generating another column with random samples called noise on line 49. As the noise is generated in the range [0,1), we’re subtracting 0.5 to shift the range to [-0.5,0.5). This noise is then added to the linear combination we created on lines 56-58. The purpose of it is to add noise to the data, because if the data was perfect, only a couple of samples would be needed to calculate the coefficients.

63  # create the LinearRegression model to be fit
64  lr = LinearRegression(featuresCol="features", labelCol="label",
65                        solver="l-bfgs", maxIter=args.max_iter)

63
64
65

# create the LinearRegression model to be fit
lr = LinearRegression(featuresCol="features", labelCol="label",
                      solver="l-bfgs", maxIter=args.max_iter)

On line 64, we instantiate a LinearRegression model from Spark’s MLlib to be fit. As the model is fitted with an input dataframe, we’re telling it which columns to search for when given a dataframe, in our case features (inputs) and label (target output).

67  # we need to format the data for the LinearRegression Model
68   vector_assembler = VectorAssembler().setInputCols(
69   ['rand', 'rand2']).setOutputCol('features') 
70  df = vector_assembler.transform(df).select(
71    "features", F.col("noisy_result").alias("label"))

67
68
69
70
71

# we need to format the data for the LinearRegression Model
vector_assembler = VectorAssembler().setInputCols(
  ['rand', 'rand2']).setOutputCol('features') 
df = vector_assembler.transform(df).select(
     "features", F.col("noisy_result").alias("label"))

MLlib’s models require the data input to be formatted in a specific way, and the VectorAssembler helps us achieve just that. We’re going from two separate columns containing float values to one DenseVector which contains both float features. In line 68 we’re instantiating the VectorAssembler and in line 70 we’re actually using it to transform our data.

73
74

# split the data into training and test set
vdf_train, df_test = df.randomSplit([0.8, 0.2])

73  # split the data into training and test set
74  vdf_train, df_test = df.randomSplit([0.8, 0.2))

As standard practice on ML problems, we’re splitting the data into training and testing. The training data is used to fit the model. The test data is used to evaluate the model on unseen data once it is fitted. It’s kind of simulating what would happen in production. We’re using Spark’s randomSplit to do this, and we’re telling it we want 80% of the data for training and 20% for testing.

76
77
78
79
80
81
82
83
84
85

# fit the model
lr_model = lr.fit(df_train)

# print some info & metrics of the fitting process
print("printing training summary info...")
trainingSummary = lr_model.summary
print(f"number of iterations: {trainingSummary.totalIterations}")
print(f"RMSE: {trainingSummary.rootMeanSquaredError}")
print(f"r2: {trainingSummary.r2}")
print("model coefficients: ", lr_model.coefficients)

76 # fit the model
77 lr_model = lr.fit(df_train)
78  
79 # print some info & metrics of the fitting process
80 print("printing training summary info...")
81 trainingSummary = lr_model.summary
82 print(f"number of iterations: {trainingSummary.totalIterations}")
83 print(f"RMSE: {trainingSummary.rootMeanSquaredError}")
84 print(f"r2: {trainingSummary.r2}")
85 print("model coefficients: ", lr_model.coefficients)

This is where the magic happens! In line 77 we’re fitting the model. In the following lines, some info and metrics of the training process are printed to stdout. Note we’re also printing the coefficients learned by the model. If things go well, these should be extremely close to those coefficients which were passed by arguments. They should be close and not exactly equal due to the noise we have introduced to the data.

87
88
89
90
91
92
93
94
95
96
97
98
99

# transform the test data
df_eval = lr_model.transform(df_test)
print("displaying predictions on evaluation data...")
df_eval.show(10)
# format the data so we can evaluate the performance of the model using
# RegressionMetrics
df_eval = df_eval.select('label', 'prediction')
metrics = RegressionMetrics(df_eval.rdd)
print("displaying metrics on test")
print(f"test data RMSE: {metrics.rootMeanSquaredError}")
print(f"test data r2: {metrics.r2}")

87 # transform the test data
88 df_eval = lr_model.transform(df_test)
89 print("displaying predictions on evaluation data...")
90 df_eval.show(10)
91 # format the data so we can evaluate the performance of the model using 92
93 # RegressionMetrics
94 df_eval = df_eval.select('label', 'prediction')
95
96  metrics = RegressionMetrics(df_eval.rdd) 
97 print("displaying metrics on test") 
98 print(f"test data RMSE: {metrics.rootMeanSquaredError}")
99 print(f"test data r2: {metrics.r2}")

 displaying generated dataframe...
+---+-------------------+--------------------+
| _1|               rand|               rand2|
+---+-------------------+--------------------+
|303| 0.4694038640011158|  0.6481209161559824|
| 95| 0.6726286905382153|  0.8981606963845883|
|161| 0.8679103354458326| 0.18119671605635224|
|448|0.30126976391012883|  0.7447454830462397|
|170|  0.864527171418423| 0.07967898764743175|
|131| 0.8658936796366256| 0.19843634271437316|
| 90| 0.9212755414082087|  0.1328917388102402|
|321| 0.4444531531651619| 0.45985908905674244|
| 26| 0.8636854530294522| 0.26834470199928806|
|447| 0.9480941000568025|0.036768371545385814|
+---+-------------------+--------------------+
only showing top 10 rows
displaying generated dataframe with labels...
+---+-------------------+--------------------+----------+-----------+------------+
| _1|               rand|               rand2|raw_result|      noise|noisy_result|
+---+-------------------+--------------------+----------+-----------+------------+
|303| 0.4694038640011158|  0.6481209161559824| 11.175248| 0.06108435|   11.236333|
| 95| 0.6726286905382153|  0.8981606963845883| 15.707894|-0.12919985|   15.578694|
|161| 0.8679103354458326| 0.18119671605635224| 10.491071|  0.3945236|   10.885594|
|448|0.30126976391012883|  0.7447454830462397| 10.460153| -0.0810913|   10.379062|
|170|  0.864527171418423| 0.07967898764743175|  9.442061| 0.43645015|    9.878511|
|131| 0.8658936796366256| 0.19843634271437316|   10.6433|-0.07997194|   10.563328|
| 90| 0.9212755414082087|  0.1328917388102402| 10.541673|0.030807901|    10.57248|
|321| 0.4444531531651619| 0.45985908905674244|  9.043122|-0.29848525|   8.7446375|
| 26| 0.8636854530294522| 0.26834470199928806| 11.320302|-0.43192774|   10.888374|
|447| 0.9480941000568025|0.036768371545385814|  9.848625| 0.11807168|    9.966697|
+---+-------------------+--------------------+----------+-----------+------------+
only showing top 10 rows
printing training summary info...
number of iterations: 4
RMSE: 0.28385880311787814
r2: 0.9954688447304679
model coefficients:  [10.000646699625818,10.005967687529866]
displaying predictions on evaluation data...
+--------------------+---------+------------------+
|            features|    label|        prediction|
+--------------------+---------+------------------+
|[0.00834606519633...|6.4147186|6.2857339058323545|
|[0.06440724048404...| 8.307216| 8.800616841502176|
|[0.11552795484457...| 7.149164| 6.903089763368193|
|[0.12749768801334...|2.6500432|2.8610826921022343|
|[0.16084911790794...| 2.955299| 2.606324673004222|
|[0.18106191852777...| 8.683829| 8.508712313017005|
|[0.19052191329248...|10.528968| 10.22865986628032|
|[0.19501743199211...|11.151053| 11.18943875216736|
|[0.20326177358481...| 9.618211| 9.652048794087703|
|[0.21711604751733...|10.206596|10.217273926029563|
+--------------------+---------+------------------+
only showing top 10 rows
displaying metrics on test
test data RMSE: 0.2923418543882163
test data r2: 0.9945363063793037

Let’s take a glance at the script’s output. The first dataframe being displayed is the result of line 38, and contains the randomly generated inputs. The second one is the result of line’s 61 execution, where we’re displaying the randomly generated values, the linear combination of those, randomly generated noise and the addition of the linear combination and the noise. Training information is then printed, including the number of iterations of the linear regression algorithm, metrics such as RMSE and r2, and the calculated model coefficients. These should be very close to the ones we’re using as inputs x1 and x2. Finally, a portion of the test dataframe is shown, including the inputs (features), the label and the model’s prediction. Metrics on the test data should closely resemble the ones for training.

Conclusion

In this first part of the series, we reviewed the high level architecture we’ll be using to run distributed data processing on AWS. For illustration purposes, we set up an extremely simple example script using pySpark, the Spark API for Python. We reviewed the script and what it does.

In part 2, we’ll dive into the details of running this script on EMR. We’ll explain the technologies to be used and set up the necessary infrastructure to run the example.

By: Maximiliano Palay and Tiziana Romani

Stay ahead of the curve on the latest trends and insights in big data, machine learning and artificial intelligence. Don’t miss out and subscribe to our newsletter!

At the latest edition of AWS re:Invent 2022, Swami Sivasubramanian (VP of database, analytics, and ML) announced Athena’s support for Apache Spark and we were really excited about it!

As proficient users of Apache Spark the headline was very well received, but we were curious about various aspects:

• Is it as easy to use as the traditional tool (mostly for SQL queries)?
• Does it have a similar interface? 
• How does it differ from what you can do in Glue Studio or in SageMaker notebooks?
• Does it support all Spark components?
• Is it as fast and efficient as the traditional query-based engine?
• What are some of its limitations?

So we went through the process of using Apache Spark for Athena so as to try to answer the questions above. Let’s first go through the steps needed to set up Athena for Apache Spark.

Setting up Athena for Apache Spark

Getting Started 

Once you get to the console’s Athena page in your region (e.g.  https://us-east-1.console.aws.amazon.com/athena/home?region=us-east-1#/landing-page), the first step is to select “Analyze your data” (as opposed to the traditional option for “Query your data”). If this is your first time using Athena for Spark, then you will need to create a workflow and choose Apache Spark:

If you already have an IAM Role suitable for Athena for Spark, you can select it. Otherwise, as in our case, AWS will create a new role with all the permissions necessary to get started. 

Such a role will have permissions to write data to a log bucket, athena access itself, and the ability to write to CloudWatch logs. In our case the role created was named AWSAthenaSparkExecutionRole-l5mlv874cc.

Once you have the workgroup created, you can select it:

Then we proceeded to create a notebook within the workgroup (a workgroup can have multiple notebooks):

Once you have created the notebook the typical jupyter-like interface will be available:

Running Spark Code

As in other AWS notebook systems the implicit spark variable will reference a Spark session that will be created on-the-fly for you:

On our first attempt to read a dataset from a CSV file stored in S3 we got access denied error:

This is due to the fact that the role we’re using for Athena does not have access to the data in the bucket where the dataset lies. However, since we know which IAM Role that is (AWSAthenaSparkExecutionRole-l5mlv874cc in our case), we can look it up in the IAM console and add the additional policy entries (by editing the existing policy or adding a new one). In our case we just edited the existing policy and added the bucket we have our dataset on:

Now we’re able to load the data and start running queries. In the screenshot below we run a Spark describe command on the dataset. 

Note:

The dataset we used here is the one we used on Chapter 4 of our book Mastering Machine Learning on AWS (you can find an easy access link here: https://www.amazon.com/Mastering-Machine-Learning-AWS-TensorFlow/dp/1789349796)which has rows representing ad impressions in different devices, each of which may or may not have resulted in users clicking on such impressions.  Such a dataset has a size of 6GB. 

Each cell execution triggers a calculation, which in turn may result in AWS launching Spark nodes in the background.  These nodes will incur in data processing costs measured by Data Processing Units (DPUs) which are the basis for AWS to charge us for this service. While we don’t control the kind or number of nodes in the background, we can set a maximum number of DPUs attached to the notebook session by editing the session details. Note that AWS does not charge us for the use of the notebook itself. It will only bill us the usage of DPUs from both the notebook’s Spark driver as well as the Spark worker nodes.  

The Jupyter notebook supports the convenient magic %%sql command for us to run SQL directly in the notebook. For that we can register a table name from a dataframe as follows:

The above query shows how many impressions resulted in clicks vs how many impressions we just showed but not clicked into. 

Integrating Pandas and Plotting

Additionally one can run heavy Spark queries that aggregate results in smaller datasets which in turn can be transformed into Pandas for convenient analysis and plotting. 

First we construct a Spark dataframe by running a SQL aggregation query. Then we transform the Spark dataframe to Pandas:

The Pandas dataframe can be viewed as usual:

As you can see in the cell above, an attempt to plot the pandas Dataframe did not show the graph! This is because we need to explicitly clear the pyplot current figure and call the %matplot plt module. In the cell below we show how this is done:

We found it surprising that this is explicitly required, but not a big problem. 

Tracking Calculations

In the session information of the notebook we can find all the calculations that were triggered as well as the duration. If you observe the table below (as well as each cell’s output) the durations are always just a few seconds. This is impressive considering our dataset was several gigs in size! 

Limitations

Our original goal was to be able to run Spark machine learning pipelines within Athena. However we later found that this is not supported, as described in Athena’s documentation:

MLlib (Apache Spark machine learning library) is not supported. For a list of supported Python libraries, see the List of preinstalled Python libraries.

An attempt to import pyspark.ml will result in python not resolving such a library. 

Takeaways

These are the main takeaways from our first steps towards using Apache Spark within Athena. 

• Athena for Spark is a great tool to quickly run data-wrangling notebooks and quick analysis with very little set up.
• Spark MLlib is not yet supported. If you want to run machine learning  jobs, we recommend you use SageMaker, EMR, or Glue. 
• As opposed to SageMaker (and Glue Notebooks), you don’t need to explicitly provision notebooks for your analysis, nor you will be charged for the use of the notebook itself However you will be charged for the DPUs of the jobs you trigger when you run the cells on your notebook. 
• Athena for Spark is extremely fast and the job provisioning is a very smooth experience. 
• It’s easy to re-open old notebooks without the need to provision a notebook server. 
• Compared to Glue Studio, we found Athena for Spark to be much simpler to use and faster to set up. However, it comes with some limitations (such as supported libraries). 

Interested in exploring the use of Apache Spark on Athena? As AWS Partners our team at Montevideo Labs has extensive experience with AWS services at scale. Contact our team to learn how we can help you in your cloud journey!

By: Montevideo Labs Engineering Team