Monday, October 3, 2022
HomeBig DataRun an information processing job on Amazon EMR Serverless with AWS Step...

Run an information processing job on Amazon EMR Serverless with AWS Step Capabilities

There are a number of infrastructure as code (IaC) frameworks out there at the moment, that can assist you outline your infrastructure, such because the AWS Cloud Improvement Package (AWS CDK) or Terraform by HashiCorp. Terraform, an AWS Companion Community (APN) Superior Know-how Companion and member of the AWS DevOps Competency, is an IaC software just like AWS CloudFormation that permits you to create, replace, and model your AWS infrastructure. Terraform supplies pleasant syntax (just like AWS CloudFormation) together with different options like planning (visibility to see the adjustments earlier than they really occur), graphing, and the flexibility to create templates to interrupt infrastructure configurations into smaller chunks, which permits higher upkeep and reusability. We use the capabilities and options of Terraform to construct an API-based ingestion course of into AWS. Let’s get began!

On this put up, we showcase methods to construct and orchestrate a Scala Spark utility utilizing Amazon EMR Serverless, AWS Step Capabilities, and Terraform. On this end-to-end answer, we run a Spark job on EMR Serverless that processes pattern clickstream knowledge in an Amazon Easy Storage Service (Amazon S3) bucket and shops the aggregation leads to Amazon S3.

With EMR Serverless, you don’t should configure, optimize, safe, or function clusters to run purposes. You’ll proceed to get the advantages of Amazon EMR, similar to open supply compatibility, concurrency, and optimized runtime efficiency for well-liked knowledge frameworks. EMR Serverless is appropriate for purchasers who need ease in working purposes utilizing open-source frameworks. It provides fast job startup, computerized capability administration, and easy value controls.

Answer overview

We offer the Terraform infrastructure definition and the supply code for an AWS Lambda perform utilizing pattern buyer consumer clicks for on-line web site inputs, that are ingested into an Amazon Kinesis Information Firehose supply stream. The answer makes use of Kinesis Information Firehose to transform the incoming knowledge right into a Parquet file (an open-source file format for Hadoop) earlier than pushing it to Amazon S3 utilizing the AWS Glue Information Catalog. The generated output S3 Parquet file logs are then processed by an EMR Serverless course of, which outputs a report detailing mixture clickstream statistics in an S3 bucket. The EMR Serverless operation is triggered utilizing Step Capabilities. The pattern structure and code are spun up as proven within the following diagram.

emr serverless application

The offered samples have the supply code for constructing the infrastructure utilizing Terraform for working the Amazon EMR utility. Setup scripts are offered to create the pattern ingestion utilizing Lambda for the incoming utility logs. For the same ingestion sample pattern, check with Provision AWS infrastructure utilizing Terraform (By HashiCorp): an instance of net utility logging buyer knowledge.

The next are the high-level steps and AWS providers used on this answer:

  • The offered utility code is packaged and constructed utilizing Apache Maven.
  • Terraform instructions are used to deploy the infrastructure in AWS.
  • The EMR Serverless utility supplies the choice to submit a Spark job.
  • The answer makes use of two Lambda capabilities:
    • Ingestion – This perform processes the incoming request and pushes the information into the Kinesis Information Firehose supply stream.
    • EMR Begin Job – This perform begins the EMR Serverless utility. The EMR job course of converts the ingested consumer click on logs into output in one other S3 bucket.
  • Step Capabilities triggers the EMR Begin Job Lambda perform, which submits the appliance to EMR Serverless for processing of the ingested log recordsdata.
  • The answer makes use of 4 S3 buckets:
    • Kinesis Information Firehose supply bucket – Shops the ingested utility logs in Parquet file format.
    • Loggregator supply bucket – Shops the Scala code and JAR for working the EMR job.
    • Loggregator output bucket – Shops the EMR processed output.
    • EMR Serverless logs bucket – Shops the EMR course of utility logs.
  • Pattern invoke instructions (run as a part of the preliminary setup course of) insert the information utilizing the ingestion Lambda perform. The Kinesis Information Firehose supply stream converts the incoming stream right into a Parquet file and shops it in an S3 bucket.

For this answer, we made the next design choices:

  • We use Step Capabilities and Lambda on this use case to set off the EMR Serverless utility. In a real-world use case, the information processing utility could possibly be lengthy working and should exceed Lambda’s timeout limits. On this case, you should use instruments like Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Amazon MWAA is a managed orchestration service makes it simpler to arrange and function end-to-end knowledge pipelines within the cloud at scale.
  • The Lambda code and EMR Serverless log aggregation code are developed utilizing Java and Scala, respectively. You should utilize any supported languages in these use circumstances.
  • The AWS Command Line Interface (AWS CLI) V2 is required for querying EMR Serverless purposes from the command line. You may as well view these from the AWS Administration Console. We offer a pattern AWS CLI command to check the answer later on this put up.


To make use of this answer, you need to full the next stipulations:

  • Set up the AWS CLI. For this put up, we used model 2.7.18. That is required with the intention to question the aws emr-serverless AWS CLI instructions out of your native machine. Optionally, all of the AWS providers used on this put up could be considered and operated through the console.
  • Be sure to have Java put in, and JDK/JRE 8 is about within the setting path of your machine. For directions, see the Java Improvement Package.
  • Set up Apache Maven. The Java Lambda capabilities are constructed utilizing mvn packages and are deployed utilizing Terraform into AWS.
  • Set up the Scala Construct Software. For this put up, we used model 1.4.7. Be sure to obtain and set up primarily based in your working system wants.
  • Arrange Terraform. For steps, see Terraform downloads. We use model 1.2.5 for this put up.
  • Have an AWS account.

Configure the answer

To spin up the infrastructure and the appliance, full the next steps:

  1. Clone the next GitHub repository.
    The offered shell script builds the Java utility JAR (for the Lambda ingestion perform) and the Scala utility JAR (for the EMR processing) and deploys the AWS infrastructure that’s wanted for this use case.
  2. Run the next instructions:
    $ chmod +x
    $ ./

    To run the instructions individually, set the appliance deployment Area and account quantity, as proven within the following instance:

    $ APP_DIR=$PWD
    $ APP_PREFIX=clicklogger
    $ STAGE_NAME=dev
    $ REGION=us-east-1
    $ ACCOUNT_ID=$(aws sts get-caller-identity | jq -r '.Account')

    The next is the Maven construct Lambda utility JAR and Scala utility bundle:

    $ cd $APP_DIR/supply/clicklogger
    $ mvn clear bundle
    $ sbt reload
    $ sbt compile
    $ sbt bundle

  3. Deploy the AWS infrastructure utilizing Terraform:
    $ terraform init
    $ terraform plan
    $ terraform apply --auto-approve

Take a look at the answer

After you construct and deploy the appliance, you may insert pattern knowledge for Amazon EMR processing. We use the next code for instance. The script has a number of pattern insertions for Lambda. The ingested logs are utilized by the EMR Serverless utility job.

The pattern AWS CLI invoke command inserts pattern knowledge for the appliance logs:

aws lambda invoke --function-name clicklogger-dev-ingestion-lambda —cli-binary-format raw-in-base64-out —payload '{"requestid":"OAP-guid-001","contextid":"OAP-ctxt-001","callerid":"OrderingApplication","part":"login","motion":"load","sort":"webpage"}' out

To validate the deployments, full the next steps:

  1. On the Amazon S3 console, navigate to the bucket created as a part of the infrastructure setup.
  2. Select the bucket to view the recordsdata.
    You need to see that knowledge from the ingested stream was transformed right into a Parquet file.
  3. Select the file to view the information.
    The next screenshot exhibits an instance of our bucket contents.
    Now you may run Step Capabilities to validate the EMR Serverless utility.
  4. On the Step Capabilities console, open clicklogger-dev-state-machine.
    The state machine exhibits the steps to run that set off the Lambda perform and EMR Serverless utility, as proven within the following diagram.
  5. Run the state machine.
  6. After the state machine runs efficiently, navigate to the clicklogger-dev-output-bucket on the Amazon S3 console to see the output recordsdata.
  7. Use the AWS CLI to examine the deployed EMR Serverless utility:
    $ aws emr-serverless list-applications 
          | jq -r '.purposes[] | choose(.title=="clicklogger-dev-loggregrator-emr-<Your-Account-Quantity>").id'

  8. On the Amazon EMR console, select Serverless within the navigation pane.
  9. Choose clicklogger-dev-studio and select Handle purposes.
  10. The Software created by the stack will probably be as proven beneath clicklogger-dev-loggregator-emr-<Your-Account-Quantity>
    Now you may evaluate the EMR Serverless utility output.
  11. On the Amazon S3 console, open the output bucket (us-east-1-clicklogger-dev-loggregator-output-).
    The EMR Serverless utility writes the output primarily based on the date partition, similar to 2022/07/28/ next code exhibits an instance of the file output:

Clear up

The offered ./ script has the required steps to delete all of the recordsdata from the S3 buckets that have been created as a part of this put up. The terraform destroy command cleans up the AWS infrastructure that you simply created earlier. See the next code:

$ chmod +x
$ ./

To do the steps manually, it’s also possible to delete the assets through the AWS CLI:

# CLI Instructions to delete the Amazon S3  

aws s3 rb s3://clicklogger-dev-emr-serverless-logs-bucket-<your-account-number> --force
aws s3 rb s3://clicklogger-dev-firehose-delivery-bucket-<your-account-number> --force
aws s3 rb s3://clicklogger-dev-loggregator-output-bucket-<your-account-number> --force
aws s3 rb s3://clicklogger-dev-loggregator-source-bucket-<your-account-number> --force
aws s3 rb s3://clicklogger-dev-loggregator-source-bucket-<your-account-number> --force

# Destroy the AWS Infrastructure 
terraform destroy --auto-approve


On this put up, we constructed, deployed, and ran an information processing Spark job in EMR Serverless that interacts with varied AWS providers. We walked by way of deploying a Lambda perform packaged with Java utilizing Maven, and a Scala utility code for the EMR Serverless utility triggered with Step Capabilities with infrastructure as code. You should utilize any mixture of relevant programming languages to construct your Lambda capabilities and EMR job utility. EMR Serverless could be triggered manually, automated, or orchestrated utilizing AWS providers like Step Capabilities and Amazon MWAA.

We encourage you to check this instance and see for your self how this general utility design works inside AWS. Then, it’s simply the matter of changing your particular person code base, packaging it, and letting EMR Serverless deal with the method effectively.

If you happen to implement this instance and run into any points, or have any questions or suggestions about this put up, please depart a remark!


Concerning the Authors

Sivasubramanian Ramani (Siva Ramani) is a Sr Cloud Software Architect at Amazon Net Providers. His experience is in utility optimization & modernization, serverless options and utilizing Microsoft utility workloads with AWS.

Naveen Balaraman is a Sr Cloud Software Architect at Amazon Net Providers. He’s keen about Containers, serverless Functions, Architecting Microservices and serving to clients leverage the ability of AWS cloud.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments