Building AWS Standard Step Functions with enabled CloudWatch Logs using a SAM template

Hey there, dear reader! As my first post I would like to share with you some knowledge on building serverless applications in AWS through SAM/CloudFormation templates.

It is most likely that you’re here because you are either a cloud enthusiast or you’re struggling with enabling CloudWatch Logs to your standard Step Function through your SAM/CloudFormation template. Well, if the latter is the case then search no more! Here, I present to you a step-by-step guide on how to do it. On the other side, if you are just looking on some new cloud content then I hope this article will allow you to discover new features within AWS Step Functions.

Why?

First of all, because AWS is awesome! The amount of services offered is huge and allows you to leverage the power of your cloud applications. Being a service used by millions, it is useful to document about how to implement this, for future reference.

At the beggining of the year, AWS anounced that this would be a supported feaure (see here). However, I had been struggling on how to enable CloudWatch logs in Step Functions using an AWS SAM template. The lack of examples in the community imposed me an obstacle in which I spent a significant amount of time trying to figure out how to do it.

With this post I present you a simple step-by-step guide on how to do this so that people can quickly have their state machines logging to CloudWatch.

Project Requirements

We will build a standard step function’s state machine which will trigger a Lambda and will log all execution-related events to CloudWatch.

Prerequisites

This guide is mostly focused to people who have already used AWS, more specifically AWS Step Functions. However, if you don’t have knowledge on this I suggest you to check the AWS Step Functions docs page, which explains what step functions is in a very clear way, and also the AWS CloudWatch docs page plus the AWS SAM page.

In terms of software, you will need the following:

• AWS CLI (installation guide): management of your AWS account.
• AWS SAM CLI (installation guide): management of your SAM application.
• NodeJS (installation guide): runtime environment for the SAM application

The installation guides are really easy to follow and are very well-written. It shouldn’t take you more than 15 minutes to get these up and running.

Simple SAM-app creation

First, let’s create a new NodeJS SAM application. The AWS SAM CLI allows us to do this in a very simple and effective manner. Just follow the commands below.

$ sam init
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1
...
Runtime: 1
Project name [sam-app]:
Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git
AWS quick start application templates:
        1 - Hello World Example
        2 - Quick Start: From Scratch
        3 - Quick Start: Scheduled Events
        4 - Quick Start: S3
        5 - Quick Start: SNS
        6 - Quick Start: SQS
        7 - Quick Start: Web Backend
Template selection: Template selection: 1
-----------------------
Generating application:
-----------------------
Name: sam-app
Runtime: nodejs12.x
Dependency Manager: npm
Application Template: hello-world
Output Directory: .
Next steps can be found in the README file at ./sam-app/README.md

After you run this you should have a new folder created with the name of the project you created (in my case I left it named with the default “sam-app”).

Hooray! We already have a simple local SAM application! Now let’s dive into the creation of the resources we want.

Creation of a Lambda Function

Go into the folder created and check the template.yaml file. You will see that there are some resources and outputs already defined. You can go ahead and delete the outputs as those will not be necessary for our application.

You shall be left with a lambda in the resources section with the below definition:

HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

What this is doing is defining a new Lambda that is specifying an event which will trigger its execution — in this case a GET request to the /hello resource of an API Gateway. Well, we don’t need this as we will not use an API Gateway in our application, so make sure to remove the “Events” property.

So here we are... Stuck with a single Lambda. Well, this is actually good as we want a Lambda to be process during the workflow execution. Great! So this will be our “ProcessingLambda”. Just rename the resource and you will end up with something like the following:

ProcessingLambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x

Mapping Definition

I decided to define a Mapping section so we don’t need to repeat hardcoded values throught the template. This is a feature that allows us to define keys for sets of named values.

Mappings:  
  StepFunctionsNames:    
    ProcessingStateMachine:      
      "Name": "Process-Lambda-State-Machine"

Note that this is not mandatory but highly recommendable in bigger template files as allows you to reference the key name instead of repeating its value everywhere.

Creation of the Step Function Resource

Alright! Right now, we have a mapping section and the lambda. Ooofff… We are almost there! On to the definition of the step function…

First, note that currently AWS SAM does not support state machines definitions directly but (from the docs):

Because AWS SAM is an extension of AWS CloudFormation, you get the reliable deployment capabilities of AWS CloudFormation. You can define resources by using AWS CloudFormation in your AWS SAM template.

This is awesome because StateMachine is supported by CloudFormation! This way, we can indeed define state machines and they will be successfully created.

Below is our state machine resource definition. You can see it has a single task state which will invoke the lambda we created before. So, for every execution of the state machine, it will just invoke the previously defined lambda and finish.

Notice that an execution role is also present. This will define what permissions our state machine has to do things. In this case, we have defined that it has permissions to invoke lambdas and access/change CloudWatch Logs.

StateMachine:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      StateMachineName: !FindInMap [ StepFunctionsNames, ProcessingStateMachine, Name ]
      DefinitionString:
        Fn::Sub:
          - |-
            {
              "Comment": "A simple workflow to invoke a lambda upon execution.",
              "StartAt": "HelloWorld",
              "States": {
                "HelloWorld": {
                  "Type": "Task",
                  "Resource": "${ProcessingLambdaArn}",
                  "End": true
                }
              }
            }
          - {
            ProcessingLambdaArn: !GetAtt [ ProcessingLambda, Arn ]
          }
      RoleArn:
        Fn::GetAtt: [ StatesExecutionRole, Arn ]
# State Machines Execution Role
StatesExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - !Sub states.${AWS::Region}.amazonaws.com
            Action: "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: StatesExecutionPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "lambda:InvokeFunction"
                  - "cloudwatch:*"
                  - "logs:*"
                Resource: "*"

Now, this is pretty cool but if you deploy this you won’t have any CloudWatch logs when executing the state machine. Why you ask? Well, at the moment we have just defined permissions for the state machine to log to CloudWatch but we haven’t actually stated that we wanted to log.

Configure State Machine Logging Properties

To define our state machine to log to CloudWatch, we have to create a new resource: a LogGroup. This will basically host log streams. In our case, we want to create a new log group to host all of our state machine related logs.

To do this, we must create the new resource, giving it an intuitive name:

ProcessingStateMachineLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Join [ "/", [ "stepfunctions", !FindInMap [ StepFunctionsNames, ProcessingStateMachine, Name ]]]

Ok, great! Almost there! Now we just have to let the state machine know where to write the logs to. For this, we need to add the following LoggingConfiguration property to the state machine resource definition so that it can start to log to the previously defined log group in CloudWatch:

DependsOn: ProcessingStateMachineLogGroup
...
Properties:
  LoggingConfiguration:
    Destinations:
      - CloudWatchLogsLogGroup:
         LogGroupArn: !GetAtt [ProcessingStateMachineLogGroup, Arn]
    IncludeExecutionData: false
    Level: 'ALL'
  ...

Note that there is a DependsOn property that we have specified. This is to state that this resource (the state machine) shall be created only after the ProcessingStateMachineLogGroup resource is created, to avoid referencing a resource that is not already created.

The level property is customizable. In our case, we will log all execution events. However, you can choose which events you would like to log. Check the docs to see what other options are available.

Final Structure

So these were the required resources to allow your state machine to log to CloudWatch. Your template file should now look like the following:

Deploy & Verify

That’s it! This was all it took for enabling CloudWatch logs in a simple Step Function workflow using a SAM template. Now, let’s push the changes so we can actually test this live.

To do so, we must execute the following operations:

• sam build: this command will build the application, and prepares it for subsequent steps in your workflow.

$ sam build
...
Build Succeeded
...

• sam deploy: deploys the application to your AWS account. As you can see there is the “ — guided” flag present. This allows you to enter in a guided interactive mode.

$ sam deploy --guided
...
Successfully created/updated stack - sam-app in us-east-1

Note: If you see the “Error: Security Constraints Not Satisfied” make sure to enter “y” when you get prompted for the following message:

“HelloWorldFunction may not have authorization defined, Is this okay? [y/N]”

otherwise it will fail. See this more information.

If everything was successful, the resources were created. To verify it, log in to your AWS account and verify that you are in the correct region (in my case, us-east-1). Afterwards, check that your resources were successfully created:

Fig. 1 — New State Machine

Fig. 2 — New Log Group

Final Result

By this point, we already have the resources created. Now, we want to make sure that the logs are successfully written to CloudWatch upon the state machine’s execution:

1- Create a new state machine execution.

Fig. 3 — Created State Machine Execution

Fig. 4 — State Machine Execution Events

2 - Check that logs were written to CloudWatch log group.

Fig. 5 — CloudWatch Logs of State Machine Execution

Note: Remember that we specified in the LoggingConfiguration property of the state machine that we wanted all events to be logged. If we wanted, we could have selected other option to choose which events to be logged.

… And that’s it! We have successfully managed to enable CloudWatch logs in a standard step function workflow. Now, you are free to enable CloudWatch logs in other state machines that you’d like to create :)

Hope you found this article useful and that it helped you to correctly enable CloudWatch Logs in a Standard Step Function Workflow!