Enterprise Java

AWS Lambda Event Source Mappings: bringing your triggers to order from chaos

Recently we introduced two new AWS Lambda event sources (trigger types) for your serverless projects on Sigma cloud IDE: SQS queues and DynamoDB Streams. (Yup, AWS introduced them months ago; but we’re still a tiny team, caught up in a thousand and one other things as well!)

While developing support for these triggers, I noticed a common (and yeah, pretty obvious) pattern on Lambda event source trigger configurations; that I felt was worth sharing.

Why AWS Lambda triggers are messed up

Lambda – or rather AWS – has a rather peculiar and disorganized trigger architecture; to put it lightly. For different trigger types, you have to put up configurations all over the place; targets for CloudWatch Events rules, integrations for API Gateway endpoints, notification configurations for S3 bucket events, and the like. Quite a mess, considering other platforms like GCP where you can configure everything in one place: the “trigger” config of the actual target function.

Configs. Configs. All over the place.

If you have used infrastructure as code (IAC) services like CloudFormation (CF) or Terraform (TF), you would already know what I mean. You need mappings, linkages, permissions and other bells and whistles all over the place to get even a simple HTTP URL working. (SAM does simplify this a bit, but it comes with its own set of limitations – and we have tried our best to avoid such complexities in our Sigma IDE.)

Maybe this is to be expected, given the diversity of services offered by AWS, and their timeline (Lambda, after all, is just a four-year-old kid). AWS should surely have had to do some crazy hacks to support triggering Lambdas from so many diverse services; and hence the confusing, scattered configurations.

Event Source Mappings: light at the end of the tunnel?

AWS Lambda Event Source

Luckily, the more recently introduced, stream-type triggers follow a common pattern:

This way, you know exactly where you should configure the trigger, and how you should allow the Lambda to consume the event stream.

No more jumping around.

This is quite convenient when you are based on an IAC like CloudFormation:

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
{
  ...
 
    // event source (SQS queue)
 
    "sqsq": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "DelaySeconds": 0,
        "MaximumMessageSize": 262144,
        "MessageRetentionPeriod": 345600,
        "QueueName": "q",
        "ReceiveMessageWaitTimeSeconds": 0,
        "VisibilityTimeout": 30
      }
    },
 
    // event target (Lambda function)
 
    "tikjs": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "FunctionName": "tikjs",
        "Description": "Invokes functions defined in \
tik/js.js in project tik. Generated by Sigma.",
        ...
      }
    },
 
    // function execution role that allows it (Lambda service)
    // to query SQS and remove read messages
 
    "tikjsExecutionRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "ManagedPolicyArns": [
          "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        ],
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Action": [
                "sts:AssumeRole"
              ],
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "lambda.amazonaws.com"
                ]
              }
            }
          ]
        },
        "Policies": [
          {
            "PolicyName": "tikjsPolicy",
            "PolicyDocument": {
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "sqs:GetQueueAttributes",
                    "sqs:ReceiveMessage",
                    "sqs:DeleteMessage"
                  ],
                  "Resource": {
                    "Fn::GetAtt": [
                      "sqsq",
                      "Arn"
                    ]
                  }
                }
              ]
            }
          }
        ]
      }
    },
 
    // the actual event source mapping (SQS queue -> Lambda)
 
    "sqsqTriggertikjs0": {
      "Type": "AWS::Lambda::EventSourceMapping",
      "Properties": {
        "BatchSize": "10",
        "EventSourceArn": {
          "Fn::GetAtt": [
            "sqsq",
            "Arn"
          ]
        },
        "FunctionName": {
          "Ref": "tikjs"
        }
      }
    },
 
    // grants permission for SQS service to invoke the Lambda
    // when messages are available in our queue
 
    "sqsqPermissiontikjs": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "Action": "lambda:InvokeFunction",
        "FunctionName": {
          "Ref": "tikjs"
        },
        "SourceArn": {
          "Fn::GetAtt": [
            "sqsq",
            "Arn"
          ]
        },
        "Principal": "sqs.amazonaws.com"
      }
    }
 
  ...
}

(In fact, that was the whole reason/purpose of this post.)

Tip: You do not need to worry about this whole IAC/CloudFormation thingy – or writing lengthy JSON/YAML – if you go with a fully automated resource management tool like SLAppForge Sigma serverless cloud IDE.

But… are Event Source Mappings ready for the big game?

AWS Lambda Event Source

They sure look promising, but it seems event source mappings do need a bit more maturity, before we can use them in fully automated, production environments.

You cannot update an event source mapping via IAC.

For example, even after more than four years from their inception, event sources cannot be updated after being created via an IaC like CloudFormation or Serverless Framework. This causes serious trouble; if you update the mapping configuration, you need to manually delete the old one and deploy the new one. Get it right the first time, or you’ll have to run through a hectic manual cleanup to get the whole thing working again. So much for automation!

The event source arn (aaa) and function (bbb) provided mapping already exists. Please update or delete the existing mapping...

Polling? Sounds old-school.

There are other, less-evident problems as well; for one, event source mappings are driven by polling mechanisms. If your source is an SQS queue, the Lambda service will keep polling it until the next message arrives. While this is fully out of your hands, it does mean that you pay for the polling. Plus, as a dev, I don’t feel that polling exactly fits into the event-driven, serverless paradigm. Sure, everything boils down to polling in the end, but still…

In closing: why not just try out event source mappings?

AWS Lambda Event Source

Ready or not, looks like event source mappings are here to stay. With the growing popularity of data streaming (Kinesis), queue-driven distributed processing and coordination (SQS) and event ledgers (DynamoDB Streams), they will become ever more popular as time passes.

You can try out how event source mappings work, via numerous means: the AWS console, aws-cli, CloudFormation, Serverless Framework, and the easy-as-pie graphical IDE SLAppForge Sigma.

Easily manage your event source mappings – with just a drag-n-drop!

In Sigma IDE you can simply drag-n-drop an event source (SQS queue, DynamoDB table or Kinesis stream) on to the event variable of your Lambda function code. Sigma will pop-up a dialog with available mapping configurations, so you can easily configure the source mapping behavior. You can even configure an entirely new source (queue, table or stream) instead of using an existing one, right there within the pop-up.

AWS Lambda Event Source

When deployed, Sigma will auto-generate all necessary configurations and permissions for your new event source, and publish them to AWS for you. It’s all fully managed, fully automated and fully transparent.

Enough talk. Let’s start!

Published on Java Code Geeks with permission by Janaka Bandara, partner at our JCG program. See the original article here: AWS Lambda Event Source Mappings: bringing your triggers to order from chaos

Opinions expressed by Java Code Geeks contributors are their own.

Janaka Bandara

A budding software engineer in the enterprise integration space, and a simplistic guy who likes to learn, appreciate, help, share, and enjoy life
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button