Building Serverless Apps with AWS, Lambda, Python & Zappa

This article was published on 5/11/2018 in TechBeacon.

Serverless architecture has quickly emerged as the up-and-coming way to develop cloud-based applications. The benefits of serverless computing—zero server maintenance, out-of-the-box scalability, and ground-up support for asynchronous workflows—are just too good to ignore. It’s a developer’s dream.

But serverless architecture comes with a catch. These operational advantages carry a hefty price for developers: laborious configuration, immature frameworks, a lack of tooling, and a dearth of established design patterns. Until recently, going serverless meant throwing out decades of established tools and practices and starting over. In a nutshell, making serverless apps can be painful.

That’s why I selected the open-source Zappa as my serverless Python framework on a recent project for a dotcom start-up. I wanted to eliminate the burden of server maintenance for the young company and to design a cost-effective cloud architecture that was cheap when traffic was low, and would scale as the company grew. But I wanted these benefits without the unpleasant side effects that I’d encountered in the past when working on serverless apps. After doing the research, it became clear that Zappa embodied the culmination of the progress that serverless technology has made over the last 18 months. Here’s why.

Serverless evolution: From SAM to Zappa

The serverless evolution began when Amazon Web Services (AWS) open sourced its Serverless Application Model (SAM). SAM is a simplified template syntax for CloudFormation, AWS’s infrastructure-as-code framework, that reduced the jaw-dropping amount of code previously required to automate the deployment of serverless apps.

SAM made configuration a little better for some uses, but it was only an incremental improvement. The fundamental problem still remained: Developing applications as Lambda functions required starting over with new design patterns and Lambda-compatible libraries. For the Python crowd, this meant kissing time-saving application frameworks such as Django and Flask goodbye.

Fortunately, the universe of supporting tools and frameworks has evolved rapidly thanks to another AWS release, version 1.0 of Chalice, a Python framework and command-line utility for serverless application development. Chalice provides some neat features, such as automatically generating an IAM security policy based on an inspection of your code (this is amazing, I’m not going to lie), but it still doesn’t provide the application scaffolding that we’ve come to rely on for rapid application development.

Enter Zappa, an open source project maintained by the smart folks at gun.io. Zappa is a full-featured command line tool that Lambda-fies Flask and Django applications. Technically speaking, it’s a framework that packages and deploys WSGI-compatible Python applications to an AWS Lambda function. But for all practical purposes, it’s an Easy Button for Flask and Django serverless deployment.

With Zappa, you can have your cake and eat it too. You can develop applications locally using your favorite IDE, continue writing code using the Python libraries that you’ve come to rely on, and utilize best practice design patterns. Then, when it’s time to deploy, you just run zappa deploy at the command line to package up your traditional app and deploy it to AWS Lambda. Voilà!

Now, of course, there is no such thing as a free lunch. Like any framework, Zappa takes a little time to learn, and it has its own quirks. But Zappa is much easier to get started with and has far fewer quirks than most open source frameworks. Given the enormous benefit it provides—the power of serverless deployment with the ease of traditional application development—it’s well worth the small investment in ramp-up time.

What I like about Zappa

Here are a few of the features that Zappa provides that I find particularly compelling:

• Documentation: The Zappa README page provides simple but comprehensive, documentation. I use it frequently and it gives me the information I’m looking for 95% of the time. The Zappa README is the README that all open source projects should aspire to.

• Quick Start Wizard: Zappa has a neat interactive wizard that helps you set up Zappa projects quickly when you’re first getting started. The command line tool asks you a handful of simple questions with smart defaults (e.g. it detects if you’re working on a Flask or Django project) and it creates the Zappa project for you.

• Automated Deployment: This is Zappa’s superpower—the ability to automagically package and deploy plain old Flask or Django apps to an AWS serverless app. With a single command, you can deploy, update or destroy an application for a specified stage (e.g. dev, staging, production).

• Automated Rollback: In addition to deployments, you can also rollback a Zappa application with a single command. Zappa uses AWS Lambda’s native versioning to do this.

• Keep Warm: One of the drawbacks of AWS Lambda for infrequently used applications is that the underlying container running the Lambda function gets decommissioned after several minutes of inactivity. This causes a noticeable delay the next time the function is called, as the container is re-provisioned. Fortunately, by default, Zappa creates a scheduled Lambda function that keeps the container warm by pinging it every 4 minutes, avoiding this performance hit.

• Settings File: Once you become familiar with Zappa, you’ll manage your project configuration directly in the Zappa settings file. This is a simple YAML or JSON file that stores your project configuration for each stage.

• Environment Variables: Within the Zappa settings file, you can store environment variables that can be accessed by your app when it’s deployed. Zappa also handles secret environment variables (e.g. a database connection string that you don’t want in source control) by allowing you to reference an S3 bucket that contains a settings file with the secrets. It’s worth mentioning here that Zappa introduced its environment variable functionality before AWS added it to Lambda. This can be a source of confusion, because your Zappa environment variables won’t show up in the AWS Console for your Lambda function. I spent a few hours one day trying to resolve this “bug” before I realized that the environment variables were there all along, just stored under-the-hood.

• Continuous Delivery: Zappa was built with CI/CD pipelines in mind. So, instead of using Zappa to deploy the application directly, you can also use it to generate a CloudFormation template and zipped code package for your own DIY deployment. As a DevOps-minded engineer, I love that this option, but as a lazy programmer, I love the full functionality of Zappa even more. For my CI/CD pipelines, I just install Zappa as a dependency in my CI service (e.g. AWS CodePipeline/CodeBuild) and call zappa deploy in my deployment scripts. This works like a charm.

• Debugging with the Tail Command: Debugging Lambda functions is a pain, as it requires wading through verbose CloudWatch logs in the AWS Console to find clues about your problem. Zappa’s tail command makes this process a little easier. With the tail command, you can watch the CloudWatch logs directly from the command line. This is nice because it avoids a trip to the AWS Console. But what’s even better is that you can filter the results with a command line option using the CloudWatch log filtering syntax. This greatly reduces the noise. I have a simple script that calls Zappa tail with a few helpful filters, allowing me to skip the useless log detritus and get right to the useful debugging info quickly.

• Default or Custom IAM Policy: By default, Zappa creates an IAM policy that provides enough permissions to get started out-of-the-box for most use cases. However, ABSOLUTELY DO NOT use this policy in production. By design, the default policy is extremely permissive and opens up a ton of permissions that you probably don’t want or need. The default policy is helpful when you’re getting started because it allows you to focus on learning Zappa, without the pain of debugging IAM permissions. However, the default policy is not suitable for production. Fortunately, Zappa allows you to specify a custom IAM policy written by you to secure your specific app, ensuring that it adheres to the principle of least privilege. Yes, writing a locked down IAM policy takes time and it’s no fun, but you have to do it to keep your customer’s data secure.

• Scheduled Lambda Jobs: Zappa allows you to run Lambda functions at regular intervals. This is helpful when you have small maintenance tasks to run, but don’t want to spend a lot of time scheduling and managing CloudWatch jobs.

• SSL Certificates: SSL always seems to be the one area of CI/CD that is resistant to automation. Fortunately, Zappa has a certify command that allows you to use DevOps-friendly certs from AWS and Let’s Encrypt (as well as SSL certs from other providers) for your Lambda’s Custom Domain. Although this is a cool feature, I usually end up managing SSL certs outside of Zappa, since creating and installing SSL certificates in a 100% automated way requires a bunch of extra code anyway.

• X-Ray Integration: X-Ray is AWS’s new service for monitoring application performance across microservices (basically, it’s AWS’s answer to New Relic). You can add X-Ray integration to your app with a simple addition to the Zappa settings file. Once this is enabled, the Python libraries that implement X-Ray instrumentation become available in your app. I really like the fact that Zappa is so quick to incorporate new, advanced AWS features into the framework, and it’s one of the reasons I selected it.

• Cognito User Pools: If you’re using AWS Cognito to manage user pools and authentication, then Zappa’s got you covered. You can specify your User Pool ID in the Zappa settings file, and Zappa will create an API Gateway Authorizer to handle the token-based authentication. Not only does this add a robust layer of security on top of your app, it also gives you easy access to user attributes like name, email and user ID in the HTTP headers that API Gateway passes to your app.

• Global Deployments: The Zappa framework has the option to deploy your application globally, in multiple AWS regions. This provides better performance for apps with geographically distributed users and creates redundancy when an entire region goes down (which happens). I’ve used Zappa to deploy multi-region apps, but I ended up doing this outside the Zappa framework. The Zappa documentation for global deployment is a bit light, and there are lot of other practical considerations when deploying apps in multiple regions (e.g. database redundancy). So, I find it easier to just manage the multi-region deployment in my own scripts.

• T-Shirts: But the best thing about Zappa is that they have T-shirts! They have a Zappa Flask t-shirt and a Django one. How cool is that?

As you can see, Zappa is a remarkably full-featured framework, especially given the newness of serverless development. But what I like the most about developing applications with Zappa is that when the framework doesn’t cover a use-case, it stays out of the way when I need to write custom code to address it. All too often, developers spend more time managing the framework than building the app—not so with Zappa. Zappa strikes the right balance between being full-featured and light-weight.

Serverless development has come a long way in a short amount of time, and with frameworks like Zappa, I dare say, it has actually become…fun.