Exploring a completely serverless API and deployment pipeline

Serverless architecture is a popular topic these days. I see a lot of potential advantages of this style but also have concerns about how to address some of the challenges. To explore this architecture I'm going to build my own serverless API and deployment pipeline. I'll document my experiences in a series of articles. Please join me in my journey! You can help shape my direction with your comments.

Areas of exploration

There are a number of areas I want to investigate in this series. In addition to the API implementation and storage backend I also want understand several other things:

• Deployment
  What does deployment look like? How to various stages of build and deploy get executed without servers? How is state managed in a multi-stage build process?

• Integrating AWS services
  How will the various AWS services work together? Where are the friction points?

• Testing at various stages
  How do we implement automated testing with the 5-minute Lambda restriction? How can we coordinate multiple testing stages and steps? How do we implement different testing environments?

• Security
  How can we use IAM properly to make sure we don't grant too much privilege and open ourselves up to vulnerabilities?

• Debugging and Monitoring
  How can you debug Lambda code without access to your debugger? What kind of monitoring works best?

A problem to build around: Media Library API

This project is built around a Media Library API. Imagine this API is the backend for an application like Google Photos. The front end calls the API to manipulate both content and metadata. I'll define content as the actual media files and metadata as information about the files like resolution and tags.

Problems to explore

The Media Library presents us with a couple of interesting challenges to address:

• Storage
  What will we use for storage and how will me manage it? S3 for the content and DynamoDB for the metadata seems like the obvious choice. How will we coordinate the two?

• Binary Data
  Since API Gateway doesn't support binary data we'll need a way to work around it.

• Others?
  Can you think of any more? Let me know in the commtents.

Architectural Constraints

No good architecture would be complete without some constraints. These will help keep me from going down too many rabbit holes and focus on the topics I want to cover.

• No EC2 instances
  I know, that's the whole point! But this is the main constraint some I'm listing it.

• Core AWS
  I plan to stick with core AWS for this project. There are a lot of great frameworks like serverless and Claudia.js, but I want to focus on the fundamentals of AWS. Sticking to core AWS also helps to highlight its limitations so it can be useful as a tool to help decide which framework you might like for your own projects.

• Node.js v4.3.2
  Support for Node 4.3.2 was recently added to Lambda and is now the recommended Node version to use. I'll also use the new [callback][callack-link] form of completing Lambdas.

• Bash + aws-cli
  I'll use bash for wrapping some of the aws cli commands.

• Suggestions?
  Hit me in those comments!

Next steps

Now that the high-level architecture is sketched out, my plan is to create the baseline deployment pipeline. The goal will be to create the simplest possible API Gateway and Lambda, deploy it, and run a simple test.

The journey has begun. Join me!

I'd love to hear your thoughts on the project! Do you have any experience building a similar API? Would you start a different way? Can you suggest any additional or different constraints? Please comment below, tweet @kennbrodhagen, or email me kennbrodhagen@gmail.com and let me know what you think.

References

• Serverless
• Claudia.js
• Lamda Node.js callback-style
• Images courtesy iStock.com/KatarzynaBialasiewicz and iStock.com/Petrovich9

Interested in more?

Sign up for my mailing list. You'll get each new article and other announcements delivered to your inbox. I promise not to spam you. Unsubscribe any time you like.