alexanderfortin

[alexanderfortin]

Deprecated

This blog has migrated to a new location -> https://a.l3x.in/

At some point I'll delete this one so please update your RSS feed accordingly ;)

[alexanderfortin]

sslnotify.me - yet another OpenSource Serverless MVP

After ~10 years of experience in managing servers at small and medium scales, and despite the fact that I still love doing sysadmin stuff and I constantly try to learn new and better ways to approach and solve problems, there's one thing that I think I've learned very well: it's hard to get the operation part right, very hard.

Instead of dealing with not-so-interesting problems like patching critical security bugs or adding/removing users or troubleshooting obscure network connectivity issues, what if we could just cut to the chase and invest most of our time in what we as engineers are usually hired for, i.e. solving non-trivial problems?

TL;DL - try the service and/or check out the code

sslnotify.me is an experimental web service developed following the Serverless principles. You can inspect all the code needed to create the infrastructure, deploy the application, and of course the application itself on this GitHub repository.

Enter Serverless

The Serverless trend has been the last to join the list of paradigms which are supposedly going to bring ops bliss into our technical lives. If Infrastructure as a Service didn't spare us from the need of operational work (sometimes arguably making it even harder), nor Platfrom as a Service was able to address important concerns like technical lock-in and price optimization, Function as a Service is brand new and feels like a fresh new way of approaching software development.

These are some of the points I find particually beneficial in FaaS:

very fast prototyping speed

sensible reduction in management complexity

low (mostly zero while developing) cost for certain usage patterns

natual tendency to design highly efficient and scalable architectures

This in a nutshell seems to be the FaaS promise so far, and I'd like to find out how much of this is true and how much is just marketing.

Bootstrapping

The Net is full of excellent documentation material for who wants to get started with some new technology, but what I find most beneficial to myself is to get my hands dirty with some real project, as I recently did with Go and OpenWhisk.

Scratching an itch also generally works well for me as source of inspiration, so after issuing a Let's Encrypt certificate for this OpenWhisk web service I was toying with, I thought would be nice to be alerted when the certificate is going to expire. To my surprise, a Google search for such a service resulted in a list of very crappy or overly complicated web sites which are in the best case hiding the only functionality I needed between a bloat of other services.

That's basically how sslnotify.me was born.

Choosing the stack

The purpose of the project was yes to learn new tools and get some hands on experience with FaaS, but doing so in the simpliest possible way (i.e. following KISS principle as much as possible), so keep in mind that many of the choices might not necessary be the "best" nor most efficient nor elegant ones, they are representative of what looked like the simpliest and most straightforward option to me while facing problems while they were rising.

That said, this is the software stack taken from the project README:

Chalice framework - to expose a simple REST API via AWS API Gateway

Amazon Web Services:

Lambda (Python 2.7.10) - for almost everything else

DynamoDB (data persistency)

S3 (data backup)

SES (email notifications)

Route53 (DNS records)

CloudFront (delivery of frontend static files via https, redirect of http to https)

ACM (SSL certificate for both frontend and APIs)

CloudWatch Logs and Events (logging and reporting, trigger for batch jobs)

Bootstrap + jQuery (JS frontend)

Moto and Py.test (unit tests, work in progress)

I know, at first sight this list is quite a slap in the face of the beloved KISS principle, isn't it? I have to agree, but what might seem an over complication in terms of technologies, is in my opinion mitigated by the almost-zero maintenance and management required to keep the service up and running. Let's dig a little more into the implementation details to find out.

Architecture

sslnotify.me is basically a daily monitoring notification system, where the user can subscribe and unsubscribe via a simple web page to get email notifications if and when a chosen SSL certificate is expiring before a certain amount of days which the user can also specify.

Under the hoods the JS frontend interacts with the backend available at the api.sslnotify.me HTTPS endpoint to register/unregister the user and to deliver the feeedback form content, otherwise polls the sslexpired.info service when the user clicks the Check now button.

The actual SSL expiration check is done by this other service which I previously developed with OpenWhisk and deployed on IBM Bluemix platform, in order to be used indipendently as a validation backend and to learn a bit more of Golang along the way, but that's for another story...

The service core functionality can be seen as a simple daily cronjob (the cron lambda triggered by CloudWatch Events) which sets the things in motion to run the checks and deliver notifications when an alert state is detected.

To give you a better idea of how it works, this is the sequence of events behind it: - a CloudWatch Event invokes the cron lambda async execution (at 6:00 UTC every day) - cron lambda invokes data lambda (blocking) to retrieve the list of users and domains to be checked - data lambda connects to DynamoDB and get the content of the user table, sends the content back to the invoker lambda (cron) - for each entry in the user table, cron lambda asyncrounosly invokes one checker lambda, writes an OK log to the related CloudWatch Logs stream, and exits - each checker lambda executes indipendently and concurrently, sending a GET request to the sslexpired.info endpoint to validate the domain; if no alert condition is present, logs OK and exits, otherwise asyncrounosly invokes a mailer lambda execution and exits - any triggered mailer lambda runs concurrently, connects to the SES service to deliver the alert to the user via email, logs OK and exits

Beside the main cron logic, there are a couple of other simpler cron-like processes:

a daily reporter lambda, collecting logged errors or exceptions and bounce/feedback emails delivered (if any) since the day before, and sending me an email with the content (if any)

an hourly backup of the DynamoDB tables to a dedicated S3 bucket, implemented by the data lambda

Challenges

Everyone interested in Serverless seems to agree on one thing: being living its infancy it is still not very mature, especially in regards of tooling around it. Between other novelties, you'll have to understand how to do logging right, how to test properly, and so on. It's all true, but thankfully there's a lot going on on this front, frameworks and tutorials and hands-on and whitepapers are popping up at a mindblowing rate and I'm quite sure it's just a matter of time before the ecosystem makes it ready for prime time.

Here though I'd like to focus on something else which I found interesting and suprisingly to me not that much in evidence yet. Before starting to getting the actual pieces together, I honestly underestimated the complexity that even a very very simple service like this could hide, not much in the actual coding required (that was the easy part) but from the infrastructure setup perspective, so I actually think that what's in the Terraform file here is the interesting part of this project that might be helpful to whoever is starting from scratch.

For example, take setting SES properly, it is non trivial at all, you have to take care of DNS records to make DKIM work, setup proper bounces handling and so on, and I couldn't find that much of help just googling around, so I had to go the hard way, i.e. reading in details the AWS official documentation, which sometimes unfortunately is not that exhaustive, especially if you plan to do things right, meaning hitting the APIs in a programmatic way instead of clicking around the web console (what an heresy!) as shown in almost every single page of the official docs.

One thing that really surprises me all the time is how broken are the security defaults suggested there. For example, when setting up a new lambda, docs usually show something like this as policy for the lambda role:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:*:*:*" ] } ] }

I didn't check but I think it's what you actually get when creating a new lambda from the web console. Mmmm... read-write permission to ANY log group and stream... seriously? Thankfully with Terraform is not that hard to set lambdas permission right, i.e. to create and write only to their dedicated stream, and in practice it looks something like this:

data "aws_iam_policy_document" "lambda-reporter-policy" { statement { actions = [ "logs:CreateLogStream", "logs:PutLogEvents", ] resources = ["${aws_cloudwatch_log_group.lambda-reporter.arn}"] } [...]

Nice and clean, and definitely safer. For more details on how Terraform was used, you can find the whole setup here. It's also taking care of crating the needed S3 buckets, especially the one for the frontend which is actually delivered via CloudFront using our SSL certificate, and to link the SSL certificate to the API backend too.

Unfortunately I couldn't reduce to zero the 'point and click' work, some was needed to validate ownership of the sslnotify.me domain (obviously a no-go for Terraform), some to deal with the fact that CloudFront distributions take a very long time to be in place.

Nevertheless, I'm quite happy with the result and I hope this could help who's going down this path to get up to speed quicker then me. For example, I was also suprised not to find any tutorial on how to setup an HTTPS static website using cloud services: lots for HTTP only examples, but with ACM and Let's encrypt out in the wild and totally free of charge, there's really no more excuse for not serving HTTP traffic via SSL in 2017.

What's missing

The more I write the more I feel there's much more to say, for example I didn't touch at all the frontend part yet, mostly because I had almost zero experience with CSS, JS, CORS, sitemaps, and holy moly how much you need to know just to get started with something so simple... the web party is really hard for me and thankfully I got great support from some very special friends while dealing with it.

Anyway, this was thought as an MVP (minimum viable product) from the very beginning, so obviously there's a huge room from improvement almost everywhere, especially on the frontend side of the fence. If you feel like giving some advise or better patch the code, please don't hold that feeling and go ahead! I highly value any feedback I receive and I'll be more then happy to hear about what you think about it.

Conclusion

I personally believe Serverless is here to stay and, it's fun to work with, and the event driven paradigm pushes you to think on how to develop highly efficient applications made up of short lived processes, which are easy to be scaled (by the platform, without you moving a finger) and naturally adapt to usage pattern, with the obvious benefits of cost reduction (you pay for the milliseconds your code is executed) and almost zero work needed to keep the infrastructure up and running. The platform under the hood is now taking care of provisioning resources on demand and scaling out as many parallel lambda executions as needed to serve the actual user load; on top of this (or, better, on the bottom of it) built-in functionalities like DynamoDB TTL fileds, S3 objects lifecycles or CloudWatch logs expirations spare us from the need of writing the same kind of scripts, again and again, to deal with basic tech needs like purging old resources.

That said, keeping it simple is in practice very hard to follow, and this case was no different. It's important not to get fooled into thinking FaaS is the panacea to every complexity pain, because it's not the case (and this is valid as a general advise, you can replace FaaS with any cool tech out there and still stands true). There are still many open questions and challenges waiting down the road, for example setting up an environment in a safe and predictible way is not as easy as it may look at a first sight, even if you're an experienced engineer; there are lots of small details you'll have to learn and figure out in order to make all the setup rock solid, but it's definitely doable and worth the initial pain of going through pages and pages of documentation. The good part is that it's very easy to start experimenting with it, and the generous free tiers offered by all the cloud providers make it even easier to step in.

What's next

Speaking of free tiers, the amazing 300USD Google Compute Cloud credit, together with the enthusiasm of some friend already using it, convinced me to play with this amazing toy they call Kubernetes; let's see if I'll be able to come up with something interesting to share, given the amount of docs and tutorials out there I highly doubt that, but never say never. Till the next time... try Serverless and have fun with it!

[alexanderfortin]

Lastversion: a Go serverless proof of concept developed on OpenWhisk

It seems that recently every OSS tool I've been using during my working day (together with many other cool projects) is written in Go, so I decided to teach myself a little bit of it having fun in the meanwhile, in the not-too-rare case I'll need to troubleshoot one of them, or simply understand better how it works. Unsurprisingly enough, the Net is full of excellent documentation material if you want to dig into Go programming without spending a single cent.

Generally I choose the next pet project trying to scratch an itch I already have, for example from time to time I want to know what's the latest stable version of one of the tools I use to have available on my workstation or on my servers in a programmatic way, so I decided to try to write a web service which will spare me the time of scraping the project's homepage in search of this precious information.

Thankfully, most of those projects I'm interested in are developed using Git with sources hosted by one of the main hosting platforms; they also tend to respect a SemVer versioning scheme (when you're lucky...) adding a git tag for each new release, so for those we can easily devise the biggest stable version reading the output of the git ls-remote command.

Where to host? Go serverless!

I'm a big fun of the serverless movement, and a tiny web service like this seems to be a perfect fit for the paradigm, so having the service running on some server which is not managed by myself feels like the only way to go. At the time writing, the only serverless platform offering good Go support (via Docker) seems to be Apache OpenWhisk (hosted on IBM Bluemix cloud).

Developing on OpenWhisk turned out to be quite fun (a special thank to James Thomas for his excellent blog posts helping me getting started smoothly), and one of the advantages of being open source, is to have a local development environment running on a Vagrant box. As if this is not already enough to make it an appealing platform, the community around the project is incredibly welcoming and most importantly very helpful troubleshooting the issues you might encounter (for example I was confused by namespaces in Bluemix and I found prompt and helpful feedback on the Slack channel which clarified my confusion very quickly).

Let's cUrl to the chase

The result of my tinkering is this publicly usable (and useless?) proof of concept:

$ curl https://lastversion.info/go { "request": "go", "result": "1.8" } $ curl https://lastversion.info/terraform { "request": "terraform", "result": "0.8.7" } # Non SSL available too $ curl lastversion.info/lastversion { "request": "lastversion", "result": "0.2.1" }

An Nginx instance (external to Bluemix) is providing SSL termination and URL rewriting, but the backend is a (Bluemix hosted) OpenWhisk action behind an OpenWhisk API gateway, consumable directly using the Bluemix endpoint:

$ wsk action list | grep /lastversion /<myaccount>_<my_namespace>/lastversion private blackbox $ wsk api-experimental list | grep /lastversion /<myaccount>_<my_namespace>/lastversion get /lastversion https://cfdbfa25-8630-4e9d-8b98-067de61009a0-gws.api-gw.mybluemix.net/lastversion/v1 $ curl https://cfdbfa25-8630-4e9d-8b98-067de61009a0-gws.api-gw.mybluemix.net/lastversion/v1?project=kubernetes { "request": "kubernetes", "result": "1.5.3" }

The project homepage is hosted on GitHub where you could also have a look at the source code, leave comments or open issues or pull requests. It has no ambition of being neither complete nor especially useful, but it might give you some hints if you're starting fresh on OpenWhisk (have a look at the build.sh script for examples on how to interact with the OpenWhisk CLI).

As a side note, if you develop on MacOS platform, you might also want to install the OpenWhisk wsk CLI tool using Homebrew and this formula I wrote] when I started to work on this project. EDIT: I've been told that on MacOS the suggested way to prototype on OpenWhisk is documented here, I didn't try it yet but you might want to start from there instead.

What's next

I'm already quite satisfied with the result considering that the only goal was to start writing Go and to learn how to deploy publicly accessible actions on OpenWhisk, but if I'll find time and will (or help) those would be some of the things I'd like to see improved:

remove the self hosted Nginx from the stack (which is providing SSL termination and URL rewriting) to make it a 100% serverless solution

remove the statically linked Git binary using a pure Go implementation of git ls-remote

add support for most of the active and popular OSS projects

add support for non-SemVer versions

add other useful(?) features based on users request if any

Follow me on Twitter if you want to get updates on my technical experiments with both Go and OpenWhisk, or get+stay in touch.

[alexanderfortin]

Breaking news

Just a quick update for everybody who knows I'm based in Berlin and might think I might have been involved in the recent terrible facts, well I was already in Bangkok when that happened so obviously I wasn't. Sorry for the late update but I'm still strugling with the jet-leg and I honestly have no idea what time it is here or there.

[alexanderfortin]

Bangkok, one-way ticket

Feels like yesterday but it was already 4 years ago I moved to Berlin, and since a little after that I've been calling the city home. Sounds like enough time to call for a new adventure.

Almost exactly 10 years after leaving Italy for a very long trip to Australia, potentially without any coming-home, at 36, here I am again, planning some random wandering close to the Equator, without any answer to the question "So when do you plan to come back?".

Many things were different those old days, but something still seems exactly the same: I've got no idea what's going to be "next", and I can't wait to live it and see how it will look like.

I can just promise I'll try to keep this blog up to date, but there's some serious investment in tropial lazyness to be done upfront, so I don't know when (and from where) next update will be. Till then... stay tuned!

[alexanderfortin]

Integrating Sonos with Spreaker RSS feed

It's a cold winter

Berlin winter is famous to be quite cold (ok, let's be honest, I'm a tech addict and can't help the urge of hack around from time to time) so recently I've spent some fun time tinkering a bit with this Internet Of Things idea: how to make my flat a little bit more automated and have fun while adding this competely superfluous automation?

First world problems

One of the things I've been missing from my dear Sonos system is a proper RSS integration with Spreaker: I love RSS feeds, they are simple and easy to use, and Spreaker is so kind to provide one for every show I like. With SoCo library is trivial to write a Python based CLI tool to add an HTTP URL with the mp3 to play, but I'm notoriously lazy, I don't want to open my laptop, fire up the browser, right-click for copying links to the clipboard, and type commands in the terminal unless strictly needed; all I want in this case is to be able to open the Sonos app and listen a podcast just tapping on the fresh new Spreaker episode when it's available.

Shopping list

To make this possible, I need quite a few (free or open source) pieces:

Python >v2.7 with SoCo and Bottle libraries

IFTTT Maker and RSS channels

a hosting provider for free GIT repository (i.e. GitHub)

some config management scripts to make this easy to install (even if I'm a Puppet fan, I took the chance to try out Ansible for this)

plus an always-on host connected to the Internet that runs Python (I use a RaspberryPi for the purpose) in the same LAN of the Sonos system (i.e. my flat).

How does it work

this is the basic workflow:

IFTTT RSS channel reads from a Spreaker feed there's a new episode available (and sends on my phone a push notification about this)

the connected IFTTT Maker channel sends an HTTP GET request to the in-house server with the URL of the episode as a parameter

this Bottle/SoCo based HTTP server receives the request, parses it filtering for the mp3 link needed by Sonos to be able to play the episode, and adds that to the Sonos queue

How to set this up

get a free IFTTT account if you don't have it yet

create a recipe that looks like this

create a SNAT rule on your router, or do whatever it takes to be able to make an HTTP GET on port TCP/9999 from the outside

Internet <---> TCP:80 (Internet Router) <---> TCP:9999 (RaspberryPi)

clone this GIT repo somewhere on your home host, edit the speakers.txt file with the IP addresses of your Sonos components, and finally:

$ ./pysonos/sonos.py

That's it. From now on, you should be able to play a new Spreaker episode as soon as it is available in the RSS feed directly from your Sonos player app.

Food for blog

I could also make the Ansible code public, or show how I'm messing around my flat with other (location based) IFTTT integrations, but for one blog entry seems already enough, so I'll keep that for future winters.

Conclusion

Well, in the end as you can see it's not that complex, all the pieces are already out there and I just needed to wrap them up, plug in my almost-forgotten RaspberryPi, and write some glue Python code, that is always great fun.

If you liked this article, please consider a donation dropping unrational beliefs like monotheistic religions, but if you really need to believe in a supernatural entity, then at least try out the Pastafarian cult first. Thanks!