Gonna be a tight squeeze in 🇩🇪. My flight leaving 🇨🇿 has been delayed at least an hour. :(
Let's encrypt created the ability to create wildcard certs last year (2018). Since I've been working on some Indieweb standard for my site I decided that getting some SSL would be good to have. Especially since I want to write my own auth server for using Indieauth. I looked into using IndieLogin but there's no way to register a site to use the auth... so here's a chance for me to write my own.
What did I have to do to get SSL setup? Since I created and use HAProxy to handle load-balancing for my site, I decided that all SSL termination should be done at the HAProxy level. This ties nicely into the ability for me to host multiple sub domains on my domain. ie indieauth.nicklang.com and have it be SSL'd. Because you know.. ssl.
Thankfully, a lot of the work has already been done for me. This post, Generate Wildcard SSL certificate using Let’s Encrypt/Certbot did most of the work for me, on how to create a cert. We can skip all the details here, and you can read the page yourself. But ultimately I was able to take the command provided and alter it to fit my needs:
certbot certonly -- manual --email firstname.lastname@example.org --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d *.nicklang.com -d nicklang.com
The differences here being I opted to not use the dns challenge method and I have 2 domains listed.
I didn't use the DNS challenge because when using 2 domains that are the same (
nicklang.com) certbot wanted me to create 2 txt records on the same domain and it would end up failing the challenge :(
By doing the default for
--preferex-challenges certbot asked me to do a DNS challenge and to do a txt file challenge. I was able to provide both of these challenges at the same time and everything worked.
After this was finished I got a notification saying where my certs have been saved too,
The next part was configuring the certs so that HAProxy can use them. The LetsEncrypt docs reference 2 sites for help with HAProxy and I chose the digital ocean one. How To Secure HAProxy with Let's Encrypt on Ubuntu 14.04. The important bit of info from this article is the idea that you want to
privatekey.pem. So I did just that. Next, I put the cert in the location that HAProxy is expecting it to be and voila, https!
So now the fun part how do I get this to work with Docker Swarm. Especially since the certs expire every 90 days, this means I'd have to go and do a Docker Secret rotation every 60 days, which is doable, but I'd rather figure a way to do fewer things, and I'm still working out the details. So I hope this post can give you some ideas.
First, we want to create a couple of volumes: 1. letsencrypt - directory where certbot will store certs 2. letsencryptbackup - directory where backup info is stored.
Next, we want to create a docker image that uses Certbot:
FROM alpine:latest RUN apk update && apk add certbot
In my docker-compose file, I've created a
certbot service and a
haproxy. The haproxy service mounts the letsencrypt volume and the certbot service mounts both.
This allows me to run the certbot service and write to the docker volume and that volume is shared to only the haproxy volume which can pick up my certs.
I know this is fairly abstract, but ultimately my workflow goes kinda like this:
First, I do a stack deploy
docker stack deploy -f docker-compose.yml nicklang_stack. This ensures that my volumes have been created and my services created. The certbot on the other hand exits right away cause there's no
CMD specified in the Dockerfile. So I want to update the service command.
docker service update --args "ping docker.com" nicklang_stack_certbot
What this does is it restarts the service with a new command, that will just keep the container alive as long as I need it to be.
Next, I will find the container id of the certbot container and then exec into the container.
Now that I'm in the certbot container I can run the command we talked about above.
And I can manually cd into the
live directory where my certs are and I can
cat fullchain.pem privatekey.pem > nicklang.com.pem. Now my cert exists in the volume where my haproxy is expecting it to be. I can now restart haproxy_service and it'll pick up the change.
This process was kinda messy I'll admit. BUT I'm hoping that it's a one-off process, just getting the cert. I would like to have the ability that I can run
cron on my swarm master that runs a service that will just
renew my certs, hopefully the renew process doesn't require any manual intervention and can be locked away to just run on its own. I'll write more about that in a couple months when it's time for me to figure that step out!