Let’s Encrypt and AutoTLS pitfalls

I fell into a couple of pitfalls when deploying a server with HTTPS support.

Auto TLS Basics

Here is the minimum server code for an HTTPS server. See https://echo.labstack.com/docs/cookbook/auto-tls for more information.

package main

import (
 "github.com/labstack/echo/v4"
 "golang.org/x/crypto/acme/autocert"
)

func main() {
 e := echo.New()
 e.GET("/", func(c echo.Context) error {
  return c.String(200, "Hello, World!")
 })
 e.AutoTLSManager.HostPolicy = autocert.HostWhitelist("example.com")
 e.AutoTLSManager.Cache = autocert.DirCache("/var/www/.cache")
 e.Logger.Fatal(e.StartAutoTLS(":443"))
}

Once you registered your domain example.com to the DNS server, you can run this code to launch an HTTPS server. AutoTLSManager , as its name suggests, will take care of TLS/SSL certificates generation by talking to Let’s Encrypt server using Automated Certificate Management Environment aka ACME protocol. You don’t have to do any manual work to get certificates. It’s quite amazing how simple it is.

Pitfalls

It works fine and you forget about you even set this up and continue whatever you wanted to do with the server and it’ll just work… until it doesn’t!

After a couple of build and deployment of the server, it suddenly stopped working. The server spews no logs and just stops handling incoming requests. After some digging, I discovered that I hit the Let’s Encryp server’s rate limit (see https://letsencrypt.org/docs/rate-limits/). This line e.AutoTLSManager.Cache = autocert.DirCache("/var/www/.cache") should enable caching of the certs, so I was puzzled why I’m hitting the rate limit.

It turned out, it was because of Docker. Because I forgot to mount the cache directory, every time I deploy the server, the container will be discarded along with the cached certs. So make sure mounting the cache directory if you are using Docker.

docker run -v /var/www/.cache:/var/www/.cache ...

At this point, it was already too late, I had to wait for a day before I can get a new cert from Let’s Encrypt.

Let’s Encrypt encourages using its staging environment (https://letsencrypt.org/docs/staging-environment/) during development. Adding the following configuration will make the server use the staging Let’s Encrypt environment.

 e.AutoTLSManager.Client = &acme.Client{
  DirectoryURL: "https://acme-staging-v02.api.letsencrypt.org/directory",
  HTTPClient: &http.Client{
   Transport: &http.Transport{
    TLSClientConfig: &tls.Config{
     InsecureSkipVerify: true,
    },
   },
  },
 }

Note that the certs returned by the staging environment is not a valid cert and cannot be used in production.

This worked for a bit but stopped working as well after a few minutes. The staging server suddenly stopped replying. I thought I might have hit some different kind of rate limit again before I realized the staging server went to a scheduled maintenance. Just how lucky I was… If you have problem with the staging server, make sure checking https://letsencrypt.status.io/ to see if the server is down so that you don’t have to waste your time like I did.

Buypass.com

Buypass (https://www.buypass.com) is an alternative to Let’s Encrypt. Because I was completely blocked by Let’s Encrypt, I decided to give it a try. Here’s the complete code to use buypass

package main

import (
 "net/http"

 "github.com/labstack/echo/v4"
 "golang.org/x/crypto/acme"
 "golang.org/x/crypto/acme/autocert"
)

func main() {
 e := echo.New()
 e.GET("/", func(c echo.Context) error {
  return c.String(200, "Hello, World!")
 })
 e.AutoTLSManager.HostPolicy = autocert.HostWhitelist("example.com")
 e.AutoTLSManager.Cache = autocert.DirCache("/var/www/.cache")
 e.AutoTLSManager.Email = "example@example.com"
 e.AutoTLSManager.Prompt = autocert.AcceptTOS
 e.AutoTLSManager.Client = &acme.Client{
  DirectoryURL: "https://api.buypass.com/acme/directory",
 }

 // Start the HTTP server that listens for ACME challenges.
 // Let's Encrypt doesn't require this because it can use the TLS-ALPN-01 challenge.
 // buypass.com diesn't seem to support that challenge, so we need to start an HTTP server
 // to listen for the HTTP-01 challenge.
 // Don't forget to open the port 80 in the firewall.
 go func() {
  httpServer := &http.Server{
   Addr:    ":80",
   Handler: e.AutoTLSManager.HTTPHandler(http.NotFoundHandler()),
  }

  err := httpServer.ListenAndServe()
  if err != nil {
   e.Logger.Fatal(err)
  }
 }()
 e.Logger.Fatal(e.StartAutoTLS(":443"))
}

A few lines were added such as Email, AcceptTOS. Most notable difference is the HTTP server go routine. There are different challenge types in the ACME protocol. Let’s Encrypt supports TLS-ALPN-01 which can be completed within HTTPS (port 443). I couldn’t find out whether Buypass support it, but at least I couldn’t make it work. So I switched to HTTP-01 challenge which requires the client (my server in this case) to listen to HTTP (port 80) where the ACME server will poll to complete the challenge. This HTTP server only needs to server the request to /.well-known/acme-challenge/. Other requests will be handled by http.NotFoundHandler which simply returns a 404 Page Not Found.

It’s also recommended to add the following CAA entries to the DNS records. I added the following records to my DNS configuration.

• example.com CAA 0 issue “buypass.no”
• example.com CAA 0 issue "letsencrypt.org"
• exmaple.com CAA 0 iodef "mailto:example@example.com"

See https://www.buypass.com/products/tls-ssl-certificates/resources-tls-ssl-certificates/articles/introduction-to-certificate-authority-authorisation-caa for more information.