Managing Let's Encrypt certs in k8s using acme.sh

Published on: Monday, Jun 5, 2023

UPDATE: I did some digging and found that there is a kubernetes native solution called cert-manager. The installation is easy and so is managing your certs. I migrated all of my certs over to cert-manager.

The Problem

In my home lab I run a small Kubernetes cluster hosting a couple of services for my home network. Managing the TLS certificates for these service can be annoying, since Let’s Encrypt certificates only last for 3 months. In order to solve this problem, I came up with a solution that automatically renews my certs using a program called acme.sh.

The Setup

The services are load balanced/exposed using Traefik. The config below show an example for one host, additional hosts can be added under rules and tls. The Certificate is stored as a secret and can be created using the following command: kubectl create secret tls _secretname –key domain.key –cert fullchain.cer. All hosts have their own certificates, following the principal of least privilege. Acme.sh has been set up as the root user, make sure the CA is set to Let’s Encrypt and you provided your API credential for the DNS challenge. I have to use the DNS challenge, since my services are not exposed to the internet. The config files for acme.sh are stored under /root/.acme.sh/.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ingress
spec:
  tls:
    - hosts:
      - git.cwoellner.com
      secretName: git-tls
  rules:
    - host: git.cwoellner.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gitea-service
              port:
                number: 3000

Listing Certs

/root/.acme.sh/acme.sh --list

The above command list all certs under management, only certs that are in here will be renewed.

Issuing a new Cert

/root/.acme.sh/acme.sh --issue --domain git.cwoellner.com --dns dns_cf

Replace dns_cf with the code for your DNS provider. The command will print out the location of your new cert.

Removing a Cert

/root/.acme.sh/acme.sh --remove --domain git.cwoellner.com

Make sure to remove unused cert, or they will be renewed.

Renewing Certs

Certs are renewed using:

/root/.acme.sh/acme.sh --renew-all

Once a cert has been renewed, the Kubernetes secret has to be overwritten.

Since I don’t want to do this by hand, I put these two actions into a systemd service and created a timer for it. If you have multiple certs, just add another ExecStart line. The timer will run the service every day and try to renew the certs, preventing your certs from ever expiring.

acme.service:

[Unit]
Description=Renew Let's Encrypt certificates using acme.sh
After=network-online.target

[Service]
Type=oneshot
# --home's argument should be where the acme.sh script resides.
ExecStart=/root/.acme.sh/acme.sh --renew-all --home /root/.acme.sh/
ExecStart=/bin/sh -c 'kubectl create secret tls git-tls --key /root/.acme.sh/git.cwoellner.com/git.cwoellner.com.key --cert /root/.acme.sh/git.cwoellner.com/fullchain.cer --dry-run=client -o yaml --save-config | kubectl apply -f -'
SuccessExitStatus=0 2

acme.timer:

[Unit]
Description=Daily renewal of Let's Encrypt's certificates

[Timer]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true

[Install]
WantedBy=timers.target