April 10, 2023
min read

Ditching ZeroSSL for Google's Public CA

Uffizzi provides all of its customers' deployments with HTTPS traffic forwarding, including appropriate certificates. For years we used `cert-manager` to provision TLS certificates from ZeroSSL. Now we're migrating to use Google's new Public Certificate Authority as described in this article.

Uffizzi provides all customer deployments with HTTPS traffic forwarding, including trusted certificates. Configuring these certificates can be challenging because there are still few public Certificate Authorities able to dynamically provision certificates in a way that Kubernetes can use. Recently we switched from using ZeroSSL to Google's Public CA. This article will demonstrate how we did this using `cert-manager`.

Certificates for every deployment!

Most Uffizzi deployments receive HTTPS requests at subdomains of `app.uffizzi.com`. For these domains matching the pattern `*.app.uffizzi.com`, we purchased a single "wildcard" certificate and installed it as the `default-ssl-certificate` for our `ingress-nginx` load balancer. This means any `Ingress` virtual load balancer that does not specify its own certificate will be protected by this one shared cert.

A few customers need to test their apps at more than one domain from a web browser's perspective. They may need to host the same application at `foo.example.app.uffizzi.com` and `bar.example.app.uffizzi.com`. A wildcard certificate only works for one "level" of a fully-qualified domain, so our `*.app.uffizzi.com` certificate cannot secure these additional subdomains. For these deployments, we dynamically provision certificates with additional hostnames.

For years we used `cert-manager` to provision TLS certificates from ZeroSSL. Their ACME service is free, but we've really gotten what we paid for. Service outages were common, and more recently ZeroSSL added undocumented rate limiting for HTTP requests to their ACME API. This change put us in the same situation as this `cert-manager` user: https://github.com/cert-manager/cert-manager/issues/5867. We began looking for alternative certificate authorities that support the ACME protocol.

About a year ago, Google announced ACME protocol support for their new Public Certificate Authority. This is the alternative we're looking for. We've configured our system to use this and below I'll show how you can too. We'll follow part of Google's tutorial, but instead of using `certbot`, we'll configure a `ClusterIssuer` and associated `Secret` for `cert-manager`. https://cloud.google.com/certificate-manager/docs/public-ca-tutorial

$ gcloud config set project uffizzi-pro-production-gke
Updated property [core/project]

$ gcloud beta publicca external-account-keys create

This will output a private key and an ID number. Take the resulting `b64MacKey` value and base64 encode it.

$ echo -n [the b64MacKey] | base64 --wrap=0

Copy that value into a YAML file specifying two Kubernetes Resources.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
  name: google-public-ca
    email: info@example.com
      keyID: 03ffa4c7d6bf48853fb5e63b106e83f1
        key: eab_hmac_key
        name: google-public-ca-eab-secret
      name: google-public-ca-tls
    server: https://dv.acme-v02.api.pki.goog/directory
    - http01:
          class: nginx
apiVersion: v1
kind: Secret
  name: google-public-ca-eab-secret
  namespace: cert-manager
type: Opaque
  eab_hmac_key: [the base64-encoded key]

Create the Kubernetes resources.

$ kubectl apply -f google-public-ca.yaml

Confirm that `cert-manager` successfully registered the credentials with Google's ACME service.

$ kubectl get clusterissuer google-public-ca -o json | jq '.status.conditions'
    "lastTransitionTime": "2023-04-06T20:33:55Z",
    "message": "The ACME account was registered with the ACME server",
    "observedGeneration": 1,
    "reason": "ACMEAccountRegistered",
    "status": "True",
    "type": "Ready"

Now specify an `Ingress` with this annotation and `cert-manager` will provision a certificate from Google's Public CA.

apiVersion: networking.k8s.io/v1
kind: Ingress
    cert-manager.io/cluster-issuer: google-public-ca

(In our case, we manage these k8s resources using terraform, specifically the `kubernetes_secret` and `kubernetes_manifest` resources.)

Google's Public Certificate Authority is completely free to use, so long as you have a GCP Project to meter its quotas. Unlike ZeroSSL, Google's quotas are published here: https://cloud.google.com/certificate-manager/docs/quotas#public_ca_request_quotas

We have not yet bumped up against these quota limits, and now Uffizzi customers are consistently provisioning certificates with many additional hostnames. This is another example of infrastructure work we do so you can deploy ephemeral environments for all of your project's branches. If you have questions about this or any of our infrastructure, join us on Slack!

Uffizzi logo
Environments as a Service
Learn More
preview icon
Empower your devs with an environment for every pull request
time icon
Avoid release delays that cost millions in missed revenue
Velocity icon
Improve development velocity at scale by up to 50%
Lifecycle icon
Plan capacity in real time with lifecycle management built in
Learn More