Server not accessible via web

The server is running but customers are unable to access their CustomOffice instance.

Symptoms¶

Blank page when navigating to the server
SSH to the server works

Problems with SWAG¶

SWAG is the docker image that we use to set up Nginx and letsencrypt: https://github.com/linuxserver/docker-swag

Letsencrypt rate limit¶

Rate limit rules:

Example where our nightly check revealed two servers with this problem: https://customoffice.slack.com/archives/C6HES5UGP/p1780979662058549

Diagnose¶

Assumes you SSH to the relevant server and that you are logged in as the service user

Start out by checking if the swag service reports healthy.

docker ps

If swag reports unhealthy we continue to inspect the logs to see what is going on.

May be worth restarting the service to get a fresh attempt to start up in the logs

docker compose -f ~/containers/proxy/docker-compose.yml restart

docker logs -f swag
...
swag  | An unexpected error occurred:
swag  | too many certificates (5) already issued for this exact set of identifiers in the last 168h0m0s, retry after 2026-06-10 14:56:23 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-identifiers
swag  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /config/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
swag  | 2026/06/09 07:52:26 [emerg] 764#764: cannot load certificate "/config/keys/cert.crt": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/config/keys/cert.crt, r) error:10000080:BIO routines::no such file)

Workaround¶

Encountering this issue on customer servers, it is not really an option to wait 24+ hours to be allowed to retry requesting a new certificate.

As a workaround, we can make a slight change to the configuration to circumvent the limit, e.g. appear to be a new requestor.

One way to do this is by adding a subdomain. In below example we apply the workaround for the server hugo.customoffice.dk.

Add new A record to the DNS in Cloudflare: sub.hugo.customoffice.dk -> IP for hugo.customoffice.dk
Add - SUBDOMAINS=sub to the environment section of ~/containers/proxy/docker-compose.yml
Make swag to re-request certificate using the new configuration

docker compose -f ~/containers/proxy/docker-compose.yml up -d

For reasons unknown it may be needed to restart swag once more to get the signed certificate.

docker compose -f ~/containers/proxy/docker-compose.yml restart

Completed work¶

Track renewal attempts¶

Obtain a count of how many renewal requests each server has made.

Gulp server status lets encrypt renewel requests

Correct swag container volume¶

Make the swag container persist whole config folder instead of just the letsencrypt subfolder.

fix(reverse proxy): Fixed the proxy container volume configuration