Notes

A small-agency hosting stack that does not need a platform team

12 October 2026

The production box for RetainerProof is one Node process, a SQLite file on a volume, and Coolify on a Hetzner server. Traefik sits in front and asks Let’s Encrypt for certificates. There is no cluster, no second region, and no person whose job title is platform. That is the stack this note describes, because it is the one I run, and because it is the one I would hand to a two-person studio that hosts a handful of custom apps and still has to answer the phone.

I am not going to tell you to install Kubernetes. I am also not going to pretend a status-page product is a hosting plan. The client-facing page is a different job. If you want the shape of that page, open a sample digest. The uptime section of a client report, and why a 15-minute homepage probe is not a one-minute monitor, is already written up as a client website uptime report. This note is the machinery under your own apps.

What this stack is for

It is for the agency that deploys a few long-lived web apps and a few static sites, and that wants to know three things without opening five dashboards. Is the process answering. Is the certificate going to expire before you notice. Can you get the data back if the disk dies. Everything else is optional until one of those three has failed you.

It is a bad stack for a product that needs to scale to a traffic spike you can name in requests per second. It is a bad stack if your promise to a client is “we will know within a minute, from three continents.” Say that promise out loud before you pick tools. If the promise is “the site is hosted, we deploy it, and we will notice if it is down before the client writes to us,” one well-watched server is enough.

Coolify is the panel I use because it is a self-hosted PaaS: Git in, container out, a proxy in front, and a place to attach a domain. The docs are public, which matters when you are the only person who will be awake.

One server, one long-lived process

Start with a single virtual machine you can reinstall from the provider’s console. Put the panel on it. Put the apps on it. Give each app its own service, its own environment variables, and its own volume if it has state. Do not SSH in and edit a file that the next deploy will overwrite. If a setting matters, it lives in the panel or in the repo.

RetainerProof is one process on purpose. The scheduler that runs full checks, and the homepage probe loop, both live in that process. If the container sleeps, the checks stop. Coolify is happy to run a web service that exits after a request. That model is wrong for anything that has to notice time passing. The Dockerfile starts Node and stays up. The health check, below, only proves the process answered. It does not prove the scheduler thread is doing its job. I still want the health check. I also look at the log once a week to see that probes are being written.

Write down, in the same place as the client roster, who can open the server console, who can open the panel, and where the SSH key is. A stack you cannot enter on a Sunday morning is a hobby. Two people should be able to get in. If you are solo, the second person is a sealed envelope or a password-manager vault someone else can open.

The health endpoint

Every app gets a URL that means “this process can serve.” Ours is GET /healthz. It returns HTTP 200 and the body ok. It does not look at Stripe, it does not send mail, and it does not render a page. The Docker image includes curl, and the image health check is curl -fsS http://127.0.0.1/healthz. Port 80 inside the container, because that is what the image listens on.

app.get("/healthz", function (_req, res) {
  res.status(200).type("text/plain").send("ok");
});

That handler is almost too small to be worth a section. The mistakes are all in how it gets used.

Coolify’s own docs distinguish a health check you set in the dashboard from a HEALTHCHECK in the Dockerfile and from a Compose health check. Dashboard HTTP checks run from inside the application container. The image has to contain curl or wget. Installing curl on the server and not in the image does nothing. I learned to treat “curl is on the box” as a different sentence from “curl is in the image.”

A rolling update, in the same docs, starts the new container while the old one is still up, waits for the new container’s health check, and only then removes the old one. If the new one fails the check, Coolify can drop it and leave the old one running. If you have no health check, the new container is treated as ready when it has started. Started is not the same as listening. A Node process can be alive and still be busy opening a database. Give the check a start period long enough for a cold boot, and keep the check itself fast.

What the endpoint should refuse to do is as important as what it does. Do not call your payment provider. Do not query a third-party API. Do not make it depend on disk space in a clever way that fails closed during a backup. A health check that flaps will train you to ignore it, and then it is decoration. If you want a stricter readiness check, add a second path, /ready, that confirms the database file opened. Point the deploy health check at /ready if a deploy must not receive traffic until the file is there. Point the outside uptime probe at /healthz or at the real homepage, depending on what you promised. Do not point both systems at a check that fails for reasons the visitor cannot see. You will get paged for a blank database while the marketing site is fine, or you will stay quiet while the marketing site is down.

Here is the order I use when I add an app.

1. Add /healthz and return a tiny body. No HTML.
2. Put curl in the image. Confirm it by running it inside a thrown-away container, not on your laptop.
3. Add the Docker HEALTHCHECK or the panel’s HTTP check. Restart so the container is recreated. Disabling a check also needs a recreate. The docs say the old container keeps the old check until then.
4. Deploy a change you know is good, with the health check on, and read the deploy log for the line that says the new container became healthy before the old one stopped.
5. Then break it on purpose in a staging service: return 500 from /healthz and confirm the new container is thrown away. If you skip this, you do not know the safety works. You know the happy path.

A passing container check is not a zero-downtime deploy. Coolify’s rolling-update note says the same thing. The app still has to finish in-flight requests when it is told to stop, and two versions have to tolerate the same database for a minute. For a small agency app that is usually true. For a migration that renames a column, it is not. Turn the rolling overlap off, or deploy that change in a window you announced. Do not discover it from a 500 in the middle of a Tuesday.

Three clocks, not one

Uptime, the certificate, and the domain expiry are three different dates. People fold them into “is the site ok” and then miss the one that does not beep. I keep them on separate lines in the roster and in my head.

Uptime, from somewhere else

Probe the public URL from a network that is not the server. A check on the same machine will succeed while the provider’s route to the rest of the internet is broken, and it will succeed while Traefik is answering on localhost and refusing the public interface. I want the probe to do what a visitor does: resolve DNS, connect on 443, get a response.

Frequency is a promise. Our homepage probes run every 15 minutes by default, down to 30 seconds, and two misses in a row email the account, not the client. A dedicated monitor checks from many regions. If a contract says “within a minute,” keep the monitor and let the client report stay the monthly letter. That split is the uptime report page. For your own infrastructure, pick a number you will still believe at 2am. Two failures before a page is enough to ignore a restart. One failure, on a 30-second probe, will page you for every deploy.

Check the URL the client actually uses. If the apex redirects to www, probe the apex too, once a day is enough, because a broken redirect is a broken site for anyone who types the name. Check the one path that makes money: /checkout, the booking page, the contact form’s thank-you URL if you can hit it without sending mail to a human. A homepage 200 with a checkout 500 is the outage that costs the relationship.

Certificates, on a shrinking lifetime

Let’s Encrypt’s own lifetime page, updated 22 July 2026, says the default certificate is still 90 days, and that this has been true since they launched. Short-lived 6-day certificates are optional. The same page says industry rules will cap public certificates at 47 days from 15 March 2029, and that Let’s Encrypt intends to be at 45 days by February 2028.

The dates in between are published, so you can put them in a calendar instead of a rumour. On 7 October 2026 they wrote that the default profile moves to 64-day certificates on 10 February 2027, unless you opt into something shorter. They expect the last 90-day certificate from that change to expire on 11 May 2027. The earlier note, 2 December 2025, puts the classic profile at 45 days on 16 February 2028. An opt-in 45-day profile already exists for people who want to try it. The CA/Browser Forum baseline requirements schedule the maximum validity at 200 days from 15 March 2026, 100 days from 15 March 2027, and 47 days from 15 March 2029.

The practical line from Let’s Encrypt, in that October 2026 post, is the one I care about: if your renewal is hard-coded to a number of days before expiry, change it to about two thirds of the lifetime. Grep your cron and your runbooks for 60, 80, and 83. Those numbers were sensible when every certificate lived 90 days. They become “renew after it has already expired” when the lifetime is 45.

On our box, Traefik asks for the certificate, and it only does that for hostnames attached to the service in Coolify. The app does not issue certificates for arbitrary customer names. I have a separate note for the status-page hostname dance, DNS and the Coolify TLS step, because people assume HTTPS appears when the CNAME does. It does not. For your own apps the lesson is smaller: a domain that is not attached in the panel will not get a certificate, no matter how green the DNS looks.

Watch the expiry date from outside anyway. Automation fails quietly. A calendar reminder at 21 days is a backstop, not the primary plan. When lifetimes hit 45 days, 21 days is already late. Move the backstop with the lifetime. I would rather have a boring alert that says “certificate for example.com expires on 3 March, 14 days left” than a clever one.

Domains, which are not certificates

The certificate can be healthy on a domain that is about to fall off the registrar. ICANN’s Expired Registration Recovery Policy requires registrars to send at least two renewal notices before a generic TLD expires, about one month before and about one week before, and another notice after expiry with restore instructions. After a registration is deleted, generic TLD registries other than sponsored ones must offer a 30-day redemption period. During redemption, DNS is off. The site and the mail go dark on purpose, as a last shove. The registrar can charge a restore fee. ICANN will not renew the name for you.

A 30-day redemption window is not a plan. It is a fine. I alert at 30 days and again at 14, and the 30-day alert is a task, not a notification. Open the registrar. Confirm auto-renew is on and that the card is not the one that expired in January. Write the expiry date into the client roster even when the client owns the registrar login. You cannot renew what you cannot see, so the monthly pass includes “ask for the date if you do not have the login.” More on that pass in the monthly note when it is up. Until then, the date belongs in your sheet this week.

Backups you have restored

A volume protects you from a container being replaced. Coolify’s storage docs say that plainly: persistent storage does not protect you from deletion, a dead disk, a lost server, or a corrupted file. An instance backup is a different object again. Coolify’s backup guide says an instance backup saves the panel’s own data, projects, settings, and deployment history. It does not save the data inside your applications, databases, or volumes. The application key that decrypts secrets stored in the panel lives in a file on disk, and the database dump does not include that file. Copy that key somewhere that is not the server. I keep it in a password manager. Without it, a restored panel cannot read the secrets it stored.

Database backups, for databases Coolify itself is running, are scheduled and engine-aware. The docs list PostgreSQL, MySQL, MariaDB, MongoDB, ClickHouse, and SQLite. The dump lands on the database’s server, under /data/coolify/backups, and can be uploaded to S3-compatible storage. If the database is stopped when the schedule fires, that run is skipped. If the S3 upload fails, the local dump can still be there. Read the execution. “Success” with an S3 warning means you have one copy, on the machine you were trying to survive the death of.

Our own database is a SQLite file on the /data volume of the app, not only a database resource inside the panel. I do not trust a single mechanism for it. The file is the product. The copy has to leave the server. CISA’s small-business backup guidance is the rule I can point at without inventing a number: three copies of important files, two kinds of media, one copy off-site. They also say to test that you can restore, and to be able to roll back at least seven days. I treat “seven days” as their floor, not a statistic about my own incidents. I have not counted my incidents in public and I am not going to invent a count.

Once a month I restore something, not everything. The drill is short because a long drill gets skipped.

1. Pick one database or one SQLite file. Not the whole machine.
2. Copy the latest off-site object to a laptop or a throwaway server. Do not restore over production.
3. Open it. For SQLite, sqlite3 backup.sqlite "PRAGMA integrity_check;" and then a query you recognise, such as the number of rows in a table you know.
4. Write one line in the work log: the date, the file name, and whether the check passed. If it failed, that line is the work for the day. Fix the backup before you fix anything cosmetic.

If you have never restored, you have a copy, not a backup. The first drill is allowed to be clumsy. The second one should fit in half an hour. CISA’s wording is “test,” not “hope.” I keep the off-site copy in a bucket that the production server can write to and that my laptop, with a different key, can read. The server’s key should not be able to delete the bucket’s older versions. Ransomware that reaches the panel will delete the backups it can see. Versioning or an immutable retention rule is the boring control. Turn it on before you need the story.

Alerts that reach a person

An alert has a recipient who can fix the thing, a condition, and a sentence. “Service degraded” is not a sentence. “example.com returned 502 at 02:14 UTC, two probes, healthz on the server is still 200” is a sentence. The first one wakes you into a scavenger hunt. The second one tells you the proxy or the app is the problem, and the process itself might be fine.

I split alerts into two pipes. The host pipe is the outside probe: DNS, TLS handshake, HTTP status. The app pipe is the container health and the disk. They page the same human at our size. They should not page the client. The client gets a report later, in their words, if the miss is part of the month they are paying for. A public status page is optional and separate. I wrote up how we treat that page, including what it deliberately leaves out, under white-label status pages. Do not point your PagerDuty-style siren at the same address you use for client mail.

Rules I actually follow:

• Two failed probes before the first page. One recovery message when it is green again. No recovery message if you never paged.
• A still-red reminder on a slow cadence, not every minute. Our product default for client sites is configurable. For the host, I use a few hours. A reminder every minute is how you mute the channel.
• Disk above a threshold you pick while calm. I use 80 percent as a ticket and 90 percent as a page. Those are choices, not a standard I can cite. The point is to have two numbers so the first one is not already an outage.
• Certificate and domain alerts go to email, in the daytime, as tasks. They are not pages unless the expiry is inside a week and auto-renew has already failed.
• The alert includes the URL, the status, the time in UTC, and a link to the log. It does not include a screenshot of a dashboard.

Test the pipe the way you test the health check. Break staging, confirm the message arrives, then fix staging. If it landed in a mailbox nobody reads, move it. A muted chat channel is not an on-call rotation. At two people, on-call means the phone is not on silent, and the runbook says so.

A runbook that fits on a page

I keep this in the repo and in the password manager. The values in brackets are yours. The shape is the point.

Service: [name]
Public URL: https://[name]
Health: https://[name]/healthz  expects 200 and the body ok
Panel: [Coolify URL]  service UUID in the panel, not in this file
Server: [provider console link]
Volume: /data  [what file lives there]
DNS: [registrar]  auto-renew [yes/no]  expires [date]
Certificate: issued by the proxy for hostnames attached in the panel
Backup: [bucket]  last restore drill [date]  key holder [name]
Alert: [address]  two failures, reminder every [n] hours
Deploy: merge to [branch]. Confirm the deploy log says the new container became healthy.
Rollback: redeploy the previous deployment in the panel. Do not edit files on the server.
If the disk is full: stop the noisy container, delete logs older than 14 days, then find the real growth.
If you cannot log in: the second person is [name], vault [where].

When something breaks, you fill the bottom of that page with what you did, and later you turn the useful part into a work-log line the client can read. The runbook itself stays technical. The client letter does not. Mixing them is how a client receives “redeployed container abc123” and still asks what they are paying for.

A first weekend, if you are starting from nothing

This is the order I would use for a studio setting the stack up once, not a suggestion that you rebuild production on a Saturday for fun.

1. Buy the server in the region you can legally put the data. Write the region down. Our app is EU-hosted by default. Yours might differ. Do not mix a client’s data residency promise with a cheap region you picked because the button was first.
2. Install the panel. Turn on the instance backup to a bucket, and copy the application key off the box the same day. Confirm one backup execution shows a file with a size above zero.
3. Deploy a hello-world container with /healthz, curl in the image, and a real hostname. Confirm HTTPS in a browser that is not on the server.
4. Add the outside probe. Pull the network cable in your head: stop the container, wait for two failures, read the alert, start the container, read the recovery.
5. Deploy the real app with a volume. Write one row into the database. Copy the file off-site. Restore it somewhere else. Read the row back.
6. Only then point a client DNS record at it. Skipping the restore and pointing DNS anyway is the usual shortcut. It feels fine until the disk goes read-only.

What I would not add yet

A second panel that nobody has restored. A service mesh. Multi-region failover you have never failed over. A log vendor that costs more than the server before you have grepped a file. An uptime product with a pile of status pages and no one assigned to the alert. Add a second server when you can say which requests it serves and how the data gets there. Until then it is a second place for a certificate to expire.

The client still does not see this machinery. They see whether the site worked and what you changed. When you want that page in their language, the sample digest is the shape. The stack in this note is how you keep the thing the digest is about from depending on your memory.

Sources

• Coolify, health checks. HTTP checks run inside the container and need curl or wget in the image.
• Coolify, rolling updates. Without a health check, a new container is treated as ready when it starts.
• Coolify, instance backup. Panel data only. It does not include application volumes. The application key is a separate file.
• Coolify, database backups. Scheduled dumps, local path, optional S3. A failed upload does not always mean the local dump failed.
• Coolify, persistent storage. A volume survives a new container. It does not survive a lost disk.
• Let’s Encrypt, certificate lifetimes. Updated 22 July 2026. The default is still 90 days.
• Let’s Encrypt, 64-day certificates. 7 October 2026. Default profile moves on 10 February 2027. Renew at about two thirds of the lifetime.
• Let’s Encrypt, from 90 to 45. 2 December 2025. Classic profile at 45 days on 16 February 2028.
• CA/Browser Forum baseline requirements. Maximum validity 200 days from 15 March 2026, 100 days from 15 March 2027, 47 days from 15 March 2029.
• ICANN, what registrants should know about ERRP. Two notices before expiry, redemption after deletion.
• ICANN, Expired Registration Recovery Policy. 30-day redemption for generic TLDs, sponsored TLDs excepted. DNS off during redemption.
• CISA, back up business data. Three copies, two kinds of media, one off-site. Test a restore.