Backups Are Useless Until You Restore Them

0. A Backup Is Not a File, but a Promise

You can write anything you want in the logs:

backup completed successfully

WAL uploaded successfully

retention completed successfully

archive is healthy

But on the day of a real disaster, PostgreSQL will not read your beautiful logs.

It will simply ask for the next WAL file.

And if you cannot provide it, the whole story ends right there.

This article is not about the internal implementation of a WAL receiver.

There was already a separate long story about that.

This article is about how I see the future development of the tool.

The real question is this:

Can I restore PostgreSQL from what my tool has been so confidently saving?

1. Backups Are a Comforting Lie

Backups are one of the most comforting illusions in infrastructure.

The command completed successfully.

A file appeared in S3.

There is a green line in the logs.

Maybe somewhere on a dashboard, healthy is even glowing.

Everyone feels relaxed and confident.

The problem is that none of this proves that recovery is possible.

It only proves that some operation completed without an error.

Some bytes moved from one place to another.

Some process returned exit code 0.

Some object storage API said, “yes, I accepted the file.”

But the recovery process does not care about our bright feelings.

It cares about only one thing:

Give me the required WAL file.

Right now.

Under the correct name.

In the correct place.

And make sure it is not corrupted.

If the file exists, PostgreSQL continues restoring history.

If the file does not exist, the history ends.

At that exact moment, “backup completed successfully” turns from a pleasant phrase into a question:

What exactly did you successfully do, actually?

2. WAL Files Are Not the Goal

When you write a WAL receiver, it is easy to become emotionally attached to WAL files.

They stream beautifully.

They appear in a directory.

They have serious-looking names like:

00000001000000000000000A

00000001000000000000000B

00000001000000000000000C

They are uploaded to remote storage.

They can be compressed.

They can be encrypted.

They can be shown in a UI.

They can be counted, sorted, checked, deleted, and downloaded again.

At some point, it starts to feel like the project is about them.

But that is a trap.

WAL files are not the product.

The product is the restored database.

Nobody wakes up at 3 a.m. thinking:

“How wonderful that I have 438 beautiful WAL files sitting in S3.”

People think differently:

“Can I bring the database back before people start calling me?”

A WAL archive is not a museum of artifacts.

It does not exist to store pretty files with long names.

It exists so PostgreSQL can replay the history of changes, the transaction log.

In other words, it exists to restore the database state through the chain:

base-backup -> WAL -> WAL -> WAL -> target recovery point

Without a base backup, WAL files are useless.

Without WAL files, a base backup quickly becomes an outdated snapshot of the past.

Without restore_command, everything together turns into a collection of files that looks like a backup system, but has not yet proven that it can work as a backup system.

3. The Real Product Is Point-in-Time Recovery

PostgreSQL recovery is not “restoring one file.”

It is restoring history.

A base backup gives the database state at a specific moment.

WAL files after that moment provide the history of changes.

The recovery process applies that history up to the required point.

In simplified form:

[base-backup]

      |

      v

000000010000000000000001

      |

      v

000000010000000000000002

      |

      v

000000010000000000000003

      |

      v

[target recovery point]

The problem is that the smallest gap in this chain can break everything.

One missing WAL file, and PostgreSQL cannot continue replay.

It is like losing one page from a legal contract.

Except the contract is a production database, and the lawyer is PostgreSQL, which simply refuses to start any further.

So the question is not:

Do I have WAL files?

The real question is:

Do I have a continuous, recoverable WAL chain

starting from a known base backup?

This is where pgrwl stops being just a “WAL file receiver.”

It becomes part of the recovery chain.

4. restore_command Is the Final Boss

There is a moment in PostgreSQL recovery where all theory ends.

That moment is restore_command.

On paper, everything looks beautiful:

base-backup + WAL archive = point-in-time recovery

But during recovery, PostgreSQL does not say:

“Show me a beautiful dashboard.”

It does not say:

“Tell me how well the uploader worked last week.”

It does not ask:

“Did you have logs and metrics?”

It asks for a specific file.

Something like this:

I need WAL file 00000001000000000000000A.

Put it here.

Return success if it worked.

And that is all.

If restore_command can fetch this file, recovery continues.

If it cannot, the whole system stops being a recovery system and becomes a sad collection of partially useful data.

That is why restore_command is the final boss of a backup system.

This is where it becomes clear whether the archive is actually usable for recovery.

For the development of pgrwl, this is an important shift in thinking.

At first, it seems that the main thing is to receive WAL:

PostgreSQL -> replication protocol -> pgrwl -> local directory

Then it seems that the main thing is to upload WAL:

local directory -> compression/encryption -> S3/SFTP

And then comes the realization:

S3/SFTP -> restore_command -> PostgreSQL recovery

That is the moment of truth.

5. A Successful Upload Is Not Proof of Successful Future Recovery

Object storage makes people optimistic.

You uploaded a file.

The API returned a successful response.

The file appeared in the bucket.

Everything looks reliable.

But for a backup system, that is not enough.

upload successful proves only transport.

It does not prove:

• that the file is complete


• that the checksum matches


• that encryption/decryption works


• that compression/decompression works


• that the file can be downloaded back


• that restore_command will find it under the correct name


• that retention will not delete it tomorrow


• that the entire WAL chain is continuous

That is exactly why a backup tool must be unpleasantly suspicious.

It is not enough to ask:

Can I upload it?

You need to ask:

Can I upload it?

Can I list it?

Can I read it back?

Can I decrypt it?

Can PostgreSQL use it during recovery?

A backup uploaded to S3 but never downloaded back even once is a motivational poster, not a recovery strategy.

6. Storage Is Where Backup Tools Become Dangerous

Cleaning up files sounds tempting.

Delete old files.

Free up space.

Clean the archive.

What could possibly go wrong?

In backup systems, almost everything.

Deleting the wrong “old file” can turn the entire chain into a useless set of data.

Bad cleanup logic thinks like this:

Delete WAL files older than 7 days.

More correct cleanup logic should think like this:

Delete only those WAL files that are definitely not needed

by any backup and by any recovery scenario.

The difference is huge.

A WAL file may look old by timestamp, but still be required to recover from a specific base backup.

If you delete a WAL that the oldest backup needs, that backup turns into beautiful garbage.

Cleanup is not housekeeping.

It is part of the recovery contract.

A backup tool should delete files with the confidence of a nervous accountant, not a shell script with rm -rf.

7. Backup Tool Development Is Based on Negative Scenarios

The most useful way to think about a backup system is not to start with positive scenarios.

• not with the receive command


• not with a beautiful CLI


• not with a dashboard

But with a disaster.

Imagine this:

the primary is unavailable

the local disk failed

the directory disappeared

a new PostgreSQL must be brought up

it must be recovered to the required point

people are waiting

coffee no longer helps

And now we ask the questions:

Where is the base backup?

Which WAL files are needed?

Are they in storage?

Can they be downloaded?

Can they be decrypted?

Can they be decompressed?

Does PostgreSQL know how to get them through restore_command?

Are there gaps in the chain?

When you design a system from this point, many features stop being “nice to have.”

A status API is not decoration.

It is a way to understand whether the receiver is alive.

WAL listing is not a toy for the UI.

It is a way to check the archive.

Backup metadata is not bureaucracy.

It is a recovery map.

Cleanup is not space saving.

It is a potentially dangerous operation.

Logging is not noise.

It is evidence.

A dashboard is not “for making things pretty.”

It is a way to quickly answer the question:

Are we okay, or do we just not yet know that we are already not okay?

8. What the Tool Should Know

At the beginning, it seems enough for the tool to know only a little:

where to read WAL from

where to write WAL

where to upload WAL

Then reality arrives with a long list of requirements.

9. What the Operator Should See

A backup system should not require archaeology.

If an operator needs to read 4,000 lines of logs to understand whether the archive is alive, the UX has already lost.

An infrastructure UI does not have to look like a spaceship.

It should quickly answer a few questions:

is the receiver alive?

is the WAL stream running?

what is the last received WAL?

what is the last uploaded WAL?

when was the last upload?

are there errors?

how many WAL files are stored locally?

how many WAL files are stored remotely?

which base backups exist?

which backup was the last successful one?

what will be required for recovery?

A green healthy label does not mean much by itself.

It is better to show evidence:

last received WAL: 00000001000000000000000A

last uploaded WAL: 000000010000000000000009

last upload: 42 seconds ago

slot: pgrwl_slot

mode: receive

storage: s3

errors: none

That is why the UI for pgrwl is not just about “making it pretty.”

It is an attempt to give the operator the state of the system without forcing them to read the tea leaves in logs.

10. Test Recovery Skeptically

The only honest test of a backup system is recovery.

Everything else is optimism.

A good test should be rough.

It should do unpleasant things:

• create PostgreSQL


• generate data


• take a base backup


• continue writing data


• stream WAL


• upload WAL to storage


• stop everything


• delete the data directory
  


• restore the base backup into a new location


• configure restore_command
  


• replay WAL


• verify that the data is in place

Even better if the tests can:

• restart the receiver


• interrupt uploads


• switch modes


• generate WAL under load


• check for missing gaps


• compare expected and restored data

A bad test:

command exited with 0

A good test:

new PostgreSQL instance started from restored backup

expected rows are present

target recovery point reached

If a backup test does not cause mild discomfort, it is probably a demo.

11. Things I Still Do Not Trust

There are parts of a backup system that I do not trust “just because.”

Not because they are necessarily broken.

But because they are too important to trust by default.

• I do not trust retention until I understand why a file can be deleted.


• I do not trust encryption until I have verified decryption.


• I do not trust compression until I have verified decompression.


• I do not trust object storage until I have read the data back.


• I do not trust the restore procedure until I have started PostgreSQL from a restored backup.


• I do not trust green badges on a dashboard if there are no concrete numbers behind them.

12. Make Recovery Boring

The goal of a backup system is not to make backups exciting.

The goal is to make recovery boring.

That means it should be:

• predictable


• documented


• verified


• observable


• repeatable


• without magic


• without heroism


• without “I think we also need to run this script here”

In an ideal world, recovery does not require a heroic engineer, three terminals, spiritual negotiations with object storage, and a random shell script from 2018.

It should be a procedure.

A boring procedure.

Because in infrastructure, boring is a compliment.

Thank you for reading!