At ITdaily, we run a QNAP running TrueNAS off-site as an exact copy of our server, ready to take over as soon as it fails. ZFS replication and a well-thought-out dataset structure ensure that both our data and our containers continue to run almost seamlessly.
The core of our infrastructure is a self-built TrueNAS server in the office, with an Intel Core i5-12600K and about fifty containers that we manage via Portainer. On it, Traefik runs as a reverse proxy for tools that need to be available outside the network (ITdaily is hosted by Combell). That server is the beating heart; if it fails, a large part of our digital operations comes to a standstill.
To mitigate that risk, we converted a QNAP TVS-h874T into a second TrueNAS machine. It is deliberately placed at a different physical location and acts as an exact mirror. The hardware is comparable: the QNAP also runs on a 12th generation Intel processor, architecturally almost identical to our Intel Core i5-12600K. This makes the two machines interchangeable without having to worry about compatibility at the CPU level.
ZFS replication as the engine
The mechanism behind the mirror is ZFS replication. Instead of copying files, our office server takes snapshots of its datasets at fixed intervals and only sends the changed blocks to the QNAP NAS. This incremental approach keeps data traffic limited: after the first full synchronization, subsequent transfers are compact.
ZFS works at the dataset level, not the disk level. This is an important fact, because it means that the underlying RAID setup of both machines does not have to be identical. One side can have a different vdev topology than the other; replication simply sends a stream of blocks that the receiving machine writes to its own pool. The data integrity of the copy remains guaranteed.
The same datasets, the same paths
The trick that allows not only the data but also the containers to move seamlessly is in the structure. Portainer stores each container as a docker-compose file in a folder structure. As long as that structure is on a replicated dataset, the QNAP has exactly the same stack definitions as the office server. We replicate the associated volumes (databases, configuration, media) as separate datasets.
The only hard requirement is that the mount paths on both machines are identical. If a container in the office gets its data from a certain path, that same path must exist on the QNAP. We solve this by giving our datasets the same names and mount points on both devices. The result: if you start the containers on the QNAP, they point to exactly the same locations as in the office.
What we specifically do NOT replicate
Not everything is allowed in the mirror. We deliberately keep two components separate: Tailscale and Portainer itself. Tailscale, which provides our secure network between both locations, maintains a machine-specific identity. If you were to replicate that, two devices would suddenly log on to the network with the same identity, resulting in conflicts.
The solution is to install Tailscale and Portainer separately on both machines before replication starts. This way, each machine gets its own identity, while we only mirror our actual data and compose files. In other words, we replicate specifically per dataset, instead of blindly transferring the entire pool.
First local, then remote
An exact copy of dozens of terabytes cannot simply be pumped over an internet connection. That is why we performed the first full synchronization locally, over a fast 10GbE network, with both machines in the same room. That initial transfer is by far the heaviest. After that, the compact incremental snapshots are sufficient.
Only after that first sync did the QNAP move to its off-site location. In principle, the device needs nothing more than power and an internet connection. The connection between both machines runs over Tailscale, which gives each machine a fixed address that does not change when the physical location changes. As a result, we hardly had to reconfigure anything after the move: replication simply picks up again, only slower than over the local network.
Failover in a few steps
The scenario for which we are building all this is the moment the office server fails. The replicated datasets on the QNAP are read-only by default. In the event of a failure, we promote them to read-write, open Portainer, and start the stacks. Because the paths are identical, the containers start running immediately.
Important to emphasize: this is not a fully automatic failover. It is a manual but fast procedure. We deliberately did NOT choose an active-active setup where both locations write simultaneously. That is significantly more complex and ZFS replication is not the right tool for it. For our needs, a well-thought-out mirror that we activate in a few steps via TrueNAS is sufficient.
Good practices for backups
A mirror is valuable, but it is not a full backup strategy in itself. Anyone who takes their data seriously should keep a few principles in mind.
The best-known rule of thumb is 3-2-1: keep at least three copies of your data, on two different media, one of which is at a different location. Our off-site QNAP fulfills exactly that last, crucial part.
Be careful of a pitfall: replication also copies your mistakes. If you accidentally delete a file, or ransomware encrypts your data, that disaster travels to the copy. That is exactly why ZFS snapshots are so valuable: they capture the state at a specific moment, so you can return to a version from before the incident. Maintain a well-thought-out retention schedule, multiple snapshots over different periods, and make them immutable where possible.
Equally essential, and often forgotten: test your recovery. A backup that you have never tried out is an assumption, not a certainty. Periodically perform a real failover test and check whether your containers and data actually start up. Add monitoring to that which warns you when a replication task fails, so you don’t only discover during a disaster that the mirror has been down for weeks.
Finally, it pays to let your off-site location truly be off-site: a different power supply, a different network (and telecom provider), ideally a different building. A second server standing next to the first protects you against a disk crash, but not against fire, theft, or a power surge. It is precisely this separation that makes the difference between a handy copy and a real disaster recovery strategy.