Like the title? “Day Two” one of those fancy, industry buzzwords for ongoing maintenance after the system is up and running. (Apparently you get three days: day zero to plan it, day one to implement it, and day two to support it. Day two can stretch into weeks, months or even years.)
This document touches upon some of the things you’ll need to do to keep the system up and available on “day two”.
If you’re an experienced admin who already knows about logical volume management, regular software updates, backups and disaster recovery planning, and all that, feel free to skip this.
When storage was provisioned, the space allocated to logical drives was intentionally small. If you have a large external device, you’ll probably want to expand your logical volumes, particularly the one that holds shared files. The easiest way to do this is using Cockpit.
Cockpit makes it easy, but you can also use the command-line. With this method, it’s two commands. First, lvextend is used to expand the logical volume. Second, resize2fs will expand the filesystem to take advantage of the extra space.
Keeping current on system software is essential for maintaining a secure and healthy operating system. The easiest way to do this is to get in the habit of logging in to Cockpit every week or so and looking at the Health panel on the System Overview page. It will tell you if the system is up to date and give you a link to click if it’s not.
You can also use the update-system.yml Ansible playbook, if you prefer the command-line. There’s also the traditional sudo apt-get update sudo apt-get upgrade method.
Hardware eventually fails. Users will inevitably say, “Oops, I didn’t mean to delete that one!” Power will go out. Basements will flood. Stuff happens. Having a good backup will make these events less stressful.
You may be familiar with the 3-2-1 backup up rule. If not, this is it in summary: Keep at least 3 copies of the data, stored on 2 different storage media, with 1 being located off-site.
Below are a few ways you can approach 3-2-1 with limited hardware and budget.
This is the least expensive, and also the least robust. Chances are good you will lose at least some data.
The good part is:
The bad part is:
If you’re going to be only relying on Nextcloud sync, make sure you keep copies of your LDAP user accounts LDIF file, any DNS zone customizations you’ve made, the local certificate authority certs and anything you have stored in the pi user’s home directory. The configuration files in /opt/docker should periodically be copied somewhere safe, as well as the contents of /var/lib/docker.
This option involves attaching another external storage device, but making it available only the root user. Important data is copied to the secondary device on a schedule.
The good part is:
The bad part is:
This option is a lot like the previous one in that it involves periodic copies with rsync, but the sync is to a storage device on another Pi. If you have an older Raspberry Pi model 2 or 3 lying around collecting dust, this can be a good option for service as an inexpensive backup host.
The good part is:
The bad part is:
There are plenty of companies that offer internet-based backup solutions. It seems like every year another list comes out with the top ten best plans, so they’re not hard to find.
The good part is:
The bad part is:
You can choose any one or any combination of the options listed above. At the very least, make sure the Nextcloud “virtual files” feature is disabled on your desktop clients to satisfy the “multiple copies” part of the 3-2-1 rule. If you have the means, set up second external storage device and a cron job for periodic, incremental copy with rsync to offer protection against accidental deletions and satisfy the “multiple storage media” part of 3-2-1. Locating that second storage device on another Pi or off-site is up to you and your budget.
Having backups is one thing. Knowing what to do when the system is down is another. Think about these things now so you’re not panicking later. Below are a few possible scenarios based on my past experience with running a Raspberry Pi 24/7.
The one problem I’ve experienced most is a corrupted microSD card. They’re really not designed for the strain of a high number of write operations and failure is more a question of when than a question of if. Having separate OS and data storage devices helps with this situation.
Back in the early implementations of the project, when I was using an USB flash drive for storage, I had a couple of them fail. Like the failed microSD cards, it’s because of pushing a device beyond what it was designed for. I’m hoping with a NAS-rated device, this external storage failure won’t happen again. But, it’s good to be prepared.
If you’re not using a backup device, the Docker persistent data in logical volumes vol01 and vol02 will be unrecoverable.
The worst situation of all is the total lose of the Pi hardware, microSD card and external storage device. In this situation all you have is your backup.
This scenarios above offer only a few possibilities of things that can go wrong. Be sure to plan for your own potential disasters while taking into account the importance of the data your storing and how difficult it is to replace.
If you have a second Pi of equal specs, run through the exercise of building the system again and recovering the data. Try make the second Pi able a close copy of the original and see if all your data is there.
Try to take into account situations not only involving loss of hardware, but also ransomeware infecting one or more of your client devices. How would you recover from these situations? Plan for it now so you won’t be surprised later.
Depending on your backupp strategy, you may not have copies of the LDAP database and DNS zones stored on the operating system’s microSD card. However, you can make an initial export of the data and keep a copy somewhere safe.
These methods do nothing to back up the configuration for LDAP and DNS servers, only the data.
The command sudo slapcat -n1 will dump the contents of the LDAP user database. This is where all the user accounts, groups, and associated password hashes are stored. Store the output in a file somewhere safe and you’ll be able to reimport later if needed.
The command sudo rndc dumpdb -zones will dump the contents of all configured DNS zones to a file. On Raspberry Pi OS 11 (Bullseye) that file’s path is /var/cache/bind/named_dump.db It’s a plain text file and can be read with any text editor. It also contains a lot of comment lines.
The important file in the CA are already in plain text. You just need to keep them somewhere safe. Be sure to get all the files: the root certificate and key, the intermediate certificate and key, as well as any host certificates and keys you have issued.
If you’ve configured your client devices to trust your CA, the intermediate and root certificates will already be in your client’s certificate store. You will not have the private keys, however. Be sure to keep copies of these in a safe place.
For more discussion on backup and other interesting things you can do with this thing you just built, see the Cloud Pi wiki
Every bad situation is a blues song waiting to happen. —Amy Winehouse