CloudPi

Provision External Storage

In this step, you’ll prepare an external storage device, either a spinning disk or a solid state drive. This device will used to hold configuration files for Docker containers; Docker containers, Docker images and Docker volumes; with additional space allocated to user files.

By the end of this step you will have:

Attached and manually identified the external storage device.
Configured the device for Logical Volume Manager (LVM).
Created and mounted logical volumes to hold Docker persistent data and user files.

If you haven’t installed Ansible yet, do that first.

Can I skip it?

You can run your entire system off of the micro-SD card if you want. You can also partition the storage device manually using command-line tools.

Summary of Commands

sudo parted /dev/sda print
ansible-playbook provision-storage.yml
df -h ; cat /etc/fstab ; sudo lvs

Why an External Disk or SSD?

With microSD cards available in capacities of 256G, 512G, or even 1TB, you might be wondering why this project uses an external storage device. The simple reason is that separating the operating system from the data helps make recovery easier when things go wrong. Another reason is price. When comparing 1TB microSD and a 1TB 2.5” NAS-rated SSD, both from the same manufacturer, the list prices are the same. Theoretically, the NAS-rated SSD should be a better fit for a device that runs 24/7.

Attaching and Identifying the External Storage

First, plug the storage device into the SATA side of the USB to SATA adapter cable plug the USB side into the Raspberry Pi. Be sure to plug it into one of the blue USB3 ports to get the best performance. To make the disk easier to identify, don’t plug in any other storage devices at this time.

Next, identify the device using the command sudo parted /dev/sda print. Pay particular attention to the first two lines.

The manufacturer and model number should match the device you plugged in.
The device should be /dev/sda and the capacity should match the device you plugged in.

Here’s an example of a 160G Hitachi spinning disk:

$ sudo parted /dev/sda print
Model: Hitachi HTS543216L9A300 (scsi)
Disk /dev/sda: 160GB
...

Partition information has been truncated for clarity.

If you don’t see /dev/sda with the manufacturer and capacity you expect, stop and figure out what’s wrong before you proceed!

Understanding the Ansible Playbook

Creating partitions and filesystems is a destructive process. Any information already on the storage device will be lost. Therefore, you should proceed with caution. To this end, the playbook has been designed to fail unless you provide it with a confirm_device variable with the device name of the storage device.

Do not override confirm_device by editing the playbook.

Use Ansible’s –extra-vars command-line option instead, like this:

ansible-playbook provision-storage.yml --extra-vars confirm_device=/dev/sda

If you don’t provide confirm_device or you provide the wrong value, you will see a fatal error whe the playbook gets to the task of Verifying device.

Running the Ansible Playbook

Copy the contents of provision-storage.yml to your Pi and run it. The playbook will take care of the following tasks.

Install the LVM tools package.
Create a new GUID partition on /dev/sda that spans the entire device.
Create a volume group called vg1, using /dev/sda1 as its physical volume.
Create three logical volumes: vol01, vol02, and vol03.
Create ext4 filesystems on the three volumes.
Create directories where needed for /opt/docker, /var/lib/docker, and /srv.
Mount /dev/vg1/vol01, /dev/vg1/vol02, and /dev/vg1/vol03 on /opt/docker, /var/lib/docker, and /srv, respectively.

Volume Sizes

The logical volume sizes are hard-coded as 5G for /opt/docker, 10G for /var/lib/docker, and 100G for /srv. They are intentionally conservative to fit the constraints of smaller storage devices. But, if you have a device with a lot of space, you can override the default sizes using the Ansible variable called logical_volumes. Alternatively, you can resize them as the your storage needs grow. The lvresize and resize2fs command-line tools will enable you to do this.

If all goes well, the output from the playbook should look like this:

$ ansible-playbook provision-storage.yml --extra-vars confirm_device=/dev/sda

PLAY [Provision external storage device for logical volumes (LVM)] *************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Verifying device] ********************************************************
ok: [localhost]

TASK [Installing LVM2] *********************************************************
changed: [localhost]

TASK [Repartitioning device] ***************************************************
changed: [localhost]

TASK [Creating volume group] ***************************************************
changed: [localhost]

TASK [Creating volumes] ********************************************************
changed: [localhost] => (item={'name': 'vol01', 'mount_point': '/opt/docker', 'size': '5G'})
changed: [localhost] => (item={'name': 'vol02', 'mount_point': '/var/lib/docker', 'size': '10G'})
changed: [localhost] => (item={'name': 'vol03', 'mount_point': '/srv', 'size': '100G'})

TASK [Creating filesystems] ***************************************************
changed: [localhost] => (item={'name': 'vol01', 'mount_point': '/opt/docker', 'size': '5G'})
changed: [localhost] => (item={'name': 'vol02', 'mount_point': '/var/lib/docker', 'size': '10G'})
changed: [localhost] => (item={'name': 'vol03', 'mount_point': '/srv', 'size': '100G'})

TASK [Creating mount-point directories] **************************************
changed: [localhost] => (item={'name': 'vol01', 'mount_point': '/opt/docker', 'size': '5G'})
changed: [localhost] => (item={'name': 'vol02', 'mount_point': '/var/lib/docker', 'size': '10G'})
changed: [localhost] => (item={'name': 'vol03', 'mount_point': '/srv', 'size': '100G'})

TASK [Mounting filesystems for Docker persistent config] ***********************
changed: [localhost] => (item={'name': 'vol01', 'mount_point': '/opt/docker', 'size': '5G'})
changed: [localhost] => (item={'name': 'vol02', 'mount_point': '/var/lib/docker', 'size': '10G'})
changed: [localhost] => (item={'name': 'vol03', 'mount_point': '/srv', 'size': '100G'})

PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=5    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Verifying Configuration

The following example shows some of the commands you can use to check the storage configuration and some typical results.

$ df -h
...
/dev/mapper/vg1-vol01  4.9G   24K  4.6G   1% /opt/docker
/dev/mapper/vg1-vol02  9.8G   24K  9.3G   1% /var/lib/docker
/dev/mapper/vg1-vol03   98G   24K   93G   1% /srv

cat /etc/fstab
...
/dev/vg1/vol01 /opt/docker ext4 defaults,noatime 0 0
/dev/vg1/vol02 /var/lib/docker ext4 defaults,noatime 0 0
/dev/vg1/vol03 /srv ext4 defaults,noatime 0 0

$ sudo lvs
LV    VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
vol01 vg1 -wi-ao----   5.00g
vol02 vg1 -wi-ao----  10.00g
vol03 vg1 -wi-ao---- 100.00g

Extraneous information has been replaced with ellipses (…) to aid clarity.

Next Steps

Now that there’s plenty of disk space, your Pi is ready to start serving files. You can jump ahead and rush right into installing Docker and Portainer to start deploying applications if you want.

Or you can take a look at some of the optional components like:

DNS to let you refer to devices by name rather than IP address.
A certificate authority (CA) to enable secure HTTPS connections to servers.
LDAP directory services to let you users maintain a single user name and password across multiple applications.

All these will help make the system more robust and user friendly, even though they’re not strictly necessary.

If you decide to pursue these optional services, start with deploying a simple web server to have something to test against.

This one sparks joy. —Marie Kondo

This site is open source. Improve this page.