Managing Storage for Container Images

By default all files created inside a container are stored on a writable container layer. This means that:

The data doesn't persist when that container no longer exists, and it can be difficult to get the data out of the container if another process needs it.
A container's writable layer is tightly coupled to the host machine where the container is running. You can't easily move the data somewhere else.
Writing into a container's writable layer requires a storage driver to manage the filesystem. The storage driver provides a union filesystem, using the Linux kernel. This extra abstraction reduces performance as compared to using data volumes, which write directly to the host filesystem.
Support is provided for any storage driver—overlay2 is a good choice.

There are two options for containers to store files in the host machine so that the files are persisted even after the container stops: volumes and bind mounts.

For more information about storage options for containers, see Manage Data in Docker in the Docker documentation.

Use a Volume for the Storage-Group

The container image has a predefined mount point for an event broker's storage-group (/var/lib/solace). If you leave the storage-group unspecified when you start the container, it will be stored in the container's writable layer. This can cause the container to run out of space in the union filesystem and prevent data from being properly migrated during upgrade. We strongly recommended allocating storage outside the union filesystem for the storage-group.

The software event broker requires storage with medium to high bandwidth/IOPs and low latency. For information about getting the best storage performance, consult the best practices documentation for your platform.

A volume is storage in a part of the host filesystem that is managed by the container runtime (for example, Docker Engine) and that has been mounted on the container's filesystem outside the writable layer. We recommend using volumes as the means to persist data for a software event broker container.

For the storage-group mounted as a volume, we recommend xfs as the filesystem, because it has better performance than ext4. If you aren't using xfs or ext4, your filesystem must support the fallocate linux command.

Use External Storage Devices in Production

For production deployments, we recommend that you assign the event broker's storage-group to an external storage device. In the container filesystem, the storage-group is stored in the directory /var/lib/solace.

For information about storage options for Kubernetes, see Storage in the Kubernetes documentation.

For information about storage options for OpenShift, see Understanding Persistent Storage in the OpenShift documentation.

Directory Ownership with Rootless Containers

As described in Rootless Containers, the container runtime makes use of a user namespace to map the UIDs and GIDs within the containers to a (potentially different) block of users it is given access to on the host. This means that if you (as a non-root user) mount a directory from the host into a container, it is owned by the root user inside the container. If you create a file in that directory as the container root user, and then look at that file on the host, you'll see it's owned by your non-root host user.

This is illustrated in the following example:

> whoami
solly

# Mount an empty folder
host> ls /home/solly/folder
host> podman run -u 0 -v /home/solly/folder:/container/volume mycontainer /bin/bash

# Create a file from inside the container
root@container> whoami
root
root@container> touch /container/volume/test

# Check the file owner from inside the container
root@container> ls -l /container/volume
total 0
-rw-r--r-- 1 root root 0 May 20 21:47 test
root@container> exit

# Check again outside the container
host> ls -l /home/solly/folder
total 0
-rw-r--r-- 1 solly solly 0 May 20 21:47 test

This isn't a problem if the container is running as root, but from a security standpoint, it's better to run containers as non-root users. However, when the container user is non-root, it can't access volumes that are owned by the root user. Taking the example above, if the container were run as a non-root user, the touch command would fail with a permission denied error.

To get around this, you can change the ownership of the directory by running chown in a podman unshare session:

podman unshare chown -R <container-user>:<container-group> <path>

The podman unshare command lets you run a command (chown in this case) in the same user namespace as your containers. Because all rootless containers that are run by a given user run inside the same user namespace, you only need to run podman unshare chown once to allow all of a user's containers to access a directory.

Podman resets the ownership of directories and files in volume mounts when it starts the rootless container for the first time, therefore, simply run the podman unshare command after you start the container for the first time. Alternatively, create an empty directory and bind mount it—in this case the correct directory/file ownership is automatically assigned.

For more information, see Rootless Containers and the podman unshare documentation.

Examples of Configuring External Storage

The following are some examples that show how to configure external storage using Docker and Podman, for both volumes and bind mounts.

Example: Docker for Linux (Using a Volume)

To assign a storage-group to a dedicated external volume, do the following:

Attach a disk, or disks, to the host. Since the specific steps for performing this task vary from one environment to the next, we recommend that you consult your environment’s documentation for instructions.
Create the new storage-group volume. For detailed instructions, refer to Use Volumes in the Docker documentation.

Create a new container that mounts the new storage-group volume and maps it to /var/lib/solace:

docker create --network=host --uts=host --shm-size=1g --ulimit core=-1 --ulimit memlock=-1 --ulimit nofile=2448:42192 \
--env 'username_admin_globalaccesslevel=admin' --env 'username_admin_password=admin' --name=solace \
--mount source=storage-group,target=/var/lib/solace  solace-pubsub-enterprise:<version>

If you are using SELinux and need to change the label of the host file or directory that is mounted into the container, you must use the --volume parameter. The --mount parameter does not support the z or Z options for modifying SELinux labels. For more information, see Configure the SELinux Label in the Docker documentation.

Example: Docker for Windows (Using a Bind Mount)

To expand the default storage capability of software event broker containers in Docker for Windows, you can make use of drives from the host, but these drives must be shared with the Docker for Windows Linux VM. Shared drives are configured in the Docker Settings menu.

To assign a directory on the host as external storage for the storage-group using a bind mount, add the following to the docker run command when you start a software event broker, replacing <host-path> with the path on the host where you want the storage-group to be mounted:

--mount type=bind,source=<host-path>,target=/var/lib/solace

The following example uses C:\storage\data as external storage for the storage-group:

> docker run --mount type=bind,source=C:\storage\data,target=/var/lib/solace -d -p 8080:8080 -p 55555:55555 ^
--shm-size=1g --env 'username_admin_globalaccesslevel=admin' --env 'username_admin_password=admin' ^
--name=solace solace-pubsub-standard:<version>

If a Windows process is using one of the ports you are requesting, the command above fails with a "Ports are not available" error. You can use a different port, or you can reserve ports so that Windows can't use them. For details, see the related article on the Solace community.

Example: Docker for Mac (Using a Bind Mount)

To expand the default storage capability of software event broker containers in Docker for Mac, you can make use of drives from the host, but these drives must be shared with the Docker for Mac Linux VM. Shared drives are configured in the Docker Settings menu.

To assign a directory on the host as external storage for the storage-group using a bind mount, add the following to the docker run command when you start a software event broker:

--mount type=bind,source=<host-path>,target=/var/lib/solace

The following example uses /mnt/solace as external storage for the storage-group:

> docker run --mount type=bind,source=/mnt/solace,target=/var/lib/solace -d -p 8080:8080 -p 55556:55555 \
--shm-size=1g --env 'username_admin_globalaccesslevel=admin' --env 'username_admin_password=admin' \
--name=solace solace-pubsub-standard:<version>

On MacOS Big Sur and later, port 55555 (the default SMF port for the software event broker) is blocked. If this port is mapped to a port on a Docker container, the container fails to start, either silently or with a "port in use" error. To avoid this problem, the compose template maps port 55554 on the host to port 55555 in the container.

Example: Rootless Podman for Linux (Using a Bind Mount)

To assign a directory on the host as external storage for the storage-group using a bind mount, do the following:

Create the directory to use as external storage for the storage-group:
```
> mkdir /home/<user-name>/storage-group
```
Change the ownership of the directory so that the container user can access it (this command assumes UID 1000 belongs to GID 0, but this does not have to be the case):
```
$ podman unshare chown 1000:0 -R /home/<user-name>/storage-group
```

Start the container:

> podman run -d -u 1000 -p 8080:8080 -p 55555:55555 \
--shm-size=1g --env 'username_admin_globalaccesslevel=admin' --env 'username_admin_password=admin' \
--mount type=bind,source=/home/<user-name>/storage-group,target=/var/lib/solace,relabel=private \
--name=solace docker.io/solace/solace-pubsub-standard:edge

As discussed above, when you create an empty directory and bind mount it, Podman correctly assigns the directory/file ownership when it starts the container.

Example: Rootless Podman for Linux (Using a Volume)

To assign a storage-group to a dedicated external volume, do the following:

Attach a disk, or disks, to the host. Since the specific steps for this task vary from one environment to the next, we recommend that you consult your environment’s documentation for instructions.
Ensure that Podman's volume path is correctly set to point to the external disk.
Create the new storage-group volume. For detailed instructions, refer to Podman volume create in the Podman documentation.

Create a new container that mounts the new storage-group volume and maps it to /var/lib/solace:

podman create -u 1000 --network=host --uts=host --shm-size=1g \
--ulimit core=-1 --ulimit memlock=-1 --ulimit nofile=2448:42192 \
--env 'username_admin_globalaccesslevel=admin' --env 'username_admin_password=admin' --name=solace \
--mount type=volume,source=storage-group,target=/var/lib/solace,relabel=private docker.io/solace/solace-pubsub-standard:edge

The host mode for --network gives the container full access to local system services such as D-bus and is therefore considered insecure.

Provide feedback