Core files generated by the Solace PubSub+ software event broker are an important tool used by Solace Customer Support to diagnose and debug issues. A core file is the record of process state that is captured by the Linux kernel when the process has been signaled for termination.
Core files can be very large—on the order of several gigabytes up to the size of the minimum cgroup limit of the container. You must take special care and consideration when you determine the storage resources to be made available to the event broker for core files.
Managing Core Files for Container Images
Because container images don't include a Linux kernel, you must manually configure your runtime environment to manage any core files that might be generated. To ensure that core files are correctly managed, you must allocate adequate storage for them and decide how they are named and where they are stored.
You can manage the properties of core files by:
Setting the Resource Limit for the Size of Core Files
The resource limit (rlimit) for the size of core files must be set high enough to allow the kernel to capture the complete process state. A Linux rlimit applies to a shell process and all its sub-processes, so its value must be set when the container is created. In some container environments, like Docker Desktop for Windows and Mac, the default rlimit for the size of core files is 0. With this configuration, no core files are created.
For most container runtimes, you can set the rlimit when you create the container by using the ulimit parameter. Where disk space permits, we recommend the core file sizerlimit for event brokers be "unlimited". For example, in Docker Engine and Podman, add the --ulimit core=-1 command line parameter when the container is created.
Configuring the Core File Pattern
Certain Linux kernel tunables determine how core files are created on the storage device. In particular, the kernel.core_pattern parameter typically contains one of the following three values:
- a filename, such as
core - an absolute path, such as
/tmp/cores - a pattern starting with a pipe symbol (
|)
When the Core File Pattern is a Filename
When the value of kernel.core_pattern is a filename, it is interpreted as a pattern for the core dump process to follow for creating core files in its working directory. For containers, this working directory exists in the mount namespace of the container process.
The most common core file pattern is core, often with kernel.core_uses_pid set to '1'. In this case, core files are generated in the event broker’s working directory (/usr/sw), which exists in the container's ephemeral (writeable) layer and consumes disk space in the container runtime's backing store. It is not possible to bind-mount the working directory as a volume.
Because core is the default core file pattern for many versions of RHEL and Ubuntu, the event broker manages these core files, freeing disk space automatically when the core files are no longer relevant. The broker automatically removes core files that are older than 14 days and those captured by Solace support tools like gather-diagnostics.
When the Core File Pattern is an Absolute Path
When kernel.core_pattern is an absolute path, the value is interpreted as a pattern for the core dump process to follow for creating core files in its mount namespace. If the defined path is not bind-mounted on the host, core files are written to the container’s ephemeral (writeable) layer, consuming disk space in the backing store of the container runtime ( typically reserved for container images).
When an absolute path is specified, the event broker cannot manage core files—it attempts to capture them on a best-effort basis using Solace support tools. Core files are large and can easily exceed the disk space allocated to the runtime’s backing store. Without management, it is possible to corrupt the backing store, causing containers running on multi-tenant hosts to crash and fail to restart. Because of this, we do not recommend this core file pattern.
When the Core File Pattern Begins with a Pipe Symbol
Since kernel version 2.6.19, Linux supports an alternate syntax for kernel.core_pattern in which the first character is a pipe symbol (|). When this alternate syntax is used, the kernel treats the rest of the pattern as a command to run, which is invoked in the initial namespaces (that is, not within the container’s process or mount namespace). The core dump is written to the standard input of that program instead of to a file. This pattern is common when a core file manager, such as apport or systemd-coredump, is being used.
When a core file manager is used, the event broker cannot manage core files or capture them using Solace support tools. Therefore, we do not recommend this core file pattern.