Managing Core Files
Core files generated by the Solace PubSub+ software event broker are an important tool used by Solace Customer Support to diagnose and debug issues. A core file is the record of process state that is captured by the Linux kernel when the process has been signaled for termination.
Core files can be very large—on the order of several gigabytes up to the size of the minimum
cgroup limit of the container. You must take special care and consideration when you determine the storage resources to be made available to the event broker for core files.
Managing Core Files for Virtual Machine Images
The event broker Virtual Machine image is delivered with the Linux kernel configured to capture, compress, and store core files in the
diagnostics storage-element. See Using System Scaling Parameters to determine how much disk space should be allocated to the
diagnostics storage-element for your broker configuration. When the disk space requirement for the
diagnostics storage-element exceeds the event broker's volume partition (approximately 9.6 GB), the use of an external disk is recommended.
The event broker fully manages core files in Virtual Machine instances, freeing disk space automatically when the core files are no longer relevant. The broker automatically removes core files that are older than 14 days and those captured by Solace support tools like
Managing Core Files for Container Images
Because container images don't include a Linux kernel, you must manually configure your runtime environment to manage any core files that might be generated. To ensure that core files are correctly managed, you must allocate adequate storage for them and decide how they are named and where they are stored.
You can manage the properties of core files by:
Setting the Resource Limit for the Size of Core Files
The resource limit (
rlimit) for the size of core files must be set high enough to allow the kernel to capture the complete process state. A Linux
rlimit applies to a shell process and all its sub-processes, so its value must be set when the container is created. In some container environments, like Docker Desktop for Windows and Mac, the default
rlimit for the size of core files is 0. With this configuration, no core files are created.
For most container runtimes, you can set the
rlimit when you create the container by using the
ulimit parameter. Where disk space permits, we recommend the core file size
rlimit for event brokers be "unlimited". For example, in Docker Engine and Podman, add the
--ulimit core=-1 command line parameter when the container is created.
Configuring the Core File Pattern
Certain Linux kernel tunables determine how core files are created on the storage device. In particular, the
kernel.core_pattern parameter typically contains one of the following three values:
- a filename, such as
- an absolute path, such as
- a pattern starting with a pipe symbol (
When the Core File Pattern is a Filename
When the value of
kernel.core_pattern is a filename, it is interpreted as a pattern for the core dump process to follow for creating core files in its working directory. For containers, this working directory exists in the mount namespace of the container process.
The most common core file pattern is
core, often with
kernel.core_uses_pid set to '1'. In this case, core files are generated in the event broker’s working directory (
/usr/sw), which exists in the container's ephemeral (writeable) layer and consumes disk space in the container runtime's backing store. It is not possible to bind-mount the working directory as a volume.
core is the default core file pattern for many versions of RHEL and Ubuntu, the event broker manages these core files, freeing disk space automatically when the core files are no longer relevant. The broker automatically removes core files that are older than 14 days and those captured by Solace support tools like
When the Core File Pattern is an Absolute Path
kernel.core_pattern is an absolute path, the value is interpreted as a pattern for the core dump process to follow for creating core files in its mount namespace. If the defined path is not bind-mounted on the host, core files are written to the container’s ephemeral (writeable) layer, consuming disk space in the backing store of the container runtime ( typically reserved for container images).
When an absolute path is specified, the event broker cannot manage core files—it attempts to capture them on a best-effort basis using Solace support tools. Core files are large and can easily exceed the disk space allocated to the runtime’s backing store. Without management, it is possible to corrupt the backing store, causing containers running on multi-tenant hosts to crash and fail to restart. Because of this, we do not recommend this core file pattern.
When the Core File Pattern Begins with a Pipe Symbol
Since kernel version 2.6.19, Linux supports an alternate syntax for
kernel.core_pattern in which the first character is a pipe symbol (
|). When this alternate syntax is used, the kernel treats the rest of the pattern as a command to run, which is invoked in the initial namespaces (that is, not within the container’s process or mount namespace). The core dump is written to the standard input of that program instead of to a file. This pattern is common when a core file manager, such as
systemd-coredump, is being used.
When a core file manager is used, the event broker cannot manage core files or capture them using Solace support tools. Therefore, we do not recommend this core file pattern.