206 to provide data protection capabilities for VM image backups and Kubernetes workloads.

Upon creation of an Mtree, it can identify the domain on which the Mtree is created. All namespace metadata as well as the file data and metadata associated with a Mtree is allocated from resources owned by that domain. The Mtree names as well as Mtree IDs are unique across the cluster. Santorini exposes a global namespace that is a union of all namespaces in all domains.

FIG. 3 illustrates details of the services related to the data path running in Santorini cluster network, under some embodiments. As shown in diagram 300, a product services layer 302 provides the necessary REST APIs and user interface utilities. The API server implements a RESTful interface, allowing many different tools and libraries can readily communicate with it. A client called kubecfg is packaged along with the server-side tools and can be used from a local computer to interact with the Kubernetes cluster.

Below layer 302, the protection software services layer 304 includes a data manager (e.g., Power Protect Data Manager, PPDM) component 305 that provides backup software functionality. Within the scale-out protection storage services layer 306, the File System Redirection Proxy (FSRP) service 307 redirects file operations in a consistent manner based on the hash of a file handle, path, or other properties to instance of the access object service 309. The access object service 309 handles protocols and a content store manager. This means that files are segmented and the Lp tree is constructed by an access object 309. The FSRP 307 redirects file system accesses in a consistent way to the access objects 309 so that any in-memory state can be reused if a file is accessed repeatedly in a short time, and it avoids taking global locks.

Also included in this layer 306 are any number of nodes (e.g., Nodes 1 to 3, as shown), each containing a dedup/compression packer and a key-value (KV) store.

As an L1 is formed, a similarity group is calculated for the L1 based on the SHA1 fingerprints of the L0 segments (or other properties). The similarity group is checked against a mapping table, which leads to consistently routing the L1 based on its similarity group to an instance of a node's dedup/compress service. If similarity groups are in a range of 0-1023, then if there are four instances of the dedup/compress service, then the instances may uniquely be responsible for the ranges 0-255, 256-511, 512-767, 768-1023, though other mappings are possible. The dedup/compress instance will deduplicate the L0 segments relative to other fingerprints within the same similarity group. Other functionality exists in the dedup/compress service such as packing segments into compression regions, containers, and blobs that will be written to an underlying object storage such as provided by the object scale service layer 301, or an object store provided by the public cloud.

Distributed key value stores are also a component of Santorini and are used to hold much of the metadata such as the namespace Btree, the Lp tree, fingerprint index, and container fingerprints. These run as containers within the Santorini cluster and are stored to low latency media such as NVMe. There is also a distributed and durable log that replaces NVRAM for Santorini.

Support Bundle Collector Service

Because of system complexity and variety of features and products, Santorini and other similar cluster systems are typically provided to users by different system providers, or ‘vendors’. Such vendors usually provide the necessary hardware and software infrastructure necessary for users to install and deploy cluster networks in their own environment. As part of this service, vendors may also provide ongoing support in the way of bug fixes, system optimization, software upgrades/updates, configuration recommendations, and so on.

As products are developed for release and deployment, bug fixes, version iterations, and configuration support are important activities of the vendor. Such activities rely on timely and efficient interchange of information between the user and vendor to satisfactorily solve any issues that the user may encounter during and after system deployment. In an embodiment, this information is encapsulated in the form of ‘support bundles’ that comprise the relevant information and data used by the vendor to identify and solve user issues. As stated above, support bundle collection is a critical feature for Santorini, and similar systems, during both customer deployment and during product development. Support bundles are collected, transferred back to the vendor, and analyzed during debugging. Similarly, during development, engineers collect and analyze support bundles to understand problems in their code.

FIG. 4 illustrates the generation and transmission of support bundles between a user and vendor, under some embodiments. As shown in FIG. 4, system 400 comprises a user system 402 having a deployed cluster network including a number of individual nodes 403 coupled to one another through network links 401. User system 402 can represent any appropriate cluster network, but the description will refer to a Santorini system, although embodiments are not so limited. Such a system may be used to implement any appropriate application 420 and file system 421, though the description may specifically mention a Data Domain-based deduplication backup application and the DDFS filesystem.

During installation, deployment, and use, system 402 may encounter various issues related to software/hardware operation, configuration, application compatibility, data corruption, and so on. Any number of issues may arise, and some may be directly addressable by the system vendor. For the embodiment of system 400, the vendor operates a backend system 410 that provides development and service/support functions to both produce the components for the user and debug, support, and improve these components as they are used. For this ongoing process, the user deployment 402 produces, usually on a periodic or on-demand basis, a support bundle 404 comprising any necessary information and data necessary for the vendor backend 410 to provide appropriate fixes, updates, patches, and so on, back to the user system. In an embodiment, the support bundle includes relevant configuration for each individual node 403 and network 401, and the overall system 402. It also includes any specific problem or bug issues, and log information for relevant events and transactions.

In general, a support bundle may include all of the logs written by the various services running within the cluster, as well as topology information listing which services are running on particular nodes and their version numbers. Hardware configuration information may also be included such as drive model numbers, MAC addresses for network cards, etc. Overall system information that may be included comprise total storage capacity, capacity currently utilized, and remaining capacity. Performance statistics may include read/write latencies, read/write throughputs, replication throughput, garbage collection performance, etc. Support bundle information can also include any identified failures such as offline nodes, hardware components, or services. An overall cluster identifier will be included in the support bundle such as a product identifier number. While this information could be gathered across the entire cluster in a single support bundle, filter criteria may also be applied to limit the information gathered to particular nodes and services.

The support bundle information 404 is provided to the vendor backend 410 through a user interface or application program interface (API) component 412 that can send and receive data and commands to and from the user system 402. The vendor system may also include debug tools 414 to correct any specific issues encountered in the user system. Other vendor components include any necessary customer service (personnel) support and/or development tools to address the issues encapsulated in the support bundle 404.

In an embodiment, the support bundle 404 is collected and provided to the vendor by a support bundle processor component 406 that includes a collector service 407 that collects support bundles from the Santorini network across all of the components and nodes in the cluster. The collector service component can be configured to specifically gather the information necessary for a Data Domain (or other application) support bundle.

The collector service 407 collects system configuration and operation information from each component (node, network, application, etc.) and collects logs from persistent volumes (PVs) on each node while applying filter criteria for one or more filters 408. The filters are configured to remove unnecessary data from the user to vendor transmission in a way that does not bottleneck the data flow. The collection is scalable by collecting from each node in parallel with a coordinator-worker design.

FIG. 5 illustrates components of a support bundle collector service, under some embodiments. System 500 illustrates component 406 of FIG. 4 in greater detail, and for an example embodiment of a Kubernetes-based implementation. As shown in FIG. 5, a support bundle request 502, as issued by a user or the system, triggers a Kubernetes API 504 call to trigger support bundle collection from operator 506. This call may include filter criteria to be applied to logs, such as time ranges, component types, and so on. This starts a scheduler 516 that creates a set of BundleJobs 518 for a node 510, where each job is based on a container image stored in an image repository 508. The BundleJob process 518 has the responsibility to create a support bundle for its area of responsibility and upload 520 the bundle to an interface or software component that will then transfer it to the vendor.

Different conditions can trigger or initiate a support bundle request. A typical use case is that the cluster has experienced some sort of problem that generates an alert that goes to the user and/or to the vendor. The user contacts vendor support (or support sometimes contacts the user first), and a support situation is initiated. The vendor directs the customer to generate a support bundle, which the user can do through a GUI or calling a REST API directly. The GUI has configuration options to set filter criteria and possibly set up a schedule for automatically generating a support bundle if that is deemed useful. Support bundles are usually initiated only when a problem is encountered and are typically not generated for normal cluster operation. The request may be configured to be automatically generated upon detection of a defined problem, error, or exception condition, or it can be manually triggered by the user upon such a condition or as needed/desired during cluster operation. An error condition automatically generating a support bundle request may be defined by any appropriate measurable situation, such as if capacity usage exceeds a threshold, processing bandwidth exceeds a maximum allowable percentage, and so on.

In an embodiment, the log files and various other information is gathered from individual nodes into a single support bundle by the DDOS bundler job 518. The support bundle 404 thus comprises individual sets of log and config/statistic information from each node, and which may comprise multiple files, that are then combined together into a single dataset (i.e., the support bundle), which can then be conveniently transferred to the vendor.

FIG. 6 is a flowchart that illustrates a method of gathering a support bundle for Data Domain in Santorini, under some embodiments. As shown in FIG. 6, process 600 involves the three main steps of gathering component statistics, 602; gathering system logs, 604; and transferring the support bundle to in interface for transfer to the vendor, 606.

In an embodiment, the component statistics gathered in step 602 comprise component statistics. Each service may be queried by the support bundle collection service, and each service will provide information to include in a support bundle. As described above, this information may include statistics about the service such as counters, performance measurements, latencies, and other details that will allow support and engineers to understand its internal workings. Any error situations identified by the service can also be included in this information.

Logging

In a cluster network, each component/product has many services, and often many different instances of each service. Services may be started, stopped, and re-started, even on different nodes. Logging the execution and issue of processes across all of the components is necessary to diagnose any issues in the system, and logs should be available across container failures. With respect to FIG. 2, generally the microservices layers 204, 206, and 208 perform the logging functions.

As shown in FIG. 6, step 604 comprises gathering system logs for the support bundle. In general, the following events are logged for all systems. FIG. 7 lists the events and items 702 that are logged for all systems. As shown in FIG. 7, the logged items 702 include system changes 704, which can comprise component availability state changes, such as nodes or processes shutting down; component failures and errors, such as device failures and recoverable memory errors; configuration changes, such as adding a device, installing an application or enabling a service; or changes to source code in production or configuration changes in a production system; and other similar system changes.

Logged items also include authentication and authorization events 706, such as authentication failures (e.g., login failures and account lockouts); failed accesses, such as insufficient permissions to access a database or transaction; or identity and access provisioning activities (e.g., creating new accounts, granting new permissions or changing existing permissions); and other similar activities. Privileged access events 708 can include access to privileged functions (e.g., using a super user level access (e.g., Linux sudo), running the administrative console or logging in as a normal user account; or all actions taken by any individual with root or administrative privileges. The audit-related activities 710 can include changing the event logging configuration or deleting event logs, or audit log failure, among others. FIG. 7 is provided for purposes of illustration and other items may also be logged as needed.

The system can implement log management using any one of a number of possible processes. A first process is to write log files directly to a persistent location. For this method, each pod must manage log rotation to maintain space within the persistent location. The technique involves limiting a log file (“log”) to a specific upper size. When that size is reached, the log file is renamed such as to ‘log.1’ and a new, empty log file is available for new log lines, where a log line is generally a single line of printed text for a log. If log.1 exists, it is renamed to ‘log.2’, and this applies to other logs with conflicting names. The number of log files is limited based on a calculation of the space allowed for logging. By rotating the log names and deleting the highest numbered log, space is managed. This may only be supported for pods with node affinity, otherwise it would be complicated to manage logs across nodes if a service restarted on different nodes. This direct writing process generally involves only simple file writes, and avoids double-write overhead; it does, however, require that each pod must manage its own internal logs, and does add some complexity to support bundle collection.

A second method of writing log files is to write the log files to local files and use a sidecar. Many pods in a Data Domain system already write logs to local files, so a sidecar can be used to track directories and file extensions for updates. The sidecar will copy the log lines to the persistent location. For this method, legacy log file usage is supported, and log management is handled by rsyslogd (or similar) pod. This method, however, imposes a double-write overhead, and the need for each pod to manage its internal logs. There is also a small risk of possible loss of log lines when log lines are transferred from the sidecar to rsyslogd. As used in this description, a ‘sidecar’ is Kubernetes container that tracks log files in a pod and can be used to transfer these files to other services, storage, etc.

A third method of writing log files is to write to STDOUT and use Fluentbit, for example, which are standard Kubernetes service to copy log files from a pod to persistent volume. Kubernetes typically redirects STDOUT to a directory such as ‘/var/log/containers{POD},’ which is generally a well-established log management process handled by rsyslogd pod. This may, however, impose a double-write overhead and possible loss of log lines. Fluentbit is a fast log processor and forwarder for popular operating systems, and tracks updates to the established directory and then copies those updates across the internal network to rsyslog, which will write the lines to the corresponding log file within its persistent volume.

In an embodiment, the logging process is implemented in a way that provides a consistent logging framework with low risk of losing log lines, and maintaining low resource overheads with no major code changes between on-premises and cloud deployments.

Logs are generally available outside of containers, but are not available if a node fails, however, the system can handle drive failures within a node. The system is configured to generate a support bundle from the logs. In system 100, a service or component crash generates a core file, which encapsulates the memory state stored in a file as a crash dump and which is used for debugging purposes. Core files and logs need to be stored persistently outside of pods, since pod memory is ephemeral, i.e., it is erased upon shutdown. Present systems typically store this information by writing directly to a single persistent volume (PV) or writing to local files through a sidecar container. Embodiments improve this method by providing additional PVs for selected services so that support bundle information is maintained in multiple locations for multi-node and multi-service environments to prevent failure of a single PV to result in log loss. Certain aggregations techniques are then used to generate a single support bundle for use by the vendor.

FIG. 8 is a flowchart that illustrates a method of collecting logs to generate a support bundle, under some embodiments. Process 800 of FIG. 8 generally illustrates a log aggregation process for a manageability process shown in FIG. 9. As shown in FIG. 9, a node 901 has storage components 902 (access object, AOB) and 904 (key-value store, KVS). For the embodiment of FIG. 9, components 902 and 904 represent special or critical components/services that require certain needs such as node affinity (services restart on the same node only in the event of failure recovery), and that need to have their logs persevere, i.e., failure of a central PV or any other event that causes losing log lines is not acceptable. For this embodiment, a dedicated PV 906 is associated with each special service, along with the central PV 916. Other non-critical components (e.g., manageability or filesystem pods) can continue to use STDOUT or the sidecar with the single PV.

For the example embodiment of FIG. 9, AOB component 902 is a type of content store that may store Lp trees, name spaces, and so on, and KVS 904 may be a key-value store for metadata. Other types of special storage may include DOB that is a segment store for a deduplication layer, and other similar resources. A respective persistent volume (PV) 906 is associated with each storage, where the PV is mounted using a PV mounter pod 908. In an embodiment the PV mounter pod 908 accesses the persistent volumes 906 used in a first logging case. The system needs a service to mount those persistent volumes so that it can access the logs to include in a support bundle. In some cases, the services that are writing to those persistent volumes may go offline due to software failures, so it cannot count on access logs through those services. The PV mounter pod represents a simple service that is unlikely to become unavailable so as to alleviate this possibility. Although specific example storage types (e.g., AOB and KVS) have been shown, it should be noted that any appropriate storage for objects and data elements may be used.

One or more services 910 using a sidecar or FluentBit 912 use the Rsyslogd function 914, which has its own PV 916 mounted. The support bundle collection job will then include logs from both rsyslog PV (916) and logging PV (906).

As shown in FIG. 8, to collect logs, the system tools copy from a node 901 mounting a PV 906 with the logs, step 802. For this process, Rsyslogd pod 914 is likely to be running, and regular pods may be down. The process adds a PV mounter pod 908 mounts all PVs with the appropriate label, 804. The process may restart if the PVs switch devices due to failures, 806 and will likely also mount/crash-dump to access core files. To create a support bundle, the system will log into to each PV mounter POD running on each node and access the logging PVs and copy the log files.

The logs information may be provided in any appropriate format and contents. For example, logs may be timestamped in ISO 8601 format and with the UTC timezone (e.g., 2020-03-18T22:00:55.040Z). This may then be converted to local time zone when displaying to users. The logs may be defined to encapsulate relevant information, such as [timestamp, severity, source, body].

In an embodiment, logs can be selected based on various different parameters or characteristics, such as keywords, component, data ranges, nodes, and so on. The configuration information that is gathered can be provided in a standard system or industry configuration, or it may be defined by individual or specifically tailored criteria, as needed.

Support bundles can be managed through any appropriate management program or process. A user interface or management program can be configured to list bundles according to a system or user defined format, and bundles can be generated, defined, deleted, or modified, as needed. The system includes mechanisms to automatically delete old or corrupted bundles to provide space awareness and management.

In an embodiment, different log levels may be used. A UI or API function may be used to configure these log levels, for example by a UI/API call to set log level for one or multiple components. The implementation will be specific to each component. Some components may use a registry, while others may use an API.

In an embodiment, different log levels are provided to differentiate among different error/exception conditions and assign logging verbosity. Logs are thus provided in levels are of different verbosity of logging for activities such as debug, error, info, warning, and so on. Any practical number of log levels may be used depending on the number and type of different errors that may occur, and higher log levels imply greater logging activity, and the verbosity can be increased to collect extra logs to debug the problem. For example, the info level may have the least amount of logging and consists of function call history. In contrast, the debug level may be the most verbose and include all of the content from the info level as well as more detailed logging information about operations within functions.

The logs can be configured at different levels of granularity, such as per component-product, component, and node. The system is configured to set different log levels for different components within Santorini. For example, it can set the Data Domain services to log at a detailed level, while all other services on all nodes remain at a default level that has minimal logging. As another example, it may set the PPDM indexing service on node 1 to debug logging, while all other services on all nodes remain at a default level that has minimal logging.

Transferring Support Bundles

In an embodiment support bundles are collected using a support bundle collector service, such as shown in FIG. 5. Once the system logs have been gathered 602 in process 600, they can be transferred to the vendor through a relevant interface, step 606.

FIG. 10 illustrates a system and process of transferring support bundles for a Data Domain application, under an example embodiment. System 950 of FIG. 10 illustrates two different example nodes 956 and 958 in a Santorini cluster. As shown in 950, a support bundle collection is triggered by a UI 952 or API 954 call. Lines 953 (illustrated in white) represent communication with each data protection component to request statistics to be included in a support bundle. These include calls from the support bundle collector in node 956 to the PPDM in node 958 and the access object and deduplication/compression processes in 956 and 958. For the example of FIG. 10, there is one instance of the support bundle collector, and this can reside on any node of the network. As illustrated, there is SSH, UI, and API access to the support bundle collector and support bundles can be transferred out of the cluster to a vendor through either a remote connection (ESE) process or manual download.

Lines 955 (illustrated in black) show logs being gathered from persistent volumes (PVs) and the rsyslogd pod that has its own persistent volume. The support bundle is then transferred out to the vendor or made available for download when users have not enabled external communication. Support bundles can be transferred with a remote connectivity process (e.g., Embedded Support Assist Solution, ESE), manual transfer, or any other appropriate method.

FIG. 11 illustrates log collection and transfers in a multi-node system, under an example embodiment. As shown in FIG. 11, cluster system 1100 contains two nodes 1102 and 1104. Each node has multiple components running such as the AOB 1106 and KVS 1108 and other manageability 1110 and filesystem 1112 components. The manageability and filesystem components run as pods that are use a sidecar mechanism to transfer logs to a persistent volume 1114 managed by rsyslogd. Those components write files to a local directory (e.g., “/var/log/containers”) and manage the log rotation and space quotas within their directory space. A sidecar is added to those pods that tails the local log files and sends the log lines to rsyslogd to be written to its persistent volume 1114. For these components, log collection happens by contacting rsyslog and copying from its attached persistent volume.

Other components in Santorini use a different mechanism for log persistence. A persistent volume is created specifically for those components and attached to those components when they start up. In one embodiment, the AOB, KVS, and several other components may have their own persistent volumes 1116 for logging. For these components, writing logs is very similar to the process described above. The components write logs to a local directory and need to manage log rotation and space quotas. The difference is that the local directory is actually on a persistent volume. For these components, the system gathers logs from their respective persistent volumes.

Some Kubernetes implementations may have limitations that prevent the system from providing persistent volumes to all components or having all components share the same persistent volume. In such a case, a combination of individual and shared persistent volumes may be used.

For the components with directly attached persistent volumes (e.g., 1116), there is a possibility that the component will be offline due to problems such as software bugs. In this case, the system can be configured to start a PV mounter pod 1118 that mounts the detached persistent volumes and copies the logs through that pod.

Besides gathering logs, the system also needs to gather system status information not currently printed to logs. In an embodiment, this is performed in a manner similar to what may be done for auto support generation. In this case, the system queries each software module to print out auto support information. In a Santorini system, a process will similarly query each component to print out its auto support information to include in the overall support bundle. The call mechanism “kubectl_exec” is used to reach each pod and trigger it to print its information. This call is performed using the black connection lines 955 shown in FIG. 10 to the PVs and debug logs from the support bundle connector from the support bundle collector.

For the embodiment of FIG. 11, any number of nodes 1102 or 1104 may be configured to collect respective support bundles, and each operates as illustrated and described for node 1102. FIG. 11 illustrates collecting a support bundle independently for different nodes in a Santorini (or similar) cluster network. In an embodiment, the support bundle collection and transfer system is scalable to provide a support bundle across an entire Santorini cluster to support scalability and space management. This is important for some networks and applications, such as deduplication backup (Data Domain) systems that raise certain challenges for scalability and space management. In such systems, collecting a support bundle can take significant time considering the number of components to collect information from across multiple nodes.

When collecting the support bundle, process 112 provides an option to collect from each node independently and transfer its portion of the support bundle separately. Compression can be applied to the support bundle to reduce space requirements. The overall support bundle then needs to be merged at the receiving side. Due to size limitations of the cluster interface, the support bundle may be divided into more manageable pieces to transfer, as needed.

In an embodiment, support bundle collection may be across the nodes in parallel. One method to send a support bundle for multiple nodes is to configure a controller module that runs from a single location and issues collection commands to all of the nodes and receives back the support bundle content to transfer out to the vendor. This would require a large space to be reserved on that node to handle the full support bundle content.

A second method is to use dedicated worker modules on each node to temporarily store content to alleviate the need for a single node to bear the entire storage requirement. This embodiment is illustrated in FIG. 12, which shows a system for collecting a support bundle from multiple nodes as a group. As shown in FIG. 12, cluster system 1200 contains three example nodes 1202, 1204, and 1206. Each node has the components and functionality illustrated and described with respect to node 1102 in FIG. 11. For this embodiment, each node includes a support bundle worker component 1203. The support bundle worker module will run on each node to collect logs from logging PV on each node and the system can then either copy these bundle to a master node, or copy it separately to the vendor backend.

A support bundle controller module 1205 in a controller node 1204 issues collection commands to all of the nodes and direct respective worker modules 1203 on those nodes to temporarily store the content on each node. In this way, the space requirement scales effectively with the number of nodes in the system. The controller module 1205 then coordinates the transfer of support bundle content from each node through the appropriate interface 1210. From this interface, the partial support bundle can be transferred to the vendor 1212, and the controller will track the progress. The controller 1205, and possibly the workers 1203 if used, would manage space usage related to support bundles. Sufficient space must exist to collect a support bundle, and older bundles should be deleted over time to maintain space for new bundles. A pre-phase process that estimates the amount of space needed for a support bundle by analyzing the existing log files may also be used to further optimize space usage.

Although FIG. 12 illustrates an embodiment for three nodes in a cluster, embodiments are not so limited. Any number of nodes, such as 16 or more may be supported depending on application and system requirements and constraints.

Another embodiment to control the size of support bundles is to create a targeted support bundle based on filter criteria. When collecting a support bundle, the API may use an optional filter criteria that can be passed from the GUI or from direct API calls. The criteria includes start/end times, component-products, components, job IDs, and so on. These criteria pass through the API call to the support bundle collector service 1205 and to each of the bundle collection jobs in workers 1203. The DD bundle job will use that filter criteria to determine which components to collect from, the time ranges from logs to collect, etc. This can dramatically shrink the collection size, and filter options can be used to build knowledge over time, and can also be configured to collect more data rather than less to avoid asking users to collect data repeatedly. One option for setting the filter criteria is that the support team member will provide the filter values when a support request is opened. Depending on the nature of the problem, it may be appropriate to gather logs for a certain period of time leading up to an error message.

In an embodiment, support bundles may be transferred manually rather than transferring directly from the cluster across the Internet. This may be for cases in which the cluster is in a dark site without Internet connectivity, or because the user wishes to perform their own security scan on the support bundle before transfer. In this situation, there is an option for the customer to download the support bundle and then send it to the vendor separately.

In an embodiment, process 112 uses certain core files for the data elements of the support bundle, such as the component statistics 602 and system logs 604. In an embodiment, ‘core’ files are files that include a captured memory state of the independent pods, rather than the cluster as a whole. A core file encapsulates the memory state of the pod (the constituent container or containers) at the time that a program crashes. Core files allow viewing and analysis of variables of the system by capturing memory states during normal operation and extraordinary events, such as system/program crashes, attacks, and so on.

Core files can be formatted, stored, and processed in any appropriate manner. In an embodiment, core files are stored in each node at: /var/lib/systemd/coredump/. Core partitions stores both kernel and process core files. Core files can be compressed with zstd (or similar) compression, and may typically be on the order of 2 GB in size after compression. Core files are named and formatted to facilitate use in a multi-node, Santorini cluster, and so include more than simply name and timestamp information, such as may be common in single node systems.

An example core file format may be:

- core.<podname>.<proc_name>.<uid>.<boot_id>.<pid>.<timestamp>.zst core.ddfsaob-1-0.ddfs.0.9a955b6ae94841c487d548874cacb863.13907.1679230726000000.zst

A core dump helper service may be used to rename a core file name once a process core file is generated. A service can also be used to delete old core files (e.g., set to run every 30 minutes) since core files may tend to be large in size. Old crash core files can be deleted to make sure crash core file used space is smaller than 30 GB (or similar space). Similarly, it can delete old application core files, to make sure application core file used space is smaller than 220 GB (or similar). It can also rename application core files if there is a core file not renamed. Nodes may be configured to store core files for defined time periods, and generally core files are retained indefinitely until more space is needed, however, periodic memory saving operations may also be performed, for example the system may be configured to ensure that there is always 30 GB (for example) free on each node.

For security reasons, access to core files may be restricted or prevented. In an embodiment, a PV mounter pod, such as pod 908 in FIG. 9, will have access to core file directory. Such a PV mounter pod has heightened permissions depending on system configuration. This can be used to transfer core files through the interface 1210 to the vendor or system administrator, where debugging can then be performed.

In an embodiment, certain support bundle automation features may also be provided in conjunction with the core files, such as using existing scripts to process cores in a utility that is used to analyze core files, such as GNU Project debugger (GDB). Such a utility typically shows the state of a running node and the value of variables within the code at the time the core was generated. This can be used to return thread back traces and other details, and avoid transferring core files. Other features include automatically creating and transferring support bundles, providing user opt-in, avoiding delays when filing support tickets, and providing targeted support bundles with the option to gather more information later.

FIG. 13 is a flowchart illustrating an overall method of providing support bundle processing for cluster file system serviceability, under some embodiments. As shown in FIG. 13, process 1300 begins by creating a support bundle to report system states and issues to a vendor, 1302. This step entails gathering system statistics and log information, as illustrated in FIG. 6. For a multi-node cluster system, support bundle information is gathered for all of the nodes by a controller module, such as shown in