In some embodiments, ODR application 180 works in connection with remote data forwarding subsystem 160 of storage system 100 to create replication facilities between pairs of similarly configured storage systems 100. Replication of data on a replication facility will be referred to herein as remote data forwarding (RDF). As described in greater detail herein, in some embodiments one of the storage systems, that is configured to implement cyber protection as a cyber recovery vault, includes a snapshot subsystem configured to create point in time copies of storage volumes upon achievement of intermittent consistency with a respective data center. A snapshot of a storage volume, such as production device 140, is point-in-time copy of the storage volume at the storage volume existed at the time that the snapshot was created.

In some embodiments, host computer runs a mainframe software application configured to manage creation and management of storage volumes and to interact with the storage systems 100 that are providing storage resources to the host computer 102, to ensure that the storage systems 100 are correctly configured to provide continuous data protection for mainframe data assets. For example, as shown in FIG. 1, in some embodiments the host computer includes Storage Volume Creation and Management System (SVCMS) 190 that is configured to interact with storage system 100 to cause the snapshot subsystem 170 of storage system 100 to create snapshots of storage volumes that are used by host computer 102 and are stored in the cyber recovery vault on a regular cadence.

In some embodiments, SVCMS 190 interacts with the storage system 100 that is being used as the cyber recovery vault to create a versioned data group of the storage volumes that are contained in the cyber recovery vault that are to be backed up. SVCMS 190 also interacts with the snapshot subsystem 170 of the storage system 100 to cause the snapshot subsystem 170 to create snapsets (groups of snapshots) of the storage volumes of the versioned data group. In this way, the mainframe host 102 can control creation of snapsets of volumes of data by the snapshot subsystem 170 within the cyber recovery vault, to create point in time recovery points of the set of storage volumes stored in the cyber recovery vault.

In some embodiments, as shown in FIG. 1, one application that may be executing on storage system 100 is a Remote Data Forwarding (RDF) application process 160, which causes selected storage volumes to be mirrored by the storage system 100 to one or more similar backup storage systems 100.

It is possible for a primary storage system 100 (R1) to perform data replication to a backup storage system 100 (R2) where the storage systems 100 are compatible and properly configured. The RDF application 160, when executed on storage system 100, enables the storage system 100 to participate in storage system level data replication between sets of mirroring pairs of storage systems 100. A set of storage systems 100 that are configured for data to be mirrored from a primary storage system 100 (R1) to a backup storage system 100 (R2) will be referred to herein as a “Data Replication Facility”. A given storage system, such as storage system 100, may operate as a primary storage system 100 R1 or backup storage system 100 R2 in many mirroring pairs, and hence multiple RDF applications 160 may simultaneously execute on storage system 100 to control participation of the storage system 100 in the mirroring operations. As used herein, the term “cascaded” data replication facility refers to a data replication facility in which data is replicated from a first data center DC1 to a second data center DC2 on a first leg of the data replication facility, and then is replicated from the second data center DC2 to a third data center DC3 on a second leg of the data replication facility. In some embodiments, as described in greater detail herein, a cascaded data replication facility is created in which the third data center DC3 is a cyber recovery vault CRy.

Data transfer among storage systems, including transfers between storage systems 100 for data replication (mirroring) functions, may take place in several ways depending on how the primary storage system R1 handles data written by the host 102 and how the backup storage system R2 acknowledges receipt of data on the data replication facility. Three example data mirroring modes will be referred to herein as synchronous data replication mode (RDF/S), asynchronous data replication mode (RDF/A), and Adaptive Copy Disk data replication mode (ADCOPY-DISK).

In synchronous data replication mode (RDF/S), data is transmitted from the primary storage system R1 to the backup storage system R2 as the data is received from the host 102, and an acknowledgement of a successful write is transmitted by the backup storage system R2 synchronously with the completion thereof. To maintain a synchronous relationship between the primary storage system R1 and the backup storage system R2, each IO from the host 102 is forwarded by the primary storage system R1 to the backup storage system R2 as it is received from host 102, and the primary storage system R1 will wait for an acknowledgment from the backup storage system R2 before issuing a subsequent IO from the host 102.

In asynchronous data replication mode (RDF/A), when data is received from the host 102, the data is written to the primary storage system R1 and a data transfer process is initiated to write the data to the backup storage system R2 on the data replication facility. The primary storage system R1 acknowledges the write operation to the host 102 before the primary storage system R1 has received an acknowledgement that the data has been received by the backup storage system R2. The use of asynchronous data replication RDF/A enables the data on the primary storage system R1 and backup storage system R2 to be one or more cycles out of synchronization, because the primary storage system R1 will continue to execute IOs prior to receipt of acknowledgments from the backup storage system R2. The use of asynchronous replication RDF/A may be beneficial in connection with sites located geographically distant from each other, for example where the distance between the primary storage system R1 and the backup storage system R2 is such that waiting for an acknowledgement from the backup storage system R2 would take considerable time and, hence, reduce responsiveness of the primary storage system R1 to the host 102.

Adaptive Copy Disk (ADCOPY-DISK) data replication mode, as that term is used herein, refers to an asynchronous type of data replication in which data is transmitted from the primary storage system R1 to the backup storage system R2 using a best-efforts type of data replication between the storage systems. In adaptive copy, the data on the backup storage system R2 may be more than one IO out of synchronization with the primary storage system R1 and, accordingly, data consistency at the backup storage system R2 is not guaranteed. Adaptive copy enables bulk copy operations to be implemented between the primary storage system R1 and the backup storage system R2, for example when there are many tracks to synchronize between the two storage systems.

In some embodiments, the storage systems are configured such that a cascaded data replication facility is able to include RDF/S data replication on the first leg and either RDF/A data replication or ADCOPY-DISK data replication on the second leg, or to use RDF/A data replication on the first leg and ADCOPY-DISK on the second leg. However, in some embodiments, the storage systems are configured to not allow creation of a cascaded data replication facility on which RDF/A data replication is used on both the first and second leg of the cascaded data replication facility.

There are many types of data reproduction facilities that may be created, which may have different topographies depending on the number of data centers and the manner in which data is configured to be replicated between the data centers. For example, FIG. 2 is a block diagram of a storage environment including four Data Centers (DC) arranged in a square data reproduction facility that also includes one or more Cyber Recovery (CR) vault sites, according to some embodiments. Although some embodiments are described herein in which the data reproduction facility is implemented using a square topography, it should be understood that other topographies may be used as well depending on the particular implementation.

In the nomenclature adopted in the figures, the letter “R” is used to refer to one or more storage volumes that has been included in a replication facility, such that data that is written to one or more of the storage volumes will be replicated on the data replication facility (R). The numbers following the letter R indicate if the storage volume is a source (designated by a number 1) or a receiver (designated by the number 2). In FIG. 2, the square topography of the data replication facility includes a first pair of data centers, DC1 and DC2, in a primary region and a second pair of data centers, DC3 and DC4, in a non-primary region.

In the data replication facility shown in FIG. 2, production site data center DC1 is the source (R11) on both a first replication session to data center DC2 (Arrow 1) and on a second replication session to data center DC3 (Arrow 2). Synchronous data replication RDF/S is used to replicate data between the data centers in the primary region (on Arrow 1 from DC1 to DC2) and asynchronous data replication RDF/A is used to replicate data between the primary and non-primary regions (on Arrow 2 from DC1 to DC3). DC2 likewise replicates data received from DC1 in a cascaded manner to DC4 over an asynchronous data replication session (Arrow 3). Further, within the non-primary region, DC3 replicates data received from DC1 in a cascaded manner to DC4 (Arrow 4) using synchronous data replication. Hence, in FIG. 2, the square data replication facility includes four data replication sessions:

- DC1→DC2 (Arrow 1: synchronous remote data forwarding RDF/S).
- DC1→DC3 (Arrow 2: asynchronous remote data forwarding RDF/A)
- DC2→DC4 (Arrow 3: asynchronous remote data forwarding RDF/A).
- DC3→DC4 (Arrow 4: synchronous remote data forwarding RDF/S)

In some embodiments, orchestrated disaster recovery application 180 is used to configure the data replication facility to enable data to be mirrored between storage systems DC1, DC2, DC3, and DC4 so that, in the event of a failure of one of the storage systems, the data remains available on one or more of the other storage systems. It should be noted that the same set of storage volumes, that originate at DC1, are replicated on each of the replication sessions (Arrows 1, 2, 3, and 4). In the event of a failure, the orchestrated disaster recovery application 180 enables failover from one storage system to another of the storage systems of the data replication facility. Although FIG. 2 shows a square-shaped data replication facility, it should be understood that data replication facilities can have different numbers of storage systems and different topographies depending on the implementation.

In FIG. 2, a Cyber Recovery (CR) vault site is connected by a replication facility (Arrow 5) with one of the production sites DC2, which is the source of production data that is to be protected using the cyber recovery vault CR2. In FIG. 2, production site DC2 is connected by a replication data facility (Arrow 5) to a cyber recovery vault CR2, although cyber recovery vault CR2 may be connected to any production site DCx. In some embodiments, Cyber Protection Automation (CPA) is implemented using Orchestrated Disaster Recovery application 180, to create copies of data at a cyber recovery vault (CRV) site CR2 that can be used for cyber recovery, for example in instances where the data maintained by the storage systems implementing the data replication facility is corrupted in a malware attack. In some embodiments, the storage volume creation and management system (SVCMS) 190 is used to create regular snapsets at the cyber recovery vault CR2, which is a physically separate, possibly airgapped, cyber recovery vault CR2 using a Cyber Protection Automation (CPA) process. Likewise, in FIG. 2, production site DC4 is connected by a replication data facility (Arrow 6) to a second cyber recovery vault CR4, although the second cyber recovery vault CR4 may be connected to any production site DCx.

An airgap, as that term is used herein, refers to the relative isolation of the storage system implementing the cyber recovery vault CRy from the production site DCx. As shown in FIG. 2, in some embodiments the production site DCx and cyber recovery vault CRy 205 are connected by RDF links 250. In an airgapped solution, these RDF links 250 are normally offline, making data flow impossible. When an airgap connection is enabled, that means that data flow is possible and the RDF links 250 are online. When an airgap connection is disabled, that means data flow is not possible because the RDF links 250 are offline. By using an airgap, it is possible to physically isolate the cyber recovery vault CRy from the production sites DCx of the replication data facility. In some embodiments, the cyber recovery vault CRy controls the state of the airgap to selectively toggle the RDF links 250 between online and offline states.

U.S. Pat. No. 11,580,065, entitled Automated Orchestration of Cyber Protection of a Set of Storage Volumes, describes a process of protecting storage volumes by creation of snapsets of the storage volumes on the production site DCx, linking the snapsets to a set of target devices on the production site DCx, and then transmission of the data from the linked target devices to the cyber recovery vault CRy to enable the snapsets to be stored in the cyber recovery vault CRy. The content of U.S. Pat. No. 11,580,065 is hereby incorporated herein by reference in its entirety. This previous solution required additional space for creation of the snapsets on the production sites, and required additional processing associated with linking the snapsets to the target devices at the production sites and transmission of the data from the target devices to the cyber recovery vault CRy.

According to some embodiments, a new space-saving implementation is provided which removes the space requirements at the source site by adding support for the creation of consistent snapsets directly in the cyber recovery vault, without first creating corresponding snapsets in the production site. Specifically, as described in greater detail herein, a process is described by which snapsets are created from R2 devices in the cyber recovery vault CR4 when intermittent consistency is achieved between the R2 storage volumes in the cyber recovery vault CR4 and the R1 storage volumes in the corresponding data center DC4. The snapsets created in the cyber recovery vault CR4 may optionally be linked to a set of target devices on the cyber recovery vault CR4.

In some embodiments, as described herein, in a cascaded data replication facility, it is not possible to use RDF/A data replication with Multi-Session Consistency (MSC) on two consecutive legs of the cascaded data replication facility. Accordingly, in instances where a first leg of a cascaded data replication facility is implemented using RDF/A+MSC, it is not possible to use this same replication technology on the second leg of the cascaded data replication facility. In instances where the cyber recovery vault CRy is on the end of the second leg, it accordingly is not possible to use RDF/A+MSC to ensure consistency of the storage volumes within the cyber recover vault CRy.

According to some embodiments, orchestrated disaster recovery cyber protection automation with intermittent consistency operates to ensure creation of consistent snapsets of storage volumes in a cyber recovery vault CRy, when the cyber recovery vault CRy is connected to a data center and ADCOPY-DISK data replication is used to mirror data to the cyber recovery vault CRy on the data replication facility.

In some embodiments, a cascaded data replication facility is created, in which asynchronous data replication (Remote Data Forwarding—Asynchronous: RDF/A) with Mulit-Session Consistency (MSC) is used to transmit data on a first leg of the cascaded data replication facility from a first data center DC1 to a second data center DCx. The cascaded data replication facility also includes a second leg in which ADCOPY-DISK data replication mode is used to transmit data from the second data center DCx to a Cyber Recovery (CR) vault CRy.

Consistency is intermittently achieved in the cyber recover vault CRy by determining that less than a maximum number of invalid tracks is owed by the second data center DCx to the cyber recovery vault CRy, and in response to this determination, temporarily suspending transmission of data on the first leg of the cascaded data replication facility while continuing to transmit data from the second data center DCx to the cyber recovery vault CRy. In response to a determination that the number of invalid tracks that is owed by the second data center DCx to the cyber recovery vault CRy has reached zero, an instance of intermittent consistency has been achieved at the cyber recovery vault CRy. Once the data of the storage volumes on the cyber recovery vault CRy in a consistent restartable state, a Storage Volume Creation and Management System (SVCMS) is used to create a snapshot of each storage volume contained in the cyber recovery vault CRy, and transmission of data on the first leg of the cascaded data replication facility is resumed. This process iterates to intermittently achieve consistency between the cyber recovery vault CRy and the second data center DCx, to create periodic snapsets of storage volumes in the cyber recovery vault CRy. Snapshots of the snapset may optionally be linked to a set of target devices at the cyber recovery vault CRy.

FIG. 3 is a block diagram of an example cascaded remote data replication facility, including a first leg (Arrow 7) on which asynchronous data replication RDF/A with Multi-Session Consistency (MSC) is used to transmit data from a first data center DC1 to a second data center DCx, and including a second leg (Arrow 8) in which ADCOPY-DISK data replication mode is used to transmit data from the second data center DCx to a Cyber Recovery (CR) vault CRy. As described in greater detail herein, in some embodiments although the relationship between the source site (DCx) and the cyber recovery vault CRy is ADCOPY-DISK, consistent snapsets in the cyber recovery vault CRy are created from R2 devices which are the RDF targets of the source site production devices through the creation of intermittent consistency in the cyber recovery vault CRy. In some embodiments, creation of intermittent consistency is achieved by temporary suspension of asynchronous data replication RDF/A on the first leg, thus enabling the number of invalid tracks on the second leg to be reduced to zero. When the number of invalid tracks on the second leg reaches zero, the cyber recovery vault CRy has achieved consistency with the source site (DCx) and a snapset of the storage volumes of the versioned data group is able to be created on the cyber recovery vault CRy. After creation of the snapset, transmission of data on the first leg of the cascaded remote data replication facility resumes.

FIG. 4 is a flow chart of an example method of protecting a set of storage volumes of source data center DCx, by creation of snapshots of the storage volumes on cyber recovery vault CRy in the cascaded remote data replication facility of FIG. 3 upon achievement of intermittent consistency, according to some embodiments. In FIG. 4, snapshot creation in the cyber recovery vault CRy is triggered by closure of an airgap between data center DCx and the cyber recovery vault CRy to bring the RDF links 250 online.

As shown in FIG. 4, in some embodiments a cascaded RDF/A+MSC to ADCOPY-DISK data replication facility is created (block 400). For example, in some embodiments a first RDF session (leg 1) is created between data center DC1 and data center DCx, on which RDF/A+MSC is used to replicate a set of storage volumes of a MSC group from data center DC1 to data center DCx (block 405). A second RDF session (leg 2) is created between data center DCx and the cyber recovery vault CRy, on which ADCOPY-DISK is used to replicate a set of storage volumes of the MSC group from data center DCx to the cyber recovery vault CRy (block 410). Because snapshots will be created from the storage volumes in the cyber recovery vault CRy, a versioned data group (VDG) is created in the cyber recovery vault CRy (block 415).

Data is then transmitted from DC1 to DCx on the first leg of the remote data replication facility (block 420). For example, in embodiments where the MSC group includes a plurality of storage volumes, IO operations on the storage volumes at DC1 are mirrored to DCx on the first leg of the remote data replication facility using RDF/A, and MSC is used to ensure consistency between DC1 and DCx.

In some embodiments, as shown in FIG. 4, where an airgap is used to protect the cyber recovery vault CRy, transmission of data from DCx to the cyber recovery vault CRy on the second leg of the cascaded data replication facility is not possible while the airgap is open. Accordingly, in embodiments where an airgap is being used to isolate the cyber recovery vault CRy, the second data center DCx waits for the airgap to be closed, at which point the RDF links 250 are brought online and transmission of data on the RDF links 250 is possible (block