402. Devices 428 may be any sort of device of a computing system, such as harddrives, graphics cards, ASICs, I/O devices, and/or other such devices. Furthermore, communicatively and/or electrically coupling together cache coherent switch on chips 402A and 402B may provide for greater system redundancy, increasing reliability.

Though the embodiment of FIG. 4 illustrates cache coherent switch on chips 402A and 402B being electrically and/or communicatively coupled between expansion port 472, other embodiments may couple various cache coherent switch on chips with other techniques, such as over a local area network (LAN), over the internet, and/or over another such network.

In certain embodiments, each of cache coherent switch on chip 402A and 402B may include their own virtual hierarchies. When coupled as in FIG. 4, the virtual hierarchies of one or both of cache coherent switch on chips 402A and 402B may be utilized for switching operations.

FIG. 5 illustrates system 500 that includes cache coherent switch on chips 502A to 502C, CPUs 504A and 504B, management 530, and devices 528. Each of cache coherent switch on chips 502A, 502B, and 502C may include their own individual virtual hierarchies. In certain embodiments, cache coherent switch on chips 502 may include a fabric manager 540 to manage resources connected to the ports (e.g., ports 516) of cache coherent switch on chips 502. The fabric manager 540 may connect to higher level management software entities (e.g., management 530) via Ethernet 568 (as, for example, Redfish over Ethernet) and/or another network or protocol (e.g., PCI protocols). Ethernet 568 may further communicatively and/or electrically couple cache coherent switch on chips 502A, 502B, and 502C and CPUs 504 and devices 528.

Fabric manager 540 may be configured to allocate and/or deallocate resources attached to the ports of cache coherent switch on chips 502 to applications running on such ports (e.g., to applications running on ASICs coupled to ports of cache coherent switch on chips 502). Fabric manager 540 may be configured to receive signals (e.g., data) from an upstream port and direct the signal to the appropriate downstream port. Various techniques for receiving and directing such signals (e.g., packet flows) are described herein. Fabric manager 540, as well as other firmware and/or software may further manage hot plug coupling by devices 528 to downstream CXL ports. Fabric manager 540 may also manage the inventory of various devices coupled to the ports of the respective cache coherent switch on chip 502.

Fabric manager 540 may be communicatively coupled to management 530 for top level management of system 500, including management of the various cache coherent switch on chips 502 described herein. Thus, in various embodiments, management 530 may be, for example, a baseboard management controller and/or another management device or server configured to provide management/orchestration. In various embodiments, management 530 may interface with fabric management 540 to provide for management of the various cache coherent switch on chips (e.g., via a specific fabric management API).

Fabric manager 540 may be implemented within firmware of cache coherent switch on chip 502 (e.g., within the firmware of a microprocessor of cache coherent switch on chip 502). Such firmware may include a system fabric manager that implements the logic for operations to be performed by switch hardware and other helper functions for implementing the API and a CXL fabric manager for implementing the front-end fabric manager APis according to the CXL specifications.

In certain embodiments, a CXL single logical device (SLD), such as device 528A, may be hot-inserted into or hot-removed from cache coherent switch on chip 502B (e.g., via port 516E, which may be a PCI and/or CXL protocol port). When such an SLD is first hot-inserted, it is assigned to fabric manager 540B. Diagnostics may be performed on the newly inserted SLD (e.g., either run as self-diagnostics by device 528A or run via diagnostics software on the processing core of cache coherent switch on chips 502). After the SLD has been determined to be ready, it can be assigned to one of the ports (e.g., port 516E) of cache coherent switch on chip 502B based on policy (e.g., due to a virtual hierarchy) or via a command (e.g., from software within system 100).

The assignment may include binding the corresponding downstream PPBs of a cache coherent switch on chip 502 to one of the vPPBs, virtual hierarchies, and host port of cache coherent switch on chip 502. The managed hot-inserted device 528A is then presented to the host port (e.g., port 516E) after its assignment to the respective virtual hierarchy to allocate device 528A. The host CPU (e.g., the CPU within the respective cache coherent switch on chip 502) may then discover device 528A (e.g., via software), load software for device 528A and begin communicating with device 528A.

FIG. 6 illustrates system 600. System 600 may illustrate cache coherent switch on chip 602. Cache coherent switch on chip 602 may include a plurality of root ports 632 and a plurality of virtual hierarchies 624. The plurality of root ports 632 may include the ports described herein, as well as, for example, that of internal components within cache coherent switch on chip 602, such as microprocessors/CPUs and/or other components. Each root port 632 may be assigned to downstream CXL protocol resources. Each virtual hierarchy 624 may include a plurality of vPPBs, where certain vPPBs 634 are associated with root ports 632 and other vPBBs 636 are associated with PPBs 638. Various multi-logical devices (MLDs) 640 may be coupled to downstream ports via certain PPBs 638.

Cache coherent switch on chip 602 may include a plurality of root ports 632. Such root ports 632 may include, for example, ports associated with a processing core of cache coherent switch on chip 602 as well as external devices. Root ports 632 may be assigned to downstream CXL resources, including embedded accelerators within system 600. Fabric manager 640 may include a processor (e.g., an ARM processor or another type of processor) and such a processor may be a part of one or more virtual hierarchies 624. Various downstream PPB ports 638 may be communicatively coupled to MLDs 640. The assignment of MLDs 640, as well as other components such as SLDs, memories, accelerators, and other such components, to certain PPBs 638 and vPPBs 636 may be controlled by fabric manager 640. Thus, fabric manager 640 may detect that a component has been coupled to a port of cache coherent switch on chip 602 and accordingly assign the component to the appropriate virtual hierarchy 624 (e.g., based on the detected type of the component). Furthermore, the appropriate PPB 638 and/or the vPPB 636 may be assigned to the component. In certain embodiments, such assignment may be based on the detected type of the component and on virtual hierarchy 624.

FIG. 7 illustrates system 700 that includes cache coherent switch on chip 702. As shown in FIG. 7, MLDs 740A and 740B are coupled to PPBs 738A and 738B, respectively. MLDs 740A and 740B include memories 714A and 714B, respectively, and are thus utilized as memory expansion. Coupling of memories 714A and 714B to system 700 allows for an increase in the amount of memory of system 700 (e.g., system 700 may be, for example, a single socket server).

In various embodiments, the amount of memory attached to a socket is limited by the number of channels that the socket supports. In certain situations, in a data-centric environment, an entire operating data set may not fit in a server's available memory, resulting in poor performance and increased latency when processing the data. Cache coherent switch on chip 702 addresses this problem by allowing for low-latency memory expansion due to memories 714A and 714B via the ports of cache coherent switch on chip 702, increasing the amount of memory available to a host CPU (beyond what could be connected directly to the CPU). Memories 714 may be DDR4, DDR5, future DDR, DRAM, PM, NVMe, Low-Power Double Data Rate (LPDDR), and/or other such appropriate memory drives which may be expanded via CXL protocol through cache coherent switch on chip 702.

Such an ability of cache coherent switch on chip 702 is particularly beneficial in providing cost and performance advantages for memory intensive applications that would otherwise require a computing device with a large memory footprint or result in poor performance in a less expensive computing device with limited memory.

FIG. 8 illustrates system 800 that includes a plurality of servers 842A and 842B. Each server 842 may include its own cache coherent switch on chip 802, a plurality of memories 814 communicatively coupled to each cache coherent switch on chip 802, and a microprocessor 804 communicatively coupled to each cache coherent switch on chip 802. Cache coherent switch on chip 802A and 802B may be communicatively coupled via fabric switch/bus 844. In various embodiments, fabric switch/bus 844 may be, for example, a switch fabric, a bus bar, and/or another such technique for communicating signals between different server devices.

As illustrated in FIG. 8, memories may be pooled between different microprocessors 804. Such memories may include memories 814 communicatively coupled to cache coherent switch on chips 802 and/or memory that is socket connected to various microprocessors 804. Thus, cache coherent switch on chips 802 may allow for pooling of memory and other resources (e.g., AI, ASICs, GPUs, SNICs, NVMe, storage, and/or other such resources) between servers 842 that are communicatively coupled via switch fabric/bus 844. As signals communicated between switch fabric/bus 844 may be similar to that of signals communicated within a single server device, cache coherent switch on chips 802 may allow for sharing of such resources in a similar manner to that described herein. In various embodiments, a plurality (two or more) of servers 842 may, accordingly, pool memory resources such as DRAM, PM, and/or other such memories. Such resources may be shared over fabric switches for memory pooling inside a server, between servers within a server rack, between various servers and racks within a data center, and/or between data centers. In a further embodiment, messages may be passed between components in a manner similar to that of the sharing of resources. Such techniques allow for reduction in the communication of messages between various components, increasing the performance of, for example, AI or ML workloads on processors.

In various embodiments, cache coherent switch on chips 802 may provide compression and/or decompression ability to conserve persistent memory as well as crypto ability to provide added security between transactions into and out of persistent memory.

In certain embodiments, a prefetched buffer scheme may be utilized at the memory source. Accordingly, in various embodiments, cache coherent switch on chips 802 may include memory prefetchers 878. Memory prefetchers 878 may be an intelligent algorithm run by the processing core of the cache coherent switch on chips 802. Memory prefetchers 878 may be an artificial intelligence (AI) or machine learning (ML) prefetcher configured to predict the addresses of future accesses to memories based on past access patterns by the hosts, and prefetch data from such memories for those addresses to store in DRAM buffers to reduce the latency of future accesses by the host applications. In certain embodiments, accelerators communicatively coupled to cache coherent switch on chip 802 may also be configured to provide prefetching when pooling resources via cache coherent switch on chips 802 between servers 842A and 842B.

In certain embodiments, disaggregated servers 842 may pool memory and/or other resources across a midplane (e.g., bus 844). Thus, for example, in a chassis or blade server, a large shared pool of memory on memory cards/blades is available to be used by server cards/blades (that could be lightweight servers, aka thin servers, with a minimal amount of their own memory connected to the CPU socket). Such memory pooling may provide cost and/or power consumption advantages by reducing the amount of unused memory and/or other resources in data center servers, as memory/resource pooling allows for greater flexibility and, thus, a lower requirement for fixed resources. Servers may also be more flexibly configured due to the advantages of resource sharing.

In a certain use case, current typical server systems may include 512 gigabyte (GB) or so of volatile memory in cloud service provider infrastructure. A portion of this memory is typically stranded due to lower memory utilization for all the applications. Additionally, certain cloud environments include highly memory intensive applications that require more than 512 GB of memory. Currently, for example, platforms allocate all the servers with 512 GB memory due to simplicity, stranding the memory resources in the majority of the servers in order to have enough capacity for edge use cases. The currently disclosed cache coherent switch on chips addresses this memory stranding problem by allowing for the sharing of CXL protocol persistent memory both inside the server system and to outside servers connected via a network.

FIG. 9 illustrates system 900 that includes server 942, switch fabric/bus 944, and memory appliance 946. Memory appliance 946 may be a shared or expansion memory for server 942. System 900 allows for memory 914A of cache coherent switch on chip 902A to be declared as a cache buffer for persistent memory ports (e.g., ports coupled to switch fabric bus 944 and, thus, memory 914C of memory appliance 946). Utilizing memory 914A as a read/write buffer hides the access time of utilizing memory appliance 946 and, thus, memory 914C.

In various embodiments, there may be both write and read flows for memory 914A. In a write flow, microprocessor 904 may indicate that writes on memory 914A are steered to a DRAM buffer port of cache coherent switch on chip 902A. For such writes, cache coherent switch on chip 902A may check to ensure that memory 914C is configured to provide buffer write/read commands to memory 914A, allowing for memory 914A to be used as a buffer for memory 914C. Thus, memory 914C is updated so that the buffer write/read address of memory 914C refers to that of memory 914A. Memory 914A may then be accordingly utilized as a buffer for memory 914C, avoiding the increase in access time of utilizing memory appliance 946.

In certain embodiments, for a read flow, microprocessor 904 may first query the buffer port of memory 914A for the wanted data. If such data is present within the buffer of memory 914A, the data may be provided to microprocessor 904. If memory 914A does not include such data, memory 914C may be queried and the requested data may be provided from memory 914C over switch fabric/bus 944.

In certain embodiments, the cache buffers of memory 914A include AI/ML prefetch algorithms. The algorithm is configured to predict the next set of addresses (expected to be fetched by the applications) and configures a direct memory access (DMA) engine to prefetch those addresses and store the data in read/write buffers, to be ready to be read by the applications. In certain embodiments, cache coherent switch on chip 902A is configured to keep statistics of hit ratios for each line that was prefetched to provide feedback to the algorithm for continuous improvement (e.g., to determine which prefetched data has been utilized).

In certain embodiments, cache coherent switch on chip 902A may provide instructions for operation of the memory prefetcher. Thus, cache coherent switch on chip 902A may be configured to determine data to be prefetched (e.g., based on the AI/ML prefetch algorithm) and provide instructions (via switch fabric/bus 944) to memory 914C to provide such prefetched data to memory 914A (via switch fabric/bus 944) for caching. Memory 914C may accordingly provide such data for buffering by memory 914A.

In certain embodiments, each upstream port of cache coherent switch on chip 902A is configured to determine whether a cache buffer port is assigned for the respective upstream port. If a cache buffer port is assigned, a further determination may be made as to which downstream port is assigned as the cache buffer port. Incoming traffic may then be accordingly provided to the assigned downstream port for cache buffer purposes.

In various embodiments, caching may be performed by memory of the switch on chip and/or memory attached to the ports of the switch on chip. Variously, cache coherent switch on chip 914A may determine whether requested data is within the cache and retrieve such data if it is present within the cache. If the data is not within the cache, a request may be provided to the coupled persistent memory for the data and the data may be accordingly provided. In certain embodiments, write requests may be provided to both the cache and the persistent memory.

FIG. 10A illustrates system 1000 that includes servers 1000A and 1000B. Each server 1000A/B includes a cache coherent switch on chip 1002, each cache coherent switch on chip 1002 communicatively/electrically coupled to CPU 1004, accelerator 1006, storage 1008, ASIC 1010, PM 1012, memory 1014, and network interface card (NIC) 1080. Each accelerator 1006 may include respective memory 1046, which may include its own cache coherent and non-cache coherent storage. Cache coherent switch on chips 1002A and 1002B may be communicatively coupled via network/bus 1044. FIG. 10A may illustrate a configuration where a cache coherent switch on chip of a first server may bridge over Ethernet to another cache coherent switch on chip of a second server.

In various embodiments, cache coherent switch on chips 1002A and 1002B may be communicatively coupled via an Ethernet connection (e.g., via network 1044). As such, cache coherent switch on chips 1002 may communicate via CXL protocol through Ethernet to allow for resource pooling and/or sharing (e.g., of memory, accelerators, and/or other devices) between different devices, server racks, and/or data centers.

In various embodiments, commands received from a host via a CXL protocol port of cache coherent switch on chips 1002 are received and terminated inside the respective cache coherent switch on chips 1002 at the CXL protocol port. Cache coherent switch on chip 1002 may then provide a corresponding command tunneled within the payload of Ethernet frames that are communicated over network 1044. Thus, cache coherent switch on chip 1002 includes a bridging function that is configured to terminate all the read and write commands (e.g., persistent memory flush commands) inside cache coherent switch on chip 1002 and provide corresponding commands over Ethernet.

NICs 1080 may be configured to allow for cache coherent switch on chips 1002s to communicate via network/bus 1044. In certain embodiments, cache coherent switch on chips 1002 may be provided for data flow between accelerators 1006 and NICs 1080 (which may be a Smart NIC) so that NICs 1080 may write directly into accelerator 1006's cache coherent memory. Such data flow allows for sending and/or receiving of cache coherent traffic over network 1044 by accelerators 1006.

The configuration of system 1000 allows for data to be communicated between components within servers 1000A and 1000B as well as between servers 1000A and 1000B without needing to be controlled by CPUs 1004. Furthermore, the components of system 1000 are decoupled from each other, with traffic controlled by respective cache coherent switch on chips 1002.

In a certain embodiments, system 1000 may be configured so that cache coherent traffic stays within respective servers 1000A and 1000B. Cache coherency within each server 1000A/B is resolved by respective CPU 1004. Cache coherent switch on chips 1002 may provide accelerator traffic over network 1044, but in certain such embodiments, such accelerator traffic may be non-cache coherent traffic. The cache coherent traffic is thus never exposed to network 1044.

In certain embodiments, (e.g., with processing core 474 within a cache coherent switch on chip, as described in FIG. 4), cache coherent switch on chips 1002 may be configured to resolve cache coherent traffic among accelerators 1006, as well as resolve cache coherency within CPU 1004. Thus, for example, cache coherent switch on chips 1002 may resolve symmetric coherency between two processing domains based on CXL protocol. The processing core within cache coherent switch on chip may receive and provide cache coherent traffic between the various components of system 1000. Thus, for example, all cache coherent traffic may first be provided to cache coherent switch on chip 1002 and cache coherent switch on chip 1002 may then provide corresponding cache coherent traffic to respective target components. Such a configuration may reduce the cache coherency traffic between accelerators and CPUs, as well as other components within system 1000, increasing the performance of accelerator dominated ML/AI workloads by alleviating the bottleneck of CPUs.

In a further embodiment of providing/receiving cache coherent traffic to accelerator 1006 over network 1044, NIC 1080 may indicate that it is providing cache coherent traffic to accelerator 1006. Upon receipt of such traffic, accelerator 1006 may provide the bias change of the coherent memory line to CPU 1004 (via cache coherent switch on chip 1002). Upon receipt, CPU 1004 may then provide snoop requests to all components within its respective server 1000A/B, and once the cache line is resolved, CPU 1004 provides a line resolved message to the requesting accelerator 1006. Upon receipt of this message, accelerator 1006 may write the received traffic from NIC 1080 into the cache coherent portion of the respective memory 1046 of accelerator 1006.

Typically, accelerator to accelerator traffic within a system is provided via a proprietary switch. Cache coherent switch on chip 1002 allows for the elimination of such a proprietary switch while providing for accelerator to accelerator traffic. Accordingly, CXL protocol data may be provided from a first accelerator 1006 to a cache coherent switch on chip 1002 and communicated to a second accelerator 1006.

In various embodiments, CPU 1004 may include a home agent configured to resolve coherent traffic. Cache coherent traffic may be resolved by the home agent of CPU 1004. However, cache coherency may also be resolved within a processing core (e.g., a processing core such as processing core 474 of cache coherent switch on chip) of the cache coherent switch on chip, removing CPU 1004 as a bottleneck.

In a typical system, when data arrives from a network, typical data flows include network to processor, processor to storage, storage to processor, and processor to accelerator. As the volume of data grows, the processor becomes a bottleneck in this type of circular cycle of data transfer.

Cache coherent switch on chip 1002 allows for data to flow through to its ultimate destination while bypassing any CPU bottleneck. Thus, cache coherent switch on chip 1002 allows for data transfer between various ports, such as between two downstream ports. Components that are coupled to cache coherent switch on chip 1002 may, accordingly, more easily transfer data between each other and bypass CPU bottlenecks. Such transfers may be of the CXL protocol format.

For data transfers between accelerators and storage devices allocated to a root port of a microprocessor of cache coherent switch on chip 1002, the transfers may be cache coherent (e.g., controlled by the microprocessor), removing the need for cache coherency to be resolved by CPU 1004. Such a configuration provides for bandwidth and latency advantages as CPU 1004 may be bypassed and may be especially beneficial for neural networks, cryptocurrency, and/or other such systems where accelerators, ASICs, and/or other devices are primarily used (e.g., during training or mining).

Each cache coherent switch on chip 1002 may be communicatively/electrically coupled with one or more of a plurality of accelerators 1006. As each cache coherent switch on chip 1002 may be communicatively/electrically coupled to one or more other cache coherent switch on chip 1002, the number of accelerators available to each of the communicatively/electrically coupled cache coherent switch on chips 1002 may be accordingly expanded across a network to encompass accelerators that are coupled to the plurality of cache coherent switch on chips 1002. Variously, cache coherent switch on chip 1002 may provide for such pooling regardless of whether the respective accelerator is assigned to CPU 1004 or a microprocessor of the cache coherent switch on chip 1002 (allowing for operation of the accelerator via cache coherent switch on chip 1002).

Thus, cache coherent switch on chip 1002 allows for creating and managing a pool of CXL protocol attached accelerators or other resources distributed across one or more cache coherent switch on chips 1002. In various embodiments, each cluster of communicatively coupled cache coherent switch on chips 1002 may include their own respective virtual hierarchies and cluster of resources. Resources within each cluster may communicate between each other accordingly as if all are connected to the same switch.

Resources within the pool (such as accelerators) may be allocated/deallocated to any application server inside a rack, aisle, data center, and/or any portion of networked data centers communicatively coupled via CXL protocol (including via CXL protocol over Ethernet or other networks). Applications servers may thus be provided with direct access to all accelerators within a cluster, removing all data transformations that are required in typical architecture (e.g., from CUDA code to RDMA protocol packets and back).

In certain embodiments, traffic passing through a first cache coherent switch on chip may be mirrored on a second cache coherent switch on chip. The mirrored traffic may then be utilized for, for example, analysis of traffic that is provided through the first cache coherent switch on chip.

FIG. 10B illustrates formats of read packet 2000A, read response packet 2000B, write packet 2000C, and write acknowledgment packet 2000D. Such packets may be used for providing resource pooling (e.g., via bridging) and persistent memory functions over Ethernet. Variously, each of packets 2000 may include preamble 2002, DA 2004, SA 2006, type 2008, command 2010, address 2012, and CRC 2016. Read packet 2000A and write acknowledgement packet 2000D may include PAD 2014. Read response packet 2000B may include read data 2018 and write packet 2000C may include write data 2020. The size of each portion of data may be indicated within FIG. 10B.

For read packet 2000A, command 2010 may include a command indicating “PM read” with length data of the packet and the intended address. For read response packet 2000B, command 2010 may indicate “PM response” with the intended address and the read data. CRC 2016 may indicate the full Ethernet frame. Address 2012 may correspond to the persistent memory's address.

For write packet 2000C, command 2010 may indicate a “PM write” with length data of the packet, the intended address, and the write data. For write acknowledgement packet 2000D, command 2010 may indicate a “PM write acknowledgement” and the intended address.

In various embodiments, compression and/or decompression may be utilized and, based on the packets, the same compression and/or decompression algorithm may be utilized for both the read initiator and the target. Compressed data may be inflated at the source and written within cache.

FIG. 11 illustrates a block diagram of an example cache coherent switch on chip with accelerator, in accordance with some embodiments. FIG. 11 illustrates system 1100 that includes cache coherent switch on chip 1102, CPU 1104 with memory 1114B, and NIC 1180. Cache coherent switch on chip 1102 includes fabric 1148 and compression and security module (CSM) 1150. CSM 1150 allows for cache coherent switch on chip 1102 to perform compression and decompression for data received.

In certain embodiments, after data arrives within cache coherent switch on chip 1102 from the network (e.g., via NIC 1180), the data is provided to CSM 1150 to be decrypted and/or decompressed. Once the data is decrypted and/or decompressed, such data is then provided to other components through one or more ports of cache coherent switch on chip 1102. Additionally, when data is provided to cache coherent switch on chip 1102 to be provided to the network via NIC 1180, CSM 1150 may first encrypt and/or compress such data before memory buffering and/or providing such data to NIC 1180 (and, thus, the network).

FIGS. 12-14 illustrate block diagrams of further examples, in accordance with some embodiments. FIG. 12 illustrates system 1200 that includes a plurality of servers 1242. Server 1242A and 1242B are communicatively coupled via switch 1252 (e.g., cache coherent switch on chips 1202A and 1202B of servers 1242A and 1242B, respectively, are communicatively coupled via switch 1252). In various embodiments, multiple such servers may be communicatively coupled via fabric switch. Coupling in such a manner may allow for such communicatively coupled servers (e.g., servers 1242A and 1242B) to pool resources such as CXL protocol or CPU socket attached memory, accelerators, and/or other such resources over fabric, increasing the amount of resources available to a system and increasing flexibility. In various embodiments, such resources may be pooled via software controlled, driver, or driver-less techniques.

In a certain instance, server 1242B may wish to share one or more of memories 1214F-J with server 1242A. A driver running within server 1242B may pin such memory through a registration routine and may provide an access key to server 1242A for access to the respective memory. Server 1242A may then access the shared memory via CXL protocol memory commands.

In another instance, server 1242B may share read/write caches with server 1242A. When recalling cached data, server 1242A may first check if the data is available locally. If the data is not available locally, a request for cached data is provided to server 1242B. Server 1242B may then provide the requested cached data either from a cache within memories 1214F-I of server 1242B or from memory 1214J communicatively coupled to microprocessor 1204B.

In other embodiments, two or more servers may be a part of the system. A local server may determine that requested data is not within its own buffer and may then communicate requests for the buffer data to each of the various servers. The various servers may provide erasure code, accordingly to the techniques described herein (e.g., within FIG. 17). The servers receiving the request may each determine whether its own caches include the requested data. Servers that include the data may then provide read responses to the requesting server and the requesting server may then receive erasure code data and replace the missing data blocks. Servers that do not include the data may provide read requests to corresponding memory, update the corresponding caches, and provide the data blocks to the requesting server. The requesting server may then reconstruct such data.

FIGS. 13A and 13B illustrate system 1300 that includes downstream bridges for supporting legacy devices. Systems 1300 of FIGS. 13A and 13B include cache coherent switch on chip 1302 and bridge 1354. Bridge 1354 may be, for example, a single root to multi root (SR2MR) or single logical device to multiple logical device (SLD2MLD) bridge. Bridge 1354 may be configured to expose a single device (e.g., device 1328) to multiple host ports.

In the embodiment of FIG. 13A, bridge 1354A may be a SR2MR bridge. In various embodiments, port 1316 may be communicatively coupled to bridge 1354A via a PCI protocol. Bridge 1354A may be accordingly communicatively coupled to device 1328 via the PCI protocol. Bridge 1354A may be implemented within cache coherent switch on chip 1302 or as a separate chip.

Bridge 1354A may include a plurality of virtual function assignments 1396A-C. Port 1316 may be coupled to device 1328 via bridge 1354A. Port 1316 may include a plurality of P2P bridges 1386A-D. Each virtual function 1396 may be associated with a corresponding P2P bridge 1386. Each virtual function 1396 may include address remap logic. In certain embodiments, port 1316 may implement physical function assignment logic to control processor 1398. Due to the matched virtual functions 1396 of bridge 1354A to P2P bridges 1386 of port 1316, device 1328 may be associated with a plurality of roots (e.g., multi-roots). The configuration of system 1300A may be utilized for single root devices and may provide for the implementation of multi-root devices while providing the security and isolation of separate virtual hierarchies.

In the embodiment of FIG. 13B, bridge 1354B may be a SLD2MLD bridge. Bridge 1354B may be implemented within cache coherent switch on chip 1302 or as a separate chip. Bridge 1354B may be communicatively coupled to PPB 1338 and, accordingly, vPPB s 1336. Bridge 1354B may provide a plurality of address remaps 1356A/B as well as provide for assignment logic such as for interrupts and resets with 1356C. Thus, single logic device 1328 coupled to bridge 1354B may be virtualized into a multi-logic device. A single logic device 1328 may be accordingly associated with a plurality of vPPBs 1336 and available as a resource and/or utilize resources from a plurality of other devices communicatively coupled to cache coherent switch on chip 1302. Utilizing the configuration of system 1300B, a single logic device may be shared and become, effectively, a multi-logic device and obtain the security and isolation benefits of a multi-logic device with a plurality of virtual hierarchies.

FIG. 14 illustrates system 1400 with cache coherent switch on chip 1402 with fabric 1448 of cache coherent switch on chip 1402 coupled to chiplets 1464. In certain embodiments, chiplet 1464 may be a memory controller chiplet that increases the efficiency and reduces the latency of memory. In other embodiments, chiplets 1464 may be other types of chiplets, such as AI inference engines, FPGAs, GPU accelerators, edge computing devices, and/or other such devices.

FIG. 15 illustrates a block diagram of an example computing system with a cache coherent switch on chip, in accordance with some embodiments. FIG. 15 illustrates system 1500 that includes cache coherent switch on chip 1502. Cache coherent switch on chip 1502 may be communicatively coupled to a resource pool. The resource pool may include a plurality of CPUs 1504A-N, devices 1528, accelerator 1506, memory 1514, storage 1508, processor 1504, and ASIC 1510. Such communicative coupling may be via a CXL protocol. Such resource pools may be within a server, within a data center, and/or communicatively coupled via Ethernet, the Internet, and/or another data connection (e.g., Bluetooth or satellite Internet).

As described herein, cache coherent switch on chip 1502 may be configured to assign one or more resources from the resource pool to applications on demand. When the application no longer requires the assigned resources, the resources may be reallocated available for other applications.

FIG. 16 illustrates a block diagram of a networked system, in accordance with some embodiments. Networked system 1600 may include a plurality of server racks 1666. Server racks 1666A and 1666B may be communicatively coupled via Ethernet 1668A and server racks 1666C and 1666D may be communicatively coupled via Ethernet 1668B. Ethernet 1668A and 1668B may be communicatively coupled via Internet 1670. Accordingly, server racks 1666A-D may all be communicatively coupled with each other.

Each of server racks 1666A-D may include their respective cache coherent switch on chips. Resource clusters may be created from devices communicatively coupled to the respective cache coherent switch on chips within a server rack (e.g., within one of server racks 1666A to D), from devices communicatively coupled via Ethernet 1668, from devices communicatively coupled via Internet 1670, and/or communicatively coupled via another technique. Accordingly, the cache coherent switch on chip disclosed herein allows for the creation of any resource cluster within a system, within a server rack, and across the server racks, creating completely fungible resources connected via a high speed CXL network or CXL protocol over fabric.

FIG. 17 illustrates a block diagram of an example cache coherent switch on chip with erasure code accelerator, in accordance with some embodiments. FIG. 17 illustrates cache coherent switch on chip 1702 with ports 1720/1722, fabric 1776, erasure code accelerator 1782, and processor 1726.

Erasure code accelerator 1782 may provide redundancy for data stored in persistent memory, non-volatile memory, random access memory, and/or other such memory communicatively coupled to cache coherent switch on chip 1702 or across a network that cache coherent switch on chip 1702 is communicatively coupled to with other cache coherent switch on chips.

Thus, erasure code accelerator 1782 may be communicatively coupled to processor 1726 and/or to memory or storage communicatively coupled to ports 1720/1722. In situations where erasure code accelerator 1782 is communicatively coupled to processor 1726, erasure code accelerator 1782 may perform read/write requests addressed to processor 1726. Erasure code accelerator 1782 thus stripes data across one or more non-volatile memory on writes and reconstructs data from such memory during reads. In the event of a non-volatile memory failure, erasure code accelerator 1782 may support reconstruction of any lost data.

In certain embodiments, cache coherent switch on chip 1702 may receive a write data flow. For a write data flow received by cache coherent switch on chip 1702, a check may be performed to determine whether the write data is assigned a virtual end point (e.g., a memory or I/O device) in a virtual hierarchy. If the write is for the virtual end point, fabric 1776 may provide the data to processor 1726. Processor 1726 may then provide the write request to erasure code accelerator 1782, identifying the port associated with the request and the erasure code technique for use. Data may then read from various CXL protocol ports of cache coherent switch on chip 1702, allowing for erasure coding to be accordingly performed by erasure code accelerator 1782 by modifying the data and recalculating the erasure coded data. The modified erasure coded data is then written to the respective CXL port (e.g., the ports where the data is read from the various CXL protocol ports). Such a technique may conserve processing resources by offloading erasure coding to erasure code accelerator 1782.

Erasure code accelerator 1782 may also provide a read data flow. In a certain embodiment, ingress logic (e.g., for a read request from a port of cache coherent switch on chip 1702) determines whether the read data flow has erasure code implemented. If erasure code has been implemented, the read request may be provided to processor 1726. Processor 1726 may then provide the read request to erasure code accelerator 1782. The read request may identify the port (and, thus, the device communicatively coupled to the port) where the read request was received. The requested read data may then read from various CXL protocol ports of cache coherent switch on chip 1702, allowing for erasure coding to be accordingly performed by erasure code accelerator 1782 to prepare new erasure coded data. The erasure coded data is then provided back to the respective requesting CXL port.

The various accelerators of cache coherent switch on chip 1702 (e.g., compression, security, erasure coding, and/or other such accelerators) and processor 1726 of cache coherent switch on chip 1702 may be utilized for provisioning of computational storage services (CSSes) to applications running on host CPUs (e.g., CPUs of the greater system containing cache coherent switch on chip 1702). For example, processor 1726 and CSM modules may serve as computational storage processors (CSPs) to provide CSSes to attached hosts. Processor 1726 may also be utilized as the host in computational storage use cases, orchestrating data movement and running of CSSes. In certain embodiments, processor 1726 may offload batch processing of CSS commands from the host CPUs.

In various embodiments, FIGS. 18-20 may illustrate features of a network-on-chip (NOC, a.k.a. switch on chip) system. FIG. 18 illustrates a block diagram of an example network-on-chip, in accordance with some embodiments. FIG. 18 illustrates system 1800 that includes ports 1802A-P, cross points 1804A-I, accelerators 1806, and fabric ports 1808. In various embodiments, the various portions of system 1800 may be communicatively coupled via communication interfaces 1812, which may be any communication interface as described herein. Though the configuration of system 1800 is that of a 3×3 cross point with two ports communicatively coupled to each cross point, other embodiments may include any other configuration of cross points and ports. Accelerators 1806 and fabric ports 1808 may be such components on a switch on chip, as described herein. Other embodiments may include other components on system 1800.

System 1800, including ports 1802A-P and cross points 1804A-I, form a switch mesh network. System 1800's topology includes ports 1802 disposed on the perimeter of the die of the switch mesh network of system 1800. Cross points 1804 may form the nodes of a switch mesh network and be communicatively coupled to certain portions of communication interfaces 1812 (e.g., referred to as “cross links”). Communication interfaces 1812 may be, for example, 512@1 gigahertz interfaces. As communication interfaces 1812 may be bidirectional, such interfaces may actually be 2×512@1 gigahertz interfaces. The configuration of system 1800 allows for a plurality of links between points (e.g., ports and/or cross points 1804) of system 1800.

In various embodiments, the instantaneous bandwidth of system 1800 may be determined by summing the bandwidth of all the links between cross points 1804. System 1800 includes cross-wise communication interfaces 1812C and 1812D as well as 1812A and 1812B between cross points 1804G and 1804C and between 1804E and 1804A (the cross points on the edge corners of the grid array), respectively. Such cross-wise communication interfaces 1812 reduces the average hop count (the average number of cross points 1804 that a signal communicated within system 1800 would encounter from an input port to an output port) from a normal grid array for signals in communication within system 1800.

Cross-wise communication interfaces 1812 allows for effective overspeed for certain signals communicated within system 1800. In certain embodiments, such overspeed may be close to 2× overspeed. Furthermore, cross-wise communication interfaces 1812 reduces potential for path conflicts. In certain such embodiments, the longest path between cross points is that of two hops, as long as all cross-links are able to receive and send signals. In certain embodiments, the following rules may be applied in order so that the longest path between cross points (e.g., for signals communicated within the mesh network) is that of two hops: 1) If the destination is on the same cross point, the path has zero hops, 2) If the destination is on an adjacent cross point, the path is a single hop directly to that cross point, 3) A two hop path between array corner cross points in the same row, follows the row, 4) A two hop path between array corner cross points in the same column, follows the column, 5) A two hop path from a corner cross point to a node adjacent to the opposite corner cross point takes its first hop to the opposite corner cross point, 6) A two hop path from a node at the center of its row or column to a destination homed at the center of its row or column goes to the array center for its first hop, and 7) A two hop path from a cross point at the center of its row or column to a destination homed at an array corner cross point goes to the array diagonally opposite its destination's cross point for its first hop.

FIG. 19 illustrates a block diagram of an example cross point architecture, in accordance with some embodiments. FIG. 19 illustrates the architecture of cross point 1900. Cross point 1900 includes ports 1902A-D, 1904A-D, 1906A-D, and 1908A-D, and array 1910. Variously, ports 1902A-D, 1904A-D, 1906A-D, and 1908A-D may be input ports, output ports, or include both input and output ports. In the embodiment illustrated in FIG. 19, cross point 1900 may be a corner cross point (e.g., located on the corner of an array) and may include 8 input ports and 8 output ports. For example, two of ports 1902A-D, 1904A-D, 1906A-D, and 1908A-D may be input ports and two of ports 1902A-D, 1904A-D, 1906A-D, and 1908A-D may be output ports.

Other configurations of input and output ports (e.g., location and/or quantity) may be utilized for various cross points, including cross points that are not corner cross points. Thus, for example, row center and column center cross points (e.g., cross points 1804B, 1804D, 1804F, and 1804H of FIG. 18) may include two additional ports on their bottom edge to mate with the array center cross point (e.g., cross point 18041) that has four ports in each direction on each edge.

Output ports may include one or more first in, first out (FIFO) queues to provide control of data outputted by the output ports (e.g., for traffic control of signals communicated). Such FIFO queues may be any appropriate size for stage flow control, such as a depth of 2 to 64, inclusive (e.g., may contain 2 to 64 signals and/or flits). In various embodiments, certain output ports may include different sized FIFO queues, depending on expected traffic. For example, FIFO queues at the station outputs (e.g., output ports that are communicatively coupled to one or more ports 1802) may be increased in size to absorb transient output port connections.

In various embodiments, input and output ports may be implemented in pairs, with an odd lane and an even lane. Signals may be allocated to either the odd lane or the even lane throughout communications within the system (e.g., within the fabric). Thus, paired inputs may communicate corresponding odd and even signals. Paired outputs may communicate corresponding odd and even signals as well. An even input may only feed an even output, and vice versa. In various embodiments, loopback connections (not shown) may be present from input to output to support communications between two bifurcated ports in the same station.

In various embodiments, a multiplexer and arbiter may be present at each cross point output. The arbiter round robins among the FIFO queues based on several criteria. The head of the FIFO queue (e.g., the oldest data entry within a queue) that wins an arbitration event is selected and clocked into an output register. The head of a FIFO queue is eligible to compete in an arbitration round if there is sufficient room in the output FIFO queues of the link partner cross point (e.g., if the associated credit register is greater than 0). If any of the FIFO queues are more than half full (e.g., more than half of their capacity is used), the arbiter rounds among those FIFO queues that are more than half full and pass the credit test (e.g., that the destination buffer has free data space). Otherwise, it rounds among those FIFO queues that are not empty and whose heads pass the credit test. If a FIFO queue's head fails a credit test, the FIFO queue is blocked until a credit grant is received.

FIG. 20 illustrates a block diagram of a system, in accordance with some embodiments. FIG. 20 illustrates system 2000 that includes cross points 2002 and 2004. Cross point 2002 includes input port 2006 and output ports 2008. Cross point 2004 includes input port 2012 and output ports 2014B. System 2000 may utilize hop by hop flow control to regulate traffic flow between cross points 2002 and 2004 (e.g., regulating traffic flow from one cross point to another cross point).

In certain embodiments, a credit system may be utilized for communication of signals and/or flits. Each cross point output port may include a set of credit registers—one credit register for each output of the corresponding linked cross point (e.g., for output port 2008B, one each for output ports 2014A-C) and the FIFO queue for the input-output port pair of the cross point (e.g., the input-output pair of input port 2006B and output port 2008B, per the prior example). The initial credit in each register is equal to the queue depth of the FIFO queue. When a flit or a signal is provided to a cross point, the input port may receive the flit and accordingly forward the flit to the appropriate output port for storage within the FIFO queue of the output port before the flit and/or signal is provided to the next cross point (e.g., output ports