210a-210d, host 230, and interconnect selectors 241-242. System 201 also includes interconnects 251-253, 261-265. Interconnect 261 operatively couples the output of interconnect selector 241 to device stack0 210a. Interconnect 263 operatively couples interface 234 to a first input of interconnect selector 241. A second input of interconnect selector 241 is operatively coupled to a first output of interconnect selector 242. A second output of interconnect selector 242 is operatively coupled to interface 235. Stack3 210d is operatively coupled to the input of interconnect selector 242. Thus, it should be understood that when interconnect selectors 241-242 are configured to interconnect device stack3 210d with interface 235 and to interface 234 with device stack 0 210a, the configuration illustrated in FIG. 2A is achieved.

When interconnect selectors 241-242 are configured to interconnect device stack3 210d with device stack 0 210a, a ring topology that does not include host 230 is achieved. This configuration is illustrated in FIG. 2C by not showing inactive interconnections 263-264.

FIGS. 3A-3C are diagrams illustrating scalable stacked device system interconnection configurations. In FIGS. 3A-3C, system 300 comprises device stacks 310a-310h, and host 330. System 300 also includes interconnect selectors 341-346. Host 330 includes interfaces 334-337. Device stacks 310a-310h may each include a plurality of memory device die (e.g., DRAMs) and one or more processing device dies. These processing device dies and/or host 330, may be, or include, an application specific integrated circuit (ASIC) device, a graphics processor unit (GPU), a central processing unit (CPU), a system-on-chip (SoC), or an integrated circuit device that includes many circuit blocks such as ones selected from graphics cores, processor cores, and MPEG encoder/decoders, etc.

Interconnect selector 341 is operatively coupled with interface 334, interconnect selector 342, and stack0 310a. Interconnect selector 342 is operatively coupled with interface 335, interconnect selector 341, and stack3 310d. Stack0 310a is operatively coupled with stack1 310b. Interconnect selector 343 is operatively coupled with interconnect selector 344, interconnect selector 345, and stack1 310b. Interconnect selector 344 is operatively coupled with interconnect selector 343, interconnect selector 346, and stack2 310c. Stack2 310c is operatively coupled with stack3 310d. Interconnect selector 345 is operatively coupled with interconnect selector 343, interface 336, interconnect selector 346, and stack4 310e. Stack4 310e is operatively coupled to stack5 310f. Stack5 310f is operatively coupled to stack6 310g. Stack6 310g is operatively coupled to stack7 310h. Interconnect selector 346 is operatively coupled with interconnect selector 344, interface 337, interconnect selector 345, and stack7 310h.

Using interconnect selectors 341-346, system 300 may be configured in a single-ring topology. System 300 may be further configured to include or exclude host 330 from this ring topology. This topology is illustrated in FIG. 3B by not showing inactive interconnections. System 300 may also be configured in a multiple-ring topology. System 300 may be further configured to either include or exclude host 330 from one or more of the multiple rings. This multiple ring topology is illustrated in FIG. 3C by not showing inactive interconnections.

FIG. 4 is a flowchart illustrating a method of operating a stacked device system. One or more steps illustrated in FIG. 4 may be performed by, for example, system 100, system 200, system 201, system 300, and/or their components. By a first device stack comprising a first set of stacked memory devices and a first set of one or more processing devices electrically coupled to, and stacked with, the first set of stacked memory devices, a first copy of a first neural network model is received (402). For example, host 230 may transmit via interconnect 254 and to device stack0 210a, programming and/or neural network model parameters (e.g., topology, interconnect weights, etc.) that can be used by device stack0 210a to implement a neural network model. Host 230 may also transmit via interconnect 254 a first partial training dataset to be used by device stack0 210a to train its copy of the neural network model.

By a second device stack comprising a second set of stacked memory devices and a second set of one or more processing devices electrically coupled to, and stacked with, the second set of stacked memory devices, a second copy of the first neural network model is received (404). For example, under instructions from host 230, device stack0 210a may relay, via interconnect 251 and to device stack1 210b, the programming and/or neural network model parameters (e.g., topology, interconnect weights, etc.) received from host 230 that can be used by device stack1 210b to implement the neural network model. Host 230 may also transmit via device stack0 210a a second partial training dataset to be used by device stack1 210b to train the neural network model.

The first copy of the first neural network model is trained using a first portion of first neural network model training data to generate first updated neural network model parameters (406). For example, device stack0 210a, using the first partial training dataset, train its copy of the neural network model to generate a first set of updated neural network model parameters. The second copy of the first neural network model is trained using a second portion of first neural network model training data (408). For example, device stack1 210b, using the second partial training dataset, train its copy of the neural network model to generate a second set of updated neural network model parameters.

The first updated neural network model parameters are communicated to the second device stack using a handshake protocol (410). For example, using a handshake protocol to set up and facilitate communication, the first set of updated neural network model parameters may be communicated from device stack0 210a to device stack 1210b.

The second updated neural network model parameters are communicated to the first device stack using the handshake protocol (412). For example, using the handshake protocol to set up and facilitate communication between device stacks 210b-210d and (optionally) host 230, the second set of updated neural network model parameters may be communicated from device stack1 210b to device stack0 210a via a ring topology that includes device stacks 210a-210d and (optionally) host 230.

FIG. 5 is a flowchart illustrating a method of reconfiguring a stacked device system. One or more steps illustrated in FIG. 5 may be performed by, for example, system 100, system 201, system 300, and/or their components. The interconnect used by a plurality of device stacks is configured in a first ring topology that includes a host device, where the device stacks include at least one processing device (502). For example, the interconnect 251-253, 261-264 of system 201 may be configured in a first ring topology that includes host 230 by configuring interconnect selector 242 to route communication between device stack3 210d and interface 235 and configuring interconnect selector 241 to route communication between interface 234 and communication device stack 210a.

The interconnect used by the plurality of device stacks is configured in a second ring topology that does not include the host device (504). For example, the interconnect 251-253, 261-265 of system 201 may be configured in a first ring topology that does not include host 230 by configuring interconnect selector 242 to route communication between device stack3 210d and interconnect selector 241 and configuring interconnect selector 241 to route communication between interconnect selector 242 and communication device stack 210a.

FIG. 6 is a flowchart illustrating a method of configuring a stacked device system to partition processing. One or more steps illustrated in FIG. 6 may be performed by, for example, system 100, system 300, and/or their components. The interconnect used by a plurality of device stacks is configured into at least two ring topologies where the interconnect used by the plurality of device stacks, and where the device stacks include at least one processing device (602). For example, the interconnect of system 300 may be configured to concurrently have both a first ring topology and a second ring topology.

The first ring topology may be made by: (a) configuring interconnect selector 343 to route communication between device stack1 310b and interconnect selector 344; and, (b) configuring interconnect selector 344 to route communication between interconnect selector 343 and device stack2 310c; and (c) either: (1) configuring interconnect selector 341 to route communication between interface 334 and device stack0 310a and configuring interconnect selector 342 to route communication between device stack3 310d and interface 335; or, (2) configuring interconnect selector 341 to route communication between interconnect selector 342 and device stack0 310a and configuring interconnect selector 342 to route communication between device stack3 310d and interconnect selector 341.

The second ring topology may be made by either: (1) configuring interconnect selector 345 to route communication between interface 336 and device stack4 310e and configuring interconnect selector 346 to route communication between device stack7 310h and interface 337; or, (2) configuring interconnect selector 345 to route communication between interconnect selector 346 and device stack4 310e and configuring interconnect selector 346 to route communication between device stack7 310h and interconnect selector 345.

The interconnect used by the plurality of device stacks is configured into a single ring topology (604). For example, the interconnect of system 300 may be configured into a single ring topology. The single ring topology may be made by: (a) configuring interconnect selector 343 to route communication between device stack1 310b and interconnect selector 345; and, (b) configuring interconnect selector 346 to route communication between device stack7 310h and interconnect selector 344; and (c) either: (1) configuring interconnect selector 341 to route communication between interface 334 and device stack0 310a and configuring interconnect selector 342 to route communication between device stack3 310d and interface 335; or, (2) configuring interconnect selector 341 to route communication between interconnect selector 342 and device stack0 310a and configuring interconnect selector 342 to route communication between device stack3 310d and interconnect selector 341.

FIGS. 7A-7B are a flowchart illustrating methods of training neural network models with a stacked device system. One or more steps illustrated in FIG. 7 may be performed by, for example, system 100, system 200, system 201, system 300, and/or their components. The interconnect used by a plurality of device stacks is configured into a first ring topology, the device stacks each including at least one processing device (702). For example, the interconnect of system 300 may be configured into a single ring topology that includes device stacks 310a-310h.

Respective copies of a first neural network model are trained using the plurality of device stacks and respective portions of a first training data set (704). For example, each device stack 310a-310h may use a unique (with respect to the other device stacks 310a-310h) set of training data (whose training data sets may overlap or be disjoint) to concurrently train their respective copies of a first neural network model. Using the first ring topology, information about the respective trained copies of the first neural network model are communicated between the plurality of device stacks (706). For example, device stacks 310a-310h may communicate their updated model parameters (e.g., weights) with the other device stacks 310a-310h by passing their updated model parameters and relaying the updated model parameters received from other device stacks 310a-310h around the single ring topology. This relaying may be part of, for example, a concurrent (parallel) training algorithm known as Ring-AllReduce.

Using the information about respective trained copies of the first neural network model reduction operations are performed using the plurality of device stacks (708). For example, device stacks 310a-310h may use their own updated model parameters and the updated model parameters from the other device stacks 310a-310h to produce new model parameters to be used in the next training iteration. This process may continue through further iterations and/or distributions of partial training data until the training of the first neural network model is complete (e.g., converges). This updating may be part of, for example, the concurrent (parallel) training algorithm known as Ring-AllReduce.

The interconnect used by the plurality of device stacks is configured into a second ring topology and a third ring topology, the second ring topology and the third ring topology including a first subset and a second subset of the plurality of device stacks, respectively (710). For example, the interconnect of system 300 may be configured to concurrently have a plurality of ring topologies. In particular, device stacks 310a-310d may be configured into a first ring topology as described and device stacks 310e-310h may be configured into a second ring topology as described herein. Respective copies of a second neural network model are trained using the first subset and respective portions of a second training data set (712). For example, each device stack 310a-310d of the first ring topology may use a unique (with respect to the other device stack 310a-310d of the first ring topology) set of training data (whose training data sets may overlap or be disjoint) to concurrently train their respective copies of a second neural network model.

Concurrent with training the second neural network model using the first subset, respective copies of a third neural network model are trained using the second subset and respective portions of a third training data set (714). For example, concurrent with the first ring topology device stacks 310a-310d training the second neural network model, each device stack 310e-310h of the second ring topology may use a unique (with respect to the other device stack 310e-310h of the second ring topology) set of training data (whose training data sets may overlap or be disjoint) to concurrently train their respective copies of a third neural network model.

Using the second ring topology, information about respective trained copies of the second neural network model is communicated between the first subset (714). For example, device stacks 310a-310d of the first ring topology may communicate their updated model parameters (e.g., weights) with the other device stacks 310a-310d of the first ring topology by passing their updated model parameters and relaying the updated model parameters received from other device stacks 310a-310d around the first ring topology. This relaying may be part of, for example, a concurrent (parallel) training algorithm known as Ring-AllReduce.

Using the information about respective trained copies of the second neural network model reduction operations are performed using the first subset (716). For example, device stacks 310a-310d of the first ring topology may use their own updated model parameters and the updated model parameters from the other device stacks 310a-310d of the first ring topology to produce new model parameters to be used in the next training iteration. This process may continue through further iterations and/or distributions of partial training data until the training of the second neural network model is complete (e.g., converges). This updating may be part of, for example, the concurrent (parallel) training algorithm known as Ring-AllReduce.

Using the third ring topology, information about respective trained copies of the third neural network model are communicated between the second subset (718). For example, device stacks 310e-310h of the second ring topology may communicate their updated model parameters (e.g., weights) with the other device stacks 310e-310h of the second ring topology by passing their updated model parameters and relaying the updated model parameters received from other device stacks 310e-310h around the second ring topology. This relaying may be part of, for example, a concurrent (parallel) training algorithm known as Ring-AllReduce.

Using the information about respective trained copies of the third neural network model, perform reduction operations using the second subset (720). For example, device stacks 310e-310h of the second ring topology may use their own updated model parameters and the updated model parameters from the other device stacks 310e-310h of the second ring topology to produce new model parameters to be used in the next training iteration. This process may continue through further iterations and/or distributions of partial training data until the training of the third neural network model is complete (e.g., converges). This updating may be part of, for example, the concurrent (parallel) training algorithm known as Ring-AllReduce.

The methods, systems and devices described above may be implemented in computer systems, or stored by computer systems. The methods described above may also be stored on a non-transitory computer readable medium. Devices, circuits, and systems described herein may be implemented using computer-aided design tools available in the art, and embodied by computer-readable files containing software descriptions of such circuits. This includes, but is not limited to one or more elements of system 100, system 200, system 201, system 300, and their components. These software descriptions may be: behavioral, register transfer, logic component, transistor, and layout geometry-level descriptions. Moreover, the software descriptions may be stored on storage media or communicated by carrier waves.

Data formats in which such descriptions may be implemented include, but are not limited to: formats supporting behavioral languages like C, formats supporting register transfer level (RTL) languages like Verilog and VHDL, formats supporting geometry description languages (such as GDSII, GDSIII, GDSIV, CIF, and MEBES), and other suitable formats and languages. Moreover, data transfers of such files on machine-readable media may be done electronically over the diverse media on the Internet or, for example, via email. Note that physical files may be implemented on machine-readable media such as: 4 mm magnetic tape, 8 mm magnetic tape, 3-½ inch floppy media, CDs, DVDs, and so on.

FIG. 8 is a block diagram illustrating one embodiment of a processing system 800 for including, processing, or generating, a representation of a circuit component 820. Processing system 800 includes one or more processors 802, a memory 804, and one or more communications devices 806. Processors 802, memory 804, and communications devices 806 communicate using any suitable type, number, and/or configuration of wired and/or wireless connections 808.

Processors 802 execute instructions of one or more processes 812 stored in a memory 804 to process and/or generate circuit component 820 responsive to user inputs 814 and parameters 816. Processes 812 may be any suitable electronic design automation (EDA) tool or portion thereof used to design, simulate, analyze, and/or verify electronic circuitry and/or generate photomasks for electronic circuitry. Representation 820 includes data that describes all or portions of system 100, system 200, system 201, system 300, and their components, as shown in the Figures.

Representation 820 may include one or more of behavioral, register transfer, logic component, transistor, and layout geometry-level descriptions. Moreover, representation 820 may be stored on storage media or communicated by carrier waves.

Data formats in which representation 820 may be implemented include, but are not limited to: formats supporting behavioral languages like C, formats supporting register transfer level (RTL) languages like Verilog and VHDL, formats supporting geometry description languages (such as GDSII, GDSIII, GDSIV, CIF, and MEBES), and other suitable formats and languages. Moreover, data transfers of such files on machine-readable media may be done electronically over the diverse media on the Internet or, for example, via email