In embodiments, controller 230 may determine which circuit to select and under which circumstances, including when to switch back to a previously used circuit, e.g., a general purpose processor. In embodiments, circuit selection may be accomplished by using a logic element, such as an XOR circuit.

FIG. 3 is a flowchart of an illustrative process for partitioning computational resources to conserve power according to various embodiments of the present disclosure. In embodiments, process 300 may begin, at step 302, when network information is received by a controller that may be coupled to any number of compute units. Each compute unit may comprise a number of CNN processors that each, in turn, may comprise any number of sub-circuits, which may be grouped together into one or more groups. In embodiments, sub-circuits may represent one or more input channels at any given time.

At step 304, controller may use the network information to decide which compute units to select such as to reduce power consumption. For example, in embodiments, the controller may select fewer than all compute units to perform certain computing operations. It is understood that this is not intended as a limitation, since the controller may select all available compute units, under certain circumstances.

At step 306, the controller may use the network information to configure one or more of the selected processors, the groups of sub-circuits, or the sub-circuits to process input channel data and/or parameters associated with a neural network layer, e.g., to obtain intermediate results. In embodiments, the controller may use power consumption, distance, or other suitable metrics to decide where to store or cache the intermediate results.

At step 308, the controller may locally store or cache intermediate results near the previously selected compute units, e.g., to reduce a length of a data path to or from the selected compute units.

Finally, at step 310, the controller may cause less than all of the processors, the groups of sub-circuits, or the sub-circuits to accumulate at least some of the intermediate results that may be used, e.g., to generate output channel data.

Single-Bit Computation

It is noted that while various embodiments may use multi-bit weight parameters and/or multi-bit input data to maintain high accuracy, especially for computationally relatively easily feasible software implementations, hardware embodiments herein may equally benefit from single-bit processing embodiments.

Although embodiments herein are given mainly in the context of single-bit weight parameters and/or single-bit input data, this is not intended as a limitation on the scope of the present disclosure. In fact, in embodiments, equal or even better results may be achieved by using variable data and/or variable network parameter sizes across a given machine learning process, for example, by allowing configurability on a per-layer basis, as discussed in greater detail below.

It is noted that single-bit computation differs significantly from multi-bit computation. In embodiments, circuitry may be dynamically selected based on a per-layer data and/or weight parameter configuration associated with a neural network. This further reduces power consumption and increases speed or throughput.

FIG. 4 is a block diagram of an exemplary circuit for dynamically selecting an optimized circuit according to various embodiments of the present disclosure. As depicted in FIG. 4, circuit 400 may comprise controller 402, CNN processor 404, multiplexers 410-412, and demultiplexers 414. In embodiments, controller 402 may be coupled to receive configuration parameters 406 may also be inputs to multiplexer 410. Exemplary configuration parameters may be single-bit and/or multi-bit configuration parameters and have varying size. Multiplexer 410 may be coupled to any number of CNN processors 404 that receive input data 403 via multiplexer 412, e.g., single-bit and/or multi-bit variable data that may be pre-selected by controller 402.

In embodiments, CNN processor 404 may be implemented as a single hardware accelerator architecture, or CNN processor 404 may comprise two or more circuits through which data may be routed. For example, one circuit may be a dedicated or standalone circuit that may be optimized for a specific task or to accommodate a particular performance metric, such as a power consumption, an output accuracy, a processing speed or throughput, or a processing capability (e.g., fixed size inputs). Another circuit may be a common hardware accelerator circuit.

In operation, controller 402 may receive configuration parameters 406, e.g., from a storage register and use them, together with information about input data 403, e.g., to determine whether input data 403 comprises single-bit data or multi-bit data and to determine a number of performance metrics, which may comprise calculated, measured, or estimated/expected values.

In embodiments, based on the performance metrics, controller 402 may determine and select a circuit configuration of CNN processors 404 from a number of possible configurations. Further, controller 402 may dynamically select configuration parameters 406 such as to prevent certain configuration parameters 406 from being processed by CNN processors 404. Similarly, controller 402 may actively suppress certain types of input data from being used or processed by CNN processors 404, e.g., on a per-layer basis, to reduce unnecessary data movements and power consumption.

For example, for portions of a neural network that have a certain bit width, controller 402 may dynamically select a low-power logic implementation, e.g., a circuit comprising combinational logic that carries less computational overhead, to perform certain operations, such as arithmetic computations, more efficiently, faster, and using less power when compared to comparable general-purpose implementations that employ general-purpose computing hardware, or even when compared to a full adder circuit or multiplier circuit.

Further, in embodiments, controller 402 may apply any number of selected configuration or network parameters 407, e.g., network parameters associated with an output from a network layer of a neural network, to a selected circuit configuration to process a subsequent layer of the network according to the selected configuration parameters 407 and/or selected input data 408. For example, to reduce a computational burden on CNN processors 404 and to increase throughput, controller 402 may select single-bit weight parameters over multi-bit weight parameters, e.g., at the expense of a slightly reduced accuracy. In embodiments, to accomplish this, prior to determining when to use a special purpose circuit, controller 402 may first determine whether a particular neural network layer processes single-bit data or single-bit weight parameters, e.g., by identifying data or weight parameter sizes/widths associated with a particular network layer.

In other words, controller 402 may determine when to use a special purpose circuit on a layer-by-layer basis. In embodiments, controller 402 may obtain layer specific information, e.g., from configuration information. It is noted that, even if the size of an input layer may be unknown, since hidden layers in a neural network are generated, advantageously, their size can be known.

It is understood that the output of CNN processors 404 may be routed through demultiplexer 414 to produce output channel data 409. It is further understood that circuit 400 illustrated in FIG. 4 is not limited to the constructional detail shown there or described in the accompanying text. Those skilled in the art will appreciate that a suitable circuit may comprise different or additional elements and/or connections, e.g., comparators, storage devices, and so on.

FIG. 5 is a flowchart of an illustrative process for increasing computational efficiency according to various embodiments of the present disclosure. In one or more embodiments, process 500 for increasing computational efficiency may begin, at step 502, when a controller receives a set of configuration parameters that also may comprise single-bit and/or multi-bit configuration parameters. The controller may further receive input data that may comprise single-bit and multi-bit input data.

At step 504, based on the input data, the controller may determine a number of performance metrics, such as a power consumption, an accuracy, or a throughput, for a circuit configuration that uses single-bit configuration parameters and for another circuit configuration that uses multi-bit configuration parameters

At step 506, the controller may use the performance metrics to determine and/or select one or more suitable circuit configurations for processing a neural network, e.g., to satisfy one or more of the performance metrics.

At step 508, the controller may apply any number of single-bit or multi-bit configuration parameters to a selected circuit according to a selected circuit configuration to process the input data using the neural network to generate an output, e.g., an intermediate result that may be used in a subsequent layer of the neural network. It is understood that the same circuit may be used for more than one circuit configuration. Alternatively, two or more circuits, each having a different circuit configuration may be used.

Memory Self-Test and Initialization

In conventional general-purpose hardware accelerator implementations, operations such as testing, initializing, and clearing a large number of parameters from the accelerator's memory invariably consume a great amount of time and power. The resulting inefficiencies are in large part caused by the fact that such operations are typically performed in software. Depending on the size of an accelerator's memory, performing these operations comes at a cost that may be a significant portion of the overall computational cost.

To reduce time and energy cost for time and energy-intensive operations, various embodiments herein take advantage of a Built-In Self-Test (BIST) circuit. BISTs are readily available as a hardware function in existing memory cores and commonly used during a memory manufacturing or production test phase. Conventionally, a BIST circuit, in a self-test phase, cycles a sequence of known test patterns through cells in a memory array to verify proper circuit operation of the memory device, wherein each pattern tests a different interaction between adjacent cells to detect potential device failures. A dedicated BIST circuit typically empties or overwrites the contents of a set of registers to a predetermined value or state, e.g., by setting contents to zero or initializing the contents with a predetermined pattern.

In contrast, embodiments herein utilize aspects of a BIST circuit and a BIST routine in the context of reducing power consumption in machine learning accelerators. In embodiments, a BIST circuit's testing routine, ordinarily used to clear or initialize registers and memory in a hardware accelerator or other computing device, is modified and controlled to terminate at a stage when the content of a set of registers has been zeroed to create a known state for the accelerator. In this manner, e.g., during a startup and/or wakeup phase of the accelerator, no additional movements or transitions are required, advantageously saving time and energy. This is particularly useful when the capacity of an accelerator is well utilized, where many memory move operations would otherwise be required.

FIG. 6 illustrates a block diagram of an exemplary system for reducing power consumption in machine learning accelerators according to various embodiments of the present disclosure. System 600 comprises master controller 602, BIST circuit 604 (e.g., one or more BIST controllers or one or more BIST circuits), hardware accelerator 606, and memory device 608. In operation, master controller 602 may configure and control both BIST circuit 604 and hardware accelerator 606, which may read from memory device 608. Memory device 608 may be any memory device known in the art that can receive a test pattern, for example, at a set of dedicated pins.

It is understood that circuit 600 illustrated in FIG. 6 may comprise different or additional elements and/or connections. For example, master controller 602 may be used to initialize memory device 608, e.g., if memory device 608 is a single-port memory that may require configuration. Also, BIST circuit 604 may be coupled to a read/write logic block, which may be coupled to memory device 608. In embodiments, instead of BIST circuit 604 instructing memory device 608 directly, BIST circuit 604 may use the read/write logic block to trigger memory device 608 to perform a series of reads and writes and confirm the success or failure thereof. It is understood that BIST circuit may be configured to ignore a certain number of erroneous read/write values per access.

In embodiments, master controller 602 may communicate a command to BIST circuit 604 to instruct BIST circuit 604 to initiate a BIST routine that causes some or all of memory device 608 to assume a predetermined state. For example, the content of registers in memory device 608 may be set to zero to create a known state for hardware accelerator 606. Advantageously, this reduces power consumption by reducing the number of movements or transitions. It is noted that selecting zeroes as the predetermined values has the additional advantage that for some memory technologies, reading a “zero” from memory and/or moving it out of memory device 608 consumes less power than reading and/or moving a “one.” Further, utilizing the zeroing feature of BIST circuit 604, advantageously, reduces hardware cost since no additional circuitry needs to be designed, implemented, or used, e.g., merely for the purpose of preparing hardware accelerator 606 to commence processing a CNN or other neural network. In embodiments, once memory device 608 has reached the predetermined state, BIST circuit 604 may terminate the BIST routine, e.g., by halting the cycling of known test patterns through memory device 608.

In various embodiments, master controller 602 may signal to BIST circuit 604 to run a test that may be used to determine whether a number of the bit cells in memory device 608 are in a properly working condition. In embodiments, test results may be used to determine a safety margin, e.g., based on a temperature, a frequency, and/or a voltage. In embodiments, master controller 602 or BIST circuit 604 may then determine a suitable frequency and/or voltage to adjust the margin, as needed. For example, if the margin is deemed too low, to reduce the risk of a device failure in memory device