330, input voltage range, transfer functions of the ADCs in the target hardware 330, weight and output noise model parameters, parameters relating to sources of non-linear errors (e.g., line resistances), process variation constants for circuit elements in target hardware 330, and other properties of the target hardware 330. In one embodiment, the known information being determined by processor 310 can be dependent on an approach being used by target hardware 330 to compute and/or approximate output currents.

Processor 310 can set or define an initial maximum input range of values (e.g., −α_max, +α_max) within which define the range of input data values to be quantized by DACs for input to the AIMC tile modeling a layer in a neural network model of the target hardware 330. In one embodiment, the processor first obtains an input distribution of the actual floating point values for input to initial layer of a trained neural network modeled at the target hardware 330 to define the maximum input range of values (e.g., between −α_max, +α_max).

Processor 310 can obtain (e.g., retrieve from memory 312) the initial maximum input range of values (e.g., −α_max, +α_max) and perform methods to modify or tune the maximum input range of values (e.g., −α_max, +α_max) that maximizes a signal to noise (SNR) ratio when performing model operations, e.g., MVM operations for inference or classification tasks. Processor 310 can use target hardware properties 318, the conductance values encoding parameters w_ijto program a circuit model 332 in one embodiment. The circuit model 332 can be a virtual model or representation of the target hardware 330 that simulates operations, parameters, and properties of target hardware 330. Circuit model 332 can compute or estimate the output current distributions of the ADCs given the input word-line voltages. In one embodiment, circuit model 332 can be realized using a behavioral-based or physics-based model, such as a SPICE model (e.g., a text-description of an AIMC that can be used by a SPICE Simulator to mathematically predict the behavior of the AIMC under varying conditions). Processor 310 can also be configured to adjust these simulated parameters and properties of circuit model 332. In an embodiment, processor 310 can program circuit model 332 to have specifications in virtual space that may be identical or substantially identical to specifications of target hardware 330 in physical space. By way of example, programming circuit model 332 using the conductance values to encode weights w_ijin circuit model 332 can allow processor 310 to simulate operations of target hardware 330.

In order to run the circuit model 332, the processor 310 can provide inputs to the circuit model 332 in order for the circuit model 332 to perform operations, such as MAC, MVM or VMM operations or like linear algebraic operation, to generate an output values 334. By way of example, the target hardware 330 can be an AIMC 10, shown in FIG. 1, and the circuit model 332 can simulate the AIMC 10. Processor 310 can determine inputs to be applied on the word lines WL₀, . . . , WL_M, and feed the determined inputs to circuit model 332. Processor 310 can be configured to determine the input to be provided to circuit model 332. Memory 312 can store a ML model dataset 320 that includes at least a training set used in training of trained ML model 314, a validation set used in an accuracy evaluation during training of trained ML model 314, or a test set used in a performance evaluation (post training) of the trained ML model 314.

In one embodiment, processor 310 can determine the input to be provided to circuit model 332 by sampling subsets of at least one of either the training set, the validation set, or the test set. For example, processor 310 may sample subsets from the training set. As another example, processor 310 may sample subsets from the validation set. As yet another example, processor 310 may sample subsets from the test set. Yet as another example, processor 310 may sample subsets from two or more of those sets, e.g., two or more combinations of the training set, the validation set and the test set. In accordance with an embodiment, processor 310 can select a specific distribution (e.g., statistical distribution) from a plurality of candidate distributions, such as input distributions 316 stored in memory 312. The selected distribution can be a distribution that best fits the training set that was used for training of trained ML model 314. Processor 310 can input the selected distribution as an input to circuit model 332. In embodiments, the input distributions are floating point number values within a max input range that are to be mapped as quantized signals, e.g., int8 bit values ranging between −128 to 127) for processing by the AIMC tile. Typically, an input distribution of floating point numbers define a Gaussian distribution of input data values (e.g., within a range between values −α (max), +α (max)) that are input to the neural network model and/or input to intermediate (i.e., hidden) layers of the neural network model. By clipping (cutting off) outlier values, the input distribution range (between −α (max), +α (max)) becomes restricted (the input range distribution is tighter), thereby increasing the signal to noise (SNR) ratio of the outputs. This input range tightening is advantageous as there are more larger inputs values leading to higher signal strengths which improves quality of MVM operations. The system of FIG. 3 automatically determines the optimal boundary values (−α (max), +α (max)) to restrict the floating point input range so that the mapping of an input floating point value to the quantized signal improves the quality of the MVM operation performed.

Processor 310 can select the first AIMC tile(s) that belong to the first layer of the model for processing and successively process data inputs at each additional AIMC tile(s) that belong to the first layer of the model. For each neural network model layer, at each AIMC tile(s) that receive(s) input data for processing through the network, the input data at a layer is recorded so that processor 310 determines the distribution of the inputs that are to be processed by the tiles at each layer. The processor 310 runs the circuit model 332 with the collected inputs, and the circuit model 332 can output current values 334 that simulate currents being outputted from bit lines BL₀, . . . , BL_Nof the AIMC 10.

In an embodiment, processor 310 can then analyze the input distribution 316 to determine an initial DAC (input) range setting of each AIMC tile such that the average SNR for all columns (with respect to the output noise) is approximately equal to a predetermined value.

In this way, processor 310 of system 300 can determine at least one property of output values 336, such as the mean, median, mode, standard deviation, entropy, skew, minimum and maximum values, an k-th quartile, or other properties. Processor 310 can adaptively update the maximum input range of values (e.g., between −α_max, +α_max) in circuit model 332 based on the determined properties of output values 336 and optimize the trade-off between the signal strength generated by aggressive clipping of the DAC input ranges to increase the tightness of the input range distribution (e.g., reduce boundary values −α_max, +α_max) and the amount of information lost. Generally, the tighter (lower in value) the input ranges used for clipping are, the stronger the signal. However, too tight an input range leads to a clipping of many input signals that are larger than the input range which causes loss of information.

In one embodiment, the processor 310 can perform optimization techniques to adjust the input range floating point values that are quantized in the circuit model 332 until optimized data input ranges 338 are determined for each column, or different groups of columns, or unit cells in target hardware 330. Processor 310 can adjust the input data range of floating point values of circuit model 332, i.e., adjust (−α (max), +α (max)) values, to cause circuit model 332 to generate updated output current values 334. Processor 310 can repeat the adjustment of the maximum input range of floating point values in circuit model 332 and update the input distribution 316 until a stopping criterion, such as when an output distribution satisfies a defined objective, e.g., optimizes the SNR of the target hardware 330. The input range of floating point values in circuit model 332 that causes the output distribution to satisfy a defined objective can be set by processor 310 as an optimized input range of floating point values 338. In one embodiment, the stopping criterion can also be an upper bound, on a maximum number of iterations or adjustments made to the adjust the input data range of floating point values in circuit model 332.

In an embodiment, the target hardware 330 can be calibrated by determining the optimized input data range of floating point values 338 that the DAC devices at the wordline inputs (WL₀, WL₁, . . . , WL_M) of target hardware 330 can receive. Target hardware 330 can be calibrated prior to being deployed to various applications for performing inference on new input data.

FIGS. 4A-4B depict a flow chart illustrating a process 400 for automatically optimizing the input range floating point values that are quantized for processing in AIMC tiles in one embodiment. Process 400 can be performed by system 300 shown in FIG. 3. In one embodiment, process 400 can be performed to automatically adapt DAC range optimization for improved SNR in each individual AIMC architectures/system. Process 400 can begin at a block 402. At block 402, a processor (e.g., processor 310) can initialize the neural network model layer index i, where i=1, . . . , N, and initialize AIMC tile index k, where tile k=1, . . . , K in an analog memory device. For example, the first AIMC tile is selected that belongs to the first layer of the model. Process 400 can proceed from block 402 to block 405. At block 405, the processor can program desired network weights to the AIMC tile. The network weights for the particular neural network model are known and correspond to different conductance values for the unit cells in a particular row/column in the tile shown in FIG. 1. The network weight values are thus programmed for each individual column of unit cells, or for groups or regions of columns of unit cells in the AIMC tile. Then, at block 408, the processor can determine whether the last model layer has been processed, i.e., whether i>N. If it is determined that the last layer has been processed, the process ends. However, if it is determined that the last layer has not been processed, the process continues to 412 where the processor can determine an input distribution (of floating point values) to the input line DACs of the first AIMC tile that receives input data at the current layer.

The next steps determine the input distribution to the current AIMC tile(s) that receive(s) input data by collecting inputs that are processed by the current layer. In an embodiment, the inputs sampled from a representative input distribution can be used or, alternatively, inputs from a training or validation set can be used. In a non-limiting embodiment, dummy data input values are input through the model, and for each layer of the neural network model, values that are input into every layer of the neural network model are recorded. The recorded data values are used to adjust the floating point values (input ranges) which are an optimal bounds used for clipping inputs for each tile of that the model layer.

Then, at 415, the processor can optionally determine and fix the ADC (output) range of each AIMC tile. The processor is programmed to ultimately determine the optimal input range such that the average SNR for all columns (with respect to the output noise) is approximately equal to a predefined value. Continuing to 418, the processor can select the first AIMC tile for the current layer and at 423 determines whether this selected tile is the last tile for this neural network layer, i.e., whether tile k>K. If it is determined that the selected tile is the last tile, the process proceeds to step 430 where the first AIMC tile is again selected and the process proceeds to step 450, FIG. 4B.

Returning to step 423, if it is determined that the selected tile is not the last tile, i.e., k≤K, the process proceeds to 425 where the processor sets the initial input range such that the average SNR for all columns (w/r/t/ the output noise) is approximately equal to a predetermined value. Then, continuing to 428, the processor then selects the next tile by incrementing k, k=k+1 and selecting the next tile k and the process returns to step 423 to further determine if the current tile is the last tile of that network model layer and if the next tile is not the last tile, the process proceeds back to 425 to set the initial input range such that the average SNR for all columns (w/r/t/ the output noise) is approximately equal to a predetermined value. Then the next tile is selected at 428 by incrementing k, k=k+1 and the next tile k is selected. The processor repeats steps 423, 425 and 428 to ensure that initial input range is such that the average SNR for all columns (w/r/t/ the output noise) is approximately equal to a predetermined value. Thus, the method determines and fixes the ADC (output) range of each AIMC tile.

In an alternate embodiment, step 428 is optional. That is, in an alternate embodiment, when the processing of a layer is completed, the input distribution for that layer is obtained for the next layer and the process then moves to the tiles of the next layer. The first input range can just be the maximum value of the input distribution collected for that layer. Thus, step 428 can alternatively entail choosing initial input range according to a methods such as taking the abs(max( )) of the input distribution or a quantile of the input distribution (in order to remove outliers).

Returning to 430, after repeating the steps 423, 425, 428 to ensure that initial input range is such that the average SNR for all columns (w/r/t/ the output noise) is approximately equal to a predetermined value, the method selects the first AIMC tile (e.g., sets k=1) and proceeds to 450, FIG. 4B. At step 450, FIG. 4B. a determination is made as to whether this current selected tile is being processed is the last tile for this neural network layer, i.e., whether tile k>K. If it is determined that the selected tile is not the last tile, the process proceeds to step 454 where the processor can determine the end-to-end system accuracy. That is, the neural network model is run with the current input range and an added residual noise level and the processor determines an accuracy of the output value of the model. In an embodiment, the added residual noise level is added to emulate what the target hardware is doing, i.e., knowing amount of noise the hardware introduces so as to simulate the effect of noise to optimize performance metric for the given hardware. That is, the residual noise value matches a noise characteristic between hardware and simulation- and can be a random gaussian vector added to the output of the MVM. In an embodiment, the output noise model can be a function that, given the inputs and weights of the network, as well as other hardware specific parameters, generates a noise vector that is added to the output of the MVM. A simple example function is one that generates a Gaussian vector, e.g., with a mean of 0 and std=0.1 as long as the output noise model mimics the behavior of the system.

Then, the network can perform an MVM operation using the data within current input range and determine an output accuracy or like error metric of the performed MVM operation. Generally, an MVM operation performed on a hardware neural network model is according to:

Y=M*x

where M is matrix and x is an input vector of floating point (e.g., FP32) values within the current distribution range. However, according to an embodiment herein, the simulated MVM operation in the method becomes:

$Y^{'} = (M + η_w) * x_int8 + η_out;$

where η_w is a modeled matrix weight noise error, x_int8 is a quantized form of the input data and η_out is the introduced residual noise error. The quantization conversion to int_8 is dependent upon the hardware, and can be based on 9 bit (Int9) or 10-bit (int10) quantization levels. The output l2 error can be determined using an l2 function to provide a difference metric between Y and Y′. This can be determined using a downstream task, e.g., performing an image classification task, using the simulated AIMC tiles. Thus, for a given initial maximum data input value range, the downstream task will classify one or more images and compute the number of images that are classified correctly to obtain a score. In an embodiment, there is passed through a plurality of images in the AMIC device emulating the neural network classification model. The method responsively computes the image classification and determines how many images are classified correctly to obtain an accuracy score. The process is iteratively repeated with modified i.e., tightened, maximum data input value range at each iteration. The tightened maximum data input value range is then chosen that gives the highest accuracy score or achieves the best performance metric, i.e., maximizes accuracy. Other tasks/metrics can be performed for use in evaluating and determining the optimal input range that gives the best performance score (accuracy).

In an embodiment, a determination of the end-to-end system accuracy at 454 can be based upon whether the “l2” _error is minimized or whether some performance metric has been maximized/minimized. In an embodiment, the accuracy can be determined by the measured SNR at the model output. The process continues to 460 where the processor can reduce the DAC input range of the selected AIMC tile, i.e., further clip (tighten) the floating point number maxima and minima (−α_max, α_max) input to the model layer. Then continuing to 465, the processor again determines the end-to-end system accuracy as in step 454, e.g., by measuring the error metric, output value accuracy, or measured output SNR. In an embodiment, the determining of the performance metric is based on an obtained model output value and includes comparing the obtained simulated initial model output against a model output value obtained from running the same neural network model operation on the corresponding hardware-based neural network model (target hardware 330) being simulated. The determined model output value error measure is based on the comparing, such that the best corresponding performance metric corresponds to a minimized error measure. In an embodiment, the input values makes use of the end-to-end system accuracy measures to determine whether the additional error introduced to the network by clipping inputs exceeds the accuracy gained by increasing the dynamic resolution of the input. For example, there exists a trade-off as more the input signals are clipped, the system SNR is increased; however information is lost about the input (due to the clipping of some input values).

Upon determining an end-to-end system accuracy at the reduced input range at 465, FIG. 4B, the method proceeds to 470 where a determination is made as to whether the end-to-end system accuracy has improved, i.e., whether the output value accuracy of the MVM operation or SNR level has increased or whether the error of the MVM operation output has decreased. If at 470, it is determined that the end-to-end system accuracy has improved, the process proceeds to 475, FIG. 4B where the processor can again reduce the DAC input range of the selected AIMC tile, i.e., further clip (tighten) the floating point number maxima and minima (−α_max, α_max) input to the current model layer. The process then returns back to 465 to again determine the end-to-end system accuracy at the reduced input range and, at 470, determine whether the end-to-end system accuracy has improved as a result. In an embodiment, the end-to-end system accuracy is a simulated accuracy using a model of the output noise. The steps 465, 470, 475 are repeated until such time that the end-to-end system accuracy does not improve, or whether a fixed maximum number of iteration have been performed. By repeating steps 465, 470, 475 the processor reduces the DAC input range of the selected AIMC tile, i.e., further clip (tighten) the floating point number maxima and minima (−α_max, α_max) input to the model layer, until there is no appreciable end-to-end system accuracy improvement. At such time it is determined that a reduced input range does not appreciably improve end-to-end system accuracy, the process returns to 480, FIG. 4B to fix the previous input bound. Then, the process proceeds to 485 where the processor selects the next tile, i.e., select next tile by incrementing k (i.e., k=k+1). After incrementing the next tile index k, the process returns to 450 where the processing of FIG. 4B is repeated with the input values for the next tile. Then the method steps of FIG. 4B are repeated for the next tile in the current layer in order to adapt the DAC input range of the next selected AIMC tile, i.e., further clip (tighten) the floating point number maxima and minima (−α_max, +α_max) input to the next tile at the current model layer. As a result of repeating steps 450 to 485 of FIG. 4B, the system determines the optimum DAC input range (floating point values to be quantized by the DACs) of each of the AIMC tiles of the current neural network model layer. Upon a determining at 450 that all AIMC tiles have been processed at the current neural network layer, i.e., that tile k>K, the process proceeds to 490 where the processor can select the next layer of the neural network model to be processed, e.g., by first incrementing the model layer index i (i.e., i=i+1), and then proceeding back to step 408, FIG. 4A. Then, the processor can process the input range of values for the next model layer and set the initial range of input values for each tile of the next layer such that the average SNR for all columns (w/r/t/ the output noise) for each tile is approximately equal to a predetermined value.

In an embodiment, the method 400 of FIGS. 4A, 4B are repeated for each tile(s) at each neural network model layer, with the end result being a vector of α_max(and corresponding −α_max) values for each layer of AIMC tile(s). The system thus automatically provides an optimized “clipping” of the input range of values, i.e., a boundary +α_max(and corresponding −α_max) value that ensures a maximum accuracy, i.e., improved SNR in the AIMC architecture.

In an embodiment, the method of FIGS. 4A-4B can be performed as part of a calibration phase and applied using the actual hardware. For example, a cell phone product can have a computing AIMC integrated circuit, e.g., for performing image recognition/classification, and prior to shipping the product, the AIMC IC can be calibrated. In the calibration, the method 400 of FIGS. 4A, 4B is performed for the AIMC to obtain the optimal input ranges (−α_max, α_max). The determined input range values −α_max, +α_maxfor the AIMC chip can be programmed or otherwise stored in registers at the AIMC IC. The input range values are quantized by the DACs that convert the floating point values x into x_int8 values and can correspond to voltage pulse lengths applied as input rows of the crossbar.

In an embodiment, in view of FIG. 4A, at step 412, the selecting the first AIMC tile(s) that belong to the first layer of the model and determining the input distribution to the current AIMC tile(s) of the current layer that receive(s) input data is repeated for each layer, i.e., the method re-samples the inputs that come into one layer, after calibrating the previous layers' input range. Alternatively, the input distribution range for current AIMC tiles of a layer can be selected first, and then only the inputs coming into each layer is sampled once. That is, in an alternative embodiment, using recorded inputs into every tile, the input ranges can be pre-determined using a method for calculating the dynamic ranges of activations which can be done with an exponential moving average, or by buffering the activations and then computing the abs(max)) on the activations or a quantile on the activations. Then, starting at the tiles of the first layer, the method iteratively tightens the input ranges for these tiles until there is no improvement observed with respect to downstream performance under the output noise model that mimics the behavior of the system. Then, without re-sampling the inputs for the next layer, the process is repeated for the next layer.

FIG. 5 illustrates an example computer or processing system or environment that may implement a system for adaptively optimizing digital-to-analog converter (DAC) input range values for improved signal-to-noise ratio in analog in-memory computing system architectures in one embodiment. As shown in FIG. 5, computing environment 100 can include an example of an environment for the execution of at least some of the computer code involved in performing the inventive methods described herein, such as the adaptive optimization of the input range values algorithm code 500. In addition to block 500, computing environment 100 includes, for example, computer 101, wide area network (WAN) 102, end user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this embodiment, computer 101 includes processor set 110 (including processing circuitry 120 and cache 121), communication fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and block 200, as identified above), peripheral device set 114 (including user interface (UI), device set 123, storage 124, and Internet of Things (IoT) sensor set 125), and network module 115. Remote server 104 includes remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, host physical machine set 142, virtual machine set 143, and container set 144.

COMPUTER 101 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer, mainframe computer, quantum computer or any other form of computer or mobile device now known or to be developed in the future that is capable of running a program, accessing a network or querying a database, such as remote database 130. As is well understood in the art of computer technology, and depending upon the technology, performance of a computer-implemented method may be distributed among multiple computers and/or between multiple locations. On the other hand, in this presentation of computing environment 100, detailed discussion is focused on a single computer, specifically computer 101, to keep the presentation as simple as possible. Computer 101 may be located in a cloud, even though it is not shown in a cloud in FIG. 1. On the other hand, computer 101 is not required to be in a cloud except to any extent as may be affirmatively indicated.

PROCESSOR SET 110 includes one, or more, computer processors of any type now known or to be developed in the future. Processing circuitry 120 may be distributed over multiple packages, for example, multiple, coordinated integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and/or multiple processor cores. Cache 121 is memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set 110. Cache memories are typically organized into multiple levels depending upon relative proximity to the processing circuitry. Alternatively, some, or all, of the cache for the processor set may be located “off chip.” In some computing environments, processor set 110 may be designed for working with qubits and performing quantum computing.

Computer readable program instructions are typically loaded onto computer 101 to cause a series of operational steps to be performed by processor set 110 of computer 101 and thereby effect a computer-implemented method, such that the instructions thus executed will instantiate the methods specified in flowcharts and/or narrative descriptions of computer-implemented methods included in this document (collectively referred to as “the inventive methods”). These computer readable program instructions are stored in various types of computer readable storage media, such as cache