Some examples, AMS devices could include: RF interface ICs, e.g., WLAN/mobile phone interfaces which use MIMO, and hence multiple identical receive and/or transmit modules; sensor integrated circuits (ICs) using multiple AMS devices, e.g., in an automotive context; and sensor arrays, either in a triple redundancy environment, or for measuring environment parameters at different locations, e.g., smoke detector or CO2 detector.

Moreover, the defects to be detected are very diverse in nature. The interdependencies of most of these options are very hard to put into a physical/mathematical model. This is because most of them are non-linear and dependent on factors like temperature and aging. The whole environment of defect detection is hence extremely complex. Moreover, even small ICs or small AMS modules have a very large number of possible defects. As a result, today's BIST techniques are not sufficiently able to handle this situation to satisfy the requirements of detecting early defects for functional safety. As mentioned, the problem is to find a point in the feature space for testing for a parametric deviation, with a low rate of false-positive and low false-negative results. Unfortunately, the feature space is usually hard to model, because it is non-linear and has a high number of dimensions and a large number of potential defects.

In an automotive driver assistance system (ADAS), the functional safety system needs to ensure fail-safe behavior, i.e., to stop the respective assistance function once a defect is detected. A defective radar system will hence alert the driver that if it is not available anymore; the driver can then continue driving without it. For this fail-safe behavior, it is sufficient to detect a defect as soon as it impacts the function.

In self-driving cars, however, requirements are stricter: functional safety needs to ensure fail-operational behavior, i.e., to provide a reduced, safe function once a defect is detected. This also requires anticipating slowly worsening defects. Many defects in AMS devices do not occur abruptly, but are slowly worsening—due to electromigration, gradually weaking of vias, breaks in metal wires, break-thru of oxides etc. Such gradually worsening defects in AMS devices first lead to as slight deviation of parametric values.

Usually, such parametric defects manifest themselves most clearly at certain conditions—some examples include: leakage paths in dielectric at high supply voltage; resistive shorts at low supply voltage; wire breaks or weak vias at low signal frequency; resistive opens at high signal frequency; increased cross talk due to particles at high signal amplitudes; and increased noise level at low signal amplitudes. Aging of AMS devices also can lead to defects.

Testing AMS devices for parametric deviations is often done by on-chip Built-In Self-Test (BIST). Such BIST engines can often apply different signal frequencies, amplitudes, etc. to the AMS device under test. They also allow measuring different response values—gain, DC voltages etc., as a function of the signal frequency.

There is a huge feature space, i.e., a space of different AMS control parameters, associated measured values to consider, and BIST settings. Today's AMS devices under test have a lot of AMS control parameters, such as for example, settings of bias currents, reference voltages, filter settings, gain settings, etc. Also, today's BIST modules allow the application of BIST stimulus parameters with a large number of different signal amplitudes, frequencies, and shapes. Further, today's BIST modules allow for the analysis of the AMS device output signal, and the BIST modules measure a large number of values of the AMS device, like frequency response, transient response, crosstalk, etc. The measured values of the AMS device may be compared either to fixed limits, or to the same value of other, identical modules. This is often possible where multiple identical AMS devices are present, e.g., in car radar ICs that may have 3-4 identical transmitter modules and 3-4 identical receiver modules.

There is hence a wide variety of different features that the BIST module may address (i.e., AMS control parameters, BIST stimulus parameters, BIST response parameters, and limits). Moreover, the defects to be detected are very diverse in nature. The interdependencies of most of these factors are very hard to put into a physical/mathematical model. This is also because most of them are non-linear and dependent on factors like temperature and aging. The whole environment of defect detection is hence extremely complex.

Today's BIST techniques are not sufficiently able to handle this situation to satisfy the requirements of detecting early defects for functional safety. There is hence no technique to reliably detect such marginal failures by perusing the mentioned feature space efficiently and to select the right feature combination suitable for defect detection. The problem is to find a point for testing a parametric defect with low false-positive and low false-negative results in a feature space which is hard to model. Further, the transfer function of the AMS device in the feature space is often non-linear (e.g., the gain at a certain frequency as function of a bias current is usually non-linear); and the number of potential defects is very high. Also, the feature space has a high number of dimensions. The requirement for the parametric test is to find a good solution with restricted effort instead of the best solution with high effort. Finally, to find this good solution, there is no means for supervised learning; in other words, there is a requirement for unsupervised learning.

The test system embodiments described herein relates to ICs with several identical AMS devices. The situation that needs to be detected is that in many points of the feature space, one of these modules behaves slightly different to the others—but in most of these points, it is not clear if this different behavior is simply a normal deviation due to manufacturing tolerances, or if it is due to a defect. In a smaller range of the feature space, however, this deviation is so large that it can be classified as a defect, i.e. as a deviation from normal function that may still allow normal function, but soon requires replacement of the AMS device to prevent the defect from further worsening that then may render the entire IC defective.

The fault to be detected will lead to a non-linear behavior in the feature space. This means, that the module's result substantially differs from the result of the other modules. In other words, the task is to detect if one module behaves such that it notably differs from the others. The selection of stimuli, and the decision if a module is considered marginally defective (or shall better be tested using another set of stimuli, to be better able to detect if it is defective or not) is hence key.

Each AMS device will have BIST hardware included. The BIST hardware may include stimulus generator that generates inputs to the AMS device to stimulate a response from the AMS device that may be tested to detect defects. Further, the BIST may include additional hardware or software that processes the output response to use in detecting defects. For example, a fast Fourier transform (FFT) may be performed on the output response. The BIST hardware may include a controller that helps to control the BIST. Such controller may control stimulus generator and the processing of the output response. The controller may also receive inputs from external sources to provide additional control of the BIST hardware.

The BIST runs are preferably performed during breaks in the operation of the AMS device. For example, in auto radar systems, especially in frequency modulated continuous wave (FMCM) systems, this is the case between one chirp and the next. This break may be approximately 5 to 100 ms in length. A complete self-test run of the AMS device often may take more than this time. However, the self-test may be intermittently executed during several breaks. In other systems, there may be regular breaks where the AMS device does not need to operate, or it may at least be possible to schedule breaks during which no harm to the functionality of the AMS device results.

Two test system embodiments will be described. FIG. 1 illustrates a first embodiment of a test system. The test system 100 includes M AMS devices 105 that include BIST hardware. The solution for detecting defects during the lifetime of the product is to run BIST tests on several identical AMS devices 105 and compare the results to find excessive deviations in one of the modules, caused by defects. The stimulated test outputs of the AMS devices 105 are processed by a processor 110. The processor 110 may be a dedicated hardware processor running software. In other embodiments, the processor may include specific hardware configured to carry out specific functions. The test system 100 may first perform post processing 112 on the stimulated test outputs of the AMS devices to obtain characteristic quantities: e.g., FFT analysis, ratio calculation, etc. For example, an FFT, filtering, or any other needed post processing 112 may be applied. Next, the processor 110 may remove systematic mismatch 114 such as the correction of shift and scaling errors due to variation in the physical location of the modules and related load variations. Then the processor 110 performs the defect likelihood estimation 116. The defect likelihood estimation 116 compares and combines the measurements and determines a defect likelihood level. The defect detector 120 receives the output of the defect likelihood estimation 116 and uses the output to detect defects. The defect detector 120 also provides test parameters to the AMS devices 105. The operation of the defect detector 120 will be described below.

The estimation of the defect likelihood level will now be described. The objective of this test system 100 is to identify defects. This is accomplished by capturing signals with on-chip analog-to-digital converters (ADCs) or related means. Defects may be identified for particular signals (or derived quantities) that may be compared to a constant, optionally mode dependent, threshold. If the signal does not comply to the requirement, a defect may be indicated. For signals which cannot be tested against static levels and that exist in a multitude of equivalent modules, these signals may be compared mutually. That is for a number of AMS devices 105 the statistics of the signals are compared.

The measurements of analog signals will always be affected by noise in the signals and the applied measurement method. Hence, measurements will by nature not be exactly the same. To that end, based on statistics, the level of deviation is computed or estimated; above a certain level of deviation (in sigma units) the signal is considered to be affected by a defect.

The defect likelihood estimation is inspired by the outlier rejection method as described in “Simplified Statistics for Small Numbers of Observations,” R. B. Dean and W. J. Dixon, Analytical Chemistry, Vol. 23, No. 4, April 1951 and “Statistical Treatment for Reaction of Deviant Values: Critical Values of Dixon's “Q” Parameter and Related Subrange Ratios of the 95% Confidence Level,” David B. Rorabacher, Analytical Chemistry, Vol. 63, No. 2, Jan. 15, 1991. They define formulas and criteria for rejecting samples from a (small) set of observations based on statistical treatment given a certain confidence level.

In this context, the samples are divided into two sets. One set includes measurements of modules that are considered to be defect-free. This set is called the reference set. A second set includes the measurements of a module that are tested for having a potential defect. This set is called the test set. The test set is compared against the reference set.

The measurements of the M modules are denoted by M₁. . . M_M. A defect may affect one or more modules. In Table 1, the test layout of the modules over the test and reference sets is depicted for defects that affect a single module. For M=4 this results in 4 test cases.

TABLE 1

Test-case #
Test-assumption
Test-set
Reference-set

1
Defect affecting one module
M1
M2 + M3 + M4

2
Defect affecting one module
M2
M1 + M3 + M4

3
Defect affecting one module
M3
M1 + M2 + M4

4
Defect affecting one module
M4
M1 + M2 + M3

In Table 2, the test layout is for a case where a defect affects not one, but two modules. For M=4 this results in 6 test-cases.

TABLE 2

Test-case #
Test-assumption
Test-set
Reference-set

5
Defect affecting two modules
M1 + M2
M3 + M4

6
Defect affecting two modules
M1 + M3
M2 + M4

7
Defect affecting two modules
M1 + M4
M2 + M3

8
Defect affecting two modules
M3 + M4
M1 + M2

9
Defect affecting two modules
M2 + M4
M1 + M3

10
Defect affecting two modules
M2 + M3
M1 + M4

For a stimulus parameter set, these test-cases are evaluated which yields a defect likelihood estimation for each case. The over-all defect likelihood estimation is obtained by taking the maximum over all cases. Note that a single set of measurements will allow for the estimation of all cases; each case is just a different allocation of the measurements to the test and reference sets. The maximum of the sigma value is taken, but a high sigma corresponds to a small likelihood of occurrence. A small probability of occurrence means a high probability of a defect. For example, if the system is defined for 50 deviations, a deviation of 66 means an almost certain probability of a defect (in particular in case the deviation is persistent).

The estimation of the defect likelihood is based on comparing the statistics (the first and second moments) of the signals (or derived quantities from the signal) assuming that the measurements of these signals are Gaussian distributed. The statistics are calculated from a number of measurements, denoted by N. In this embodiment it is assumed that N=5. Though testing a simulation, it has been determined that N=5 provides the ability to detect defects. Other values of N may be used as well, but there is a tradeoff between N and how much time the testing takes. So keeping N as low as possible reduces the impact of doing defect testing on the operation of the system. Hence, N may be set to be less than or equal to 5, 10, 15, 20, or 25. The tradeoff for increasing N is greater ability to detect defects, but at the expense of testing time. This increased testing them then means that it will take longer to find a defect.

Generalized ratio expressions for the first (mean) and second (variance) moments will now be described. The following two generalized expressions are proposed to derive the likelihood of a defect being present based on the measurements in the test set and the reference set:

r
mean

=

mean
(
test_set
)

-

mean
(
ref_set
)

std
⁡
(
ref_set
)

;
and

r
var

=

std
⁡
(

test
set

)

std
⁡
(

ref
set

)

.

The ratio expression for r_meancan yield negative and positive values and has an expected value of 0. Very negative or positive ratio values are unlikely and may indicate a defect or severe parameter shift. In the next section, a mapping function is described that relates any ratio value r_meanto the likelihood of occurrence under normal conditions based upon sigma values.

The ratio expression for r_varcan yield only positive values. Ratio values close to 0 or very large ratio values are unlikely and may indicate the presence of a defect or severe parameter shift.

The mapping function from the r-values to an s-value (sigma value) will now be described. The distributions of the r_meanand r_varratio values need to be known for mapping such ratio values to the defect likelihood of occurrence as indicated by sigma values. The probability distributions of the ratio-test values may be estimated or characterized by means of, e.g., Monte-Carlo simulation, production testing or statistical modeling. Given the distributions of the r_meanand r_varratio values, the likelihood of occurrence of each ratio-test result is defined by the cumulative distribution function (cdf). Using the cdf, the r-values can be mapped to s-values that represent the corresponding deviation in sigma units for a normal distribution. For this mapping function

s
=

func_r2s
⁢

(
r
)

it holds that

cdf
⁡
(
r
)

=

normcdf
⁡
(
s
)

,

where cdf(r) is the estimated cumulative distribution function of the ratio values r_meanand r_varrespectively, and normcdf(s) is the normal cumulative distribution function of the standard normal distribution, evaluated at the values in r and s respectively.

The distributions of r_meanand r_var, and hence the r-to-s mapping functions, depend on the actual signal-under-test (its statistical properties) and the test-case size (the sizes of the test-set and the size of reference set).

To that end, the distributions are estimated by means of e.g., Monte-Carlo simulation, production testing, or statistical modeling. Once the estimated distributions of r_meanand r_varare obtained, a mathematical function s=func_r2s(r) is fitted such that for the mapped values s holds:

cdf
⁡
(
r
)

=

normcdf
⁡
(
s
)

.

Table 3 shows the relation between s and normcdf(s) based upon a normal distribution:

TABLE 3

s
normcdf(s)*100%

−4
0.003%

−3
0.14%

−2
2.3%

−1
15.9%

0
50%

1
84.1

2
97.7

3
99.87

4
99.997

FIGS. 2A and 2B illustrate the mapping functions from r_meanand r_var, respectively, to s for nt=5 and nr=15 assuming a signal with a standard-normal distribution. In FIG. 2A the plot 202 illustrates the mapping between r_meanand s_mean. In FIG. 2B the plot 204 illustrates the mapping between r_varand s_var.

If a device or product is designed for a 5 (or 6) sigma variation, detection of a s-value larger then 5 (or 6) means that the product is operating out-of-spec which might indicate the presence of a defect or severe parameter-shift. To combine the r_meanand r_varlikelihoods, the maximum of the correspondingly absolute values of the s-values shall be taken: S=max(abs(s_mean), abs(s_var)). This may then be compared to a threshold value in an alike test that indicates if there is a defect present.

To show the robustness of the defect likelihood estimation function(s), the defect likelihood for 100,000 experimental cases in 4 scenarios as listed in Table 4 has been evaluated. FIGS. 2C-2F illustrate plots of the probability density function for the alike test for each of the experimental cases listed in Table 3. For each of these cases, 5 values for the test set and 15 values for the reference set were computed; these values were created using a random function that follows a gaussian distribution; mean and standard deviation as input to the random number function were chosen according to the scenario mentioned in the respective line in Table 3. Such a value could, for example, be the gain of an amplifier, or a bias voltage of an ADC, DAC, PLL VCO, or the like.

TABLE 4

Test-set
Reference-set

(nt = 5)
(nr = 15)

Scenario
Mean
Std-dev
mean
Std-dev
Note

1
10.0
0.1
10.0
0.1
Reference

2
11.5
0.1
10.0
0.1
Deviating Mean

3
10.0
0.4
10.0
0.1
Deviating Std-dev

4
11.5
0.4
10.0
0.1
Deviating Mean &

Std-dev

The first scenario is for reference purposes where both the test and reference set have equal statistics. When setting the threshold for sigma to, for example, 4, there are nearly no false alerts, i.e., in nearly no case would the defect-less reference module be flagged as defective. Falsely flagged cases may be caught by re-testing (possibly with a higher number of samples) as described below. FIG. 2C illustrates a plot of the probability density 205 of the alike test for the first scenario.

The second scenario has a test set with a deviating mean that is supposed to be due to a defect. FIG. 2D illustrates a plot of the probability density 210 of the alike test for the second scenario. As can be seen, the probability density 210 has shifted significantly upward and is centered around a little more than 8 sigma.

The third scenario has a test set with an increased standard deviation that is supposed to be due to a defect. FIG. 2E illustrates a plot of the probability density 215 of the alike test for the third scenario. As can be seen, the probability density 215 has shifted significantly upward and is centered around a little more than 6 sigma.

The fourth scenario has a test set with both a deviating mean and an increased standard deviation that is supposed to be due to a defect. FIG. 2F illustrates a plot of the probability density 220 of the alike test for the second scenario. As can be seen, the probability density 220 has shifted significantly upward and is centered around a little more than 8 sigma.

With the above-mentioned threshold of sigma=4, nearly all cases of a defective module would be detected. This illustrates that based upon such modelling, threshold values may be set to achieve a specified probability of detection while limiting the false alarm rate.

FIG. 3 illustrates a defect detection method carried out by the test system. The defect detection method 300 selects of stimuli of the AMS devices 105 using the Nelder-Mead algorithm based upon pairwise module test using the alike test for comparing small number of measurements. The Nelder-Mead approach is an approach for finding the maximum in a large, non-linear multi-dimensional feature space and is often applied to nonlinear optimization problems for which derivatives may not be known. Herein, the maximum means the maximum deviation of the measurements of the reference module vs. the module under test.

The alike test approach allows comparing different modules when only a small number of measurements of the same parameter are present, e.g., only 5 measurements of the gain, under a certain stimulus parameter setting. In case of, e.g., 100 measurements, it would be easy to calculate the statistics both of the reference module(s), and the module under test. In reality, however, 100 measurements would take too long time, given the many points in the feature space. The alike test is applied in a unique way and with a unique ratio and mapping for obtaining the likelihood of the presence of a defect for comparing to a critical level.

The defect detection method 300 begins with the generation of a directed random set of stimulus parameters P 305. Directed random means that some of the parameters are enumerated while others are chosen randomly. This may be specified with engineering input or other information regarding the source and likelihood of detects in the AMS devices 105. For example, if the AMS device under test is known to be susceptible to certain types of defects, for example at high frequencies, then the random parameter generation may give more weight to making high frequency measurements. In other embodiments this selection may be completely random over the parameter space.

Next, the defect detection method 300, for this stimulus parameter set P, executes the BIST operation N times 310 to obtain sufficient measurements for pairwise comparison of the M modules and calculating estimates of the defect likelihood as described above. The calculated defect likelihood estimates r_meanand r_varmay be mapped to s_meanand s_varas described above to determine an estimate of overall defect likelihood using the alike test, where the estimate is denoted as S_overall, which is the maximum of the defect probabilities of all module pairs.

Next, the defect detection method 300 determines if stimulus parameters are approaching a potential defect 315. This may be done by having a second threshold S_approachthat indicates that the current stimulus parameters may be near stimulus parameters that would indicate a defect. The approach threshold S_approachis less than a default threshold S_ththat will be used later to identify a defect. If the defect detection method 300 is not approaching a defect, then the defect detection method 300 returns to step 305 and randomly generates a new set of stimulus parameters P and then continues.

If the defect detection method 300 is approaching a defect, then the defect detection method 300 continues to step 320 executes an initial step of the Nelder-Mead algorithm. The initial step of the Nelder-Mead algorithm will generate updated stimulus parameters in order to find parameters that may indicate a defect. Then the detection method 300 repeats the N measurements using the updated parameters from the Nelder-Mead algorithm and estimates the defect likelihood and calculates S_overallagain.

Then the defect detection method 300 compares S_overallwith a threshold value S_th330. If S_overallis greater than S_th, then a defect has been found. When a defect is found, the defect may be reconfirmed 335 by running the test again (i.e., steps 325 and 333). In this defect confirmation 335, the number of measurements may include N tests, but alternatively, during the defect conformation 335 more than N measurements may be taken and used. Then the defect detection method 300 reports a defect 340. This report may include information regarding the specific AMS devices 105 that demonstrates a defect. In an alternative embodiment, the defect confirmation 335 may be omitted.

If a defect is not found at step 330 or the defect is not confirmed at step 335, the defect detection method 300 may again determine if a defect is being approached 345 by comparing S_overallthe approach threshold S_approach. If the defect detection method 300 is not approaching a defect, then the defect detection method 300 returns to step 304 and randomly generates a new set of stimulus parameters P and then continues. In another embodiment, a count of the number of iterations through steps 350, 325, 330, and 345 may be kept and compared to a threshold counter value at step 345 in addition to checking against S_overall. If the number of iterations exceed the threshold counter value then the defect detection method 300 continues to step 305. If the number of iterations does not exceed the threshold counter value the defect detection method continues to step 350. This provides a way to limit the amount of time spent searching for parameters that uncover a defect condition.

If the defect detection method 300 is approaching a defect, then the defect detection method 300 continues to step 350 where an iteration step of the Nelder-Mead algorithm is performed to update the stimulus parameters to continue searching for parameter set that finds a possible defect in the AMS devices 105. This new parameter set is chosen by the search algorithm such that the new parameter set leads to a sharper difference of the output values of the test module versus the reference module. Then the defect detection method 300 continues to step 325.

The test system 100 may be implemented using a combination of hardware and software as follows. A processor may be used to carry out the post processing 112, systematic mismatch removal 114, defect likelihood estimation 116, and defect detection method 300. These processes may be handled by a processor in the system as a background process. Computer software would be run on the processor to carry out these functions. Further, parts of the BIST, e.g., calculation of the response (e.g., FFT, response values like gain, phase shift, distortion etc.) may optionally be calculated by the processor (especially if the processor is on a chip implementing the BIST or AMS.) The BIST device in AMS devices 105 may include: a stimulus module to provide stimulus signals of selectable frequency, amplitude, waveform etc.; a capture module to capture the AMS device response for later calculation of the response; and a control/calculation module to calculate the response.

FIG. 4 illustrates a second embodiment of a test system. The test system 400 uses a generative adversarial network (GAN) to select the stimulus parameters and the weights used to produce a weighted average of the other AMS devices. In general, tasks similar to the system testing described herein have been successfully addressed with by artificial neural networks. This especially relates to the system test application addressed herein where the relationship between the control parameters and the resulting effect on the behavior of the AMS devices is not known (and they cannot be known, given the many different positions where a defect can occur). Because modelling is not possible, learning the system behavior may be accomplished with a GAN.

A GAN includes two neural networks that contest with each other in the form of a zero-sum game, where one agent's gain is another agent's loss. Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. This may be done using unsupervised learning.

The core idea of a GAN is based on the indirect training through the discriminator, another neural network that can tell how realistic the input seems, which itself is also being updated dynamically. This means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator. This enables the model to learn in an unsupervised manner.

A real-life analogy of a GAN is provided using the following example. Imagine there is a game with three players: a government mint which prints legitimate money; a counterfeiter which produces fake currency hoping to make some quick criminal profits; and a police officer who wants to tell the difference between the real and fake currency.

During each round of the game, the mint and the counterfeiter each present a batch of money (real and fake, respectively) to the police officer. The mint can be considered a constant—it only produces the same type of real money over and over again. However, the counterfeiter is trying to produce fake money that looks as realistic as possible. At the same time, the police officer is doing their best to accurately distinguish fake money from real money.

At the end of each round, the scores of the police officer and the counterfeiter are tallied up. If the counterfeiter succeeded in fooling the police, the counterfeiter gets a good score and the police gets a low score and vice versa if the police correctly mark the counterfeit as fake currency. Based on these results, the counterfeiter adjusts their printing techniques, and the police officer sharpens their sensitivity to the telltale traces of fake currency. The players then start a new round with better counterfeiting and detection skills.