RELATED APPLICATIONS 0001 This application is a continuation-in-part of U.S. patent appication Ser. No. 08/404,024, filed Mar. 14, 1995 and issued Jul. 16, 1996 as U.S. Pat. No. 5,537,119, which is a continuation-in-part of U.S. patent application Ser. No. 08/171,327 filed Dec. 21, 1993, now U.S. Pat. No. 5,406,289.

FIELD OF THE INVENTION 0002 The invention relates generally to computerized techniques for processing data obtained from radiation reflections used to track multiple discrete object.

BACKGROUND OF THE INVENTION 0003 The invention relates generally to computerized techniques for processing data obtained from radar to track multiple discrete objects. 0004 There are many situations where the courses of multiple objects in a region must be tracked. Typically, radar is used to scan the region and generate discrete images or snapshots based on sets of returns or observations. In some types of tracking systems, all the returns from any one object are represented in an image as a single point unrelated to the shape or size of the objects. Tracking is the process of identifying a sequence of points from a respective sequence of the images that represents the motion of an object. The tracking problem is difficult when there are multiple closely spaced objects because the objects can change their speed and direction rapidly and move into and out of the line of sight for other objects. The problem is exacerbated because each set of returns may result from noise as well as echoes from the actual objects. The returns resulting from the noise are also called false positives. Likewise, the radar will not detect all echoes from the actual objects and this phenomena is called a false negative or missed detect error. For tracking airborne objects, a large distance between the radar and the objects diminishes the signal to noise ratio so the number of false positives and false negatives can be high. For robotic applications, the power of the radar is low and as a result, the signal to noise ratio can also be low and the number of false positives and false negatives high. 0005 In view of the proximity of the objects to one another, varied motion of the objects and false positives and false negatives, multiple sequential images should be analyzed collectively to obtain enough information to properly assign the points to the proper tracks. Naturally, the larger the number of images that are analyzed, the greater the amount of information that must be processed. 0006 While identifying the track of an object, a kinematic model describing the object's location, velocity and acceleration may be generated. Such a model provides the means by which the object's future motion can be predicted. Based upon such a prediction, appropriate action may be initiated. For example, in a military application there is a need to track multiple enemy aircraft or missiles in a region to predict their objective, plan responses and intercept them. Alternatively, in a commercial air traffic control application there is a need to track multiple commercial aircraft around an airport to predict their future courses and avoid collision. Further, in these and other applications, such as robotic applications, may use radar, sonar, infrared or other object detecting radiation bandwidths for tracking objects. In particular, in robotic applications reflected radiation can be used to track a single object which moves relative to the robot (or vice versa) so the robot can work on the object. 0007 Consider the very simple example of two objects being tracked and no false positives or false negatives. The radar, after scanning at time t1, reports objects at two locations in a first observation set. That is, it returns a set of two observations o11, o12. At time t2 it returns a similar set of two observations o21, o22) from a second observation set. Suppose from prior processing that track data for two tracks T1 and T2 includes the locations at to of two objects. Track T1 may be extended through the points in the two sets of observations in any of four ways, as may track T2. The possible extensions of T1 can be described as: T1, o11, o21, T1, o12, o22, T1, o12, o21 and T1, o12, o22. Tracks can likewise be extended from T2 in four possible ways including, T2, o12, o21. FIG. 1 illustrates these five (out of eight) possible tracks (to simplify the problem for purposes of explanation) . The five track extensions are labeled h11, h12, h13, h14, and h21 wherein h11 is derived from T1, o11, o21, h12 is derived from T1, o11, o22, h13 is derived from T1, o12, o21, h14 is derived from T1, o12, o22, and h21 is derived from (T2, o11, o21. The problem of determining which such track extensions are the most likely or optimal is hereinafter known as the assignment problem. 0008 It is known from prior art to determine a figure of merit or cost for assigning each of the points in the images to a track. The figure of merit or cost is based on the likelihood that the point is actually part of the track. For example, the figure of merit or cost may be based on the distance from the point to an extrapolation of the track. FIG. 1 illustrates costs 21 22 21 modeled target characteristics. The function to calculate the cost will normally incorporate detailed characteristics of the sensor, such as probability of measurement error, and track characteristics, such as likelihood of track maneuver. 0009 FIG. 2 illustrates a two by two by two matrix, c, that contains the costs for each point in relation to each possible track. The cost matrix is indexed along one axis by the track number, along another axis by the image number and along the third axis by a point number. Thus, each position in the cost matrix lists the cost for a unique combination of points and a track, one point from each image. FIG. 2 also illustrates a 0, 1 assignment matrix, z, which is defined with the same dimensions as the cost matrix. Setting a position in the assignment matrix to one means that the equivalent position in the cost matrix is selected into the solution. The illustrated solution matrix selects the h14, h21 solution previously described. Note that for the above example of two tracks and two snapshots, the resulting cost and assignment matrices are three dimensional. As used in this patent application, the term dimension means the number of axes in the cost or assignment matrix while size refers to the number of elements along a typical axis. The costs and assignments have been grouped in matrices to facilitate computation. 0010 A solution to the assignment problem satisfies two constraintsfirst, the sum of the associated costs for assigning points to a track extension is minimized and, second, if no false positives or false negatives exist, then each point is assigned to one and only one track. 0011 When false positives exist, however, additional hypothetical track extensions incorporating the false positives will be generated. Further note that the random locations of false positives will, in general, not fit well with true data and such additional hypothetical track extensions will result in higher costs. Also note that when false negative errors exist, then the size of the cost matrix must grow to include hypothetical track extensions formulated with gaps (i.e., data omissions where there should be legitimate observation data) for the false negatives. Thus, the second criteria must be weakened to reflect false positives not being as signed and also to permit the gap filler to be multiply assigned. With hypothetical cost calculated in this manner then the foregoing criteria for minimization will tend to materialize the false negatives and avoid the false positives. 0012 For a 3-dimensional problem, as is illustrated in FIG. 1, but with N1 (initial) tracks, N2 observations in scan 1, N3 observations in scan 2, false positives and negatives assumed, the assignment problem can be formulated as: 1

\begin{matrix} (a) Minimize :_{i_{1} = 0}^{N_{1}}_{i_{2} = 0}^{N_{2}}_{i_{3} = 0}^{N_{3}} c_{i_{1} i_{2} i_{3}} z_{i_{1} i_{2} i_{3}} (b) Subject to :_{i_{2} = 1}^{N_{2}}_{i_{3} = 1}^{N_{3}} z_{i_{1} i_{2} i_{3}} = 1, i_{1} = 1,, N_{1} (c)_{i_{1} = 1}^{N_{1}}_{i_{3} = 1}^{N_{3}} z_{i_{1} i_{2} i_{3}} 1, i_{2} = 1, N_{2} (d)_{i_{1} = 1}^{N_{1}}_{i_{2} = 1}^{N_{2}} z_{i_{1} i_{2} i_{3}} 1, i_{3} = 1, N_{3} (e) z_{i_{1} i_{2} i_{3}} {0, 1} z_{i_{1} i_{2} i_{3}} & [1.0] \end{matrix}

0013 where c is the cost and z is a point or observation assignment, as in FIG. 2. 0014 The minimization equation or equivalently objective function 1.0 (a) specifies the sum of the element by element product of the c and z matrices. The summation includes hypothesis representations zi1i2i3 with observation number zero being the gap filler observation. Equation 1.0(b) requires that each of the tracks T1, . . . TN1 be extended by one and only one hypothesis. Equation 1.0 (c) relates to each point or observation in the first observation set and requires that each such observation, except the gap filler, can only associate with one track but because of the less than condition it might not associate with any track. Equation 1.0 (d) is like 1.0 (c) except that it is applicable to the second observation set. Equation 1.0 (e) requires that elements of the solution matrix z be limited to the zero and one values. 0015 The only known method to solve Problem Formulation 1.0 exactly is a method called Branch and Bound. This method provides a systematic ordering of the potential solutions so that solutions with a same partial solution are accessible via a branch of a tree describing all possible solutions whereby the cost of unexamined solutions on a branch are incrementally developed as the cost for other solutions on the branch are determined. When the developing cost grows to exceed the previously known minimal cost (i.e., the bound) then enumeration of the tree branch terminates. Evaluation continues with a new branch. If evaluation of the cost of a particular branch completes, then that branch has lower cost than the previous bound so the new cost replaces the old bound. When all possible branches are evaluated or eliminated then the branch that had resulted in the last used bound is the solution. If we assume that N1N2N3n and that branches typically evaluate to half there full length, then workload associated with branch and bound is proportional to 2

{(n! | \frac{n}{2}!)}^{2} .

0016 This workload is unsuited to real time evaluation. 0017 The Branch and Bound algorithm has been used in past research on the Traveling Salesman Problem. Messrs. Held and Karp showed that if the set of constraints was relaxed by a method of Lagrangian Multipliers (described in more detail below) then tight lower bounds could be developed in advance of enumerating any branch of the potential solution. By initiating the branch and bound algorithm with such a tight lower bound, significant performance improvements result in that branches will typically evaluate to less than half their full length. 0018 Messrs. Frieze and Yadagar in dealing with a problem related to scheduling combinations of resources, as in job, worker and work site, showed that Problem Formulation 1.0 applied. They further described a solution method based upon an extension of the Lagrangian Relaxation previously mentioned. The two critical extensions provided by Messrs. Frieze and Yadagar were: (1) an iterative procedure that permitted the lower bound on the solution to be improved (by hill climbing described below) and (2) the recognition that when the lower bound of the relaxed problem was maximized, then there existed a method to recover the solution of the non-relaxed problem in most cases using parameters determined at the maximum. The procedures attributed to Messrs. Frieze and Yadagar are only applicable to the 3-dimensional problem posed by Problem Formulation 1.0 and where the cost matrix is fully populated. However, tracking multiple airborne objects usually requires solution of a much higher dimensional problem. 0019 FIGS. 1 and 2 illustrate an example where look ahead data from the second image improved the assignment accuracy for the first image. Without the look ahead, and based only upon a simple nearest neighbor approach, the assignments in the first set would have been reversed. Problem Formulation 1.0 and the prior art only permit looking ahead one image. In the prior art it was known that the accuracy of assignments will improve if the process looks further ahead, however no practical method to optimally incorporate look ahead data existed. Many real radar tracking problems involve hundreds of tracks, thousands of observations per observation set and matrices with dimensions in the range of 3 to 25 including many images of look ahead. 0020 It was also known that the data assignment problem may be simplified (without reducing the dimension of the assignment problem) by eliminating from consideration for each track those points which, after considering estimated limits of speed and turning ability of the objects, could not physically be part of the track. One such technique, denoted hereinafter the cone method, defines a cone as a continuation of each previously determined track with the apex of the cone at the end of the previously defined track. The length of the cone is based on the estimated maximum speed of the object and the size of the arc of the cone is based on the estimated maximum turning ability of the object. Thus, the cone defines a region outside of which no point could physically be part of the respective track. For any such points outside of the cones, an infinite number could be put in the cost matrix and a zero could be preassigned in the assignment matrix. It was known for the tracking problem that these elements will be very common in the cost and selection matrices (so these matrices are sparse). 0021 It was also known in the prior art that one or more tracks which are substantially separated geographically from the other tracks can be separated also in the assignment problem. This is done by examining the distances from each point to the various possible tracks. If the distances from one set of points are reasonably short only in relation to one track, then they are assigned to that track and not further considered with the remainder of the points. Similarly, if a larger group of points can only be assigned to a few tracks, then the group is considered a different assignment problem. Because the complexity of assignment problems increases dramatically with the number of possible tracks and the total number of points in each matrix, this partitioning of the group of points into a separate assignment problem and removal of these points from the matrices for the remaining points, substantially reduces the complexity of the overall assignment problem. 0022 A previously known Multiple Hypothesis Testing (MHT) algorithm (see Blackman, Multiple-Target Tracking with Radar Applications, Chapter 10, Artech House Norwood Mass., 1986) related to formulation and scoring. The MHT procedure describes how to formulate the sparse set of all reasonable extension hypothesis (for FIG. 1 the set h11. . . h24) and how to calculate a cost of the hypothesis Ti, o1j, o2k based upon the previously calculated cost for hypothesis T1, o1j. The experience with the MHT algorithm, known in the prior art, is the basis for the assertion that look ahead through k sets of observations results in improved assignment of observations from the first set to the track. 0023 In theory, the MHT procedure uses the extendable costing procedure to defer assignment decision until the accumulated evidence supporting the assignment becomes overwhelming. When it makes the assignment decision it then eliminates all potential assignments invalidated by the decision in a process called pruning the tree. In practice, the MHT hypothesis tree is limited to a fixed number of generations and the overwhelming evidence rule is replaced by a most likely and feasible rule. This rule considers each track independently of others and is therefore a local decision rule. 0024 A general object of the present invention is to provide an efficient and accurate process for assigning each point object in a region from multiple images to a proper track and then taking an action based upon the assignments. 0025 A more specific object of the present invention is to provide a technique of the foregoing type which determines the solution of a k-dimensional assignment problem where k is greater than or equal to three.

SUMMARY OF THE INVENTION 0026 The present invention relates to a method and apparatus for tracking objects. In particular, the present invention tracks movement or trajectories of objects by analyzing radiation reflected from the objects, the invention being especially useful for real-time tracking in noisy environments. 0027 In providing such a tracking capability, a region containing the objects is repeatedly scanned to generate a multiplicity of sequential images or data observation sets of the region. One or more points (or equivalently observations), in each of the images or observation sets are detected wherein each such observation either corresponds to an actual location of an object or is an erroneous data point due to noise. Subsequently, for each observation detected, figures of merit or costs are determined for assigning the observation to each of a plurality of previously determined tracks. *Afterwards, a first optimization problem is specified which includes: 0028 (a) a first objective function for relating the above mentioned costs to potential track extensions through the detected observations (or simply observations); and 0029 (b) a first collection of constraint sets wherein each constraint set includes constraints to be satisfied by the observations in a particular scan to which the constraint set is related. In general, there is a constraint for each observation of the scan, wherein the constraint indicates the number of track extensions to which the observation may belong. 0030 In particular, the first optimization problem is formulated, generated or defined as an M-dimensional assignment problem wherein there are M constraint sets in the first collection of constraint sets (i.e., there are M scans being examined) and the first objective function minimizes a total cost for assigning observations to various track extensions wherein terms are included in the cost, such that the terms have the figures of merit or costs for hypothesized combinations of assignments of the observations to the tracks. Subsequently, the formulated M-dimensional assignment problem is solved by reducing the complexity of the problem by generating one or more optimization problems each having a lower dimension and then solving each lower dimension optimization problem. That is, the M-dimensional assignment problem is solved by solving a plurality of optimization problems each having a lower number of constraint sets. 0031 The reduction of the M-dimensional assignment problem to a lower dimensioned problems is accomplished by relaxing the constraints on the points of one or more scans thereby permitting these points to be assigned to more than one track extension. In relaxing the constraints, terms having penalty factors are added into the objective function thereby increasing the total cost of an assignment when one or more points are assigned to more than one track. Thus, the reduction in complexity by this relaxation process is iteratively repeated until a sufficiently low dimension is attained such that the lower dimensional problem may be solved directly by known techniques. 0032 In one embodiment of the invention, each k-dimensional assignment problem 2<kM, is iteratively reduced to a k1 dimensional problem until a 2-dimensional problem is specified or formulated. Subsequently, the 2-dimensional problem formulated is solved directly and a recovery technique is used to iteratively recover an optimal or near-optimal solution to each k-dimensional problem from a derived (k1) dimensional problem k2,3,4, . . .M. 0033 In performing each recovery step (of obtaining a solution to a k-dimensional problem using a solution to a (k1)-dimensional problem) an auxiliary function, is utilized. In particular, to recover an optimal or near-optimal solution to a k-dimensional problem, an auxiliary function, k-1, k4,5, . . . , M, is specified and a region or domain is determined wherein this function is maximized, whereby values of the region determine the penalty factors of the (k1)-dimensional problem such that another 2-dimensional problem can be formulated which determines a solution to the k-dimensional problem using the penalty factors of the (k1)-dimensional problem. 0034 Each Auxiliary function k of both lower dimensional problem penalty factors and a solution at the dimension k at which the penalized cost function is solved directly (typically a 2-dimensional problem). Further, in determining, for auxiliary function 4 determined, and utilized to identify the peak region. Thus, gradients are used for each of the approximation functions 3 4 determining penalty factors penalty factors until tm1 is used in determining the penalty factors for the M1 dimensional problem. Subsequently, once the M-dimensional problem is solved (using a 2-dimensional problem to go from an (M1) dimensional solution to an M-dimensional solution), one or more of the following actions are taken based on the track assignments: sending a warning to aircraft or a ground or sea facility, controlling air traffic, controlling anti-aircraft or anti-missile equipment, taking evasive action, working on one of the objects. 0035 According to one feature of this first embodiment of the present invention, the following steps are also performed before the step of defining the auxiliary function. A preliminary auxiliary function is defined for each of the lower dimensional problems having a dimension equal or one greater than the dimension at which the penalized cost function is solved directly. The preliminary auxiliary function is a function of lower order penalty values and a solution at the dimension at which the penalized cost function was solved directly. In determining a gradient of the preliminary auxiliary function, step in the direction of the gradient to identify a peak region of the preliminary auxiliary function and determine penalty factors at the peak region. Iteratively repeat the defining, gradient determining, stepping and peak determining steps to define auxiliary functions at successively higher dimensions until the auxiliary function at 6-18 (k1) dimension is determined. In an alternative second embodiment of the present invention, instead of reducing the dimentiality of the M-dimensional assignment problem by a single dimension at a time, a plurality of dimensions are relaxed simultaneously. This new strategy has the advantage that when the M-dimensional problem is relaxed directly to a 2-dimensional assignment problem, then all computations may be performed precisely without utilizing an auxiliary function such as k embodiment. More particularly, the second embodiment solves the first optimization problem (i.e., the M- dimensional assignment problem) by specifying (i.e., creating, generating, formulating and/or defining) a second optimization problem. The second optimization problem includes a second objective function and a second collection of constraint sets wherein: 0036 a) the second objective function is a combination of the first objective function and penalty factors or terms determined for incorporating M-m constraint sets of the first optimization problem into the second objective function; 0037 b) the constraint sets of the second collection include only m of the constraint sets of the first collection of constraints, 0038 wherein 2<mM1. Note that, once the second optimization problem has been specified or formulated, an optimal or near-optimal solution is determined and that solution is used in specifying (i.e, creating, generating, formulating and/or defining) a third optimization problem of M-m dimensions (or equivalently constraint sets) . The third optimization problem is subsequently solved by decomposing it using the same procedure of this second embodiment as was used to decompose the first optimization problem above. Thus, a plurality of instantiations of the third optimization problem are specified, each successive instantiation having a lower number of dimensions, until an instance of the third optimization problem is a two dimensional assignment problem which can be solved directly. Subsequently, whenever an instance of the third optimization problem is solved, the solution is used to recover a solution to the instance of the first optimization problem from which this instance of the third optimization was derived. Thus, an optimal or near-optimal solution to the original first optimization problem may be recovered through iteration of the above steps. 0039 As mentioned above, the second embodiment of the present invention is especially advantageous when m2, since in this case all computations may be performed precisely and without utilizing auxiliary functions. 0040 Other features and benefits of the present invention will become apparent from the detailed description with the accompanying drawings contained hereinafter.

BRIEF DESCRIPTION OF THE FIGURES 0041 FIG. 1 is a graph of images or data sets generated by a scan of a region and possible tracks within the images or data sets according to the prior art. 0042 FIG. 2 illustrates cost and assignment matrices for the data sets of FIG. 1 according to the prior art. 0043 FIG. 3 is a block diagram of the present invention. 0044 FIG. 4 is a flow chart of a process according to the prior art for solving a 3-dimensional assignment problem. 0045 FIG. 5 Is a flow chart of a process according to the present invention for solving a k-dimensional assignment problem where k is greater than or equal to 3. 0046 FIG. 6 is a graph of various functions used to explain the present invention. 0047 FIG. 7 is another block diagram of the present invention for solving the k-dimensional assignment problem where k is greater than or equal to 3. 0048 FIG. 8 is a flowchart describing the procedure for solving a n-dimensional assignment problem according to the second embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 0049 Referring now to the other figures in detail wherein like reference numerals indicate like elements throughout the several views, FIG. 3 illustrates a system generally designated 100 for implementing the present invention. System 100 comprises, for example, a radar station 102 (note sonar, microwave, infrared and other radiation bandwidths are also contemplated) for scanning a region which may be, for example, an aerial region (in aerial surveillance applications) or a work region (in robotic applications) and generating signals indicating locations of objects within the region. The signals are input to a converter 104 which converts the signals to data points or observations in which each object (or false positive) is represented by a single point. The output of the converter is input to and readable by a computer 106. As described in more detail below, the computer 106 assigns the points to respective tracks, and then displays the tracks and extrapolations of the tracks on a monitor 110. Also, the computer 106 determines an appropriate action to take based on the tracks and track extensions. For example, in a commercial application at an airport, the computer can determine if two aircraft being tracked are on a collision course and if so, signal a transmitter 112 to warn each aircraft, or if a scheduled take-off will pose the risk of collision, delay the take-off. For a military application on a ship or base, the computer can determine subsequent coordinates of enemy aircraft and send the coordinates to an antiaircraft gun or missile 120 via a communication channel 122. In a robotic application, the computer controls the robot to work on the proper object or portion of the object. 0050 The invention generates k-dimensional matrices where k is the number of images or sets of observation data in the look ahead window plus one. Then, the invention formulates a k-dimensional assignment problem as in 1.0 above. 0051 The k-dimensional assignment problem is subsequently relaxed to a (k1)-dimensional problem by incorporating one set of constraints into the objective function using a Lagrangian relaxation of this set. Given a solution of the (k1)-dimensional problem, a feasible solution of the k-dimensional problem is then reconstructed. The (k1)-dimensional problem is solved in a similar manner, and the process is repeated until it reaches the two-dimensional problem that can be solved exactly. The ideas behind the Lagrangian relaxation scheme are outlined next. 0052 Consider the integer programming problem 3

\begin{matrix} Minimize v (z) = c^{T} z Subject to : Az b Bz d z_{i} is an integer for i I & [1.0.1] \end{matrix}

0053 where the partitioning of the constraints is natural in some sense. Given a multiplier vector u0, the Lagrangian relaxation of 1.0.1 relative to the constraints Bzd is defined to be 4

\begin{matrix} ^{T T} Subject to : Az b z_{i} is an integer for i I & [1.0.2] \end{matrix}

0054 If the constraint set Bzd is replaced by Bzd, the nonnegativity constraint on u is removed. cTzuT(Bzd) is the Lagrangian relative to the constraints Bzd, and hence the name Lagrangian relaxation. The following fact gives the relationship between the objective functions of the original and relaxed problems. 0055 FACT A.1. If {overscore (z)} is an optimal solution to 1.0.1, then v({overscore (z)}) for all u0. If an optimal solution {circumflex over (z)} of 1.0.2 is feasible for 1.0.2, then {circumflex over (z)} is an optimal solution for 1.0.1 and 0056 Algorithm k k28 0 converging to the solution {overscore (u)} of Maximize 0 and a corresponding sequence of feasible solutions zkk0 of 1.0.1 as follows: 0057 1. Generate in initial approximation u0. 0058 2. Given uk, choose a search direction sk and a search distance k k k k k estimate uk by uk1ukk k 0059 3. Given uk1 and a feasible solution {circumflex over (z)}k1(uk1) of 1.0.2, recover a feasible solution zk1(uk1) of the integer programming problem 1.0.1. 0060 4. Check the termination criteria. If the algorithm is not finished, set kk1 and return to Step 2. Otherwise, terminate the algorithm. 0061 If {circumflex over (z)} is an optimal solution of 1.0.1, then

kv({overscore (z)})(zk)

0062 Since the optimal solution {circumflex over (z)}{circumflex over (z)}(u) of 1.0.2 is usually not a feasible solution of 1.0.1 for any choice of the multipliers, 0063 0064 0065 0066 ppk in z1i . . . ikk in problem 1.1 below indicates that the track extension represented by zi1k. . . ik includes a false positive in the pth observation set. Note that this implies that a hypothesis be formed incorporating an observation with k1 gap fillers, e.g. , z0 . . . 0ip0 . . . 0, ip0. Thus, the resulting generalization of Problem Formulation 1.0 without the less than complication within the constraints is the following k-Dimensional Assignment Problems in which k3: 5

\begin{matrix} Minimize_{i_{1} = 0}^{N_{1}}_{i_{k} = 0}^{N_{k}} c_{i_{1} i_{k}} z_{i_{1} i_{k}} Subject to :_{i_{2} = 0}^{N_{2}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}} = 1, i_{1} = 1, N_{1}_{i_{1} = 0}^{N_{1}}_{i_{j - 1} = 0}^{M_{j - 1}}_{i_{j + 1} = 0}^{M_{j + 1}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}} = 1, for i_{j} = 1,, N_{j} and j = 2,, k - 1,_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}} z_{i_{1} i_{k}} = 1, i_{k} = 1, N_{k} z_{l_{1} i_{k}} {0, 1} i_{n} n = 1,, k & (1.1) \end{matrix}

0067 where c and z are similarly dimensioned matrices representing costs and hypothetical assignments. Note, in general, for tracking problems these matrices are sparse. 0068 After formulating the k-dimensional assignment problem as in 1.1, the present invention solves the resulting problem so as to generate the outputs required by devices 110, 112, 122 and 130. For each observation set Oi received from converter 104 at time ti where i1. . . , N1, the computer processes Oi in a batch together with the other observation sets Oik1, . . . , Oi and the track Tik, i.e., Tj is the set of all tracks that have been defined up to but not including Oj. (Note, bold type designations refer to the vector of elements for the indicated time, i.e., the set of all observations in the scan or tracks existing at the time, etc.) The result of this processing is the new set of tracks Tik1 and a set of cost weighted possible solutions indicating how the tracks might extend to the current time ti. At time ti1 the batch process is repeated using the newest observation set and deleting the oldest. Thus, there is a moving window of observation sets which is shifted forward to always incorporate the most recent observation set. The effect is that input observation sets are reused for k1 cycles and then on the observation set's k-th reuse each observation within the observation set is integrated into a track. 0069 FIG. 7 illustrates various processes implemented upon receipt of each observation set. Except for the addition of the k-dimensional assignment solving process 300 and the modification to scoring process 154 to build data structures suitable for process 300, all processes in FIG. 7 are based upon prior art. The following processes 150 and 152 extend previously defined tracks hi1 based on new observations. Gate formulation and output process 156 determines, for each of the previously defined tracks, a zone wherein the track may potentially extend based on limits of velocity, maneuverability and radar precision. One such technique to accomplish this is the cone method described previously. The definition of the zone is passed to gating process 150. When a new observation set Oi is received, the gating process 150 will match each member observation with the zone for each member of the hypothetical set hi1. After all input observations from Oi are processed the new hypothesis set hi is generated by extending each track of the prior set of hypothetical tracks hi1 either with missed detect gap fillers or with all new observation elements satisfying the track's zone. This is a many to many matching in that each hypothesis member can be extended to many new observations and each new observation can be used to extend many hypotheses. It, however, is not a full matching in that any hypothesis will neither be matched to all observations nor vice versa. It is this matching characteristic that leads to the sparse matrices involved in the tracking process. Subsequently, gating 150 forwards the new hypothesis set hi to filtering process 152. Filtering process 152 determines a smooth curve for each member of hi. Such a smooth curve is more likely than a sharp turn from each point straight to the next point. Further, the filtering process 152 removes small errors that may occur in generating observations. Note that in performing these tasks, the filtering process 152 preferably utilizes a minimization of a least squares test of the points in a track hypothesis or a Kalman Filtering approach. 0070 As noted above, the foregoing track extension process requires knowledge of a previous track. For the initial observations, the following gating process 158 and filtering process 160 determine the previous track based on initial observations. In determining the initial tracks, the points from the first observation set form the beginning points of all possible tracks. After observation data from the next observation set is received, sets of simple two point straight line tracks are defined. Then, promotion, gate formulation, and output step 162 determines a zone in which future extensions are possible. Note that filtering step 160 uses curve fitting techniques to smooth the track extensions depending upon the number of prior observations that have accumulated in each hypothesis. Further note that promotion, gate formulation and output process 162 also determines when sufficient observations have accumulated to form a basis for promoting the track to processes 150 and 152 as described above. 0071 The output of each of the filtering processes 152 and 160 is a set of hypothetical track extensions. Each such extension contains the hypothetical initial conditions (from the previous track), the list of observations incorporated in the extension, and distance between each observation and the filtered track curve. Scoring process 154 determines the figure of merit or cost of an observation being an extension of a track. In one embodiment, the cost is based on the above-mentioned distance although the particular formula for determining the cost is not critical to the present invention. A preferred formula for determining the cost utilizes a negative log likelihood function in which the cost is the negative of the sum of: (a) the logs of the distances normalized by sensor standard deviation parameters, and (b) the log likelihoods for events related to: track initiation, track termination, track maneuver, false negatives and false positives. Note that track maneuvers are detected by comparing the previous track curve with the current extension. Further note that some of the other events related to, for example, false negatives and false positives are detected by analyzing the relative relationship of gap fillers in the hypothesis. Thus, after determining that one of these events occurred, a cost for it can be determined based upon suitable statistics tables and system input parameters. The negative log likelihood function is desirable because it permits effective integration of the useful components. Copies of the set of hypothetical track extensions which are scored are subsequently passed directly to one of the gate formulation and output steps 156 and 162. Note that the scoring process 154 also arranges the actual scores in a sparse matrix based upon observation identifiers, and passes them to k-dimensional assignment problem solving process 300. 0072 The assignment solving process 300 is described below. Its output is simply the list of assignments which constitute the most likely solution of the problem described by Equation 1.1. Note that both gate formulation and output processes 156 and 162 use (at different times) the list of assignments to generate the updated track history Ti to eliminate or prune alternative previous hypotheses that are prohibited by the actual assignments in the list, and subsequently to output any required data. Also note that when one of the gate formulation and output processes 156 and 162 accomplish these tasks, the process will subsequently generate and forward the new set of gates for each remaining hypothesis and the processes will then be prepared to receive the next set of observations. In one embodiment, the loop described here will generate zones for a delayed set of observations rather than the subsequent set. This permits processes 156 and 162 to operate on even observation sets while the scoring step 154 and k-dimensional solving process operate on odd sets of observations, or vice versa. 0073 The assignment solving process 300 permits the present invention to operate with window size of k1 for k3. The upper limit on k depends only upon the computational power of the computer 106 and the response time constraints of system 100. The k1 observation sets within the processing window plus the prior track history result in a k-dimensional Assignment Problem as described by Problem Formulation 1.1. The present invention solves this generalized problem including the processes required to consider false positives and negatives, and also the processes required to consider sparse matrix problem formulations. 0074 I. A FIRST EMBODIMENT OF THE k-DIMENSIONAL ASSIGNMENT SOLVER 300 0075 In describing a first embodiment of the k-dimensional assignment solver 300, it is worthwhile to also discuss the process of FIG. 4 which is used by the solver 300. FIG. 4 illustrates use of the Frieze and Yadagar process as shown in prior art for transforming a 3-dimensional assignment problem into a 2-dimensional assignment problem and then use a hill climbing algorithm to solve the 3-dimensional assignment problem. The solver 300 uses a Lagrangian Relaxation technique (well known in the art) to reduce the dimension of an original k-dimensional assignment problem (k>3) down to a 3-dimensional problem and then use the process of FIG. 4 to solve the 3-dimensional problem. Further note that the Lagrangian Relaxation technique is also utilized by the process of FIG. 4 and that in using this technique the requirement that each point is assigned to one and only one track is relaxed. Instead, an additional cost, which is equal to a respective Lagrangian Coefficient u, is added to the cost or objective function 1.0(a) whenever a point is assigned to more than one track. This additional cost can be picked to weight the significance of each constraint violation differently, so this additional cost is represented as a vector of coefficients u which are correlated with respective observation points. Hill climbing will then develop a sequence of Lagrangian Coefficients sets designated (u0,. . . uj, uj1, . . . , up). That correspond to an optimum solution of the 2-dimensional assignment problem. The assignments at this optimum solution are then used to recover the assignment solution of the 3-dimensional assignment problem. 0076 In step 200 of FIG. 4, initial values are selected for the u0 coefficients. Because the Lagrangian Relaxation process is iterative, the initial values are not critical and are all initially selected as zero. In step 202, these additional costs are applied to the objective function 1.0(a). With the addition of the costs u, the goal is still to assign the points which minimize the total cost. This transforms Equation 1.0(a) , written for k3 and altered to exclude mechanisms related to false positives and negatives, into objective function 2.1(a). In the first iteration it is not necessary to consider the u matrix because all u values are set to zero. To relax the requirement that each point be assigned to one and only one track, the constraint Equation 1.0 (d) is deleted, thereby permitting points from the last image to be assigned to more than one track. Note that while any axis can be chosen for relaxation, observation constraints are preferably relaxed. The effect of this relaxation is to create a new problem which must have the same solution in the first two axes but which can have a differing solution in the third axis. The result is constraints 2.1(bd). 6

\begin{matrix} (a) Minimize :_{i_{1} = 0}^{N_{1}}_{i_{2} = 0}^{N_{2}} {(c_{i_{1} i_{2} i_{3}} - u_{j_{i_{3}}})} z_{i_{1} i_{2} i_{3}} (b) Subject to :_{i_{2} = 1}^{N_{2}}_{i_{3} = 1}^{N_{3}} z_{i_{1} i_{2} i_{3}} = 1, i = 1,, N_{1} (c)_{i_{1} = 1}^{N_{1}}_{i_{3} = 1}^{N_{3}} z_{i_{1} i_{2} i_{3}} 1, i_{2} = 1,, N_{2} (d) z_{i_{1} i_{2} i_{3}} {0, 1} z_{i_{1} i_{2} i_{3}} & [2.1] \end{matrix}

0077 Step 204 then generates from the 3-dimensional problem described by Problem Formulation 1.0 a new 2-dimensional problem formulation which will have the same solution for the first two indices. As Problem Formulation 2.1 has no constraints on the 3rd axis, any value within a particular 3rd axis can be used in a solution, but using anything other than the minimum value from any 3-rd axis has the effect of increasing solution cost. Conceptually, the effect of step 204 is to change the 3-dimensional arrays in Problem Formulation 2.1 into 2-dimensional arrays as shown in Problem Formulation 2.2 and to generate the new 2-dimensional matrix mi1i2 defined as shown in Equation 2.3. 7

\begin{matrix} (a) Minimize :_{i_{1} = 0}^{N_{1}}_{i_{2} = 0}^{N_{2}} {\begin{matrix} Min : \\ (c_{i_{1} i_{2} i_{3}} - u_{j_{i_{3}}}) \\ i_{3} \end{matrix}} z_{i_{1} i_{2}} (b) Subject to :_{i_{2} = 1}^{N_{2}}_{i_{3} = 1}^{N_{3}} z_{i_{2} i_{3}} = 1, i_{1} = 1,, N_{1} (c)_{i_{1} = 1}^{N_{1}}_{i_{3} = 1}^{N_{3}} z_{i_{1} i_{3}} 1, i_{2} = 1,, N_{2} (d) z_{i_{1} i_{2}} {0, 1} z_{i_{1} i_{2}} & [2.2] \\ m_{i_{1} i_{2}} = Min : \arg minimize {c_{i_{1} i_{2}} - u_{j_{i}} | i = 1,, N_{k}} & [2.3] \end{matrix}

0078 The cost or objective function for the reduced problem as defined by 2.2(a), if evaluated at all possible values of u is a surface over the domain of U3. This surface is referred to as u 0079 u 0080 uj 0081 coefficients uj1 whose corresponding Problem Formulation 2.2 problem solution is a cost value closer to the peak of 0 u 0082 up 0083 dimensional assignment problem requires solving the Equation 2.2 problem corresponding to up. 0084 In step 206, the two dimensional problem is solved directly using a technique known to those skilled in the art such as Reverse Auction for the corresponding cost and solution values. This is the evaluation of one point on the surface or for the first iteration u0 0085 Thus, after this first iteration, the points have been assigned based on all u values being arbitrarily set to zero. Because the u values have been arbitrarily assigned, it is unlikely that these assignments are correct and it is likely that further iterations are required to properly assign the points. Step 208 determines whether the points have been properly assigned after the first iteration by determining if for this set of assignments whether a different set of u values could result in a higher total cost. Thus, step 208 is implemented by determining the gradient of objective function 2.2(a) with respect to uj. If the gradient is substantially non-zero (greater than a predetermined limit) then the assignments are not at or near enough to the peak of the u 0086 uj1 0087 Hill climbing Step 212 determines the uj1 values based upon the uj values, the direction resulting from protecting the previous gradient into the U3 domain, and a step size. The solution value of the 2-dimensional problem is the set of coefficients that minimize the 2-dimensional problem and the actual cost at the minimum. Those coefficients augmented by the coefficients stored in mi1i2 permit the evaluation (but not the minimization) of the cost term in Problem Formulation 2.1. These two cost terms are lower and upper bounds on the actual minimized cost of the 3-dimensional problem, and the difference between them in combination with the gradient is used to compute the step size. 0088 With this new set of u values, steps 202-210 are repeated as a second iteration. Steps 212 and 202-210 are repeated until the gradient as a function of u determined in step 208 is less than the predetermined limit. This indicates that the up values which locate the peak area of the u surface are determined and that the corresponding Problem Formulation 2.2 has been solved. Step 214 will attempt to use the assignments that resulted from this particular 2-dimensional assignment problem to recover the solution of the 3-dimensional assignment problem as described below. If the limit was chosen properly so that the u values are close enough to the peak, this recovery will yield the proper set of assignments that rigidly satisfies the constraint that each point be assigned to one and only one track. However, if the u values are not close enough to the peak, then the limit value for decision 210 is reduced and the repetition of steps 212 and 202-210 is continued. 0089 Step 214 recovers the 3-dimensional assignment solution by using the assignment values determined on the last iteration through step 208. Consider the 2-dimensional z assignment matrix to have 1's in the locations specified by the list L1(ai, bi)i1N. If the 3-dimensional z matrix is specified by placing 1's at the location indicated by the list L2(ai, bi, maib1){haeck over (N)}1 then the result is a solution of Problem Formulation 2.1. Let L3(maib1){haeck over (N)}1 be the list formed by the third index. If each member of L3 is unique then the L2 solution satisfies the third constraint so it is a solution to Problem Formulation 1.0. When this is not the case, recovery determines the minimal substitutions required within list L3 so that it plus L1 will be a feasible solution, i.e., a solution which satisfies the constraints of a problem formulation, but which may not optimize the objective function of the problem formulation. This stage of the recovery process is formulated as a 2-dimensional assignment problem: Form a new cost matrix ci,ji,j1N where ci,jcaibij, for j1. . . Ni and the Ni term is the total number of cost elements in the selected row of the 3-dimensional cost matrix. Attempt to solve this 2-dimensional problem for the minimum using two constraints sets. If a feasible solution is found then the result will have the same form as list L1. Replace the first set of indexes by the indicated (a1, bi) pairs taken from list L1 and the result will be a feasible solution of Problem Formulation 1.1. If no feasible solution to the new 2-dimensional problem exists then further effort to locate the peak of u 0090 0091 0092 0093 0094 by the set of observations o{overscore (y)}i1, . . . , M1, is not valid. Because the matrix is sparse the list of cost elements is stored as a packed list, and then for each dimension of the matrix, a vector of a variable length list of pointers to the cost elements is generated and stored. This organization means that for a particular observation o{overscore (y)} the jth list in the ith vector will be a list of pointers to all hypotheses in which o{overscore (y)} participates. This structure is further explained in the following section dealing with problem partitioning. 0095 The objective of the assignment solving process is to select from the set of all possible combinations of track extensions a subset that satisfies two criteria. First, each point in the subset of combinations should be assigned to one and only one track and therefore, included in one and only one combination of the subset, and second, the total of the scoring sums for the combinations of the subset should be minimized. This yields the following M-dimensional equations where kM: 8

\begin{matrix} (a) Minimize : v_{k} (z^{k}) =_{i_{1} = 0}^{N_{1}}_{i_{k} = 0}^{N_{k}} c_{i_{1} i_{k}}^{k} z_{i_{1} i_{k}}^{k} (b) Subject to :_{i_{2} = 0}^{N_{2}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}}^{k} = 1 i_{1} = 1, N_{1} (c)_{i_{1} = 0}^{N_{1}}_{i_{j - 1} = 0}^{N_{j - 1}}_{i_{j + 1} = 0}^{N_{j + 1}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}}^{k} = 1 for i_{j} = 1, N_{j} and j = 2, k - 1 (d)_{i_{1} = 0}^{N_{1}}_{i_{k - 1}}^{N_{k - 1}} z_{i_{1} i_{k}}^{k} = 1 i_{k} = 1, N_{k} (e) z_{i_{1} i_{k}}^{k} {0, 1} for all i_{1} l_{k} & [3.1] \end{matrix}

0096 and where ck is the cost matrix cilk. . . ik which is a function of the distance between the observed point zk and the smoothed track determined by the filtering step, and vk is the cost function. This set of equations is similar to the set presented in Problem Formulation 1.1 except that it includes the subscript and superscript k notation. Thus, in solving the M-dimensional Assignment Problem the invention reduces this problem to an M-1 dimensional Assignment Problem and then to an N-2 dimensional Assignment Problem, etc. Further, the symbol k3, . . . ,M customizes Problem Formulation 3.1 to a particular relaxation level. That is, the notation is used to reference data from levels relatively removed as in ck1 are the cost coefficients which existed prior to this level of relaxed coefficients ck. Note that actual observations are numbered from 1 to Ni where Ni is the number of observations in observation set i. Further note that the added 0 observation in each set of observations is the unconstrained gap filler. This element serves as a filler in substituting for missed detects, and a sequence of these elements including only one true observation represents the possibility that the observation is a false positive. Also note that by being unconstrained a gap filler may be used in as many hypotheses as required. 0097 While direct solution to 3.1 would give the precise assignment, the solution of k-dimensional equations directly for large k is too complex and time consuming for practice. Thus, the present invention solves this problem indirectly. 0098 The following is a short description of many aspects of the present invention and includes some steps according to the prior art. The first step in solving the problem indirectly is to reduce the complexity of the problem by the previously known and discussed Lagrangian Relaxation technique. According to the Lagrangian Relaxation technique, the absolute requirement that each point is assigned to one and only one track is relaxed such that for some one image, points can be assigned to more than one track. However, a penalty based on a respective Lagrangian Coefficient uk is added to the cost function when a point in the image is assigned to more than one track. The Lagrangian Relaxation technique reduces the complexity or dimension of the formulation of the assignment problem because constraints on one observation set are relaxed. Thus, the Lagrangian Relaxation is performed iteratively to repeatedly reduce the dimension until a 2-dimensional penalized cost function problem results as in Problem Formulation 2.1. This 2-dimensional problem is solved then directly by a previously known technique such as Reverse Auction. The penalized cost function for the 2-dimensional problem defines a valley or convex shaped surface which is a function of various sets of ukk3, . . . ,M penalty values and one set of assignments for the points in two dimensions. That is, for each particular u3 there is a corresponding 2-dimensional penalized cost function problem and its solution. Note that the solution of the 2-dimensional penalized cost function problem identifies the set of assignments for the particular u3 values that minimize the penalized cost function. However, these assignments are not likely to be optimum for any higher dimensional problem because they were based on an initial arbitrary set of uk values. Therefore, the next step is to determine the optimum assignments for the related 3-dimensional penalized cost function problem. There exists a 2-dimensional hill shaped function 9

u^{3} u^{k} | k > 3}

0099 values originally assigned. Then, the gradient of the hill-shaped 0100 u3 0101 previously selected for the one point on the hill (corresponding to the minimum of the penalized cost

u3 0102 which the corresponding problem will result in the peak of the characteristic. The next task is to recover the solution of the proper u4 values for the 4-dimensional problem. The foregoing hill climbing process will not work again because the foregoing hill climbing process when required to locate the 4-dimensional u4 values for the peak of 3 0103 exact definition of the 3 0104 case of 2 3 0105 the iteration can result in a greater than approximation of 3 0106 function 0107 uk 0108 gradient of the 0109 u4 u4 0110 results in a new 3-dimensional problem and requires the 2-dimensional hill climbing based upon new 2-dimensional problems. At the peak of the 3-dimensional to a complete solution. This process is repeated iteratively until the uk values that result in a corresponding solution at the peak of the highest order 0111 0112 Coefficient uM penalty values initially equal to zero. The Lagrangian Coefficients associated with the Mth constraint set are a NM1 element vector. The reduction of this M-dimensional problem in step 322 to a M1 dimensional problem uses the two step process described above. First, a penalty based on the value of the respective uM coefficient is added to the cost function when a point is assigned to more than one track and then the resultant cost function is minimized. However, during the first iteration, the penalty is zero because all uM values are initially set to zero. Second, the requirement that no point from any image can be assigned to more than one track is relaxed for one of the images. In the extreme this would allow a point from the relaxed image to associate with every track. However, the effect of the previous penalty would probably mean that such an association would not minimize the cost. The effect of the two steps in combination is to remove a hard constraint while adding the penalty to the cost function so that it operates like a soft constraint. For step 322 this two step process results in the following penalized cost function problem with kM: 10

\begin{matrix} \begin{matrix} (a)_{k}^{k} Minimize :_{k}^{k k} \\ =_{i_{1} = 0}^{N_{1}}_{i_{k} = 0}^{N_{k}} c_{i_{1} i_{k}}^{k} z_{i_{1} i_{k}}^{k} - \\ _{i_{k} = 0}^{N_{k}} u_{i_{k}}^{k}_{i_{1} = 0}^{N_{1}}_{i_{k - 1}}^{N_{k - 1}} z_{i_{1} i_{k}}^{k} - 1 \\ =_{i_{1} = 0}^{N_{1}}_{i_{k} = 0}^{N_{k}} (c_{i_{1} i_{k}}^{k} - u_{i_{k}}^{k}) z_{i_{1} i_{k}}^{k} + \\ _{i_{k} = 0}^{N_{k}} u_{i_{k}}^{k} \end{matrix} & [3.2] \\ (b) Subject to :_{i_{2} = 0}^{N_{2}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}}^{k} = 1 i_{1} = 1, N_{1} (c)_{i_{1} = 0}^{N_{1}}_{i_{j - 1} = 0}^{N_{j - 1}}_{i_{j + 1} = 0}^{N_{j + 1}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}}^{k} = 1 for i_{j} = 1, N_{j} and j = 2, k - 1 (d) z_{i_{1} i_{k}}^{k} {0, 1} for all i_{1} l_{k} \end{matrix}

0113 Because the constraint on single assignment of elements from the last image has been eliminated, a M1 dimensional problem can be developed by eliminating some of the possible assignments. As shown in Equations 3.3, this is done by selecting the smallest cost clement from each of the Mth axis vectors of the cost matrix. Reduction in this manner yields a new, lower order penalized cost function defined by Equations 3.3 which has the same minimum cost as does the objective function defined by 3.2(a) above. 0114 The cost vectors are selected as follows. Define an index array mi1 . . . ik1k and new cost array ci1k1. . . ik1 by: 11

\begin{matrix} m_{i_{1} i_{k}}^{k} = Min : \arg minimize {c_{i_{1} i_{k}}^{k} - u_{i_{k}}^{k} | i_{k} = 0, 1,, N_{k}} c_{i_{1} i_{k - 1}}^{k - 1} = c_{i_{1} i_{k - 1} m_{i_{1} i_{k}}^{k}}^{k} for (i_{1},, i_{k - 1}) (0,, 0) c_{0,, 0}^{k - 1} =_{i_{k} = 0}^{N_{k}} \min {0, c_{0 0}^{k} - u_{i_{k}}^{k}} & [3.3] \end{matrix}

0115 The resulting M1 dimensional problem is (where kM): 12

\begin{matrix} _{k}^{k}_{k - 1}^{k - 1}_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}} c_{i_{1} i_{k - 1}}^{k - 1} z_{i_{1} i_{k - 1}}^{k - 1} Subject to :_{i_{2} = 0}^{N_{2}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}}^{k - 1} = 1 i_{1} = 1, N_{1}_{i_{1} = 0}^{N_{1}}_{i_{j - 1} = 0}^{N_{j - 1}}_{i_{j + 1} = 0}^{N_{j + 1}}_{i_{k} = 0}^{N_{k}} z_{i_{1} i_{k}}^{k - 1} = 1 for i_{j} = 1,, N_{j} and j = 2, k - 1_{i_{1} = 0}^{N_{1}}_{i_{k - 2} = 0}^{N_{k - 2}} z_{i_{1} i_{k - 1}}^{k - 1} = 1 i_{k - 1} = 1, N_{k - 1} z_{i_{1} i_{k} {0, 1}}^{k} {0, 1} for all i_{1} l_{k - 1} & [3.4] \end{matrix}

0116 Assignment Problem 3.1 and Problem Formulation 3.4 differ only in the dimension M vs. M1, respectively. An optimal solution to 3.4 is also an optimal solution to equation 3.2. This relationship is the basis for an iterative sequence of reductions indicated by steps 320-332 through 330-332 and 200-204 in which the penalized cost function problem is reduced to a two dimensional problem. As these formula will be used to describe the processing at all levels, the lowercase k is used except where specific reference to the top level is needed. In step 206, the 2-dimensional penalized cost function is solved directly by the prior art Reverse Auction technique. Each execution of 206 produces two results, the set of z2 values that minimize the problem and the cost that results v2 when these z values are substituted into the objective function 3.4(a). 0117 In step 208, according to the prior art, solution z2 values are substituted into the 2-dimensional derivative of the 2 u3 0118 should be adjusted so as to perform the hill climbing function. As was previously described the objective is to produce a sequence of u13 values which ends when the up3 value in the domain of the peak of the 0119 0120 uik 0121 to the peak of 2 0122 flow moves to step 214 which will attempt to recover the 3-dimensional solution as previously described. When further adjustment is required then the flow progresses to step 212 and the new values of uk are computed. At the 2-dimensional level the method of the prior art could be used for the hill climbing procedure. However, it is not practical to use this prior art hill climbing technique to determine the updated Lagrangian Coefficients uk or the Max on the next (or any) higher order 0123 0124 uk 0125 values which result in z values that are closer to the proper z values for the highest k-dimension. This hill climbing process (which is different than that used in the prior art of FIG. 4 for recovering only the three dimensional solution) is used iteratively at all lower dimensions k of the M-dimensional problem (including the 3-dimensional level where it replaces prior art) even when k is much larger than three. FIG. 6 helps to explain this new hill climbing technique and illustrates the original k-dimensional cost function vk of Problem Formulation 3.1. However, the actual k-dimensional cost surface vk defined by 3.1(a) comprises scalar values at each point described by k-dimensional vectors and as such can not be drawn. Nevertheless, for illustration purposes only, FIG. 6 ignores the reality of ordering vectors and illustrates a concave function vk (zk) to represent Equation 3.1. The vk surface is illustrated as being smooth to simplify the explanation although actually it can be imagined to be terraced. The goal of the assignment problem is to find the values of {overscore (z)}k; these values minimize the k-dimensional cost function vk. 0126 For purposes of explanation, assume that in FIG. 6, k4 (the procedure is used for all k3). This problem is reduced by two iterations of Lagrangian Relaxation to a 2-dimensional penalized cost function j(k1)i. This cost function, and all other cost functions described below are also non-smooth and continuous but are illustrated in FIG. 6 as smooth for explanation purposes. Solving the j(k1)i problem results in one set of z2 assignments and the value of (k1)i at the point uij2. A series of functions j(k1)i, . . . , l(k1)i each generated from a different u3 are shown. The particular series illustrates the process of locating the peak of (k1)i. The 2-dimensional penalized cost functions j(k1)i, . . . ,l(k1)i can be solved directly. Each such solution provides the information required to calculate the next u3 value. Each iteration of the hill climbing improves the selection of u3 values, i.e., yields 2 solution is closer to those at the solution of the 3 The result of solving l(k1)i is values that are on both 0127 (k1)1 and k k 0128 k 0129 and 3.4 were solved at all possible values of uk the function k k k 0130 than the vk surface except at the peak as described in Equation 3.5 and its maximum occurs where the vk surface is minimum. Because the k 0131 the k k 0132 vk surface. The k 0133 minimization problem described by 3.1(a). Let {overscore (z)}k be the unknown solution to Problem Formulation 3.1 and note that:

k(uk)vk({overscore (z)}k)vk1(zk)3.5

0134 Consequently, the zk values at the peak of k greatest lower bound on the cost of the relaxed problem), can be substituted into the k-dimensional penalized cost function to determine the proper assignments. Consequently, the present invention attempts to find the maximum on the k surface. However, it is not possible to use the prior art hill climbing to hill climb to the peak of k definition of k functions. As the solution of lk in that higher order u values are not yet optimized, its value can be larger than the true peak of k lower bound on vk and it can not be used to recover the required solution. 0135 Instead, the present invention defines all auxiliary function k 0136 dimensional penalized cost function problem, lower order zk values and uk values determined previously by the reduction process. The k k and its gradient is used for hill climbing to its peak. The zk values at the peak of the k into Problem Formulation 3.2 to determine the proper assignments. To define the k 0137 explicitly makes the function k uk 0138 order sets of Lagrangian Coefficients with the expanded notation: k uk uk1 uk is defined recursively, using the k

\begin{matrix} _{3} (u^{3}) = v_{2} +_{i_{3} = 0}^{N_{3}} u_{i_{3}}^{3} =_{3} (u^{3}; u^{4},, u^{K}) & [3.6] (a) \end{matrix}

0139 where v2 is the solution value for the most recent 2-dimensional penalized cost function problem. For k>3 14

\begin{matrix} _{k} (u^{3},, u^{k - 1}; u^{k}) = {\begin{matrix} _{k} (u^{k}; u^{k + 1},, u^{k}) & If known \\ _{k - 1} (u^{3},, u^{k - 2}; u^{k - 1}) +_{i_{k} = 0}^{N_{k}} u_{i_{k}}^{k} & otherwise \end{matrix} & [3.6] (b) \end{matrix}

0140 From the definition of k 0141 compared with Problem Formulation 3.2): 15

_{k} (u^{k}; u^{k + 1},, u^{M}) = v_{k - 1} (z^{k - 1}) +_{i_{k} = 0}^{N_{k}} u_{i_{k}}^{k}

0142 it follows that: 16

_{3} (u^{3}) = v_{2} +_{i_{3} = 0}^{N_{3}} u_{i_{3}}^{3} =_{3} (u^{3}; u^{4},, u^{M}) v_{3}^{3} (z^{3})

0143 and with that Equation 3.5 is extended to:

k(u3, . . . , uk1; uk)k(uk; uk1, . . . uM)vk({overscore (u)}k)vk(uk)3.7

0144 This relationship means that either k k hill climbing to update the Lagrangian Coefficients u. k the preferred choice, however it is only available when the solution to Problem Formulation 3.2 is a feasible solution to Problem Formulation 3.1 (as in hill climbing from the second to third dimension which is why prior art worked). For simplicity in implementation, the function k 0145 that it equals k 0146 therefore always used, even for hill climbing from the second dimension. The use of the 0147 u 0148 u 0149 k 0150 climbing/peak detection are to determine the gradient of the k 0151 and then move up the k 0152 increasing portion of the gradient. As shown in FIG. 6, any upward step on the k 0153 the kj uk 0154 is closer to the ideal set of uk values which correspond to the minimum of the vk function. While it is possible to iteratively step up this k 0155 and then determine if the peak has been reached, the present invention optimizes this process by determining the single step size from the starting point at the minimum of ki that will jump to the peak and then calculating the uk values at the peak. Once the f. u s u at the peak at k determined, then the uk values can be substituted into Problem Formulation 3.2 to determine the proper assignment. (However, in a more realistic example, where k is much greater than three, then the uk values at the peak of the along with the lower order uk values and those assigned and yielded by the reduction steps are used to define the next higher level 0156 k k uk 0157 k 0158 determine the gradient of each 0159 0160 bundle) are determined at several points on the k in the region of the starting point (i.e., minimum of ki) and then averaged. Statistically, the averaged result should point toward the peak of the k Subgradient Algorithm (Wolfe75, Wolfe79) for minimization was previously known in another environment to determine a gradient of a non-smooth surface using multiple subgradients and can be used with modification in the present invention. Wolfe's algorithm is further described in A Method of Conjugate Subgradients for Minimizing Nondifferentiable Functions page 147-173 published by Mathematical Programming Study 3 in 1975 and Finding the Nearest Point in a Polytope page 128-149 published by Mathematical Programming Study 11 in 1976. The modification to Wolfe's algorithm uses the information generated for k uk3 ukk2 ukk1 for calculating the subgradients. The definition of a subgradient v of k uk3 ukk 0161 subdifferential set defined as:

k(u)RNk1k(uk3, . . . , ukk1; u)(k(uk3, . . . , uk1; uk)vT(uuk) uRNk1

0162 (where vT is the transpose of v) 0163 Next, a subgradient vector is determined from this function. If zk is the solution of Problem Formulation 3.2, then differentiating k u3 uk1 uk uikk evaluating the result with respect to the current selection matrix zk yields a subgradient vector: 17

g = (0, (1 -_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k}} z_{i_{1}}^{k}, i_{k} i_{k} = 1,, N_{k}))

0164 The iterative nature of the solution process at each dimension yields a set of such subgradients. Except for a situation described below where the resultant averaged gradient does not point toward the peak, the most recent set of such subgradients are saved and used as the bundle for the peak finding process for this dimension. For example, at the k level there is a bundle of subgradients of the k the minimum of the ki surface determined as a result of solving Problem Formulation 3.1 at all lower levels. This bundle can be averaged to approximate the gradient. Alternately, the previous bundle can be discarded so as to use the new value to initiate a new bundle. This choice provides a way to adjust the process to differing classes of problems, i.e., when data is being derived from two sensors and the relaxation proceeds from data derived from one sensor to the other then the prior relaxation data for the first sensor could be detrimental to performance on the second sensors data. 0165 I.2.3. Step Size and Termination Criterion 0166 After the average gradient of the k determined, the next step is to determine a single step that will jump to the peak of the k 0167 is to first specify an arbitrary step size, and then calculate the value of k 0168 gradient. If the k 0169 this probably means that the step has not yet reached the peak. Consequently, the step size is doubled and a new k value determined and compared to the previous one. This process is repeated until the new k 0170 previous one. At that time, the step has gone too far and crossed over the peak. Consequently, the last doubling is rolled-back and that step size is used for the iteration. If the initial estimated k 0171 step size is decreased by 50%. If this still results in a smaller k 0172 previous step size is decreased by 25%. The following is a more detailed description of this process. 0173 With a suitable bundle of subgradients determined as just described. Wolfe's algorithm can be used to determine the effective subgradient (d) and the Upgraded value uu1k. From the previous iteration, or from an initial condition, there exists a step length value (t). The value,

uujktd

0174 is calculated as an estimate of uj1k. To determine if the current step size is valid the we evaluate k u3 uk2 u If the result represents an improvement then we double the step size. Otherwise we halve the step size. In either case a new u is calculated. The doubling or halving continues until the step becomes too large to improve the result, or until it becomes small enough to not degrade the result. The resulting suitable step size is saved with d as part of the subgradient bundle. The last acceptable u is assigned to uj1k. 0175 Three distinct criterion are used to determine when ujk is close enough to {overscore (u)}k: 0176 1. The Wolfe's algorithm criterion of d0 given that the test has been repeated with the bundle containing only the most recent subgradient. 0177 2. The difference between the lower bound k uk upper bound vk (zk, uk) being less than a preset relative threshold. (Six percent was found to be an effective threshold for radar tracking problems.) 0178 3. An iteration count being exceeded. 0179 The use of limits on the iteration are particularly effective for iterations at the level 3 through n-1 levels, as these iterations will be repeated so as to resolve higher order coefficient sets. With limited iterations the process is in general robust enough to improve the estimate of upper order Lagrangian Coefficients. By limiting the iteration counts then the total processing time for the algorithm becomes deterministic. That characteristic means the process can be effective for real time problems such as radar, where the results of the last scan of the environment must be processed prior to the next scan being received. 0180 I.2.4. Solution Recovery 0181 The following process determines if the uk values at what is believed to be the peak of the k 0182 approximate the uk values at the peak of the corresponding k function. This is done by determining if the corresponding penalized cost function is minimized. Thus, the uk values at the peak of the k 0183 Formulation 3.2 to determine a set of z assignments for k1 points. During the foregoing reduction process, Problem Formulation 3.4 yielded a tentative list of k selected z points that can be described by their indices as: 18

{(i_{l}^{j} i_{k = 1}^{j})}_{j = 1}^{N_{0}}

0184 where N0 is the number of cost elements selected into the solution. One possible solution of Problem Formulation 3.1 is the solution of Problem Formulation 3.2 which is described as 19

{(i_{l}^{j} i_{k - 1}^{j} m_{i_{1} i_{k - 1}}^{k})}_{j = 1}^{N_{0}}

0185 with mi1k. . . ik1 as it was defined in Problem Formulation 3.3. If this solution satisfies the k-th constraint set then it is the optimal solution. 0186 However, if the solution for Problem Formulation 3.2 is not feasible (decision 355), then the following adjustment process determines if a solution exists which satisfies the k-th constraint while retaining the assignments made in solving Problem Formulation 3.4. To do this, a 2-dimensional cost matrix is defined based upon all observations from the k-th set which could be used to extend the relaxed solution.

hjlcilk, . . . , ik1, 1 for l0, . . . , Nk

0187 and j0, . . . , N03.80188 If the resulting 2-dimensional assignment problem, 20

\begin{matrix} Minimize:_{j = 0}^{N_{0}}_{l = 0}^{N_{1}} h_{j1} w_{j1} Subject to:_{l = 0}^{N_{k}} w_{j1} = 1 j = 1,, N_{0}_{l = 0}^{N_{k}} w_{j1} = 1 j = 1,, N_{0} w_{j1} {0, 1} j = 0,, N_{0}, l = 0,, N_{k} & [3.9] \end{matrix}

0189 has a feasible solution then the indices of that solution map to the solution of Problem Formulation 3.1 for the k-dimensional problem. The first index in each resultant solution entry is the pointer back to an element of the 21

{(i_{1}^{j} i_{k - 1}^{j} m_{i_{1} i_{k - 1}}^{k})}_{j = 1}^{N_{0}}

0190 list. That element supplies the first k-1 indices of the solution. The second index of the solution to the recovery problem is the kth index of the solution. Together these indexes specify the values of zk that solve Problem Formulation 3.1 at the kth level. 0191 If Problem Formulation 3.9 does not have a feasible solution then the value of uik which was thought to represent uPk is not representative of the actual peak and further iteration at the kth level is required. This decision represent the branch path from step 214 and equivalent steps. 0192 I.3. Partitioning 0193 A partitioning process is used to divide the cost matrix that results from the Scoring step 154 into as many independent problems as possible prior to beginning the relaxation solution. The partitioning process is included with the Problem Formulation Step 310. The result of partitioning is a set of problems to be solved, i.e., there will be p1 problems that consist of a single hypothesis, p2 problems that consist of two hypothesis, etc. Each such problem is a group in that one or more observations or tracks are shared between members of the group. 0194 In partitioning to groups no consideration is given to the actual cost values. The analysis depends strictly on the basis of two or more cost elements sharing the same specific axis of the cost matrix. In a 2-dimensional case two cost elements must be in the same group if they share a row or a column. If the two elements are in the same row, then each other cost clement that is also in the same row, as well as any cost elements that are in columns occupied by members of the row must be included in the group. The process continues recursively. In literature it is referred to as Constructing a Spanning Forest. The k-dimensional case is analogous to the 2-dimensional case. The specific method we have incorporated is a depth first search, presented by Aho, Hopecraft, and Ullman, in Design and Analysis of Computer Algorithms, section 5.2, published by Addison-Westley, Mass. 0195 The result of partitioning at level M is the set of problems described as Piji1, pj and j1, . . . ,N, where N is the total number of hypothesis. The problems labeled Pili1, . . . , p1 are the cases where their is only one choice for the next observation at each scan and that observation could be used for no other track, i.e., it is a single isolated track. The hypothesis must be in included in the solution set and no further processing is required. 0196 As hypothesis are constructed the first clement is used to refer to the track id. Any problem in the partitioned set which does not have shared costs in the first scan represent a case where a track could be extended in several ways but none of the ways share an observation with any other track. The required solution hypothesis for this case is the particular hypothesis with the maximum likelihood. For this case all likelihoods are determined as was described in Scoring and the maximum is selected. 0197 In addition to partitioning at the M level, partitioning is applied to each subsequent level M1, . . . , 2. For each problem that was not eliminated by either by the prior selection partitioning is repeated, ignoring observations that are shared in the last set of observations. Partitioning recognizes that relaxation will eliminate the last constraint set and thus partitioning is feasible for the lower level problems that will result from relaxation. This process is repeated for all levels down to k3. The full set of partitionings can be performed in the Problem Formulation Step 310, prior to initiating the actual relaxation steps. The actual 2-dimensional solver used in step 206 includes an equivalent process so no advantage would be gained by partitioning at the k2 level. 0198 There are two possible solution methods for the remaining problems. Branch and Bound as was previously described, or assigned to both tracks. The fruitfulness of partitioning increases for lower and lower levels of relaxation. 0199 The following is a more detailed description of the partitioning method. Its application at all but the M level depends upon the relaxation process described in this invention. The recursive partitioning is therefore a distinct part of this invention. The advantage of this method is greatly enhanced by the sparse cost matrix resulting from tracking problems. However the sparse nature of the problem requires special storage and search techniques. 0200 A hypothesis is the combination of the cost element cn the selection variable zn and all observations that made up the potential track extension. It can be written as, hncn, zn, onkokik1, . . . , Mi1, . . . , Nk, i.e., cost, selection variable and observations in the hypothesis. n1, . . . , N where N is the total number of hypotheses in the problem. While the cost and assignment matrices were previously referenced, these matrices are not effective storage mechanisms for tracking applications. Instead the list of all hypothesis and sets of lists for each dimension that reference the relaxation method that this invention describes. If any of the partitions have 5-10 possible tracks and less than 50 to 20 hypotheses, then the prior art Branch and Bound algorithm generally executes faster than does the relaxation due to its reduced level of startup overhead. The Branch and Bound algorithm is executed against all remaining M level problems that satisfy the size constraint. For the remainder the Relaxation algorithm is used. The scheduling done in Problem Formulation allows each Problem Formulation 3.2 cost matrix resulting from the first step of relaxation to be partitioned. The resulting partitions can be solved by any of the four methods, isolated track direct inclusion, isolated track tree evaluation, small group Branch and Bound or an additional stage of relaxation as has been fully described. 0201 The partitioning after each level of Lagrangian Relaxation is effective because when a problem is relaxed, the constraint that requires each point to be assigned to only one track is eliminated (for one image at a time). Therefore, two tracks previously linked by contending for one point will be unlinked by the relaxation which permits the point to be the hypothesis set are stored. The hypothetical set in list form is: 22

{h_{n}}_{n = 1}^{n = N}

0202 For each dimension k1 . . . M there exists a set of lists, with each list element being a pointer to a particular hypothesis: 23

L_{k_{i}} = {p_{k_{j}} | k_{j} = 1,, N_{k_{i}}}_{i = 1, k = 1}^{i = N_{k}, k = M}

0203 where Nki is number of hypothesis containing the i-th observation from scan k. This structure is a multiply linked list in that any observation is associated with a set of pointer to all hypothesis it participates in, and any hypothesis has a set of pointers to all observations that formed it. (These pointers can be implemented as true pointers or indices depending upon the particular programming language utilized.) 0204 Given this description of the matrix storage technique then the partitioning technique is as follows: Mark the first list Lki and follow out all pointers in that list to the indicated hypothesis hPki for i1, . . . , Nki. Mark all located hypothesis, and for each follow pointers back the particular Lok for k1, . . . M. Those L's if not previously marked get marked and also followed out to hypothesis elements and back to L's. When all such L's or h's being located are marked, then an isolated sub-problem has been identified. All marked elements can be removed from the lists and stored as a unique problem to be solved. The partitioning problem then continues by again starting at the first residual L set. When none remain, the original problem is completely partitioned. 0205 Isolated problems can result from one source track having multiple possible extensions or from a set of source tracks contending for some observations. Because one of the indices of k (in our implementation it is k1) indicate the source track then it is possible to categorize the two problem types by observing if the isolated problem includes more than one L-list from the index level associated with tracks. 0206 I.4. Subsequent Track Prediction 0207 As noted above, after the points are assigned to the respective tracks, some action is taken such as controlling take-offs and landings to avoid collision, advising airborne aircraft to change course, warning airborne aircraft of an adjacent aircraft in a commercial environment, or aiming and firing an anti-aircraft (or anti-missile) missile, rocket or projectile, or taking evasive action in a military environment. Also, the tracking can be used to position a robot to work on an object. For some or all of these applications, usually the tracks which have just been identified are extended or extrapolated to predict a subsequent position of the aircraft or other object. The extrapolation can be done in a variety of prior art ways; for example, straight line extensions of the most likely track extension hypothesis, parametric quadratic extension of the x versus time, y versus time, etc. functions that are the result of the filtering process described earlier, least square path fitting or Kalman filtering of just the selected hypothesis. 0208 The process of gating, filtering, and gate generation as they were previously described require that a curve be fit through observations such that fit of observations to the likely path can be scored and further so that the hypothetical target future location can be calculated for the next stage of gating. In the implementation existing, quadratic functions have been fit through the measurements that occur within the window. A set of quadratics is used, one per sensor measurement. To calculate intercept locations these quadratics can be converted directly into path functions. Intercept times are then calculated by prior methods based upon simultaneous solution of path equations. 0209 The use of the fitted quadratics is not as precise as more conventional filters like the Kalman Filter. They are however much faster and sufficiently accurate for the relative scores required for the Assignment problem. When better location prediction is required then the assignment problem is executed to select the solution hypothesis and based upon the observations in those hypothesis the more extensive Kalman filter is executed. The result is tremendous computation savings when compared with the Kalman Filter being run on all hypothetical tracks. 0210 Based on the foregoing, apparatus and methods have been disclosed for tracking objects. However, numerous modifications and substitutions can be made without deviating from the scope of the present invention. For example, the foregoing 0211 uk values are used to eliminate the higher than approximation characteristic of the k 0212 0213 k 0214 be as efficient it seems likely that the method would converge. Therefore, the invention has been disclosed by way of example and not limitation, and reference should be made to the following claims to determine the scope of the present invention. 0215 II. AN ALTERNATIVE EMBODIMENT OF THE MULTI-DIMENSIONAL ASSIGNMENT SOLVING PROCESS 0216 The following description provides an alternative second embodiment of the multi-dimensional assignment solving process of section I.1. However, in order to clearly describe the present alternative embodiment, further discussion is first presented regarding the data assignment problem of partitioning measurements into tracks and false alarms and, in particular, the representation of multi-dimensional assignment problems. 0217 II.1. Formulation of the Assignment Problem 0218 The goal of this section is to explain the formulation of the data association problems, and more particularly multi-dimensional assignment problems, that govern large classes of problems in centralized or hybrid centralized-sensor level multisensor/multitarget tracking. The presentation is brief; technical details are presented for both track initiation and maintenance in A. B. Poore, Multi-dimensional assignment formulation of data association problems arising from multitarget tracking and multisensor data fusion, Computational Optimization and Applications, 3 (1994), pp. 27-57, for nonmaneuvering targets and in Ingemar J. Cox, Pierre Hansen, and Beta Julesz, eds., Partitioning Data Sets, DIMACS Series in Discrete Mathematics and Theoretical Computer Science. American Mathematical Society, Providence, R.I., v. 19, Feb. 1995, pp. 169-198 for maneuvering targets. Note that the present formulation can also be modified to include target features (e.g., size and type) into the scoring process 154. 0219 The data assignment problems for multisensor and multitarget tracking considered here are generally posed as that of maximizing the posterior probability of the surveillance region (given the data) according to 24

\begin{matrix} Maximize {\frac{P (= | Z^{M})}{P (=^{0} | Z^{M})} |^{*}} & [5.1] \end{matrix}

0220 where ZM represents M data sets, of the data (and thus induces a partition of the data), * the finite collection of all such partitions,

* 0

0221 P(ZM) is the posterior probability of a partition true given the data ZM. The term partition is defined below. 0222 Consider M observation sets Z(k) (k1, . . . ,N) each of Nk observations 25

{z_{i_{k}}^{k}}_{i_{k} = 1}^{N_{k}}

0223 and let ZM denote the cumulative data set defined by 26

\begin{matrix} Z (k) = {z_{i_{k}}^{k}}_{i_{k} = 1}^{N_{k}} and Z^{M} = {Z (1),, Z (M)}, & [5.2] \end{matrix}

0224 respectively. In multisensor data fusion and multitarget tracking the data sets Z(k) may represent different classes of objects, and each data set can arise from different sensors. For track initiation the objects are measurements that must be partitioned into tracks and false alarms. In the formulation of track extensions a moving window over time of observations sets is used. The observation sets will be measurements which are: (a) assigned to existing tracks, (b) designated as false measurements, or (c) used for initiating tracks. However, note that alternative data objects instead of observation sets may also be fused such as in sensor level tracking wherein each sensor forms tracks from its own measurements and then the tracks from the sensors are fused in a central location. Note that, as one skilled in the art will appreciate, both embodiments of the present invention may be used for this type of data fusion. 0225 The data assignment problem considered presently is represented as a case of set partitioning defined in the following way. First, for notational convenience in representing tracks, a zero index is added to each of the index sets in 5.2, a gap filler z0k is added to each of the data sets Z(k) in 5.2, track of data is defined as (zi11, . . . , ziNN) where ik and zikk can now assume the values of 0 and z0k, respectively. A partition of the data will refer to a collection of tracks of data or track extensions wherein each observation occurs exactly once in one of the tracks of data and such that all data is used up. Note that the occurrence of the gap filler is unrestricted. The gap filler z0k serves several purposes in the representation of missing data, false observations, initiating tracks, and terminating tracks. Note that a reference partition may be defined which is a partition in which all observations are declared to be false. 0226 Next under appropriate independence assumptions it can be shown that: 27

\begin{matrix} \frac{P (= | Z^{M})}{P (=^{0} | Z^{M})} L_{} \underset{(i_{1} i_{N})}{} L_{i_{1} i_{M}}, & [5.3] \end{matrix}

0227 where Lilk. . . iM is a likelihood ratio containing probabilities for detection, maneuvers, and termination as well as probability density functions for measurement errors, track initiation and termination. Then if ci1k. . . iNlnLi1k, . . . iM, 28

\begin{matrix} - \ln \frac{P (| Z^{M})}{P (^{0} | Z^{M})} = \underset{(i_{1},, i_{M})}{} c_{i_{1} i_{N}} . & [5.4] \end{matrix}

0228 Using 5.3 and the zero-one variable zi1k . . . iN1 if (i1, . . . , iM) 29

\begin{matrix} _{i_{1} = 0}^{N_{1}}_{i_{M} = 0}^{N_{M}} c_{i_{1} i_{M}} z_{i_{1} i_{M}}_{i_{2} = 0}^{N_{2}}_{i_{M} = 0}^{N_{M}} z_{i_{1} i_{M}} = 1, l_{1} (c)_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}}_{i_{k + 1} = 0}^{N_{k + 1}}_{i_{M} = 0}^{N_{M}} z_{i_{1} i_{M}} = 1, for i_{k} = 1,, N_{k} and k = 2,, M - 1, (d)_{i_{1} = 0}^{N_{1}}_{i_{N - 1} = 0}^{N_{M - 1}} z_{i_{1} i_{M}} = 1, i_{M} = 1,, N_{M} (e) z_{i_{1} i_{M}} {0, 1} for all i_{1},, i_{M} & [5.5] \end{matrix}

0229 where c0 . . . 0 is arbitrarily defined to be zero. Here, each group of sums in the constraints 5.5 (b)-5.5(e) represents the fact that each non-gap filler observation occurs exactly once in a track of data. One can modify this formulation to include multiassignments of one, some, or all the actual observations. The assignment problem 5.5 is changed accordingly. For example, if zikk is to be assigned no more than, exactly, or no less than nlk k times, then the 1 in the appropriate one of the constraints 5.5 (b)-5.5(d) is changed to , , nikk, respectively. 0230 Expressions for the likelihood ratios Li1k. . . iM such as 5.5 are well known. In particular, discussions of these ratios may be found in: A. B. Poore, Multi-dimensional assignment formulation of data association problems arising from multitarget tracking and multisensor data fusion, 0231 Computational Optimization and Applications, 3 (1994), pp. 27-57; and Ingemar J. Cox, Pierre Hansen, and Beta Julesz, eds., Partitioning Data Sets, DIMACS Series in Discrete Mathematics and Theoretical Computer Science. American Mathematical Society, Providence, R.I., v. 19, 1995, pp. 169-198. Furthermore, the likelihood ratios are easily modified to include target features and to account for different sensor types. Also note that in track initiation, the M observation sets provide observations from M sensors, possibly all the same. Additionally note that for track maintenance, a sliding window of M observation sets and one data set containing established tracks may be used. In this latter case, the formulation is the same as above except that the dimension of the assignment problem is now M1. 0232 II.2. Overview of the New Lagrangian Relaxation Algorithms 0233 Having formulated an M-dimensional assignment problem 5.5, we now turn to a description of the Lagrangian relaxation algorithm. Subsection II.2.1 below presents many of the relaxation properties associated with the relaxation of an n-dimensional assignment problem to an m-dimensional one via a Lagrangian relaxation of n-m sets of constraints wherein m<nM and preferably in the present invention embodiment nm>1. Although any nm constraint sets can be relaxed, the description here is based on relaxing the last n-m sets of constraints and keeping the first m sets. Given either an optimal or suboptimal solution of the relaxed problem, a technique for recovering a feasible solution of the n-dimensional problem is presented in subsection II.2.2 below and an overview of the Lagrangian relaxation algorithm is given in subsection II.2.3. 0234 The following notation will be used throughout the remainder of the work. Let M be an integer such that M3 and let n3, . . . ,M. The n-dimensional assignment problem is 30

\begin{matrix} (a) Minimize : v_{n} (z) =_{i_{1} = 0}^{N_{1}}_{i_{n} = 0}^{N_{m}} c_{i_{1} i_{n}}^{n} z_{i_{1} i_{n}}^{n} (b) Subject to :_{i_{2} = 0}^{N_{2}}_{i_{n} = 0}^{N_{m}} z_{i_{1} i_{n}}^{n} = 1, i_{1} = 1,, N_{1}, (c)_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}}_{i_{k + 1} = 0}^{N_{k - 1}}_{i_{n} = 0}^{N_{m}} z_{i_{1} i_{n}}^{n} = 1, for i_{k} = 1,, N_{k} and k = 2,, n - 1, (d)_{i_{1} = 0}^{N_{1}}_{i_{n - 1} = 0}^{N_{m - 1}} z_{i_{1} i_{n}}^{n} = 1, i_{n} = 1,, N_{m}, (e) z_{i_{1} i_{n}}^{n} {0, 1} for all i_{1},, i_{n} & [6.1] \end{matrix}

0235 To ensure that a feasible solution of 6.1 always exists, all variables with exactly one nonzero index (i.e., variables of the form z0 . . . 0ik0 . . . 0n for ik0) are assumed free to be assigned and the corresponding cost coefficients are well-defined. 0236 II.2.1. The Lagrangian Relaxed Assignment Problem 0237 The n-dimensional assignment problem 6.1 has n sets of constraints. A (Nk1)-dimensional multiplier vector associated with the k-th constraint set will be denoted by uk(u0k, u1k, . . . , uMkk) with u0k0 and k1, . . . , n. The n-dimensional assignment problem 6.1 is relaxed to an m-dimensional assignment problem by incorporating n-m of the n sets of constraints into the objective function 6.1 (a). Although any constraint set can be relaxed, the description of the relaxation procedure for 6.1 will be based on the relaxation of the last n-m sets of constraints. The relaxed problem is 31

\begin{matrix} \begin{matrix} _{m n}^{m + 1 n} Minimize_{m n}^{n M + 1 n} \\ _{i_{1} = 0}^{N_{1}}_{i_{n} = 0}^{N_{m}} c_{i_{1} i_{n}}^{n} z_{i_{1} i_{n}}^{n} + \\ _{k = m + 1}^{n}_{i_{k} = 0}^{N_{k}} u_{i_{k}}^{k}_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}}_{i_{k + 1} = 0}^{N_{k + 1}}_{i_{n} = 0}^{N_{m}} z_{i_{1} i_{n}}^{n} - 1 \\ _{i_{1} = 0}^{N_{1}}_{i_{n} = 0}^{N_{m}} [c_{i_{1} i_{n}}^{n} +_{k = m + 1}^{n} u_{i_{k}}^{k}] z_{i_{1} i_{n}}^{n} - \\ _{k = m + 1}^{n}_{i_{k} = 0}^{N_{k}} u_{i_{k}}^{k} \end{matrix} & [6.2] \\ Subject to :_{i_{2} = 0}^{N_{2}}_{i_{n} = 0}^{N_{n}} z_{i_{1}}^{n} i_{n} = 1, i_{1} = 1,, N_{1},_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}}_{i_{k + 1} = 0}^{N_{k + 1}}_{i_{n} = 0}^{N_{m}} z_{i_{1} i_{n}}^{n} = 1, for i_{k} = 1,, N_{k} and k = 2,, m, z_{i_{1} i_{n}}^{n} {0, 1} for all i_{1},, i_{n} . \end{matrix}

0238 wherein we have added the multiplier u0k0 for notational convenience. Thus, the multiplier ukRNk1 with u0k0 is fixed. 0239 An optimal (or suboptimal) solution of 6.2 can be constructed from that of an m-dimensional assignment problem. To show this, define for each (il, . . . , im) an index (jm1, . . . , jn) (jm1(i1, . . . , im), . . . , jn(il, . . . , im)) and a new cost function ci1m. . . im by 32

\begin{matrix} (j_{m + 1},, j_{n}) = \arg \min {c_{i_{1} i_{n}}^{n} +_{k = m + 1}^{n} u_{1 k}^{k} | i_{k} = 0,, N_{k} and k = m + 1,, n} c_{i_{1} i_{m}}^{m} = c_{i_{1} i_{m} j_{m + 1} j_{n}}^{n} +_{k = m + 1}^{n} u_{j_{k}}^{k} for (i_{1},, i_{m}) (0,, 0) c_{0 0}^{m} =_{i_{m + 1} = 0}^{N_{m + 1}}_{i_{n} = 0}^{N_{m}} Minimum {0, c_{0 0 i_{m + 1} i_{n}}^{n} +_{k = m + 1}^{n} u_{j_{k}}^{k}} & [6.3] \end{matrix}

0240 (If (jm1, . . . , jn) is not unique, choose the smallest such jm1, amongst those (nm)-tuples with the same jm1 choose the smallest jm2, etc., so that (jm1, . . . , jn) is uniquely defined.) Using the cost coefficients defined in this way, the following m-dimensional assignment problem is obtained: 33

\begin{matrix} {\hat{}}_{m n}^{m + 1 n}_{m n}^{m m + 1 n} v_{m} (z^{m})_{i_{1} = 0}^{N_{1}}_{i_{m} = 0}^{N_{m}} c_{i_{1} i_{m}}^{m} z_{i_{1} i_{m}}^{m} Subject to :_{i_{2} = 0}^{N_{2}}_{i_{m} = 0}^{N_{m}} z_{i_{1} i_{m}}^{m} = 1, i_{1} = 1,, N_{1},_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}}_{i_{k + 1} = 0}^{N_{k + 1}}_{i_{m} = 0}^{N_{m}} z_{i_{1} i_{m}}^{m} = 1, for i_{k} = 1,, N_{k} and k = 2,, m - 1,_{i_{1} = 0}^{N_{1}}_{i_{m - 1} = 0}^{N_{n - 1}} z_{i_{1} i_{m} = 1}^{m}, i_{m} = 1, N_{m} z_{i_{1} i_{m}}^{m} {0, 1} for all i_{1},, i_{m} . & [6.4] \end{matrix}

0241 As an aside, observe that any feasible solution zn of 6.1 yields a feasible solution zm of 6.4 via the construction 34

z_{i_{1} i_{m}}^{m} = {\begin{matrix} 1 & if z_{i_{1} i_{n - 1} i_{n}}^{n} = 1 for some (i_{m + 1},, i_{n}) \\ 0 & otherwise . \end{matrix}

0242 Thus, the m-dimensional assignment problem 6.4 has at least as many feasible solutions of the constraints as the original problem 6.1. 0243 The following Fact A.2 has been shown to be true. It states that an optimal solution of Problem Formulation 6.2 can be computed from that of Problem Formulation 6.4. Furthermore, the converse to this fact is provided in Fact A.3. Moreover, if the solution of either of these two problems 6.2 or 6.4 is -optimal (i.e., the objective function associated with the suboptimal solution is within of the objective function associated with the optimal solution), then so is the other. 0244 Fact A.2. Let wm be a feasible solution to problem 6.4 and define wn by 35

\begin{matrix} w_{i_{1} i_{n}}^{n} = w_{i1 im}^{m} if (i_{m + 1}, i_{n}) = (j_{m + 1}, j_{n}) and (i_{1},, i_{m}) (0,, 0) w_{i_{1} i_{n} = 0}^{n} if (i_{m + 1}, i_{n}) (j_{m + 1}, j_{n}) and (i_{1},, i_{m}) (0,, 0) w_{0 0 i_{m + 1} i_{n} = 1}^{n} if c_{0 0 im + 1 in}^{n} +_{k = m + 1}^{n} u_{i_{k}}^{k} 0 w_{0 0 i_{m + 1} i_{n} = 0}^{n} if c_{0 0 im + 1 in}^{n} +_{k = m + 1}^{n} u_{i_{k}}^{k} > 0 & [6.5] \end{matrix}

0245 Then wn is a feasible solution of the Lagrangian relaxed problem 6.2 and mnn m1 n {circumflex over ()}mn(wm; um1, . . . , un) km1nikMk0uikk0246 If, in addition, wm is optimal for 6.4, then wn is an optimal solution of 6.2 and mnm1 n mn(um1, . . . , un)km1nikMk0uikk0247 With the exception of one equality being converted to an inequality, the following Fact is a converse of Fact A.2 and has also been shown to be true. 0248 Fact A.3. Let wn be a feasible solution to problem 6.2 and define wm by 36

\begin{matrix} w_{i_{1} i_{m}}^{m} =_{i_{m + 1} = 0}^{N_{m + 1}}_{i_{n} = 0}^{N_{m}} w_{i1 in}^{n} for (i_{1},, i_{m}) (0,, 0) w_{0 0}^{m} = 0 if (i_{1},, i_{m}) = (0,, 0) and c_{0 0 i_{m + 1} i_{n}}^{n} +_{k = m - 1}^{n} u_{i_{k}}^{k} > 0 for all (i_{m + 1},, i_{n}) w_{0 0}^{m} = 1 if (i_{1},, i_{m}) = (0,, 0) and c_{0 0 i_{m + 1} i_{n}}^{n} +_{k = m + 1}^{n} u_{i_{k}}^{k} 0 for some (i_{m + 1},, i_{n}) . & [6.6] \end{matrix}

0249 Then wm is a feasible solution of the problem 6.4 and mnn24 {circumflex over ()}mn(wm; um1, . . . , un) km1nikMk0uikk0250 If, in addition, wn is optimal for 6.2, then wm is an optimal solution of 6.4, mnn m1 n 0251 {circumflex over ()}mn(wn; um1, . . . , un)km1nikMk0uikk, and mnm1 n mn(um1, . . . , un)km1nikMk0uikk . 0252 II.2.2. The Recovery Procedure 0253 The next objective is to explain a recovery procedure or method for recovering a solution to the n-dimensional problem of 6.1 from a relaxed problem having potentially substantially fewer dimensions than 6.1. Note that this aspect of the alternative embodiment of the present invention is substantially different from the method disclosed in the first embodiment of the multi-dimensional assignment solving process of section I.1 of this specification. Further, this alternative embodiment provides substantial benefits in terms of computational efficiency and accuracy over the first embodiment, as will be discussed. Thus, given a feasible (optimal or suboptimal) solution wm of 6.4 (or wn if Problem Formulation 6.2 is constructed via Fact A.2), an explanation is provided here regarding how to generate a feasible solution zn of 6.1 which is close to wm in a sense to be specified. We first assume that no variables in 6.1 are preassigned to zero (this assumption will be removed shortly). The difficulty with the solution wn is that it need not satisfy the last n-m sets of constraints in 6.1. Note, however, that if wm is an optimal solution for 6.4 and wn (constructed as in Fact A.2) satisfies the relaxed constraints, then wn is optimal for 6.1. The recovery procedure described here is designed to preserve the 0-1 character of the solution wm of 6.4 as far as possible. That is, if wi1m. . . im0 and i10 for at least one l1, . . . ,m, then the corresponding feasible solution zn of 6.1 is constructed so that zi1n. . . in1 for some (im1, . . . ,in). Note that by this reasoning, variables of the form z0 . . . 0im1n . . . in can be assigned to a value of 1 in the recovery problem only if w0 . . . 0m1. However, variables z0 . . . 0im1n . . . in in will be treated differently in the recovery procedure in that they can be assigned 0 or 1 independent of the value w0 . . . 0m. This increases the feasible set of the recovery problem, leading to a potentially better solution. 0254 Let 37

{(i_{1}^{j},, i_{m}^{j})}_{j = 1}^{N_{0}}

0255 be an enumeration of indices of wm (or the first m indices of wn constructed as in Fact A.2) such that wi1mj. . . imj1 and (i1j, . . . ,imj)(0, . . . , 0). Set (i10, . . . , im0)90, . . . , 0) for j0 and define 0256 cjim1nm1. . . incilnj. . . imjim1. . . in for ik0, . . . , Nk; 0257 km1, . . . , n; j0, . . . ,N0.6.70258 Let Y denote the solution of the (nm1)-dimensional assignment problem: 38

\begin{matrix} Minimize_{j = 0}^{N_{0}}_{i_{m + 1} = 0}^{N_{m + 1}}_{i_{n} = 0}^{N_{m}} c_{{ji}_{m + 1} i_{n}}^{n - m + 1} y_{{ji}_{m + 1} i_{n}} Subject to_{i_{m + 1} = 0}^{N_{m + 1}}_{i_{n} = 0}^{N_{m}} y_{{ji}_{m + 1} i_{n}} = 1, j = 1,, N_{0},_{j = 0}^{N_{0}}_{i_{m + 1} = 0}^{N_{m + 1}}_{i_{k - 1} = 0}^{N_{k - 1}}_{i_{k + 1} = 0}^{N_{k + 1}}_{i_{n} = 0}^{N_{m}} y_{{ji}_{m + 1} i_{n}} = 1, for i_{k} = 1,, N_{k} and k = m + 1,, n - 1,_{j = 0}^{N_{0}}_{i_{m + 1} = 0}^{N_{m + 1}}_{i_{n - 1} = 0}^{N_{m - 1}} y_{{ji}_{m + 1} i_{n}} = 1, i_{n} = 1,, N_{m}, y_{{ji}_{m + 1} i_{n}} {0, 1} for all j, i_{m + 1},, i_{n} . & [6.8] \end{matrix}

0259 The recovered feasible solution zn of 6.1 corresponding to the multiplier set um, . . . , un is then defined by 39

\begin{matrix} z_{i_{1} i_{n}}^{n} = {\begin{matrix} 1 & if (i_{1},, i_{m}) = (i_{1}^{j},, i_{m}^{j}) for some \\ j = 0,, N_{0} and Y_{{ji}_{m + 1}} i_{n} = 1 \\ 0 & otherwise . \end{matrix} & [6.9] \end{matrix}

0260 This recovery procedure is valid as long as all cost coefficients cn are defined and all zero-one variables in zn are free to be assigned. Note that modifications are necessary for sparse problems. If the cost coefficient ci1nj. . . imjim1. . . in is well defined and the zero-one variable zi1nj. . . imjim1. . . in is free to be assigned to zero or one, then define cjim1nm1. . . in ci1nj. . . imjim1. . . in as in 6.7 with zjim1nm1. . . in being free to be assigned. Otherwise, zjim1nm1. . . in is preassigned to zero. To ensure that a feasible solution exists, we now only need ensure that the variables zj0 . . . 0nm1 are free for j0, 1, . . . ,N0. (Recall that all variables of the form z0 . . . ik. . . 0n are free (to be assigned) and all coefficients of the form c0 . . . ik. . . 0n are well defined for k1, . . . ,n.) This is accomplished as follows. If the cost coefficient ci1nji2j . . . imj0 . . . 0 is well defined and zi1nji2j . . . imj 0 . . . 0 is free, then define cj0. . . onm1ci1nji2j . . . imj0 . . . 0 with zj0 . . . 0nm1 being free. 0261 Otherwise, since all variables of the form z0 . . . ik. . . 0n are known feasible and have well-defined costs, put 40

c_{j0 0}^{n - m + 1} =_{k = 1, i_{k}^{j} 0}^{m} c_{0 0 i_{k}^{j} 0 0}^{n} .

0262 II.3. The Multi-Dimensional Lagrangian Relaxation Algorithm for the n-Dimensional Assignment Problem Starting with the M-dimensional assignment problem 6.1, i.e., nM, the algorithm described below is recursive in that the M-dimensional assignment problem is relaxed to an m-dimensional problem by incorporating (nm) sets of constrains into the objective function using Lagrangian relaxation of this set. This problem is maximized with respect to the Lagrange multipliers, and a good suboptimal solution to the original problem is recovered using an (nm1)-dimensional assignment problem. Each of these two (the m-dimensional and the (nm1)-dimensional assignment problems) can be solved in a similar manner. 0263 More precisely, reference is made to FIG. 8 which presents a flowchart of a procedure embodying the multi-dimensional relaxation algorithm, referred to immediately above. This procedure, denoted MULTI_DIM_RELAX in FIG. 8, has three formal parameters, n, PROB_FORMULATION and ASSIGNMT_SOLUTION, which are used to transfer information between recursive instantiations of the procedure. In particular, the parameter, n, is an input parameter supplying the dimension for the multi-dimensional assignment problem (as in 6.1) which is to be solved (i.e., to obtain an optimal or near-optimal solution). The parameter, PROB_FORMULATION, is also an input parameter supplying the data structure(s) used to represent the n-dimensional assignment problem to be solved. Note that PROB_FORMULATION at least provides access to the cost matrix, cn, and the observation sets whose observations are to be assigned. Additionally, the parameter, ASSIGNMT_SOLUTION, is used as an output parameter for returning a solution of a lower dimensioned assignment problem to an instantiation of MULTI_DIM_RELAX which is processing a higher dimensioned assignment problem. 0264 A description of FIG. 8 follows. Assuming MULTI_DIM_RELAX is initially invoked with M as the value for the parameter n and the PROB_FORMULATION having a data structure(s) representing an M-dimensional assignment problem as in 5.5, in step 500 an integer m, 2m<n is chosen such that the constraint sets or dimensions m1, . . . , n are to be relaxed so that an m-dimensional problem formulation as in 6.2 results. In step 504 an initial approximation is provided for u0m1, . . . , u0n. Subsequently, in step 508 the above initial values for um0m1, . . . , u0n are used in an iterative process in determining ({overscore (u)}m1, . . . ,{overscore (u)}n) which maximizes mnm1 n kRMk1 and km1, . . . , n for a feasible solution wn subject to the constraints of Problem Formulation 6.2. Note that by maximizing mnm1 n constraints that were relaxed are being forced to be satisfied and in so doing information is built into a solution of this function for solving the input assignment problem in PROB_FORMULATION. Further note that a non-smooth optimization technique is used here and that a preferred method of determining a maximum in step 508 is the bundle-trust method of Schramm and Zowe as described in H. Schramm and J. Zowe, A version of the bundle idea for minimizing a non-smooth function: Conceptual idea, convergence analysis, numerical results, SIAM Journal on Optimization, 2 (1992), pp. 121-152. This method, along with various other methods for determining the maximum in step 508, are discussed below. 0265 Also note that for m>2, a solution to the optimization problem of step 508 is NP-hard and therefore cannot be solved optimally. That is, there is no known computationally tractable method for guarantying an optimal solution. Thus, there are two possibilities: either (a) allow m to be greater than 2 and use auxiliary functions similar to those disclosed in the first embodiment of the k-dimensional assignment solver 300 in section I to compute a near-optimal solution, or (b) make m2 wherein algorithms such as the forward/reverse auction algorithm of D. P. Bertsekas and D. A. Castaon in their paper, A forward/reverse auction algorithm for asymmetric assignment problems, Computational Optimization and Applications, 1 (1992), pp. 277-298, provides an optimal solution. 0266 If option (a) immediately above is chosen, then the auxiliary functions are used as approximation functions for obtaining at least a near-optimal solution to the optimization problem of step 508. Note that the auxiliary functions used depend on the value of m. Thus, auxiliary functions used when m3 will likely be different from those for m4. But in any case, the optimization procedure is guided by using the merit function, 2n3 n 2-dimensional assignment problem for guiding the maximization process. 0267 Alternatively, if option (b) above is chosen, then two important advantages result: the optimization problem of step 508 can be always solved optimally and without using auxiliary approximation functions. Thus, better solutions to the original M-dimensional assignment problem are likely since there is no guarantee that when the non-smooth optimization techniques are applied to the auxiliary functions the techniques will yield an optimal solution to step 508. Furthermore, it is important to note that without auxiliary functions, the processing in step 508 is both conceptually easier to follow and more efficient. 0268 Subsequently, in step 512 of FIG. 8, the solution ({overscore (u)}m1, . . . , {overscore (u)}n) and wn is used in determining an optimal solution, wm to Problem Formulation 6.4 as generated according to 6.3 and Fact A.3. 0269 In step 516, the solution wn is used in defining the cost matrix cnm1 as in 6.7. Subsequently, if nm12, then the assignment problem 6.8 may be solved straightforwardly using known techniques such as forward/reverse auction algorithms. Following this, in step 528, the solution to the 2-dimensional assignment problem is assigned to the variable ASSIGNMT_SOLUTION and in step 532 ASSIGNMT_SOLUTION is returned to a dimension three level recursion of MULTI_DIM_RELAX for solving a 3-dimensional assignment problem. 0270 Alternatively, if in step 520, nm1>2, then in step 536 the data structure(s) representing a problem formulation as in 6.8 is generated and assigned to the parameter, PROB_FORMULATION. Subsequently, in step 540 a recursive copy of MULTI_DIM_RELAX is invoked to solve the lower dimensional assignment problem represented by PROB_FORMULATION. Upon the completion of step 540, the parameter, ASSIGNMT_SOLUTION, contains the solution to the nm1 dimensional assignment problem. Thus, in step 544, the nm1 solution is used to solve the n-dimensional assignment problem as discussed regarding equations 6.9. Finally, in steps 548 and 552 the solution to the n-dimensional assignment problem is returned to the calling program so that, for example, it may be used in taking one or more actions such as (a) sending a warning to aircraft or sea facility; (b) controlling air traffic; (c) controlling anti-aircraft or anti-missile equipment; (d) taking evasive action; or (e) surveilling an object. 0271 II.3.1. Comments on the Various Procedures Provided by FIG. 8 0272 There are many procedures described by FIG. 8. One such procedure is the first embodiment of the multi-dimensional assignment solving process of section I.1. That is, by specifying mn1 in step 500, a single set of constraints is relaxed in step 508. Thus, one set of constraints incorporated is into the objective function via the Lagrangian problem formulation,resulting in an mn1 dimensional problem. The relaxed problem is subsequently maximized in step 512 with respect to the corresponding Lagrange multipliers and then a feasible solution is reconstructed for the n-dimensional problem using a two-dimensional assignment problem. The second procedure provided by FIG. 8 is a novel approach which is not suggested by the first embodiment of section I.1. In fact, the second procedure is somewhat of a mirror image of the first embodiment in that n2 sets of constraints are simultaneously relaxed, yielding immediately an m2 dimensional problem in step 512. Thus, a feasible solution to the n-dimensional problem is then recovered using a recursively obtained solution to an n1 dimensional problem via step 540. In this case, the function values and subgradients of 2n3 n optimally via a two dimensional assignment problem. The significant advantage here is that there is no need for the merit or auxiliary functions, k embodiment of the multi-dimensional assignment solving process of section I.1 above and also there is no need for the more general merit or auxiliary functions mn subsection II.4.2 below. Further, note that all function values and subgradients used in the non-smooth maximization process are computed exactly (i.e., optimally) in this second procedure. Moreover, problem decomposition is now carried out for the n-dimensional problem; however, decomposition of the n1 dimensional recovery problem (and all lower recovery problems) is performed only after the problem is formulated. 0273 Between these two procedures are a host of different relaxation schemes based on relaxing nm sets of constraints to an m-dimensional problem (2<m<n), but these all have the same difficulties as the procedure for the first embodiment of section I.1 in that the relaxed problem is an NP-hard problem. To resolve this difficulty, we use an auxiliary or merit function mn notation k n1,n m<n1, the recovery procedure is still based on an NP-hard nm1 dimensional assignment problem. Note that the partitioning techniques similar to those discussed in Section I.3 may be used to identify the assignment problem with a layered graph and then to find the disjoint components of this graph. In general, all relaxed problems can be decomposed prior to any non-smooth computations because their structure stays fixed throughout the algorithm of FIG. 8. However, all recovery problems cannot be decomposed until they are formulated, as their structure changes as the solutions to the relaxed problems change. 0274 II.4. Details and Refinements Relating to the Flowchart of FIG. 8 0275 Further explanation is provided here on how various steps of FIG. 8 are solved. Note that the refinements presented here can significantly increase the speed of the relaxation procedure of step 508. 0276 II.4.1 Maximization of the Non-smooth Function mnm1 n 0277 One of the key steps in the Lagrangian relaxation algorithm in section II.3 is the solution of the problem

Maximize mnm1 n kRMk1; km1, . . . , n7.1

0278 where u0k0 for all km1, . . . ,n. The evaluation of mnm1 ncorresponding minimization problem 6.2 Zn varies for each instance of (um1, . . . ,un). The following discussion provides some properties of these functions. 0279 Fact A.4. Let um1, . . . , n be multiplier vectors associated with the (m1)st through the nth set of constraints 6.1, let mn objective function value of the n-dimensional assignment problem in equation 6.1, let zn be any feasible solution of 6.1, and let {overscore (z)}n be an optimal solution of 6.1. Then, mnm1 n um1, . . . , un and

mnm1 n vn({overscore (z)}n)vn(zn)7.2a

0280 Furthermore,

m1nm m1 n mnm1 n 7.2a

0281 for all ukRMk1 with u0k0 and km, . . . , n. 0282 Most of the procedures for non-smooth optimization are based on generalized gradients called subgradients, given by the following definition. 0283 Definition. At u(um1, . . . , un) the set mn subdifferential of mn

mnRMm1ix . . . xRMn1mnmnqT(wu)wRMm11x . . . xRMn17.3

0284 A vector qmn0285 If zn is an optimal solution of 6.2 computed during evaluation of mn mn inn yields the following in-th component of a subgradient g of 41

\begin{matrix} _{m n} g_{0}^{k} = 0 g_{i_{k}}^{k} =_{i_{1} = 0}^{N_{1}}_{i_{k - 1} = 0}^{N_{k - 1}}_{i_{k + 1} = 0}^{N_{k + 1}}_{i_{n} = 0}^{N_{m}} z_{i_{1}}^{n} i_{n}^{- 1} for i_{k} = 1,, N_{k} and k = m + 1,, n . & [7.4] \end{matrix}

0286 If zn is the unique optimal solution of 6.2, mn0287 mn 0288 unique, then there are finitely many such solutions, say zn (1, . . . , zn (K). Given the corresponding subgradients, g1, . . . , gk, the subdifferential 0289 g1, . . . , gK. 0290 II.4.2 Mathematical Description of a Merit or Auxiliary Function 0291 For real-time needs, one must address the fact that the non-smooth optimization problem of step 508 (FIG. 8) requires the solution of an NP-hard problem for m>2. One approach to this problem is to use the following merit or auxiliary function to decide whether a function value has increased or decreased sufficiently in the line search or trust region methods: 42

\begin{matrix} _{m n} ({\overline{u}}^{3},, {\overline{u}}^{m}; u^{m + 1},, u^{n}) = {\begin{matrix} _{m n} (u^{m + 1},, u^{n}) & if m = 2 \\ or (6.2) is solved optimally, \\ _{2 n} ({\overline{u}}^{3},, \overset{m}{\overline{u}}; u^{m + 1},, u^{n}) & otherwise . \end{matrix} & [7.5] \end{matrix}

0292 where the multipliers {overscore (u)}3, . . . , {overscore (u)}m that appear in lower order relaxations used to construct (suboptimal) solutions of the m-dimensional relaxed problem (6.2) have been explicitly included. Note that mn 0293 be solved optimally if m2. For sufficiently small problems (6.2) or (6.4), one can more efficiently solve the NP-hard problem by branch and bound. This is the reason for the inclusion of the first case; otherwise, the relaxed function 2n 0294 the merit function provides a lower bound for the optimal solution follows directly from Fact A.4 and the following fact. 0295 Fact A.5. Given the definition of mn 0296 mn3 m m1 nmnm1 n 0297 {overscore (u)}3, . . . , {overscore (u)}m, um1, . . . , un.7.60298 The actual function value used in the optimization phase is mn 0299 with the solution zi1n. . . in being a suboptimal solution constructed from a relaxation procedure applied to the m-dimensional problem. Again, the use of these lower order relaxed problems is the reason for the inclusion of the multipliers {overscore (u)}3, . . . , {overscore (u)}m. 0300 To explain how the merit function is used, suppose there is a current multiplier set (uoldm1, . . . , uold n) and it is desirable to update to a new multiplier set (unewm1, . . . , unewn) via (unewm1, . . . , unewn)(uo1 dm1un)old (m1 nmnold old old old3 m m1 n old old3 m obtained during the relaxation process used to compute a high quality solution to the relaxed m-dimensional assignment problem (6.2) at (um1, . . . , un)(uoldm1, . . . , uoldn). The decision to accept (unewm1, . . . , unewn) is then based on mnold old3 m uoldm1, . . . , uoldn)<mnnew new new new2 m m1 n stopping criteria commonly used in line searches. Again, ({overscore (u)}new3, . . . , {overscore (u)}newm) represents the multiplier set used in the lower level relaxation procedure to construct a high quality feasible solution to the m-dimensional relaxed problem (6.2) at (um1, . . . , un)(unewm1, . . . , unewn). The point is that each time one changes (um1, . . . ,un) and uses the merit function mn3 m m1 n generally change the lower level multipliers ({overscore (u)}3, . . . , {overscore (u)}m). 0301 An illustration of this merit function for mn1 is given in PARTITIONING MULTIPLE DATA SETS, MULTIDIMENSIONAL ASSIGNMENTS AND LAGRANGIAN RELAXATION, by Aubrey B. Poore and Nenad Rijavec, and in QUADRATIC ASSIGNMENT AND RELATED PROBLEMS, DIMACS SERIES IN DISCRETE MATHEMATICS AND THEORETICAL COMPUTER SCIENCE, in American Mathematical Society, Providence, R.I., Vol. 16, 1994, pp. 25-37 edited by Panos M. Pardalos and Henry Wolkowicz. 0302 II.4.3 Non-smooth Optimization Methods 0303 By Fact A.4 the function mn affine, and concave, so that the negative of mn Thus the problem of maximizing mn optimization. Since there is a large literature on such problems, only a brief discussion of the primary classes of methods for solving such problems is provided. A first class of methods, known as subgradient methods, are reviewed and analyzed in the book by N. Z. Shor, Minimization Methods for Non-Differentiable Functions, Springer-Verlag, New York, 1985. Despite their relative simplicity, subgradient methods have some drawbacks that make them inappropriate for the tracking problem. They do not guarantee a descent direction at each iteration, they lack a clear stopping criterion, and exhibit poor convergence (less than linear). 0304 A more recent and sophisticated class of methods are the bundle methods; excellent developments are presented in the books of Hiriart-Urruty and Lemarchal: J.-B. Hiriart-Urruty and C. Lemarchal, Convex Analysis and Minimization Algorithms I and II, Springer-Verlag, Berlin, 1993; and the book of K. C. Kiwiel: Methods of Descent for Non-Differentiable Optimization, Lecture Notes in Mathematics, Vol. 1133, Springer-Verlag, Berlin, 1985. Bundle methods retain a set (or bundle) of previously computed subgradients to determine the best possible descent direction at the current iteration. Because of the non-smoothness of mn may not provide a sufficient decrease in mn bundle algorithms take a null step, wherein the bundle is enriched by a subgradient close to uk. As a result, bundle methods are non-ascent methods which satisfy the relaxed descent condition mnk1 mnk k1uk. These methods have been shown to have good convergence properties. In particular, bundle method variants have been proven to converge in a finite number of steps for piecewise affine convex functionals in K. C. Kiwiel, An Aggregate Subgradient Method for Non-smooth Convex Minimization, Mathematical Programming 27, (1983) pp. 320-341, and H. Schramm and J. Zowe, A version of the bundle idea for minimizing a non-smooth function: Conceptual idea, convergence analysis, numerical results, SIAM Journal on Optimization, 2 (1992), pp. 121-152. 0305 All of the above non-smooth optimization methods require the computation mn mn the computation of the relaxed cost coefficients 6.3. In both the first and second procedures discussed in section II.3.1, maximization of 2n In the most efficient implementations presently known of these two procedures, it was found that at least 95% of the computational effort of the entire procedure is spent in the evaluation of the relaxed cost coefficients 6.3 as part of computing 2n that makes as efficient use of the subgradients and function values as is possible, even at the cost sophisticated manipulation of the subgradients. In evaluating three different bundle procedures: (a) the conjugate subgradient method of Wolfe used in section I.1 of the first embodiment of the present invention; (b) the aggregate subgradient method of Kiwiel (described in K. C. Kiwiel, An Aggregate Subgradient Method for Non-smooth Convex Minimization, Mathematical Programming 27, (1983) pp. 320-341); and (c) the bundle trust method of Schramm and Zowe (described in H. Schramm and J. Zowe, A version of the bundle idea for minimizing a non-smooth function: Conceptual idea, convergence analysis, numerical results, SIAM Journal on Optimization, 2 (1992), pp. 121-152), it was determined that for a fixed number of non-smooth iterations, say, ten, the bundle-trust method provides good quality solutions with the fewest number of function and subgradient evaluations of all the methods, and is therefore the preferred method. 0306 II.4.4 The Two Dimensional Assignment Problem 0307 The forward/reverse auction algorithm of Bertsekas and Castaon (as described in D. P. Bertsekas and D. A. Castaon, A forward/reverse auction algorithm for asymmetric assignment problems, Computational Optimization and Applications, 1 (1992), pp. 277-298) is used to solve the many relaxed two dimensional problems that occur in the course of execution. 0308 II.4.5. Initial Multipliers and Hot Starts 0309 The effective use of hot starts is fundamental for real-time applications. A good initial set of multipliers can significantly reduce the number of non-smooth iterations (and hence the number of mn quality recovered solution. Further, there are several ways that multipliers can be reused. First, if a n-dimensional problem is relaxed to a m-dimensional problem, relaxation provides the multiplier set um1, um2, . . . , un. These can be used as the initial multipliers for the nm1 dimensional recovery problem for nm1>2. This approach has also worked well to reduce the number of non-smooth iterations during recovery. 0310 Further, for track maintenance, initial feasible solutions are generated as follows. When a new scan of information (a new observation set) arrives from a sensor, one can obtain an initial primal feasible solution by matching new reports to existing tracks via a two dimensional assignment problem. This is known as the track-while-scan (TWS) approach. Thus, an initial primal solution exists and then we use the above relaxation procedure is to make improvements to this TWS solution. Also for track maintenance, multipliers are available from a previously solved and closely related multi-dimensional assignment problems for all but the new observation set. 0311 II.4.6. Local Search Methods 0312 Given a feasible solution of the multi-dimensional assignment problem, one can consider local search procedures to improve this result, as described in C. H. Papadimitriou and K. Steiglitz, Combinatorial Optimization: Algorithms and Complexity, Prentice-Hall, Englewood Cliffs, N.J., 1982, and J. Pearl, Heuristics: intelligent search strategies for computer problem solving, Addison-Wesley, 1984. One method is the idea of k-interchanges. Since for sparse problems only those active arcs that participate in the solution are stored, it is difficult to efficiently identify eligible variable exchange sets with some data structures for solving the assignment problem. However, a local search that may be more promising is to use the two dimensional assignment solver in the following way. Given a feasible solution to the multi-dimensional assignment problem, the indices that correspond to active arcs in the solution are enumerated. Subsequently, one coordinate is freed to formulate a two dimensional assignment problem with one index corresponding to the enumeration and the other to the freed coordinate, and then solve a two dimensional assignment problem to improve the freed index position. 0313 II.4.7. Problem Decomposition 0314 The procedures described thus far are all based on relaxation. Due to the sparsity of assignment problems, however, frequently a decomposition of the problem into a collection of disjoint components can be done wherein each of the components can be solved independently. Due to the setup costs of Lagrangian relaxation, a branch and bound procedure is generally more efficient for small components, say ten to twenty feasible 0-1 variables (i.e., zi1 . . . in). Otherwise, the relaxation procedures described above are used. Perhaps the easiest way to view the decomposition method is to view the reports or measurements as a layered graph. A vertex is associated with each observation point, and an edge is allowed to connect two vertices only if the two observations belong to at least one feasible track of observations. Given this graph, the decomposition problem can then be posed as that of identifying the connected subcomponents of a graph which can be accomplished by constructing a spanning forest via a depth first search algorithm, as described in A. V. Aho, J. E. Hopcroft, and J. D. Ullman, The Design and Analysis of Computer Algorithms, Addison-Wesley Publishing Company, Reading, Mass., 1974. Additionally, decomposition of a different type might be based on the identification of bi- and tri-connected components of a graph and enumerating on the connections. Here is a technical explanation. Let 0315 Zzi1i2. . . inzi1i2 . . . in is not preassigned to zero denote the set of assignable variables. Define an undirected graph G(N,A) where the set of nodes is

Nzinnn1, . . . , n; in1, . . . , Nm

0316 and arcs, 0317 A(zjnn, zjll)n1, jn0, j10 and there exists zi1i2 . . . inZ 0318 such that jnin and jlil. 0319 Note that the nodes corresponding to zero index have not been included in the above defined graph, since two variables that have only the zero index in common can be assigned independently. Connected components of the graph are then easily found by constructing a spanning forest via a depth first search. (A detailed algorithm can be found in the book by Aho, Hopcroft and Ullman cited above). Furthermore, this procedure is used at each level in the relaxation, i.e., is applied to each assignment problem 3.1 for n3, . . . ,M. 0320 The original relaxation problem is decomposed first. All relaxed assignment problems can be decomposed a priori and all recovery problems can be decomposed only after they are formulated. Hence, in the n-to-(n-1) case, we have n-2 relaxed problems that can all be decomposed initially, and the recovery problems that are not decomposed (since they are all two dimensional). In the n-to-2 case, we have only one relaxed problem that can be decomposed initially. This case yields n-3 recovery problems, which can be decomposed only after they are formulated. 0321 The ever-increasing demand for sensor surveillance systems is to accurately track and identify objects in real-time, even for dense target scenarios and in regions of high track contention. The use of multiple sensors, through more varied information, has the potential to greatly improve target state estimation and identification. However, to take full advantage of the available data, a multiple frame data association approach is needed. The most popular such approach is an enumerative technique called multiple hypothesis tracking (MHT). As an enumerative technique, MHT can be too computationally intensive for real-time needs because this problem is NP-hard. A promising approach is to utilize the recently developed Lagrangian relaxation algorithms for the multidimensional assignment problem; however, there are many other potentially good approaches to these assignment problems such as LP relaxation combined with an interior point method, GRASP, and parallelization. 0322 These data association problems are fundamentally important in tracking. Thus, our first objective is to present an overview of the tracking problem and the context within which these data association problems fit. Following a brief review of the probabilistic framework for data association, the second objective is to formulate two models for track initiation and maintenance as multidimensional assignment problems. This formulation is based on a window moving over the frames of data. The first and simpler model uses the same window length for track initiation and maintenance, while the second model uses different lengths. As the window moves over the frames of data, one creates a sequence of these multidimensional assignment problems and there is an overlap region of frames common to adjacent windows. Thus, given the solution of one problem, one can develop a warm start for the solution to the next problem in the sequence, as shown hereinafter. Such information is critically important to the design of real-time algorithms.

Overview of the Tracking Problem 0323 Tracking and data fusion are complex subjects of specialization that can only be briefly summarized as they are related to the subject of this paper. The processing of track multiple targets using data from one or more sensors is typically partitioned into two stages or major functional blocks, namely, signal processing and data process. The first stage is the signal processing that takes raw sensor data and outputs reports. Reports are sometimes called observations, threshold exceedances, plots, hits, or returns, depending on the type of sensor. The true source of each report is usually unknown and can be due to a target of interest, a false signal, or persistent background objects that can be moving in the field of view of the sensor. 0324 Two principal functions of data processing are tracking and discrimination or target identification. The discrimination or target identification function distinguishes between tracks that are for targets of interest and other tracks such as those due to background. Also, if there is enough information, each track is classified as to the type of target it appears to represent. Target identification and discrimination will not be discussed further in this paper except to comment here on the use of attributes (including identification information) in the data association. 0325 There are two types of attributes or features, namely, discrete valued variables and continuous valued variables. Available attributes of either or both types should be used in computing the log likelihoods or scores for data association. In discussing tracking in this paper, the attributes will not be mentioned explicitly but it should be understood that some reports and some tracks may include attributes information that should be and can be used in the track processing (both data association and filtering) if it is useful.

Tracking Functions 0326 The tracking function accepts reports for each frame of data and constructs or maintains a target state estimate, the variance-covariance matrix of the state estimation error, and refined estimates (or probabilities) of the target attributes. The state estimate typically includes estimates of target position and velocity in three dimensions and possibly also acceleration and other variables of interest. 0327 The tracking function is typically viewed in two stages, namely, data association and filtering. Also, the process of constructing new tracks, called track initiation, is different from the process of updating existing tracks, called track maintenance. 0328 In track maintenance, the data association function decides how to relate the reports from the current frame of data to the previously computed tracks. In one approach, at most one report is assigned to each track, and in other approaches, weights are assigned to the pairings of reports to a track. After the data association, the filter function updates each target state estimate using the one or more (with weights) reports that were determined by the data association function. A filter commonly used for multiple target tracking and data fusion is the well know Kalman filter (or the extended Kalman filter in the case of nonlinear problems) or a simplified version of it. 0329 In track initiation, typically a sequence of reports is selected (one from each of a few frames of data) to construct a new track. In track initiation, the filtering function constructs a target state estimate and related information based on the selected sequence of reports. The new track is later updated by the track maintenance processing of the subsequent frames of reports. 0330 In some trackers, there is also a tentative tracking function for processing recently established tracks until there is enough confidence to include them in track maintenance. While for simplicity of presentation, tentative tracking will not be included in the processing discussed in this paper, the techniques discussed could readily include one or more tentative tracking functions. 0331 There have been numerous approaches developed to perform the data association function. Since optimal data association is far too complex to implement, good but practical sub-optimal approaches are pursued. Data association approaches can be classified in a number of ways. On way to classify data association approaches is based on the number of data frames used in the association process. In single frame data association for track maintenance, hard decisions are made each frame as to how the reports are to be related to the tracks. Some single frame approaches include: individual nearest neighbor, PDA, JPDA, and global nearest neighbor (sequential most probable hypothesis tracking), which uses a two dimensional assignment algorithm. In most single frame data association approaches, only one track per object is carried forward to be used in processing the next frame of reports. 0332 Multiple frame data association is more complex and frequently involves purposely carry forward more than one track per target to be used in processing the next frame of reports. By retaining more than one track per target, the tracking performance is improved at the cost of increased processing load. The best known multiple frame data association approach is an enumerative technique called multiple hypothesis tracking (MHT) which enumerates all the global hypothesis with various pruning rules. 0333 As shown hereinafter, the log likelihood of the probability of a global hypothesis can be decomposed into the sum of scores of each track that is contained in the hypothesis. As a consequence, the most probable hypothesis can be identified with the aid of a non-enumerative algorithm for the assignment so formulated.

Interface Between Track Initiation And Track Maintenance 0334 There are a number of ways that track initiation and track maintenance functions can interact. Two ways that are especially pertinent to the methods of this paper will be discussed in this section. 0335 One approach is to treat these two functions sequentially, by first assigning reports to tracks and then using the information that remains to form new tracks. A better approach is to integrate both processes where initiating tracks compete for the new frame of reports on an equal basis, i.e., at the same time. 0336 In the integrated approach, the data association for track initiation and track maintenance processing are combined and conducted simultaneously. One assignment array is created that includes the track scores for potential tracks for both track initiation and track maintenance. The first dimension of the assignment problem includes only all the tracks either created or updated in the processing of the prior frames of reports. Each of the remaining dimensions accommodates one frame of reports. 0337 After the assignment algorithm finds a solution, each of the previously established tracks are updated with the report assigned to it from the second dimension of the assignment array. The remaining reports in the second dimension that were in the assignment solution are firmly established as track initiators. The unassigned reports in the second dimension of the assignment array are discarded as false signals. The processing of the next frame of reports repeats this process using these updated tracks and the newly identified initiators in the first dimension of the cost array for processing the new frame of reports. Details of this approach are discussed hereinafter. 0338 The integrated approach just discussed, uses the same number of frames of reports for both track initiation and track maintenance. The goal in using the multi-dimensional assignment algorithm is to provide improved performance while minimizing the amount of processing required. Typically, track initiation will benefit from more frames of reports than will track maintenance. Thus a second approach that integrates track initiation and track maintenance is discussed hereinafter, wherein the number of frames of reports is not the same for these two functions. This is a novel approach that is introduced in this paper.

Multiple Sensor Processing 0339 There are many advantages and many ways to combining data from multiple sensors. There are also many ways of categorizing the different algorithmic architectures for processing data from multiple sensors. One approach outlines four generic algorithmic architectures. Two of these generic architectures are especially pertinent to this paper and are summarized briefly. 0340 In the Centralized Fusion algorithmic architecture, reports are combined from the various sensors to form global tracks. This algorithmic architecture is also called Central Level Tracking, Centralized Algorithmic Architecture, or simply the Type IV algorithmic architecture. In track maintenance for example, if a single frame data association is used, then for data association a frame of reports from one sensor is processed with the latest global tracks; then the global tracks are updated by the filter function. After the processing of this frame of reports is completed, a frame of the reports from another (or the same) sensor is processed with these updated global tracks. This process is continued as new frames of data become available to the system as a whole. 0341 In Centralized Fusion, using the multi-dimensional assignment algorithm for the data association with multiple sensors is similar to the processing of data from a single sensor. Instead of using multiple frames of reports from a single sensor, the multiple frames of reports come from multiple sensors. The frames of reports from all the sensors are ordered based on the nominal time each frame of reports is obtained and independent of which sensor provided the reports. In this way the approaches discussed in this paper can be extended to process reports from multiple sensors using the Centralized Fusion algorithmic architecture. 0342 The second pertinent algorithmic architecture is Track Fusion. This approach is also called the Hierarchical or Federated algorithmic architecture, sensor level tracking or simply the Type II algorithmic architecture. In track maintenance for example, a processor for each sensor computes single sensor tracks; these tracks are then forwarded to the global tracker to compute global tracks based on data from all sensors. 0343 After the first time a sensor processor forwards tracks to the global tracker, then subsequent tracks for the same targets are cross-correlated with the existing global tracks. This track-to-track cross-correlation is due to the common history of the current sensor tracks and the tracks from the same sensor that were forwarded earlier to the global tracker. The processing must take this cross-correlation into account and there are a number of ways of compensating for this-cross-correlations. One method for dealing with this cross-correlation is to decorrelate the sensor tracks that are sent to the global tracker. There are a variety of ways to achieve this decorrelation and some are summarized in Drummond, 1996, Signal Data Proc. of Small Targets 1995,-2561:369-383. 0344 Once the sensor tracks are decorrelated they can be processed by the global tracker in almost the same way as reports are processed. In the case of track fusion the association process is referred to as track (or track-to-track) association rather than data (or report-to-track) association. If the sensor tracks are decorrelated, the global tracker can process the tracks from the various sensor processors in much the same way that the global tracker of Centralized Fusion processes reports. Accordingly the methods described in this paper can be readily extended to processing data from multiple sensors using either the Centralized Fusion or Track Fusion or even a hybrid combination of both algorithmic architectures.

Formulation of the Data Association Problem 0345 The goal of this section is to briefly outline the probabilistic framework for the data association problems presented in this work. The technical details are presented elsewhere. The data association problems for multisensor and multitarget tracking considered in this work are generally posed as that of maximizing the posterior probability of the surveillance region (given the data) according to

Maximize p(ZN1)*(3.1)

0346 where ZN1 represents N1 data sets, indices of the data (and thus induces a partition of the data), 0347 0 0348 partition, and P( ZN1) is the posterior probability of a partition is defined below; however, this framework is currently sufficiently general to cover set packings and coverings. 0349 Consider N1 data sets Z(k)(k1, . . . , N1), each consisting of Mk actual reports and a dummy report z0k, and let denote the cumulative data set defined by 43

\begin{matrix} Z (k) = {z_{i_{k}}^{k}}_{i_{k} = 0}^{M_{k}} and Z^{N + 1} = {Z (1),, Z (N + 1)}, & (3.2) \end{matrix}

0350 respectively. (The dummy report z0k serves several purposes in the representation of missing data, false reports, initiating tracks, and terminating tracks.) In multisensor data fusion and multitarget tracking the data sets Z(k) may represent different classes of objects, and each data set can arise from different sensors. For track initiation the objects are reports that must be partitioned into tracks and false alarms. In our formulation of track maintenance, which uses a moving window, one data set will be tracks and remaining data sets will be reports which are assigned to existing tracks, as false reports, or to initiating tracks. 0351 We specialize the problem to the case of set partitioning defined in the following way. Define a track of data as zi11, . . . , ziN1N1 where each ik and zikk can assume the values of 0 and z0k, respectively. A partition of the data will refer to a collection of tracks of data wherein each report occurs exactly once in one of the tracks of data and such that all data is used up; the occurrence of dummy report is unrestricted. The reference partition is that in which all reports are declared to be false. 0352 Next, under appropriate independence assumptions one can show 44

\begin{matrix} \frac{P (= | Z^{N + 1})}{P (=^{0} | Z^{N + 1})} L \underset{(i_{1} i_{N + 1})}{} L_{i_{1} i_{N + 1}}, & (3.3) \end{matrix}

0353 Li1 . . . iN1 is a likelihood ratio containing probabilities for detection, maneuvers, and termination as well as probability density functions for report errors, track initiation and termination. Define 45

\begin{matrix} c_{i_{1} i_{N + 1}} = - \ln L_{i_{1} i_{N + 1}}, z_{i_{1} i_{N - 1}} = {\begin{matrix} 1 & if (z_{i_{1} i_{N + 1}}) are assigned to as a track, \\ 0 & otherwise . \end{matrix} & (3.4) \end{matrix}

0354 Then, recognizing that 46

\frac{P (= | Z^{N + 1})}{P (=^{0} | Z^{N + 1})} = (i_{1},, i_{N + 1}) c_{i_{1} i_{N + 1},}

0355 the problem (3.1) can be expressed as the N1-dimensional assignment problem 47

\begin{matrix} Minimize_{i_{1} = 0}^{M_{1}}_{i_{N + 1} = 0}^{M_{N + 1}} c_{i_{1} i_{N + 1}} z_{i_{1} + i_{N + 1}} Subject to_{i_{2} = 0}^{M_{2}}_{i_{N + 1} = 0}^{M_{N + 1}} z_{i_{1} i_{N + 1}} = 1, i_{1} = 1,, M_{1},_{i_{1} = 0}^{M_{1}}_{i_{k - 1} = 0}^{M_{k - 1}}_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{N + 1} = 0}^{M_{N + 1}} z_{i_{1} i_{N + 1}} = 1, for i_{k} = 1,, M_{k} and k = 2,, N,_{i_{1} = 0}^{M_{1}}_{i_{N + 1 - 1} = 0}^{M_{N}} z_{i_{1} i_{N + 1} = 1, i_{N + 1} = 1,, M_{N + 1}} z_{i_{1} i_{N + 1}} {0, 1} for all i_{1},, i_{N + 1}, & (3.5) \end{matrix}

0356 where c0 . . . 0 is arbitrarily defined to be zero. Here, each group of sums in the constraints represents the fact that each non-dummy report occurs exactly once in a track of data. One can modify this formulation to include multiassignments of one, some, or all the actual reports. The assignment problem (3.5) is changed accordingly. For example, if zikk is to be assigned no more than, exactly, or no less than nikk times, then the 1 in the constraint (3.5) is changed to , , nikk, respectively. In making these changes, one must pay careful attention to the independence assumptions, which need not be valid in many applications. Expressions for the likelihood ratios can be found in the work of Poore, 1994, Computational Optimization & Appl., 3:27-57, and the references therein.

Track Initiation and Maintenance 0357 In this section we explain two multiframe assignment formulations to the track initiation and maintenance problem. The continued use of all prior information is computationally intensive for tracking, so that a window sliding over the frames of reports is used as the framework for track maintenance and track initiation within the window. The objectives are to describe an improved version of a simple method and then to put this into a more general framework in which track initiation and maintenance have different length moving windows.

The First Approach to Track Maintenance and Initiation 0358 The first method as explained in this section is an improved version of our first track maintenance scheme and uses the same window length for track initiation and maintenance after the initialization step. The process is to start with a window of length N1 anchored at frame one. In the first step one there is only track initiation in that we assume no prior existing tracks. In the second and all subsequent frames, there is a window of length N anchored at frame k plus a collection of tracks up to frame k. This window is denoted by k; k1, . . . , kN. The following explanation of the steps is much like mathematical induction in that we explain the first step and then step k to step k1. 0359 Track Maintenance and Initiation: Step 1. Let 48

\begin{matrix} {i_{1} (l_{2}), i_{2} (l_{2}), i_{N} (l_{2}), i_{N + 1} (l_{2})}_{l_{2} = 1}^{L_{2}} & (4.1) \end{matrix}

0360 be an enumeration of all those zero-one variables in the solution of the assignment problem (3.5) (i.e., zi1i2 iN11) excluding all the false reports in the solution (i.e., all those zero-one variables with exactly one nonzero index) and zero-one variables in the solution for which (i1, i2 )(0,0). (The latter can correspond to tracks that initiate on frames three and higher.) These denote our initial tracks. 0361 Consider only the first two index sets in this enumeration (4.1) and add the zero index l20 with the corresponding values of ii and i2 being zero. Thus, the enumeration is now 49

{i_{1} (l_{2}), i_{2} (l_{2})}_{l_{2} = 0}^{L_{2}} .

0362 The notation T2(l2)(zi1(l2), zi2(l2) ) will be used for this pairing. Suppose now that the next data set, i.e., the (N2)th set, is added to the problem. 0363 To explain the costs for the new problem, one starts with the hypothesis that a partition * being true is now conditioned on the truth of the pairings on the first two frames being correct. Corresponding to the sequence T2 (l2), zi3, . . . , zin2, the likelihood function is then given by 50

\begin{matrix} L \underset{{T_{2} (l_{2}), z_{i_{3}},, z_{i_{N + 2}}}}{}, L_{l_{2} i_{3} i_{N + 2}} where L_{l_{2} i_{3} i_{N + 2}} L_{T_{2} (l_{2}),} L_{z_{i_{3}},, z_{i_{N + 2}}} = L_{T_{2} (l_{2})} L_{z_{i_{3}},, z_{l_{N + 2}}}, & (4.2a) \end{matrix}

0364 Next, define the cost and the corresponding zero-one variable 51

\begin{matrix} C_{l_{2} i_{3} i_{N + 2}} = - \ln L_{l_{2} i_{3} i_{N + 2}}, z_{l_{2} i_{3} i_{N + 2}} = {\begin{matrix} 1 & if {z_{i_{3}}^{3},, z_{i_{N + 2}}^{N + 2}} is assigned to Track T_{2} (l_{2}), \\ 0 & otherwise, \end{matrix} & (4.2b) \end{matrix}

0365 respectively. Then the track maintenance problem Maximize L* assignment 52

Minimize_{l_{2} = 0}^{L_{2}}_{i_{3} = 0}^{M_{3}}_{i_{N + 2} = 0}^{M_{N + 2}} c_{l_{2} i_{3} i_{N + 2}} z_{l_{2} i_{3} i_{N + 2}}

Subject to :_{i_{3} = 0}^{M_{3}}_{i_{N + 2} = 0}^{M_{N + 2}} z_{l_{2} i_{3} i_{N + 2}} = 1, l_{2} = 1, l_{2},, L_{2},_{l_{2} = 0}^{L_{2}}_{i_{4} = 0}^{M_{4} = 0}_{i_{N + 2} = 0}^{M_{N + 2} = 0} z_{l_{3} i_{3} i_{N + 2}} = 1, i_{3} = 1,, M_{3},_{l_{2} = 0}^{L_{2}}_{i_{3} = 0}^{M_{3}}_{i_{p - 1} = 0}^{M_{p - 1}}_{i_{p + 1} = 0}^{M_{p + 1}}_{i_{N + 2} = 0}^{M_{N + 2}} z_{l_{2} i_{3} i_{N + 2}} = 1

for i_{p} = 1,, M_{p} and p = 4,, N + 2 - 1, z_{i_{1} i_{N + 1}}_{i_{N + 2 - 1} = 0}^{M_{N + 2 - 1}} z_{l_{2} i_{3} i_{N + 2}} = 1, i_{N + 2} = 1,_{l_{2} i_{3} i_{N + 2}} {0, 1} for all l_{2} i_{3},, i_{N + 2}

0366 Track Maintence and Initiation: Step k. At the beginning of the kth step, we solve the following (N1)-dimensional assignment problem. 53

\begin{matrix} Minimize :_{l_{k} = 0}^{L_{k}}_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{k + N} = 0}^{M_{k + N}} c_{l_{k} i_{k + 1} i_{k + N}} z_{l_{k} i_{k + 1}} i_{k + N} Subject to :_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{k + N} = 0}^{M_{k + N}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, l_{k} = 1, L_{k},_{i_{k} = 0}^{L_{k}}_{i_{k + 2} = 0}^{M_{k + 2}}_{i_{k + N} = 0}^{M_{k + N}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, i_{k + 1} = 1, M_{k + 1},_{l_{k} = 0}^{L_{k}}_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{p - 1} = 0}^{M_{p - 1}}_{i_{p + 1} = 0}^{M_{p + 1}}_{i_{k + N} = 0}^{M_{k + N}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, for i_{p} = 1,, M_{p} and p = k + 2,, N + K - 1,_{l_{k} = 0}^{L_{k}}_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{k + 1 - N - 2} = 0}^{M_{k + 1 + N - 2}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, i_{k + N} = 1,, M_{k + N} z_{l_{k} i_{k + 1} i_{k + N}} {0, 1} for all l_{k}, i_{k + 1},, i_{k + N} . & (4.4) \end{matrix}

0367 where for the first step l1 and L1 are replaced by i1 and M1, respectively. The first index lk in the subscripts corresponds to the sequence of tracks 54

{T_{k} (l_{k})}_{l_{k} = 0}^{L_{k}}

0368 where Tk(lk)zi11(lk) , . . . , zikk(lk) is a track of data from the solution of the problem prior to the formulation of (4.4) or if k1 then it is just the first data set in frame one. 0369 Suppose problem (4.4) has been solved and let the solution, i.e., those zero-one variables equal to one, be enumerated by 55

\begin{matrix} {(l_{k} (l_{k + 1}), i_{k + 1} (l_{k + 1}),, i_{k + N} (l_{k + 1}))}_{l_{k + 1} = 1}^{L_{k + 1}} & (4.5) \end{matrix}

0370 with the following exceptions. 0371 a. All zero one variables for which (lk, ik1)(0,0) are excluded. Thus, tracks that initiate on frames after the (k1)th are not included in the list. 0372 b. All zero-one variables whose subscripts have the form lk0 and exactly one nonzero index in the remaining indices ik1, . . . , ikN are excluded. These correspond to false reports on frames pk1, . . . , kN. 0373 c. All variables (z1kik1. . . ikN) for which (ik1, . . . , ikN)(0,0, . . . , 0) and ik(lk)0 in (4.5). In other words, the reports on the last N1 frames in the 56

{T_{k} (l_{k}),_{i_{k + N}}^{z_{k + 1}}}

0374 are all dummy. Any solution with this feature corresponds to a terminated track. 0375 Given the enumeration (4.5), one now fixes the assignments on the first two index sets in the list (4.5). The zero index lk10 is added to the enumeration to specify (lk (0(, ik1(0))(0,0) that is used to represent false reports and tracks that initiate on frame k2 or later, so that the enumeration (4.5) is now 57

\begin{matrix} {(l_{k} (l_{k + 1}), i_{k + 1} (l_{k + 1}))}_{l_{k + 1} = 0}^{L_{k + 1}} . & (4.6) \end{matrix}

0376 Then, for lk11, . . . , Lk1, the lik1th such track is denoted by Tk1(lk1)Tk(lk(lk1)), zk1ik1 (lk1) and the (N1)-tuple Tk1(lk1), zik2k2, . . . , zik1Nk1N will denote a track Tk1(lk1) plus a set of reports zik2k2, . . . , zik1Nk1N, actual or dummy, that are feasible with the track Tk1(lk1). The (N1)-tuple Tk1(0), zik2k2, . . . , zik1Nk1N will denote a track that initiates in the sliding window, i.e., on subsequent frames. A false report in the sliding window is one with all but one non-zero index ip for some pk2, . . . , k1N in the (N1)-tuple Tk1(0), zik2k2, . . . , zik1Nk1N. 0377 The corresponding hypothesis about a partition * being true is now conditioned on the truth of the Lk1 tracks existing at the beginning of the N-frame window. (Thus the assignments prior to this sliding window are fixed.) The likelihood function is given by 58

\begin{matrix} L_{} = \underset{T_{k_{1} (l_{k + 1})}, z_{i_{k + 2}},, z_{i_{k + 1 + N}}}{} L_{l_{h + 1} i_{h + 1 + N}}, where L_{l_{k - 1} i_{k + 2} i_{k + 1 + N}} = L_{T_{k + 1} (l_{k + 1})}, L_{z_{i_{k + 2}}^{k + 2}},, L_{z_{i_{k + 1 + N}}^{k + 1 + N}} L_{T_{k + 1} (0)} = 1. & (4.7a) \end{matrix}

0378 Next, define the cost and the zero-one variable by said at least one inert gas has an amount, on a molar basis, that is greater than the amount of said oxygen in said pressurized medium; clk1ik2. . . ik1NlnLlk1ik2. . . ik1NlnLTk1(lk1)zik2. . . zik1N(4.7b) 59

z_{l_{k + 1} i_{k + 2} i_{k + 1 + N}} = {\begin{matrix} 1 & if {z_{i_{k + 2}}^{k + 2},, z_{i_{k + 1 + N}}^{k + 1 + N}} \\ 0 & otherwise \end{matrix}

0379 is assigned to Tk1(lk1), respectively, so that the track extension problem, which was originally formulated as Maximize L*, can be expressed as exactly the same multi-dimensional assignment in (3.4) but with k replaced by k1. Thus, we do not repeat it here.

The Second Approach to Track Maintenance and Initiation 0380 In the first approach the window lengths were the same for both track maintenance and initiation. This can be inefficient in that one can usually use a shorter window for track maintenance than for track initiation. This section addresses such a formulation. 0381 The General Step k. To formulate a problem for track initiation and maintenance we consider a moving window centered as frame k of length IJ1 denoted by kI, . . . , k, . . . , kJ. In this representation, the window length for track maintenance is J and that for initiation, IJ1. The objective will be to explain the situation at this center and then the move to the same length window at center k1 denoted by k1I, . . . , k1, . . . , k1J, i.e., by moving the frame one frame at time to the right. The explanation from the first step follows hereinafter. 0382 The notation for a track of data is

Tk(lk)zi11(lk), . . . , zipp(lk), . . . , zikk(lk)(4.8)

0383 where the index lk is used for an enumeration of those reports paired together. We also use the notation Tp,k(lk) to denote the sequence of reports belonging to track Tk(lk) but restricted to frames prior to and including p. Thus,

Tk(lk)Tk,k(lk)

Tk(lk)zi11(lk), . . . , zipp(lk)(4.9)

Tk,k(lk)Tp,k(lk)ziplpl(lk), . . . , zik1k1(lk) ,zikk(lk) for pk1.

0384 Given this notation for the tracks and partition of the data in the frames kI, . . . , k, . . . , kJ , LTp,k(lk) will denote the accumulated likelihood ratio up to and including frame p(pk) for a track that is declared as existing on frame k as a solution of the assignment problem. In this notation, the likelihood for Tp,k(lk) and that of the association of zik1k1, . . . , zikNkN with track Tk(lk) is given by 0385 LTk(lk)LTp,k(lk) Lip1(lk)ik(ik) for any pk1, 0386 LTk(lk)ik1ikWLTp,k(ik)Lip1(ik)ik(lk)ikN for any pk1,(4.10) 0387 respectively. 0388 The cost for the assignment of zik1k1, . . . , zikNkN to track Tk(lk) and the corresponding zero one variable are given by 60

\begin{matrix} \begin{matrix} c_{l_{k} i_{k + 1} i_{k + J}}^{T} = - \ln [L_{T_{k} (l_{k})} L_{i_{k + 1} i_{k + J}}] \\ = - \ln [L_{T_{p}, k (l_{k})} L_{i_{p + 1} (l_{k}) i_{k} (l_{k}) i_{k + 1} i_{k + J}}] \end{matrix} and & (4.11) \end{matrix}

z_{l_{k} i_{k + 1} i_{k + J}}^{T} = {\begin{matrix} 1 & if {z_{i_{k + 1}}^{k + 1},, z_{i_{k + N}}^{k + N}} \\ 0 & otherwise \end{matrix}

0389 respectively. Likewise, for costs associated with the false reports on the frames k-I to k and as associated with zik1k1, . . . ,zikJkJ and the corresponding zero-one variable are given by: 62

z_{i_{k - 1} i_{k} i_{k + 1} i_{k + J}}^{F} = {\begin{matrix} 1 & if {z_{i_{k - I}},, z_{i_{k} i_{k + 1}},, z_{k_{k + J}}} \\ are assigned as a track, \\ 0 & otherwise . \end{matrix}

0390 For the window of frames in the range kI, . . . , k, the easiest explanation of the partition of the reports is based on the definitions 0391 Tp,kipzipp belongs to one of the tracks listed on frame k, 0392 i.e., to one of the Tk(lk) for some lk1, . . . , Lk 0393 Fp,k(1, . . . , Mp)tp,k)(0),(4.13) 0394 so that all of those reports not used in the tracks Tk(lk) on frame p are put into the set of available reports Fp,k for the formation of tracks over the entire window. Thus, we formulate the assignment problem as 63

Minimize_{p = k - I}^{k} \underset{i_{p} F_{p, k}}{}_{r - k + 1}^{k + J}_{i_{r} = 0}^{M_{r}} c_{i_{k - I} i_{k}, i_{k +}^{1} i_{k +}^{J}}^{F} z_{ik - I ikik + 1 ik + J}^{F}

Subject to :_{i_{3} = 0}^{M_{3}}_{i_{N + 2} = 0}^{M_{N + 2}} z_{l_{2} i_{3} i_{N + 2}} = 1, l_{2} = 1, l_{2},, L_{2},_{l_{2} = 0}^{L_{2}}_{i_{4} = 0}^{M_{4} = 0}_{i_{N + 2} = 0}^{M_{N + 2} = 0} z_{l_{3} i_{3} i_{N + 2}} = 1, i_{3} = 1,, M_{3},_{l_{2} = 0}^{L_{2}}_{i_{3} = 0}^{M_{3}}_{i_{p - 1} = 0}^{M_{p - 1}}_{i_{p + 1} = 0}^{M_{p + 1}}_{i_{N + 2} = 0}^{M_{N + 2}} z_{l_{2} i_{3} i_{N + 2}} = 1

for i_{k} = 1,, M_{k} and k = 2,, N,_{i_{1} = 0}^{M_{1}}_{i_{N + 1 - 1} = 0}^{M_{n}} z_{i_{1} i_{N + 1}} = 1, i_{N + 1} = 1, M_{N + 1}

z_{i_{1}} l_{N + 1} {0, 1} for all i_{1},, i_{N + 1}

0395 Suppose problem (4-14) has been solved and let the solution, i.e., those zero-one variables equal to one, be 64

\begin{matrix} {z_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1}) i_{k + j} (l_{k + 1})}^{T enumerated by}}_{l_{k + 1} = 1}^{{\overline{L}}_{k + 1}} {z_{l_{k - I} (l_{k + 1}) i_{k} (l_{k + 1}) i_{k} (l_{k + 1}) i_{k + J} (l_{k + 1})}^{F}}_{l_{k + 1} = {\overline{L}}_{k + 1} + 1}^{L_{k + 1}} & (4.15) \end{matrix}

0396 with the following exceptions. 0397 a. All zero one variables in the second list for which (ik1, . . . , ik, ik1)(0, . . . , 0, 0) are excluded. Thus, tracks that initiate on frames after the (k1)th are not included in the list. 0398 b. All false reports axe excluded, i.e., all zero-one variables in the second list whose subscripts have exactly one nonzero index. 0399 c. All variables zlkik1. . . ikNT for which (ik1, . . . , ikN)(0, 0, . . . , 0) and Z(p)Tk(lk)0 for pkK, . . . , k where K0 is user specified. Thus the track Tk(lk) is terminated if it is not observed over KJ1 frames. 0400 Given the enumeration (4.15), one now fixes the assignments on the all index sets up to and including the (k1)th index sets. 65

\begin{matrix} T_{k + 1} (l_{k + 1}) = {\begin{matrix} (T_{k} (l_{k} (l_{k + 1})), z_{i_{k + 1} (l_{k + 1})}^{k + 1}) \\ for l_{k + 1} = 1,, {\overline{L}}_{k_{1}} \\ (z_{i_{k - I} (l_{k + 1})}^{k - I},, z_{i_{k} (l_{k + 1})}^{k}, z_{i_{k + 1} (l_{k + 1})}^{k + 1}) \end{matrix} for l_{k + 1} = 1,, {\overline{L}}_{k + 1} + 1,, L_{k + 1} & (4.16) \end{matrix}

0401 Thus one can now formulate the assignment problem for the next problem exactly as in (4-14) but with k replaced by k1. Thus, we do not repeat it here. 0402 The Initial Step. Here is one explanation for the initial step. First, assume that NIJ. In this case, we start the track initiation with a solution of (3.5). Let 66

{i_{1} (l_{2}), l_{2} (l_{2}) i_{N} (l_{2}), i_{N + 1} (l_{2})}_{l_{2} = 0}^{L_{2}}

0403 be an enumeration of the solution set of (3.5), i.e., those zero-one variables zi1(l2)i2(l2)iN1(l2)1, including z00.01 corresponding to l20, but excluding all those zero-one variables that are assigned to one and correspond to false reports (i.e., there is exactly one nonzero index in the subscript of zi1i2. . . iN1), all those zero-one variables that are assigned to one and correspond to tracks that initiate on frames higher than I2. Then we fix the data association decisions corresponding to the reports in our list of tracks prior to and including frame k1I2. This defines the k for problem (4.4) and one can then continue the development by adding a frame to the window as in the general case. 0404 If IJ>N, then one possibility is to start the process with N1 frames, and assuming JN, proceed as before replacing I by NJ for the moment, and continue to add frames without lopping off the first frame in the window until reaches a window of length IJ1. Then we proceed as in the previous paragraph. 0405 If IJ<N, then one can solve the track initiation problem (3.5), formulate the problem with the center of the window at k1N1J, enumerate the solutions as above, and lop off the first NJI frames. Then, we proceed just as in the case IJN. 0406 A primary objective in this work has been to demonstrate how multidimensional assignment problems arise in the tracking environment. The problem of track initiation and maintenance has been formulated within the framework of a moving window over the frames of data. The solution of these NP-hard, noisy, large scale, and sparse problems to the noise level in the problem is fundamental to superior track estimation and identification. Thus, one must utilize the special structure in the problems as well take advantage of special information that is available. Since these moving windows are overlapping, there are some algorithm efficiencies that can be identified and that take advantages of the overlap in the windows from one frame of reports to the next. Here is an example of the use of a primal solution of one problem to warm start the solution of the next problem in the sequence. 0407 Suppose we have solved problem (4.4) and have enumerated all those zero-one variables in the solution of (4.4) as in (4.5). Add the zero index lk10, so that the enumeration is

(lk(lk1), ik1(lk1), . . . , ikN(lk1))(5.1)

0408 With this enumeration one can define the cost by

clk1ik1Nclk(lk1), ik1(lk1), . . . , iN1(lk1)ik1N(5.2)

0409 and the two dimensional assignment problem 67

\begin{matrix} _{2} Minimize_{l_{k + 1} = 0}^{L_{k + 1}}_{i_{k + 1 + N} = 0}^{L_{k + 1 + N}} c_{l_{k + 1} i_{k + 1 + N}}^{2} z_{l_{k + 1} i_{k + 1 + N}}^{2} V_{2} (z^{2}) Subject to_{i_{k + 1 + N} = 0}^{M_{k} + 1 + N} z_{l_{k + 1} i_{k + 1 + N}}^{2} = 1, l_{k + 1} = 1,, L_{k + 1}_{l_{k + 1} = 0}^{L_{k + 1}} z_{l_{k + 1} i_{k + 1 + N}}^{2} = 1, i_{k + 1 + N} = 1,, M_{k + 1 + N} z_{l_{k + 1} i_{k + 1 + N}}^{2} {0, 1} for all l_{l_{k + 1}}, i_{k + 1 + N} . & (5.3) \end{matrix}

0410 Let w be an optimal or feasible solution to this two-dimensional assignment problem and define 68

\begin{matrix} {\hat{z}}_{i_{k + 1} i_{k + N} i_{k - 1 + N}} = {\begin{matrix} 1 & if (i_{k + 1},, i_{k + N}) = (i_{k + 1} (l_{k + 1},, i_{k + N} (l_{k + 1})) \\ and w_{l_{k + 1} i_{k + 1 + N}} = 1 for some l_{k + 1} = 1,, L_{k + 1} \\ or if (l_{k + 1}, i_{k + 1 + N}) = 0, 0 \\ 0 & otherwise . \end{matrix} & (5.4) \end{matrix}

0411 This need not satisfy the constraints in that there are usually many objects left unassigned. Thus, one can complete the assignment by using the zero-one variables in (4.4) with k replaced by k1 with exactly one nonzero index corresponding to any unassigned object or data report. 0412 For the dual solutions, the multipliers arising from the solution of the two dimensional assignment problem (5.3) corresponding to the second variable, i.e., 69

{u_{i_{k + 1 + N}}^{k + 1 + N}}_{i_{k + 1 + N} = 0}^{M_{k + 1 + N}} .

0413 These are good initial values for use in a relaxation scheme 11,12. Finally, note that one can also develop a warm start for problem (4.14) in a similar fashion. 0414 Multidimensional assignment problems govern the central problem of data association in multisensor and multitarget tracking, i.e., the problem of partitioning observations from multiple scans of the surveillance region and from either single or multiple sensors into tracks and false alarms. This fundamental problem can be stated as 70

\begin{matrix} Minimize_{i_{1} = 0}^{M_{1}}_{i_{N} = 0}^{M_{N}} c_{i_{1} i_{N}} z_{i_{1} i_{N}} Subject to :_{i_{2} = 0}^{M_{2}}_{i_{N} = 0}^{M_{N}} z_{i_{1} i_{N}} = 1,_{i_{1} = 1, l,, M_{1}},_{i_{1} = 0}^{M_{1}}_{i_{p - 1} = 0}^{M_{p - 1}}_{i_{p + 1} = 0}^{M_{p + 1} = 0}_{i_{N} = 0}^{M_{N}} z_{i_{1} i_{N} = 1} for i_{p} = 1,, M_{p} and p = 2,, N - 1,_{i_{1} = 0}^{M_{1}}_{i_{n - 1} = 0}^{M_{N - 1}} z_{i_{1} i_{N}} = 1, i_{N} = 1,, M_{N}, z_{l_{1} i_{N}} {0, 1} for all i_{1},, i_{N}, for i_{k} = 1,, M_{k} and k = 2,, N, z_{i_{1}} 1_{N + 1} {0, 1} for all i_{1},, i_{N + 1} & (1.1) \end{matrix}

0415 where c0 . . . 0 is arbitrarily defined to be zero and is included for notational convenience. One can modify this formulation to include multiassignments of one, some, or all of the actual reports. The zero index is used in representing missing data, false alarms, initiating and terminating tracks. In these problems, we assume that all zero-one variables Z zi1. . .iN with precisely one nonzero index are free to be assigned and that the corresponding cost coefficients are well-defined. (This is a valid assumption in the tracking environment.) Although not required, these cost coefficients with exactly one nonzero index can be translated to zero by cost shifting without changing the optimal assignment. Finally, this formulation is of sufficient generality to include the symmetric problem and the asymmetric inequality problem. 0416 The data association problems in tracking that are formulated as (1.1) have several characteristics. They are normally sparse, the cost coefficients are noisy and the problem is NP-hard, but it must be solved in real-time. The only known methods for solving this NP-hard problem optimally are enumerative in nature, with branch-and-bound being the most efficient; however, such methods are much too slow for real-time applications. Thus one must resort to suboptimal approaches. Ideally, any such algorithm should solve the problem to within the noise level, assuming, of course, that one can measure this noise level in the physical problem and the mathematical method provides a way to decide if the criterion has been met. 0417 There are many algorithms that can be used to construct suboptimal solutions to NP-hard combinatorial optimization problems. These include greedy (and its many variants), relaxation, simulated annealing, tabu search, genetic algorithms, and neural netork algorithms. For the three dimensional assignment problem, Pierskalla developed the tri-substitution method, which is a variant of the simplex method. Frieze and Yadegar introduced a method based on Lagrangian relaxation in which a feasible solution is recovered using information provided by the relaxed solution. Our choice of approaches is strongly influenced by the need to balance real-time performance and solution quality. Lagrangian relaxation based methods have been used successfully in prior tracking applications. An advantage of these methods is that they provide both an upper and lower bound on the optimal solution, which can then be used to measure the solution quality. These works extend the method of Frieze and Yadegar to the multidimensional case.

Probabilistic Framework for Data Association. (ABP) 0418 The goal of this section is to explain the formulation of the data association problems that governs large classes of data association problems in centralized or hybrid centralized-sensor level multisensor/multitarget tracking. The presentation is brief; technical details are presented for both track initiation and maintenance in the work of Poore. The formulation is of sufficient generality to cover the MHT work of Reid, Blackman and Stein and modifications by Kurien to include maneuvering targets. As suggested by Blackman, this formulation can also be modified to include target features (e.g., size and type) into the scoring function. The recent work of Poore and Drummond significantly extends the work of this section to new approaches for multiple sensor centralized tracking. Future work will involve extensions to track-to-track correlation. 0419 The data association problems for multisensor and multitarget tracking considered in this work are generally posed as that of maximizing the posterior probability of the surveillance region (given the data) according to

Maximize P(ZN)*(2.1)

0420 where ZN represents N data sets, of the data (and thus induces a partition of the data), 0421 0422 0 0423 P(ZN) is the posterior probability of a partition true given the data ZN. The term partition is defined below; however, this framework is currently sufficiently general to cover set packings and coverings. 0424 Consider N data sets Z(k) (kI, . . . , N) each of Mk reports 71

{z_{i_{k}}^{k}}_{i_{k} = 1}^{M_{k}},

0425 and let ZN denote the cumulative data set defined by 72

\begin{matrix} Z (k) = {z_{i_{k}}^{k}}_{i_{k} = 1}^{M_{k}} and Z^{N} = {Z (1),, Z (N)} & (2.2) \end{matrix}

0426 respectively. In multisensor data fusion and multitarget tracking the data sets Z(k) may represent different classes of objects, and each data set can arise from different sensors. For track initiation the objects are measurements that must be partitioned into tracks and false alarms. In our formulation of track maintenance, which uses a moving window over time, one data set will be tracks and remaining data sets will be measurements which are assigned to existing tracks, as false measurements, or to initiating tracks. In sensor level tracking, the objects to be fused are tracks. In centralized fusion, the objects may all be measurements that represent targets or false reports, and the problem is to determine which measurements emanate from a common source. 0427 We specialize the problem to the case of set partitioning defined in the following way. First, for notational convenience in representing tracks, we add a zero index to each of the index sets in (2.2), a dummy report z0k to each of the data sets Z(k) in (2.2), and define a track of data as (zi11, ziNN) where ik and zikk can now assume the values of 0 and z0k respectively. A partition of the data will refer to a collection of tracks of data wherein each report occurs exactly once in one of the tracks of data and such that all data is used up; the occurrence of dummy report is unrestricted. The dummy report z0k serves several purposes in the representation of missing data, false reports, initiating tracks, and terminating tracks. The reference partition is that in which all reports are declared to be false. 0428 Next under appropriate independence assumptions one can show 73

\begin{matrix} \frac{P (= | Z^{N})}{P (=^{0} | Z^{N})} L \underset{(i_{1} i_{N})}{} L_{i_{1} i_{N}}, & (2.3) \end{matrix}

0429 Li1. . . iN is a likelihood ratio containing probabilities for detection, maneuvers, and termination as well as probability density functions for measurement errors, track initiation and termination. Then if ci1iN1nLi1. . . iN

\begin{matrix} \ln \frac{P (| Z^{N})}{P (^{0} | Z^{N})} = \underset{(i_{1},, i_{N}) c_{i_{1} i_{N}}}{} . & (2.4) \end{matrix}

0430 Using (2.3) and the zero-one variable ziliN1 if (i1, . . . , iN) and 0 otherwise, one can then write the problem (2.1) as the following N-dimensional assignment problem: 75

\begin{matrix} Minimize_{i_{1} - i_{N}} c_{i_{1} i_{N}} z_{i_{1} i_{N}} \begin{matrix} Subject to_{i_{2} i_{3} i_{N}} z_{i_{1} i_{N}} = 1 & (i_{1} = 1,, M_{1}) \end{matrix} \begin{matrix} _{i_{1} i_{3} i_{N}} z_{i_{1} i_{N}} = 1 & (i_{2} = 1,, M_{2}) \end{matrix}_{i_{1} i_{p - 1} i_{p + 1} i_{N}} z_{i_{1} i_{N}} = 1 (i_{p} = 1,, M_{p} and p = 2, N - 1)_{i_{1} i_{2} i_{N - 1}} z_{i_{1} i_{N}} = 1 (i_{N} = 1,, M_{N}) z_{i_{1} i_{N}} {0, 1} for all i_{1},, i_{N} . & (2.5) \end{matrix}

0431 where c0 . . . 0 is arbitrarily defined to be zero. Here, each group of sums in the constraints represents the fact that each-non-dummy report occurs exactly once in a track of data. One can modify this formulation to include multiassignments of one, some, or all the actual reports. The assignment problem (2.5) is changed accordingly. For example, if zikk is to be assigned no more than, exactly, or no less than nikk times, then the 1 in the constraint (2.5) is changed to , , nikk, respectively. Modifications for group tracking and multiresolution features of the surveillance region will be addressed in future work. In making these changes, one must pay careful attention to the independence assumptions that need not be valid in many applications. 0432 Expressions for the likelihood ratios Li1, . . . , iN, can be found in the work of Poore. These expressions include the developments of Reid, Kurien, and Stein and Blackman. What's more, they are easily modified to include target features and to account for different sensor types. In track initiation, the N data sets all represent reports from N sensors, possibly all the same. For track maintenance, we use a sliding window of N data sets and one data set containing established tracks. The formulation is the same as above except that the dimension of the assignment problem is now N1.0433 To explain the costs for the new problem, one starts with the hypothesis that a partition * conditioned on the truth of the pairings on the first two frames being correct. Corresponding to the sequence (T2(l2), zi3, . . . ,ziN2), the likelihood function is then given by 76

\underset{{T_{2} (l_{2}), z_{i_{3}},, z_{i_{N + 2}}}}{} L_{l_{2} i_{3} i_{N + 2}} where

L_{l_{2} i_{3} i_{N + 2}} L_{T_{2} (l_{2})}, z_{i_{3}}, Z_{i_{N + 2}} = L_{T_{2} (l_{2})} L_{z_{i_{3}}}, z_{i_{N + 2}}, L_{T_{2}} = 1.

0434 Next, define the cost and the corresponding zero-one variable 77

\begin{matrix} C_{l_{2} i_{3} i_{N + 2}} = - \ln L_{l_{2} i_{3} i_{N + 2}}, z_{l_{2} i_{3} i_{N + 2}} = {\begin{matrix} 1 & if {z_{i_{3}}^{3},, z_{i_{N + 2}}^{N + 2}} is assigned to Track T_{2} (l_{2}), \\ 0 & otherwise \end{matrix} & (4.2b) \end{matrix}

0435 respectively. Then the track maintenance problem Maximize L* 0436 assignment 78

Minimize_{l_{2} = 0}^{L_{2}}_{i_{3} = 0}^{M_{3}}_{i_{N + 2} = 0}^{M_{N + 2}} c_{l_{2} i_{3} i_{N} + 2} z_{l_{2} i_{3} i_{N + 2}}

Subject to :_{i_{3} = 0}^{M_{3}}_{i_{N + 2} = 0}^{M_{N + 2}} z_{l_{2} i_{3} i_{N + 2}} = 1, l_{2} = 1, l_{2},, L_{2},_{l_{2} = 0}^{L_{2}}_{i_{4} = 0}^{M_{4} = 0}_{i_{N + 2} = 0}^{M_{N + 2} = 0} z_{l_{3} i_{3} i_{N + 2}} = 1, i_{3} = 1,, M_{3},_{l_{2} = 0}^{L_{2}}_{i_{3} = 0}^{M_{3}}_{i_{p - 1} = 0}^{M_{p - 1}}_{i_{p + 1} = 0}^{M_{p + 1}}_{i_{N + 2} = 0}^{M_{N + 2}} z_{l_{2} i_{3} i_{N + 2}} = 1

for i_{k} = 1,, M_{k} and k = 2,, N,_{i_{1} = 0}^{M_{1}}_{i_{N + 1 - 1} = 0}^{M_{N}} z_{i_{1} i_{N + 1}} = 1, i_{N + 1} = 1, M_{N + 1}

z_{i_{1}} 1_{N + 1} {0, 1} for all i_{1},, i_{N + 1}

0437 Track Maintence and Initiation: Step k. At the beginning of the kth step, we solve the following (N1)-dimensional assignment problem. 79

\begin{matrix} Minimize_{l_{k} = 0}^{L_{k}}_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{k + N} = 0}^{M_{k + N}} c_{l_{{ki}_{k + 1}} i_{k + N}} z_{l_{{ki}_{k + 1}} i_{k + N}} Subject to :_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{k + N} = 0}^{M_{k + N}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, l_{k} = 1, L_{k},_{i_{k} = 0}^{L_{k}}_{i_{k + 2} = 0}^{M_{k + 2}}_{i_{k + N} = 0}^{M_{k + N}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, l_{k} = 1, L_{k},_{i_{k} = 0}^{L_{k}}_{i_{k + 2} = 0}^{M_{k + 2}}_{i_{k + N} = 0}^{M_{k + N}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, i_{k + 1} = 1, M_{k + 1},_{l_{k} = 0}^{L_{k}}_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{p - 1} = 0}^{M_{p - 1}}_{i_{p + 1} = 0}^{M_{p + 1}}_{i_{k + N} = 0}^{M_{k + N}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, for i_{p} = 1,, M_{p} and p = k + 2,, N, + k + 1, & (4.4) \end{matrix}

_{l_{k} = 0}^{L_{k}}_{i_{k + 1} = 0}^{M_{k + 1}}_{i_{k + 1 + N - 2} = 0}^{M_{k - 1 + N - 2}} z_{l_{k} i_{k + 1} i_{k + N}} = 1, i_{k + N} = 1,, M_{k + N}

z_{l_{k} i_{k + 1} l_{k + N}} {0, 1} for all l_{k}, i_{k + 1},, i_{k + N}

0438 where for the first step l1 and L1 are replaced by il and Ml, respectively. The first index lk in the subscripts corresponds to the sequence of tracks 81

{T_{k} (l_{k})}_{l_{k}}^{L_{k}} = 0

0439 where Tk(lk)zdi 11(lk), . . . , zikk(lk) is a track of data from the solution of the problem prior to the formulation of (4.4) or if k1 then it is just the first data set in frame one. 0440 Suppose problem (4.4) has been solved and let the solution, i.e., those zero-one variables equal to one, be enumerated by 82

\begin{matrix} {(l_{k} (l_{k + 1}), i_{k + 1} (l_{k + 1}),, i_{k + N} (l_{k + 1}))}_{l_{k + 1} = 1}^{L_{k + 1}} & (4.5) \end{matrix}

0441 with the following exceptions. 0442 a. All zero one variables for which (lk, ik1)(0,0) are excluded. Thus, tracks that initiate on frames after the (k1)th are not included in the list. 0443 b. All zero-one variables whose subscripts have the form lk0 and exactly one nonzero index in the remaining indices ik1, . . . , ikN are excluded. These correspond to false reports on frames pk1, . . . , kN. 0444 c. All variables zlki1 . . . ikN) for which (ik1, . . . , ikN)(0, 0, . . . , 0) and ik(lk)0 in (4.5). In other words, the reports on the last N1 frames in the Tk(lk), zikNk1 are all dummy. 0445 Any solution with this feature corresponds to a terminated track. 0446 Given the enumeration (4.5), one now fixes the assignments on the first two index sets in the list (4.5). The zero index lk10 is added to the enumeration to specify (lk(0), ik1 (0))(0, 0) that is used to represent false reports and tracks that initiate on frame k2 or later, so that the enumeration (4.5) is now 83

\begin{matrix} {(l_{k} (l_{k + 1}), i_{k + 1} (l_{k + 1}))}_{l_{k + 1}}^{L_{k + 1}} = 0. & (4.6) \end{matrix}

0447 Then, for lk11, . . . , Lk1, the lk1th such track is denoted by Tk1(lk1)Tk(lk(lk1)), zik1k1, and the (n1)-tuple Tk1(lk1), zik2k2, . . . , zik1Nk1N will denote a track Tk1(lk1) plus a set of reports zik2k2, . . . , zik1Nk1N, actual or dummy, that are feasible with the track Tk1(lk1). The (N1)-tuple Tk1(0), zik2k2, . . . , zik1Nk1N, will denote a track that initiates in the sliding window, i.e., on subsequent frames. A false report in the sliding window is one with all but one non-zero index ip for some pk2 , . . . , k1N in the (N1)-tuple Tk1(0), zik2k2, . . . , zik1Nk1N. 0448 The corresponding hypothesis about a partition * being true is now conditioned on the truth of the Lk1 tracks existing at the beginning of the N-frame window. (Thus the assignments prior to this sliding window are fixed.) The likelihood function is given by 84

\begin{matrix} L_{} = \underset{T_{k_{1} (l_{k + 1})}, z_{i_{k + 2}},, z_{i_{k + 1 + N}}}{} L_{l_{h + 1} i_{h + 1 + N}}, where L_{l_{k + 1} i_{k + 2} i_{k + 1 + N}} = L_{T_{k + 1} (l_{k + 1})}, L_{z_{i_{k + 1}}^{k + 2}},, L_{z_{i_{k + 1 + N}}^{k + 1 + N}} L_{T_{k + 1} (0)} = 1. & (4.7a) \end{matrix}

0449 Next, define the cost and the zero-one variable by 85

\begin{matrix} Z_{l_{k + 1} i_{k + 1 + N}} = {\begin{matrix} 1 & if {z_{i_{k + 2}}^{k + 2},, z_{i_{k + 1 + N}}^{k + 1 + N} \\ 0 & otherwise, \end{matrix} & (4.7b) \end{matrix}

0450 respectively, so that the track extension problem, which was originally formulated as Maximize L as exactly the same multi-dimensional assignment in (3.4) but with k replaced by k1. Thus, we do not repeat it here.

The Second Approach to Track Maintenance and Initiation 0451 In the first approach the window lengths were the same for both track maintenance and initiation. This can be inefficient in that one can usually use a shorter window for track maintenance than for track initiation. This section addresses such a formulation. 0452 The General Step k. To formulate a problem for track initiation and maintenance we consider a moving window centered as frame k of length IJ1denoted by kI, . . . , k, . . . , kJ In this representation, the window length for track maintenance is J and that for initiation, IJ1. The objective will be to explain the situation at this center and then the move to the same length window at center k1 denoted by k1I, . . . , k1, . . . , k1J, i.e., by moving the frame one frame at time to the right. The explanation from the first step follows hereinafter. 0453 The notation for a track of data is

Tk(lk)zi11(lk), . . . , zipp(lk), . . . , zikk(lk)(4.8)

0454 where the index lk is used for an enumeration of those reports paired together. We also use the notation Tp,k (lk) to denote the sequence of reports belonging to track Tk(lk) but restricted to frames prior to and including p. Thus, 0455 Tk(lk)Tk,k(lk) 0456 Tk(lk)zi11(lk), . . . , zipp(lk)(4.9) 0457 Tk,k(lk)Tp,k(lk)zip1p1(lk), . . . , zik131 1(lk), zikk(lk) for pk1. 0458 Given this notation for the tracks and partition of the data in the frames k-I, . . . , k, . . . , kJ, LTp,k(lk) will denote the accumulated likelihood ratio up to and including frame p(pk) for a track that is declared as existing on frame k as a solution of the assignment problem. In this notation, the likelihood for Tp,k(lk) and that of the association of zik1k1, . . . , zikNkN with track Tk(lk) is given by 0459 LTk(lk)LTp,k(lk)Lip1(lk)ik(lk) for any pk1, 0460 LTk(lk)ik1ikNLTp,k(lk)Lip1(lk)ik(lk)ikN(4.10) respectively. 0461 The cost for the assignment of zik1k1, . . . , zikNkN to track Tk(lk) and the corresponding zero one variable are given by 86

\begin{matrix} \begin{matrix} c_{l_{k} i_{k + 1} i_{k + J}}^{T} = - \ln [L_{T_{k} (l_{k})} L_{i_{k + 1} i_{k + J}}] \\ = - \ln [L_{T_{p, k} (l_{k})} L_{i_{p + 1} (l_{k}) i_{k} (l_{k}) i_{k + 1} i_{k + J}}] \end{matrix} and & (4.11) \end{matrix}

z_{l_{k} i_{k + 1} i_{k + J}}^{T} = {\begin{matrix} 1 & {z_{i_{k - 1}}^{k + 1},, z_{i_{k + 1}}^{k + N}} is assigned to track T_{k} (l_{k}), \\ 0 & otherwise \end{matrix}

0462 respectively. Likewise, for costs associated with the false reports on the frames k-I to k and as associated with zik1k1, . . . , zikJkJ and the corresponding zero-one variable are given by: 88

\begin{matrix} c_{i_{k - I}}^{F} i_{_{i} i_{k + 1}} i_{k + J} = - \ln L_{i_{k - I} i_{k} i_{k + 1} i_{k + J}} Z_{i_{k - 1} i_{k} i_{k + 1} i_{k + J}}^{} = {\begin{matrix} 1 & {if z_{i_{k - r}},, z_{i_{k - I}},, {zi}_{k + J}} \\ are assigned as a track, \\ 0 & otherwise . \end{matrix} & (4.12) \end{matrix}

0463 For the window of frames in the range kI, . . . , k, the easiest explanation of the partition of the reports is based on the definitions 0464 Fp,kipzipp belongs to one of the tracks listed on frame k, i.e., to one of the Tk(lk) for some lk1, . . . , Lk,(4.13) 0465 Tp,k(1, . . . , MpTp,k)0, so that all of those reports not used in the tracks Tk(lk) on frame p are put into the set of available reports Fp,k for the formation of tracks over the entire window. Thus, we formulate the assignment problem as 89

Minimize_{p = k - I}^{k} \underset{i_{p} F_{p, k}}{}_{r = k + 1}^{k + J}_{i_{r} = 0}^{M_{r}} c_{i_{k - r} i_{k}, i_{k + 1} i_{k + J}}^{F} z_{i_{k - I} i_{k} i_{k + 1} i_{k + J}}^{F}

Subject to :_{o = k - I, p q}}^{k} \underset{i_{p} F_{p, k}}{}_{r = k + 1}^{k + J}_{i_{r} = 0}^{M_{r}} z_{i_{k - I}}^{F} i_{k} i_{k + 1} i_{k + J} = 1

for i_{q} F_{q, k} {0} and q = k - I, k,_{p = k - I}^{k} \underset{i_{p} F_{p, k}}{}_{{r = k + 1, r q}}^{k + J}_{i_{r} = 0}^{M_{r}} for i_{q} = 1,, m_{q} and

q = k + 1,, k + J, z_{i_{k - 1} i_{k} i_{k + 1} i_{k + J}} {0, 1} for all i_{k - I},, i_{k}, i_{k + 1},, i_{k + J}

_{p = k + l}^{k + J}_{i_{r} = 0}^{M_{r}} z_{l_{k} i_{k + 1} i_{k + J}}^{T} = 1, i_{q} = 1,, M_{q}, for q = k + 1,, k + J, z_{l_{k} i_{k + 1} i_{k + J}} {0, 1} for all l_{k} > 1, i_{k + 1},, i_{k + J} .

0466 Suppose problem (4.14) has been solved and let the solution, i.e., those zero-one variables equal to one, be enumerated by 90

\begin{matrix} {z_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1}) i_{k + J} (l_{k + 1})}^{T}}_{L_{k + 1 = 1}}^{{\overline{L}}_{K + 1}} {z_{i_{k - I} (l_{k + 1}) i_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1}) i_{k + J} (l_{k + 1})}^{F}}_{L_{k + 1 = {\overline{L}}_{k + 1} + 1}}^{L_{K + 1}} & (4.15) \end{matrix}

0467 with the following exceptions. 0468 a. All zero one variables in the second list for which (ikI, . . . , ik, ik1) (0, . . . ,0,0) are excluded. Thus, tracks that initiate on frames after the (k1 )th are not included in the list. 0469 b. All false reports are excluded, i.e., all zero-one variables in the second list whose subscripts have exactly one nonzero index. 0470 c. All variables zlkik1T, . . . , ikN for which (ik1, . . . , ikN(0,0, . . . , 0) and z(p)Tk(lk)0 for pkK, . . . , k where K0 is user specified. Thus the track Tk(lk) is terminated if it is not observed over KJ1 frames. 0471 Given the enumeration (4.15), one now fixes the assignments on the all index sets up to and including the (k1 )th index sets. 91

\begin{matrix} T_{k + 1} (l_{k + 1}) = {(\begin{matrix} T_{k} (l_{k} (l_{k + 1})), z_{i_{k + 1}}^{k + 1} (l_{k + 1})) for l_{k + 1} = 1,, {\overline{L}}_{k + 1} \\ (z_{i_{k - 1}}^{k - I} (l_{k + 1}),, z_{i_{k} (l_{k + 1}))}^{k}, {z_{i}}_{k + 1}^{k + 1} (l_{k + 1}) for l_{k + 1} = {\overline{L}}_{k + 1} + 1,, l_{k + 1} \end{matrix} & (4.16) \end{matrix}

0472 Thus one can now formulate the assignment problem for the next problem exactly as in (4.14) but with k replaced by k1. Thus, we do not repeat it here. 0473 The Initial Step. Here is one explanation for the initial step. First, assume that NIJ. In this case, we start the track initiation with a solution of (3.5). Let 92

{i_{1} (l_{2}), i_{2} (l_{2}), i_{N} (l_{2}), i_{N + 1} (l_{2})}_{l_{2} = 0}^{L_{2}}

0474 be an enumeration of the solution set of (3.5), i.e., those zero-one variables zi1(i2)i2(l2) . . . iN1(l2)1, including z00. . . 01 corresponding to l20, but excluding all those zero-one variables that are assigned to one and correspond to false reports (i.e., there is exactly one nonzero index in the subscript of zi1i2. . . iN1), all those zero-one variables that are assigned to one and correspond to tracks that initiate on frames higher than I2. Then we fix the data association decisions corresponding to the reports in our list of tracks prior to and including frame k1I2. This defines the k for problem (4.4) and one can then continue the development by adding a frame to the window as in the general case. 0475 If IJ>N, then one possibility is to start the process with N1 frames, and assuming JN, proceed as before replacing I by NJ for the moment, and continue to add frames without lopping off the first frame in the window until reaches a window of length IJ1. Then we proceed as in the previous paragraph. 0476 If IJ<N, then one can solve the track initiation problem (3.5), formulate the problem with the center of the window at k1N1J, enumerate the solutions as above, and lop off the first NJI frames. Then, we proceed just as in the case IJN. 0477 A primary objective in this work has been to demonstrate how multidimensional assignment problems arise in the tracking environment. The problem of track initiation and maintenance has been formulated within the framework of a moving window over the frames of data. The solution of these NP-hard, noisy, large scale, and sparse problems to the noise level in the problem is fundamental to superior track estimation and identification. Thus, one must utilize the special structure in the problems as well take advantage of special information that is available. Since these moving windows are overlapping, there are some algorithm efficiencies that can been identified and that take advantages of the overlap in the windows from one frame of reports to the next. Here is an example of the use of a primal solution of one problem to warm start the solution of the next problem in the sequence. 0478 Suppose we have solved problem (4.4) and have enumerated all those zero-one variables in the solution of (4.4) as in (4.5). Add the zero index lk10, so that the enumeration is 93

\begin{matrix} {(l_{k} (l_{k + 1}), i_{k + 1} (l_{k + 1}),, i_{k + N} (l_{k + 1}))}_{l_{k + 1} = 0}^{L_{k + 1}} & (5.1) \end{matrix}

0479 With this enumeration one can define the cost by

clk1iklNclk(lk1),ik1(lk1), . . . , iN1(lk1)ik1N(5.2)

0480 and the two dimensional assignment problem 94

\begin{matrix} _{2} = Minimize_{l_{k + 1} = 0}^{L_{k + 1}}_{i_{k + 1 + N} = 0}^{M_{k - 1 + N}} c_{l_{k + 1}}^{2} i_{k + 1 + N} z_{l_{k + 1} i_{k + 1 + N}}^{2} v_{2} (z^{2}) Subject to :_{i_{k + 1 + N} = 0}^{M_{k + 1 + N}} z_{l_{k + 1} i_{k + 1 + N}}^{2} = 1, l_{k + 1} = 1,, L_{k + 1},_{l_{k + 1} = 0}^{L_{k + 1}} Z_{l_{k + 1} i_{K + 1 + N}}^{2} = 1, l_{k + 1 + N} = 1,, M_{k + 1 + N}, z_{l_{k + 1} i_{k + 1 + N}}^{2} {0, 1} for all l_{l_{k + 1}}, i_{k + 1 + N} . & (5.3) \end{matrix}

0481 Let w be an optimal or feasible solution to this two-dimensional assignment problem and define 95

\begin{matrix} {\overline{z}}_{i_{k + 1} i_{k + N} i_{k + 1 + N}} = {\begin{matrix} 1 & if (i_{k + 1},, i_{k + N} = (i_{k + 1} (l_{k + 1},, i_{k + N} (l_{k + 1})) \\ and w_{l_{k + 1} i_{k + 1 + N}} = 1 for some l_{k + 1} = 1,, L_{k + 1} \\ or if (l_{k + 1}, i_{k + 1 + N}) = (0, 0), \\ 0 & otherwise \end{matrix} & (5.4) \end{matrix}

0482 This need not satisfy the constraints in that there are usually many objects left unassigned. Thus, one can complete the assignment by using the zero-one variables in (4.4) with k replaced by k1 with exactly one nonzero index corresponding to any unassigned object or data report. 0483 For the dual solutions, the multipliers arising from the solution of the two dimensional assignment problem (5.3) corresponding to the second variable, i.e., 96

{u_{i_{k + 1 + N}}^{k + 1 + N}}_{i_{k + 1 + N} = 0}^{M_{k + 1 + N}} .

0484 These are good initial values for use in a relaxation scheme 11,12. Finally, note that one can also develop a warm start for problem (4.14) in a similar fashion. 0485 Multidimensional assignment problems govern the central problem of data association in multisensor and multitarget tracking, i.e., the problem of partitioning observations from multiple scans of the surveillance region and from either single or multiple sensors into tracks and false alarms. This fundamental problem can be stated as 97

\begin{matrix} Minimize_{i_{1} = 0}^{M_{1}}_{i_{N} = 0}^{M_{N}} c_{i_{1} i_{N}} z_{i_{1} i_{N}} Subject to :_{i_{2} = 0}^{M_{2}}_{i_{N} = 0}^{M_{N}} z_{i_{1} i_{N}} = 1,_{i_{1} = 1, l,, M_{1}},_{i_{1} = 0}^{M_{1}}_{i_{p - 1} = 0}^{M_{p - 1}}_{i_{p + 1} = 0}^{M_{p + 1} = 0}_{i_{N} = 0}^{M_{N}} z_{i_{1} i_{N} = 1} for i_{p} = 1,, M_{p} and p = 2,, N - 1,_{i_{1} = 0}^{M_{1}}_{i_{n - 1} = 0}^{M_{N - 1}} z_{i_{1} i_{N}} = 1, i_{N} = 1,, M_{N}, z_{l_{1} i_{N}} {0, 1} for all i_{1},, i_{N}, for i_{k} = 1,, M_{k} and k = 2,, N, z_{i_{1}} 1_{N + 1} {0, 1} for all i_{1},, i_{N + 1} & (1.1) \end{matrix}

0486 where c0 . . . 0 is arbitrarily defined to be zero and is included for notational convenience. One can modify this formulation to include multiassignments of one, some, or all of the actual reports. The zero index is used in representing missing data, false alarms, initiating and terminating tracks. In these problems, we assume that all zero-one variables zil. . . iN with precisely one nonzero index are free to be assigned and that the corresponding cost coefficients are well-defined. (This is a valid assumption in the tracking environment.) Although not required, these cost coefficients with exactly one nonzero index can be translated to zero by cost shifting without changing the optimal assignment. Finally, this formulation is of sufficient generality to include the symmetric problem and the asymmetric inequality problem. 0487 The data association problems in tracking that are formulated as (1.1) have several characteristics. They are normally sparse, the cost coefficients are noisy and the problem is NP-hard, but it must be solved in real-time. The only known methods for solving this NP-hard problem optimally are enumerative in nature, with branch-and-bound being the most efficient; however, such methods are much too slow for real-time applications. Thus one must resort to suboptimal approaches. Ideally, any such algorithm should solve the problem to within the noise level, assuming, of course, that one can measure this noise level in the physical problem and the mathematical method provides a way to decide if the criterion has been met. 0488 There are many algorithms that can be used to construct suboptimal solutions to NP-hard combinatorial optimization problems. These include greedy (and its many variants), relaxation, simulated annealing, tabu search, genetic algorithms, and neural netork algorithms. For the three dimensional assignment problem, Pierskalla developed the tri-substitution method, which is a variant of the simplex method. Frieze and Yadegar introduced a method based on Lagrangian relaxation in which a feasible solution is recovered using information provided by the relaxed solution. Our choice of approaches is strongly influenced by the need to balance real-time performance and solution quality. Lagrangian relaxation based methods have been used successfully in prior tracking applications. An advantage of these methods is that they provide both an upper and lower bound on the optimal solution, which can then be used to measure the solution quality. These works extend the method of Frieze and Yadegar to the multidimensional case.

Probabilistic Framework for Data Association. (ABP) 0489 The goal of this section is to explain the formulation of the data association problems that governs large classes of data association problems in centralized or hybrid centralized-sensor level multisensor/multitarget tracking. The presentation is brief; technical details are presented for both track initiation and maintenance in the work of Poore. The formulation is of sufficient generality to cover the MHT work of Reid, Blackman and Stein and modifications by Kurien to include maneuvering targets. As suggested by Blackman, this formulation can also be modified to include target features (e.g., size and type) into the scoring function. The recent work of Poore and Drummond significantly extends the work of this section to new approaches for multiple sensor centralized tracking. Future work will involve extensions to track-to-track correlation. 0490 The data association problems for multisensor and multitarget tracking considered in this work are generally posed as that of maximizing the posterior probability of the surveillance region (given the data) according to

Maximize p(ZN)*(2.1)

0491 where ZN represents N data sets, of the data (and thus induces a partition of the data), 0492 0493 0 0494 P(ZN) is the posterior probability of a partition true given the data ZN. The term partition is defined below; however, this framework is currently sufficiently general to cover set packings and coverings. 0495 Consider N data sets Z(k) (kI, . . . , N) each of Mk reports 98

{z_{i_{k}}^{k}}_{i_{k} = 1}^{M_{k}},

0496 and let ZN denote the cumulative data set defined by 99

\begin{matrix} Z (k) = {z_{i_{k}}^{k}}_{i_{k} = 1}^{M_{k}} and Z^{N} = {Z (1),, Z (N)} & (2.2) \end{matrix}

0497 respectively. In multisensor data fusion and multitarget tracking the data sets Z(k) may represent different classes of objects, and each data set can arise from different sensors. For track initiation the objects are measurements that must be partitioned into tracks and false alarms. In our formulation of track maintenance, which uses a moving window over time, one data set will be tracks and remaining data sets will be measurements which are assigned to existing tracks, as false measurements, or to initiating tracks. In sensor level tracking, the objects to be fused are tracks. In centralized fusion, the objects may all be measurements that represent targets or false reports, and the problem is to determine which measurements emanate from a common source. 0498 We specialize the problem to the case of set partitioning defined in the following way. First, for notational convenience in representing tracks, we add a zero index to each of the index sets in (2.2), a dummy report z0k to each of the data sets Z(k) in (2.2), and define a track of data as (zil1, ziNN) where ik and zikk can now assume the values of 0 and z0k, respectively. A partition of the data will refer to a collection of tracks of data wherein each report occurs exactly once in one of the tracks of data and such that all data is used up; the occurrence of dummy report is unrestricted. The dummy report z0k serves several purposes in the representation of missing data, false reports, initiating tracks, and terminating tracks. The reference partition is that in which all reports are declared to be false. 0499 Next under appropriate independence assumptions one can show 100

\begin{matrix} \frac{P (= | Z^{N})}{P (=^{0} | Z^{N})} L \underset{(i_{1} i_{N})}{} L_{i_{1} i_{N}}, & (2.3) \end{matrix}

0500 Lil . . . iN is a likelihood ratio containing probabilities for detection, maneuvers, and termination as well as probability density functions for measurement errors, track initiation and termination. Then if 0501 Using (2.3) and the zero-one variable zi1 . . . N1 i1, . . . N) 0502 0503 .i f(i ,i 0504

101

\begin{matrix} Minimize_{i_{1} i_{N}} c_{i_{1} i_{N}} z_{i_{1} i_{N}} \begin{matrix} Subject to_{i_{2} i_{3} i_{N}} z_{i_{1} i_{N}} = 1 & (i_{1} = 1,, M_{1}) \end{matrix} \begin{matrix} _{i_{1} i_{3} i_{N}} z_{i_{1} i_{N}} = 1 & (i_{2} = 1,, M_{2}) \end{matrix}_{i_{1} i_{p - 1} i_{p + 1} i_{N}} z_{i_{1} i_{N}} = 1 (i_{p} = 1,, M_{p} and p = 2, N - 1)_{i_{1} i_{2} i_{N - 1}} z_{i_{1} i_{N}} = 1 (i_{N} = 1,, M_{N}) z_{i_{1} i_{N}} {0, 1} for all i_{1},, i_{N} . & (2.5) \end{matrix}

0505 where c0 . . .0 is arbitrarily defined to be zero. Here, each group of sums in the constraints represents the fact that each-non-dummy report occurs exactly once in a track of data. One can modify this formulation to include multiassignments of one, some, or all the actual reports. The assignment problem (2.5) is changed accordingly. For example, if z1kk is to be assigned no more than, exactly, or no less than nikk times, then the 1 in the constraint (2.5) is changed to , , nikk, respectively. Modifications for group tracking and multiresolution features of the surveillance region will be addressed in future work. In making these changes, one must pay careful attention to the independence assumptions that need not be valid in many applications. 0506 Expressions for the likelihood ratios Li1, . . . , iN, can be found in the work of Poore. These expressions include the developments of Reid, Kurien, and Stein and Blackman. What's more, they are easily modified to include target features and to account for different sensor types. In track initiation, the N data sets all represent reports from N sensors, possibly all the same. For track maintenance, we use a sliding window of N data sets and one data set containing established tracks. The formulation is the same as above except that the dimension of the assignment problem is now N1. 0507 Overview of the Lagrangian Relaxation Algorithm. (ABP) Having discussed the N-dimensional assignment problem (1.1), we now turn to a description of the Lagrangian relaxation algorithm. The algorithm will proceed iteratively for a loop k1, . . . , N2. At the completion, there remains one twodimensional assignment problem that provides the last step which yields an optimal (sometimes) or near-optimal solution to the original N-dimensional assignment problem. In step k of this loop (summarized in Section 4) one starts with the following (Nk1)-dimensional assignment problem with one change in notation. If k1, then the index notation ll and Ll are to be replaced by i1 and Ml, respectively. 102

\begin{matrix} Minimize_{l_{k} i_{k + 1} i_{N}} c_{l_{k} i_{k + 1} i_{N}} c_{N}^{N - k + 1} z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} Subject to_{i_{k + 1} i_{N}} z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} = 1, l_{k} = 1,, L_{k},_{l_{k} i_{k + 2} i_{N}} z_{l_{k} i_{k + 1} i_{k + 1} i_{N}}^{N - k + 1} = 1, i_{k + 1} = 1,, M_{k + 1},_{l_{k} i_{k + 1} i_{p + 1} i_{N}} z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} = 1, for i_{p} = 1,, M_{p} and p = k + 2,, N - 1,_{l_{k} i_{k + 1} i_{N - k + 1}} z_{l_{i} i_{k + 1} i_{N}}^{N - k + 1} = 1, i_{N} = 1,, M_{N}, z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} {0, 1} for all l_{k}, i_{k + 1},, i_{N} & (3.1) \end{matrix}

0508 To ensure that a feasible solution of (3.1) always exists for a sparse problem, all variables with exactly one nonzero index (i.e., variables of the form zlk00Nk1 for lk, . . . , Lk and zl0iP00Nk1 for ip1, . . . , Mip and pk12, . . . , N) assumed free to be assigned with the corresponding cost coefficients being well-defined. This assumption is valid in the tracking environment. 0509 Subsection 3.1 presents some of the properties associated with the Lagrangian relaxation of (3.1) based on relaxing the last (Nk)-sets of constraints to a two-dimensional one. A new approach to the problem of recovering a high quality feasible solution of the original (Nk1 )-dimensional problem given a feasible solution (optimal or suboptimal) of the relaxed two-dimensional problem is described hereinafter. A summary of the relaxation algorithm is given hereinafter, following which we establish the maximization of the Lagrangian dual (an important aspect of the relaxation procedure) to be an unconstrained nonsmooth optimization problem and then present a method for computing the subgradients. 0510 The Lagrangian Relaxed Assignment Problem. The (Nk1)-dimensional problem (3.1) has (Nk1) sets of constraints. A (Mp1)-dimensional multiplier vector (i.e., p RMp1) associated with the p-th constraint set will be denoted by up(u0p, u1p, . . . , uMpp) with u0p0 being fixed for each pk2, . . . , N and included for notational convenience only. Now, the (Nk1)dimensional assignment problem (3.1) is relaxed to a two-dimensional assignment problem by incorporating (Nk1) sets of constraints into the objective function via the Lagrangian. Although any (Nk1) constraint sets can be relaxed, we choose the last (Nk1) sets of constraints for convenience. The relaxed problem is 103

\begin{matrix} _{N - k + 1} (u^{k + 2},, u^{N}) Minimize_{N - k + 1} (z^{N - k + 1}, u^{k + 2}, u^{N}) Minimize_{l_{k} l_{k + 1} i_{N}} c_{l_{k} i_{k + 1 i_{N}}}^{N - k + 1} z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} +_{p = k + 2}^{N}_{i_{p} = 0}^{M_{p}} u_{i_{p}}^{p} [_{l_{k} i_{k + 1} i_{p - 1} i_{p + 1} i_{N}} - 1] Minimize_{l_{k} i_{k + 1} i_{N}} [c_{l_{k} 1_{k + 1}}^{N - k + 1} +_{p = k + 2}^{N}_{1_{p} = 0}^{M_{p}} u_{i_{p}}^{p}] z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} -_{p = k - 2}^{N}_{l_{p} = 0}^{M_{p}} u_{i_{p}}^{p} Subject To_{i_{k + 1} i_{k + 2} i_{N}} z_{l_{k} l_{k + 1} l_{k + 2} l_{N}}^{N - k + 1} = 1, l_{k} = 1, L_{k},_{l_{k} i_{k + 2} i_{N}} z_{l_{k} l_{k + 1} i_{k + 2} i_{N}}^{N - k + 1} = 1, i_{k + 1} = 1,, M_{k + 1} & (3.2) \end{matrix}

0511 One of the major steps in the algorithm is the maximization of Nk1(uk2, . . . , uN) with respect to the multipliers (uk2, . . . , uN). It turns out that Nk1 is a concave, continuous, and piecewise affine function of the multipliers (uk2, . . . , uN), so that the maximization of Nk1 is a problem of nonsmooth optimization. Since many of these algorithms require a function value and a subgradient of Nk1 at any required multiplier value (uk2, . . ., uN) , we address this problem in the next subsection. We note, however, that there are other ways to maximize Nk1 and the next subsection just addresses one such method.

Properties Lagrangian Relaxed Assignment Problem 0512 For a function evaluation of Nk1, we show that an optimal (or suboptimal) solution of this relaxed problem (3.2) can be constructed from that of a two-dimensional assignment problem. Then, the nonsmooth characteristics of Nk1 are addressed, followed by a method for computing the function value and a subgradient. 0513 Evaluation of Nk1. Define for each (lk, ik1) an index (jk2, . . . ,jN)(jk2(lk, ik1), . . . , jN(lk, ik1) ) and a new cost function clkik12 by 104

\begin{matrix} (j_{k + 2} (l_{k}, i_{k + 1}),, j_{N} (l_{k}, i_{k + 1})) = \arg \min {c_{l_{k} i_{k + 1} i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{o} | & (3.3) \\ i_{p} = 0, 1,, M_{p} and p = k + 2,, N} c_{l_{k} i_{k + 1}}^{2} = c_{l_{k} i_{k + 1} j_{k + 2} (l_{k}, i_{k + 1}) j_{N} (l_{k}, i_{k + 1})}^{n - k + 1} +_{p = k + 2}^{N} u_{j_{p} (l_{k}, i_{k + 1})}^{p} for (l_{k}, i_{k + 1}) (0, 0) c_{00}^{2} =_{i_{k + 2} l_{N}} Minimum {0, c_{00 i_{k + 2} i_{n}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p}} \end{matrix}

0514 Given an index pair (lk, ik1), (jk2, . . . , jN) need not be unique, resulting in the potential generation of several subgradients (3.11). 105

\begin{matrix} {\hat{}}_{N - k + 1} (U^{k + 2},, u^{N}) = Minimize {\hat{}}_{N - k + 1} (z^{2}; u^{k + 2},, u^{N})_{l_{k} = 0}^{L_{k}}_{i_{k + 1} = 0)}^{L_{k - 1}} c_{l_{k} i_{k + 1}}^{2} z_{l_{k} i_{k + 1}}^{2} -_{p = k + 2}^{N}_{i_{p} = 0}^{M_{p}} u_{i_{p}}^{p} Subj . To_{i_{k + 1} = 0}^{M_{k + 1}} z_{l_{k} i_{k + 1}}^{2} = 1, l_{k} = 1,, L_{k},_{l_{k} = 0}^{L_{k}} z_{l_{k} i_{k + 1}}^{2} = 1, i_{k} = 1, M_{k + 1}, z_{l_{k} i_{k + 1}}^{2} {0, 1} for all l_{k}, i_{k + 1} . & (3.4) \end{matrix}

0515 As an aside, two observations are in order. The first is that the search procedure needed for the computation of the relaxed cost coefficients in (3.3) is the most computationally intensive part of the entire relaxation algorithm. The second is that a feasible solution zNk1 of a sparse problem (3.1) yields a feasible solution z2 of (3.4) via the construction 106

Z_{l_{k} i_{k + 1}}^{2} = {\begin{matrix} 1, & if z_{l_{k} i_{k + 1} i_{k + 2} i_{N}}^{N - k + 1} = 1, for some (i_{k + 2}, i_{N}); \\ 0, & otherwise . \end{matrix}

0516 thus, there are generally solutions other than the one nonzero index solution. 0517 The following Theorem 3.1 gives a method for evaluating Nk1 and states that an optimal solution of (3.2) can be computed from that of (3.4). Furthermore, if the solution of either of these two problems is -optimal, then so is the other. The converse is contained in Theorem 3.2. 0518 Theorem 3.1. Let 2 dimensional assignment problem (3.4) and define Nk1

107

\begin{matrix} _{_{1} i_{k + 2} N}^{1} =_{l_{k} i_{k + 1}}^{2} if (i_{k + 2}, i_{N}) = (j_{k + 2},, j_{N}) and (l_{k}, i_{k + 1}) (0, 0) & (3.5) \\ _{i_{k + 1} i_{k + 2} N}^{k + 1} = 0 if (i_{k + 2}, i_{N}) (j_{k + 2},, j_{N}) and (l_{k}, i_{k + 1}) (0, 0) \\ _{0 i_{k + 2} i_{N}}^{k + 1} = 1 if c_{00 i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p} 0; \\ _{0 i_{k + 2} i_{N}}^{k + 1} = 0 if c_{00 i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p} > 0. \end{matrix}

0519 Then Nk1 0520 problem and Nk1Nk1 k2. . . , n {circumflex over ()}Nk12 k2 N0521 addition, 2 0522 Nk1 0523 Nk1k2 n {circumflex over ()}Nk1k2 N0524 Proof. Let Nk1 2 0525 Nk1Nk1 N {circumflex over ()}Nk12 k2 N) 0526 objective function values of (3.2) and (3.4), respectively. Direct verification shows that Nk1 in (3.2) and Nk1Nk1 k2 N {circumflex over ()}Nk12 k2 . . . , N remainder of the proof, assume that 2 0527 Let XNk1

108

\begin{matrix} _{l_{k} i_{k - 1}}^{2} =_{i_{k - 2} i_{N}}_{l_{k} i_{k + 1} i_{k + 2} i_{N}}^{N - k + 1} for (l_{k}, i_{k + 1}) (0, 0);_{00}^{2} = 1 if (l_{k}, i_{k + 1}) = (0, 0) and c_{00 i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p} 0 for some (i_{k + 2},, i_{N});_{00}^{2} = 0 if (l_{k}, i_{k + 1}) = (0, 0) and C_{00 i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p} > 0 for all (i_{k + 2},, i_{N}) . & (3.6) \end{matrix}

0528 Note that X2 0529 Nk1XNk1 N{circumflex over ()}Nk1(X2 k2 N{circumflex over ()}Nk1 (2 k2 N Nk1 k2 N0530 XNk1 Nk1 0531 optimal solution of (3.2). Next, Nk1k2 N {circumflex over ()}Nk1 (N2 N Nk1 k2 N {circumflex over ()}Nk1(2 k2 N 2 0532 With the exception of one equality being converted to an inequality, the following theorem is a converse of Theorem 5 3.1. 0533 Theorem 3.2. Let Nk1 0534 and define 2

109

\begin{matrix} _{l_{k} i_{k - 1}}^{2} =_{i_{k - 2} i_{N}}_{l_{k} i_{k + 1} i_{k + 2} i_{N}}^{N - k + 1} for (l_{k}, i_{k + 1}) (0, 0);_{00}^{2} = 1 if (l_{k}, i_{k + 1}) = (0, 0) and C_{00 i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p} 0 for some (i_{k + 2},, i_{N});_{00}^{2} = 0 if (l_{k}, i_{k + 1}) = (0, 0) and C_{00 i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p} > 0 for all (i_{k + 2},, i_{N}) . & (3.7) \end{matrix}

0535 Then 2 Nk1Nk1; k2 N{circumflex over ()}Nk1(2 NNk1 2 0536 (3.4) Nk1Nk1 k2 N{circumflex over ()}Nk1(2 k2 N and Nk1(k2 N {circumflex over ()}Nk1k2 N0537 Proof: Let Nk1 2 Nk1(Nk1 k2 N and {circumflex over ()}Nk1(2 k2 N0538 objective function values of (3.2) and (3.4), respectively. 0539 Direct verification shows that 2 0540 (3.4) and Nk1Nk1 k2 N{circumflex over ()}Nk1(2 k2 N0541 remainder of the proof, assume that Nk1 0542 and construct 2 {overscore ()}Nk1 0543 by replacing Nk1 {overscore ()}Nk1 0544 Nk1{overscore ()}Nk1 k2 N{circumflex over ()}Nk1(2 k2 NNk1Nk1; k2 N Nk1 0545 inequality is in fact an equality. This proves the theorem. 0546 The Nonsmooth Optimization Problem. Next, we address the nonsmooth properties of the function Nk1k2 N explained in the following theorem. 0547 Theorem 3.4 Let Nk1 Nk1Nk1) be the object function value of the (Nk1 ) -dimensional assignment problem in equation (3.1) corresponding to a feasible solution zNk1 of the constraints in (3.1), and let {overscore (z)}Nk1 be an optimal solution of (3.1). Then, Nk1k2 Nis piecewise affine, concave and continuous in k2 N and

Nk1(uk2, . . . , uN)vNk1({overscore (z)}Nk1)vNk1(zNk1).

0548 Most of the algorithms for nonsmooth optimization are based on generalized gradients, called subgradients, given by the following definition for a concave function. 0549 Definition 3.5. At u(uk2, . . . , uN) the set Nk1 (u) is called a subdifferential of Nk1 and is defined for the concave function Nk1(u) by

Nk1(u)gRMk21x . . . x RMm1Nk1()Nk1(u)gT(u) for all RMk21 x . . . xRMN1.

0550 where g0p0pu0p0 are all permanently fixed. (Recall that these were used for notational conveience only.) A vector gNk1 is called a subgradient. 0551 There is a large literature on such problems, and we have tried a variety of methods including subgradient methods and bundle methods. Of these, we have determined that for a fixed number of nonsmooth iterations (e.g., twenty) , the bundle trust method Schramm and Zowe provides excellent quality solutions with the fewest number of function and subgradient evaluations, and is therefore our currently recommended method. 0552 An Algorithm for Computing Nk1 a Subgradient. Most current software for maximizing the concave function Nk1 requires the value of the function and a subgradient at a point (uk2, . . . , uN). Based on the previous two sections, this can be summarized as follows. 0553 1. Starting with problem (3.1), form the relaxed problem (3.2); 0554 2. To solve (3.2) optimally, defined the two dimensional assignment problem (3.4) via the transformation (3.3); 0555 3. Solve the two-dimensional problem (3.4) optimally; 0556 4. Reconstruct the optimal solution, say Nk1, of (3.2) via equation (3.5) as in Theorem 3.1; 0557 5. Compute the function 110

\begin{matrix} \begin{matrix} _{N - k + 1} (u^{k + 2},, u^{N})_{N - k + 1} ({\hat{}}^{N - k + 1}; u^{k + 2},, u^{N}) \\ _{l_{k} i_{k + 1}, i_{N}} c_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} {\hat{}}_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} \end{matrix}_{p = k + 2}^{N}_{I_{p} = 0}^{M_{p}} u_{i_{p}}^{p} [_{l_{k} i_{k + 1} i_{p - 1} i_{p + 1} i_{N}} {\hat{}}_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} - 1]; & (3.10) \end{matrix}

0558 6. A subgradient is given by 111

\begin{matrix} \begin{matrix} g_{i_{p}}^{p} = \frac{_{N - k + 1} (u^{k + 2},, u^{N})}{u_{i_{p}}^{p}} \\ = [_{l_{k} i_{k + 1} i_{p - 1} i_{p + 1} i_{N}} {\hat{}}_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} - 1] \end{matrix} for i_{p} = 1,, M_{p} and p = k + 2,, N . & (3.11) \end{matrix}

0559 Several remarks are in order. First, the optimal solution of the minimiation problem (3.2) is required before one can remove the minimum sign, replace zNk1 by the minimizer Nk1 and differentiate with respect to u1pp to obtain a subgradient as in (3.11). If Nk1 is an approximate solution of (3.2), then the subgradient and function values are only approximate with accruacy depending on that of Nk1. Although one can evaluate the sums in (3.10) and (3.11) in a straightforward manner, another method is based on the following observation. Given the multiplier vector uk2, . . . , uN), let (lk(lk1), ik1(lk1)lk1Lk10, be an enumeration of indices lk, ik1 of w2 (or the first 2 indices of wNk1 constructed in equation 3.5)) such that wlk(lk1), ik1(lk1)1 for (lk(lk1), ik1(lk1))(0,0) and (lk(lk1), ik1 (lk1)(0,0) for lk10 regardless of whether w0021 or not. (The latter can only improve the recovered feasible solution.) The evaluation of the bracketed quantity in (3.11) for a specific ip>1 and pk2, . . . ,N is one minus the number of times the index value ip appears the (pk1)th position of the (Nk1)tuple in the list 112

{(({l_{k} (l_{k + 1}), i_{k + 1} (l_{k + 1}), j_{k + 2},, j_{N})})}_{l_{k + 1} = 0}^{L_{k + 1}} .

0560 Finally, we have presented a method for computing one subgradient. If Nk1 is the unique optimal solution of (3.2), then Nk1(u) is differentiable at u, g is just the gradient at u, and the subdifferential Nk1(u)g is a singleton. If, corresponding to u,Nk1 is not unique, then there are finitely many such solutions, say Nk1(1), . . . , Nk1 (m) with respective subgradients g(1), . . . , g(m), the subdifferential (u) is the convex hull of g(1), . . . , g(m). These nonunique solutions arise in two ways. First, if the optimal solution of the two dimensional assignment problem is not unique, then one can general all optimal solutions of the two dimensional assignment problem (3.4). Corresponding to each of these solutions and to each index pair (lk, ik1)in each solution, compute the indices (jk2, . . . , jN) in (3.3). If these j's are not unique, then we can enumerate all the possible optimal solutions Nk1 of (3.2). Given these solutions, one can then compute the corresponding subgradients from (3.11). 0561 In most nonsmooth optimization algorithms, one uses an epsilon-subdifferential 0562 Definition 3.x. At u(uk2, . . . , uN the set Nk1 (u) is called an epsilon subdifferential of Nk1 and is defined for the concave funtion Nk1 (u) by

Nk1(u)gRMk21 x . . . xRMN1Nk1()Nk1(u)gT(u) for all RMk21x . . . xRMN1.(3.9)

0563 where g0p0pu0p0 are all permanently fixed. (Recall that these were used for notational convenicence only.) A vector g Nk1(u) is called a subgradient. 0564 HOW DO YOU COMPUTE g Nk1(u) 0565 The Recovery Procedure. The next objective is to explain a recovery procedure, i.e., given a feasible (optimal or suboptimal) solution 2 Nk1 via Theorem 3.1), generate a feasible solution zNk1 of equation (3.1) which is close to 2 specified. We first assume that no variables in (3.1) are presassigned to zero; this assumption will removed shortly. The difficulty with the solution Nk1 it need not satisfy the last (Nk1) sets of constraints in (3.1). (Note, however, that if 2 (3.4) and Nk1 relaxed constraints, then Nk1 recovery proceudre described here is designed to preserve the 0-1 character of the solution 2 0566 If lkik121 and lk0 or ik10, the corresponding feasible solution zNk1 of (3.1) is construed so that zl0ip0 . . . 0Nk11 for some (ik2, . . . ,iN). By this reasoning, variables of the form z00ik2iNNk1 can be assigned a value of one in the recovery problem only if 0021. However, variables z00ik2iNNk1 will be treated differently in the recovery procedure in that they can be assigned either zero or one independent of the value of 002. This increases the feasible set of the recovery problem, leading to a potentially better solution. 0567 Let (lk(lk1), ik1(lk1)lk1Lk10 be an enumeration of indices lk, ik1 of 2 0568 equation (5)) such that lk(lk1),ik1(lk1)21 for (lk(lk1), ik1(lk1))(0,0) and (lk(lk1), ik1(lk1))(0,0) for lk10 regardless of whether 0021 or not. To define the Nk dimensional assignment problem that resores feasibility, let 113

\begin{matrix} C_{l_{2} i_{3} i_{N}}^{N - 1} = c_{l_{i} (l_{2}) i_{2} (l_{2}) i_{3} i_{N}}^{N} for k = 1 \begin{matrix} C_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} = C_{l_{k} l_{k + 1} i_{k + 1} l_{k + 1} i_{k + 2} i_{N}}^{N - k + 1} \\ = c_{i_{1} (l_{2 k + 1}) i_{2} (l_{2 k + 1}) i_{3} (l_{3 k + 1}) i_{k} (l_{kk + 1}) i_{k + 1} (l_{k + 1}) i_{k + 2 i_{N}}}^{N} \end{matrix} for k 2 and l_{k} = 0,, L_{k} & (3.12) \end{matrix}

0569 where

lm . . . k1lmolm1o . . . olk(lk1)lm(lm1( . . . (lk(lk1)) . . . ))(3.13)

0570 for m2, . . .k1 and where o denotes function composition. For example l23l2(l3) and l234l2o l3(l4)l2(l3(l4)). 0571 Then the (Nk)-dimensional assignment problem that restores feasibility is 114

\begin{matrix} Minimize_{l_{k + 1} i_{k + 2} i_{N}} c_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} z_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} Subject to_{i_{k + 2} i_{N}} z_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} = 1, l_{k + 1} = 1,, L_{k + 1},_{i_{k + 1} i_{k + 3} i_{N}} z_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} = 1, i_{k + 2} = 1,, M_{k + 2},_{l_{k} i_{k + 1} i_{p - 1} i_{p + 1} i_{N}} z_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} = 1, for i_{p} = 1,, M_{p} and p = K + 3,, N - 1,_{l_{k + 1} i_{k + 2} i_{N - 1}} z_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} = 1, i_{N} = 1,, M_{N} z_{l_{k + 1} i_{k + 2} i_{N}}^{N - k} {0, 1} for all l_{k + 1}, i_{k + 2},, i_{N} . & (3.14) \end{matrix}

0572 Let Y be an optimal or feasible solution to this Nk dimensional assignment problem. The recovered feasible solution zN is defined by 115

\begin{matrix} N_{i_{1} i_{2} i_{3} i_{N}} = {\begin{matrix} 1 & \begin{matrix} if i_{1} = i_{1} (l_{2 k + 1}), i_{2} = i_{2} (l_{2 k + 1}), \\ i_{3} = i_{3} (l_{2 k + 1}), i_{k} = i_{k} (l_{kk + 1}), \\ i_{k + 1} (l_{k + 1}) and Y_{l_{k + 1} i_{k + 2} i_{N}} = 1, \end{matrix} \\ 0 & otherwise . \end{matrix} & (3.14) \end{matrix}

0573 Said in a different way, the recovered reasible solution zN corresponding to the multiplier set uk2, . . . , uN is then defined by 116

z_{i_{1} (l_{2 k + 1}) i_{2} (l_{2 k + 1}) i_{3} (l_{3 k + 1}) i_{k} (l_{kk + 1}) i_{k + 1} (l_{k + 1}) i_{k + 2} i_{N}}^{N} = {\begin{matrix} 1 & if Y_{k_{k + 1} i_{k + 2} i_{N}} = 1; \\ 0 & otherwise \end{matrix}

0574 where lmk1 is defined in (3.13) and denotes function composition. 0575 The next objective is to remove the asumption that all cost coefficients are defined and all zero-one variables are free to be assigned. We first note that the above construction of a reduced dimension assignment problem (3.14) is valid as long as all cost coefficients cNk are defined and all zero-one variables in zNk are free to be assigned. Modifications are necessary for sparse problems. If the cost coefficient clk(lk1)ik1(lk1)ik2. . . iNNk1 is well-defined and the zero-one variable zlk(lk1)ik1(lk1)ik2. . . iNNk1 free to be assigned to zero or one, then define clk1ik2. . . iNNkclk(lk1)ik1(lk1)ik2. . . iNNk1 as in (3.12) with zlk1ik2. . . iNNk being free to be assigned. Otherwise, zlk1ik2. . . iNNk is preassigned to zero or the corresponding arc is not allowed in the feasible set of arcs. To ensure that a feasible solution exists, we now only need ensure that the variables zlk10 . . . 0Nk are free to be assigned for lk10, 1, . . . , Lk1 with the corresponding cost coefficients being well-defined. (Recall that all variables of the form zlk0 . . . 0Nk1 and z0 . . . 0ip0 . . . 0Nk1 are free (to be assigned) and all coefficients of the form clk0 . . . 0Nk1 and 0 . . . 0ip0 . . . 0Nk1 are well-defined for lk0, . . . ,Lk and ip0, . . . , Mp for pk1, . . . , N.) This is accomplished as follows: If the cost coefficient clk(lk1) ik1(lk1)0 . . . 0Nk1 is well-defined and zlk(lk1) ik1(lk1)0 . . . 0Nk1 is free, the define clk10 . . . 0Nkclk(lk1)ik1(lk1)0 . . . 0Nk1 with zlk10 . . . 0Nk being free. Otherwise, since all variables of the form zlk0 . . . 0Nk and z0ik10 . . . 0Nk are free to be assigned with the corresponding costs being well-defined, set clk10 . . . 0Nkclk(lk1)00 . . . 0Nk c0ik1(1k1)0 . . . 0Nk1 where the first term is omitted if lk(lk1)0 and the second, if ik1(lk1)0. For lk(lk1)0 and ik1(lk1)0, define c0 . . . 0Nkc0 . . .0Nk1. 0576 The last step kN2. The description of the algorithm ends with kN2. The resulting recovery problem defined in (3.14) with kN2 is 117

_{2} Minimize_{l_{N - 1} = 0}^{L_{N - 1}}_{i_{N} = 0}^{M_{N}} c_{l_{N - 1} i_{N}}^{2} z_{l_{N - 1} i_{N}}^{2} V_{2} (z^{2})

Subject to_{i_{N} = 0}^{M_{N}} z_{l_{N - 1} i_{N}}^{2} = 1, 1_{N - 1} = 1,, L_{N - 1},_{l_{N - 1} = 0}^{L_{N - 1}} z_{l_{N - 1} i_{N}}^{2} = 1, l_{N} = 1,, M_{N}, z_{l_{N - 1} i_{N}}^{2} {0, 1} for all l_{N - 1}, i_{N} .

0577 Let Y be an optimal or feasible solution to this 2-dimensional assignment problem. The recovered feasible solution zN is defined by 118

\begin{matrix} {\hat{z}}_{i_{1} i_{2} i_{3} i_{N}}^{N} = {\begin{matrix} 1 & \begin{matrix} if i_{1} = i_{1} (l_{2 N - 1}), i_{2} = i_{2} (l_{2 N - 1}), \\ i_{3} = i_{3} (l_{3 N - 1}),, \\ i_{N - 2} = i_{N - 2} (l_{N - 2 N - 1}), i_{N - 1} (l_{N - 1}) and \\ Y_{l_{N - 1} i_{N}} = 1 \end{matrix} \\ 0 & otherwise . \end{matrix} & (3.18) \end{matrix}

0578 Said in a different way, the recovered feasible solution zN is then defined by 119

\begin{matrix} {\hat{z}}_{i_{1 (l_{2 N - 1}) i_{2} (l_{2 N - 1}) i_{3} (l_{2 N - 1}) i_{N - 2} (l_{N - 2 N - 1})} i_{N - 1} (l_{N - 1}) i_{N}}^{N} = {\begin{matrix} 1 & if Y_{l_{N - 1} i_{N}} =; \\ 0 & otherwise \end{matrix} & (3.19) \end{matrix}

0579 where lmN1 is defined in (3.13) and denotes function composition. To complete the description, let 120

{{(l_{N - 1} (l_{N}), i_{N} (l_{N})}}_{l_{N} = 0}^{L_{N}}

0580 be an enumeration of indices lN1, iN of Y such that YlN1(lN), iN(lN)1 for (lN1(lN), iN(lN))(0,0) and (lN1(lN), iN(Ln))(0,0) for lN0 regardless of whether Y001 or not. Then the recovered feasible solution can be written as 121

{\hat{z}}_{i_{1} i_{2} i_{3} i_{N}}^{N} = {\begin{matrix} 1 & \begin{matrix} if i_{1} = i_{1} (l_{2 N - 1}), i_{2} = i_{2} (l_{2 N - 1}), \\ i_{3} = i_{3} (l_{3 N - 1}),, \\ i_{N - 1} = i_{N - 1} (l_{N - 1 N}), i_{N} = i_{N} (l_{N}); \end{matrix} \\ 0 & otherwise . \end{matrix}

0581 The Upper and Lower Bound. The upper bound on the feasible solution is given by

vN({circumflex over (z)}N)il. . . iN cil. . . iNN {circumflex over (z)}il. . . iNN(3.21)

0582 and the lower by N3. . . , uN) for any multiplier value (u3, . . . , uN). In particular, we have

N(u3, . . . , uN)N({overscore (u)}3, . . . , {overscore (u)}N)vN({overscore (z)}N)vN({circumflex over (z)}N)(3.22)

0583 where (u3, . . . , uN) is any multiplier value, (3. . . , N maximizer of N3. . . , uN), N multidimensional assignment problem (3.1) and {circumflex over (z)}N is the recovered feasible solution defined by (3.20). Finally, we conclude with the observation that VN({circumflex over (z)})v2(Y) where Y is the optimal solution of (3.17) used in the construction of {circumflex over (z)} in equations (3.18)-(3.20). 0584 Reuse of Multipliers. Since the most computationally expensive part of the algorithm occurs in the maximization of Nk1 starts or multipliers close to the optimal are fundamentally important for real-time speed. The purpose of this section is to demonstrate that the multiplier set obtained at stage k1 provide good starting values for those obtained at step k1. Theorem 3.xx. Let (u3, . . . , uN) denote a multiplier set obtained in the maximum of N3. . . , uN). Then this multiplier set satisfies the string of inequalities

n(u3, . . . , uN)N1(u4, . . . , u5). . . 3(uN)2vn({circumflex over (z)}),

0585 where after the first step in the maximization of N multipliers are not changed in the remaining steps. Furthermore, to improve this inequiality, let (u2k,Nk1, . . . , uN,Nk1)denote a maximizer of Nk1(uk2, . . . , uN). Then, we have 122

\begin{matrix} _{N - k + 1} (u^{k + 2},, u^{N})_{N - k + 1} (u^{2 + k, N - k + 1},, u^{N, N - k + 1})_{N - k} (u^{3 + k, n - k + 1},, u^{N, N - k + 1})_{N - k} (u^{3 + k, N - k},, u^{N, N - k})_{3} 10 u^{N, 3})_{2} V_{N} (\hat{z}) & (3.xxx) \end{matrix}

0586 Inequalities depend on solution of 2d assignment problem. Proof: Suppose we have a value of the multipliers (uk2, . . . , uN) obtained in the maximization of

Nk1(uk2, . . . , uN)Minimize Nk1(zNk1; uk2, . . . , uN)

Subject to ik1ik2iN zlki k1Nk1ik2. . . 1, lk1, . . . , Lk,

lkik2iN Zlkik1ik2. . . iNNk1 1, ik11, . . . , Mk1.

0587 where (3.xx) 123

_{N - k + 1} (z^{N - k + 1}; u^{k + 2},, u^{N}) =_{l_{k} i_{k + 1} i_{N}} c_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} +_{p = k + 2}^{N}_{i_{p} = 0}^{M_{p}} [_{l_{k} i_{k + 1} i_{p - 1} l_{p l} i_{N}} z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} - 1] =_{l_{k} i_{k + 1} i_{N}} [c_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p}] z_{l_{k} i_{k + 1} i_{N}}^{N - k + 1} -_{p = k + 2}^{N}_{i_{p} = 0}^{M_{p}} u_{i_{p}}^{p}

0588 These need not be the maximizer; however, we do assume that we have solved the above minimization problem optimally to evaluate Nk1 . Just as in the definition of the earlier recovery problem, let 124

{{(l_{k} (l_{k + 1}), i_{k + 1} (l_{k + 1})}}_{l_{k + 1} = 0}^{L_{k + 1}}

0589 be an enumeration of indices lk, ik1) of the optimal solution w2 of problem (3.4) (or the first 2 indices of the solution wNk1 of the relaxed problem (3.2) constructed in equation (3.5)) such that wlk2(lk1), ik1(lk1)1 for (lk(lk1), ik1(lk1))(0,0) and (lk(lk1), ik1(lk1))(0,0) for lk10 regardless of whether w002 1 or not. 0590 Then, the final value of the objective function can be expressed as

Nk1(uk2, . . . , uN)

Minimize Nk1(zNk1; uk2, . . . , uN)

Subject to ik2iN zlkNk1(lk1)ik1(lk1)ik2. . . iN1, . . . , Lk1

0591 where 125

\begin{matrix} _{n - k + 1} (z^{N - k + 1}; u^{k + 2},, u^{N} =_{l_{k + 1} i_{k + 2} i_{N}} [c_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1}) i_{k + 2} i_{N}}^{N - k + 1} +_{p = k + 2}^{N} u_{i_{p}}^{p}] z_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k - 1}) i_{k + 2} i_{N}}^{N - k + 1} -_{p = k + 2}^{N}_{i_{p} = 0}^{M_{p}} u_{i_{p}}^{p} . & (3.xxx) \end{matrix}

0592 If now another constrain set, say the (k2)th set, is added to this problem, one has 126

{\overline{}}_{N = k + 1} (u^{k + 1}, u^{N}) Minimize {\overline{}}_{N - k + 1} (z^{N - k + 1}; u^{k + 2},, u^{N} Minimize_{l_{k + 1} i_{k + 2} i_{N}} [c_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1}) i_{k + 2} i_{N}}^{N k 1}_{p = k + 2}^{N} u_{i_{p}}^{p}] z_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1})}^{N k 1}_{p = k + 2}^{N}_{i_{p} = 0}^{M_{p}} u_{i_{p}}^{p} Subject to_{i_{k + 2} i_{N}} z_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1}) i_{k + 2} i_{N}}^{N k 1} 1, i_{k + 2} 1,, L_{k + 1},_{l_{k + 1} i_{k + 3} i_{N}} z_{l_{k} (l_{k + 1}) i_{k + 1} (l_{k + 1}) i_{k + 2} i_{N}}^{N k 1} 1, i_{k + 2} 1,, M_{k + 2} .

0593 Since the constraint lk1di k2. . . iN zlkNk1(lk1)ik1(lk1)ik2. . . iN1 for ik21, . . . , Mk2 is now imposed, the feasible region is smaller, so that one has

Nk1(uk2, . . . , uN){overscore ()}Nk1(uk2, . . . , uN){overscore ()}Nk1(uk3, . . . , uN)Nk1(uk3, . . . , uN

0594 where the fact that {overscore ()}Nk1 does not depend on the multiplier set uk2 due the added constraint set. Also, the last equivalence follows from the fact that Nk1k3. . . , uN) is the relaxed problem (3.xx) (with k replaced by k1) for the recovery problem (3.xx) in step k of the above algorithm. Continuing this way, one can compute (u3, . . . , uN) at the first step (k1), fix them thereafter, and perform no optimization at the subsequent steps to obtain

N(u3, . . . ,uN)N1(u4, . . . , uN). . . 3(uN)2vN({circumflex over (z)})

0595 where 2 0596 To explain how to improve this inequality, let (u2k,Nk1, . . . , uN,Nk1) denote a maximizer of Nk1(uk2, . . . , uN) . Then by the same reasoning one has (3.xx) as stated in the theorem. Q.E.D. 0597 Summary of the Lagrangian Relaxation Problem. (APB) Given the multidimensional assignment problem 127

\begin{matrix} Minimize_{i_{1} i_{N}} c_{i_{1} i_{N}} {z_{i_{1} i}}_{N}_{i_{2} i_{3} i_{N}} z_{i_{1} i_{N}} = 1 (i_{1} = 1,, M_{1}) Subject to_{i_{1} i_{3} i_{N}} z_{i_{1} i_{N}} = 1 (i_{2} = 1,, M_{2})_{i_{1} i_{p - 1} i_{p + 1} i_{N}} z_{i_{1} i_{N}} (i_{p} = 1,, M_{p} and p = 2,, N - 1)_{i_{1} i_{2} i_{N - 1}} z_{i_{1} i_{N}} = 1 (i_{n} = 1,, m_{N}) z_{i_{1} i_{N} {0, 1} for all i_{1},, i_{N}} & (4.1) \end{matrix}

0598 where c0 . . . 0N is arbitrarily defined to be zero and is included for notational convenience and where the superscript N on both c and z is not an exponent, but denotes the dimension of the subscripts and the assignment problem as stated in (2.5). In relaxed and recovery problems c0 . . . 0N need not be zero In this problem, we assume that all zero-one variables zil. . . iNN with precisely one nonzero index are free to be assigned and that the corresponding cost coefficients are well-defined. (This is a valid assumption in the tracking environment.) Although not required, these cost coefficients with exactly one nonzero index can be translated to zero by cost shifting without changing the optimal assignment. 0599 Having explained many of the relaxation features, it is now appropriate to present the Lagrangian relaxation algorithm, which iteratively constructs a feasible solution to the N-dimensional assignment problem (4.1). 0600 Algorithm 4.1 (Lagrangian Relaxation Algorithm). To construct a high quality feasible solution, enoted by {circumflex over (z)}N, of the assignment problem (4.1), proceed as follows: 0601 0. Initialize the multipliers (uk2, . . . , uN), e.g., (uk2, . . . , uN)(0,0). 0602 For k1, . . . , N2, do 0603 1. For the Lagrangian relaxed problem (3.2) from the problem (3.1) by relaxing the last (Nk1) sets of constraints. 0604 2. Use a nonsmooth optimization technique to solve

Maximize Nk1uPRMP1 for pk2, . . . ,N

0605 with u0P0 being fixed(4.2)0606 where Nk1(uk2, . . . , uN) is defined by equation (3.2). The algorithm following problem (3.9) provides one method for computing a function value and a subgradient out of the subdifferential at (uk2, . . . , uN), as required in most nonsmooth optimization techniques. 0607 3. Given an approximate or optimal maximizer of (4.2), say k2, . . . , N), let 2 0608 assignment problem (3.4) corresponding to this maximizer of Nk1(uk2, . . . , uN). 0609 4. Formulate the recovery (N-k)-dimensional problem (3.13), modified as discussed at the end of subsection (3.3) for sparse problems. At this stage, {circumflex over (z)}N as defined in (3.15) contains the alignment of the indices i1, . . . , ik1. 0610 Formulate the final two-dimensional problem (3.17) and complete the final recovered solution {circumflex over (z)}N as in (3.20). 0611 To explain the lower and upper bounds, let N defined in (3.2) with k1, let VN(zN) be the objective function value of the N-dimensional assignment problem in equation (2.5) corresponding to a feasible solution zN of the constraints in (2.5), and let zNk1 0612 (2.5). Then,

N(u3, . . . , uN)vn({overscore (z)}N)vN({circumflex over (z)}N)(4.3)

0613 is the desired inequality. 0614 COMMENTS: 0615 1. Step 2 is the computational part of the algorithm. Evaluating Nk1 search procedure requires 99% of the computing time in the algorithm. This part uses two dimensional assignment algorithms, a search over a large number of indices, and a nonsmooth optimization algorithm. It is the second part (the search) that consumes 99% of the computational time and this is almost entirely parallelizable. 0616 2. In track maintenance, the warm starts for the initial multipliers for the first step are available. For the relaxation procedure, initial multipliers are available in step 2 from the prior step in the algorithm. 0617 3. There are many variations on the above algorithm. For example, one can compute a good solution on the first pass (k1) and not perform the optimization in (2) thereafter. This yields a great solution. Thus one can continue the optimization at the first pass, and immediately compute quality feasible solutions to the problem. 0618 Lagrangian Relaxation Algorithm for the 3-Dimensional Algorithm. (ABP) In this section, we illustrate the relaxation algorithm for the three dimensional assignment problem. Having discussed the general relaxation scheme, 128

\begin{matrix} Minimize_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}}_{i_{3} = 0}^{M_{3}} c_{i_{1} i_{2} i_{3}} z_{i_{1} i_{2} i_{3}} Subject to_{i_{2} = 0}^{M_{2}}_{i_{3} = 0}^{M_{3}} z_{i_{1} i_{2} i_{3}} = 1, i_{1} = 1,, M_{1},_{i_{1} = 0}^{M_{1}}_{i_{3} = 0}^{M_{3}} z_{i_{1} i_{2} i_{3}} = 1, i_{2} = 1,, M_{2},_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}} z_{i_{1} i_{2} i_{3}} = 1, i_{3} = 1,, M_{3} z_{i_{1} i_{2} i_{3}} (0, 1) for all i_{1} i_{2} i_{3}, & (5.1) \end{matrix}

0619 To ensure that a feasible solution of (5.1) always exists for a sparse problem, all variables with exactly one nonzero index (i.e., variables of the form zi100, zi100, z0i20, and z00i3 for ip1, Mp and p1,2,3 are assumed free to be assigned with the corresponding cost coefficients being well-defined. This assumption is valid in the tracking environment **refPoa, **refPob. 0620 The Lagrangian Relaxed Assignment Problem. The three dimensional assignment problem (5.1) has three sets of constraints and one can describe the relaxation by relaxing any of the three sets of constraints, the description here is based on relaxing the last set of constraints. A (M31)-dimensional multiplier vector (i.e., u3Rm31) associated with the 3th constraint set will be denoted by u3(u03, u13, . . . , uM33)T with u030 being fixed for notational convenience. (The zero multiplier u03 is used for notational convenience.) Now, the three dimensional assignment problem (5.1) is relaxed to a two-dimensional assignment problem by incorporating last set of constraints into the objective function via the Lagrangian. Although any constraint set can be relaxed, we choose the last set of constraints for convenience. The relaxed problem is 129

\begin{matrix} \begin{matrix} _{3} (u^{3}) Minimize_{3} (z^{3}; u^{3}) \\ _{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}}_{i_{3} = 0}^{M_{3}} c_{i_{1} i_{2} i_{3}}^{3} z_{i_{1} i_{2} i_{3}}^{3} + \\ _{I_{3} = 0}^{M_{3}} U_{i_{3}}^{3} [_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}} z_{i_{1} i_{2} i_{3}}^{3} - 1] \\ Minimize_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}}_{i_{3} = 0}^{M_{3}} [c_{i_{1} i_{2} i_{3}}^{3} + u_{i_{3}}^{3}] z_{i_{1} i_{2} i_{3}}^{3} - \\ _{i_{3} = 0}^{M_{3}} u_{i}^{3} \end{matrix} \begin{matrix} Subject to_{i_{2} = 0}^{M_{2}}_{i_{3} = 0}^{M_{3}} z_{i_{1} i_{2} i_{3}}^{3} = 1, i + 1 = 1, M_{1} \\ _{i_{1} = 0}^{M_{1}}_{i_{3} = 0}^{M_{3}} z_{i_{1} i_{2} i_{3}}^{3} = 1, i + 1 = 1, M_{2} \end{matrix} & (5.2) \end{matrix}

0621 One of the major steps in the algorithm is the maximization of 330622 u3. It turns out that 3 0623 piecewise affine function of the multiplier vector u3, so that the maximization of 3 0624 Since many of these algorithms require a function value and a subgradient of 3 30625 address this problem in the next subsection. We note, however, that there are other ways to maximize 3 0626 subsection addresses but one such method.

Properties Lagrangian Relaxed Assignment Problem 0627 For a function evaluation of 3 0628 suboptimal) solution of this relaxed problem (5.2) can be constructed from that of a two-dimensional assignment problem. Then, the nonsmooth characteristics of 3 0629 followed by a method for computing a subgradient. 0630 Evaluation of 3. (i1, i2) 0631 j3j3(i1, i2) and a new cost function ci1i22 by: 130

\begin{matrix} j_{3} j_{3} (i_{1}, i_{2}) = \arg \min {c_{i_{1} i_{2} i_{3}}^{3} + u_{i_{3}}^{3} i_{3} = 0, 1,, M_{3}} c_{i_{1} i_{2}}^{2} = c_{j_{3}}^{3} for (i_{1}, i_{2}) (0, 0) c_{00}^{2} =_{i_{3} = 0}^{M_{3}} Minimum {0, c_{00 i_{3}}^{3} + u_{i_{3}}^{3}} & (5.3) \end{matrix}

0632 Given an index pair (i1, i2), j3 need not be unique, resulting in the potential generation of several subgradients (5.11). (We further discuss this issue at the end of Subsection 5.2.3.) 0633 Now, define 131

\begin{matrix} \overline{} (u_{3}) = Minimize_{3} (z^{2}; u^{3})_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}} c_{i_{1} i_{2}}^{2} z_{i_{1} i_{2}}^{2} -_{i_{3} = 0}^{M_{3}} Subject to_{l_{2} = 0}^{M_{2}} z_{i_{1} i_{2}}^{2} = 1, 1,, M_{1},_{i_{1} = 0}^{M_{1}} z_{i_{1} i_{2}}^{2} = 1, 1,, M_{2}, z_{i_{1} i_{2}}^{2} {0, 1} for all i_{1}, i_{2} . & (5.4) \end{matrix}

0634 As an aside, two observations are in order. The first is that the search procedure needed for the computation of the relaxed cost coefficients in (5.3) is the most computationally intensive part of the entire relaxation algorithm. The second is that a feasible solution z3 of a sparse problem (5.1) yields a feasible solution z2 of (5.4) via the construction 132

z_{i_{1} i_{2}}^{2} = {\begin{matrix} 1, & if z_{i_{1} i_{2}, i_{3}}^{3} = 1 for some (i_{1}, i_{2}, i_{3}); \\ 0, & otherwise . \end{matrix}

0635 Thus the two-dimensional assignment problem (5.4) has feasible solutions other than those with exactly one nonzero index if the original problem (5.1) does. 0636 The following Theorem 5.1 states that an optimal solution of (5.2) can be computed from that of (5.4). Furthermore, if the solution of either of these two problems is then so is the other. The converse is contained in Theorem 5.2. 0637 Theorem 5.1. Let 2 0638 dimensional assignment problem (5.4) and define w3 by

wi1i2i33l wi1i22 and (i1, i2)(0,0);(5.5)

wi1i2i330 if i3j3 and (i1, i2)(0,0);

w00i331 if c00i33ui330;

w0013301 if c00i33ui33>0.

0639 Then w3 is a feasible solution of the Lagrangian relaxed problem and 3 3 2 3 2 0640 optimal for the two dimensional problem, then w3 is an optimal solution of the relaxed problem and 3(u3){overscore ()}3(u3). 0641 With the exception of one equality being converted to an inequality, the following theorem is a converse of Theorem 5.1. 0642 Theorem 5.2. Let w2 be a feasible solution to problem (5.2) and define w2 by 133

\begin{matrix} w_{i_{1} i_{2}}^{2} =_{i_{3} = 0}^{M_{3}} w_{i_{1} i_{2} i_{3}}^{3} for (i_{1}, i_{2}) (0, 0) w_{00}^{2} = 0 if (i_{1}, i_{2}) = (00) and c_{00 i_{3}}^{3} + u_{i_{3}}^{3} > 0 for all i_{3} w_{00}^{2} = 1 if (i_{1}, i_{2}) = (00) and c_{00 i_{3}}^{3} + u_{i_{3}}^{3} 0 for all i_{3} . & (5.7) \end{matrix}

0643 Then w2 is a feasible solution of the problem (5.4) and 3(w3; u3)3(w2; u3). If, in addition, w3 is optimal for (5.2), then w2 is an optimal solution of (5.4), 3(w3; u3)3(w2; u3) and 3(u3){overscore ()}3(u3).

The Nonnsmooth Optimization Problem

An Algorithm for Computing 3 and a Subgradient 0644 Given u3 the problem of computing 33 subgradient of 330645 1. Starting with problem (5.1), form the relaxed problem (5.2) 0646 2. To solve (5.2) optimally, define the two dimensional assignment problem (5.4) via the transformation (5.3) 0647 3. Solve the two-dimensional problem (5.4) optimally. 0648 4. Reconstruct the optimal solution, say 3 equation (5.6) as in Theorem 5.1. 0649 5. Then 134

\begin{matrix} _{3} (u^{3})_{3} ({\hat{w}}^{3}; u^{3}) =_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}}_{i_{3} = 0}^{M_{3}} c_{i_{1} i_{2} i_{3}}^{3} {\hat{w}}_{i_{1} i_{2} i_{3}}^{3} +_{i_{3} = 0}^{M_{3}} u_{i_{3}}^{3} [_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}} {\hat{w}}_{i_{1} i_{2} i_{3}}^{3} - 1] & (5.10) \end{matrix}

0650 6. A subgradient is given by substituting 3 objective function (5.2), erasing the minimum sign, and taking the partial with respect to ui33. The result is 135

\begin{matrix} g_{i_{3}}^{3} = \frac{_{3} (u^{3})}{u_{i_{3}}^{3}} =_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}} {\hat{w}}_{i_{1} i_{2} i_{3}}^{3} - 1 for i_{3} = 1,, M_{3} . & (5.11) \end{matrix}

0651 The Recovery Procedure. The next objective is to explain a recovery procedure, i.e., given a feasible (optimal or suboptimal) solution w2 of (5.4) (or w3 of (5.2) constructed in Theorem 5.1), generate a feasible solution z3 of equation (5.1) which is close to w2 in a sense to be specified. We first assume that no variables in (5. 1) are preassigned to zero; this assumption will be removed shortly. The difficulty with the solution w3 in Theorem 5.1 is that it need not satisfy the third set of constraints in (5.1). (Note, however, that if w2 is an optimal solution for (5.4) and w3, as constructed in Theorem 5.1, satisfies the relaxed constraints, then w3 is optimal for (5.1).) The recovery procedure described here is designed to preserve the 0-1 character of the solution w2 of (5.4) as far as possible: If wi1i221 and i1`0 or i20, the corresponding feasible solution z3 of (5.1) is constructed so that zi1i2i331 for some (i1, i2, i3). By this reasoning, variables of the form z00i33, can be assigned a value of one in the recovery problem only if w0021. However, variables z00i33 will be treated differently in the recovery procedure in that they can be assigned either zero or one independent of the value w002. This increases the feasible set of the recovery problem, leading to a potentially better solution. 0652 Let 136

{{(i_{1} (l_{2}), i_{2} (l_{2})}}_{l_{2} = 0}^{L_{1}}

0653 be an enumeration of indices i1, i2 of w2 (or the first 2 indices of w3 constructed in equation (5)) such that wi12(l2), i2(l2)1 for (i1(i2),i2(l2))(0,0) and (i1(i2), i2(l2))(0,0) for l20 regardless of whether w0021 or not. (The latter can only improve the quality of the feasible solution.) 0654 Next to define the two-dimensional assignment problem that restores feasibility, let

cl2l32ci13(l2)i2(l2)i3 for l20, . . . ,L1(5.12)

0655 Then the two-dimensional assignment problem that restores feasibility is 137

\begin{matrix} Minimize_{l_{2} = 0}^{L_{2}}_{i_{3} = 0}^{M_{3}} c_{i_{2} i_{3}}^{2} z_{l_{2} i_{3}}^{2} Subject to_{i_{3} = 0}^{M_{3}} z_{l_{2} i_{3}}^{2} = 1, l_{2} = 1,, L_{2},_{l_{2} = 0}^{L_{2}} z_{l_{2} i_{3}}^{2} = 1, i_{3} = 1,, M_{3}, z_{l_{2} i_{3}}^{2} {0, 1} for all l_{2}, i_{3} . & (5.14) \end{matrix}

0656 The next objective is to remove the assumption that all cost coefficients are defined and all zero-one variables are free to be assigned. We first note that the above construction of a reduced dimension assignment problem (5.13) is valid as long as all cost coefficients c2 are defined and all zero-one variables in z2 are free to be assigned. Modifications are necessary for sparse problems. If the cost coefficient ci13(l2)i2(l2)i3 well-defined and the zero-one variable zi13(l2)i2(l2)i3 is free to be assigned to zero or one, then define c12i32ci13(l2)i2(l2)i3 as in (5.12) with zi13(l2)i2(l2)i3 being free to be assigned. Otherwise, zl2i32 is preassigned to zero or the corresponding arc is not allowed in the feasible set of arcs. To ensure that a feasible solution exists, we now only need ensure that the variables z1202 are free to be assigned for l20,1, . . . ,L1 with the corresponding cost coefficients being well-defined. (Recall that all variables of the form zi1003, z0i203 and z00i33 are free (to be assigned) and all coefficients of the corresponding form ci1003, c0i203 and c00i33.) This is accomplished as follows: If the cost coefficient ci1l2)i2(l2)03 is well-defined and zi1(l2)i2(l2)03 is free, then define cl202ci2(l2)i2(l2)0 with z1202 being free. Otherwise, since all variables of the form zi103 and z0i2o3 are free to be assigned with the corresponding costs being well-defined, set cl202ci1(l2)003c0i2(l2)03 where the first term is omitted if i1(l2)0 and the second, if i2(l2)0. for i1(l2)0 and i2(l2)0, define c002c0003. 0657 The final recovery problem. The recovery problem for the case N3 is 138

\begin{matrix} Minimize \underset{Y}{\overset{X}{}} \underset{Y}{\overset{X}{}} c_{l_{2} i_{3}}^{2} z_{l_{2} i_{3}}^{2} Subject to_{i_{3} = 0}^{M_{3}} z_{l_{2} i_{3}}^{2} = 1, l_{2} = 1,, L_{2},_{l_{2} = 0}^{L_{2}} z_{l_{2} i_{3}}^{2} = 1, l_{3} = 1,, M_{3}, z_{l_{2} i_{3}}^{2} {0, 1} for all l_{2}, i_{3} . & (5.14) \end{matrix}

0658 Let Y be an optimal or feasible solution to this 2-dimensional assignment problem. The recovered feasible solution z3 is defined by 139

\begin{matrix} z_{i_{1} i_{2} i_{3}}^{3} = {\begin{matrix} 1 & if i_{1} = i_{1} (l_{2}), i_{2} = i_{2} (l_{2}) and Y_{l_{2} i_{3}} = 1; \\ 0 & otherwise . \end{matrix} & (5.15) \end{matrix}

0659 Said in another way, let (l2(l3), i3(l3)l30L3 be an enumeration of indices l2, i3 of Y such that Yl2(l3), u3(l3)1 for (l2(l3),i3(l3)(0,0) and (l2(l3),i3(l3)(0,0) for l30 regardless of whether Y001 or not. Then the recovered feasible solution can be written as 140

\begin{matrix} {\hat{z}}_{i_{1} i_{2} i_{3}}^{3} = {\begin{matrix} 1 & if i_{1} = i_{1} (l_{12}), i_{2} = i_{2} (l_{12}), i_{3} = l_{3} (l_{3}) \\ 0 & otherwise \end{matrix}; & (5.16) \end{matrix}

0660 The upper bound on the feasible solution is given by 141

\begin{matrix} V_{3} ({\hat{z}}^{3}) =_{i_{1} = 0}^{M_{1}}_{i_{2} = 0}^{M_{2}}_{i_{3} = 0}^{M_{3}} c_{i_{1} i_{1} i_{3}}^{3} {\hat{z}}_{i_{1} i_{2} i_{3}}^{3} & (5.17) \end{matrix}

0661 and the lower by 330662 In particular, we have

3(u3)3({overscore (u)}3)v3({overscore (z)}3)v3({circumflex over (z)}3)

0663 where u3 is any multiplier value, ({overscore (u)}30664 33{overscore (z)}3 0665 assignment problem and {overscore (z)}3 is the recovered feasible solution defined by (5.16). 0666 Other Relaxations for the Multidimensional Assignment Problem.(ABP) In this section, we briefly describe other possible relaxations and their implications. These include linear programming relaxations and the corresponding duals. 0667 Recall that one starts with the definition of the zero-one variable ziN. . . iN1 and cost coefficients to form the problem 0668 0669 where c0 . . . 0 is arbitrarily defined to be zero. Here, each group of sums in the constraints represents the fact that each non-dummy report occurs exactly once in a track of data. One can modify this formulation to include multiassignments of one, some, or all the actual reports. The assignment problem (2.5) is changed accordingly. For example, if zikk is to be assigned no more than, exactly, or no less than nikk times, then the 1 in the constraint (2.5) is changed to , , nikk respectively. Modifications for group tracking and multiresolution features of the surveillance region will be addressed in future work. In making these changes, one must pay careful attention to the independence assumptions that need not be valid in many applications. 0670 Again, the recent work of Poore and Drummond significantly extends the formulation of the track maintenance and initiation to new approaches. The discussions of this section apply equally to those formulations. 0671 The Linear Programming Relaxation. In the linear programming relaxation, one replaces the zero-one variables constraints

zi1iN0,1 for all i1, . . . , iN(6.2)

0672 with the constraint

0zi1. . . iN1 for all i1, . . . iN(6.2)

0673 Then, the problem (6.1) can be formulated as a linear programming problem with the constraint (6.2) in (6.1) replaced by (6.3) with a special block structure to which the Dantzing-Wolfe decomposition is applicable. Of course, after solving this problem, one must now recover the zero-one character of the variables in (6.1) and there are many ways to do this, such as using the two dimensional assignment problems. Commercial software is also available. 0674 The Linear Programming Dual and Partial Lagrangian Relaxations. Given the linear programming relaxation, one can formulate the dual problem or the partial Lagrangian relaxation duals with respect to any number of constraints. In particular, this is precisely what is done in Section 3 on the Lagrangian relaxation algorithm presented. The much broader class of algorithms provided in the U.S. Pat. No. 5,537,119 (Aubrey B. Poore, Jr., Method and System for Tracking Multiple Regional Objects by MultiDimensional Relaxation, U.S. Pat. No. 5,537,119, filed Mar. 14, 1995, issue date of Jul. 16, 1996 ) can be modified to remove the zero-one character when one relaxes M sets of constraints to an NM dimensional problem and recovers with an NM1 dimensional problem. This avoids the problems associated with the NP-hard NM and NM1 dimensional problems. However, to restore the zero-one character, one can do it sequentially with an assignment problem or with one of the many zero-one rounding techniques. These formulations are easy to work out and thus the details are omitted. 0675 The complete dual problem is another way of solving the LP problem and may indeed be more efficient in certain applications. In addition, the solution of this dual problem may provide excellent initial approximations to the multipliers for Lagrangian relaxations. 0676 Multisensor Resolutions Problems and Their Solution via Linear Nonlinear Assignment Problems and Inequality Constraints. (ABP) The mathematical development for inequality constraints is understood by Aubrey B. Poore, but this work has not been written up. This feature is important in combining information from sensors with different resolutions such an IR and Radar. 0677 Formulation and Algorithms for the Distributed Sensor Tracking: Track-to-Track Correlations. (ABP) 0678 Hot Starts for Track Maintenance and Initiation: Bundle Modifications (ABP). 0679 Thus reuse of multipliers and the first proof that this reuse is actually valid was presented in this section. The reuse in track maintenance is demonstrated and discussed in the work of Poore and Drummond xx. The only item left is the modification of the bundle of subgradients for the use with the multiplier values as one goes through the recovery problem as in Section 3.6 or as one moves the window in track maintenance. It is this aspect of the nonsmooth optimization that adds an order of magnitude to the speed of the algorithms. This work is based on both a mathematical proof as in Section 3.6 as well as computational experience and heuristics. 0680 Adaptive Window Size Selection (ABP). 0681 These data association algorithms, based on the multidimensional assignment problem, range from sequential processing to many frames of data processed all at once. The data association problem for multisensor multitarget tracking is formulated using a window sliding over the frames of data from the sensors. This discussion focuses on the work of Poore and Drummond and not on the formulations in Section 2. Firm data association decisions for existing track are made at, say frame k, with the most recent frame being kM. Decisions after frame k are soft decisions. Reports not in the confirmed tracks are used to initiate tracks over frames numbered kI to kM. 0682 The length of these windows varies from sequential processing, which corresponds precisely to I0 and M1, to user defined large values of I and M. (In the case of sequential processing, we also have a temporary track file of dynamically feasible tracks, but incorrect data association.) There is a marked change in performance over this range. As the two windows become exceedingly long, there is little statistically significant information gained and indeed performance can degenerate slightly. Thus, one must manually find the correct window lengths for performance in a given scenario and the algorithms do not change for a changing environment. Thus we propose an adaptive method for adjusting the window lengths. (The method has been highly successful for selecting the correct window length for multistep methods in ordinary differential equations.) 0683 1. Compute the solution for different window lengths that overlap and differ by one or two frames. 0684 2. Compare the solution quality, i.e., the value of the objective function, for two adjacent window lengths. 0685 3. If there is a predefined improvement in either direction, we then, for stability, repeat the exercise for a shorter or longer on the first try. If there is consistent information, we adjust the window size in the indicated direction. This can be given a well defined mathematical formula in terms of the assignment problems of different dimensions. 0686 Algorithm Enhancements due to Data Structures (ABP). 0687 Construction and enumeration of the Best K-Solutions. 0688 New Nonsmooth Optimization Techniques. 0689 Sensitivity Analysis. 0690 For sequential processing in which the two dimensional assignment algorithm is used, one can use the LP sensitivity analysis theory and easily obtain the corresponding answers. For the multiframe processing, the optimal solution is not available; however, there are still two approaches one can use. First, the basic algorithm can perform the same sensitivity analysis at each stage (loop) of the algorithm as is done in the two dimensional assignment problem since the evaluation of function Nk1 is equivalent to a two dimensional assignment problem. Alternately, one could use an LP relaxation of the assignment problem and base the sensitivity on the resulting LP problem. We currently see this as an important step in finding even better solutions to the assignment problem if so desired. 0691 New Auction Algorithms (ABP & PS). In this section we present a new auction algorithm for the two dimensional assignment problem. 0692 An important step in solving N-dimensional assignment problems for N3, is finding the optimal solution of the 2-dimensional assignment problem. In particular, we wish to solve the 2-dimensional problem which includes the zero-index. T3,pically this problem can be thought of as trying to find either a minimum cost or maximum utility in assigning person to objects, tenants to apartments, or teachers to classrooms. We will follow the work of Bertsekas and call the first index set persons and the second index set objects. The statement of this problem is given below (1) when the number of persons is m and the number of objects is n. 142

\begin{matrix} Minimize_{i = 0}^{m}_{j = 0}^{n} c_{i_{j}}_{i_{j}} \begin{matrix} Subject to_{j = 0}^{n}_{i_{j}} = 1 for all i = 1,, m \\ _{i = 0}^{m}_{i_{j}} = 1 \\ _{i_{J}} {0, 1} \end{matrix} & (1) \end{matrix}

0693 There are a couple of assumptions which we will make about (1). First of all, we assume that ci0 and c0j are well-defined and the corresponding variables x10 x0j free to be assigned. Second if a cost cij happens to be undefined, then the corresponding variable xij x to 0. In effect if cij is undefined, we simply remove this cost and variable from inclusion in the problem. 0694 Notice that there are no constraints on the number of times person 0 and object 0 can be assigned. But notice that the first constraint requires that each non-zero person i must be assigned exactly once. Similarly, the second constraint forces each non-zero object j to be assigned exactly one time. Finally, the last constraint gives an integer solution, although we will see shortly that this constraint can be relaxed to admit any solution xij0. One reason for requiring that all of the costs of the form ci0 and c0j be defined is so that we are guaranteed a feasible solution exists for the given problem. 0695 Relaxation of One Constraint. 0696 0697 Relaxing the last constraint, we obtain the following problem: 143

\begin{matrix} Minimize_{i = 0}^{m}_{j = 0}^{n} c_{i_{j}}_{i_{j}} +_{j = 0}^{n} u_{j}^{2}_{i = 0}^{m}_{ij} - 1 \\ _{i = 0}^{m}_{j = 0}^{n} [c_{ij} + u_{j}^{2}]_{ij} -_{j = 0}^{n} u_{j}^{2} \end{matrix}

\begin{matrix} Subject to_{j = 0}^{n}_{i_{j}} = 1 for all i = 1,, m \\ _{i_{J}} {0, 1} \end{matrix}

0698 This lower bound is achieved and the second constraint in the original problem is satisfied if the following conditions are met: 144

For i = 0,_{0 j} = {\begin{matrix} 1 & if c_{0 j} + u_{j}^{2} 0 \\ 0 & otherwise \end{matrix}} For i 0,_{ij} = {\begin{matrix} 1 & if j = \arg \min {c_{ik} + u_{k}^{2} k = 0, 1,, n} \\ 0 & otherwise \end{matrix}}

0699 All of j's are assigned only one time. 0700 The relaxation of the first constraint is analogous and would lead to similar results with i and j exchanged and the introduction of the multiplier u1. 0701 Relaxation of Both Constraints. 0702 This time we will relax both constraints and using duality theory obtain a relationship between the multipliers u1 and u2. 0703 a bunch of equations, theorems and the such here. 0704 0705 From the above relaxation, we arrive at the following complementary slackness conditions which have been relaxed by a small value . 0706 Definition: An assignment S and a multiplier set (u1,u2) are said to satisfy -complementary slackness (CS) if 0707 cijui1uj20 for all (i,j) S, 0708 cijui1uj2 for all (i,j) A. 0709 Forward Auction Algorithm. 0710 (1) Select any unassigned person i 0711 (2) Determine the following quantities: 0712 jiarg min cikuk2k A(i)0713 vicijiuji2 0714 wimin cikuk2k A(i), kji0715 In the selection of j1 above, if a tie occurs between 0 and any non-zero index k, then select ji as k. Otherwise, if there is a tie between two or more non-zero indices, the choice of ji is arbitrary. Also if A(i) consists of only one element, then set wi. 0716 (3) Update the multipliers and the assignment: 0717 If ji0, then 0718 (a) Add (i,0) to the assignment. 0719 (b) Update ui1:ci0. 0720 If ji0, then 0721 (a) Add (i,ji) to the assignment. 0722 (b) Remove (i,ji) from the assignment if ji was previously assigned. 0723 (c) Update uji2:uji2(w1v1)wicij1. 0724 (d) Update ui1:(cijiuji2)wi. 0725 Reverse Auction Algorithm. 0726 (1) Select any unassigned object j, such that c0juj2>0. 0727 (2) Determine the following quantities:0728 ijarg min cjkuk1k B(j)0729 jcijjuij2 0730 jmin ckjuk1k B(j), kij0731 In the selection of ij. above, if a tie occurs between 0 and any non-zero index k, then select ji as k. Otherwise, if there is a tie between two or more non-zero indices, the choice of ji is arbitrary. Also if B(j) consists of only one element, then set j . 0732 (3) Update the multipliers and the assignment: If ij0, then 0733 (a) Add (0,j) to the assignment. 0734 (b) Update uj2:c0j. 0735 If ij0, then 0736 (a) Add (ij,j) to the assignment. 0737 (b) Remove (ij,j) from the assignment if (ij,j) was previously assigned. 0738 (c) Update u1j1:uij1(jj)jcijj. 0739 (d) Update uj2:(cijjuijj1)j. 0740 Combined Forward/Reverse Auction Algorithm 0741 1. Assume that u2 is given as an arbitrary multiplier. 0742 2. Adjust the value of u2 for each object j as follows:

If cojuj2<0, then set uj2:c0j.

0743 3. Run iterations of the Forward Auction Algorithm until all persons become assigned. 0744 4. Run iterations of the Reverse Auction Algorithm until all of the objects become assigned. 0745 Note at the completion of the Forward auction step we have the following conditions satisfied: 0746 c0juj20 for all objects j. 0747 cijui1uj20 for all (i,j) S. 0748 cijuj2kmincikuk2 for all (i,j) S. 0749 Thus we can prove the following proposition. Proposition: If we assume that c0juj20 at the start of the Forward Auction Algorithm and all of the persons are assigned via a forward step, then we have:

cijui1uj2 for all (i,j) A.

cijui1uj20 for all (i,j) S.

cijuj2kmincikuk2 for all (i,j) S.

0750 Optimality of the Algorithm 0751 Theorem: -CS preserved during every forward and reverse iteration. 0752 Theorem: If a feasible solution exists, then the resulting solution is with m of being optimal for the Combined Forward Reverse Algorithm. 0753 Implementation Specifics Parallelization. 0754 Here are but a few comments. 0755 Although the algorithm appears to be serial in nature, its primary computational requirements are almost entirely parallelizable. Thus parallelization is planned. 0756 Step 2 is the computational part of the algorithm. Evaluating Nk1 x x x procedure requires 99% of the computing time in the algorithm. This part uses two dimensional assignment algorithms, a search over a large number of indices, and a nonsmooth optimization algorithm. It is the second part (the search) that consumes 99% of the computational time and this is almost entirely parallelizable. Indeed, there are two dimensional assignment solvers that are highly parallelizable. Thus, we need but parallelize the nonsmooth optimization solver to have a reasonably complete parallelization. 0757 If a sensitivity analysis is desired or if one is interested in computing several near-optimal solutions, a parallel processor with a few powerful processors and good communication such as on the Intel Paragon would be most beneficial. 0758 The foregoing discussion of the invention has been presented for purposes of illustration and description. Further, the description is not intended to limit the invention to the form disclosed herein. Consequently, variation and modification commensurate with the above teachings, within the skill and knowledge of the relevant art, are within the scope of the present invention. The embodiment described hereinabove is further intended to explain the best mode presently known of practicing the invention and to enable others skilled in the art to utilize the invention as such, or in other embodiments, and with the various modifications required by their particular application or use of the invention. It is intended that the appended claims be construed to include alternative embodiments of the invention to the extent permitted by the prior art.