2 Triangular Input Normal Pairs
In this section, we present the fundamental representation results for the reduction to triangular normal form.
An input pair, , is input normal (IN) if and only if
The input pair is a TIN pair if it is input normal and is lower triangular.
IN pairs are not required to be stable or controllable. (From (2.1), must be at least marginally stable.) In , ‘input balanced” has a more restrictive definition of (2.1) and the additional requirement that the observability Grammian be diagonal. We do not impose any such condition on the observability Grammian. We choose this language so that ‘normal” denotes restrictions on only one Grammian while ‘balanced” denotes simultaneous restrictions on both Grammians.
Thus is an IN pair if and only if the concatenated matrix is row orthogonal. When is input normal, then the identity matrix solves Stein’s equation (also known as the discrete Lyapunov equation):
For stable , , So is input normal when or equivalently when the basis, , constitutes an orthonormal set.
Let and be equivalent IN pairs, with stable and controllable. Then is unitary.
Proof: Both and solve the Stein equation. Since is stable and controllable, the solution of Stein’s equation is unique.
Every stable, controllable input pair , is similar to a lower triangular input normal pair with . The order of the eigenvalues of may be specified arbitrarily. If is real and has real eigenvalues, then and may be chosen to be real.
Proof: Let be the unique Cholesky lower triangular factor of with positive diagonal entries: . Here is invertible. We set and . Note satisfies (2.1). By Schur’s unitary triangularization theorem (Horn and Johnson 2.3.1), there exists a unitary matrix, such that is lower triangular. The proof of the real Schur form is in Sec. 7.4.1 of  and Theorem 2.3.4 of . The eigenvalues of may be placed in any order . Clearly, and satisfy (2.1) and therefore .
We now study the number of equivalent TIN pairs:
Let and be LT matrices with and let and be unitarily similar: with unitary. Let be the number of distinct eigenvalues. Partition , and into blocks corresponding to the repeated eigenvalue blocks. Let be the multiplicity of the th eigenvalue, . Then has block diagonal form: , where is a unitary matrix of size .
Proof: From . If , then and have no common eigenvalues. By Lemma 7.1.5 of , . Repeating this argument shows for . By orthogonality, for . We continue this chain showing that for , etc. Proof by finite induction.
Let be an matrix with distinct eigenvalues. Then is unitarily similar to triangular matrix with ordered eigenvalues and is unique up to diagonal unitary similarities: , where .
If is invertible and is TIN, then is the Cholesky factor of times a diagonal unitary matrix:
Let be a TIN pair with rank , then
where is a diagonal unitary matrix. In the real case, is a signature matrix.
Proof: Let be the Cholesky factor of . Clearly some unitary with . By the uniqueness of the LQ factorization, and are unique. Note is lower triangular and therefore diagonal.
3 Band Fraction Representations of Single Input Tin Pairs.
For the single input case (), TIN pairs have an explicit band fraction representation , , where and are bidiagonal. The representation is parameterized by the eigenvalues of . For each eigenvalue, , of , we define , and . We define the matrices:
where the top row contains the diagonal elements of or and the bottom row contains the elements of first subdiagonal of the bidiagonal matrices, and .
Proof: Explicit evaluation shows .
The Hautus criterion states that if a stable matrix, , is nonderogatory (There is only one Jordan block for each eigenvalue of in the eigendecomposition of .), then there is a vector such that is controllable. Thus the Hautus criterion together with Theorem 2.3 imply that
Let be a stable, nonderogatory matrix. Then there exists a similarity transformation, , such that has the bidiagonal fraction representation of Theorem 3.1.
The bidiagonal fraction can be casted in a more aestetic form , where and are defined as
with and .
In a series of excellent articles [19, 2, 13, 17, 15, 14], a number of researchers have constructed IN filters using cascade realizations. We now consider the case single input case where has real eigenvalues. In this case, the cascade realization of the IN filter is triangular. By Corollary 2.5, there is a unique triangular state advance matrix with the given eigenvalue ordering (up to diagonal similarity transformations with elements). Thus all state space constructions of IN filters are simply different realizations of the same pair (up to sign flips of the coordinates). In Section 5, we discuss the numerics of the various realizations of IN filters.
We now give a more constructive proof of equivalence of the band fraction representation with the orthogonal basis functions of . Given a set of decay rates/poles of the frequency responses, , Ninness and Gustafsson derive a set of orthonormal basis functions in the frequency domain:
where is or . From this, we derive the relation:
for with and . Equating like powers of yields
with . This shows that the bidiagonal matrix fraction representation generates the same basis functions as in . The -th component of is to be identified with the -th lead of the -th orthonormal transfer function, .
We show that if the eigenvalues are sorted in order of increasing magnitude, then does not have large elements. We now give two results concerning the bidiagonal matrix fraction representation . The first result shows that if the eigenvalues are sorted in order of increasing magnitude, then the bidiagonal factorization is well-conditioned. We then give the eigenvectors of .
A. Explicit inversion.
We can interpret the bidiagonal matrix fraction representation as an LR factorization of the augmented system: . We now explicitly invert in (3.1) and show that the numerical conditioning of the matrix inversion is good without pivoting. We define the condition number of an invertible matrix, , as , where or [1, 3]. Here is the maximum column sum norm and is the maximum row sum norm.
Let be a stable TIN pair (). If the eigenvalue magnitudes are in ascending order, , then for and .
Proof: The LR factorization is . Now is unit lower triangular with . So
for . Note and for and . Thus . This implies that the LR factorization of is numerically stable without pivoting.
We now evaluate the eigenvectors of the single input TIN system using the parameterization (3.1)-(3.2) for the case of distinct eigenvalues. It is well known that the eigenvectors of triangular matrices satisfy a recursion formula. Let be the -th component of the -th eigenvector with for . Write and diag . The eigenvector equation becomes or element by element:
We set , for and solve the recursion for when :
where the bracketed term is equal to for .
In the next section, we show that real single input IN pairs with complex conjugate eigenvalues have a bidiagonal fraction representation as well. However, we do not give an explicit parameterization of and in terms of the eigenvalues.
4 Mimo Band Fraction Representations of Tin Pairs.
We now prove that generic TIN pairs have a band fraction representation: , , where and are triangular matrices of lower bandwidth and is upper triangular. The banded matrix fraction structure allows state space updates, , in multiplications using (1.3). Theorem 4.3 and Theorem 4.4 show how to parameterize TIN input pairs using parameters.
Let satisfy , where is a diagonal, positive definite matrix and is LT and is an matrix. Let have nonvanishing principal subminors, for , then has an unique band fraction representation: , where and are LT matrices of bandwidth and and for .
Proof: By Theorem 3.2.1 of , has a unique has a unique representation where R is UT and is LT with . Let be the submatrix of containing columns through . Since A is LT, is LT of bandwidth . Note . By Theorem 4.3.1 of , has bandwidth . We set and
For , every controllable TIN pair has nonvanishing principal minors. The condition that have nonvanishing principal minors for is generically true. If has an LR decomposition and a vanishing principal minor for , then the induced TIN pair, , does not have a unique LR decomposition of with . From
, the singular values ofare singular values of . When the condition that is relaxed, other band fraction representations may be generated by , , where is a nonsingular diagonal matrix.
The band fraction structure implies that the matrix is upper triangular with upper bandwidth . We now examine parameterizations of the band fraction representation. We define
Let denote the set of all upper triangular matrices with upper bandwidth . Let be the set of such that for and with a positive first nonzero element in the th row.
Note has nonzero coordinates while is an dimensional manifold. We now show that we can construct a TIN pair from any in using the LQ decomposition.
Let have rank and define and as the LQ decomposition of . Define and . Then has lower bandwidth and is invertible and is a TIN pair.
Proof: Note rankrank. Let . Note . By Theorem 4.3.1 of , has bandwidth . From , is triangular and implies is a TIN pair.
We now examine the map , where . From Theorem 9.1 of , has nonzero principal minors for if and only if does. We can rescale and by a nonsingular diagonal matrix : and , and preserve the induced TIN pair, . This freedom and complex phase equivalence allows us to restrict our consideration to :
For each TIN pair, , with nonvanishing principal minors in for , there exists a unique and a unique diagonal unitary matrix such that is generated by : .
Proof: Let be the unique band fraction decomposition of with . For an arbitrary diagonal unitary matrix , the set of band fraction representations of is , where is nonsingular and diagonal. Suppose both and generate an equivalent version of , i.e. . Thus for some and where is nonsingular and diagonal and is diagonal and unitary. Since , the condition shows that and are positive matrices. The requirement that forces and .
We can parameterize input normal systems using for . There are two difficulties with this parameterization. First, the representations for in contain many equivalent TIN representations corresponding to different orderings of the eigenvalues of . Second, in the parameterization, the eigenvalues of are only known after one solves by computing the LQ decomposition of .
We now discuss the computational speed of various system representations.
The bidiagonal band fraction representation, , requires only multiplications for a state vector update in a single input system. First, we compute in multiplications. We then solve the bidiagonal matrix equation in multiplications since . multiplications. The matrix operations in the band fraction advance may be implemented using the dtbmv and dtbtrs routines from the optimized LAPACK software . We caution that if has nonreal eigenvalues, then and are complex. Thus the computational advantage of the bidiagonal matrix fraction representation is primarily limited to the case when the eigenvalues of are real.
In , Heuberger et al. propose a cascade realization of IN filters. This cascade representation requires multiplications for a state vector update in a single input system. A different cascade realization of SISO IN pairs is given in Figure 1 of . The Ninness and Gustafson representation uses a lossless cascade followed by a set of AR(1) models . This realization requires at least multiplications for a state update.
The direct form representations  have even faster state advances, but these representations are not IN and can have very ill-conditioned Grammians [11, 14]. We advise against using high order models that are not in input normal form. The tridiagonal form  is another fast representation that is not input normal.
In [12, 18], embedded lossless representations are constructed. For , these embedded lossless filters multiplications per advance . These embedded filters include the cost of evaluating while the IN filter representation has an additional cost of multiplications to compute , so we should credit the embedded filters with multiplications, yielding multiplications. Unfortunately, embedded lossless systems are not input normal.
In , we give a different representation of Hessenberg and triangular input pairs by treating as a projection of a product of Givens rotations. The multiplication count for these Givens product representations is , which for large is twice as large as the band fraction representations. If Householder transformations are used, the multiplication count is again asymptotically. The Givens product representations are a MIMO generalization of the cascade architecture. This approach may be more natural for multivariate case. In the case, the band fraction representation has the advantage that the eigenvalues of are parameters of the system. This eigenvalue parameterization is convenient when the eigenvalues are prespecified or adaptively estimated.
6 Fast System Identification
The band fraction representations may be used for the rapid identification of impulse responses. We remain in the framework where a fixed basis of orthogonal basis functions are given as described in [12, 19, 13, 17, 15, 14] and summarized in the introduction. We then use the band fraction representation of Sections 3-4, so is given. The state vector evolves according to (1.3
). We effectively compute the second moment matrices,and , where is the forgetting factor. The unknown coefficients, , using recursive least squares (RLS). At each time step, we reestimate by solving . This is the normal equations for the least squares estimate of . The th component of represents the th orthonormal transfer function, , applied to the sequence . The resulting estimate of the the th lead of the impulse response is . To solve for ., we recommend using a QR update of the square root of normal equations . Even faster methods are available that use the displacement rank structure of [9, 16]. The interested reader can consult [6, 14] for a comprehensive description of adaptive estimation.
The input normal filter representations are advantageous for many reasons: First, constant , when the ‘true” state space model is used and the forcing noise is white. Thus the regression for is well-conditioned. Similarly, IN filter structures are resistant to roundoff error . Thus IN representations will time asymptotically satisfy the ansatz need by least mean squares (LMS) identification algorithms . This leads to a second advantage of IN filters: Gradient algorithms such as the least mean squares (LMS) algorithm often perform well enough in certain applications to obviate the need for more complicated and computationally intensive RLS algorithms. Third, an update of is possible . Finally, when the advance matrix is triangular, the IN pairs form nested families: is TIN for when is TIN. This nesting may be used in adaptive order selection.
These are advantages of all TIN filter representations. For single input pairs, The advantages of the band fraction representation over the cascade representation are primarily numerical, the faster computational speed ( multiplications versus or more) and the availability of fast linear algebra software for banded matrices.
We have shown that TIN pairs generically have a matrix fraction representation, , where and have lower bandwidth . This structure allows fast state space updates, , and fast solution of matrix equations: . The total operations count is multiplications. We believe that our work is the first study of system representations, where is a matrix fraction of banded matrices, .
For single input filter design, our representations are parameterized explicitly in terms of the eigenvalues, . When is bidiagonal, solving requires only multiplications and may be implemented as a systolic array with order independent latency. As showed in Theorem 3.3, if the eigenvalues are ordered in ascending magnitude, then the matrix inversion is well-conditioned and suitable for fixed point operations. For system identification, the covariance of tends to the identity matrix when the forcing noise is white. Thus least squares regression is both well-conditioned and computationally fast. Since is triangular, one can compute the evolution of all embedded models of dimension simply by projecting the model of dimension onto the first coordinates. Thus the triangular structure simplifies the evaluation of this nested family of models for model order selection.
-  G. H. Golub and C. F. Van Loan, Matrix Computations, third edition, Baltimore: John Hopkins University Press, 1996.
-  P.S.C. Heuberger, P.M.J. Van den Hof, and O.H. Bosgra, ‘A generalized orthonormal basis for linear dynamical systems,” IEEE Trans. Aut. Cont., vol. 40, pp. 451-465, 1995.
-  N. J. Higham, Accuracy and Stability of Numerical Algorithms, Philadelphia: SIAM Press, 1996.
-  R. A. Horn and C. R. Johnson, Matrix analysis, Cambridge: Cambridge University Press, 1985.
-  E. Anderson, et al. LAPACK User’s Guide Philadelphia: SIAM Press, 1999.
-  L. Ljung and T. Söderström, Theory and practice of recursive identification, Cambridge, MA: The MIT Press, 1983.
-  T. McKelvey and A. Helmersson, ‘State-space parameterizations of multivariate linear systems using tridiagonal matrix forms,” Proc. 35th IEEE Conf. on Decision and Control, pp. 1666-1671, IEEE Press, 1996.
B. Moore, ‘Principal components analysis in linear systems: controllability, observability, model reduction,”IEEE Trans. Aut. Cont., vol. 26, pp. 17-32.
-  A. Mullhaupt and K. S. Riedel, ‘Fast identification of innovations filters,” IEEE Trans. Signal Processing, vol. 45, pp. 2616-2619, 1997.
-  A. Mullhaupt and K. S. Riedel, ‘Hessenberg and Schur output normal pair representations,” submitted.
-  A. Mullhaupt and K. S. Riedel, ‘Bounds on the condition number of solutions of the Stein equation,” submitted.
-  R. A. Roberts and C. T. Mullis, Digital Signal Processing, Addison Wesley, Reading, MA, 1987.
-  B. Ninness and F. Gustafsson, ‘A unifying construction of orthonormal bases for system identification,” IEEE Trans. Aut. Cont. vol. 42, pp. 515-521, 1997.
-  B. Ninness, ‘The utility of orthonormal bases in system identification,” Technical Report 9802, Dept. of EECE, University of Newcastle, Australia, 1998.
-  B. Ninness, H. Hjalmarsson and F. Gustafsson, ‘The fundamental role of orthonormal bases in system identification,” IEEE Trans. Aut. Cont. vol. 44, pp. 1384-1407, 1999.
-  A. H. Sayed and T. Kailath, ‘A state-space approach to adaptive RLS filtering,” IEEE Signal Processing Magazine, vol. 11, pp. 18-60, 1994.
-  P.M.J. Van den Hof, P.S.C. Heuberger, and J. Bokor, ‘System identification with generalized orthonormal basis functions,” Automatica, vol. 31, pp. 1821-1831, 1995.
-  A.J. van der Veen and M. Viberg, ‘Minimal continuous state-space parametrizations,” In Proc. Eusipco, Trieste (Italy), pp.523-526, 1996.
-  B. Wahlberg, ‘System identification using Laguerre models,” IEEE Trans. Aut. Cont., vol. 36, pp. 551-562, 1991.