Angles between subspaces and the Rayleigh-Ritz method

AA00000069_00001.pdf

ANGLES BETWEEN SUBSPACES AND THE RAYLEIGH-RITZ METHOD
by
Peizhen Zhu
M.S., University of Colorado Denver, 2009
A thesis submitted to the
Faculty of the Graduate School of the
University of Colorado in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
Applied Mathematics
2012


This thesis for the Doctor of Philosophy degree by
Peizhen Zhu
has been approved
by
Andrew Knyazev, Advisor
Julien Langou, Chair
Ilse Ipsen
Dianne OLeary
Beresford Parlett
October 26, 2012
li


Zhu, Peizhen (Ph.D., Applied Mathematics)
Angles between subspaces and the Rayleigh-Ritz method
Thesis directed by Professor Andrew Knyazev
ABSTRACT
This thesis concerns four interdependent topics. The first one is principal angles
(or canonical angles) between subspaces (PABS) and their tangents. The second topic
is developing algorithms to compute PABS in high accuracy by using sines of half-
angles. The third topic is the Rayleigh quotient (RQ) and the eigenvalue perturbation
theory for bounded self-adjoint operators in Hilbert spaces. The last, but not least,
topic is the Rayleigh-Ritz (RR) method and mixed majorization-type error bounds
of Ritz value and eigenvalue perturbations.
PABS serve as a classical tool in mathematics, statistics, and applications, e.g.,
data mining. Traditionally, PABS are introduced and used via their cosines. The tan-
gents of PABS have attracted relatively less attention, but are important for analysis
of convergence of subspace iterations for eigenvalue problems. We explicitly construct
matrices, such that their singular values are equal to the tangents of PABS, using or-
thonormal and non-orthonormal bases for subspaces, and orthogonal projectors.
Computing PABS is one of the basic important problems in numerical linear
algebra. To compute PABS and the principal vectors in high accuracy for large,
i.e., near tt/2, and small angles, we propose new algorithms by using the sines of
half-angles. We generalize our algorithms to computing PABS in the A-based scalar
product. We also describe a numerically robust and fast way to compute PABS from
the upper triangular matrices of the QR factorizations. Perturbation analysis for the
sines and cosines of half-PABS is provided using the majorization technique. The
results of numerical tests are given, supporting the theory.
iii


Next we turn our attention to the eigenvalue perturbation theory. If x is an
eigenvector of a self-adjoint bounded operator A in a Hilbert space, then the RQ p{x)
of the vector x is an exact eigenvalue of A. In this case, the absolute change of the
RQ \p(x) p(y) \ becomes the absolute error in an eigenvalue p(x) of A approximated
by the RQ p(y) on a given vector y. There are three traditional kinds of bounds of the
eigenvalue error: a priori bounds via the angle between vectors; a posteriori bounds
via the norm of the residual; mixed type bounds using both the angle and the norm
of the residual. In this work, we examine these three kinds of bounds and derive new
identities and bounds for the change in the RQ with respect to the change of the
vectors. We propose a unifying approach to prove known bounds of the spectrum,
analyze their sharpness, and derive new sharper bounds.
Finally, we consider bounds of the absolute changes of Ritz values and eigenvalue
approximation errors. The RR method is a widely used technique for computing
approximations to eigenvalues and eigenvectors of a Hermitian matrix A, from a given
subspace. We take advantage of PABS and the residual matrix to derive several new
majorization-type mixed bounds that improve and generalize the existing results. Our
bounds hold for subspaces not necessary A-invariant and extend the mixed bound of
the absolute change of the vector RQ to multidimensional subspaces.
The form and content of this abstract are approved. I recommend its publication.
Approved: Andrew Knyazev
IV


ACKNOWLEDGMENT
I wish to express my sincere gratitude to my advisor, Professor Andrew Knyazev,
for his generous instruction, continuous patience, endless encouragement, and support
throughout my graduate studies. Without his guidance, the present dissertation
would not be possible. I just cannot thank Professor Knyazev enough.
The PhD committee Chair, Professor Julien Langou, has done a great job orga-
nizing the defense. I would also like to thank the rest of my committee members,
Professors Ilse Ipsen, Dianne 0;Leary, and Beresford Parlett, for their time, attention
and insightful comments.
I would like to express my appreciation to Dr. Merico Argentati, who has been a
knowledgeable and friendly source of advice. His comments are very valuable.
I am grateful to the faculty and my fellow students from the Department of
Mathematical and Statistical Sciences. In particular, I would like to thank Professor
Richard Lundgren and Professor Leopoldo Franca for referring and introducing me
to Professor Knyazev and his group. I would like to thank my fellow students Henc
Bouwmeester, Eric Sullivan, Kannanut Chamsri, Yongxia Kuang, and Minjeong Kim
for sharing their various knowledge.
In addition, I would like to express my gratitude to the Bateman family and the
Department of Mathematical and Statistical Sciences for their invaluable support.
My research and work on the thesis have also been supported by NSF DMS
grants 0612751 and 1115734, the PI Professor Andrew Knyazev. I have presented
some preliminary results of the thesis at the Conference on Numerical Linear Algebra
Austin, TX, July 20, 2010. My talk has been accepted to the Householder Symposium
XVIII on Numerical Linear Algebra June 12-17, 2011, but canceled because of my
pregnancy. I acknowledge the support of the conference organizers.
Finally, I would like to thank my husband Xinhua and my parents for their endless
support and love, and my daughter Jessie for being an inspiration.


TABLE OF CONTENTS
Figures .................................................................. viii
Tables...................................................................... ix
Chapter
1.Introduction............................................................... 1
1.1 Overview............................................................ 1
1.2 Notation............................................................ 5
2. Principal angles between subspaces (PABS) and their tangents.............. 7
2.1 Definition and basic properties of PABS............................. 7
2.2 Tangents of PABS................................................... 10
2.2.1 tan 0 in terms of the orthonormal bases of subspaces... 11
2.2.2 tan 0 in terms of the bases of subspaces............... 16
2.2.3 tan in terms of the orthogonal projectors on subspaces ... 21
2.3 Conclusions ....................................................... 26
3. New approaches to compute PABS ....................................... 27
3.1 Algebraic approaches to compute sin (and cos (.) 29
3.2 Geometric approaches to compute sin ).............................. 37
3.3 Computing sin and cos ff) without using orthonormal bases . 42
3.4 Computing sin ()and cos (in the A-based scalar product ... . 49
3.5 Matlab implementation.............................................. 58
3.6 Numerical tests ................................................... 60
3.7 Discussion......................................................... 67
3.8 Conclusions ....................................................... 70
4. Majorization inequalities................................................ 71
4.1 Definition and properties of majorization.......................... 71
4.2 Some known majorization results ................................... 73
4.3 Unitarily invariant norms, symmetric gauge functions, and majorization 75
vi


4.4 New majorization inequalities for singular values and eigenvalues . 77
4.5 Perturbation analysis for principal angles........................... 81
4.6 Conclusions ......................................................... 92
5. Bounds for the Rayleigh quotient and the spectrum of self-adjoint operators 94
5.1 Introduction......................................................... 94
5.2 Short review of some known error bounds for eigenvalues ............. 96
5.3 Key identities for RQ................................................ 98
5.3.1 Identities for the norm of the residual r (x) = Ax p(x)x . 99
5.3.2 Identities for the absolute change in the RQ.................100
5.4 Deriving some known eigenvalue error bounds from our identities ..104
5.5 New a posteriori bounds for eigenvalues..............................105
5.6 Improving known error bounds for eigenvectors........................108
5.7 Conclusions .........................................................Ill
6. Majorization bounds for the absolute change of Ritz values and eigenvalues 112
6.1 Rayleigh-Ritz method.................................................114
6.2 Perturbation of eigenspaces..........................................115
6.3 Majorization-type mixed bounds.......................................117
6.4 Discussion...........................................................126
6.5 Conclusions .........................................................130
7. Conclusions and future work.................................................131
Appendix
A. MATLAB code.................................................................133
References.....................................................................150
Vll


FIGURES
Figure
2.1 The PABS in 2D.........................................................11
2.2 Geometrical meaning of r =3(3)................................ 24
3.1 Smes and cosines or nalr-angles........................................ 32
3.2 Sine of half-angle between x and y in M.n.............................. 38
3.3 Errors in principal angles: p = 20 top and p as a function of n such that
p = n/2 bottom; left for Algorithm 3.1 and Algorithm 3.4 with A = /,
and right for Algorithm sine-cosine and Algorithm sine-cosine with A L 62
3.4 Errors in the principal angles for Algorithms 3.1, 3.2, and sine-cosine based
algorithm with D = diag(10_16rand^1^) and p = 20...................... 63
3.5 Error distributions for individual angles: top three for Algorithm 3.1,
Algorithm 3.2, and Algorithm sine-cosine; bottom three for Algorithms
3.4, 3.5, and sine-cosine with A =1................................... 65
3.6 Errors in the principal angles for Algorithms 3.3 and 3.6.............. 66
3.7 Sine of angles as functions of k for Algorithms 3.4, 3.5, and 3.6 (top and
left on bottom); Errors as functions of condition number for Algorithms
3.4, 3.5, and 3.6 (right on bottom)................................... 67
4.1 Geometry of majorization in 2D and 3D.................................. 72
6.1 Comparison of the absolute change of Ritz values for the mixed bounds
and for the a priori bounds with the exact absolute change of Ritz values
(left); The corresponding tangent of the largest principal angle (right)..129
6.2 Comparing the mixed type bound of the absolute change of Ritz values
with the exact absolute change of Ritz values.........................129
viii


TABLES
Table
2.1 Different matrices in T\ matrix T G J7 using orthonormal bases...... 15
2.2 Different expressions for T......................................... 25
2.3 Different formulas for T with 0(< tt/2: dim(< dim(V) (left);
dim( dim(3^) (right)........................................... 26
3.1 Algorithm 3.1...................................................... 33
3.2 Algorithm 3.2...................................................... 41
3.3 Algorithm 3.3...................................................... 46
3.4 Algorithm 3.4...................................................... 53
3.5 Algorithm 3.5...................................................... 54
3.6 Algorithm 3.6...................................................... 57
3.9 Computed sines and cosines of principal angles.................... 61
3.10 Algorithms to compute PABS and PABS m the A-based scalar product. 68
3.11 Comparison for algorithms to compute PABS......................... 69
3.12 Comparison for algorithms to compute PABS m the A-based scalar prod-
uct (1)............................................................. 69
IX


1.Introduction
1.1 Overview
This dissertation mainly consists of four parts. The first part deals with principal
angles (or canonical angles) between subspaces (PABS) and their tangents. The
second part develops algorithms to compute PABS in high accuracy by using sines
of half-angles. The third part studies the Rayleigh quotient (RQ) and the eigenvalue
perturbation theory for bounded self-adjoint operators in Hilbert spaces. The last part
explores the Rayleigh-Ritz (RR) method and properties of Ritz values and eigenvalues
of Hermitian matrices.
The history of PABS dates to Jordan in 1875 [40]. In statistics, the cosines of
principal angles and principal vectors are interpreted as canonical correlations and
canonical variables, correspondingly. PABS are widely used in many applications,
e.g., in ecology, information retrieval, random processes, face recognition, and system
identification; e.g., [27, 39, 43, 45, 78]. The cosines, sines, or tangents of PABS
are commonly used to measure the accuracy of numerical computation of invariant
subspaces of matrices; e.g., [47, 51,64, 73].
The sines and especially cosines of PABS are well studied; e.g., in [9, 74]. Tangents
of PABS also appear in applications, but are less investigated. In [10, 20, 52, 71,73],
researchers define the tangents of PABS of the same dimension via the norm or
singular values of some matrices, and use the tangents as analytic tools. Specifically,
the tangents of PABS are related to the singular values of a matrixwithout explicit
formulation of the matrixin [73, p. 231-232] and [71, Theorem 2.4, p. 252]. The
tangent of the largest principal angle is obtained from the norm of a matrix in [10, 20].
In [52], the tangents are used to derive convergence rate bounds of subspace iterations.
In Chapter 2, we briefly review the concepts and the most fundamental and im-
portant properties of PABS. We give explicit matrices, such that their singular values
are the tangents of PABS. We construct matrices using three different scenarios:
1


orthonormal bases, non-orthonormal bases, and orthogonal projectors. Using a geo-
metric approach, new constructions of such matrices are obtained. In addition, our
results include the formulations of matrices consistent with those in [10, 20, 52, 71,73].
Furthermore, we present the tangents of PABS with possibly different dimensions.
Computing PABS is one of the basic important problems in numerical linear
algebra. An efficient method, based on singular value decomposition (SVD) for cal-
culating cosines of PABS, is proposed by Bjorck and Golub [9]. The cosines of prin-
cipal angles and principal vectors are obtained by using the SVD of XHY^ where the
columns of matrices X and Y form orthonormal bases for the subspaces X and 3^,
correspondingly.
However, the cosine-based algorithm cannot provide accurate results for small an-
gles, i.e., angles smaller than 10-8 are computed as zero in double precision arithmetic
with EPS ^ 10-16. For the same reason, the sine-based algorithm loses accuracy for
computing large principal angles, i.e., close to tt/2. A remedy is to combine sine
and cosine based algorithms proposed by Knyazev and Argentati [48], where small
and large angles are computed separately. They introduced a threshold to compute
the angles less than the threshold using the sine-based approach and to compute the
angles greater than the threshold using the cosine-based approach. However, this
algorithm is complicated, and introduces a new potential difficultyaccurate com-
putation of the principal vectors corresponding to the PAB^ m the neighborhood of
the threshold.
In Chapter 3, we propose new algorithms to compute PABS and the principal
vectors using the sines of half-PABS. Our new algorithms provide accurate results
for all PABS m one sweep and can be implemented simply and efficiently. We also
propose a method to compute PABS for sparse data. Furthermore, we generalize our
algorithms to compute the principal angles and the principal vectors in an A-based
scalar product as well, where Aisdu positive definite Hermitian matrix. Perturbation
2


analysis for the sines and cosines of half-PABS is provided in this work. Results of
extensive numerical tests are included.
In Chapter 4, we briefly review some existing majorization inequalities based on
the singular values and the eigenvalues of matrices. Using majorization technique, we
bound the absolute changes of eigenvalues of two Hermitian matrices in terms of the
singular values of the difference of two matrices. We also use majorization technique
to bound the changes of sines and cosines of half-PABS, and of tangents of PABS.
If x is an eigenvector of a self-adjoint bounded operator A in a Hilbert space, then
the RQ p{x) of the vector x is an exact eigenvalue of A. In this case, the absolute
change of the RQ \p(x) p(y) \ becomes the absolute error in an eigenvalue p(x) of A
approximated by the RQ p(y) on a given vector y. There are three traditional kinds of
bounds of eigenvalue errors: a priori bounds [49, 53, 67] via the angle between vectors
x and y\ a posteriori bounds [13, 44, 64] via the norm of the residual Ay p(y)y of a
vector y\ and mixed type bounds [47, 75] using both the angle and the norm of the
residual. In Chapter 5, we propose a unified approach to prove known bounds of the
spectrum, analyze their sharpness, and derive new sharper bounds.
The RR method [64, 73] is a classical method for eigenvalue approximation of
Hermitian matrices. A recent paper by Knyazev and Argentati [51] presents a priori
error bounds of Ritz values and eigenvalues in terms of the principal angles between
subspaces X and 3^ using a constant. In [5, 64, 73], researchers derive a posterior
bounds of eigenvalues in terms of the singular values of the residual matrix. Another
a posterior bound obtained by Mathias [56] uses the residual matrix and the gap.
In Chapter 6, we take advantage of the principal angles and the residual matrix
to present new majorization-type mixed bounds. We improve the bounds in [61,75]
if one of the subspaces is A-invariant. In addition, our bounds are more general, since
they hold for subspaces not necessary A-invariant. We generalize the mixed bound
of the absolute change of the RQ \p(x) p(y)\ to multidimensional subspaces. This
3


work also provides majorization-type mixed bounds for Ritz values and eigenvalues,
and for RQ vector perturbation identities for application in generalized eigenvalue
problems. We compare our new bounds with the known ones.
We present our conclusions and recommend some further work in Chapter 7.
The main new results obtained in the thesis are summarized as follows:
1. We analyze the tangents of PABS related to the SVD of explicitly constructed
matrices. New properties of PABS are obtained.
2. Original numerical algorithms are developed, based on the sines of half-PABS,
to compute PABS and principal vectors for column ranges of dense and sparse
matrices. Moreover, we generalize our algorithms to compute PABS in the
A-based scalar product.
3. We use majorization techniques to present novel perturbation analysis for sines
and cosines of half-PABS.
4. A unifying approach is proposed, based on new RQ vector perturbation iden-
tities, to prove bounds of the operator spectrum, analyze their sharpness, and
derive new sharp bounds.
5. We present several majorization-type mixed bounds for the absolute changes
of Ritz values and the RR absolute eigenvalue approximation error. The
majorization-type mixed bounds are in terms of the principal angles and the
residual matrix.
4


1.2 Notation
A = AH A Hermitian matrix in Cnxn or a self-adjoint bounded operator.
S(A) The vector or singular values of A and 5(A) = [si(A)^..., sn(A)]
S+(A) The vector of positive singular values of A.
S^(A) The singular values of A are arranged in nonincreasing order.
Sm^(A) The vector of the largest m singular values of A m nonincreasing order.
Si(A) The vector of the smallest m singular values of A in nonincreasing order.
5 max (^4) The largest singular value of A.
^min (^4) The smallest singular value of A.
w Vector norm of x.
Pll Spectral norm of the matrix A, i.e., \\A\\ = smax(A).
PI Frobenius norm of A.
IPIII Unitarily invariant norm of A.
Subspaces of Cnxn.
1 The orthogonal complement of the subspace X.
The column space of a matrix X.
Px Orthogonal projector on the subspace X or TZ(X).
P^ Orthogonal projector on the subspace
A The vector or eigenvalues of A and (A) = [A^A)n(A)].
A The eigenvalues of A are arranged in nonincreasing order.
0(^,3^) The vector of principal angles between the subspaces X and 3^, and 0(y) = [6*1...(9P].
H^,y) Principal angles are arranged in nonincreasing order. 5


;^) Principal angles are arranged in nondecreasing order.
(0,/2) The vector of principal angles in (0,tt/2).
eA(x,y) The vector of principal angles in the A-based scalar product between X and 3^ .
max(x,y) The largest angle between subspaces X and 3^.
min(x,y) The smallest angle between subspaces X and 3^.
rank i A) The rank of the matrix A.
x -< y The vector x is majorized by the vector y.
'w y The vector x is weakly majorized by y.
The absolute value of the vector x.
At Moore-Penrose inverse of A.
re re(A) =
pol .. The square root of the matrix AHA^ i.e., \A\po\ = {AHA)1/2. An inner product, associated with a norm by 2 =(*,*).
_B The scalar product induced by BJ where B is positive definite
The condition number of a matrix A in the 2 norm, such that _ Smax()/Smin().
p(x) The Rayleigh quotient of A with respect to the vector x.
H A real or complex Hilbert space.
spec(A) Spectrum of the bounded operator A.
6


2. Principal angles between subspaces (PABS) and their tangents1
The concept of principal angles between subspaces (PABS) is first introduced by
Jordan [40] in 1875. Hotelling [37] defines PABS in the form of canonical correlations
in statistics in 1936. Numerous researchers work on PABS; see, e.g., our pseudo-
random choice of initial references [18, 38, 50, 73, 74, 79], out of tens of thousand of
Internet links for the keywords principal angles and canonical angles.
In this chapter, we first briefly review the concept and some important properties
of PABS in Section 2.1. Traditionally, PABS are introduced and used via their sines
and more commonly, because of the connection to canonical correlations, cosines. The
tangents of PABS have attracted relatively less attention. Our interest to the tangents
of PABS is motivated by its applications in theoretical analysis of convergence of
subspace iterations for eigenvalue problems.
We review some previous work on the tangents of PABS in Section 2.2. The main
goal of this chapter is explicitly constructing a family of matrices such that their
singular values are equal to the tangents of PABS. We form these matrices using sev-
eral different approaches: orthonormal bases for subspaces in Subsection 2.2.1, non-
orthonormal bases in Subsection 2.2.2, and orthogonal projectors in Subsection 2.2.3.
Throughout this chapter, we also discover new properties of PABS.
2.1 Definition and basic properties of PABS
In this section, we remind the reader of the concept of PABS and some funda-
mental properties of PABS. First, we recall that an acute angle between two nonzero
vectors x and y is defined as
cos0(x,y) = \xHy\,
with xHx = yHy =1,where 0 < 6(x^y) < tt/2.
1The material of this chapter is based on our manuscript, P. Zhu and A. V. Knyazev, Principal
angles between two subspaces and their tangents^ submitted to Linear Algebra and its Applications.
Also, it is available at http://arxiv.Org/abs/l209.0523.
7


The definition of an acute angle between two nonzero vectors can be extended to
PABS; see, e.g., [9, 24, 30, 37].
Definition 2.1.1 Let C Cn and Y C Cn 6e dim()=p and
dim(3^) = q. Let m = min (p, q). The principal angles
0(^,3^) = [6*i,...,6*m] where 9k G [0,tt/2], fc =1,...,m,
between X and y are recursively defined by
=cos(4) = maxmaxlxHyl = I
subject to
IIII = IMI =ixIlxi = > yHVi = > i =1,...,k -l.
The vectors {x\,..., xm} and {...ym} are called the principal vectors.
An alternative definition of PABS is proposed in [9, 30] based on the singular
value decomposition (SVD) and reproduced here as the following theorem. It is
shown in [24] that the SVD approach to define PABS is a direct consequence of the
recursive definition.
Theorem 2.1.2 Let the columns of matrices X G Cnxp and Y G Cnxq form or-
thonormal bases for the subspaces X and y, correspondingly. Let the SVD of XHY
he UTjVh ^ where U and V are unitary maznces and ^ is a p by q diagonal matrix
with the real diagonal elements Si(XhY)^ ..., sm(XHY) in nonincreasing order with
m = min(p, q). Then
cseW j) = s (xHy) = [Sl(xHy).. (xHy)]
where denotes the vector of principal angles between X and y arranged in
nondecreasing order and 5(A) denotes the vector of singular values of A. Moreover,
the principal vectors associated with this pair of subspaces are given by the first m
columns of XU and YV, correspondingly.


Theorem 2.1.2 implies that PABS are symmetric, i.e. 0(^,3^) = 0(3^,^), and
unitarily invariant, since (UX)H(UY) = XHY for any unitary matrix U G Cnxn.
A number of other important properties of PABS have been established, for finite
dimensional subspaces, e.g., in [23, 38, 50, 73, 74, 79], and for infinite dimensional
subspaces in [18, 53]. We list a few of the most useful properties of PABS below.
Property 2.1.3 [74] In the notation of Theorem 2.1.2, let p > q and q [XX] he a unitary matrix. Let Si(X^Y) > > sq(X^Y) he singular values of
XfY. Then
sk{XfY) = sm(9q+1_k), 1..¢.
Relationships of principal angles between X and 3^, and between their orthogonal
complements are thoroughly investigated in [38, 50, 53]. Let the orthogonal comple-
ments of the subspaces X and 3^ be denoted by and correspondingly. The
nonzero principal angles between X and 3^ are the same as those between and
y-1. Similarly, the nonzero principal angles between X and y1-are the same as those
between X1-and 3^.
Property 2.1.4 [38, 50, 53] For the subspaces X, y and their orthogonal comple-
ments, we have
i. [ey),..] = [e)...].
w/zere ere are max(n dim(dim(Y)0) additional Os on the left and
max(dim(A,) + dim(3^) n,0) additional Os on the right.
[00...0] = [0(y)0...0].
where there are dim(X)Q) additional Os on the left and there
are max(dim(A,) dim(3^),0) additional Os on the right.
* [f...f)]=[f (),...]
where there are max(dim(A,) dim(3^),0) additional ir/2s on the left and
9


max(dim(A,) + dim(3^) n,0) additional Os on the right.
These statements are widely used to discover new properties of PABS.
2.2 Tangents of PABS
The tangents of PABS serve as an important tool in numerical matrix analysis.
For example, in [10, 20], the authors use the tangent of the largest principal angle
derived from a norm of a specific matrix. In [71, Theorem 2.4, p. 252] and [73, p. 231-
232] the tangents of PABS, related to singular values of a matrixwithout an explicit
matrix formulationare used to analyze perturbations or invariant subspaces. In [52],
the tangents are used to derive the convergence rate bounds of subspace iterations.
In [10, 20, 52, 71,73], the two subspaces have the same dimensions.
Let the orthonormal columns of matrices X, X_l, and Y span the subspaces
the orthogonal complement of X ^ and 3^, correspondingly. According to Theo-
rem 2.1.2 and Property 2.1.3, 008 0(^,3^) = S(XHY) and sin0(^,3^) = S(X^Y).
The properties of sines and especially cosines of PABS are well investigated; e.g., see
in [9, 74]. The tangents of PABS have attracted relatively less attention. One could
obtain the tangents of PABS by using the sines and the cosines of PABS directly,
however, this is not the most efficient or accurate approach in some situations.
In this work, we construct a family T of explicitly given matrices, such that the
singular values of the matrix T are the tangents of PABS, where T G J7. We form T
in three different ways. First, we derive T as the product of matrices whose columns
form the orthonormal bases of subspaces. Second, we present i1 as the product of
matrices with non-orthonormal columns. Third, we form T as the product of or-
thogonal projections on subspaces. For better understanding, we provide a geometric
interpretation or singular values of T and of the action of T as a linear operator.
Our constructions include the matrices presented in [10, 20, 52, 71,73]. Further-
more, we consider the tangents of principal angles between two subspaces not only
with the same dimension, but also with different dimensions.
10


2.2.1 tan in terms of the orthonormal bases of subspaces
The sines and cosines of PABS are obtained from the singular values of explicitly
given matrices, which motivate us to construct T as follows: if matrices X and Y
have orthonormal columns and XHY is invertible, let T = X^Y (^XHY^j \ then
tan 0(^,3^) can be equal to the positive singular values of T. We begin with an
example in two dimensions in Figure 2.1.
Figure 2.1: The PABS in 2D.
cos /3 sin/3 cos (6 + P)
Let X = = , and Y =
sin/3 cos /3 sin (8 + P)
where 0 < 61 < tt/2 and /3 G [0,tt/2]. Then, XHY = cos^ and X^Y = sin^.
Obviously, tan6* coincides with the singular value of T = X^Y ^XHY^ \ If 61 = tt/2,
then the matrix XHY is singular, which demonstrates that, in general, we need to
use its Moore-Penrose Pseudoinverse to form our matrix T = X^Y
The Moore-Penrose Pseudoinverse is well known. For a fixed matrix A G Cnxm,
the Moore-Penrose Pseudoinverse is the unique matrix A"' G Cmxn satisfying
AAM = = #A4=A4#=AM.
11


Some properties of the Moore-Penrose Pseudoinverse are listed as follows.
HA UTjVh is the singular value decomposition of A, then = VY^UH.
HA has full column rank, and B has full row rank, then {AB)^ .
However, this formula does not hold for general matrices A and
AA' is the orthogonal projector on the range of A, and A"'A is the orthogonal
projector on the range of AH.
For additional properties of the Moore-Penrose Pseudoinverse, we refer the reader
to [73]. Now we are ready to prove that the intuition discussed above, suggesting to
try T = XfY (XhY)\ is correct. Moreover, we cover the case of the subspaces X
and 3^ having possibly different dimensions.
Theorem 2.2.1 Lei X G Cnxp have orthonormal columns and he arbitrarily com-
pleted to a unitary matrix [XX\. Let the matrix Y G Cnxq he such that YHY = I.
Then the positive singular values S+(T) of the matrix T = XfY [XhyY satisfy
tanQ(7l(X)^7l(Y)) = [oc, S+(T)^ (2.2.1)
where TZ(.) denotes the matrix column range.
Proof: Let [y y_j_] be unitary, then [x X]H[YY] unitary, such that
[xx]h[yy]
q n q
V xhy xhy
n p XfY X^Y
Applying the classical CS-decomposition (CSD), e.g., [22, 28, 62, 63], we get
H
[xx]h[yy]
XhY XhY
XfY X^Y
t/l
u2
D


12


where U\ G W(p), U2 G U(n p), Vi G W(g), and V2 G U(n q). The symbol U{k)
denotes the family of unitary matrices of the size k. The matrix D has the following
structure:
r s qrs npq-\-r s prs
D =
r s prs 1 Cl 0 0H Si I
npq-\-r 0 I
s Si
qrs I 0H
where C\ = diag(cos(6li),...,cos(6l8)), and S\ = diag(sin(6li),...,sin(6s)) with k G
(0, tt/2) for fc =1,...,s, which are the principal angles between the subspaces TZ(Y)
and TZ(X). The matrices C\ and S\ could be empty. Matrix 0 is the matrix of zeros
and does not have to be square. / denotes the identity matrix. We may have different
sizes of / in D.
Therefore,
T = XfY {XhY)] = U2 0 Si Vfv, I Cl t Uf = u2 0 SiC^1
I 0 0H
Hence, S^(T) = (tan(6li),...,tan(6l8)), where 0 < 61 < 62 < - < 6S < tt/2.
From the matrix 15, we obtain 5" = 5" (diag(/C"i0)). According to The-
orem 2.1.20(7^(X)7^(y)) = [0,0,6^...tt/2, ...tt/2]where (9 G (0tt/2)
for fc =1,...,s, and there are min(q r s^p r s) additional tt/2^s and r additional
07s on the right-hand side, which completes the proof.
Remark 2.2.2 If we know the subspaces Xy and their orthogonal complements
explicitlywe can obtain the exact sizes of block matrices in the matrix D. Let us
13


consider the decomposition of the space Cn into an orthogonal sum of five subspaces
as in [11,31,53]7
Cn =
^ere00 =01 = tny10 =n =ny(=(X)
and y = TZ{Y)). Using Tables 1 and 2 in [53] and Theorem 3.12 in [41]7 we get
dim() = r
dim(9?ti) = g r s,
diir^SDtn) = n n <7 + r
dim(9?ti)=p r s,
w/zere SDt = =2%
=1n (01)
=1n (10n)
9% n (10)
n0?tol
and s = dim(SDt^) = dim(dJly) = dim(dJlx^) = dim(SDt^?). Thus, we can represent
D as
dim(OTc dim(OT)/2 dim(OT10) dimfOTn) dim(OT)/2 dim(OT01)
dim(OT) i 0H
dim(OT)/2 c\ Si
0 I
dim(OTn) 0 I
dim(OT)/2 Si
I 0H
In adaition^ it is possible to permute trie flrst q columns or the last n q columns of
D, or the first p rows or the last n p rows and to change the sign of any column or
urn to obtain the variants of the CSD.
14


Remark 2.2.3 In Theorem 2.2.1, there are additional dim(Xny) 0\s and additional
min (dim( H Y)dim( H )ocs in [,...0, S"+(T)0,0].
Due to the fact that the angles are symmetric, i.e., 0(^,3^) = 9( the
matrix T in Theorem 2.2.1 could be substituted with Yj1 X(YHXy. Moreover, the
nonzero angles between the subspaces X and 3^ are the same as those between
the subspaces and .Hence, T can be presented as XHYj_(X^Yj_y and
YHX_L(Y(1X_Ly. Furthermore, for any matrix T, we have S(T) = which
implies that all conjugate transposes of T hold in Theorem 2.2.1. Let T denote a
family of matrices, such that the singular values of the matrix in T are the tangents
of PABS. To sum up, any of the formulas for T in J7 in the first column of Table 2.1
can be used in Theorem 2.2.1.
Table 2.1: Different matrices in T\ matrix T G J7 using orthonormal bases.
XlY (XhyY ZVY :1^)t
YfX{YHXy Fy1(1^
xhy{x^yY PXY(X^YY
YHX{YfXy PyX(YfXy
{yhxYyhx
{xhyYxhy (xHyyxHpyi
{YfXYYfX (Yfxyrfpx
{XfY^XfY {XfY^XfPy
Using the fact that the singular values are invariant under unitary multiplications,
we can also use for in Theorem 2.2.1, where is ail orthogonal
projector onto the subspace Thus, we can use T as in the second column in
Table 2.1, where ,,and denote the orthogonal projectors on the subspace
, and ,correspondingly. Note that =/ and =/ _Py.
15


Remark 2.2.4 From the proof of Theorem 2.2.1 ^ we immediately derive that
XfY {XhY)] = -(YfXyYfX and YfX(YHXy = -{XfY^XfY.
Remark 2.2.5 Let XfY {XhY)] = U2^iU1 and YfX(YHXy = V2E2Vh be the
SVDs, where Ui and (z =1,2) are unitary matrices and (z =1,2) are diagonal
matrices. Let the diagonal elements of Si and S2 be in the same order. Then the
principal vectors associated with the pair of subspaces X and y are given by the
columns of XUi and YVi.
2.2.2 tan in terms of the bases of subspaces
In the previous section, we use matrices X and Y with orthonormal columns
to formulate T. An interesting question is whether we can choose non-orthonormal
bases for these subspaces to get the same result as in Theorem 2.2.1. It turns out
that we can choose a non-orthonormal basis for one of the subspaces X or y. Before
we present the theorem, we need the following lemma.
Lemma 2.2.6 For integers q the matrixY G Cnxq be such thatXHY has full rank. Let us define Z = Y(YHY)~1^2,
f = Xfz (XHZ)\ and T = XfY {XHY)]. Then T = T.
Proof: The matrix XHY(p x q) has full rank and q < which implies that the
matrix Y has full rank, i.e., dim(7?.(y)) = dim(7l(XHY)) = q. So, the matrix YHY is
nonsingular. Moreover, the columns of Z = Y(YHY)~1^2 are orthonormal and span
TZ(Y). Using the fact that if A is of full column rank, and B is of full row rank, then
(ABy = it follows that (XHZY = {XHY{YHY)~l/2y = {YHY)l/2{XHYy.
Finally, we have
f = Xfz {XHZ)] = XfY{YHY)~l/2{YHY)l/2{XHY)] = XfY{XHY)] = T.
16


In general, if g > p, the matrices T and T are different. For example, let
1 0 0 ii
X = 0 = 10 , and Y = 0 1
0 0 1 0 0
Then, we have =[1i]and [1/2 1/2.Thus,
T^XfY (X^y)f = [1/2 0]^-
On the other hand, we obtain T = X^Z = [ ]-^.
From the proof of Lemma 2.2.6, we see that the matrix (XHZy is not equal to
(YHY)1^2(XHYy for the case q > since in general the formula {AB)^ does
not hold. Furthermore, in the example above we have
cos0(^(X),^(y)) = s(XhY(YhY)~1/2)=1,
which implies that tan0(7?.(X),7?.(y)) = 0. However, s(T)=1/2, which also shows
that the condition q < p is necessary in the following theorem.
Theorem 2.2.7 For integers q < p7 let X G (Cnxp have orthonormal columns and be
arbitrarily completed to a unitary matrix [XXi]. Let Y 6 Cnxq be such that XhY
has full rank. Let T = XY [XhyY Then
tan0(^(X),^(y)) = [5+(T),
where there are dim(TZ(X) r\TZ(Y)) 0\s on the right.
Proof: Combining Theorem 2.2.1, Remark 2.2.3, and Lemma 2.2.6 directly
proves this theorem.
Remark 2.2.8 We make the explicit matrix T in Theorem 2.2.1 connection to the
implicit matrix which is related to the tangents of angles described m [73, 71].Using
the idea described in [73, p. 231-232]7 [71, Theorem 2.4, p. 252]7 and [52]7 we construct


Z as X+X_\_T. Since [XHYy XHY = Ithe following identities Y = PxY + Pp^Y =
XXhY + XX^Y = ZXhY imply that TZ(Y) C TZ(Z). By direct calculation^ we
easily obtain that
XhZ = Xh(X + XT) = Id,iidZHZ= (X + XT)h(X + XT) = I + ThT.
Thus, XHZ (yZHZ) = (/ + thT)~1/2 is Herrnitian positive definite. The ma-
trix Z{ZUZ)~xl2 by construction has orthonormal columns which span the space Z.
Moreover, we observe that
S (^XHZ(ZHZ)-^ = A ((/ + ThT)~-^ =[(1 + (T))-^ ,...,(l + s2p (T))-1],
where A[.) denotes the vector of eigenvalues.
Therefore, tanQ(7l(X)^7l(Z)) = [^(T), 0,..., 0] and dim(X) = dim(Z). From
Theorem 2.2.7, we have tan0()7r/2)(7^(X),7^(y)) = tan0()7r/2)(7^(X),7^(Z)), where
{/2) denotes all PABS in {0,it/2). In other words, the angles in {0,it/2) be-
tween subspaces TZ(X) and TZ(Y) are the same as those between subspaces TZ(X)
andTZ(Z). For the case p = q) we can see that Q(TZ(X)^TZ(Y)) = Q(TZ(X)^TZ(Z)).
We give an explicit matrix X^Y [XHY^j 1 for the implicit matrix P7 where S(P)=
tanQ(7l(X)^7l(Y))? in [73, p. 231-232] and [71, Theorem 2.4, p. 252]. The case
p = q is considered in [52]. We extend their result to the case q Corollary 2.2.9 Using the notation of Theorem 2.2.1^ let p = q and he an
orthogonal projection on the subspace TZ(X_\_). We have
tan0((X)T)) = S .
Proof: Since the singular values are invariant under unitary transforms, it is easy
to obtain S (^X^Y (^XHY^j 1 j = 5" Y . Moreover, the
mber of the singular values of y (X1 is p, hence we obtain the result.
Above we always suppose that the matrix Y has full rank. Next, we derive a
similar result as in Theorem 2.2.7, if y is not a full rank matrix.
18


Theorem 2.2.10 Let [X X_l] he a unitary matrix with X G Cnxp. Let Y G
£nxq an(^ ran]^(y) = rank(X-^y) < p. Then the positive singular values of
T = X^Y [XHY^satisfy tanQ(TZ(X)^7l(Y)) = [^(T), 0,..., 0], where there are
dim(TZ(X) DTZ(Y)) 0Js on the right
Proof: Let the rank of Y be r, thus r < p. Let the SVD of Y be UTjVh ^ where U is
an n x n unitary matrix; S is an n x g real rectangular diagonal matrix with diagonal
entries Si,...,0,..., 0 ordered by decreasing magnitude such that Si > S2 > >
> 0, and V is du q x q unitary matrix. Since rank(y)=r < g, we can compute a
reduced SVD such that Y = Ur^V/1. Only the r column vectors of U and the r row
vectors of VH^ corresponding to nonzero singular values are used, which means that
is an r x r invertible diagonal matrix. Let
Z1 = Y (YhY)+1/2 V = UrErVrH [V (Sf Sr)_1 K^]1/2
Let C = VrH Vr 1 VrH Vrj then C is Hermitian and invertible. More-
over, we have Zi = Irxr. Since the matrix Y has rank r, the columns of ^ are
orthonormal and the range of ^ is the same as the range of Y. Therefore, we have
tan e(TZ(X),n(Y)) = tan e(TZ(X),n(Z1)).
Moreover, let Ti=X^_Z\ (^X11 . According to Theorem 2.2.1, it follows that
tan ((X)=[)0,0]
where there are dim(TZ(X) Pi TZ{Zi)) 07s on the right. Our task is now to show that
Ti=T. By direct computation, we have
Ti=X^UrErC (XHUrEry
= X^UrErCCj (XHUr^ry
= X^UrEr (XHUrEry.
19


The second equality is based on [XHUrYjrCy = {XHUrYjr^\ since rank [XHY^j =
r. Since the matrix C is invertible, we obtain the last equality. On the other hand,
T = XfY (XHY)j = XfUr^rVrH (XHUrErVrH)j
= XfUr^rVrH (VrH)j (XHUrEry
= X^UrEr (XHUrEry.
Hence, tan0(7^(X)7^(y)) = [5"(T)0..0] which completes the proof.
Next, we provide an alternative way to construct T by adopting only the trian-
gular matrix from one QR factorization.
Corollary 2.2.11 Let X G Cnxp and Y G Cnxq with q < p be matrices of full
rank, and also let XHY he full rank. Let the triangular part R of the reduced QR
factorization of the matrix L = [XY] he
Rn R12
R = .
O R22
Then the positive singular values of the matrix T = R22(Rv2ysatisfy
tan0(^(X),^(y)) = [5+(T),
where there are dim(TZ(X) r\TZ(Y)) 0\s on the right.
Proof: Let the reduced QR factorization of L be
Rn R12
0 R22
where the columns of [Qx Q2] are orthonormal, and TZ(Qi) = X. We directly obtain
Y = Q\R\2 + Q2-R22* Multiplying by on both sides of the above equality for Y,
we get i?i2 = Q\Y. Similarly, multiplying by we have R22 = Combining
these equalities with Theorem 2.2.10 completes the proof.
20
[xy] = [q1q2]


2.2.3 tan in terms of the orthogonal projectors on subspaces
In this section, we present T using orthogonal projectors only. Before we proceed,
we need a lemma and a corollary.
Lemma 2.2.12 [73] If U and V are unitary matrices, then for any matrix A? we
have (UAVy = VHAHjH.
Corollary 2.2.13 Let A e Cnxn be a block matrix, such that
A
B 012
21 22
and 022
21
H 12 H 22
A
Proof: Let the SVD of B be where 2^ is a p x g diagonal matrix, and
U\ and V\ are unitary matrices. Then there exist unitary matrices U2 and V2, such
that
A
Thus,
ih s On v/1
u2 2l 22 _ v2h_
A
v^i s 12 t u? v^i st H 21 u?
21 22 U2H H 12 H 22 u2h
Since St = RSf, we have A
?2 ^2
as desired.

Theorem 2.2.14 Let PxPyi and Py be orthogonal projectors on the subspaces X
X1-and y, correspondingly. Then the positive singular values S+(T) of the matrix
T = Px^Py (PxPy)j satisfy
tan 0(^, 3^) = [oc, ^(T),
21


w/zere ere are addonaZ min (dim()dim( ny))ocs and addonaZ
dim( Pi on e
Proof: Let the matrices [XX_l] and [yy_j_] be unitary, where TZ(X) = X and
TZ(Y) = 3^. By direct calculation, we obtain
XH
PxPy[YY]=
Xf
XHY 0
O 0
The matrix PxPy can be written as
PxPy = [XX]
XHY 0 yh
0 0 Yf
By Lemma 2.2.12 and Corollary 2.2.13, we have
{pxpyy=[YY] {XhYY0 XH
0 0 _xf_
In a similar way, we get
Px^Py = [XX] XfY 0 yh
0 0 Yf
(2.2.2)
(2.2.3)
(2.2.4)
LetT = /VP3(/^P3)tandS = Xfr(XHr)tso
T=[XX] XfY 0 {xhyYo XH =[x x] BO XH
0 0 0 0 00 _xf_
The singular values are invariant under unitary multiplication. Hence, S^(T)=
S"+(_B). From Theorem 2.2.1, we have tan 0(Y) = 10...00,5"+(T)0...0].
Moreover, from Remark 2.2.3 we can obtain the exact mbers for 0 and oc.
Remark 2.2.15 From the proof of Theorem 2.2.1^ we can obtain the properties for
sines and cosines related to orthogonal projectors. According to representation (2.2.2)
22


( XHY 0 \
V O 0
we have S (P^Py) = S
equality (2.2.4) and Property 2.1.4 we have
S(P^Py) = S
[cos 0(^, 3^), 0,..., 0]. Similarly^ from
cos
XfYO
0 0
[l...lsin(ty)0..0]
(y),
where there are nmx(dim(y) dim(X)0) additional Is on the right.
Furthermorewe have
Xf (Px~Py)[YY]= -XfY O
XH 0 xhy
which implies that the singular values of Px Py are the union of the singular values
of XY and XhYisuch that S Py) = [SXiY)SXhYi)' .By Property
2.1.3, it follows that
S(,i)(Px Py) = [sin0(^/2)(<;f,^),sin0(^/2)(<;f,^)],
where 5,(i)(*) denotes the singular values in (0,1);see also [50, 60] for differ-
ent proofs.
We note that the null space of is the orthogonal sum H ).The
range of () is thus y flfl)"1which is the orthogonal complement of the
null space of PxPy- Moreover, (P^Py)1 is an oblique projector, because (P^Py)^ =
^(P^Py)^)2 which can be obtained by direct calculation using equality (2.2.3). The
oblique projector () projects on y along .Thus, we have ()=
(PxPy)^ Consequently, we can simplify T = P^Py (PxPy)^ as T = Pxi. (PxPy)^
in Theorem 2.2.14.
To gain geometrical insight into T, in Figure 2.2 we choose an arbitrary unit
vector z. We project z on y along then project on the subspace which is
23



Figure 2.2: Geometrical meaning of 71=3(;f_Py).
interpreted as Tz. The red segment in the graph is the image of T under all unit
vectors. It is straightforward to see that s(l.i=||T|| = tan(6,).
Using Property 2.1.4 and the fact that principal angles are symmetric with respect
to the subspaces A" and the expressions (3; and 3(;f3;) can
also be used in Theorem 2.2.14.
Remark 2.2.16 According to Remark 2.2,4 and Theorem 2.2.1^, we have
Furthermore^ by the geometrical properties of the oblique projector {PxPyY, it follows
that (PyPy)' = = (PyPy)'Px. Thereforewe obtain
3(3)=()=(-)()=-((/^))'
From equalities above, we see that there are several different expressions for 1 m
Theorem 2.2.14, which a/re summarized in Table 2.2.
)=)=(-)().
Similarlywe have


Table 2.2: Different expressions for T.
Px^Py{PxPy)] {PxPy)]
PyPX(PyPXV Py^PyP^Y
Px Py^ (Px^ Py^)t PAP^Py^Y
PyPxi(PyiPx^y Py(Py^P^Y
(PX-Py)(PXPyY (Py PX)(PyPXY
Theorem 2.2.17 Let ^y) < ir/2. Then,
1.If dim(X) < dim(3^)7 we have PxPy{PxPyV = Px(PxPyV = Px-
2. If dim(X) > dim(3^)7 we have {PxPy)^PxPy = {PxPyVPy = Py-
Proof: Let the columns of matrices X and Y form orthonormal bases for the
subspaces X and 3^, correspondingly. Since 0(^,3^) < tt/2, the matrix XHY has
full rank. Using the fact that XHY(XHYy = / for dim(X) < dim(3^) and com-
bining identities (2.2.2) and (2.2.3), we obtain the first statement. Identities (2.2.2)
and (2.2.3)together with the fact that = / for dim($ dim(V)
imply the second statement.
Remark 2.2.18 From Theorem 2.2.17, for the case dim(X) dim(y) we have
Px^PxPy)] = {PxPy)] ~ Px{PxPy)] = {PXPy)] ~ Px-
Since the angles are symmetric, using the second statement in Theorem 2.2.17 we
have (PXPyyPy = (PXPyy Py.
On the other handfor the case dim(X) dim[y) we obtain
{PXPy)]Py. = (PXPyy ~ Py and (P^x )tp^ = (P^Py^ Px^.
25


To sum upthe following formulas for T in Table 2.3 can also he used in Theo-
rem 2.2.14- alternative proof for T = (PxPyY Py is provided by Drmac in [20]
/or f/ze parcwZar case dim(A") = dim(Y).
Table 2.3: Different formulas for T with 0(Y) < tt/2: dim(< dim(3^) (left);
dim( dim(V) (right).
(PxPyV Px (Pp^PyV Py
(- (
Remark 2.2.19 Finally, we note that our choice of the space = Cn may appear
natural to the reader familiar with the matrix theory^ but in fact is somewhat mis-
leading. The principal angles (and the corresponding principal vectors) between the
subspaces X C H and y C H are exactly the same as those between the subspaces
X C X + y and y C X + yi.e.we can reduce the space H to the space X + y C H
without changing PABS.
This reduction changes the definition of the subspaces X1-and y1-and thus of
the matrices Xj_ and Yj_ that span the column spaces X1-and y-1. All our statements
that use the subspaces X1-and y1-or the matrices Xj_ and Yj_ therefore have their
new analogs, if the space X + y substitutes for %.
2.3 Conclusions
In this chapter, we have briefly reviewed the concept and some important prop-
erties of PABS. We have constructed explicit matrices such that their singular values
are equal to the tangents of PABS. Moreover, we form these matrices using several
different approaches, i.e., orthonormal bases for subspaces, non-orthonormal bases for
subspaces, and projectors. We use some of the statements in the following chapters.
26


3. New approaches to compute PABS2
Computing PABS is one of the most important problems in numerical linear
algebra. Let X G Cnxp and Y G Cnxq be full column rank matrices and let X = TZ(X)
and 3^ = TZ(Y). In this chapter, we are interested in the case n p and n q.
Traditional methods for computing principal angles and principal vectors are based
on computations of eigenpairs of the eigenvalue problems. The cosines of PABS are
eigenvalues of the generalized eigenvalue problem (see [9, 30])
O xhy u XHX 0 u
=A
YhX 0 V 0 yhy V
(3.0.1)
The generalized eigenvalue problem above is equivalent to the eigenvalue problems
for a pair of matrices
(XhX)-1/2XhY(YhY)-1YhX(XhX)~1/2u = X2u,
{YHY)-l/2YHX{XHX)-lXHY{YHY)~l/2v = \2v,
which can be found in most multivariate statistics books, e.g., see [39].
Let the columns of matrices X and Y form orthonormal bases for the subspaces
X and 3^, correspondingly. The matrix on the right side of (3.0.1) is identity, since
XHX I and YHY = I. We notice that solving the eigenvalue problem in (3.0.1) is
the same as solving the SVD of XHY. Computing PABS based on the SVD has been
established by Bjorck and Golub [9]. The cosines of principal angles and principal
vectors are obtained from the SVD of XHY^ which is also called the cosine-based
algorithm. Drmac [21] shows this algorithm to be mixed stable and recommends QR
factorizations with the complete pivoting for computing X and Y from original data.
The cosine-based algorithm in principle cannot provide accurate results for small
angles in computer arithmetic, i.e., angles smaller than 10-8 are computed as zero
2Kahan has attracted our attention to the half angles approach and given us his note [42], via
Parlett. We thank Kahan and Parlett for their help.
27


in double precision arithmetic when EPS ~ 10-16. This problem is pointed out and
treated in [9], where a sine-based algorithm for computing the small principal angles
is proposed. The sines of PABS are obtained from the SVD of (/ XXH)Y. For the
same reason, the sine-based algorithm has trouble computing large, i.e., dose to
tt/2, angles in the presence of round-ofT errors.
The sine and cosine based algorithms combined by Knyazev and Argentati [48],
where small and large angles are computed separately, are being used to cure this
problem. They use sine-based algorithm to compute the angles less than a threshold
(e.g., tt/4) and cosine-based algorithm to compute the angles greater than this thresh-
old. Cosine-based angles are obtained from the SVD of XHY and sine-based angles
are obtained from the SVD of (/ XXH)Y. However, such algorithms introduce a
new difficultyaccurate computation of the principal vectors corresponding to PABS
in the neighborhood of the threshold between the small and large angles.
To compute all angles with high accuracy, another approach is by computing the
sines of half-PABS mentioned in [9, 42]. In this chapter, we propose several new al-
gorithms to compute the sines of half-PABS. Our algorithms provide accurate results
for all PABS in one sweep and can be implemented simply and more efficiently, com-
pared to the sine-cosine algorithms. Moreover, we also propose methods to compute
all PABS for large sparse matrices.
Numerical solution of Hermitian generalized eigenvalue problems requires compu-
tation of PABS in general scalar products, specifically, in an A-based scalar product
for a Hermitian positive definite matrix A, which may only be available through a
function performing the product of A times a vector. Our implementation also allows
computation of PABS in general scalar products.
The remainder of this chapter is organized as follows. We first propose a method
to compute the sines and cosines of half-PABS by one SVD of [XF], which is proved
by using properties of Jordan-Wielandt matrix. In Section 3.2, we provide a method
28


to compute sines of half-PABS based on the geometrical insight view of half-PABS.
Both methods for computing PABS developed in Sections 3.1 and 3.2 are based on
the orthonormal bases of a pair of subspaces.
For better computational efficiency, we propose algorithms adopting only the R
factors of QR factorizations without formulating the orthonormal bases of input ma-
trices in Section 3.3. Furthermore, we provide algorithms to compute all principal
angles, not only for dense matrices, but also for sparse-type structure matrices. Gen-
eralizations of these algorithms to compute PABS in the A-based scalar product are
presented in Section 3.4. A MATLAB implementation and results of numerical tests
are included in Sections 3.5 and 3.6. Finally, we give a brief discussion for the new
algorithms and compare these algorithms with the sine-cosine based algorithm.
3.1 Algebraic approaches to compute sin (and cos (
Theorem 3.1.1 Let the columns of matrices X G Cnxp and Y G Cnxg7 where q < p
and p + q Let L = [XY] and let the SVD of L be IHVHwhere is a diagonal matrix with
diagonal elements in nonincreasing order. Then
/n
cos

and sin)=
where Sq^(L) denotes the vector of the largest q singular values of L in nonincreasing
order and S^(L) denotes the vector of the smallest q singular values of L in nonin-
creasing order. Moreover^ we have
Ur V2U2-Ur
V O V ^
where = / with Uu Ux G Cpxq; VHV = VHV = / with F, F G Cqxq;
and the matrix \U1JJ2} is unitary. Then, the principal vectors of 0^ associated with
this pair of subspaces are given by the columns of
= XUx and Fy = YV.
29


Alternativelythe principal vectors of 0^ associated with this pair of subspaces are
given by the columns of
and Vy = YV.
Proof: Let the SVD of the matrix XHY be UTjVh ^ where U and V are unitary and S
is a diagonal matrix with the real diagonal elements si(XhY)^ ..., sq(XHY) arranged
in nonincreasing order. Due to [4, 35, 82], the eigenvalues of the (p + q) x (p + q)
Hermitian matrix
0 xhy
yhx o
(3.1.1)
are Si(XHy), ...Sg(XHy) together with (p g) zeros. The matrix in (3.1.1)
is sometimes called the Jordan-Wieiandt matrix corresponding to XHY. Since
(XhY)V = UE and (YHX)U = VEH, it follows that
XHYvk = sk(XHY)uk, (3.1.2)
YHXuk = sk(XHY)vk, (3.1.3)
for fc =1.. .^q. Thus, and ~^k
vk vk
are eigenvectors of (3.1.1) corresponding
to the eigenvalues Sk and correspondingly. Further (see [35, p. 418]), we present
the eigenvalue decomposition of matrix in (3.1.1) as
0 xhy
=Q diag (Si,0,0,_i,...Si)Q
YhX O
where Q G C)x ) is a unitary matrix. Let the first q columns of U be denoted
as [/i, i.e., U\ = [^i,w2, ,^g]. Let U = JJ2] and V = bi2,...Let
U\ = Uq-i^ , ^i]and V = [vq^ ...^Vi]. The matrix Q can be written as
Q =
Ux V2U2 -Ui
V O V
30
(3.1.4)


Consequently, the eigenvalues of
0 xhy
yhx o
(3.1.5)
are [1 + si,1 + S2,...,1 + 1,...,1,11sg_i,...,1si] and the columns of
Q in (3.1.4) are the eigenvectors of (3.1.5).
Furthermore, we can rewrite (3.1.5) as follows:
= [XY]H[XY]- (3.1.6)
0 xhy
yhx o
Let the SVD of the matrix [xy] be UYjVh^ where S is a diagonal matrix and the
matrices U and V are unitary. Since cos2(6,/2)=(1 + cos(6,))/2 and sin2(6,/2)=
(1cos(6l))/2, we have
S = diag (a/^cos (0/2)11v^sin (0/2)).
Moreover, V = Q. By Theorem 2.1.2, the principal vectors of 0^ associated with
this pair of subspaces are given by the columns of = XU\ and Vy = YV. Also,
the principal vectors of 0^ associated with this pair of subspaces are given by the
columns of = XU\ and Vy = YV.
Remark 3.1.2 In Theorem we obtain PABS for the case p + q < n. In a
similar way, we can compute PABS for p + q>n. Suppose p + q>n, then
dim( Pi=dim(+ dim(V) dim( U
> dim(+ dim(V) n
=p + q n.
Thereforethere are at least p + q n zero angles between the subspaces X and y.
Furthermorewe obtain the sines and cosines of half-PABS by the singular values of
31


L as follows:
cos
and
=[5^_)0,0].
There are p + q n zeros after S^^L) in 1 /^[S^^L)) We also note that
the principal vectors are not unique if Sk(L) = s^+i(L).
Theoretically speaking, PABS can be obtained from the eigenvalues and eigen-
vectors of equality (3.1.5). However, this result does not appear to be of practical
interest in computing PABS. Calculating the sine or cosine squared gives inaccu-
rate results for small angles in exact arithmetic. Moreover, we should avoid using
XHY We can achieve a better understanding of this from the following example.
Let x =[1 and y =[1le1]^* They are both normalized in double-precision
and equality (3.1.5) results in a zero angle no matter what, since xHy =1.
Figure 3.1: Sines and cosines of half-angles.
To illustrate Theorem 3.1.1, we again take x =[l ]^ and y =[lle1]^ as
1 1
an example. We have L = . Computing the singular values of L, we get
0 1e-10
sin(6,/2) = 5e 11 and cos(6,/2) = 0. Obviously, we lose accuracy in cos(6,/2) in
this case. As shown in Figure 3.1, cos(6,/2) loses accuracy for small angles in double-
32


precision. Since we always have |sin/(6,/2)| > |cos/(6,/2)| for 0 G [0,tt/2], the better
choice is to use sin(6,/2) to calculate PABS.
Combining Theorem 3.1.1 and Remark 3.1.2, we formulate an algorithm to com-
pute all PABS. This algorithm provides accurate computation for small and large
angles. The Algorithm 3.1 is based on computing sin(6,/2) which is obtained from the
smallest min(p, q) singular values of L. Moreover, the principal vectors are obtained
from the right singular vectors of L corresponding to the smallest min(p, q) singular
values for the case p + q case p + q>n.
Table 3.1: Algorithm 3.1.
Algorithm 3.1
Input: matrices X\ and Y\ with the same number of rows.
1. Compute orthonormal bases X e CnXf} and e Cnxg of 7^(Xi) and 7^(Yi)correspondingly.
2. Set L = [Xy], compute the SVD of L: =svd(L, 0).
3. If p -\- q < n, take the smallest m = min (p, q) singular values of L denoted as si,...,sm.
If p -\- q > n, take the smallest n max(p, q) singular values of L and add p -\- q n zeros
singular values and denote as si,...,sm.
4. Compute the principal angles in nonincreasing order for A:=1,...,m:
k = 2 arcsin (sk/V2).
5. Compute corresponding principal coefficients:
Ui = ~^V{1 :p,p-\-q m + 1:p -\- q) and Vi=\/2V (p -\- l :p-\-q:p-\-q m-\-l:p -\- q).
6. Compute the principal vectors:
Vx = XUx and ^ = ^1.
Output: the principal angles ...:m between the column-space of matrices X\ and Yi,the
corresponding principal vectors Vx and and the principal coefficients 171 and Vi
Remark 3.1.3 In the first step of Algorithm 3.17 we use the MATLAB function
ORTHUPDATE.rn (see Appendix A) to compute the orthonormal bases of the input
matrices. The function ORTHUPDATE.rn is backward compatible with the built-in
33


MATLAB function ORTH.m. Our function ORTHUPDATE.m provides three choices
to compute an orthonormal basis with option opts.fast being 0,1 and 2. The or-
thogonalization is performed using the SVD with the option opts.fast being 0, which
performs the same as ORTH.m. With opts.fast =17 the function ORTHUPDATE.m
performs QR factorization with pivoting. The function ORTHUPDATE.m only works
for a full-column rank matrix if opts.fast = 2. The algorithm is developed as follows:
suppose Xx is an input matrix, then we compute an upper triangular matrix R from
a Cholesky factorizationsuch that R = cho\{XfXi)then we take the orthonormal
basis as XiR~l.
Let us consider the cost of these three algorithms as mentioned above to compute
the orthonormal basis of the input matrix. In generalthere are two steps to com-
pute the SVD, e.g.7 see [8,12,19]. The first step can he done using Householder
reflections to reduce Xi to a bidiagonal matrix. The second step reduces the bidi-
agonal matrix form to diagonal form iteratively by a variant of the QR algorithm.
According to [8, p.90]7 [12]7 and [19, Chap. 11]7 the total cost of the complete SVD is
about (3 + a)np2 + ll/3p3 flops7 where a = 4 if standard givens transformations are
used in QR algorithm^ a = 2 if fast given transformations are used. For the detailed
transformations7 the reader is referred to [8, Section 2.6].
Generally^ the cost of QR factorization based on Householder reflections (see [77])
is about 2np2 2/3ps flops for an n hyp matrix with n>p. The Cholesky factorization
for the matrix X(IX1 is about l/3ps flops. Among these three choices for computing
the orthonormal basis, the opts.fast being 2 runs fastestbut is the least reliable. The
most robust and slowest one is based on the SVD-based algorithmsince the cost
of SVD is about two times the cost of QR. For detailed information concerning the
stability of these three factorizations7 see [77].
In our actual code, we use the economy-size factorization QR and SVD in the
function ORTHUPDATE.m. We do not know the exact number of flops in MATLAB.
34


In [57] 7 it states that floating-point operations are no longer the dominant factor in
execution speed with modern computer architectures. Memory references and cache
usage are the most important.
Remark 3.1.4 Golub and Zha [30] point out that PABS are sensitive to perturbation
for ill-conditioned Xx and Y A proper column scaling of Xx and Yi can improve
condition numbers of both matrices and reduce the error for PABS, since column
scaling does not affect PABS. Numerical tests by Knyazev and Argentati [48] show that
proper column scaling improves accuracy when the orthonormal bases are computed by
using the SVD. However, there is no need to scale the columns when the orthonormal
bases are obtained by the QR factorization with complete pivoting^ e.g.7 see [21,48].
Also, it is pointed out in [21] that one good way to compute the orthogonal bases
is to use the unit lower trapezoidal matrices Lx and Ly from the LU factorizations
of Xi and Y\ with pivoting^ since TZ(Xi) = TZ{Lx) ctnd Tl(Yi) = TZ(Ly).
Remark 3.1.5 We note that the matrix size of L in Algorithm 3.1 is n by p + q. If
n is large, in order to reduce the cost of the SVD of L, we can reduce the matrix L
to a (p + q) by (p + q) triangular matrix R by QR factorizationsuch that L = QR.
We can obtain PABS by the SVD of R.
In the following, we will derive a different formula to compute the sines and
cosines of half-PABS.
Lemma 3.1.6 Let the maznces X and Y he described as in Theorem 3.1.1. Lei the
columns of matrix Qx^y form an orthonormal basis for the column-space of L, where
L={XY}. LetL = Q+yL. Then
where extra Os may need to he added on the left side to match the size. In particular,
35


if we take Qx-\-y = [X X]7 then
, I XhY
L = .
OXfY
Proof: By direct computation, we have
LHL = LHQx+yQHx+yL
=LH Px+yL
= lhl,
where Px-\-y is an orthogonal projector on the column-space of L. So, the positive
singular values of L coincide with the positive singular values of L. If we take Qx-\-y =
[X X_l] then
^ I XhY
l=[xx]h[xy}=
OXfY
as expected.
Remark 3.1.7 Lei the matrix Xi G Cnxp have full column rank. Let Yi G Cnxq and
let [Q1Q2] be an orthonormal matrix obtained from the reduced QR factorization of
We take X = Qi7 thus we have
, I XhY
L = .
OQ^Y
The size of L is small, since max^size(L)) is p + q. However, this L is not attractive
to be used to compute PABS in practice. In some situations, we have no information
about the rank of the input matrix XiSuppose we use the QR factorization to obtain
the full-column rank matrix \ from the matrix Xlt In this method, to get PABS we
have to do the QR factorizations for X!Y![ iYi] and plus one SVD of .The
computational cost is similar to the method described in Remark 3.1.5. Moreover,
applying the orthonormal matrix [Q\ Q2}, which is obtained from the QR factorization
36


[ may increase the inaccuracy for small angles. Especially ij [X\ Y\] is ill-
conditioned, we may obtain [Q\ Q2] inaccurately.
Let us highlight that updating the SVD of L by appending a row or column and
deleting a row or column is well investigated, e.g., see [8, Section 3.4]. In the last
lemma of this section, we discuss briefly the relationship between the sines and the
cosines of half of the principal angles between TZ{Xi) and ^(Vi) and those of TZ{Xi)
and w]), where Xi G Cnxp and Yi G Cnxq. For convenience, let p + q < n and
q < p.
Lemma 3.1.8 Let the principal angles between TZ(Xi) and TZ(Yi) with Xi G Cnxp
and Yi G Cnxg7 where p + q < n and q < p, fee 6*i,...,6*g arranged in nondecreasing
order. Let the principal angles between TZ(Xi) and TZdYiv]) be §1arranged
in nondecreasing order. Then
( n Vhl
M 2
COS I > cos > cos
>
> cos
> cos
>
71
71
>sm >S1
sm
~ > > sin { $ 1 > sin (4) > sin ,1
Proof: Let the columns of matrices X and Y form orthonormal bases for the sub-
spaces TZ{Xi) and correspondingly. Let the columns of matrix [1 u] form the
orthonormal basis for ([1^ w]). Moreover, let L = [XY] and Z By Theo-
rem 3.1.1we have S^(L) = v^[cs(-(9i)...cos(|11sin(-6^)...sinQi^)]
and (2) = W cos ,cos ,1,1,sin )...sin ().
Applying the interlace property of the singular values [28, p. 449], the results are
obtained.
3.2 Geometric approaches to compute sin
Let x and y be unit vectors. The HausdorfT metric between vectors x and y is
defined as
2sin -6(x,y) = M{\\xq-y\\ :qeC,\q\ =1}.
(3.2.1)
37


In particular, if x and y in Rn are such that xHy > 0 (see Figure 3.2), then
2sin (^e(x,y)^j = \\x y\\.
Similarly, the HausdorfT metric between two fc-dimensional subspaces X and 3^, e.g.,
Figure 3.2: Sine of half-angle between x and y in Wn.
see [66], is defined as
$ (2sin (())...2Sin ()))=inf{|||X0-y||| }
(3.2.2)
where U{k) is the family of unitary matrices of size k and the columns of matrices
X and Y form orthonormal bases for the subspaces X and 3^, correspondingly. The
function $() is a symmetric gauge function [5] and ||| ||| denotes the corresponding
unitarily invariant norm [55]. See [66] for further information concerning the sines
of half-PABS related to the HausdorfT metric between two subspaces. In particu-
lar, taking the same minimization problem for Frobenius norm in (3.2.2), we have
(see [73])
2 sin ( ^0(^,3^)
min \\XQ-Y\\f
Qeu(k)
F
38


This is a special case of the orthogonal Procrustes problem. The solution Q of the
Procrustes problem can be obtained by using the orthogonal polar factor of XHY [28]:
let the SVD of XHY be UT.VH ^ then the optimal Q is given by Q = UVH. We use
this construction of Q to demonstrate that S(XQ Y) = 2 sin ) 3^)/2), which is
mentioned in Kahan s lecture note [42].
Theorem 3.2.1 [42] Let the columns of matrices X G Cnxp and Y G Cnxq (q < p)
form orthonormal bases for the subspaces X and y, correspondingly. Let L = XQ
Y and B = XHY, where QHQ = / and Q is the unitary factor from the polar
decomposition of B. Then
sin
(3.2.3)
Proof: Let B = U [C 0]H VH be the SVD of 5, where U and V are unitary
matrices. The matrix ¢7 is a diagonal matrix with cos (6k) on its diagonal, where
9k for k =1,...are the principal angles between the subspaces X and 3^. Let
Q = U[10}H VH. We have
LhL = (XQ-Y)h(XQ-Y)
= 21- QhXhY YhXQ
=21 2VCVh
= 2V(I-C)Vh,
where / ¢7 is a diagonal matrix with 1cos (6k) for fc =1,...,g on its diagonal.
Note that
qHxhy = v[i]UHU
0
VH = VCVH.
Therefore, the singular values of L are y 2(1cos k) for k =
sin2 (9/2)=(1cos6l)/2, we obtain sin (0^/2) = S^(L)/2.
1...g. Since

39


Remark 3.2.2 We can rewrite L as XU[j ]H YV. Moreover, if we use the
reduced SVD of such that B = UrCVrH with Ur G Cpxq and Vr G Cgxg, then L can
he written as
XUrVrH-Y or XUr-YVr.
The formulation XUr YVr to compute the sines of half-PABS is also mentioned in
[9]7 but no detailed information provided.
Remark 3.2.3 From the proof of Theorem 3.2.17 we also can get
cos (0/2) = S(XQ + Y)/2.
Moreoverif we use the reduced SVD of Bsuch that B = UrCVrHwe have
cos (0/2) = S(XUr + YVr)/2-
Theoretically speaking, the principal vectors associated with the pair of the sub-
spaces X and 3^ are the columns of the matrices XUr and YVr) correspondingly, where
Ur and Vr are obtained from the reduced SVD of B. However, the cosine-based al-
gorithm fails to accurately compute the small principal angles and the corresponding
principal vectors in double precision, which is pointed out in [48]. For example, we
have small angles 10-9,10-10,10-11, and 10-12, which will produce multiple singular
values 1 with multiplicity 4 in double precision based on the cosine-based algorithm.
In fact, there are four small distinct angles with exact arithmetic. This implies that
the principal vectors corresponding to small principal angles may not be computed
accurately in double precision using the SVD of B.
To obtain the principal vectors corresponding to small angles in high accuracy,
one way is by using the SVD of L in Theorem 3.2.1. Let SVD of L be UTjVh. The
principal vectors on the subspace 3^ are the columns of Vy^ where Vy = YV and the
principal vectors Vx on the subspace X can be obtained by Vx = XQV.
40


The formulation for principal angles on the subspaces X may not be obvious. In
our actual code, we reformulate L in Theorem 3.2.1 as XUr YVr in Remark 3.2.2, an
alternative approach to get the principal vectors associated with all angles is by the
SVD of XUr YVr. Let the SVD of XUr YVr be Ui^iV^1 ^ then the principal vectors
associated with the pair of two subspaces are the columns of XUrVi and YVrV\. Our
numerical tests support this result.
We formulate Algorithm 3.2 to computing all principal angles and principal vec-
tors by using L = XUr YVr as in remark 3.2.2.
Table 3.2: Algorithm 3.2.
Algorithm 3.2
Input: matrices X\ and Y\ with the same number of rows.
1. Compute orthonormal bases X and Y of column spaces of X\ and Yi,correspondingly.
2. Compute the SVD of B ^ X^Y: | ^ ^ ^ ^ )(
3. Let L = XUr YVr and compute the singular values of L for A:=1,...,min (p, q).
4. Compute the principal angles: k = 2arcsin (s^/2).
5. Compute the right singular vectors Vi of L, then the principal coefficients are
Ui = UrVi and V1 = VrV1.
6. Compute the principal vectors:
= XUl and Vy = YVl
Output: the principal angles :min(p,q) between the column-space of X\ and Y\ matrices,
the corresponding principal vectors Vx and and the principal coefficients Ui and Vi.
41


3.3 Computing sin (and cos (without using orthonormal bases
Our algorithms described above and the cosine-sine based algorithm compute
PABS accurately by using the orthonormal bases for subspaces. If the input matri-
ces are large sparse, then computing an orthogonal basis for each of the subspaces
may destroy sparsity of the given spaces. The cosine-sine based algorithm coded by
Knyazev and Argentati is not capable of handling the sparse-type structure matrices
available in MATLAB. In this section, we propose algorithms to compute the sines
of half-PABS by only using the upper triangular matrices R from QR factorizations
without forming the orthonormal bases or input matrices Xi and Y\. Moreover, we
provide an algorithm to compute PABS not only for dense matrices but also for sparse
matrices.
Theorem 3.3.1 Let Xi G Cnxp andYi G Cnxq he matrices of full rank where p + q <
n and q < p. The principal angles between TZ{Xi) and TZ(Yi) can be computed as
follows.
1.Compute the triangular part R of the reduced QR factorization of the matrix
L\ = such that
where Rn G Cpxp7 R12 G Cpxq7 and R22 G Cqxq.
2. Compute the triangular part Ry of the reduced QR factorization of the matrix Y
Rll Rl2
3. Let L
.We have
R22Ryl
cos
42


Proof: Let the reduced QR factorization of L\ be
[x1y1] = [g1Q2]
Ru Rl2
0 R22
(3.3.1)
where the columns of the matrix [QiQ2] G Cx(p+9) are orthonormal. Thus, the
matrix Xi can be written as Xi = QiRn. Let the reduced QR factorization of Yx be
QyRy. Therefore,
[-^1 ^i]=[Qi Qy]
Combining (3.3.1) and (3.3.2), we obtain
[Qi Qy] = [Qi Q2]
Rn
Ry
(3.3.2)
Rn Rl2
O R22
Rh1
Ry1
(3.3.3)
[Qi Q2]
(3.3.4)
/ RuRy1
R22Ryl
Since the singular values of a matrix are invariant under orthogonal transformations,
we have
I R^Ry1
S
[Qi Qy^j
s
O R22Ryl
According to Theorem 3.1.1, we obtain the result.

Remark 3.3.2 Let the SVD of L in Theorem 3.3.1 be UTjVh and let
u = Vw{\ :p^q + 1:p + q) and Vi = V2V(p +1:p + q^q + 1:p + q).
Then the principal vectors of 0^ associated with this pair of subspaces are given by
the columns ofVx = XiR^U andVy = Y\RylVi^ correspondingly.
Remark 3.3.3 Note that the matrix size of L in Theorem 3.3.1 is (p + q) by ip + q)-
The cost of the SVD of L in Theorem 3.3.1 should he less than the cost of the SVD
43


of L in Theorem 3.1.1. The method discussed m Theorem 3.3.1 may he an efficient
alternative to compute PABS.
Remark 3.3.4 The R factor is easy to update whenever a column or row is appended
to the matrix L\, see [8, Section 3.2].
Theorem 3.3.5 Under the assumptions of Theorem 3.3.1, we have
cos(e^) = S^RnRy1)
sin(0'1-) = Si(R22Ry1)-
Proof: From (3.3.1) and (3.3.2), we obtain
= Q1R12 + Q2R22 QyRy-
By multiplying Ry1 on both sides of above equality, we get
QiR^Ry1 + Q2-R22-R31=Qy-
By multiplying on both sides of (3.3.7), we have
^12^' = QiQy-
Note that Q1Q2 = * From Theorem 2.1.2 of PABS, we obtain equality (3.3.5).
Similarly, by multiplying on both sides of (3.3.7), we have
R22Ryl = Q^Qy-
Since TZ([XiYi\) C TI([QiQ2])j from Property 2.1.3 we get equality (3.3.6).
(3.3.5)
(3.3.6)
(3.3.7)
Remark 3.3.6 In Theorems 3.3.1 and 3.3.5, the matrix Ry is obtained from the QR
factorization of Ylt In fact, we could get Ry from the QR factorization of
R
22
Let the reduced QR factorization
R12
R22
be QyRy. Then,
= [Qi Q2]Qy^y-
44


Moreover, let B = [QiQ2\Qy- We have BHB = I. HenceAy is the triangular
matrix of the QR factorization of Ylt An alternative method to compute the cosines
of PABS using LQ factorization can he found in [17].
Remark 3.3.7 In the Householder QR factorization, we apply a succession of ele-
mentary unitary matrices Qi on the left of the initial matrix = Li=so
that the upper-triangular matrix R is obtained. Let Qp denote the pth unitary matrix.
The matrix = L\ after p steps is
QpQpi QiL\
0 L?
where [Rn R12] contains the first p rows of the final upper-triangular matrix R. Let
L =
I RuRy1
0 L^Ry1
where Ry is the upper-triangular matrix obtained from the QR
Rn
factorization ojY\ or
Then we have
.cos ()= and ()=(
2. cos(0^) = S^(RuRy1) and sin(0^) =
To summarize, we formulate Algorithm 3.3 to compute all PABS with dense or
sparse matrices. This algorithm provides accurate computation for small and large
angles.
45


Table 3.3: Algorithm 3.3.
R-
4. Compute L
Algorithm 3.3
Input: matrices X\ G Cnxp and Y\ G Cnxq with full column rank .
1. Compute the triangular part R of the QR factorization of L\ = [X\ Yi], e.g.,
qr(Li,0) if Xi and Fi are sparse,
triu(qr(Li,0)) otherwise.
2. Compute R\2 = R(1:p:p -\-1:p -\- q) and R22 = R(p + 1end:p +1:p -\- q).
3. Compute the triangular part Ry of the QR factorization of
_Rti
Speye(rt Ru/Ry lfx, and y, are sparse,
sparse(a p:p) 11221
eye(p) 12/%
zeros(a-p,p) R^^/Ry
where a = min(size(Li)).
5. If p -\- q < n, compute the smallest m = min(p, q) singular values si,...,sm of L and
corresponding m right singular vectors V. The principal angles are computed as
=2 arcsin (Sfc/V^) for =1...m.
6. If p -\- q > n, compute the smallest n max(p, q) singular values of LH and corresponding
n max(p, q) right-singular vectors V\. Moreover, there are p -\- q n zeros singular values
and the p -\- q n right-singular vectors V2 corresponding to the largest p -\- q n singular
values are computed. Let V =[V\ V^]* principal angles are computed as
otherwise,

2 arcsin (sk/V2) for =1... n max(p, q).
0
for =n 7 1...m.
7. Compute the principal vectors:
lx = V2X1/R(1 :p,l:p)V(l p, ) and V3 = ^Y1/RyV(p-\-1+ ).
Output: the principal angles i:...:m and the corresponding principal vectors Vx and Vy.
Remark 3.3.8 In the first step of Algorithm 3.3, we apply the MATLAB function
qr to compute the upper triangular matrix for both the dense-type and sparse-type
matrix Lx. SuiteSparseQR developed by Davis [16] in MATLAB R2009has re-
placed the prior sparse QR factorization in MATLAB provided by Gilbert [26]7 where
46


SuiteSparseQR is a multithreaded rriultifrontal sparse QR factorization method.
In step 3, for dense matrices Xi andYi it is cheaper to compute the triangular part
R12
Ry from the QR factorization of
R
22
than the QR factorization of Y\. However,
factorizing a sparse matrix implies fill-in in the factors. Thus,
R
12
R
22
may not he as
sparse as Y\. So, in step 3, we can compute Ry from the sparse matrix Y\. A good
column ordering can help limit the worst-case fill-in. A suitable column ordering of
A is usually obtained by the minimal degree analysis on AHA. For a more detailed
account of finding a good column orderingthe reader is referred to [33].
Remark 3.3.9 In step 57 if we compute a few principal angles for the large sparse
matrix L, it is better to use the function svds(L) than svd(full(L)). However, to com-
pute all singular values svd(-) will usually perform better than svds(-) in MATLAB.
See SVDS MATLAB function reference for detailed information. In our actual code,
we use svd(full(L)) instead of svds(L) to compute all PABS of sparse matrices.
Let us finally present new simple proofs for some properties of orthogonal pro-
jectors on the subspaces X and 3^.
Corollary 3.3.10 [23] Lei Px and Py he orthogonal projectors on the subspaces X
and ycorrespondingly. We have
+ A)=1 + CS (eJ,7T/2)(t^))1-CS (eJ,7T/2)(t^))
where 5,()i)U(i)2) denotes the vector of singular values in (0,1)U (1,2).
Proof: By direct calculation, we have
XH
Px + Py = [XY]
Yh
47


Let the SVD of [xy] be UEVH. Then
Px + Py = UmHfjH.
Applying Theorem 3.1.1, the result is obtained directly.
In the following corollary, we will present the sines and cosines of half-PABS
which are relative to these orthogonal projectors.
Corollary 3.3.11 Let Px and Py he orthogonal projectors on the subspaces X and
y, correspondingly. Then,
,w)(= W (eW2)(u))(>2)(^)).
Proof: We have
PX + Py=[PXPy][PXPy]H.
Thus, the singular values of [p^ Py] are the square root of the eigenvalues of Px + Py-
Since the eigenvalues of Px + Py are the same as the singular values of Px + Py^ using
Corollary 3.3.10 completes the proof.
Remark 3.3.12 Note that we can write [px py] = [xy]
XH 0
0 Yh
.From this
representation^ it is clear that the principal vectors can be obtained directly from the
right singular vectors of [p^ Py].
48


3.4 Computing sin (and cos (in the A-based scalar product
Numerical solution of Hermitian generalized eigenvalue problems requires compu-
tation of PABS in general scalar products, specifically, in an A-based scalar product
for a Hermitian positive definite (HPD) matrix A. In this section, we will develop
methods to compute PABS m the A-based scalar product based on sines of half-PABS
in the A-based scalar product. To proceed, we need the definitions of the A-based
scalar product [69] and PABS m the A-based scalar product.
For an HPD matrix A, let
(*, -)A = (A-, -k denote the scalar product induced by A.
|| |U = y (A*,*)2 = \\A1/2 ||2 denote the corresponding vector norm induced
by A
|| m = || A1/2 A-1/21|2 denote the corresponding matrix norm induced by A.
The following denmtion of PABS m the A-based scalar product, consistent with the
definition of PAB^ m Section 2.1, is proved by Knyazev and Argentati in [48].
Definition 3.4.1 Let C Cn 6e 6aces p = dim ()and g = dim (Y).
Let m = min (p, q). The principal angles
QA{X,y) = [Al)- ,^m], where 0Ak G [,tt/2], fc =1,...,m,
between X and y in the A-based scalar product are recursively defined by
sk = cos(0Ak) = max max | (x,y)A \ = \ (xk,yk)A \ = \x^Ayk\,
subject to
\\^\\a = WvWa =1,xHAxi = 0, yHAyi = 0, z =1,...,fc -1.
The vectors {xm} and {yiym} are called principal vectors relative to the
A-based scalar product.
49


Theorem 3.4.2 [48] Let the columns of matrices X G Cnxp and Y G Cnxq form
A-orthonormal bases for the subspaces X and y, correspondingly. Let the SVD of
the matrix XHAY he UTjVh, where U and V are unitary matrices and Tj is a p
by q diagonal matrix with the real diagonal elements si(XHAY)..sm(XHAY) in
nonincreasing order with m = min(p,g). Then
cos Q\(X,y) = Sl(XHAY) = [Sl(XHAY),..., sm(XHAY)],
where 0^ is the vector of principal angles in the A-based scalar product between X
and y arranged in nondecreasing order. Moreover, the principal vectors relative to
the A-based scalar product associated with this pair of subspaces are given by the first
m columns ofVx = XU and Vy = YV, correspondingly.
Now we derive theorems about sines and cosines of half-PABS in the A-based
scalar product. The following theorem is the extension of Theorem 3.1.1.
Theorem 3.4.3 In the notation of Theorem 3.4-2, let q < p and p + q < n. Let
L = Al/2[XY] and let the SVD of L be UTyH^ where ^ is a diagonal matrix with
diagonal elements in nonincreasing order. Then
cos
(3.4.1)
Moreover% we have
lh V2U2 -th
V O V '
where U^UX = U^1U1=I with U^lh e Cpxq; VHV = VHV = I with V,V e Cqxq;
the matrix [jj1 JJ2] is unitary. Then, the principal vectors in the A-based scalar product
of associated with this pair of subspaces X and y are given by the columns of
= XUx and VS = in/.
50


Alternatively, the principal vectors in the A-based scalar product f\ associated with
the pair of subspaces are given by the columns of
= XUx and
Proof: By the assumption that the columns of matrices X G Cnxp and Y G Cnxq
are A-orthonormal bases for the subspaces X and 3^, correspondingly, it follows that
XHAX = / and YHAY = L Let SVD of the matrix XHAY be UY,VH, where
S = diag(Si, ...) with si >...>% and s=cos 6^ (Y) for fc =1...g.
Using a similar process as in the proof of Theorem 3.1.1, we have the eigenvalues of
u 0 xhay
[XY]hA[XY] = I +
yhax o
(3.4.2)
are [1 + Si,...,1 + 1,...,1,1,1Si] and the corresponding eigenvectors
are the columns of
Q =
IA V2U2 -Ui
V O V
(3.4.3)
where U\ is the first q columns of [/, such that U\ = [^i,* * ^uq]. Moreover, U =
[U\ U2]. Let V = [^i,^2? * then V = [vq)vq_i)... ,^i] and U\ = [uq)uq_i)... ,^i].
Since the eigenvalues of (3.4.2) are equal to the square of the singular values of
L = A1^2[XY]J we obtain (3.4.1). Moreover, let the SVD of L be U'EVH. It is easy
to obtain V = Q. Based on the definition of PABS in the A-based scalar product via
SVD, we have that the principal vectors of 0^ in the A-based scalar product are the
columns of XUi and YV. Similarly, the principal vectors of 0^ in the A-based scalar
product are the columns { XU\ and YV.
Higham and Al-Mohy [34] give an algorithm to compute the square root of A.
It costs about 15|n3 flops for an n by n Hermitian positive definite matrix A. For
computational reasons, when the size of A is large, we need to avoid dealing with the
51


square root of A explicitly. From the proof of Theorem 3.4.3, let L\ = [Xy]^A[Xy]
and p + q < n then
cs2 ()=an(i 2 ()=3(li). (3.4.4)
However, using the SVD of the matrix [XY]HA[XY] may not produce small angles
accurately. Because we need to take the square root of the singular values of Li to
obtain the sines or cosines of half-PABS. In other words, the PABS smaller than 10-8
will be zero in double precision.
One way to cure the problem is to replace L in Theorem 3.4.3 by L = R[XY]
or L = K[XY] where A = RHR (Cholesky factorization) or A is known as A =
KHK. For this case, we need to know A exactly or give A as KHK. However,
sometimes A may not be available as a matrix but only as some function performing
the multiplication of A by a given vector. We can update L by using the following
lemma [48, Lemma 6.1].
Lemma 3.4.4 The SVD of the matrix AX^2L coincides with the SVD of the matrix
Q^AL, where Ql is a matrix with A-orthonormal columns, which span the same
column-space as columns of matrix L.
In Theorem 3.2.1, we present a formula of the sines of half-PABS. The following
theorem is an extension of Theorem 3.2.1.
Theorem 3.4.5 In the notation of Theorem 3.4-2, let L = Al/2[XUi YV], where
U\ is the first q columns of U. Let q Sin t)=(). 3.4.5)
Proof: Prom Theorem 3.2.1, we have XHAY UTjVh. Since q < we have
XHAY = U\CVH^ where ¢7 is a diagonal matrix with cos (Ak) n its diagonal, and
Ak for fc =1,...,g are the principal angles between X and 3^ in the A-based scalar
52


product. It is worth noting that U\CVH is the thin SVD of XHAY. Then
LhL = {XUx YV)HA{XUl YV)
=U^XhAXUx U^XhAYV vhyhaxu1 + vhyhayv
=21 -2C,
where / ¢7 is a diagonal matrix with 1cos (Ak) for fc =1,...,g on its diagonal.
Since the singular values of L are the square root of the eigenvalues of LHthe proof
is completed.
By analogy with Algorithms 3.1 and 3.2, we can obtain Algorithms 3.4 and 3.5
for computing the principal angles in the A-based scalar product according to The-
orems 3.4.3 and 3.4.5. In Algorithms 3.4 and 3.5, we use the function ORTHUP-
DATE.m (see Appendix A) with option opts.A = A to compute an A-orthogonal
basis, which is compatible with the function ORTH A.m presented in [48].
Table 3.4: Algorithm 3.4.
Algorithm 3.4
Input: matrices X\ and Y\ with the same number of rows.
1. Compute A-orthonormal bases X G Cnxp of 7^(Xi) and Y e Cnxq of 1Z{Yi).
2. Compute L\ = [XY] and Qlx = orthupdate(Li,opts.A).
3. Compute the SVD of L = Q^iAL\, such that = svd(L, 0).
4. Take the smallest m {p ~\~ q j) singular values of L with j = size(QiJ1,2) and add
P Q ~ j zeros denoted as si,...,sm with m = min(p, q).
5. Compute the principal angles in the A-based scalar product:
k = 2 arcsin (s^/a/2) for A:=1,...,m.
{Ui = \/2V(1:p:p -\- q m -\-1:p -\- q)

Vx = ^/2V (p~\~l :p-\-q,p-\-q-m-\-l:p-\-q)
7. Compute the principal vectors in the A-based scalar product: Vx = XUi and
Output: the principal angles ...:m between column-space of matrices X\ and Y\ in the
A-based scalar product, and the corresponding principal vectors Vx and
53


Table 3.5: Algorithm 3.5.
Algorithm 3.5
Input: matrices X\ and Y\ with the same number of rows.
1. Compute A-orthonormal bases X and Y of IZ(Xi) and 7^(Yi), correspondingly.
2. Compute the SVD of B =| ^ =Svd(^ rank(X) > rank(y),
3. Compute L\ = XUr YVr and Qlx = orthupdate(Li,opts.A).
4. Compute SVD of L = Q^iAL\.
5. Take the singular values of L for A:=1,...,m, where m = min(size(L)). Then
the principal angles in the A-based scalar product are
{2 arcsin (s^/2) for /c =1,...,m,
0 for /c = m + 1,...,min(p, q).
6. Take the right singular vectors Vi of then the principal vectors in the A-based
scalar product associated with the pair of subspaces are the columns of
Vx = XU^ and =IT%.
Output: the principal angles :min(p,q) between column-space of matrices X\ and Y\
in the A-based scalar product, and the corresponding principal vectors Vx and Vy.
Remark 3.4.6 The first and second steps of Algorithm 3.5 can be replaced by
the Cholesky factorization. Let the matrices Xi and Y\ have full column rank.
Let the Cholesky factorization of X^AXi and Y^AYi he R^Rx RyRy^ cor-
respondingly, where Rx and Ry are upper triangular matrices. Thenwe have
B = (R^)~1X^1 AYi(Ry)~1 ^ which implies that cos^^,^) = S^(B).
If the input matrices are ill-conditioned, the A-orthonormal bases may be com-
puted inaccurately. Next, we develop an algorithm to compute PAB^ m the A-based
scalar product without forming A-orthonormal bases. To proceed, we use two lem-
mas. The following lemma links the PAB^ m the A-based scalar product with the
PAB^ m the standard scalar product, e.g., see [48].
Lemma 3.4.7 [48] Let A = KHK for some matrix K. The principal angles between
54


subspaces X and y relative to the scalar product coincide with the principal
angles between subspaces KX and Ky relative to the standard scalar product (*, ).
The following lemma links A-orthogonal QR factorization with QR factorization
in the standard product.
Lemma 3.4.8 Let A-orthogonal QR factorization of S he QaR, where Q^AQa = I
and R is an upper triangular matrix. Then the QR factorization of A1/2S is QR vnth
respect to the standard inner product with Q = A1/2Qa-
Proof: The proof is straightforward, since Q^AQa = (A1/2Qa)h(A1/2Qa) = /.
Remark 3.4.9 Alternative ways to compute PABS in the A-based scalar product are
by using the SVD of the triangular part R obtained from A-orthogonal QR factoriza-
tions of [XY] and [XU! YV] in Theorems 3.4.3 and 3.4.5.
Theorem 3.4.10 Let Xi G Cnxp and Yi G Cnxq he matrices of full rank where
p + q < n and q A-based scalar product can be computed as follows.
1. Compute the triangular part R of the reduced A-orthogonal QR factorization of
the matrix L\ = [Xi Yi\, such that
Rn R12
R =
0 i?22
where Rn G Cpxp, R12 G Cpxq, and R22 e Cqxq.
2. Compute the triangular part Ry of the reduced A-orthogonal QR factorization
of the matrix Y
55


3. Let L
I RnRy
Then
R22Ry1
cos(0^) = S^(Ri2Ry1) and sin(0^) = S^(R22Ry1)-
Proof: Using Lemma 3.4.7, we take K = A1/2. Computing the principal angles
between the subspaces X and 3^ in the A-based scalar product is equivalent to com-
puting the principal angles between the subspaces Al/2X and Al/2y in the standard
scalar product.
Therefore, to compute PABS in the A-based scalar product, we can replace L\
with L\ = Al^2[Xi Yi\ in the first statement of Theorem 3.3.1 and replace Y\ with
Al/2Y\ in the second statement of Theorem 3.3.1. In other words, we can obtain trian-
gular matrices R and Ry by the reduced QR factorizations of the matrix A1^2[x1 Y\]
and A1/2!7!, correspondingly. Using Lemma 3.4.8, we obtain the results as stated.
Remark 3.4.11 In particular, if [Xi Y\] has full column rank, we can obtain R and
Ry by the Cholesky factorizations of matrices [XiYi\h A[XiYi\ andY^AYi, such
that [x: Y1]hA[x1 Yx] = RHR and Y^AYt = R§Ry.
Remark 3.4.12 One approach to constructing A-orthogonal QR factorization of a
matrix is to use the modified Gram-Schmidt process with respect to the A-based scalar
product. See the algorithm below. The MATLAB code gmsa.m to compute the trian-
gular part R by the A-orthogonal QR factorization is provided in Appendix A.
According to Theorem 3.4.10, we formulate Algorithm 3.6 to compute all PABS
in the A-based scalar product by computing the triangular upper matrices from A-
orthogonal QR factorizations.
56


Algorithm: modified Gram-Schmidt with respect to the A-based scalar product
Input: matrix V = . ? ^n]
for i =1:n Rii ~ =
Qi = i j
for j =1 n
j =qfAvj
V3 = Rijqi
end
end
Output: triangular matrix R.
Table 3.6: Algorithm 3.6.
Algorithm 3.6
Input: matrices X\ G Cnxp and Y\ G Cnxq with full column rank .
1. Compute the triangular part R of A-orthogonal QR factorization of L\ = [X\ Y\]-
2. Compute R12 = R(l : p:p -\-1:p -\- q) and R22 = R(p +1p -\- q:p -\- l p ~\~ q).
3. Compute the triangular part Ry of A-orthogonal QR factorization of Y.
4. Compute L^[eye(rt .
zeros(*?,p) R22/Ry
5. Compute the smallest m = min(p, q) singular values si,...,sm of L and the
corresponding m right singular vectors V. The principal angles are computed as
k = 2 arcsin (s^/a/2) for A:=1,...,m.
6. Compute the principal vectors:
lx = ^/2Xi/R{l : p,1:p)V(l p, ) and V3 = ^Y1/RyV(p-\-1:p-\-q,:).
Output: the principal angles ^1,...,and the corresponding principal vectors Vx and
^ in the A-based scalar product.
Finally, we obtain some properties of A-orthogonal projectors. We use the rep-
resentation for the A-orthogonal projector as in [48, 72]. Let Px be an A-orthogonal
projector on the subspace such that Px = XXHA^ where XHAX = I.
57


Theorem 3.4.13 Let Px (md Py be A-orthogonal projectors on the subspaces X and
y, correspondingly. Then
St,W) (A^(PX + Py)A~^) =
1 + CS K,/2)^^)) > 1-CS
In particular,
\\px + Py\\A = i + cs(eAiain(x,y)),
where y) the smallest angle between the subspaces X and y in the A-based
scalar product.
Proof: Let the columns of matrices X and Y form A-orthonormal bases for the
subspaces X and 3^, correspondingly. We have
A1/2(PX + Py)A~1/2 = A1/2(XXHA + YYhA)A~1/2
= A1/2{XXh + YYh)A1/2
= a1/2[xy][xy]ha1/2.
Let L = A1/2[xy], then Al/2{PX + Py)A~1/2 = LLH. Thus, the eigenvalues of
Al/2{PX + Py)A~1/2 are the square of the singular values of L. Notice that we could
add zeros or delete zeros from the singular values of L in order to match the eigen-
values of Al/2{PX + Py)A~1^2. Moreover, Al/2{PX + Py)A~1^2 is positive definite and
Hermitian. The singular values of A1^2(Px + Py)A~1^2 are the same as the eigenvalues.
From Theorem 3.4.3, we obtain the result.
3.5 Matlab implementation
In this section, we describe our code SUBSPACEUPDATE.m in detail. Our func-
tion has more features, compared to the built-in MATLAB function SUBSPACE.m
and the function SUBSPACEA.m in Mathworks written by Knyazev and Argen-
tati [48]. Our function returns principal angles, principal vectors, and normalized
principal coefficients. Our function specifies options. Below is a list of options which
SUBSPACEUPDATE takes:
58


opts.fast=0 Using the algorithm as described in Algorithm 3.1 or Algorithm 3.4 for the A-based scalar product.
opts.fast=l Using the algorithm as described in Algorithm 3.2 or Algorithm 3.5 for the A-based scalar product.
opts.fast=2 Using cosine-based algorithm or cosine-based algorithm for the A-based scalar product.
opts.A Use A-based scalar product, where A is positive definite.
opts.R=0 Use the orthonormal bases or use the A-orthonormal bases for input matrices.
opts.R=l Use the triangular upper matrices from the QR factorization or the A-orthonormal QR factorization of input matrices.
opts.number Find k smallest principal angles, if the number is k.
opts.disp Diagnostic information display level 0,1,2.
With opts.fast = 0, the code is written based on Algorithm 3.1 and Algorithm 3.4.
To compute the principal angles in the A-based scalar product, we add one more
option, such that opts.A = A. With opts.fast =1,the code is written based on
Algorithm 3.2 and Algorithm 3.5. With opts.fast = 2, the code is written based on
the cosine-based algorithm, which computes all principal angles, but it loses accuracy
for small angles. With opts.R=l, we use Algorithm 3.3 and Algorithm 3.6 to compute
principal angles and principal vectors with respect to the standard inner product and
the A-based scalar product, correspondingly. Moreover, we use Algorithm 3.3 to
compute the principal angles for large sparse-type data. We summarize the options
which algorithms 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6 take in the following table.
Name option Name option
Alg. 3.1 opts.fast = 0 Alg. 3.2 opts.fast =1 Alg. 3.3 opts.R =1 Alg. 3.4 opts.fast = 0, opts.A Alg. 3.5 opts.fast =1,opts.A Alg. 3.6 opts.R =1,opts.A
A listing for the function ORTHUPDATE.m is provided in Appendix A.
59


3.6 Numerical tests
We perform the numerical tests for our code SUBSPACEUPDATE.m based on
algorithms 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6 described in previous sections for full-
type structure matrices and Algorithm 3.3 for sparse-type structure matrices. For
comparison, we also run the code SUBSPACEA.m [48] based on the sine-cosine based
algorithm which can be obtained from MathWorks. Numerical tests are performed
on Windows Intel Duo Core and 64-bit operating system, running MATLAB R2011b.
Our first example is taken from [9, 48] with p =13 and n = 26. The matrix Xi
is orthogonal, while Y\ is an n by p Vandermonde matrix. In our code, all compu-
tations are performed in double precision in MATLAB. We observe that our results
in Table 3.9 are coinciding with those of [48] within twelve decimal digits and are
consistent with those of [9] within four decimal digits.
In the next test, we take the example from [48]. Let p = q < n/2. We define n
by p matrices
^1=[I of and Y1 = [i D }H,
where is a diagonal matrix of the size of p such that D = diag(di,...^dp) and
> 0 for fc =1,...,p. / is the identity matrix of the size p and O are zero matrices
of appropriate sizes. The exact values of sines and cosines of principal angles between
column spaces of matrices Xi and Y\ are given by
sinWfc)=
cos(9k)=
4
V1 + 4
1
V1 + 4
for k =1,...,p. Assuming that dkS are arranged in the non decreasing order, then
k^ are in the nonincreasing order. Let k be computed from our algorithms. The
60


Table 3.9: Computed sines and cosines of principal angles.
k sin(6fc) cs(0k)
1 0.9998785422971522 0.01558527040901074
2 0.9981506873301972 0.06078820101183188
3 0.6456213362708716 0.7636577048336605
4 0.4928094246236336 0.8701372713555736
5 0.3975367883303482 0.9175862367777712
6 0.3370430714820327 0.9414892288103764
7 0.2700504602152627 0.9628461709626958
8 0.2156943479780999 0.9764609302221479
9 0.1418470818351141 0.9898885823035148
10 0.1387517672025158 0.9903271919412184
11 0.06089682091193156 0.9981440663565656
12 0.05942261363977260 0.9982329151997635
13 1.072148118598322e-017 1.000000000000000
collective error in principal angles is measured as the following sum:
sin(
+ +
(3.6.1)
We multiply matrices Xi and Y\ by a random orthogonal matrix U\ of the size n on
the left to get
X2 = [/iX! and y2 = U^.
Also we multiply matrices by random orthogonal p by p matrices t/x and t/y, corre-
spondingly, on the right to obtain
X3 = X2Ux and y3 = Y2Uy.
We highlight that the principal angles and the condition numbers are invariant with
respect to these two transformations.
61


Error Growth with Problem Size
50 100 150 200 250 300 350 400 450 500
problem size: p=20
Error Growth with Problem Size
50 100 150 200 250 300 350 400 450 500
problem size: p=20
Figure 3.3: Errors in principal angles: p = 20 top and p as a function of n such
that p n/2 bottom; left for Algorithm 3.1 and Algorithm 3.4 with A I^ and right
for Algorithm sine-cosine and Algorithm sine-cosine with A l.
In the following tests, we start to check the scalability of the code of Algo-
rithms 3.1, 3.2, 3.3, and Algorithms 3.4, 3.5, 3.6 with A = /, for well-conditioned
cases. We plot the collective error in (3.6.1) by increasing the size of the problem n
for the principal angles between X3 and I3.
In our first two tests, we use the same D as in [48]. The diagonal entries of D
are chosen as uniformly distributed random numbers on the interval (0,1). On the
top part of Figure 3.3, we fix p = q = 20 and increase the size of the problem n
and plot the collective error, for the principal angles between X3 and Y3j against the
value n/2. The left side graph of Figure 3.3 presents for Algorithm 3.1 and the right
side graph presents for the sine-cosine based algorithm in [48]. We get the similar
observation as in [48] that the average error grows approximately two times with a
ten time increase in the problem size in both graphs. The behavior of Algorithm 3.1
62


and sine-cosine based algorithm is very similar. On the bottom part of Figure 3.3,
we increase p = q = n/2. We observe that the average error grows with the same
pace as the problem size in both right and left graphs. Algorithm 3.4 and sine-cosine
based algorithm with A I behave very similarly in Figure 3.3. We also try other
analogous tests which verify the similar behavior of these algorithms. We also test
Algorithms 3.2, 3.3 and Algorithms 3.5, 3.6 with A = I. We obtain similar graphs as
these displayed in Figure 3.3, so some graphs are not shown here.
Error Growth with Problem Size
Figure 3.4: Errors in the principal angles for Algorithms 3.1, 3.2, and sine-cosine
based algorithm with D = diag(10_16rand^1^) and p = 20.
In the second test, we focus on testing for small angles. We set p = g = 20 and
select every diagonal element of in the form 10_16rand where rand(is again
a uniformly distributed random number on the interval(0,1).We increase the size of
the problem n and plot the collective error, for the principal angles between X3 and
Y3. We compare the collective errors of our new Algorithms 3.1 and 3.2 with those
of the cosine-sine based algorithm in Figure 3.4. In this figure, we get good accuracy
with errors less than 6 x 10-15 for these three methods. Moreover, among these three
methods, Algorithm 3.1is the most accuracy and Algorithm 3.2 is the least accuracy.
63


In the third test, we turn our attention to the errors for individual PABS,
| sin(4) sin(4)| + | cos(4) cos(4)| for fc =1...p
instead of the sum of errors in (3.6.1). We fix a small p = q and n =100 and compute
the principal angles between X2 and Y2 and between X3 and I3 500 times. In [48],
they test several different combinations of angles less than tt/4. One of the worse-case
scenario is taking
D = diag (l,0.5,10-11,10~12,10~13,5 x 10~15, 2 x 10~15,10~15,10~16, ).
We repeat this test for our Algorithms 3.1,3.2 and the sine-cosine based algorithm,
which is presented in Figure 3.5. The top three graphs of Figure 3.5 show the dis-
tributions of the error for individual angles between X3 and I3 for Algorithm 3.1,
Algorithm 3.2 and sine-cosine based algorithm in 500 runs. In Figure 3.5, smaller an-
gles are further away from the view of reader. Again, we observe that Algorithm 3.1
achieves the best stability and Algorithm 3.2 achieves the least stability for small
angles among these three algorithms. Three graphs on the bottom of Figure 3.5
demonstrate the performance of Algorithm 3.4, Algorithm 3.5 and sine-cosine based
algorithm with A I for the same problem.
Surprisingly, the errors for small individual angles are getting smaller for Algo-
rithm 3.4 with A = I. The main reason is the following. We use Ql^ which has
A-orthonormal columns, to apply in Algorithm 3.4. Ql should return as an n by
k matrix in the exact arithmetic, but in double precision QLl returns as an n by fc
matrix, where k < k in this case. In our code, we add k k zero angles. In this test,
it returns two or three zero angles in Algorithm 3.4. Since the exact three smallest
angles are less than 4 x 10-16, it returns that the error for the three smallest angles
are less than 4 x 10-16 for Algorithm 3.4.
Also, we did the same test for Algorithms 3.3 and 3.6 (see Figure 3.6), which
do not require the orthonormal bases and A-orthonormal bases, correspondingly. In
64


Alg. 3.1
Alg. 3.2
(10'15
angle number reversed
Figure 3.5: Error distributions for individual angles: top three for Algorithm 3.1,
Algorithm 3.2, and Algorithm sine-cosine; bottom three for Algorithms 3.4, 3.5, and
sine-cosine with A I.
angle number reversed
Alg. 3.1,A=I
Alg. 3.2,A=l
angle number reversed
angle number reversed
Sine-cosine Alg,A=l

200
<1-15

angle number reversed
Sine-cosine based Alg
65


Figure 3.6: Errors in the principal angles for Algorithms 3.3 and 3.6.
Figure 3.6, we get high accuracy for both algorithms.
In the next test, we will analyze the stability of our algorithms for ill-conditioned
scalar products. We take A = Ak =10~kI + H for k =1,...,16 as in [48], where
/ is the identity matrix and H is the Hilbert matrix of the order twenty with hij =
l/(z + j 1)for z, j =1,...,20. We take Xi and Yi as G and F in [48], such that
Xx = eye(20,10) and Yi is equal to the last ten columns of the Vandermonde matrix
with elements yiyj = i20~j for z, j =1,...,20, with k(Yi) ^ 1013. We apply the LU
factorization to Yi with partial pivoting in order to reduce the condition number of
Y\. Therefore, we get the lower triangular part matrix Y2j such that TZ(Yi) = 71(Â¥2).
As k increases, the subspaces X and 3^ are invariant, but the condition number of A
increases, since f^(A) ^ 10^. We plot the sines of all ten principal angles as function of
k in logarithmic scale. Also, as in [48], we plot the error, which is defined as follows.
error = \\V^AVX I\\ + \\Vy AVy I\\ + || diag(cos(0)) V^AVy\\,
where and Vy are principal vectors of the pair of subspaces.
In Figure 3.7, we observe that the results of our new algorithms are quite similar
as those of the sine-cosine based algorithm in [48]. The error increases as the condition
number of A increases. Furthermore, among Algorithms 3.4, 3.5, and 3.6, the error
o o o o o
o o o o o
5 4 3 2 1
szJ9qiu
66


Alg. 3.4
Alg. 3.5
Alg. 3.6 Error vs. Condition Number
Figure 3.7: Sine of angles as functions of k for Algorithms 3.4, 3.5, and 3.6 (top
and left on bottom); Errors as functions of condition number for Algorithms 3.4, 3.5,
and 3.6 t^ngnt on bottom).
of Algorithm 3.5 is the smallest for large k. In particular, for fc =16 and k{A) ^ 1018
our code for Algorithm 3.4 returns zeros for 6 smallest angles, which leads to gaps
occurring in Figure 3.7 for Algorithm 3.4. The reason is the same as we have explained
in the previous test. We lose accuracy in the step for A-orthonormal. In Figure 3.7
(bottom), we observe that Algorithm 3.6 is not stable for ill-conditioned A. The
reason is that the modified Gram-Schmidt algorithm with respect to the A-based
scalar product is not stable for ill-conditioned matrix.
3.7 Discussion
In this section, we briefly discuss algorithms which we have presented in the
previous sections, and give a short summary for the comparison for these algorithms
67


and the sine-cosine based algorithm. The algorithms to compute PABS and PABS in
the A-based scalar product are summarized in Tables 3.10.
Table 3.10:_ Algorithms to compute PABS and PABS in the A-based scalar product.
Name Algorithm Name Algorithm (A-based scalar product)
Alg.3.1 X = orthupdate(Xi) Y = orthupdate(Yi) svd of [x y] Alg.3.4 X = orthupdate(Xi, opts.A) Y = orthupdate(Yi, opts.A) Li=[xy] Qlx = rthupdate(Li, opts.A) SVD of
Alg.3.2 X = orthupdate(Xi) Y = orthupdate(Yi) SVD ofXHY: UTVh SVD oiXU -YV Alg.3.5 X = orthupdate(Xi, opts.A) Y = orthupdate(Yi, opts.A) SVD of XhAY: UTVh Li=XU -YV Ql = orthupdate(Li, opts.A) SVD of Q^ALx
Alg.3.3 R = qi([X1Y1}) Ry = qr([i^12 R22Y) / E12i^l SVD of y Alg.3.6 R from A-orthogonal QR of [X\ Yi] Ry from A-orthogonal QR of Y\ / SVD of y 1
Sine-cosine X = orthupdate(Xi) Y = orthupdate(Yi) SVD of XhY SVD of (/ XXh)Y Sine-cosine X = orthupdate(Xi, opts.A) Y = orthupdate(Yi, opts.A) SVD of XhAY Li=(I -XXhA)Y Qlx = rthupdate(Li, opts.A) SVD of Q^ALx
We remark that we use the function ORTHUPDATE.m with the default value,
such that opts.fast is 0, which is is backward compatible with MATLAB built-in
function orth to compute the orthonormal bases. The function orthupdate.m with
option opts.A is backward compatible with ortha.m function in [48], which is more
expensive than A-orthogonal QR factorization using the function mgsa.m based on
68


our MATLAB tests.
Next, we summarize the comparison for algorithms to compute PABS in Ta-
ble 3.11 and PABS in the A-based scalar product in Table 3.12.
Table 3.11: Comparison for algorithms to compute PABS
Name Advantage Disadvantage
Alg.3.1 most accurate expensive for very large n
Alg.3.2 the cost is the same as the cost of less accuracy for small angles com-
sine-cosine based algorithm pare to Alg.3.1 and Alg. sine-cosine
Alg.3.3 worth for sparse matrix
Sine-cosine the cost is the same as Alg.3.2 cannot compute principal vectors in good accuracy corresponding to PABS in the neighborhood of the threshold
Table 3.12: Comparison for algorithms to compute PABS in the A-based scalar
product (1).
Name Advantage Disadvantage
Alg.3.4 if A is available as RHR or KH PABS can be obtained in most high accuracy. if the function orth lose accuracy for [XI"], then those angles return as zeros
Alg.3.5 good accuracy for an ill-conditioned matrix A
Alg.3.6 cheap lose accuracy for an ill-conditioned matrix A and the accuracy depends on MSG algorithm
Sine-cosine good accuracy for an ill-conditioned matrix A lose accuracy for principal vectors corresponding PABS near thresh- old
69


3.8 Conclusions
In this chapter, we have derived several new methods to compute all PABS and
PABS in the A-based scalar product accurately. The algorithms are based on com-
puting the sines of half angles using the algebraic and geometric approaches. We have
also described a numerically robust and fast method to compute PABS of sparse-type
and dense-type matrices using the basis of triangular matrices from QR factoriza-
tions. We conclude this work with several numerical tests. The code is robust and
can provide accurate angles not only for large scale matrices but also for large sparse
matrices.
There remains a number of problems associated with this technique, such as de-
termining orthonormal column in the A-based scalar product to make the fourth step
in Algorithm 3.5 more accurate. Though alternative ways to compute A-orthonormal
basis are available, such as a variant of the Gram-Schmidt algorithm [72] or the mod-
ified Gram-Schmidt as we use for the A-orthogonal QR factorization, those methods
do not fit well for our purpose.
70


4. Majorization inequalities
Majorization is a topic of much interest in various areas of mathematics, e.g., ma-
trix theory, numerical analysis, combinatorics, statistics, and probability. Majoriza-
tion is one of the most powerful techniques that can be used to derive inequalities in
a concise way. In particular, majorization relations among eigenvalues and singular
values of matrices produce a variety of norm inequalities in matrix theory. For a
general discussion of majorization, see Mashall, Olkin and Arnold [55], Bhatia [5],
Ando [1], and Horn and Johnson [36].
In this chapter, we present weak majorization inequalities for singular values and
eigenvalues of Hermitian matrices. We are concerned with the singular values for
the difference of two Hermitian matrices A B bounded by the singular values of
AT TB and T~lA BT~1^ where T is positive definite. The extensions of these
bounds applied to eigenvalues are provided. We also present the relations involving
the difference of eigenvalues of A and B with a matrix AT TB^ where T is invertible.
Our work is inspired by Bhatia, Kittaneh and Li [7], Bhatia, Davis, and Kittaneh [6].
Also, in this chapter we consider the absolute change in the principal angles
between subspaces X and 3^, where subspaces are perturbed to X and y. We use weak
majorization to measure the change. We also provide some basic results involving the
absolute change for the tangents of PABS and present a perturbation analysis for
sines and cosines of half-PABS.
4.1 Definition and properties of majorization
Definition 4.1.1 Weak majorization and majorization
Let and G Rn he the vectors obtained by rearranging the coordinates of vectors x
and y in algebraically nonincreasing orderdenoted by ...x\n\ and ...y\n\
such that X[i] > X[2] > X[n] and y[i] > y[2] > y[n] We say that x is weakly
majorized by yusing the notation x w yif
k k
< ^V[i\, l 71


If in addition we say that x is majorized or strongly majorized
by y7 using the notation x -< y.
For example, we have the following strong majorization:
[]t1.
Moreover, we can see that all vectors with sum majorize the vector with uniform
components, i.e.,
Geometrically, for G the relation x < y holds if and only if x is in the
convex hull of all vectors obtained by permuting the coordinates of y, i.e., see [55].
In Figure 4.1(left), for a fixed vector y we see that all vectors x on the blue segment
are majorized by y, such that [yiJ^y2 x y. In Figure 4.1(right), for
a fixed vector y we see that all vectors x on the area bounded with blue edges are
majorized by y, such that [yi+f 333] x y.
Figure 4.1: Geometry of majorization in 2D and 3D.
We remark that x < y means Xi < ^ for z =1,...,n. Therefore, x < y
implies x - all reflective and transitive [3, 5], but not symmetric, e.g., y -< x and x -< y do
not imply x = y. By the characterization of majorization, Polya proves in [65] that
72


nondecreasing convex functions preserve the weak majorization. In other words, if
y ~ notation f(x) = [f(xi),f(x2),. , f(xn)].
The following two results provide ways to preserve majorization. These two basic
majorization inequalities of vectors can be obtained by straightforward algebra. Let
nonnegative vectors x, y, and v be decreasing and of the same size. If x - xu - However, the converse is not true in general. Moreover, if x - xu - you - 4.2 Some known majorization results
In this section, we briefly review some existing majorization inequalities based on
singular values and eigenvalues of matrices. We provide these well known majorization
results as they serve as a foundation for our new majorization inequalities for singular
values, eigenvalues, and the absolute change in PABS.
In the rest of this thesisA( ), S(.)and Q denote the ordered vectors,
i.e(.)=(.)S(.)=(.)and = .
Theorem 4.2.1 [Lidskii-Mirsky-Wielandt Theorem] [5, 55] If A and B are Hermitian
matricesthen
(A-S).
Theorem 4.2.2 [55, Fan theorem p. 287] For any square matrices A and B, we have
S(A + B) ^ S(A) + S(B).
Theorem 4.2.3 [5, Mirsky^ Theorem] For any square matrices A and 5, we have
\S(A)-S(B)\ ^ S(A-B).
73


Theorem 4.2.4 [5, p. 71,Lidskii^ Theorem] For Hermitian matrices A and B)
|(A)- (_B)| - 5).
Let the eigenvalues of A and B be in nonincreasing order, such that
A(A) = [Ai(A),..., An(A)] and A(B) = [Ai(5),..., An(5)].
There are two inequalities that follow from Theorem 4.2.4. One is called WeyPs
theorem:
max^ |A^(A) A^(5)| < ||A B\\.
The other (see Stewart and Sun [73]) is I()(_B)|2 < ||A 5||i?, which
is a special case of the HofTman-Wielandt theorem established for normal matrices.
The results mentioned above describe the eigenvalues and the singular values of
the difference or the sum of matrices. Next we turn to the singular values of the
product of matrices.
Theorem 4.2.5 [51] For general matrices A and B, we have
S(AB) - If needed, we add zeros to match the sizes of vectors on either side.
Theorem 4.2.6 [5, p. 94, Theorem IV.2.5] Let A and B he square matrices. Then,
for any t > 0 we have
S\AB) - where St(A) = [s^(A),..., s^(A)].
We can see that Theorem 4.2.o is a generalization of Theorem 4.2.5, if A and B
are square matrices. However, Theorem 4.2.5 also works for rectangular matrices.
4.3 Umtarily invariant norms, symmetric gauge functions, and
majorization
74


In this section, we discuss the relationship between unitarily invariant norms,
symmetric gauge functions, and weak majorization.
Definition 4.3.1 Unitarily invariant norms
A norm \ \\ \ \\ on the space of matrices is called unitarily invariantif
\\\UAV\\\ = \\\A\\\,
for all unitary matrices U and V.
For example, the 2-norm, Ky-Fan k norms, Schatten p-norms, and Frobenius
norm are all unitarily invariant norms.
Ky-Fan k norms: ||^4|U = si (^) ? where fc =1,...,min(m, n).
Schatten p-norms: || A|=(E^(m,n)^4)for p > 1.
The Frobenius norm is a special case of a Schatten p-norm for p 2.
Definition 4.3.2 Symmetric gauge functions
A function $ : ^ R called a symmetric gauge function if
^ is a norm,
$(-P = $( for any ^ G and permutation matrix P,
It is absolute, $(|CI) = ¢(0*
The relationship between unitarily invariant norms and symmetric gauge func-
tions is well investigated by von Neumann, e.g., see [35, p. 438, Theorem 7.4.24].
He shows that if $ is a symmetric gauge function, the matrix function defined by
$(S,(A)) is a unitarily invariant norm. Conversely, for every unitarily invariant norm,
there is a corresponding symmetric gauge function $ such that |||A||| = $(S,(A)).
Considering singular value decomposition, it is immediately apparent that a uni-
tarily invariant norm on a matrix A is a function of the singular values of A. Let
75


the SVD of A be UT,VH, then |||A||| = \\\UT,VH\\\ = |||S|||. The following theo-
rem provides an intimate link between unitarily invariant norm inequalities and weak
majorization inequalities.
Theorem 4.3.3 [5] Let A and B he square matrices. Then 5(A) - only i/|||A||| < |||-B||| for every unitarily invariant norm.
From this theorem, we see that the inequalities of unitarily invariant norms are
equivalent to weak majorization of singular values of matrices. For a general discus-
sion of the connection between unitarily invariant norms, symmetric gauge functions,
and majorization, the reader is referred to [5].
Theorem 4.3.4 [5, p. 254] Let A and B he square matrices and AB he a Hermitian
matrix. Then
\\\AB\\\<\\\re(BA)\\\,
for every unitarily invariant norm, or equivalently
S(AB) ^wS(re(BA)),
where re(A) denotes (A + AH)/2.
Theorem 4.3.5 [7] Let A and B he square matrices and AB he a normal matrix.
Then
111(1^1^)111 <111(1^1^)111,
for every unitarily invariant norm, or equivalently
S (\AB\l^) ^ 5 (|5A|^),
where |A|pi=(AHA)1/2.
76


Theorem 4.3.6 [5, p. 236 VIII.3.9] Let A and B he square matrices such that A =
JD\J~l and B = TD2T~1^ where J and T are invertible matrices7 and D\ and D2
are real diagonal matrices. Then
|||diag(A(A))-diag(A(5))||| < [k{J)k{T)]1/2\\\A-B\\l
for every unitarily invariant norm, or equivalently
\A(A)-A(B)\ [k(J)k(T)]1/2S(A-B),
vjhem did^g{A{J\y)denotes a diagonal matrix tuhose diagonal eutries armugai in de-
creasing order are the eigenvalues of A, and k(.) denotes the condition number.
Theorem 4.3.7 [6] Let A and B he Hermitian and T he positive definite. Then
smin(T)\\\A-B\\\<\\\AT-TB\\\,
for every unitarily invariant norm, or equivalently^
Smin(T)S(A B) ^ where smin(T) denotes the smallest singular value ofT.
4.4 New majorization inequalities for singular values and eigenvalues
In this section, we apply the results of Sections 4.2 and 4.3 to derive weak ma-
jorization inequalities for the difference of singular values and eigenvalues of matrices.
Lemma 4.4.1 If A and B are square matrices, then
8(1?)4/2(4/2(
Proof: First, we show that S = Sl^2{T) for all T G Cnxn. Since |T|pi is
semi-positive definite and Hermitian, it follows that 1/2 (|T|pi)=S"1/2 (|T|pi). Also,
it follows that \T\^ is semi-positive definite and Hermitian. Therefore, S =
77


(|T|3) = A1/2 (|T|pol).Moreover, it is easy to check that S"(|T|pol)=S"(T). So,
we have
^(l^li?)=A1/2(|T|pol) = 51/2(T).
Setting T = AB) it follows that
S{\AB\1J^=S1/2{AB).
Applying Theorem 4.2.6 for S1^2(AB)J we obtain the result.
Lemma 4.4.2 For a square matrix A? we have
S (re(A)) 5(A).
Proof: From Theorem 4.2.2 we have
)=^^+).
Since S(A) = S(AH)J we have S (re(A)) - The following theorem is inspired by Bhatia, Kittaneh and Li [7]. We use the
same technique as Bhatia, Kittaneh and Li used in [7, Theorem 2.2] to prove the
majorization inequality of singular values of AT TB and T~lA BT~l.
Theorem 4.4.3 Let A and B he Hermitian and T he positive definite. Then
(A TS) S (T-U ST-1).
Proof: Since A and B are Hermitian, we have \ (A 5)2|p1[A B)2. Thus,
S(A-B) = S{\(A- B)2^) =S{\(A-B) T~^T^ (A 5)|^).
Applying Theorem 4.3.5, we have
-B) 78


Using Lemma 4.4.1, it follows that
S(A -B)^WS ((T1/2 (A B)2T~l/2\1^
w /2 (T1/2 (4 B) T1/2)/2 (T-1/2 (4 B) T-1/2).
Since the nondecreasing convex functions preserve the weak majorization, e.g., x2 for
x > 0, we square both sides to get
S)w S (T1/2 (A S) T1/2) S (T_1/2 (A S) T_1/2).
Using Theorem 4.3.4,
S2(A S)w S (re [(A S) T]) S (re [T1 (A S)]) . (4.4.1)
Since BT TB and BT~l T~lB are skew-Hermitian matrices, i.e., CH ¢7, thus,
re (AT TB) = re [(A B)T + (BT TB)} = re [(A B) T] (4.4.2)
and
re {T~lA BT~l) = re \T~l (A- B)- (BT-1-T~lB)]
=re [T^iA-B)]. (4.4.3)
By Lemma 4.4.2, we obtain
5 [re (AT TB)] - 5 [re {T~lA BT~1)] - Substituting (4.4.2) and (4.4.3) into (4.4.1), together with (4.4.4) and (4.4.5), we
obtain the result as desired.
Remark 4.4.4 In [7, Theorem 2.2] 7 the authors prove the inequality for unitarily
invariant norm such that \\\A B\\\2 < \\\AT TB\\\ |||T_1A 5T_1|||7 which is
equivalent to (see [7, Proposition 2.1])
s^AT-TB)^ (y s^T^A-
79


for fc =1,2... ,n. However^ the weak majorization inequality in our Theorem 4-4-3
implies that
Y, s'(A (s^at (T~lA BT~1))
i=i i=i
for k =1,2 ..., n.
Lemma 4.4.5 Under the assumptions of Theorem we have
S2 (A S) (T-1)(AT TS).
Proof: Since T~lA BT~l T~l (AT TB) T_1, we have
S (T-M ST-1)^ S (T-1)S (AT TS)S (T-1).
By Theorem 4.4.3, this lemma is proved.
Lemma 4.4.6 Let A and B he Hermitian and T he invertible. We have
1. sminCO|-J(-).
2. |A(A) A(B)\2 - 3. |A(A) AB)|2 w 52 (T1) 52 (AT TS).
Proof: Let the SVD of T be UHVH, where U and V are n by n unitary matrices
and L is diagonal, ^mce 1is invertible, it follows that S is positive definite.
AT-TB = AUEVh UEVhB = U (UHAUE EVHBV) VH.
Since the singular values are invariant under unitary transforms, we have
S {AT TB) = S (UhAUE EVHBV) . (4.4.6)
Taking A := UHAU, B := VHBV, and T := S in Theorem 4.3.7, we obtain
Smin(S)5([/HA[/ w . (4.4.7)
80


Applying Theorem 4.2.4, we have
|(A)B)| = |([/(V^SV| w S (. (4.4.8)
Combining (4.4.6), (4.4.7), and (4.4.8), we obtain the first statement.
Next, we prove the second statement. We have
T~lA BT~l = VY,~lUHA BVY,~lUH = F {Y,~lUHAU VHBVY,~l) UH.
Therefore, we have 5 {T~lA BT~l) = 5 {Y,~lUHAU VHBVY,~l).
By Theorem 4.4.3, we have
S2 (^ S S .
By using (4.4.8), we get the second argument. Similarly, applying Lemma 4.4.5 we
obtain the third statement.
Remark 4.4.7 It is worth noting that Theorem 4-2-4 ^ one special case of the first
result of Lemma 4.4.6. In Lemma 4.4.6if we take T as an identity matrixthe first
result of Lemma 4.4.6 becomes \A(A) A(B)\ S (A B).
4.5 Perturbation analysis for principal angles
In this section, we deal with the absolute change in the principal angles between
the subspaces X and 3^, where the subspace A is perturbed to obtain a subspace
X. We establish some perturbation bounds for PABS using majorization techniques.
First, we start with a review of inequalities for the absolute changes of angles between
vectors. For vectors x and y, if the vector x is perturbed to the new vector x, we
have (see [50, 66])
\0(x,y) 0(x,y)\ < 9{x,x).
The perturbation bound for the cosines of angles is
| cos {d{x) y)) cos {d{x) y)) \ < sin {9{x) x))
81


and the perturbation bound the sines of angles is
sin {9{x^ y)) sin (6(x^ y)) \ < sin {9{x^ x)).
For the subspaces X and 3^,let the subspace X be perturbed to the new subspace
with dim(=dim(=dim(3^) =p. There exist results for majorization
inequalities for PABS similar to the angles between vectors, e.g., see [50, 66, 82].
Theorem 4.5.1 [66] |0(^, 3^) 0(^, 3^)| ^ 0(^, ^).
We note that the angles 0(^,3^), 0^,3^), and Q(X^X) are in nonincreasing order.
The left side of the above majorization inequality is
y)-e1(x,y)\,..., \ep(x,y) ep(x,3)|).
Theorem 4.5.1 deals with the principal angles themselves. The next two theorems
deal with the sines and cosines of the principal angles.
Theorem 4.5.2 [50] cos (0(Y)) cos (0(3^)) w sin (0(1
Theorem 4.5.3 [50, 82] sin (0(Y)) sin (0())w sin (0(1
Remark 4.5.4 Generally, we have similar results for perturbation bounds for an-
gles in the A-based scalar product as those for angles in the standard product.
By Lemma 3.4.7, we know that the principal angles between the subspaces X and
y relative to the A-based scalar product are equal to the principal angles between
KX and Ky in the standard productwhere A = KHK. By Theorem 4.5.1we
have \S(KX,Ky) S(KX,Ky)\ - a(y)\ a(.Thereforewe conclude that if the perturbation bounds for
the absolute change of principal angles in the standard product hold, then we have
similar perturbation bounds for the absolute change of principal angles in the A-based
scalar product. For tms reason^ in the following we only deal with the perturbation
bounds for the principal angles in the standard product.
82


What happens with the tangents of PABS, if one subspace is perturbed? To see
what to expect, we derive bounds for vectors first. Let 6(x^y) < tt/2 and 6(x^y) <
tt/2. By direct calculation,
tan (6(x^ y)) tan (6(x^ y))=
sin(6l(x, y)) cos(6l(x, y)) cos(6l(x, y) sin(6l(x, y))
cos(6l(x, y)) cos(6l(x, y))
sin(d(x,y) 9(x,y))
cos(6l(x, y)) cos(6l(x, y))
Moreover,
\sm(d(x,y) d(x,y))\=sm(\d(x,y) 9(x,y)\)
< sin {9{x) x)).
Notice that {x^y) {x^y) G (tt/2,tt/2).
Thus,
|tan (d(x,y)) tan (6(x, y))| <
sin {9{x) x))
cos(6l(x, y)) cos(6l(x, y))
In particular, assuming that 6(x^y) < tt/4 and 6(x^y) < tt/4, we have
cos (0(x, y) + 9(x, y)) + cos (0(x, y) 9(x, y))
(4.5.1)
cos(6l(x, y)) cos(6l(x, y))
Since cosine is decreasing in [0,tt/2), we obtain
2
{d{x) y)) tan {9{x) y))\ < 2tan {9{x) x)).
(4.5.2)
In the following Theorem, we replace subspaces spanned by the vectors x, y, and
x with multidimensional subspaces X^ 3^, and X^ correspondingly. We extend the
inequalities of (4.5.1) and (4.5.2) to the general case.
Theorem 4.5.5 If dim(X) = dim(^) = dim(3^)7 0(^,3^) < tt/2 and 0(^,3^) <
tt/27 then
sin (0(
tan (0(^, 3^)) tan ^0(^, 3^) j

(0max(^,^))CS (0max(^, 3^))'
COS
83


In addition, if 0(^, 3^) < tt/4 and 0(^, 3^) < tt/47 we have
tan (0(^, 3^)) tan ( 0(^, 3^)
2 sin (0(

cos
(<9max (
Proof: Considering the properties of the tangents of PABS, e.g., see Table 2.2, we
have tan (0(^, 3^)) = [S+ (^Py (PyPx)^ and similarly
tan (e(X, y)) = [5+ (p^ (PyP^)
The difference between tan (0(1Y)) and tan ((Y)) can be written as
S
{PyPx)]
(P;f)t) except for some zero singular values.
S
{PyPx)]
S
{PyPx)]) S(Py. {PyPX)] ~ Py. (PyP^)
S(Py.)S((PyP^-(PyP^).
The first majorization inequality is based on Theorem 4.2.3. From Theorem 4.2.5,
we obtain the second majorization inequality. The difference between (PyP^Y and
(PyP^y can be treated as a perturbation for oblique projections [79] or as a pertur-
bation of pseudoinverse [70]. We use the technique of Wedin [79]. We have
(PyPXY {PyP^ = {PyPX)\l ~ (PyP^) ~ (I ~ (PyP^PyP^.
We Know that () is an oblique projector onto along and j () is
an oblique projector onto along Thus,
{PyPx)] = {PyPx)]Py and I (PyPxy = (/ {PyPx)]){I Px).
Similarly, we have
(PyP^Y = P^PyP^ and (/ (PyP^Y) = (/ Py)(I ~ (PyP^).
Therefore,
S {(PyPxy ~ (PyP^Y) = S (~(I (PyPXY)(I PX)P^PyP^)
^WS(I- (PyP^) S {PX^) S ((PyP^)
^ 84


The first majorization inequality follows from Theorem 4.2.5. Thus,
tan (0(^, 3^)) tan ^0(^, 3^) j
sin (0(

cos (0max(^,^))cs (dmax(x,y)y
Furthermore, let 0(^, 3^) < tt/4 and 0^, 3^) < tt/4, by direct computation, we have
COS (9max (X, 3^)) cos (^dmax(x, y)^j > cos (^dmax(x, X)^j . m
In Section 3, we propose several algorithms to compute principal angles based on
the sine of half-PABS. Next we present perturbation bounds for the sine and cosine
of half-PABS, where subspaces are perturbed.
Proposition 4.5.6
cos2 ( ) 3^) ) cos2 ( ) 3^)
sin2 -0(^, 3^) - sin2 -0(^, 3^)
sin ( X)
~ Proof: Since cos2(6,/2)=(1 + cos(6,))/2 and sin2(6,/2)=(1cos(6l))/2, by Theo-
rem 4.5.2 it is easy to obtain the following two results,
and
cos2 -0(^, 3^) - cos2 -0(^, 3^)
sin2 -0(^, 3^) - sin2 -0(^, 3^)
- sin ( 0(^,
w ; sin (0(
Moreover, sin (0(,))=2 sin (0(,)/2) cos (0(,)/2) .
Results in Proposition 4.5.6 are weak majorization estimates for the square of
sines or cosines of half principal angles. Next, we deal with the perturbation bound for
sines or cosines of half-angles. Before we proceed, we present the following theorem,
e.g., [70], from which two new results are derived.
Theorem 4.5.7 [70] Lei X,Y ^ Cnxp with XHX = / and YHY = L If 2p < n,
85


there are unitary matrices Q, U and V such that
] V r
p i
QXU = and QYV = p Q
n p 0 n 2p 0
where T = diag(7i,...,7P) and Q = diag(oi,...^ujp) satisfy
< 7i << % !>...> > 0;
7 + =1 k /p.
(4.5.3)
Lemma 4.5.8 Let cri > ...> (Jp and (J\> ... > ap he sines of half of the principal
angles between subspaces and correspondingly. Thenfor k =..p
Wk Cfk\ < sin
-x(
(4.5.4)
Proof: Without loss of generality, we assume that the dimension of the subspaces
X^y and X is less than n/2, otherwise we could deal with the complementary sub-
spaces oi and A. Let the columns of X and Y be orthonormal bases for the
subspaces X and 3^, correspondingly. Applying Theorem 4.5.7, we have
s(xu-YV) = s\ [/_r -n ol
where U^V^Y and Q are as described in Theorem 4.5.7 and T = 8(0(^, 3^)). Fur-
thermore, by Theorem 3.2.1 S(XU YV) = 2sin(0(y)/2). In a similar way,
we obtain 2sin(0(Y)/2)
=5
-f
where f = cos(0(Y)).
86


Therefore
2 sin -0(^, 3^) - 2 sin -0(^, 3^)
/ I-T \ ( i-t \
5 -Q S -Q
V 0 / V 0 /
^7/? S
r-r
Ci-n
s\ \ (t-r)2 +(n-^2
We note that (f T) + (Cl fl) is a diagonal matrix with diagonal elements
2 cos (^9k(X^ y) k(X^ 3^) j for fc =1,...,p. Hence,
- 2 sin y)j 2 sin 3^)
According to Theorem 4.5.1,
Therefore,
2 sin Q(^, y)^j 2 sin Q(^, 3^)
which completes the proof.
0(^,3^)-0(^,3^)
-
Lemma 4.5.9 Let fii > ... > fip and jli > ... > ftp be cosines of half of the principal
angles between subspaces Xy and A!correspondingly. Thenfor k =.p7
\^k fik\< sin (^9max(X, X)^j . (4.5.5)
Proof: In the notation of the proof of Lemma 4.5.8, we have
S(XU + YV) = 2 cos (0(^, 3^)/2).
The rest of proor is similar to the proof in Lemma 4.5.8.


Remark 4.5.10 In [82]7 Zhang and Li developed majorization inequalities for con-
cave functions of principal angles. They state that if f : [0,tt/2] ^ [,) is a
nondecreasing concave function, then
y)) - y)) ^ Since sin(x/2) is a nondecreasing concave function on [0,tt/2]7 we have
sin -0(^,3^) -sin -0(^,3^)
^ Although the result we obtain in Lemma 4.5.8 is not as strong as the above weak
majorization inequality of sines of half-PABS, we use a different technique to prove
(4.5.4). Furthermore^ we use the same technique to prove the bound for the change of
cosines of half-PA BS in Lemma 4.5.9.
Proposition 4.5.11 Let
a = cos (0(X, y)/2) cos y)/2)j
b = sin (e(X, y)/2) sin (e(x, y)/2^j .
Then
[|a||6|]t
>/2 sin ( -Q{X) X) ) 0,
Proof: Let columns of matrices X, X and Y be orthonormal bases for the subspaces
X and 3^, correspondingly. Let the SVD of XHX be UTjVh. By Theorem 3.1.1
we have
and
v/2
cos ( -Q{X,y) ) ,sin ( -Q{X,y)
S[[xu Y]
V2
cos ( -0(X,y) ) ,sin ( -(x,y)
S[[xv Y]
88


We may need to add zeros to the vectors on the right side. Moreover, we have
5
Y])-S[[XV Y]
S
-XV o]
[S(XU-XV),
[S(XUVH X),
2sin ( 0(f))0..
From Theorem 4.2.3, we obtain weak majorization for the first inequality. The last
equality is obtained by Theorem 3.2.1. Let a = cos (0(Y)/2) cos (0(Y)/2)
and 6 = sin(0(Y)/2) sin (0(Y)/2) Then we have
v^[K |6|]
2 sin -0(X,X) ,0,
which completes the proof.

Remark 4.5.12 Let a = and b
majorization, we have
+z w z
sm
where
..bp]. From the definition of
> l P
a, = cos (9t(X, y)/2) cos [9t(X, y)/2^
bt = sin y)/2) sin (%{X, 3^)/2)
Moreover^ it is worth noting that the bound for the sum of the absolute changes of the
sines of half-PABS and the absolute changes of cosines of half-PABS in Proposition
4-5.11 are sharp? since the constant on the right side of Proposition ^.5.11 is \^2 and
it will be at least 2 by combining Lemmas 4.5.8 and 4.5.9.
All results for the perturbation bounds for changes of angles presented above use
the condition that one of subspaces is perturbed. Now we assume both subspaces X
and 3^ are perturbed to subspaces X and 3^, correspondingly. We obtain the following
lemma which can be easily obtained from Theorems 4.5.1, 4.5.2, and 4.5.3.
89


Corollary 4.5.13 If dim(X) = dim(^) = dim(3^) = dim(3^) we have
|e(- (| ^ e(e) + (
.cos (0(,y)) cos (0(,5)w sin (0(,
sin (0(,y)) sin (0(,))w sin (0(,
+ sin (0(3^,^)).
+ sin (0(3^,^)).
Proof: We prove the first statement,
\e(x,y)-e(x,y)\ = \e(x,y)-e(x,y) + e(x,y)-e(x,y)\
< \e(x,y) e(x,y)\ + \e(x,y) e(x,y)\
- The proof for the other two is similar.
Similarly, we bound the squares or smes and cosines of half-PABS as follows.
Corollary 4.5.14
cos2 ( -0(y) ) cos2 ( -0(y)
sin2 -0(^,3^) -sin2 -0(^,3^)
- Theorem 4.5.15 Letai > .. crp anddi > ... > (7P he sines of half of the principal
angles between subspaces X and Xycorrespondingly. Thenfor k =1p
\(7k 5-fcl < sin X) ) + sin -6,max(3^, .y)).
(4.5.6)
Proof: First, by Lemma 4.5.8, for k 1,...we have
Wk -fc| < sin -9max(X, X)
Second, we apply a similar statement to the sine of half-PABS between subspaces
X^y and correspondingly. We have


Using the triangle inequality, we obtain the result as desired.

Similarly, we can establish an estimate of the cosine of half-PABS with respect
to perturbations of two subspaces.
Theorem 4.5.16 Let > ... > and fli jlp be cosines of half of the
principal angles between subspaces and correspondingly. Thenfor k =
1...
\f^k t^k\ ^ sin ^-6,max(A,, + sin y)^j * (4.5.7)
Let us emphasize that we can obtain similar perturbation bounds for sines or
cosines of half-PABS with respect to the A-based scalar product as in Lemmas 4.5.8
and 4.5.9, and Theorems 4.5.15 and 4.5.16.
Let us now turn our attention to a perturbation analysis in terms of matrices Xi
and Xi + AXi that generate subspaces X and X. Often a subspace is defined by
the column space of a matrix. We are interested in estimating the sensitivity of a
column space of a matrix. There are several results provided by Sun [74], and Golub
and Zha [29] involving the perturbation analysis for the principal angles of a pair of
matrices. The bounds in [74] are improved in [29] without restricting the dimensions
of two subspaces to be equal. Moreover, Knyazev and Argentati [48] generalize results
of [29, 74] to the A-based scalar product.
Lemma 4.5.17 [29, 48, 74] Let X and X be column-spaces of matrices Xx and
Xi + AX1? correspondingly. Let ka(Xi) = smax(A1/2X1)/smin(A1/2X1) denote the
corresponding A-based condition number of Xlt Then
s\n(eA_(x,x))< ^(^i)llpf/^||11-
In particular, for A = I, we have
91

PAGE 1

ANGLESBETWEENSUBSPACESANDTHERAYLEIGH-RITZMETHOD by PeizhenZhu M.S.,UniversityofColoradoDenver,2009 Athesissubmittedtothe FacultyoftheGraduateSchoolofthe UniversityofColoradoinpartialfulllment oftherequirementsforthedegreeof DoctorofPhilosophy AppliedMathematics 2012

PAGE 2

ThisthesisfortheDoctorofPhilosophydegreeby PeizhenZhu hasbeenapproved by AndrewKnyazev,Advisor JulienLangou,Chair IlseIpsen DianneO 0 Leary BeresfordParlett October26,2012 ii

PAGE 3

Zhu,PeizhenPh.D.,AppliedMathematics AnglesbetweensubspacesandtheRayleigh-Ritzmethod ThesisdirectedbyProfessorAndrewKnyazev ABSTRACT Thisthesisconcernsfourinterdependenttopics.Therstoneisprincipalangles orcanonicalanglesbetweensubspacesPABSandtheirtangents.Thesecondtopic isdevelopingalgorithmstocomputePABSinhighaccuracybyusingsinesofhalfangles.ThethirdtopicistheRayleighquotientRQandtheeigenvalueperturbation theoryforboundedself-adjointoperatorsinHilbertspaces.Thelast,butnotleast, topicistheRayleigh-RitzRRmethodandmixedmajorization-typeerrorbounds ofRitzvalueandeigenvalueperturbations. PABSserveasaclassicaltoolinmathematics,statistics,andapplications,e.g., datamining.Traditionally,PABSareintroducedandusedviatheircosines.ThetangentsofPABShaveattractedrelativelylessattention,butareimportantforanalysis ofconvergenceofsubspaceiterationsforeigenvalueproblems.Weexplicitlyconstruct matrices,suchthattheirsingularvaluesareequaltothetangentsofPABS,usingorthonormalandnon-orthonormalbasesforsubspaces,andorthogonalprojectors. ComputingPABSisoneofthebasicimportantproblemsinnumericallinear algebra.TocomputePABSandtheprincipalvectorsinhighaccuracyforlarge, i.e.,near = 2,andsmallangles,weproposenewalgorithmsbyusingthesinesof half-angles.WegeneralizeouralgorithmstocomputingPABSinthe A -basedscalar product.WealsodescribeanumericallyrobustandfastwaytocomputePABSfrom theuppertriangularmatricesoftheQRfactorizations.Perturbationanalysisforthe sinesandcosinesofhalf-PABSisprovidedusingthemajorizationtechnique.The resultsofnumericaltestsaregiven,supportingthetheory. iii

PAGE 4

Nextweturnourattentiontotheeigenvalueperturbationtheory.If x isan eigenvectorofaself-adjointboundedoperator A inaHilbertspace,thentheRQ x ofthevector x isanexacteigenvalueof A .Inthiscase,theabsolutechangeofthe RQ j x )]TJ/F18 11.9552 Tf 11.081 0 Td [( y j becomestheabsoluteerrorinaneigenvalue x of A approximated bytheRQ y onagivenvector y: Therearethreetraditionalkindsofboundsofthe eigenvalueerror:aprioriboundsviatheanglebetweenvectors;aposterioribounds viathenormoftheresidual;mixedtypeboundsusingboththeangleandthenorm oftheresidual.Inthiswork,weexaminethesethreekindsofboundsandderivenew identitiesandboundsforthechangeintheRQwithrespecttothechangeofthe vectors.Weproposeaunifyingapproachtoproveknownboundsofthespectrum, analyzetheirsharpness,andderivenewsharperbounds. Finally,weconsiderboundsoftheabsolutechangesofRitzvaluesandeigenvalue approximationerrors.TheRRmethodisawidelyusedtechniqueforcomputing approximationstoeigenvaluesandeigenvectorsofaHermitianmatrix A ,fromagiven subspace.WetakeadvantageofPABSandtheresidualmatrixtoderiveseveralnew majorization-typemixedboundsthatimproveandgeneralizetheexistingresults.Our boundsholdforsubspacesnotnecessary A -invariantandextendthemixedboundof theabsolutechangeofthevectorRQtomultidimensionalsubspaces. Theformandcontentofthisabstractareapproved.Irecommenditspublication. Approved:AndrewKnyazev iv

PAGE 5

ACKNOWLEDGMENT Iwishtoexpressmysinceregratitudetomyadvisor,ProfessorAndrewKnyazev, forhisgenerousinstruction,continuouspatience,endlessencouragement,andsupport throughoutmygraduatestudies.Withouthisguidance,thepresentdissertation wouldnotbepossible.IjustcannotthankProfessorKnyazevenough. ThePhDcommitteeChair,ProfessorJulienLangou,hasdoneagreatjoborganizingthedefense.Iwouldalsoliketothanktherestofmycommitteemembers, ProfessorsIlseIpsen,DianneO 0 Leary,andBeresfordParlett,fortheirtime,attention andinsightfulcomments. IwouldliketoexpressmyappreciationtoDr.MericoArgentati,whohasbeena knowledgeableandfriendlysourceofadvice.Hiscommentsareveryvaluable. IamgratefultothefacultyandmyfellowstudentsfromtheDepartmentof MathematicalandStatisticalSciences.Inparticular,IwouldliketothankProfessor RichardLundgrenandProfessorLeopoldoFrancaforreferringandintroducingme toProfessorKnyazevandhisgroup.IwouldliketothankmyfellowstudentsHenc Bouwmeester,EricSullivan,KannanutChamsri,YongxiaKuang,andMinjeongKim forsharingtheirvariousknowledge. Inaddition,IwouldliketoexpressmygratitudetotheBatemanfamilyandthe DepartmentofMathematicalandStatisticalSciencesfortheirinvaluablesupport. MyresearchandworkonthethesishavealsobeensupportedbyNSFDMS grants0612751and1115734,thePIProfessorAndrewKnyazev.Ihavepresented somepreliminaryresultsofthethesisattheConferenceonNumericalLinearAlgebra Austin,TX,July20,2010.MytalkhasbeenacceptedtotheHouseholderSymposium XVIIIonNumericalLinearAlgebraJune12-17,2011,butcanceledbecauseofmy pregnancy.Iacknowledgethesupportoftheconferenceorganizers. Finally,IwouldliketothankmyhusbandXinhuaandmyparentsfortheirendless supportandlove,andmydaughterJessieforbeinganinspiration. v

PAGE 6

TABLEOFCONTENTS Figures.......................................viii Tables........................................ix Chapter 1.Introduction...................................1 1.1Overview.................................1 1.2Notation.................................5 2.PrincipalanglesbetweensubspacesPABSandtheirtangents.......7 2.1DenitionandbasicpropertiesofPABS................7 2.2TangentsofPABS............................10 2.2.1tanintermsoftheorthonormalbasesofsubspaces.....11 2.2.2tanintermsofthebasesofsubspaces............16 2.2.3tanintermsoftheorthogonalprojectorsonsubspaces...21 2.3Conclusions...............................26 3.NewapproachestocomputePABS......................27 3.1Algebraicapproachestocomputesin )]TJ/F16 7.9701 Tf 6.675 -4.976 Td [( 2 andcos )]TJ/F16 7.9701 Tf 6.675 -4.976 Td [( 2 ........29 3.2Geometricapproachestocomputesin )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 ...............37 3.3Computingsin )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 andcos )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 withoutusingorthonormalbases.42 3.4Computingsin )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 andcos )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 inthe A -basedscalarproduct....49 3.5Matlabimplementation.........................58 3.6Numericaltests.............................60 3.7Discussion................................67 3.8Conclusions...............................70 4.Majorizationinequalities............................71 4.1Denitionandpropertiesofmajorization...............71 4.2Someknownmajorizationresults...................73 4.3Unitarilyinvariantnorms,symmetricgaugefunctions,andmajorization75 vi

PAGE 7

4.4Newmajorizationinequalitiesforsingularvaluesandeigenvalues..77 4.5Perturbationanalysisforprincipalangles...............81 4.6Conclusions...............................92 5.BoundsfortheRayleighquotientandthespectrumofself-adjointoperators94 5.1Introduction...............................94 5.2Shortreviewofsomeknownerrorboundsforeigenvalues......96 5.3KeyidentitiesforRQ..........................98 5.3.1Identitiesforthenormoftheresidual r x = Ax )]TJ/F18 11.9552 Tf 11.955 0 Td [( x x ..99 5.3.2IdentitiesfortheabsolutechangeintheRQ..........100 5.4Derivingsomeknowneigenvalueerrorboundsfromouridentities..104 5.5Newaposterioriboundsforeigenvalues................105 5.6Improvingknownerrorboundsforeigenvectors............108 5.7Conclusions...............................111 6.MajorizationboundsfortheabsolutechangeofRitzvaluesandeigenvalues112 6.1Rayleigh-Ritzmethod..........................114 6.2Perturbationofeigenspaces.......................115 6.3Majorization-typemixedbounds....................117 6.4Discussion................................126 6.5Conclusions...............................130 7.Conclusionsandfuturework..........................131 Appendix A.MATLABcode.................................133 References ......................................150 vii

PAGE 8

FIGURES Figure 2.1ThePABSin2D................................11 2.2Geometricalmeaningof T = P X ? P Y P X P Y y ................24 3.1Sinesandcosinesofhalf-angles........................32 3.2Sineofhalf-anglebetween x and y in R n ..................38 3.3Errorsinprincipalangles: p =20topand p asafunctionof n suchthat p = n= 2bottom;leftforAlgorithm3.1andAlgorithm3.4with A = I andrightforAlgorithmsine-cosineandAlgorithmsine-cosinewith A = I .62 3.4ErrorsintheprincipalanglesforAlgorithms3.1,3.2,andsine-cosinebased algorithmwith D =diag )]TJ/F16 7.9701 Tf 6.586 0 Td [(16rand ;p and p =20.............63 3.5Errordistributionsforindividualangles:topthreeforAlgorithm3.1, Algorithm3.2,andAlgorithmsine-cosine;bottomthreeforAlgorithms 3.4,3.5,andsine-cosinewith A = I: ....................65 3.6ErrorsintheprincipalanglesforAlgorithms3.3and3.6..........66 3.7Sineofanglesasfunctionsof k forAlgorithms3.4,3.5,and3.6topand leftonbottom;ErrorsasfunctionsofconditionnumberforAlgorithms 3.4,3.5,and3.6rightonbottom......................67 4.1Geometryofmajorizationin2Dand3D..................72 6.1ComparisonoftheabsolutechangeofRitzvaluesforthemixedbounds andfortheaprioriboundswiththeexactabsolutechangeofRitzvalues left;Thecorrespondingtangentofthelargestprincipalangleright..129 6.2ComparingthemixedtypeboundoftheabsolutechangeofRitzvalues withtheexactabsolutechangeofRitzvalues...............129 viii

PAGE 9

TABLES Table 2.1Dierentmatricesin F :matrix T 2F usingorthonormalbases......15 2.2Dierentexpressionsfor T ..........................25 2.3Dierentformulasfor T with X ; Y <= 2:dim X dim Y left; dim X dim Y right...........................26 3.1Algorithm3.1.................................33 3.2Algorithm3.2.................................41 3.3Algorithm3.3.................................46 3.4Algorithm3.4.................................53 3.5Algorithm3.5.................................54 3.6Algorithm3.6.................................57 3.9Computedsinesandcosinesofprincipalangles...............61 3.10AlgorithmstocomputePABSandPABSinthe A -basedscalarproduct.68 3.11ComparisonforalgorithmstocomputePABS...............69 3.12ComparisonforalgorithmstocomputePABSinthe A -basedscalarproduct.....................................69 ix

PAGE 10

1.Introduction 1.1Overview Thisdissertationmainlyconsistsoffourparts.Therstpartdealswithprincipal anglesorcanonicalanglesbetweensubspacesPABSandtheirtangents.The secondpartdevelopsalgorithmstocomputePABSinhighaccuracybyusingsines ofhalf-angles.ThethirdpartstudiestheRayleighquotientRQandtheeigenvalue perturbationtheoryforboundedself-adjointoperatorsinHilbertspaces.Thelastpart explorestheRayleigh-RitzRRmethodandpropertiesofRitzvaluesandeigenvalues ofHermitianmatrices. ThehistoryofPABSdatestoJordanin1875[40].Instatistics,thecosinesof principalanglesandprincipalvectorsareinterpretedascanonicalcorrelationsand canonicalvariables,correspondingly.PABSarewidelyusedinmanyapplications, e.g.,inecology,informationretrieval,randomprocesses,facerecognition,andsystem identication;e.g.,[27,39,43,45,78].Thecosines,sines,ortangentsofPABS arecommonlyusedtomeasuretheaccuracyofnumericalcomputationofinvariant subspacesofmatrices;e.g.,[47,51,64,73]. ThesinesandespeciallycosinesofPABSarewellstudied;e.g.,in[9,74].Tangents ofPABSalsoappearinapplications,butarelessinvestigated.In[10,20,52,71,73], researchersdenethetangentsofPABSofthesamedimensionviathenormor singularvaluesofsomematrices,andusethetangentsasanalytictools.Specically, thetangentsofPABSarerelatedtothesingularvaluesofamatrix|withoutexplicit formulationofthematrix|in[73,p.231-232]and[71,Theorem2.4,p.252].The tangentofthelargestprincipalangleisobtainedfromthenormofamatrixin[10,20]. In[52],thetangentsareusedtoderiveconvergencerateboundsofsubspaceiterations. InChapter2,webrieyreviewtheconceptsandthemostfundamentalandimportantpropertiesofPABS.Wegiveexplicitmatrices,suchthattheirsingularvalues arethetangentsofPABS.Weconstructmatricesusingthreedierentscenarios: 1

PAGE 11

orthonormalbases,non-orthonormalbases,andorthogonalprojectors.Usingageometricapproach,newconstructionsofsuchmatricesareobtained.Inaddition,our resultsincludetheformulationsofmatricesconsistentwiththosein[10,20,52,71,73]. Furthermore,wepresentthetangentsofPABSwithpossiblydierentdimensions. ComputingPABSisoneofthebasicimportantproblemsinnumericallinear algebra.Anecientmethod,basedonsingularvaluedecompositionSVDforcalculatingcosinesofPABS,isproposedbyBjorckandGolub[9].ThecosinesofprincipalanglesandprincipalvectorsareobtainedbyusingtheSVDof X H Y ,wherethe columnsofmatrices X and Y formorthonormalbasesforthesubspaces X and Y correspondingly. However,thecosine-basedalgorithmcannotprovideaccurateresultsforsmallangles,i.e.,anglessmallerthan10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(8 arecomputedaszeroindoubleprecisionarithmetic with EPS 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(16 .Forthesamereason,thesine-basedalgorithmlosesaccuracyfor computinglargeprincipalangles,i.e.,closeto = 2.Aremedyistocombinesine andcosinebasedalgorithmsproposedbyKnyazevandArgentati[48],wheresmall andlargeanglesarecomputedseparately.Theyintroducedathresholdtocompute theangleslessthanthethresholdusingthesine-basedapproachandtocomputethe anglesgreaterthanthethresholdusingthecosine-basedapproach.However,this algorithmiscomplicated,andintroducesanewpotentialdiculty|accuratecomputationoftheprincipalvectorscorrespondingtothePABSintheneighborhoodof thethreshold. InChapter3,weproposenewalgorithmstocomputePABSandtheprincipal vectorsusingthesinesof half -PABS.Ournewalgorithmsprovideaccurateresults forallPABSinonesweepandcanbeimplementedsimplyandeciently.Wealso proposeamethodtocomputePABSforsparsedata.Furthermore,wegeneralizeour algorithmstocomputetheprincipalanglesandtheprincipalvectorsinan A -based scalarproductaswell,where A isapositivedeniteHermitianmatrix.Perturbation 2

PAGE 12

analysisforthesinesandcosinesofhalf-PABSisprovidedinthiswork.Resultsof extensivenumericaltestsareincluded. InChapter4,webrieyreviewsomeexistingmajorizationinequalitiesbasedon thesingularvaluesandtheeigenvaluesofmatrices.Usingmajorizationtechnique,we boundtheabsolutechangesofeigenvaluesoftwoHermitianmatricesintermsofthe singularvaluesofthedierenceoftwomatrices.Wealsousemajorizationtechnique toboundthechangesofsinesandcosinesofhalf-PABS,andoftangentsofPABS. If x isaneigenvectorofaself-adjointboundedoperator A inaHilbertspace,then theRQ x ofthevector x isanexacteigenvalueof A .Inthiscase,theabsolute changeoftheRQ j x )]TJ/F18 11.9552 Tf 11.398 0 Td [( y j becomestheabsoluteerrorinaneigenvalue x of A approximatedbytheRQ y onagivenvector y .Therearethreetraditionalkindsof boundsofeigenvalueerrors:aprioribounds[49,53,67]viatheanglebetweenvectors x and y ;aposterioribounds[13,44,64]viathenormoftheresidual Ay )]TJ/F18 11.9552 Tf 11.523 0 Td [( y y ofa vector y ;andmixedtypebounds[47,75]usingboththeangleandthenormofthe residual.InChapter5,weproposeauniedapproachtoproveknownboundsofthe spectrum,analyzetheirsharpness,andderivenewsharperbounds. TheRRmethod[64,73]isaclassicalmethodforeigenvalueapproximationof Hermitianmatrices.ArecentpaperbyKnyazevandArgentati[51]presentsapriori errorboundsofRitzvaluesandeigenvaluesintermsoftheprincipalanglesbetween subspaces X and Y usingaconstant.In[5,64,73],researchersderiveaposterior boundsofeigenvaluesintermsofthesingularvaluesoftheresidualmatrix.Another aposteriorboundobtainedbyMathias[56]usestheresidualmatrixandthegap. InChapter6,wetakeadvantageoftheprincipalanglesandtheresidualmatrix topresentnewmajorization-typemixedbounds.Weimprovetheboundsin[61,75] ifoneofthesubspacesis A -invariant.Inaddition,ourboundsaremoregeneral,since theyholdforsubspacesnotnecessary A -invariant.Wegeneralizethemixedbound oftheabsolutechangeoftheRQ j x )]TJ/F18 11.9552 Tf 12.14 0 Td [( y j tomultidimensionalsubspaces.This 3

PAGE 13

workalsoprovidesmajorization-typemixedboundsforRitzvaluesandeigenvalues, andforRQvectorperturbationidentitiesforapplicationingeneralizedeigenvalue problems.Wecompareournewboundswiththeknownones. WepresentourconclusionsandrecommendsomefurtherworkinChapter7. Themainnewresultsobtainedinthethesisaresummarizedasfollows: 1.WeanalyzethetangentsofPABSrelatedtotheSVDofexplicitlyconstructed matrices.NewpropertiesofPABSareobtained. 2.Originalnumericalalgorithmsaredeveloped,basedonthesinesofhalf-PABS, tocomputePABSandprincipalvectorsforcolumnrangesofdenseandsparse matrices.Moreover,wegeneralizeouralgorithmstocomputePABSinthe A -basedscalarproduct. 3.Weusemajorizationtechniquestopresentnovelperturbationanalysisforsines andcosinesofhalf-PABS. 4.Aunifyingapproachisproposed,basedonnewRQvectorperturbationidentities,toproveboundsoftheoperatorspectrum,analyzetheirsharpness,and derivenewsharpbounds. 5.Wepresentseveralmajorization-typemixedboundsfortheabsolutechanges ofRitzvaluesandtheRRabsoluteeigenvalueapproximationerror.The majorization-typemixedboundsareintermsoftheprincipalanglesandthe residualmatrix. 4

PAGE 14

1.2Notation A = A H AHermitianmatrixin C n n oraself-adjointboundedoperator. S A Thevectorofsingularvaluesof A and S A =[ s 1 A ;:::;s n A ]. S + A Thevectorofpositivesingularvaluesof A S # A Thesingularvaluesof A arearrangedinnonincreasingorder. S m # A Thevectorofthelargest m singularvaluesof A in nonincreasingorder. S # m A Thevectorofthesmallest m singularvaluesof A in nonincreasingorder. s max A Thelargestsingularvalueof A s min A Thesmallestsingularvalueof A k x k Vectornormof x k A k Spectralnormofthematrix A ,i.e., k A k = s max A k A k F Frobeniusnormof A jjj A jjj Unitarilyinvariantnormof A X ; Y Subspacesof C n n X ? Theorthogonalcomplementofthesubspace X : R X Thecolumnspaceofamatrix X P X Orthogonalprojectoronthesubspace X or R X P X ? Orthogonalprojectoronthesubspace X ? : A Thevectorofeigenvaluesof A and A =[ 1 A ;:::; n A ]. # A Theeigenvaluesof A arearrangedinnonincreasingorder. X ; Y Thevectorofprincipalanglesbetweenthesubspaces X and Y and X ; Y =[ 1 ;:::; p ]. # X ; Y Principalanglesarearrangedinnonincreasingorder. 5

PAGE 15

" X ; Y Principalanglesarearrangedinnondecreasingorder. ;= 2 Thevectorofprincipalanglesin ;= 2 : A X ; Y Thevectorofprincipalanglesinthe A -basedscalarproduct between X and Y max X ; Y Thelargestanglebetweensubspaces X and Y : min X ; Y Thesmallestanglebetweensubspaces X and Y : rank A Therankofthematrix A x y Thevector x ismajorizedbythevector y x w y Thevector x isweaklymajorizedby y j x j Theabsolutevalueofthevector x A y Moore-Penroseinverseof A re A re A = A + A H 2 j A j pol Thesquarerootofthematrix A H A ,i.e., j A j pol = A H A 1 = 2 h ; i Aninnerproduct,associatedwithanormby kk 2 = h ; i : h ; i B Thescalarproductinducedby B ,where B ispositivedenite. A Theconditionnumberofamatrix A inthe2norm,suchthat A = s max A =s min A : x TheRayleighquotientof A withrespecttothevector x H ArealorcomplexHilbertspace. spec A Spectrumoftheboundedoperator A 6

PAGE 16

2.PrincipalanglesbetweensubspacesPABSandtheirtangents 1 TheconceptofprincipalanglesbetweensubspacesPABSisrstintroducedby Jordan[40]in1875.Hotelling[37]denesPABSintheformofcanonicalcorrelations instatisticsin1936.NumerousresearchersworkonPABS;see,e.g.,ourpseudorandomchoiceofinitialreferences[18,38,50,73,74,79],outoftensofthousandof Internetlinksforthekeywordsprincipalangles"andcanonicalangles". Inthischapter,werstbrieyreviewtheconceptandsomeimportantproperties ofPABSinSection2.1.Traditionally,PABSareintroducedandusedviatheirsines andmorecommonly,becauseoftheconnectiontocanonicalcorrelations,cosines.The tangentsofPABShaveattractedrelativelylessattention.Ourinteresttothetangents ofPABSismotivatedbyitsapplicationsintheoreticalanalysisofconvergenceof subspaceiterationsforeigenvalueproblems. WereviewsomepreviousworkonthetangentsofPABSinSection2.2.Themain goalofthischapterisexplicitlyconstructingafamilyofmatricessuchthattheir singularvaluesareequaltothetangentsofPABS.Weformthesematricesusingseveraldierentapproaches:orthonormalbasesforsubspacesinSubsection2.2.1,nonorthonormalbasesinSubsection2.2.2,andorthogonalprojectorsinSubsection2.2.3. Throughoutthischapter,wealsodiscovernewpropertiesofPABS. 2.1DenitionandbasicpropertiesofPABS Inthissection,weremindthereaderoftheconceptofPABSandsomefundamentalpropertiesofPABS.First,werecallthatanacuteanglebetweentwononzero vectors x and y isdenedas cos x;y = j x H y j ; with x H x = y H y =1,where0 x;y = 2. 1 Thematerialofthischapterisbasedonourmanuscript,P.ZhuandA.V.Knyazev, Principal anglesbetweentwosubspacesandtheirtangents ,submittedtoLinearAlgebraanditsApplications. Also,itisavailableathttp://arxiv.org/abs/1209.0523. 7

PAGE 17

Thedenitionofanacuteanglebetweentwononzerovectorscanbeextendedto PABS;see,e.g.,[9,24,30,37]. Denition2.1.1 Let X C n and Y C n besubspaceswith dim X = p and dim Y = q: Let m =min p;q : Theprincipalangles X ; Y =[ 1 ;:::; m ] ; where k 2 [0 ;= 2] ;k =1 ;:::;m; between X and Y arerecursivelydenedby s k =cos k =max x 2X max y 2Y j x H y j = j x H k y k j ; subjectto k x k = k y k =1 ;x H x i =0 ;y H y i =0 ;i =1 ;:::;k )]TJ/F15 11.9552 Tf 11.955 0 Td [(1 : Thevectors f x 1 ;:::;x m g and f y 1 ;:::;y m g arecalledtheprincipalvectors. AnalternativedenitionofPABSisproposedin[9,30]basedonthesingular valuedecompositionSVDandreproducedhereasthefollowingtheorem.Itis shownin[24]thattheSVDapproachtodenePABSisadirectconsequenceofthe recursivedenition. Theorem2.1.2 Letthecolumnsofmatrices X 2 C n p and Y 2 C n q formorthonormalbasesforthesubspaces X and Y ,correspondingly.LettheSVDof X H Y be U V H ; where U and V areunitarymatricesand isa p by q diagonalmatrix withtherealdiagonalelements s 1 X H Y ;:::;s m X H Y innonincreasingorderwith m =min p;q .Then cos X ; Y = S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y = s 1 )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y ;:::;s m )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y ; where X ; Y denotesthevectorofprincipalanglesbetween X and Y arrangedin nondecreasingorderand S A denotesthevectorofsingularvaluesof A .Moreover, theprincipalvectorsassociatedwiththispairofsubspacesaregivenbytherst m columnsof XU and YV; correspondingly. 8

PAGE 18

Theorem2.1.2impliesthatPABSaresymmetric,i.e. X ; Y = Y ; X ,and unitarilyinvariant,since UX H UY = X H Y foranyunitarymatrix U 2 C n n AnumberofotherimportantpropertiesofPABShavebeenestablished,fornite dimensionalsubspaces,e.g.,in[23,38,50,73,74,79],andforinnitedimensional subspacesin[18,53].WelistafewofthemostusefulpropertiesofPABSbelow. Property2.1.3 [74] InthenotationofTheorem2.1.2,let p q and q n )]TJ/F18 11.9552 Tf 12.044 0 Td [(p ,let [ XX ? ] beaunitarymatrix.Let s 1 X H ? Y s q X H ? Y besingularvaluesof X H ? Y .Then s k X H ? Y =sin q +1 )]TJ/F19 7.9701 Tf 6.586 0 Td [(k ;k =1 ;:::;q: Relationshipsofprincipalanglesbetween X and Y ,andbetweentheirorthogonal complementsarethoroughlyinvestigatedin[38,50,53].Lettheorthogonalcomplementsofthesubspaces X and Y bedenotedby X ? and Y ? ,correspondingly.The nonzeroprincipalanglesbetween X and Y arethesameasthosebetween X ? and Y ? : Similarly,thenonzeroprincipalanglesbetween X and Y ? arethesameasthose between X ? and Y : Property2.1.4 [38,50,53] Forthesubspaces X Y andtheirorthogonalcomplements,wehave 1. # X ; Y ; 0 ;:::; 0 = # X ? ; Y ? ; 0 ;:::; 0 : wherethereare max n )]TJ/F15 11.9552 Tf 13.037 0 Td [(dim X )]TJ/F15 11.9552 Tf 13.037 0 Td [(dim Y ; 0 additional0sontheleftand maxdim X +dim Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(n; 0 additional0sontheright. 2. # X ; Y ? ; 0 ;:::; 0 = # X ? ; Y ; 0 ;:::; 0 : wherethereare maxdim Y )]TJ/F15 11.9552 Tf 12.596 0 Td [(dim X ; 0 additional0sontheleftandthere are maxdim X )]TJ/F15 11.9552 Tf 11.955 0 Td [(dim Y ; 0 additional0sontheright. 3. 2 ;:::; 2 ; # X ; Y = 2 )]TJ/F15 11.9552 Tf 11.956 0 Td [( X ; Y ? ; 0 ;:::; 0 ; wherethereare maxdim X )]TJ/F15 11.9552 Tf 13.76 0 Td [(dim Y ; 0 additional = 2 s ontheleftand 9

PAGE 19

maxdim X +dim Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(n; 0 additional0sontheright. ThesestatementsarewidelyusedtodiscovernewpropertiesofPABS. 2.2TangentsofPABS ThetangentsofPABSserveasanimportanttoolinnumericalmatrixanalysis. Forexample,in[10,20],theauthorsusethetangentofthelargestprincipalangle derivedfromanormofaspecicmatrix.In[71,Theorem2.4,p.252]and[73,p.231232]thetangentsofPABS,relatedtosingularvaluesofamatrix|withoutanexplicit matrixformulation|areusedtoanalyzeperturbationsofinvariantsubspaces.In[52], thetangentsareusedtoderivetheconvergencerateboundsofsubspaceiterations. In[10,20,52,71,73],thetwosubspaceshavethesamedimensions. Lettheorthonormalcolumnsofmatrices X X ? ,and Y spanthesubspaces X theorthogonalcomplement X ? of X ,and Y ,correspondingly.AccordingtoTheorem2.1.2andProperty2.1.3,cos X ; Y = S X H Y andsin X ; Y = S X H ? Y : ThepropertiesofsinesandespeciallycosinesofPABSarewellinvestigated;e.g.,see in[9,74].ThetangentsofPABShaveattractedrelativelylessattention.Onecould obtainthetangentsofPABSbyusingthesinesandthecosinesofPABSdirectly, however,thisisnotthemostecientoraccurateapproachinsomesituations. Inthiswork,weconstructafamily F ofexplicitlygivenmatrices,suchthatthe singularvaluesofthematrix T arethetangentsofPABS,where T 2F .Weform T inthreedierentways.First,wederive T astheproductofmatriceswhosecolumns formtheorthonormalbasesofsubspaces.Second,wepresent T astheproductof matriceswithnon-orthonormalcolumns.Third,weform T astheproductoforthogonalprojectionsonsubspaces.Forbetterunderstanding,weprovideageometric interpretationofsingularvaluesof T andoftheactionof T asalinearoperator. Ourconstructionsincludethematricespresentedin[10,20,52,71,73].Furthermore,weconsiderthetangentsofprincipalanglesbetweentwosubspacesnotonly withthesamedimension,butalsowithdierentdimensions. 10

PAGE 20

2.2.1 tan intermsoftheorthonormalbasesofsubspaces ThesinesandcosinesofPABSareobtainedfromthesingularvaluesofexplicitly givenmatrices,whichmotivateustoconstruct T asfollows:ifmatrices X and Y haveorthonormalcolumnsand X H Y isinvertible,let T = X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ,then tan X ; Y canbeequaltothepositivesingularvaluesof T .Webeginwithan exampleintwodimensionsinFigure2.1. Figure2.1: ThePABSin2D. Let X = 2 6 4 cos sin 3 7 5 ;X ? = 2 6 4 )]TJ/F15 11.9552 Tf 11.291 0 Td [(sin cos 3 7 5 ; and Y = 2 6 4 cos + sin + 3 7 5 ; where0 <= 2and 2 [0 ;= 2].Then, X H Y =cos and X H ? Y =sin : Obviously,tan coincideswiththesingularvalueof T = X H ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 .If = = 2, thenthematrix X H Y issingular,whichdemonstratesthat,ingeneral,weneedto useitsMoore-PenrosePseudoinversetoformourmatrix T = X H ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y y TheMoore-PenrosePseudoinverseiswellknown.Foraxedmatrix A 2 C n m ; theMoore-PenrosePseudoinverseistheuniquematrix A y 2 C m n satisfying AA y A = A;A y AA y = A y ; AA y H = AA y ; A y A H = A y A: 11

PAGE 21

SomepropertiesoftheMoore-PenrosePseudoinversearelistedasfollows. If A = U V H isthesingularvaluedecompositionof A ,then A y = V y U H If A hasfullcolumnrank,and B hasfullrowrank,then AB y = B y A y However,thisformuladoesnotholdforgeneralmatrices A and B AA y istheorthogonalprojectorontherangeof A ,and A y A istheorthogonal projectorontherangeof A H ForadditionalpropertiesoftheMoore-PenrosePseudoinverse,wereferthereader to[73].Nowwearereadytoprovethattheintuitiondiscussedabove,suggestingto try T = X H ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y y ,iscorrect.Moreover,wecoverthecaseofthesubspaces X and Y havingpossiblydierentdimensions. Theorem2.2.1 Let X 2 C n p haveorthonormalcolumnsandbearbitrarilycompletedtoaunitarymatrix [ XX ? ] .Letthematrix Y 2 C n q besuchthat Y H Y = I Thenthepositivesingularvalues S + T ofthematrix T = X H ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y y satisfy tan R X ; R Y =[ 1 ;:::; 1 ;S + T ; 0 ;:::; 0] ; .2.1 where R denotesthematrixcolumnrange. Proof: Let[ YY ? ]beunitary,then [ XX ? ] H [ YY ? ] isunitary,suchthat [ XX ? ] H [ YY ? ] = qn )]TJ/F18 11.9552 Tf 11.955 0 Td [(q p n )]TJ/F18 11.9552 Tf 11.955 0 Td [(p 2 6 4 X H Y X H Y ? X H ? Y X H ? Y ? 3 7 5 : ApplyingtheclassicalCS-decompositionCSD,e.g.,[22,28,62,63],weget [ XX ? ] H [ YY ? ]= 2 6 4 X H Y X H Y ? X H ? Y X H ? Y ? 3 7 5 = 2 6 4 U 1 U 2 3 7 5 D 2 6 4 V 1 V 2 3 7 5 H ; 12

PAGE 22

where U 1 2U p ;U 2 2U n )]TJ/F18 11.9552 Tf 12.418 0 Td [(p ;V 1 2U q ,and V 2 2U n )]TJ/F18 11.9552 Tf 12.418 0 Td [(q .Thesymbol U k denotesthefamilyofunitarymatricesofthesize k .Thematrix D hasthefollowing structure: D = 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 rsq )]TJ/F19 7.9701 Tf 6.587 0 Td [(r )]TJ/F19 7.9701 Tf 6.586 0 Td [(sn )]TJ/F19 7.9701 Tf 6.587 0 Td [(p )]TJ/F19 7.9701 Tf 6.587 0 Td [(q + rsp )]TJ/F19 7.9701 Tf 6.586 0 Td [(r )]TJ/F19 7.9701 Tf 6.586 0 Td [(s r I O H s C 1 S 1 p )]TJ/F19 7.9701 Tf 6.587 0 Td [(r )]TJ/F19 7.9701 Tf 6.587 0 Td [(s O I n )]TJ/F19 7.9701 Tf 6.586 0 Td [(p )]TJ/F19 7.9701 Tf 6.586 0 Td [(q + r O )]TJ/F18 11.9552 Tf 9.299 0 Td [(I s S 1 )]TJ/F18 11.9552 Tf 9.299 0 Td [(C 1 q )]TJ/F19 7.9701 Tf 6.587 0 Td [(r )]TJ/F19 7.9701 Tf 6.587 0 Td [(s I O H 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 ; where C 1 =diagcos 1 ;:::; cos s ,and S 1 =diagsin 1 ;:::; sin s with k 2 ;= 2for k =1 ;:::;s ,whicharetheprincipalanglesbetweenthesubspaces R Y and R X .Thematrices C 1 and S 1 couldbeempty.Matrix O isthematrixofzeros anddoesnothavetobesquare. I denotestheidentitymatrix.Wemayhavedierent sizesof I in D: Therefore, T = X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(X H Y y = U 2 2 6 6 6 6 4 O S 1 I 3 7 7 7 7 5 V H 1 V 1 2 6 6 6 6 4 I C 1 O 3 7 7 7 7 5 y U H 1 = U 2 2 6 6 6 6 4 O S 1 C )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 1 O H 3 7 7 7 7 5 U H 1 : Hence, S + T =tan 1 ;:::; tan s ,where0 < 1 2 s <= 2. Fromthematrix D ,weobtain S )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H Y = S diag I;C 1 ;O : AccordingtoTheorem2.1.2, R X ; R Y =[0 ;:::; 0 ; 1 ;:::; s ;= 2 ;:::;= 2],where k 2 ;= 2 for k =1 ;:::;s ,andtherearemin q )]TJ/F18 11.9552 Tf 10.045 0 Td [(r )]TJ/F18 11.9552 Tf 10.045 0 Td [(s;p )]TJ/F18 11.9552 Tf 10.045 0 Td [(r )]TJ/F18 11.9552 Tf 10.045 0 Td [(s additional = 2'sand r additional 0'sontheright-handside,whichcompletestheproof. Remark2.2.2 Ifweknowthesubspaces X Y andtheirorthogonalcomplements explicitly,wecanobtaintheexactsizesofblockmatricesinthematrix D .Letus 13

PAGE 23

considerthedecompositionofthespace C n intoanorthogonalsumofvesubspaces asin [11,31,53] C n = M 00 M 01 M 10 M 11 M ; where M 00 = XY ; M 01 = XY ? ; M 10 = X ? Y ; M 11 = X ? Y ? X = R X and Y = R Y .UsingTables1and2in [53] andTheorem3.12in [41] ,weget dim M 00 = r; dim M 10 = q )]TJ/F18 11.9552 Tf 11.955 0 Td [(r )]TJ/F18 11.9552 Tf 11.955 0 Td [(s; dim M 11 = n )]TJ/F18 11.9552 Tf 11.955 0 Td [(p )]TJ/F18 11.9552 Tf 11.955 0 Td [(q + r; dim M 01 = p )]TJ/F18 11.9552 Tf 11.955 0 Td [(r )]TJ/F18 11.9552 Tf 11.955 0 Td [(s; where M = M X M X ? = M Y M Y ? with M X = X M 00 M 01 ? ; M X ? = X ? M 10 M 11 ? ; M Y = Y M 00 M 10 ? ; M Y ? = Y ? M 01 M 11 ? ; and s =dim M X =dim M Y =dim M X ? =dim M Y ? : Thus,wecanrepresent D as 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 dim M 00 dim M = 2dim M 10 dim M 11 dim M = 2dim M 01 dim M 00 I O H dim M = 2 C 1 S 1 dim M 01 O I dim M 11 O )]TJ/F18 11.9552 Tf 9.298 0 Td [(I dim M = 2 S 1 )]TJ/F18 11.9552 Tf 9.298 0 Td [(C 1 dim M 10 I O H 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 : Inaddition,itispossibletopermutetherst q columnsorthelast n )]TJ/F18 11.9552 Tf 11.98 0 Td [(q columnsof D,ortherst p rowsorthelast n )]TJ/F18 11.9552 Tf 11.708 0 Td [(p rowsandtochangethesignofanycolumnor rowtoobtainthevariantsoftheCSD. 14

PAGE 24

Remark2.2.3 InTheorem2.2.1,thereareadditional dim XY 0'sandadditional min )]TJ/F15 11.9552 Tf 5.48 -9.683 Td [(dim X ? Y ; dim XY ? 1 0 s in [ 1 ;:::; 1 ;S + T ; 0 ;:::; 0] : Duetothefactthattheanglesaresymmetric,i.e., X ; Y = Y ; X ,the matrix T inTheorem2.2.1couldbesubstitutedwith Y H ? X Y H X y : Moreover,the nonzeroanglesbetweenthesubspaces X and Y arethesameasthosebetween thesubspaces X ? and Y ? .Hence, T canbepresentedas X H Y ? X H ? Y ? y and Y H X ? Y H ? X ? y .Furthermore,foranymatrix T ,wehave S T = S T H ,which impliesthatallconjugatetransposesof T holdinTheorem2.2.1.Let F denotea familyofmatrices,suchthatthesingularvaluesofthematrixin F arethetangents ofPABS.Tosumup,anyoftheformulasfor T in F intherstcolumnofTable2.1 canbeusedinTheorem2.2.1. Table2.1: Dierentmatricesin F :matrix T 2F usingorthonormalbases. X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y y P X ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y y Y H ? X Y H X y P Y ? X Y H X y X H Y ? X H ? Y ? y P X Y ? X H ? Y ? y Y H X ? Y H ? X ? y P Y X ? Y H ? X ? y Y H X y Y H X ? Y H X y Y H P X ? X H Y y X H Y ? X H Y y X H P Y ? Y H ? X ? y Y H ? X Y H ? X ? y Y H ? P X X H ? Y ? y X H ? Y X H ? Y ? y X H ? P Y Usingthefactthatthesingularvaluesareinvariantunderunitarymultiplications, wecanalsouse P X ? Y X H Y y for T inTheorem2.2.1,where P X ? isanorthogonal projectorontothesubspace X ? .Thus,wecanuse T asinthesecondcolumnin Table2.1,where P X P Y ,and P Y ? denotetheorthogonalprojectorsonthesubspace X Y ,and Y ? ,correspondingly.Notethat P X ? = I )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X and P Y ? = I )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y 15

PAGE 25

Remark2.2.4 FromtheproofofTheorem2.2.1,weimmediatelyderivethat X H ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y y = )]TJ/F15 11.9552 Tf 9.299 0 Td [( Y H ? X ? y Y H ? X and Y H ? X Y H X y = )]TJ/F15 11.9552 Tf 9.298 0 Td [( X H ? Y ? y X H ? Y: Remark2.2.5 Let X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y y = U 2 1 U 1 and Y H ? X Y H X y = V 2 2 V 1 ; bethe SVDs,where U i and V i i =1 ; 2 areunitarymatricesand i i =1 ; 2 arediagonal matrices.Letthediagonalelementsof 1 and 2 beinthesameorder.Thenthe principalvectorsassociatedwiththepairofsubspaces X and Y aregivenbythe columnsof XU 1 and YV 1 2.2.2 tan intermsofthebasesofsubspaces Intheprevioussection,weusematrices X and Y withorthonormalcolumns toformulate T .Aninterestingquestioniswhetherwecanchoosenon-orthonormal basesforthesesubspacestogetthesameresultasinTheorem2.2.1.Itturnsout thatwecanchooseanon-orthonormalbasisforoneofthesubspaces X or Y .Before wepresentthetheorem,weneedthefollowinglemma. Lemma2.2.6 Forintegers q p ,let [ XX ? ] beaunitarymatrixwith X 2 C n p .Let thematrix Y 2 C n q besuchthat X H Y hasfullrank.Letusdene Z = Y Y H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 ^ T = X H ? Z )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Z y ,and T = X H ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y y .Then T = ^ T: Proof: Thematrix X H Y p q hasfullrankand q p ,whichimpliesthatthe matrix Y hasfullrank,i.e.,dim R Y =dim R X H Y = q: So,thematrix Y H Y is nonsingular.Moreover,thecolumnsof Z = Y Y H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 areorthonormalandspan R Y .Usingthefactthatif A isoffullcolumnrank,and B isoffullrowrank,then AB y = B y A y ,itfollowsthat X H Z y = X H Y Y H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 y = Y H Y 1 = 2 X H Y y : Finally,wehave ^ T = X H ? Z )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H Z y = X H ? Y Y H Y )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 Y H Y 1 = 2 X H Y y = X H ? Y X H Y y = T: 16

PAGE 26

Ingeneral,if q>p ,thematrices T and ^ T aredierent.Forexample,let X = 2 6 6 6 6 4 1 0 0 3 7 7 7 7 5 ;X ? = 2 6 6 6 6 4 00 10 01 3 7 7 7 7 5 ; and Y = 2 6 6 6 6 4 11 01 00 3 7 7 7 7 5 : Then,wehave X H Y =[ 11 ]and X H Y y =[ 1 = 21 = 2 ] H .Thus, T = X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y y =[ 1 = 20 ] H : Ontheotherhand,weobtain ^ T = X H ? Z )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(X H Z y =[ 00 ] H : FromtheproofofLemma2.2.6,weseethatthematrix X H Z y isnotequalto Y H Y 1 = 2 X H Y y forthecase q>p ,sinceingeneraltheformula AB y = B y A y does nothold.Furthermore,intheexampleabovewehave cos R X ; R Y = s X H Y Y H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 =1 ; whichimpliesthattan R X ; R Y =0 : However, s T =1 = 2 ; whichalsoshows thatthecondition q p isnecessaryinthefollowingtheorem. Theorem2.2.7 Forintegers q p ,let X 2 C n p haveorthonormalcolumnsandbe arbitrarilycompletedtoaunitarymatrix [ XX ? ] .Let Y 2 C n q besuchthat X H Y hasfullrank.Let T = X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y y .Then tan R X ; R Y =[ S + T ; 0 ;:::; 0] ; wherethereare dim R X R Y 0'sontheright. Proof: CombiningTheorem2.2.1,Remark2.2.3,andLemma2.2.6directly provesthistheorem. Remark2.2.8 Wemaketheexplicitmatrix T inTheorem2.2.7connectiontothe implicitmatrixwhichisrelatedtothetangentsofanglesdescribedin [73,71] .Using theideadescribedin [73,p.231-232] [71,Theorem2.4,p.252] ,and [52] ,weconstruct 17

PAGE 27

Z as X + X ? T: Since )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y y X H Y = I ,thefollowingidentities Y = P X Y + P X ? Y = XX H Y + X ? X H ? Y = ZX H Y implythat R Y R Z .Bydirectcalculation,we easilyobtainthat X H Z = X H X + X ? T = I and Z H Z = X + X ? T H X + X ? T = I + T H T: Thus, X H Z )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(Z H Z )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 = I + T H T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 isHermitianpositivedenite.Thematrix Z Z H Z )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 byconstructionhasorthonormalcolumnswhichspanthespace Z Moreover,weobservethat S X H Z Z H Z )]TJ/F17 5.9776 Tf 7.782 3.258 Td [(1 2 = I + T H T )]TJ/F17 5.9776 Tf 7.782 3.258 Td [(1 2 = h )]TJ/F15 11.9552 Tf 5.479 -9.683 Td [(1+ s 2 1 T )]TJ/F17 5.9776 Tf 7.782 3.258 Td [(1 2 ;:::; )]TJ/F15 11.9552 Tf 5.479 -9.683 Td [(1+ s 2 p T )]TJ/F17 5.9776 Tf 7.782 3.258 Td [(1 2 i ; where denotesthevectorofeigenvalues. Therefore, tan R X ; R Z =[ S + T ; 0 ;:::; 0] and dim X =dim Z : From Theorem2.2.7,wehave tan ;= 2 R X ; R Y =tan ;= 2 R X ; R Z ; where ;= 2 denotesallPABSin ;= 2 : Inotherwords,theanglesin ;= 2 betweensubspaces R X and R Y arethesameasthosebetweensubspaces R X and R Z .Forthecase p = q; wecanseethat R X ; R Y = R X ; R Z Wegiveanexplicitmatrix X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 fortheimplicitmatrix P ,where S P = tan R X ; R Y ,in [73,p.231-232] and [71,Theorem2.4,p.252] .Thecase p = q isconsideredin [52] .Weextendtheirresulttothecase q p: Corollary2.2.9 UsingthenotationofTheorem2.2.7,let p = q and P X ? bean orthogonalprojectiononthesubspace R X ? .Wehave tan R X ; R Y = S P X ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 : Proof: Sincethesingularvaluesareinvariantunderunitarytransforms,itiseasy toobtain h S X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ; 0 ;:::; 0 i = S P X ? Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 .Moreover,the numberofthesingularvaluesof P X ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 is p ,henceweobtaintheresult. Abovewealwayssupposethatthematrix Y hasfullrank.Next,wederivea similarresultasinTheorem2.2.7,if Y isnotafullrankmatrix. 18

PAGE 28

Theorem2.2.10 Let [ XX ? ] beaunitarymatrixwith X 2 C n p .Let Y 2 C n q and rank Y =rank )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H Y p .Thenthepositivesingularvaluesof T = X H ? Y )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H Y y satisfy tan R X ; R Y =[ S + T ; 0 ;:::; 0] ; wherethereare dim R X R Y 0'sontheright. Proof: Lettherankof Y be r ,thus r p .LettheSVDof Y be U V H ,where U is an n n unitarymatrix;isan n q realrectangulardiagonalmatrixwithdiagonal entries s 1 ;:::;s r ; 0 ;:::; 0orderedbydecreasingmagnitudesuchthat s 1 s 2 s r > 0,and V isa q q unitarymatrix.Sincerank Y = r q ,wecancomputea reducedSVDsuchthat Y = U r r V H r .Onlythe r columnvectorsof U andthe r row vectorsof V H ,correspondingtononzerosingularvaluesareused,whichmeansthat r isan r r invertiblediagonalmatrix.Let Z 1 = Y )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(Y H Y +1 = 2 V r = U r r V H r h V r )]TJ/F15 11.9552 Tf 5.479 -9.684 Td [( H r r )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 V H r i 1 = 2 V r : Let C = V H r h V r )]TJ/F15 11.9552 Tf 5.48 -9.683 Td [( H r r )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 V H r i 1 = 2 V r ,then C isHermitianandinvertible.Moreover,wehave Z H 1 Z 1 = I r r .Sincethematrix Y hasrank r ,thecolumnsof Z 1 are orthonormalandtherangeof Z 1 isthesameastherangeof Y .Therefore,wehave tan R X ; R Y =tan R X ; R Z 1 : Moreover,let T 1 = X H ? Z 1 )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(X H Z 1 y .AccordingtoTheorem2.2.1,itfollowsthat tan R X ; R Z 1 =[ S + T 1 ; 0 ;:::; 0] ; wheretherearedim R X R Z 1 0'sontheright.Ourtaskisnowtoshowthat T 1 = T .Bydirectcomputation,wehave T 1 = X H ? U r r C )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H U r r C y = X H ? U r r CC y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H U r r y = X H ? U r r )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H U r r y : 19

PAGE 29

Thesecondequalityisbasedon )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H U r r C y = C y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H U r r y ,sincerank )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y = r .Sincethematrix C isinvertible,weobtainthelastequality.Ontheotherhand, T = X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y y = X H ? U r r V H r )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H U r r V H r y = X H ? U r r V H r )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(V H r y )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H U r r y = X H ? U r r )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H U r r y : Hence,tan R X ; R Y =[ S + T ; 0 ;:::; 0]whichcompletestheproof. Next,weprovideanalternativewaytoconstruct T byadoptingonlythetriangularmatrixfromoneQRfactorization. Corollary2.2.11 Let X 2 C n p and Y 2 C n q with q p bematricesoffull rank,andalsolet X H Y befullrank.Letthetriangularpart R ofthereducedQR factorizationofthematrix L =[ XY ] be R = 2 6 4 R 11 R 12 OR 22 3 7 5 : Thenthepositivesingularvaluesofthematrix T = R 22 R 12 y satisfy tan R X ; R Y =[ S + T ; 0 ;:::; 0] ; wherethereare dim R X R Y 0'sontheright. Proof: LetthereducedQRfactorizationof L be [ XY ]=[ Q 1 Q 2 ] 2 6 4 R 11 R 12 OR 22 3 7 5 ; wherethecolumnsof[ Q 1 Q 2 ]areorthonormal,and R Q 1 = X .Wedirectlyobtain Y = Q 1 R 12 + Q 2 R 22 : Multiplyingby Q H 1 onbothsidesoftheaboveequalityfor Y weget R 12 = Q H 1 Y: Similarly,multiplyingby Q H 2 wehave R 22 = Q H 2 Y: Combining theseequalitieswithTheorem2.2.10completestheproof. 20

PAGE 30

2.2.3 tan intermsoftheorthogonalprojectorsonsubspaces Inthissection,wepresent T usingorthogonalprojectorsonly.Beforeweproceed, weneedalemmaandacorollary. Lemma2.2.12 [73] If U and V areunitarymatrices,thenforanymatrix A ,we have UAV y = V H A y U H : Corollary2.2.13 Let A 2 C n n beablockmatrix,suchthat A = 2 6 4 BO 12 O 21 O 22 3 7 5 ; where B isa p q matrixand O 12 ;O 21 ,and O 22 arezeromatrices.Then A y = 2 6 4 B y O H 21 O H 12 O H 22 3 7 5 : Proof: LettheSVDof B be U 1 V H 1 ; whereisa p q diagonalmatrix,and U 1 and V 1 areunitarymatrices.Thenthereexistunitarymatrices U 2 and V 2 ,such that A = 2 6 4 U 1 U 2 3 7 5 2 6 4 O 12 O 21 O 22 3 7 5 2 6 4 V H 1 V H 2 3 7 5 : Thus, A y = 2 6 4 V 1 V 2 3 7 5 2 6 4 O 12 O 21 O 22 3 7 5 y 2 6 4 U H 1 U H 2 3 7 5 = 2 6 4 V 1 V 2 3 7 5 2 6 4 y O H 21 O H 12 O H 22 3 7 5 2 6 4 U H 1 U H 2 3 7 5 : Since B y = V 1 y U H 1 ,wehave A y = 2 6 4 B y O H 21 O H 12 O H 22 3 7 5 asdesired. Theorem2.2.14 Let P X P X ? and P Y beorthogonalprojectorsonthesubspaces X X ? and Y ,correspondingly.Thenthepositivesingularvalues S + T ofthematrix T = P X ? P Y P X P Y y satisfy tan X ; Y =[ 1 ;:::; 1 ;S + T ; 0 ;:::; 0] ; 21

PAGE 31

wherethereareadditional min )]TJ/F15 11.9552 Tf 5.479 -9.684 Td [(dim X ? Y ; dim XY ? 1 0 s andadditional dim XY 0'sontheright. Proof: Letthematrices[ XX ? ]and[ YY ? ]beunitary,where R X = X and R Y = Y : Bydirectcalculation,weobtain 2 6 4 X H X H ? 3 7 5 P X P Y [ YY ? ]= 2 6 4 X H YO OO 3 7 5 : Thematrix P X P Y canbewrittenas P X P Y =[ XX ? ] 2 6 4 X H YO OO 3 7 5 2 6 4 Y H Y H ? 3 7 5 : .2.2 ByLemma2.2.12andCorollary2.2.13,wehave P X P Y y =[ YY ? ] 2 6 4 X H Y y O OO 3 7 5 2 6 4 X H X H ? 3 7 5 : .2.3 Inasimilarway,weget P X ? P Y =[ X ? X ] 2 6 4 X H ? YO OO 3 7 5 2 6 4 Y H Y H ? 3 7 5 : .2.4 Let T = P X ? P Y P X P Y y and B = X H ? Y )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(X H Y y ,so T =[ X ? X ] 2 6 4 X H ? YO OO 3 7 5 2 6 4 X H Y y O OO 3 7 5 2 6 4 X H X H ? 3 7 5 =[ X ? X ] 2 6 4 BO OO 3 7 5 2 6 4 X H X H ? 3 7 5 : Thesingularvaluesareinvariantunderunitarymultiplication.Hence, S + T = S + B .FromTheorem2.2.1,wehavetan X ; Y =[ 1 ;:::; 1 ;S + T ; 0 ;:::; 0]. Moreover,fromRemark2.2.3wecanobtaintheexactnumbersfor0and 1 Remark2.2.15 FromtheproofofTheorem2.2.14,wecanobtainthepropertiesfor sinesandcosinesrelatedtoorthogonalprojectors.Accordingtorepresentation 2.2.2 22

PAGE 32

wehave S P X P Y = S 0 B @ 2 6 4 X H YO OO 3 7 5 1 C A =[cos X ; Y ; 0 ;:::; 0] : Similarly,from equality 2.2.4 andProperty2.1.4wehave S P X ? P Y = S 0 B @ 2 6 4 X H ? YO OO 3 7 5 1 C A = cos X ? ; Y ; 0 ;:::; 0 =[1 ;:::; 1 ; sin X ; Y ; 0 ;:::; 0] ; wherethereare maxdim Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(dim X ; 0 additional 1 s ontheright. Furthermore,wehave 2 6 4 X H ? X H 3 7 5 P X )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y [ YY ? ]= 2 6 4 )]TJ/F18 11.9552 Tf 9.299 0 Td [(X H ? YO OX H Y ? 3 7 5 ; whichimpliesthatthesingularvaluesof P X )]TJ/F18 11.9552 Tf 11.093 0 Td [(P Y aretheunionofthesingularvalues of X H ? Y and X H Y ? ,suchthat S P X )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y = S X H ? Y ;S X H Y ? : ByProperty 2.1.3,itfollowsthat S ; 1 P X )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y = sin ;= 2 X ; Y ; sin ;= 2 X ; Y ; where S ; 1 denotesthesingularvaluesin ; 1 ;seealso [50,60] fordierentproofs. Wenotethatthenullspaceof P X P Y istheorthogonalsum Y ? )]TJ/F21 11.9552 Tf 5.48 -9.684 Td [(YX ? .The rangeof P X P Y y isthus Y )]TJ/F21 11.9552 Tf 5.48 -9.684 Td [(YX ? ? ; whichistheorthogonalcomplementofthe nullspaceof P X P Y .Moreover, P X P Y y isanobliqueprojector,because P X P Y y = )]TJ/F15 11.9552 Tf 5.48 -9.684 Td [( P X P Y y 2 whichcanbeobtainedbydirectcalculationusingequality2.2.3.The obliqueprojector P X P Y y projectson Y along X ? .Thus,wehave P Y P X P Y y = P X P Y y : Consequently,wecansimplify T = P X ? P Y P X P Y y as T = P X ? P X P Y y inTheorem2.2.14. Togaingeometricalinsightinto T ,inFigure2.2wechooseanarbitraryunit vector z .Weproject z on Y along X ? ,thenprojectonthesubspace X ? whichis 23

PAGE 33

Figure2.2: Geometricalmeaningof T = P X ? P Y P X P Y y interpretedas Tz .Theredsegmentinthegraphistheimageof T underallunit vectors.Itisstraightforwardtoseethat s T = k T k =tan UsingProperty2.1.4andthefactthatprincipalanglesaresymmetricwithrespect tothesubspaces X and Y ,theexpressions P Y ? P X P Y P X y and P X P Y ? P X ? P Y ? y can alsobeusedinTheorem2.2.14. Remark2.2.16 AccordingtoRemark2.2.4andTheorem2.2.14,wehave P X P Y ? P X ? P Y ? y = P X P X ? P Y ? y = )]TJ/F24 11.9552 Tf 11.291 13.27 Td [( P X ? P X P Y y H : Furthermore,bythegeometricalpropertiesoftheobliqueprojector P X P Y y ,itfollows that P X P Y y = P Y P X P Y y = P X P Y y P X : Therefore,weobtain P X ? P Y P X P Y y = P X ? P X P Y y = P Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X P X P Y y : Similarly,wehave P Y ? P X P Y P X y = P Y ? P Y P X y = P X )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y P Y P X y = )]TJ/F24 11.9552 Tf 11.291 9.684 Td [()]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(P Y P Y ? P X ? y H : Fromequalitiesabove,weseethatthereareseveraldierentexpressionsfor T in Theorem2.2.14,whicharesummarizedinTable2.2. 24

PAGE 34

Table2.2: Dierentexpressionsfor T P X ? P Y P X P Y y P X ? P X P Y y P Y ? P X P Y P X y P Y ? P Y P X y P X P Y ? P X ? P Y ? y P X P X ? P Y ? y P Y P X ? P Y ? P X ? y P Y P Y ? P X ? y P X )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y P X P Y y P Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X P Y P X y Theorem2.2.17 Let X ; Y <= 2 .Then, 1.If dim X dim Y ,wehave P X P Y P X P Y y = P X P X P Y y = P X : 2.If dim X dim Y ,wehave P X P Y y P X P Y = P X P Y y P Y = P Y : Proof: Letthecolumnsofmatrices X and Y formorthonormalbasesforthe subspaces X and Y ,correspondingly.Since X ; Y <= 2,thematrix X H Y has fullrank.Usingthefactthat X H Y X H Y y = I fordim X dim Y andcombiningidentities2.2.2and2.2.3,weobtaintherststatement.Identities2.2.2 and2.2.3,togetherwiththefactthat X H Y y X H Y = I fordim X dim Y implythesecondstatement. Remark2.2.18 FromTheorem2.2.17,forthecase dim X dim Y wehave P X ? P X P Y y = P X P Y y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X P X P Y y = P X P Y y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X : Sincetheanglesaresymmetric,usingthesecondstatementinTheorem2.2.17we have P X ? P Y ? y P Y = P X ? P Y ? y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y ? : Ontheotherhand,forthecase dim X dim Y weobtain P X P Y y P Y ? = P X P Y y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y and P X ? P Y ? y P X = P X ? P Y ? y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X ? : 25

PAGE 35

Tosumup,thefollowingformulasfor T inTable2.3canalsobeusedinTheorem2.2.14.Analternativeprooffor T = P X P Y y )]TJ/F18 11.9552 Tf 11.462 0 Td [(P Y isprovidedbyDrmacin [20] fortheparticularcase dim X =dim Y Table2.3: Dierentformulasfor T with X ; Y <= 2:dim X dim Y left; dim X dim Y right. P X P Y y )]TJ/F18 11.9552 Tf 11.956 0 Td [(P X P X P Y y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y P X ? P Y ? y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y ? P X ? P Y ? y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X ? Remark2.2.19 Finally,wenotethatourchoiceofthespace H = C n mayappear naturaltothereaderfamiliarwiththematrixtheory,butinfactissomewhatmisleading.Theprincipalanglesandthecorrespondingprincipalvectorsbetweenthe subspaces XH and YH areexactlythesameasthosebetweenthesubspaces XX + Y and YX + Y ,i.e.,wecanreducethespace H tothespace X + YH withoutchangingPABS. Thisreductionchangesthedenitionofthesubspaces X ? and Y ? andthusof thematrices X ? and Y ? thatspanthecolumnspaces X ? and Y ? .Allourstatements thatusethesubspaces X ? and Y ? orthematrices X ? and Y ? thereforehavetheir newanalogs,ifthespace X + Y substitutesfor H 2.3Conclusions Inthischapter,wehavebrieyreviewedtheconceptandsomeimportantpropertiesofPABS.Wehaveconstructedexplicitmatricessuchthattheirsingularvalues areequaltothetangentsofPABS.Moreover,weformthesematricesusingseveral dierentapproaches,i.e.,orthonormalbasesforsubspaces,non-orthonormalbasesfor subspaces,andprojectors.Weusesomeofthestatementsinthefollowingchapters. 26

PAGE 36

3.NewapproachestocomputePABS 2 ComputingPABSisoneofthemostimportantproblemsinnumericallinear algebra.Let X 2 C n p and Y 2 C n q befullcolumnrankmatricesandlet X = R X and Y = R Y .Inthischapter,weareinterestedinthecase n p and n q Traditionalmethodsforcomputingprincipalanglesandprincipalvectorsarebased oncomputationsofeigenpairsoftheeigenvalueproblems.ThecosinesofPABSare eigenvaluesofthegeneralizedeigenvalueproblemsee[9,30] 2 6 4 OX H Y Y H XO 3 7 5 2 6 4 ~ u ~ v 3 7 5 = 2 6 4 X H XO OY H Y 3 7 5 2 6 4 ~ u ~ v 3 7 5 : .0.1 Thegeneralizedeigenvalueproblemaboveisequivalenttotheeigenvalueproblems forapairofmatrices X H X )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 X H Y Y H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y H X X H X )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 ^ u = 2 ^ u; Y H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 Y H X X H X )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 X H Y Y H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 ^ v = 2 ^ v; whichcanbefoundinmostmultivariatestatisticsbooks,e.g.,see[39]. Letthecolumnsofmatrices X and Y form orthonormal basesforthesubspaces X and Y ,correspondingly.Thematrixontherightsideof3.0.1isidentity,since X H X = I and Y H Y = I: Wenoticethatsolvingtheeigenvalueproblemin3.0.1is thesameassolvingtheSVDof X H Y .ComputingPABSbasedontheSVDhasbeen establishedbyBjorckandGolub[9].Thecosinesofprincipalanglesandprincipal vectorsareobtainedfromtheSVDof X H Y ,whichisalsocalledthecosine-based algorithm.Drmac[21]showsthisalgorithmtobemixedstableandrecommendsQR factorizationswiththecompletepivotingforcomputing X and Y fromoriginaldata. Thecosine-basedalgorithminprinciplecannotprovideaccurateresultsforsmall anglesincomputerarithmetic,i.e.,anglessmallerthan10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(8 arecomputedaszero 2 Kahanhasattractedourattentiontothehalfanglesapproachandgivenushisnote[42],via Parlett.WethankKahanandParlettfortheirhelp. 27

PAGE 37

indoubleprecisionarithmeticwhen EPS 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(16 .Thisproblemispointedoutand treatedin[9],whereasine-basedalgorithmforcomputingthesmallprincipalangles isproposed.ThesinesofPABSareobtainedfromtheSVDof I )]TJ/F18 11.9552 Tf 11.721 0 Td [(XX H Y: Forthe samereason,thesine-basedalgorithmhastroublecomputinglarge",i.e.,closeto = 2,anglesinthepresenceofround-oerrors. ThesineandcosinebasedalgorithmscombinedbyKnyazevandArgentati[48], wheresmallandlargeanglesarecomputedseparately,arebeingusedtocurethis problem.Theyusesine-basedalgorithmtocomputetheangleslessthanathreshold e.g., = 4andcosine-basedalgorithmtocomputetheanglesgreaterthanthisthreshold.Cosine-basedanglesareobtainedfromtheSVDof X H Y andsine-basedangles areobtainedfromtheSVDof I )]TJ/F18 11.9552 Tf 12.209 0 Td [(XX H Y .However,suchalgorithmsintroducea newdiculty|accuratecomputationoftheprincipalvectorscorrespondingtoPABS intheneighborhoodofthethresholdbetweenthesmallandlargeangles. Tocomputeallangleswithhighaccuracy,anotherapproachisbycomputingthe sinesof half -PABSmentionedin[9,42].Inthischapter,weproposeseveralnewalgorithmstocomputethesinesofhalf-PABS.Ouralgorithmsprovideaccurateresults forallPABSinonesweepandcanbeimplementedsimplyandmoreeciently,comparedtothesine-cosinealgorithms.Moreover,wealsoproposemethodstocompute allPABSforlargesparsematrices. NumericalsolutionofHermitiangeneralizedeigenvalueproblemsrequirescomputationofPABSingeneralscalarproducts,specically,inan A -basedscalarproduct foraHermitianpositivedenitematrix A ,whichmayonlybeavailablethrougha functionperformingtheproductof A timesavector.Ourimplementationalsoallows computationofPABSingeneralscalarproducts. Theremainderofthischapterisorganizedasfollows.Werstproposeamethod tocomputethesinesandcosinesofhalf-PABSbyoneSVDof[ XY ],whichisproved byusingpropertiesofJordan-Wielandtmatrix.InSection3.2,weprovideamethod 28

PAGE 38

tocomputesinesofhalf-PABSbasedonthegeometricalinsightviewofhalf-PABS. BothmethodsforcomputingPABSdevelopedinSections3.1and3.2arebasedon theorthonormalbasesofapairofsubspaces. Forbettercomputationaleciency,weproposealgorithmsadoptingonlythe R factorsofQRfactorizationswithoutformulatingtheorthonormalbasesofinputmatricesinSection3.3.Furthermore,weprovidealgorithmstocomputeallprincipal angles,notonlyfordensematrices,butalsoforsparse-typestructurematrices.GeneralizationsofthesealgorithmstocomputePABSinthe A -basedscalarproductare presentedinSection3.4.AMATLABimplementationandresultsofnumericaltests areincludedinSections3.5and3.6.Finally,wegiveabriefdiscussionforthenew algorithmsandcomparethesealgorithmswiththesine-cosinebasedalgorithm. 3.1Algebraicapproachestocompute sin )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 and cos )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 Theorem3.1.1 Letthecolumnsofmatrices X 2 C n p and Y 2 C n q ,where q p and p + q n ,formorthonormalbasesforthesubspaces X and Y ,correspondingly. Let L =[ XY ] andlettheSVDof L be ~ U ~ ~ V H ,where ~ isadiagonalmatrixwith diagonalelementsinnonincreasingorder.Then cos 2 = 1 p 2 S q # L andsin # 2 = 1 p 2 S # q L ; where S q # L denotesthevectorofthelargest q singularvaluesofLinnonincreasing orderand S # q L denotesthevectorofthesmallest q singularvaluesofLinnonincreasingorder.Moreover,wehave ~ V = 1 p 2 2 6 4 U 1 p 2 U 2 )]TJ/F24 11.9552 Tf 10.902 3.022 Td [(b U 1 VO b V 3 7 5 ; where U H 1 U 1 = b U H 1 b U 1 = I with U 1 ; b U 1 2 C p q ; V H V = b V H b V = I with V; b V 2 C q q ; andthematrix [ U 1 ;U 2 ] isunitary.Then,theprincipalvectorsof associatedwith thispairofsubspacesaregivenbythecolumnsof V X = XU 1 and V Y = YV: 29

PAGE 39

Alternatively,theprincipalvectorsof # associatedwiththispairofsubspacesare givenbythecolumnsof V X = X b U 1 and V Y = Y b V: Proof: LettheSVDofthematrix X H Y be U V H ,where U and V areunitaryand isadiagonalmatrixwiththerealdiagonalelements s 1 X H Y ;:::;s q X H Y arranged innonincreasingorder.Dueto[4,35,82],theeigenvaluesofthe p + q p + q Hermitianmatrix 2 6 4 OX H Y Y H XO 3 7 5 .1.1 are s 1 X H Y ;:::; s q X H Y togetherwith p )]TJ/F18 11.9552 Tf 12.739 0 Td [(q zeros.Thematrixin3.1.1 issometimescalledtheJordan-Wielandtmatrixcorrespondingto X H Y .Since X H Y V = U and Y H X U = V H ,itfollowsthat X H Yv k = s k X H Y u k ; .1.2 Y H Xu k = s k X H Y v k ; .1.3 for k =1 ;:::;q .Thus, 2 6 4 u k v k 3 7 5 and 2 6 4 )]TJ/F18 11.9552 Tf 9.299 0 Td [(u k v k 3 7 5 areeigenvectorsof3.1.1corresponding totheeigenvalues s k and )]TJ/F18 11.9552 Tf 9.298 0 Td [(s k ,correspondingly.Furthersee[35,p.418],wepresent theeigenvaluedecompositionofmatrixin3.1.1as 2 6 4 OX H Y Y H XO 3 7 5 = Q diag s 1 ;s 2 ;:::;s q ; 0 ;:::; 0 ; )]TJ/F18 11.9552 Tf 9.298 0 Td [(s q ; )]TJ/F18 11.9552 Tf 9.299 0 Td [(s q )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ;:::; )]TJ/F18 11.9552 Tf 9.298 0 Td [(s 1 Q H ; where Q 2 C p + q p + q isaunitarymatrix.Lettherst q columnsof U bedenoted as U 1 ,i.e., U 1 =[ u 1 ;u 2 ;:::;u q ].Let U =[ U 1 U 2 ]and V =[ v 1 ;v 2 ;:::;v q ].Let b U 1 =[ u q ;u q )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ;:::;u 1 ]and b V =[ v q ;v q )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 ;:::;v 1 ].Thematrix Q canbewrittenas Q = 1 p 2 2 6 4 U 1 p 2 U 2 )]TJ/F24 11.9552 Tf 10.902 3.022 Td [(b U 1 VO b V 3 7 5 : .1.4 30

PAGE 40

Consequently,theeigenvaluesof I + 2 6 4 OX H Y Y H XO 3 7 5 .1.5 are[1+ s 1 ; 1+ s 2 ;:::; 1+ s q ; 1 ;:::; 1 ; 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(s q ; 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(s q )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 ;:::; 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(s 1 ]andthecolumnsof Q in3.1.4aretheeigenvectorsof3.1.5. Furthermore,wecanrewrite3.1.5asfollows: I + 2 6 4 OX H Y Y H XO 3 7 5 =[ XY ] H [ XY ] : .1.6 LettheSVDofthematrix[ XY ]be ~ U ~ ~ V H ,where ~ isadiagonalmatrixandthe matrices ~ U and ~ V areunitary.Sincecos 2 = 2=+cos = 2andsin 2 = 2= )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos = 2,wehave ~ =diag p 2cos )]TJ/F15 11.9552 Tf 5.479 -9.684 Td [( = 2 ; 1 ;:::; 1 ; p 2sin )]TJ/F15 11.9552 Tf 5.48 -9.684 Td [( # = 2 : Moreover, ~ V = Q .ByTheorem2.1.2,theprincipalvectorsof associatedwith thispairofsubspacesaregivenbythecolumnsof V X = XU 1 and V Y = YV: Also, theprincipalvectorsof # associatedwiththispairofsubspacesaregivenbythe columnsof V X = X b U 1 and V Y = Y b V: Remark3.1.2 InTheorem3.1.1,weobtainPABSforthecase p + q n .Ina similarway,wecancomputePABSfor p + q n .Suppose p + q n ,then dim XY =dim X +dim Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(dim X[Y dim X +dim Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(n = p + q )]TJ/F18 11.9552 Tf 11.955 0 Td [(n: Therefore,thereareatleast p + q )]TJ/F18 11.9552 Tf 12.44 0 Td [(n zeroanglesbetweenthesubspaces X and Y Furthermore,weobtainthesinesandcosinesofhalf-PABSbythesingularvaluesof 31

PAGE 41

L asfollows: cos 2 = 1 p 2 S q # L andsin # 2 = 1 p 2 [ S # n )]TJ/F19 7.9701 Tf 6.587 0 Td [(p L ; 0 ;:::; 0] : Thereare p + q )]TJ/F18 11.9552 Tf 11.069 0 Td [(n zerosafter S # n )]TJ/F19 7.9701 Tf 6.586 0 Td [(p L in 1 = p 2[ S # n )]TJ/F19 7.9701 Tf 6.587 0 Td [(p L ; 0 ;:::; 0] .Wealsonotethat theprincipalvectorsarenotuniqueif s k L = s k +1 L Theoreticallyspeaking,PABScanbeobtainedfromtheeigenvaluesandeigenvectorsofequality3.1.5.However,thisresultdoesnotappeartobeofpractical interestincomputingPABS.Calculatingthesineorcosinesquaredgivesinaccurateresultsforsmallanglesinexactarithmetic.Moreover,weshouldavoidusing X H Y .Wecanachieveabetterunderstandingofthisfromthefollowingexample. Let x =[ 10 ] H and y =[ 11e )]TJ/F15 11.9552 Tf 9.298 0 Td [(10 ] H .Theyarebothnormalizedindouble-precision andequality3.1.5resultsinazeroanglenomatterwhat,since x H y =1. Figure3.1: Sinesandcosinesofhalf-angles. ToillustrateTheorem3.1.1,weagaintake x =[ 10 ] H and y =[ 11e )]TJ/F15 11.9552 Tf 9.298 0 Td [(10 ] H as anexample.Wehave L = 2 6 4 11 01e )]TJ/F15 11.9552 Tf 9.298 0 Td [(10 3 7 5 .Computingthesingularvaluesof L ,weget sin = 2=5 e )]TJ/F15 11.9552 Tf 12.773 0 Td [(11andcos = 2=0.Obviously,weloseaccuracyincos = 2in thiscase.AsshowninFigure3.1,cos = 2losesaccuracyforsmallanglesindouble32

PAGE 42

precision.Sincewealwayshave j sin 0 = 2 jj cos 0 = 2 j for 2 [0 ;= 2],thebetter choiceistousesin = 2tocalculatePABS. CombiningTheorem3.1.1andRemark3.1.2,weformulateanalgorithmtocomputeallPABS.Thisalgorithmprovidesaccuratecomputationforsmallandlarge angles.TheAlgorithm3.1isbasedoncomputingsin = 2whichisobtainedfromthe smallestmin p;q singularvaluesof L .Moreover,theprincipalvectorsareobtained fromtherightsingularvectorsof L correspondingtothesmallestmin p;q singular valuesforthecase p + q n .Forcompleteness,wealsoprovidethealgorithmforthe case p + q n Table3.1: Algorithm3.1. Algorithm3.1 Input :matrices X 1 and Y 1 withthesamenumberofrows. 1.Computeorthonormalbases X 2 C n p and Y 2 C n q of R X 1 and R Y 1 ,correspondingly. 2.Set L =[ XY ],computetheSVDof L : h ~ U; ~ ; ~ V i =svd L; 0. 3.If p + q n ,takethesmallest m =min p;q singularvaluesof L denotedas s 1 ;:::;s m If p + q n ,takethesmallest n )]TJ/F8 9.9626 Tf 9.963 0 Td [(max p;q singularvaluesof L andadd p + q )]TJ/F11 9.9626 Tf 9.962 0 Td [(n zeros singularvaluesanddenoteas s 1 ;:::;s m 4.Computetheprincipalanglesinnonincreasingorderfor k =1 ;:::;m : k =2arcsin )]TJ/F11 9.9626 Tf 4.566 -8.07 Td [(s k = p 2 : 5.Computecorrespondingprincipalcoecients: b U 1 = )]TJ 7.748 8.241 Td [(p 2 ~ V : p;p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(m +1: p + q and b V 1 = p 2 ~ V p +1: p + q;p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(m +1: p + q : 6.Computetheprincipalvectors: V X = X b U 1 and V Y = Y b V 1 : Output :theprincipalangles 1 ;:::; m betweenthecolumn-spaceofmatrices X 1 and Y 1 ,the correspondingprincipalvectors V X and V Y ,andtheprincipalcoecients b U 1 and b V 1 Remark3.1.3 IntherststepofAlgorithm3.1,weusetheMATLABfunction ORTHUPDATE.m seeAppendixA tocomputetheorthonormalbasesoftheinput matrices.ThefunctionORTHUPDATE.misbackwardcompatiblewiththebuilt-in 33

PAGE 43

MATLABfunctionORTH.m.OurfunctionORTHUPDATE.mprovidesthreechoices tocomputeanorthonormalbasiswithoptionopts.fastbeing0,1and2.TheorthogonalizationisperformedusingtheSVDwiththeoptionopts.fastbeing0,which performsthesameasORTH.m.Withopts.fast =1 ,thefunctionORTHUPDATE.m performsQRfactorizationwithpivoting.ThefunctionORTHUPDATE.monlyworks forafull-columnrankmatrixifopts.fast =2 .Thealgorithmisdevelopedasfollows: suppose X 1 isaninputmatrix,thenwecomputeanuppertriangularmatrix R from aCholeskyfactorization,suchthat R =chol X H 1 X 1 ,thenwetaketheorthonormal basisas X 1 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Letusconsiderthecostofthesethreealgorithmsasmentionedabovetocompute theorthonormalbasisoftheinputmatrix.Ingeneral,therearetwostepstocomputetheSVD,e.g.,see [8,12,19] .TherststepcanbedoneusingHouseholder reectionstoreduce X 1 toabidiagonalmatrix.ThesecondstepreducesthebidiagonalmatrixformtodiagonalformiterativelybyavariantoftheQRalgorithm. Accordingto [8,p.90] [12] ,and [19,Chap.11] ,thetotalcostofthecompleteSVDis about + a np 2 +11 = 3 p 3 ops,where a =4 ifstandardgivenstransformationsare usedinQRalgorithm, a =2 iffastgiventransformationsareused.Forthedetailed transformations,thereaderisreferredto [8,Section2.6] Generally,thecostofQRfactorizationbasedonHouseholderreections see [77] isabout 2 np 2 )]TJ/F15 11.9552 Tf 9.56 0 Td [(2 = 3 p 3 opsforan n by p matrixwith n p .TheCholeskyfactorization forthematrix X H 1 X 1 isabout 1 = 3 p 3 ops.Amongthesethreechoicesforcomputing theorthonormalbasis,theopts.fastbeing2runsfastest,butistheleastreliable.The mostrobustandslowestoneisbasedontheSVD-basedalgorithm,sincethecost ofSVDisabouttwotimesthecostofQR.Fordetailedinformationconcerningthe stabilityofthesethreefactorizations,see [77] Inouractualcode,weusetheeconomy-sizefactorizationQRandSVDinthe functionORTHUPDATE.m.WedonotknowtheexactnumberofopsinMATLAB. 34

PAGE 44

In [57] ,itstatesthatoating-pointoperationsarenolongerthedominantfactorin executionspeedwithmoderncomputerarchitectures.Memoryreferencesandcache usagearethemostimportant. Remark3.1.4 GolubandZha [30] pointoutthatPABSaresensitivetoperturbation forill-conditioned X 1 and Y 1 .Apropercolumnscalingof X 1 and Y 1 canimprove conditionnumbersofbothmatricesandreducetheerrorforPABS,sincecolumn scalingdoesnotaectPABS.NumericaltestsbyKnyazevandArgentati [48] showthat propercolumnscalingimprovesaccuracywhentheorthonormalbasesarecomputedby usingtheSVD.However,thereisnoneedtoscalethecolumnswhentheorthonormal basesareobtainedbytheQRfactorizationwithcompletepivoting,e.g.,see [21,48] Also,itispointedoutin [21] thatonegoodwaytocomputetheorthogonalbases istousetheunitlowertrapezoidalmatrices L X and L Y fromtheLUfactorizations of X 1 and Y 1 withpivoting,since R X 1 = R L X and R Y 1 = R L Y Remark3.1.5 Wenotethatthematrixsizeof L inAlgorithm3.1isnby p + q .If n islarge,inordertoreducethecostoftheSVDof L ,wecanreducethematrix L toa p + q by p + q triangularmatrix R byQRfactorization,suchthat L = QR WecanobtainPABSbytheSVDof R Inthefollowing,wewillderiveadierentformulatocomputethesinesand cosinesofhalf-PABS. Lemma3.1.6 Letthematrices X and Y bedescribedasinTheorem3.1.1.Letthe columnsofmatrix Q X + Y formanorthonormalbasisforthecolumn-spaceof L ,where L =[ XY ] .Let ^ L = Q H X + Y L .Then [ S ^ L ; 0 ;:::; 0]= S L ; whereextra0smayneedtobeaddedontheleftsidetomatchthesize.Inparticular, 35

PAGE 45

ifwetake Q X + Y =[ X ^ X ? ] ,then ^ L = 2 6 4 IX H Y O ^ X H ? Y 3 7 5 : Proof: Bydirectcomputation,wehave ^ L H ^ L = L H Q X + Y Q H X + Y L = L H P X + Y L = L H L; where P X + Y isanorthogonalprojectoronthecolumn-spaceof L .So,thepositive singularvaluesof ^ L coincidewiththepositivesingularvaluesof L .Ifwetake Q X + Y = [ X ^ X ? ],then ^ L =[ X ^ X ? ] H [ XY ]= 2 6 4 IX H Y O ^ X H ? Y 3 7 5 ; asexpected. Remark3.1.7 Letthematrix X 1 2 C n p havefullcolumnrank.Let Y 1 2 C n q and let [ Q 1 Q 2 ] beanorthonormalmatrixobtainedfromthereducedQRfactorizationof [ X 1 Y 1 ] .Wetake X = Q 1 ,thuswehave ^ L = 2 6 4 IX H Y OQ H 2 Y 3 7 5 : Thesizeof ^ L issmall,since maxsize ^ L is p + q: However,this ^ L isnotattractive tobeusedtocomputePABSinpractice.Insomesituations,wehavenoinformation abouttherankoftheinputmatrix X 1 .SupposeweusetheQRfactorizationtoobtain thefull-columnrankmatrix ^ X 1 fromthematrix X 1 .Inthismethod,togetPABSwe havetodotheQRfactorizationsfor X 1 Y 1 [ ^ X 1 Y 1 ] andplusoneSVDof ^ L .The computationalcostissimilartothemethoddescribedinRemark3.1.5.Moreover, applyingtheorthonormalmatrix [ Q 1 Q 2 ] ,whichisobtainedfromtheQRfactorization 36

PAGE 46

[ ^ X 1 Y 1 ] ,mayincreasetheinaccuracyforsmallangles.Especiallyif [ ^ X 1 Y 1 ] isillconditioned,wemayobtain [ Q 1 Q 2 ] inaccurately. LetushighlightthatupdatingtheSVDof L byappendingaroworcolumnand deletingaroworcolumniswellinvestigated,e.g.,see[8,Section3.4].Inthelast lemmaofthissection,wediscussbrieytherelationshipbetweenthesinesandthe cosinesofhalfoftheprincipalanglesbetween R X 1 and R Y 1 andthoseof R X 1 and R [ Y 1 u ],where X 1 2 C n p and Y 1 2 C n q .Forconvenience,let p + q
PAGE 47

Inparticular,if x and y in R n aresuchthat x H y> 0seeFigure3.2,then 2sin 1 2 x;y = k x )]TJ/F18 11.9552 Tf 11.955 0 Td [(y k : Similarly,theHausdormetricbetweentwo k -dimensionalsubspaces X and Y ,e.g., Figure3.2: Sineofhalf-anglebetween x and y in R n see[66],isdenedas 2sin 1 2 1 X ; Y ;:::; 2sin 1 2 k X ; Y =inf fjjj X ^ Q )]TJ/F18 11.9552 Tf 11.955 0 Td [(Y jjj : ^ Q 2U k g ; .2.2 where U k isthefamilyofunitarymatricesofsize k andthecolumnsofmatrices X and Y formorthonormalbasesforthesubspaces X and Y ,correspondingly.The function isasymmetricgaugefunction[5]and jjjjjj denotesthecorresponding unitarilyinvariantnorm[55].See[66]forfurtherinformationconcerningthesines ofhalf-PABSrelatedtotheHausdormetricbetweentwosubspaces.Inparticular,takingthesameminimizationproblemforFrobeniusnormin3.2.2,wehave see[73] min ^ Q 2U k k X ^ Q )]TJ/F18 11.9552 Tf 11.955 0 Td [(Y k F = 2sin 1 2 X ; Y F : 38

PAGE 48

ThisisaspecialcaseoftheorthogonalProcrustesproblem.Thesolution ^ Q ofthe Procrustesproblemcanbeobtainedbyusingtheorthogonalpolarfactorof X H Y [28]: lettheSVDof X H Y be U V H ,thentheoptimal ^ Q isgivenby ^ Q = UV H .Weuse thisconstructionof ^ Q todemonstratethat S X ^ Q )]TJ/F18 11.9552 Tf 11.318 0 Td [(Y =2sin X ; Y = 2,whichis mentionedinKahan'slecturenote[42]. Theorem3.2.1 [42] Letthecolumnsofmatrices X 2 C n p and Y 2 C n q q p formorthonormalbasesforthesubspaces X and Y ,correspondingly.Let L = X ^ Q )]TJ/F18 11.9552 Tf -422.702 -23.98 Td [(Y and B = X H Y ,where ^ Q H ^ Q = I and ^ Q istheunitaryfactorfromthepolar decompositionof B .Then sin # 2 = 1 2 S # L : .2.3 Proof: Let B = U [ CO ] H V H betheSVDof B ,where U and V areunitary matrices.Thematrix C isadiagonalmatrixwithcos k onitsdiagonal,where k for k =1 ;:::;q aretheprincipalanglesbetweenthesubspaces X and Y .Let ^ Q = U [ IO ] H V H : Wehave L H L = X ^ Q )]TJ/F18 11.9552 Tf 11.955 0 Td [(Y H X ^ Q )]TJ/F18 11.9552 Tf 11.955 0 Td [(Y =2 I )]TJ/F15 11.9552 Tf 14.64 3.022 Td [(^ Q H X H Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(Y H X ^ Q =2 I )]TJ/F15 11.9552 Tf 11.956 0 Td [(2 VCV H =2 V I )]TJ/F18 11.9552 Tf 11.956 0 Td [(C V H ; where I )]TJ/F18 11.9552 Tf 12.297 0 Td [(C isadiagonalmatrixwith1 )]TJ/F15 11.9552 Tf 12.297 0 Td [(cos k for k =1 ;:::;q onitsdiagonal. Notethat ^ Q H X H Y = V [ IO ] U H U 2 6 4 C O 3 7 5 V H = VCV H : Therefore,thesingularvaluesof L are p 2 )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos k for k =1 ;:::;q: Since sin 2 = 2= )]TJ/F15 11.9552 Tf 11.956 0 Td [(cos = 2,weobtainsin )]TJ/F15 11.9552 Tf 5.479 -9.683 Td [( # = 2 = S # L = 2 : 39

PAGE 49

Remark3.2.2 Wecanrewrite L as XU [ IO ] H )]TJ/F18 11.9552 Tf 12.978 0 Td [(YV .Moreover,ifweusethe reducedSVDof B ,suchthat B = U r CV H r with U r 2 C p q and V r 2 C q q ; then L can bewrittenas XU r V H r )]TJ/F18 11.9552 Tf 11.955 0 Td [(Y or XU r )]TJ/F18 11.9552 Tf 11.956 0 Td [(YV r : Theformulation XU r )]TJ/F18 11.9552 Tf 11.788 0 Td [(YV r tocomputethesinesofhalf-PABSisalsomentionedin [9] ,butnodetailedinformationprovided. Remark3.2.3 FromtheproofofTheorem3.2.1,wealsocanget cos = 2= S X ^ Q + Y = 2 : Moreover,ifweusethereducedSVDof B ,suchthat B = U r CV H r ,wehave cos = 2= S XU r + YV r = 2 : Theoreticallyspeaking,theprincipalvectorsassociatedwiththepairofthesubspaces X and Y arethecolumnsofthematrices XU r and YV r ,correspondingly,where U r and V r areobtainedfromthereducedSVDof B .However,thecosine-basedalgorithmfailstoaccuratelycomputethesmallprincipalanglesandthecorresponding principalvectorsindoubleprecision,whichispointedoutin[48].Forexample,we havesmallangles10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(9 ; 10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(10 ; 10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(11 ,and10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(12 ,whichwillproducemultiplesingular values1withmultiplicity4indoubleprecisionbasedonthecosine-basedalgorithm. Infact,therearefoursmalldistinctangleswithexactarithmetic.Thisimpliesthat theprincipalvectorscorrespondingtosmallprincipalanglesmaynotbecomputed accuratelyindoubleprecisionusingtheSVDof B Toobtaintheprincipalvectorscorrespondingtosmallanglesinhighaccuracy, onewayisbyusingtheSVDof L inTheorem3.2.1.LetSVDof L be ^ U ^ ^ V H .The principalvectorsonthesubspace Y arethecolumnsof V Y ,where V Y = Y ^ V andthe principalvectors V X onthesubspace X canbeobtainedby V X = X ^ Q ^ V 40

PAGE 50

Theformulationforprincipalanglesonthesubspaces X maynotbeobvious.In ouractualcode,wereformulate L inTheorem3.2.1as XU r )]TJ/F18 11.9552 Tf 10.24 0 Td [(YV r inRemark3.2.2,an alternativeapproachtogettheprincipalvectorsassociatedwithallanglesisbythe SVDof XU r )]TJ/F18 11.9552 Tf 10.008 0 Td [(YV r .LettheSVDof XU r )]TJ/F18 11.9552 Tf 10.007 0 Td [(YV r be U 1 1 V H 1 ,thentheprincipalvectors associatedwiththepairoftwosubspacesarethecolumnsof XU r V 1 and YV r V 1 .Our numericaltestssupportthisresult. WeformulateAlgorithm3.2tocomputingallprincipalanglesandprincipalvectorsbyusing L = XU r )]TJ/F18 11.9552 Tf 11.955 0 Td [(YV r asinremark3.2.2. Table3.2: Algorithm3.2. Algorithm3.2 Input :matrices X 1 and Y 1 withthesamenumberofrows. 1.Computeorthonormalbases X and Y ofcolumnspacesof X 1 and Y 1 ,correspondingly. 2.ComputetheSVDof B = X H Y : 8 < : [ U r ;C;V r ]=svd B; 0ifrank X rank Y ; [ V r ;C;U r ]=svd B H ; 0otherwise : 3.Let L = XU r )]TJ/F11 9.9626 Tf 9.963 0 Td [(YV r andcomputethesingularvalues s k of L for k =1 ;:::; min p;q 4.Computetheprincipalangles: k =2arcsin s k = 2 : 5.Computetherightsingularvectors V 1 ofL,thentheprincipalcoecientsare ^ U 1 = U r V 1 and ^ V 1 = V r V 1 : 6.Computetheprincipalvectors: V X = X ^ U 1 and V Y = Y ^ V 1 Output :theprincipalangles 1 ;:::; min p;q betweenthecolumn-spaceof X 1 and Y 1 matrices, thecorrespondingprincipalvectors V X and V Y ,andtheprincipalcoecients b U 1 and b V 1 41

PAGE 51

3.3Computing sin )]TJ/F16 7.9701 Tf 6.675 -4.976 Td [( 2 and cos )]TJ/F16 7.9701 Tf 6.675 -4.976 Td [( 2 withoutusingorthonormalbases Ouralgorithmsdescribedaboveandthecosine-sinebasedalgorithmcompute PABSaccuratelybyusingtheorthonormalbasesforsubspaces.Iftheinputmatricesarelargesparse,thencomputinganorthogonalbasisforeachofthesubspaces maydestroysparsityofthegivenspaces.Thecosine-sinebasedalgorithmcodedby KnyazevandArgentatiisnotcapableofhandlingthesparse-typestructurematrices availableinMATLAB.Inthissection,weproposealgorithmstocomputethesines ofhalf-PABSbyonlyusingtheuppertriangularmatrices R fromQRfactorizations withoutformingtheorthonormalbasesofinputmatrices X 1 and Y 1 .Moreover,we provideanalgorithmtocomputePABSnotonlyfordensematricesbutalsoforsparse matrices. Theorem3.3.1 Let X 1 2 C n p and Y 1 2 C n q bematricesoffullrankwhere p + q n and q p .Theprincipalanglesbetween R X 1 and R Y 1 canbecomputedas follows. 1.Computethetriangularpart R ofthereducedQRfactorizationofthematrix L 1 =[ X 1 Y 1 ] ,suchthat R = 2 6 4 R 11 R 12 OR 22 3 7 5 ; where R 11 2 C p p R 12 2 C p q ,and R 22 2 C q q 2.Computethetriangularpart R Y ofthereducedQRfactorizationofthematrix Y 1 3.Let L = 2 6 4 IR 12 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y OR 22 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y 3 7 5 .Wehave cos 2 = 1 p 2 S q # L andsin # 2 = 1 p 2 S # q L : 42

PAGE 52

Proof: LetthereducedQRfactorizationof L 1 be [ X 1 Y 1 ]=[ Q 1 Q 2 ] 2 6 4 R 11 R 12 OR 22 3 7 5 ; .3.1 wherethecolumnsofthematrix[ Q 1 Q 2 ] 2 C n p + q areorthonormal.Thus,the matrix X 1 canbewrittenas X 1 = Q 1 R 11 .LetthereducedQRfactorizationof Y 1 be Q Y R Y : Therefore, [ X 1 Y 1 ]=[ Q 1 Q Y ] 2 6 4 R 11 O OR Y 3 7 5 : .3.2 Combining3.3.1and3.3.2,weobtain [ Q 1 Q Y ]=[ Q 1 Q 2 ] 2 6 4 R 11 R 12 OR 22 3 7 5 2 6 4 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 11 O OR )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y 3 7 5 .3.3 =[ Q 1 Q 2 ] 2 6 4 IR 12 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y OR 22 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y 3 7 5 : .3.4 Sincethesingularvaluesofamatrixareinvariantunderorthogonaltransformations, wehave S [ Q 1 Q Y ] = S 0 B @ 2 6 4 IR 12 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y OR 22 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y 3 7 5 1 C A : AccordingtoTheorem3.1.1,weobtaintheresult. Remark3.3.2 LettheSVDof L inTheorem3.3.1be ~ U ~ ~ V H andlet b U = p 2 ~ V : p;q +1: p + q and b V 1 = )]TJ 9.298 10.473 Td [(p 2 ~ V p +1: p + q;q +1: p + q : Thentheprincipalvectorsof # associatedwiththispairofsubspacesaregivenby thecolumnsof V X = X 1 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 11 b U and V Y = Y 1 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y b V 1 ; correspondingly. Remark3.3.3 Notethatthematrixsizeof L inTheorem3.3.1is p + q by p + q ThecostoftheSVDof L inTheorem3.3.1shouldbelessthanthecostoftheSVD 43

PAGE 53

of L inTheorem3.1.1.ThemethoddiscussedinTheorem3.3.1maybeanecient alternativetocomputePABS. Remark3.3.4 The R factoriseasytoupdatewheneveracolumnorrowisappended tothematrix L 1 ,see [8,Section3.2] Theorem3.3.5 UndertheassumptionsofTheorem3.3.1,wehave cos = S # R 12 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y .3.5 sin # = S # R 22 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y : .3.6 Proof: From3.3.1and3.3.2,weobtain Y 1 = Q 1 R 12 + Q 2 R 22 = Q Y R Y : Bymultiplying R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y onbothsidesofaboveequality,weget Q 1 R 12 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y + Q 2 R 22 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y = Q Y : .3.7 Bymultiplying Q H 1 onbothsidesof3.3.7,wehave R 12 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y = Q H 1 Q Y : Notethat Q H 1 Q 2 =0.FromTheorem2.1.2ofPABS,weobtainequality3.3.5. Similarly,bymultiplying Q H 2 onbothsidesof3.3.7,wehave R 22 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y = Q H 2 Q Y : Since R [ X 1 Y 1 ] R [ Q 1 Q 2 ],fromProperty2.1.3wegetequality3.3.6. Remark3.3.6 InTheorems3.3.1and3.3.5,thematrix R Y isobtainedfromtheQR factorizationof Y 1 .Infact,wecouldget R Y fromtheQRfactorizationof 2 6 4 R 12 R 22 3 7 5 LetthereducedQRfactorization 2 6 4 R 12 R 22 3 7 5 be ^ Q Y ^ R Y .Then, Y 1 =[ Q 1 Q 2 ] ^ Q Y ^ R Y : 44

PAGE 54

Moreover,let B =[ Q 1 Q 2 ] ^ Q Y .Wehave B H B = I .Hence, ^ R Y isthetriangular matrixoftheQRfactorizationof Y 1 .Analternativemethodtocomputethecosines ofPABSusingLQfactorizationcanbefoundin [17] Remark3.3.7 IntheHouseholderQRfactorization,weapplyasuccessionofelementaryunitarymatrices Q i ontheleftoftheinitialmatrix L 1 = L 1 =[ X 1 Y 1 ] ; so thattheupper-triangularmatrix R isobtained.Let Q p denotethe p th unitarymatrix. Thematrix L 1 = L 1 after p stepsis Q p Q p )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 :::Q 1 L 1 = 2 6 4 R 11 R 12 0 L p 1 3 7 5 ; where [ R 11 R 12 ] containstherst p rowsofthenalupper-triangularmatrix R .Let ^ L = 2 6 4 IR 12 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y 0 L p 1 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y 3 7 5 where R Y istheupper-triangularmatrixobtainedfromtheQR factorizationof Y 1 or 2 6 4 R 12 L p 1 3 7 5 : Thenwehave 1. cos 2 = 1 p 2 S q # ^ L andsin # 2 = 1 p 2 S # q ^ L : 2. cos = S # R 12 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y andsin # = S # L p 1 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y Tosummarize,weformulateAlgorithm3.3tocomputeallPABSwithdenseor sparsematrices.Thisalgorithmprovidesaccuratecomputationforsmallandlarge angles. 45

PAGE 55

Table3.3: Algorithm3.3. Algorithm3.3 Input :matrices X 1 2 C n p and Y 1 2 C n q withfullcolumnrank. 1.Computethetriangularpart R oftheQRfactorizationof L 1 =[ X 1 Y 1 ],e.g., R = 8 < : qr L 1 ; 0if X 1 and Y 1 aresparse, triuqr L 1 ; 0otherwise. 2.Compute R 12 = R : p;p +1: p + q and R 22 = R p +1: end;p +1: p + q 3.Computethetriangularpart R Y oftheQRfactorizationof 2 4 R 12 R 22 3 5 4.Compute L = 8 > > > > > > > < > > > > > > > : 2 4 speye p R 12 =R Y sparse a )]TJ/F11 9.9626 Tf 9.962 0 Td [(p;p R 22 =R Y 3 5 if X 1 and Y 1 aresparse, 2 4 eye p R 12 =R Y zeros a )]TJ/F11 9.9626 Tf 9.963 0 Td [(p;p R 22 =R Y 3 5 otherwise, where a =minsize L 1 : 5.If p + q n ,computethesmallest m =min p;q singularvalues s 1 ;:::;s m of L and corresponding m rightsingularvectors ^ V .Theprincipalanglesarecomputedas k =2arcsin )]TJ/F11 9.9626 Tf 4.566 -8.07 Td [(s k = p 2 for k =1 ;:::;m: 6.If p + q n ,computethesmallest n )]TJ/F8 9.9626 Tf 9.963 0 Td [(max p;q singularvalues s i of L H andcorresponding n )]TJ/F8 9.9626 Tf 9.962 0 Td [(max p;q right-singularvectors ^ V 1 .Moreover,thereare p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(n zerossingularvalues andthe p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(n right-singularvectors ^ V 2 correspondingtothelargest p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(n singular valuesarecomputed.Let ^ V =[ ^ V 1 ^ V 2 ] : Theprincipalanglesarecomputedas k = 8 < : 2arcsin )]TJ/F11 9.9626 Tf 4.566 -8.07 Td [(s k = p 2 for k =1 ;:::;n )]TJ/F8 9.9626 Tf 9.963 0 Td [(max p;q : 0for k = n )]TJ/F11 9.9626 Tf 9.962 0 Td [(q +1 ;:::;m: 7.Computetheprincipalvectors: V X = p 2 X 1 =R : p; 1: p ^ V : p; :and V Y = p 2 Y 1 =R Y ^ V p +1: p + q; : : Output :theprincipalangles 1 ;:::; m andthecorrespondingprincipalvectors V X and V Y Remark3.3.8 IntherststepofAlgorithm3.3,weapplytheMATLABfunction qrtocomputetheuppertriangularmatrixforboththedense-typeandsparse-type matrix L 1 .SuiteSparseQRdevelopedbyDavis [16] inMATLABR2009a,hasreplacedthepriorsparseQRfactorizationinMATLABprovidedbyGilbert [26] ,where 46

PAGE 56

SuiteSparseQRisamultithreadedmultifrontalsparseQRfactorizationmethod. Instep3,fordensematrices X 1 and Y 1 itischeapertocomputethetriangularpart R Y fromtheQRfactorizationof 2 6 4 R 12 R 22 3 7 5 thantheQRfactorizationof Y 1 .However, factorizingasparsematriximpliesll-ininthefactors.Thus, 2 6 4 R 12 R 22 3 7 5 maynotbeas sparseas Y 1 .So,instep3,wecancompute R Y fromthesparsematrix Y 1 .Agood columnorderingcanhelplimittheworst-casell-in.Asuitablecolumnorderingof A isusuallyobtainedbytheminimaldegreeanalysison A H A .Foramoredetailed accountofndingagoodcolumnordering,thereaderisreferredto [33] Remark3.3.9 Instep5,ifwecomputeafewprincipalanglesforthelargesparse matrix L ,itisbettertousethefunction svdsL than svdfullL .However,tocomputeallsingularvalues svd willusuallyperformbetterthan svds inMATLAB. SeeSVDSMATLABfunctionreferencefordetailedinformation.Inouractualcode, weuse svdfullL insteadof svdsL tocomputeallPABSofsparsematrices. Letusnallypresentnewsimpleproofsforsomepropertiesoforthogonalprojectorsonthesubspaces X and Y Corollary3.3.10 [23] Let P X and P Y beorthogonalprojectorsonthesubspaces X and Y ,correspondingly.Wehave S # ; 1 [ ; 2 P X + P Y = h 1+cos ;= 2 X ; Y ; 1 )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos # ;= 2 X ; Y i ; where S ; 1 [ ; 2 denotesthevectorofsingularvaluesin ; 1 [ ; 2 Proof: Bydirectcalculation,wehave P X + P Y =[ XY ] 2 6 4 X H Y H 3 7 5 : 47

PAGE 57

LettheSVDof[ XY ]be ~ U ~ ~ V H .Then P X + P Y = ~ U ~ ~ H ~ U H : ApplyingTheorem3.1.1,theresultisobtaineddirectly. Inthefollowingcorollary,wewillpresentthesinesandcosinesofhalf-PABS whicharerelativetotheseorthogonalprojectors. Corollary3.3.11 Let P X and P Y beorthogonalprojectorsonthesubspaces X and Y ,correspondingly.Then, S # ; 1 [ ; p 2 [ P X P Y ]= p 2 cos 1 2 ;= 2 X ; Y ; sin 1 2 # ;= 2 X ; Y : Proof: Wehave P X + P Y =[ P X P Y ][ P X P Y ] H : Thus,thesingularvaluesof[ P X P Y ]arethesquarerootoftheeigenvaluesof P X + P Y Sincetheeigenvaluesof P X + P Y arethesameasthesingularvaluesof P X + P Y ,using Corollary3.3.10completestheproof. Remark3.3.12 Notethatwecanwrite [ P X P Y ]=[ XY ] 2 6 4 X H 0 0 Y H 3 7 5 : Fromthis representation,itisclearthattheprincipalvectorscanbeobtaineddirectlyfromthe rightsingularvectorsof [ P X P Y ] : 48

PAGE 58

3.4Computing sin )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 and cos )]TJ/F16 7.9701 Tf 6.675 -4.977 Td [( 2 inthe A -basedscalarproduct NumericalsolutionofHermitiangeneralizedeigenvalueproblemsrequirescomputationofPABSingeneralscalarproducts,specically,inan A -basedscalarproduct foraHermitianpositivedeniteHPDmatrix A .Inthissection,wewilldevelop methodstocomputePABSinthe A -basedscalarproductbasedonsinesofhalf-PABS inthe A -basedscalarproduct.Toproceed,weneedthedenitionsofthe A -based scalarproduct[69]andPABSinthe A -basedscalarproduct. ForanHPDmatrix A ,let h ; i A = h A ; i 2 denotethescalarproductinducedby A kk A = p h A ; i 2 = k A 1 = 2 k 2 denotethecorrespondingvectornorminduced by A kk A = k A 1 = 2 A )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 k 2 denotethecorrespondingmatrixnorminducedby A ThefollowingdenitionofPABSinthe A -basedscalarproduct,consistentwiththe denitionofPABSinSection2.1,isprovedbyKnyazevandArgentatiin[48]. Denition3.4.1 Let X ; Y C n besubspaceswith p =dim X and q =dim Y : Let m =min p;q : Theprincipalangles A X ; Y =[ A 1 ;:::; A m ] ; where A k 2 [0 ;= 2] ;k =1 ;:::;m; between X and Y inthe A -basedscalarproductarerecursivelydenedby s k =cos A k =max x 2X max y 2Y jh x;y i A j = jh x k ;y k i A j = j x H k Ay k j ; subjectto k x k A = k y k A =1 ;x H Ax i =0 ;y H Ay i =0 ;i =1 ;:::;k )]TJ/F15 11.9552 Tf 11.955 0 Td [(1 : Thevectors f x 1 ;:::;x m g and f y 1 ;:::;y m g arecalledprincipalvectorsrelativetothe A -basedscalarproduct. 49

PAGE 59

Theorem3.4.2 [48] Letthecolumnsofmatrices X 2 C n p and Y 2 C n q form A -orthonormalbasesforthesubspaces X and Y ,correspondingly.LettheSVDof thematrix X H AY be U V H ,where U and V areunitarymatricesand isa p by q diagonalmatrixwiththerealdiagonalelements s 1 X H AY ;:::;s m X H AY in nonincreasingorderwith m =min p;q .Then cos A X ; Y = S # X H AY = s 1 X H AY ;:::;s m X H AY ; where A isthevectorofprincipalanglesinthe A -basedscalarproductbetween X and Y arrangedinnondecreasingorder.Moreover,theprincipalvectorsrelativeto the A -basedscalarproductassociatedwiththispairofsubspacesaregivenbytherst m columnsof V X = XU and V Y = YV; correspondingly. Nowwederivetheoremsaboutsinesandcosinesofhalf-PABSinthe A -based scalarproduct.ThefollowingtheoremistheextensionofTheorem3.1.1. Theorem3.4.3 InthenotationofTheorem3.4.2,let q p and p + q n .Let L = A 1 = 2 [ XY ] andlettheSVDof L be ~ U ~ ~ V H ,where ~ isadiagonalmatrixwith diagonalelementsinnonincreasingorder.Then cos A 2 = 1 p 2 S q # L andsin # A 2 = 1 p 2 S # q L : .4.1 Moreover,wehave ~ V = 1 p 2 2 6 4 U 1 p 2 U 2 )]TJ/F24 11.9552 Tf 10.902 3.022 Td [(b U 1 VO b V 3 7 5 ; where U H 1 U 1 = b U H 1 b U 1 = I with U 1 ; b U 1 2 C p q ; V H V = b V H b V = I with V; b V 2 C q q ; thematrix [ U 1 U 2 ] isunitary.Then,theprincipalvectorsinthe A -basedscalarproduct of A associatedwiththispairofsubspaces X and Y aregivenbythecolumnsof V X = XU 1 and V Y = YV: 50

PAGE 60

Alternatively,theprincipalvectorsinthe A -basedscalarproductof # A associatedwith thepairofsubspacesaregivenbythecolumnsof V X = X b U 1 and V Y = Y b V: Proof: Bytheassumptionthatthecolumnsofmatrices X 2 C n p and Y 2 C n q are A -orthonormalbasesforthesubspaces X and Y ,correspondingly,itfollowsthat X H AX = I and Y H AY = I .LetSVDofthematrix X H AY be U V H ,where =diag s 1 ;:::;s q with s 1 s q and s k =cos A k X ; Y for k =1 ;:::;q UsingasimilarprocessasintheproofofTheorem3.1.1,wehavetheeigenvaluesof [ XY ] H A [ XY ]= I + 2 6 4 OX H AY Y H AXO 3 7 5 .4.2 are[1+ s 1 ;:::; 1+ s q ; 1 ;:::; 1 ; 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(s q ;:::; 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(s 1 ]andthecorrespondingeigenvectors arethecolumnsof Q = 1 p 2 2 6 4 U 1 p 2 U 2 )]TJ/F24 11.9552 Tf 10.902 3.022 Td [(b U 1 VO b V 3 7 5 ; .4.3 where U 1 istherst q columnsof U ,suchthat U 1 =[ u 1 ;u 2 ;:::;u q ] : Moreover, U = [ U 1 U 2 ].Let V =[ v 1 ;v 2 ;:::;v q ],then b V =[ v q ;v q )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ;:::;v 1 ]and b U 1 =[ u q ;u q )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ;:::;u 1 ]. Sincetheeigenvaluesof3.4.2areequaltothesquareofthesingularvaluesof L = A 1 = 2 [ XY ],weobtain3.4.1.Moreover,lettheSVDof L be ~ U ~ ~ V H .Itiseasy toobtain ~ V = Q .BasedonthedenitionofPABSinthe A -basedscalarproductvia SVD,wehavethattheprincipalvectorsof A inthe A -basedscalarproductarethe columnsof XU 1 and YV .Similarly,theprincipalvectorsof # A inthe A -basedscalar productarethecolumnsof X b U 1 and Y b V HighamandAl-Mohy[34]giveanalgorithmtocomputethesquarerootof A Itcostsabout15 2 3 n 3 opsforan n by n Hermitianpositivedenitematrix A .For computationalreasons,whenthesizeof A islarge,weneedtoavoiddealingwiththe 51

PAGE 61

squarerootof A explicitly.FromtheproofofTheorem3.4.3,let L 1 =[ XY ] H A [ XY ] and p + q n then cos 2 A 2 = 1 2 S q # L 1 andsin 2 # A 2 = 1 2 S # q L 1 : .4.4 However,usingtheSVDofthematrix[ XY ] H A [ XY ]maynotproducesmallangles accurately.Becauseweneedtotakethesquarerootofthesingularvaluesof L 1 to obtainthesinesorcosinesofhalf-PABS.Inotherwords,thePABSsmallerthan10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(8 willbezeroindoubleprecision. Onewaytocuretheproblemistoreplace L inTheorem3.4.3by L = R [ XY ] or L = K [ XY ]where A = R H R Choleskyfactorizationor A isknownas A = K H K .Forthiscase,weneedtoknow A exactlyorgive A as K H K .However, sometimes A maynotbeavailableasamatrixbutonlyassomefunctionperforming themultiplicationof A byagivenvector.Wecanupdate L byusingthefollowing lemma[48,Lemma6.1]. Lemma3.4.4 TheSVDofthematrix A 1 = 2 L coincideswiththeSVDofthematrix Q H L AL ,where Q L isamatrixwith A -orthonormalcolumns,whichspanthesame column-spaceascolumnsofmatrix L InTheorem3.2.1,wepresentaformulaofthesinesofhalf-PABS.Thefollowing theoremisanextensionofTheorem3.2.1. Theorem3.4.5 InthenotationofTheorem3.4.2,let L = A 1 = 2 [ XU 1 )]TJ/F18 11.9552 Tf 12.116 0 Td [(YV ] ,where U 1 istherst q columnsof U: Let q p .Wehave sin # A 2 = 1 2 S # L : .4.5 Proof: FromTheorem3.2.1,wehave X H AY = U V H .Since q p ,wehave X H AY = U 1 CV H ,where C isadiagonalmatrixwithcos A k onitsdiagonal,and A k for k =1 ;:::;q aretheprincipalanglesbetween X and Y inthe A -basedscalar 52

PAGE 62

product.Itisworthnotingthat U 1 CV H isthethinSVDof X H AY .Then L H L = XU 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(YV H A XU 1 )]TJ/F18 11.9552 Tf 11.956 0 Td [(YV = U H 1 X H AXU 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(U H 1 X H AYV )]TJ/F18 11.9552 Tf 11.955 0 Td [(V H Y H AXU 1 + V H Y H AYV =2 I )]TJ/F15 11.9552 Tf 11.956 0 Td [(2 C; where I )]TJ/F18 11.9552 Tf 12.06 0 Td [(C isadiagonalmatrixwith1 )]TJ/F15 11.9552 Tf 12.06 0 Td [(cos A k for k =1 ;:::;q onitsdiagonal. Sincethesingularvaluesof L arethesquarerootoftheeigenvaluesof L H L ,theproof iscompleted. ByanalogywithAlgorithms3.1and3.2,wecanobtainAlgorithms3.4and3.5 forcomputingtheprincipalanglesinthe A -basedscalarproductaccordingtoTheorems3.4.3and3.4.5.InAlgorithms3.4and3.5,weusethefunctionORTHUPDATE.mseeAppendixAwithoptionopts.A= A tocomputean A -orthogonal basis,whichiscompatiblewiththefunctionORTHA.mpresentedin[48]. Table3.4: Algorithm3.4. Algorithm3.4 Input :matrices X 1 and Y 1 withthesamenumberofrows. 1.Compute A -orthonormalbases X 2 C n p of R X 1 and Y 2 C n q of R Y 1 2.Compute L 1 =[ XY ]and Q L 1 =orthupdate L 1 ; opts.A. 3.ComputetheSVDof L = Q H L 1 AL 1 ,suchthat h ~ U; ~ ; ~ V i =svd L; 0 : 4.Takethesmallest m )]TJ/F8 9.9626 Tf 9.963 0 Td [( p + q )]TJ/F11 9.9626 Tf 9.962 0 Td [(j singularvaluesof L with j =size Q L 1 ; 2andadd p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(j zerosdenotedas s 1 ;:::;s m with m =min p;q : 5.Computetheprincipalanglesinthe A -basedscalarproduct: k =2arcsin )]TJ/F11 9.9626 Tf 4.566 -8.07 Td [(s k = p 2 for k =1 ;:::;m 6.Take 8 < : b U 1 = )]TJ 7.749 8.241 Td [(p 2 ~ V : p;p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(m +1: p + q b V 1 = p 2 ~ V p +1: p + q;p + q )]TJ/F11 9.9626 Tf 9.963 0 Td [(m +1: p + q 7.Computetheprincipalvectorsinthe A -basedscalarproduct: V X = X b U 1 and V Y = Y b V 1 : Output :theprincipalangles 1 ;:::; m betweencolumn-spaceofmatrices X 1 and Y 1 inthe A -basedscalarproduct,andthecorrespondingprincipalvectors V X and V Y : 53

PAGE 63

Table3.5: Algorithm3.5. Algorithm3.5 Input :matrices X 1 and Y 1 withthesamenumberofrows. 1.Compute A -orthonormalbases X and Y of R X 1 and R Y 1 ,correspondingly. 2.ComputetheSVDof B = X H AY : 8 < : [ U r ;C;V r ]=svd B; 0ifrank X rank Y ; [ V r ;C;U r ]=svd B H ; 0otherwise. 3.Compute L 1 = XU r )]TJ/F11 9.9626 Tf 9.963 0 Td [(YV r and Q L 1 =orthupdate L 1 ; opts.A : 4.ComputeSVDof L = Q H L 1 AL 1 : 5.Takethesingularvalues s k of L for k =1 ;:::;m ,where m =minsize L .Then theprincipalanglesinthe A -basedscalarproductare k = 8 < : 2arcsin s k = 2for k =1 ;:::;m; 0for k = m +1 ;:::; min p;q : 6.Taketherightsingularvectors V 1 of L ,thentheprincipalvectorsinthe A -based scalarproductassociatedwiththepairofsubspacesarethecolumnsof V X = XU r V 1 and V Y = YV r V 1 Output :theprincipalangles 1 ;:::; min p;q betweencolumn-spaceofmatrices X 1 and Y 1 inthe A -basedscalarproduct,andthecorrespondingprincipalvectors V X and V Y Remark3.4.6 TherstandsecondstepsofAlgorithm3.5canbereplacedby theCholeskyfactorization.Letthematrices X 1 and Y 1 havefullcolumnrank. LettheCholeskyfactorizationof X H 1 AX 1 and Y H 1 AY 1 be R H X R X and R H Y R Y ; correspondingly,where R X and R Y areuppertriangularmatrices.Then,wehave B = R H X )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 X H 1 AY 1 R Y )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 ; whichimpliesthat cos A X ; Y = S # B : Iftheinputmatricesareill-conditioned,the A -orthonormalbasesmaybecomputedinaccurately.Next,wedevelopanalgorithmtocomputePABSinthe A -based scalarproductwithoutforming A -orthonormalbases.Toproceed,weusetwolemmas.ThefollowinglemmalinksthePABSinthe A -basedscalarproductwiththe PABSinthestandardscalarproduct,e.g.,see[48]. Lemma3.4.7 [48] Let A = K H K forsomematrix K: Theprincipalanglesbetween 54

PAGE 64

subspaces X and Y relativetothescalarproduct h ; i A coincidewiththeprincipal anglesbetweensubspaces K X and K Y relativetothestandardscalarproduct h ; i : Thefollowinglemmalinks A -orthogonalQRfactorizationwithQRfactorization inthestandardproduct. Lemma3.4.8 Let A -orthogonalQRfactorizationof S be Q A R ,where Q H A AQ A = I and R isanuppertriangularmatrix.ThentheQRfactorizationof A 1 = 2 S is ^ QR with respecttothestandardinnerproductwith ^ Q = A 1 = 2 Q A : Proof: Theproofisstraightforward,since Q H A AQ A = A 1 = 2 Q A H A 1 = 2 Q A = I: Remark3.4.9 AlternativewaystocomputePABSinthe A -basedscalarproductare byusingtheSVDofthetriangularpart R obtainedfrom A -orthogonalQRfactorizationsof [ XY ] and [ XU 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(YV ] inTheorems3.4.3and3.4.5. Theorem3.4.10 Let X 1 2 C n p and Y 1 2 C n q bematricesoffullrankwhere p + q n and q p .Theprincipalanglesbetween R X 1 and R Y 1 relativetothe A -basedscalarproductcanbecomputedasfollows. 1.Computethetriangularpart R ofthereduced A -orthogonalQRfactorizationof thematrix L 1 =[ X 1 Y 1 ] ,suchthat R = 2 6 4 R 11 R 12 OR 22 3 7 5 ; where R 11 2 C p p R 12 2 C p q ,and R 22 2 C q q 2.Computethetriangularpart R Y ofthereduced A -orthogonalQRfactorization ofthematrix Y 1 55

PAGE 65

3.Let L = 2 6 4 IR 12 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y OR 22 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y 3 7 5 .Then cos A 2 = 1 p 2 S q # L andsin # A 2 = 1 p 2 S # q L ; cos A = S # R 12 R )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 Y andsin # A = S # R 22 R )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Y : Proof: UsingLemma3.4.7,wetake K = A 1 = 2 .Computingtheprincipalangles betweenthesubspaces X and Y inthe A -basedscalarproductisequivalenttocomputingtheprincipalanglesbetweenthesubspaces A 1 = 2 X and A 1 = 2 Y inthestandard scalarproduct. Therefore,tocomputePABSinthe A -basedscalarproduct,wecanreplace L 1 with L 1 = A 1 = 2 [ X 1 Y 1 ]intherststatementofTheorem3.3.1andreplace Y 1 with A 1 = 2 Y 1 inthesecondstatementofTheorem3.3.1.Inotherwords,wecanobtaintriangularmatrices R and R Y bythereducedQRfactorizationsofthematrix A 1 = 2 [ X 1 Y 1 ] and A 1 = 2 Y 1 ,correspondingly.UsingLemma3.4.8,weobtaintheresultsasstated. Remark3.4.11 Inparticular,if [ X 1 Y 1 ] hasfullcolumnrank,wecanobtain R and R Y bytheCholeskyfactorizationsofmatrices [ X 1 Y 1 ] H A [ X 1 Y 1 ] and Y H 1 AY 1 ,such that [ X 1 Y 1 ] H A [ X 1 Y 1 ]= R H R and Y H 1 AY 1 = R H Y R Y : Remark3.4.12 Oneapproachtoconstructing A -orthogonalQRfactorizationofa matrixistousethemodiedGram-Schmidtprocesswithrespecttothe A -basedscalar product.Seethealgorithmbelow.TheMATLABcodegmsa.mtocomputethetriangularpart R bythe A -orthogonalQRfactorizationisprovidedinAppendixA. AccordingtoTheorem3.4.10,weformulateAlgorithm3.6tocomputeallPABS inthe A -basedscalarproductbycomputingthetriangularuppermatricesfrom A orthogonalQRfactorizations. 56

PAGE 66

Algorithm:modiedGram-Schmidtwithrespecttothe A -basedscalarproduct Input :matrix V =[ v 1 ;:::;v n ] : for i =1: n R ii = p v H i Av i ; q i = v i =R ii ; for j = i +1: n R ij = q H i Av j ; v j = v j )]TJ/F11 9.9626 Tf 9.963 0 Td [(R ij q i ; end end Output :triangularmatrix R: Table3.6: Algorithm3.6. Algorithm3.6 Input :matrices X 1 2 C n p and Y 1 2 C n q withfullcolumnrank. 1.Computethetriangularpart R of A -orthogonalQRfactorizationof L 1 =[ X 1 Y 1 ]. 2.Compute R 12 = R : p;p +1: p + q and R 22 = R p +1: p + q;p +1: p + q 3.Computethetriangularpart R Y of A -orthogonalQRfactorizationof Y: 4.Compute L = 2 4 eye p R 12 =R Y zeros q;p R 22 =R Y 3 5 : 5.Computethesmallest m =min p;q singularvalues s 1 ;:::;s m of L andthe corresponding m rightsingularvectors ^ V .Theprincipalanglesarecomputedas k =2arcsin )]TJ/F11 9.9626 Tf 4.566 -8.07 Td [(s k = p 2 for k =1 ;:::;m: 6.Computetheprincipalvectors: V X = p 2 X 1 =R : p; 1: p ^ V : p; :and V Y = p 2 Y 1 =R Y ^ V p +1: p + q; : : Output :theprincipalangles 1 ;:::; m ,andthecorrespondingprincipalvectors V X and V Y inthe A -basedscalarproduct. Finally,weobtainsomepropertiesof A -orthogonalprojectors.Weusetherepresentationforthe A -orthogonalprojectorasin[48,72].Let P X bean A -orthogonal projectoronthesubspace X ,suchthat P X = XX H A; where X H AX = I 57

PAGE 67

Theorem3.4.13 Let P X and P Y be A -orthogonalprojectorsonthesubspaces X and Y ,correspondingly.Then S # ; 1 [ ; 2 )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(A 1 = 2 P X + P Y A )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 = h 1+cos A ;= 2 X ; Y ; 1 )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos # A ;= 2 X ; Y i : Inparticular, k P X + P Y k A =1+cos A min X ; Y ; where A min X ; Y isthesmallestanglebetweenthesubspaces X and Y inthe A -based scalarproduct. Proof: Letthecolumnsofmatrices X and Y form A -orthonormalbasesforthe subspaces X and Y ,correspondingly.Wehave A 1 = 2 P X + P Y A )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 = A 1 = 2 XX H A + YY H A A )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 = A 1 = 2 XX H + YY H A 1 = 2 = A 1 = 2 [ XY ][ XY ] H A 1 = 2 : Let L = A 1 = 2 [ XY ],then A 1 = 2 P X + P Y A )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 = LL H : Thus,theeigenvaluesof A 1 = 2 P X + P Y A )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 arethesquareofthesingularvaluesof L .Noticethatwecould addzerosordeletezerosfromthesingularvaluesof L inordertomatchtheeigenvaluesof A 1 = 2 P X + P Y A )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 .Moreover, A 1 = 2 P X + P Y A )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 ispositivedeniteand Hermitian.Thesingularvaluesof A 1 = 2 P X + P Y A )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 arethesameastheeigenvalues. FromTheorem3.4.3,weobtaintheresult. 3.5Matlabimplementation Inthissection,wedescribeourcodeSUBSPACEUPDATE.mindetail.Ourfunctionhasmorefeatures,comparedtothebuilt-inMATLABfunctionSUBSPACE.m andthefunctionSUBSPACEA.minMathworkswrittenbyKnyazevandArgentati[48].Ourfunctionreturnsprincipalangles,principalvectors,andnormalized principalcoecients.Ourfunctionspeciesoptions.Belowisalistofoptionswhich SUBSPACEUPDATEtakes: 58

PAGE 68

opts.fast=0UsingthealgorithmasdescribedinAlgorithm3.1orAlgorithm3.4 forthe A -basedscalarproduct. opts.fast=1UsingthealgorithmasdescribedinAlgorithm3.2orAlgorithm3.5 forthe A -basedscalarproduct. opts.fast=2Usingcosine-basedalgorithmorcosine-basedalgorithmfor the A -basedscalarproduct. opts.AUse A -basedscalarproduct,where A ispositivedenite. opts.R=0Usetheorthonormalbasesorusethe A -orthonormalbasesfor inputmatrices. opts.R=1UsethetriangularuppermatricesfromtheQRfactorizationor the A -orthonormalQRfactorizationofinputmatrices. opts.numberFind k smallestprincipalangles,ifthenumberis k opts.dispDiagnosticinformationdisplaylevel0 ; 1 ; 2. Withopts.fast=0,thecodeiswrittenbasedonAlgorithm3.1andAlgorithm3.4. Tocomputetheprincipalanglesinthe A -basedscalarproduct,weaddonemore option,suchthatopts.A= A .Withopts.fast=1,thecodeiswrittenbasedon Algorithm3.2andAlgorithm3.5.Withopts.fast=2,thecodeiswrittenbasedon thecosine-basedalgorithm,whichcomputesallprincipalangles,butitlosesaccuracy forsmallangles.Withopts.R=1,weuseAlgorithm3.3andAlgorithm3.6tocompute principalanglesandprincipalvectorswithrespecttothestandardinnerproductand the A -basedscalarproduct,correspondingly.Moreover,weuseAlgorithm3.3to computetheprincipalanglesforlargesparse-typedata.Wesummarizetheoptions whichalgorithms3.1,3.2,3.3,3.4,3.5,and3.6takeinthefollowingtable. Nameoption Nameoption Alg.3.1opts.fast=0 Alg.3.4opts.fast=0,opts.A Alg.3.2opts.fast=1 Alg.3.5opts.fast=1,opts.A Alg.3.3opts.R=1 Alg.3.6opts.R=1,opts.A AlistingforthefunctionORTHUPDATE.misprovidedinAppendixA. 59

PAGE 69

3.6Numericaltests WeperformthenumericaltestsforourcodeSUBSPACEUPDATE.mbasedon algorithms3.1,3.2,3.3,3.4,3.5,and3.6describedinprevioussectionsforfulltypestructurematricesandAlgorithm3.3forsparse-typestructurematrices.For comparison,wealsorunthecodeSUBSPACEA.m[48]basedonthesine-cosinebased algorithmwhichcanbeobtainedfromMathWorks.Numericaltestsareperformed onWindowsIntelDuoCoreand64-bitoperatingsystem,runningMATLABR2011b. Ourrstexampleistakenfrom[9,48]with p =13and n =26.Thematrix X 1 isorthogonal,while Y 1 isan n by p Vandermondematrix.Inourcode,allcomputationsareperformedindoubleprecisioninMATLAB.Weobservethatourresults inTable3.9arecoincidingwiththoseof[48]withintwelvedecimaldigitsandare consistentwiththoseof[9]withinfourdecimaldigits. Inthenexttest,wetaketheexamplefrom[48].Let p = q n= 2.Wedene n by p matrices X 1 =[ IO ] H and Y 1 =[ IDO ] H ; where D isadiagonalmatrixofthesizeof p suchthat D =diag d 1 ;:::;d p and d k > 0for k =1 ;:::;p I istheidentitymatrixofthesize p and O arezeromatrices ofappropriatesizes.Theexactvaluesofsinesandcosinesofprincipalanglesbetween columnspacesofmatrices X 1 and Y 1 aregivenby sin k = d k p 1+ d 2 k ; cos k = 1 p 1+ d 2 k ; for k =1 ;:::;p: Assumingthat d k 'sarearrangedinthenondecreasingorder,then k 'sareinthenonincreasingorder.Let ~ k becomputedfromouralgorithms.The 60

PAGE 70

Table3.9: Computedsinesandcosinesofprincipalangles. k sin k cos k 1 0.9998785422971522 0.01558527040901074 2 0.9981506873301972 0.06078820101183188 3 0.6456213362708716 0.7636577048336605 4 0.4928094246236336 0.8701372713555736 5 0.3975367883303482 0.9175862367777712 6 0.3370430714820327 0.9414892288103764 7 0.2700504602152627 0.9628461709626958 8 0.2156943479780999 0.9764609302221479 9 0.1418470818351141 0.9898885823035148 10 0.1387517672025158 0.9903271919412184 11 0.06089682091193156 0.9981440663565656 12 0.05942261363977260 0.9982329151997635 13 1.072148118598322e-017 1.000000000000000 collectiveerrorinprincipalanglesismeasuredasthefollowingsum: r sin 1 )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ 1 2 + + sin p )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ p 2 +.6.1 r cos 1 )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos ~ 1 2 + + cos p )]TJ/F15 11.9552 Tf 11.956 0 Td [(cos ~ p 2 : Wemultiplymatrices X 1 and Y 1 byarandomorthogonalmatrix U 1 ofthesize n on thelefttoget X 2 = U 1 X 1 and Y 2 = U 1 Y 1 : Alsowemultiplymatricesbyrandomorthogonal p by p matrices U X and U Y ,correspondingly,ontherighttoobtain X 3 = X 2 U X and Y 3 = Y 2 U Y : Wehighlightthattheprincipalanglesandtheconditionnumbersareinvariantwith respecttothesetwotransformations. 61

PAGE 71

Figure3.3: Errorsinprincipalangles: p =20topand p asafunctionof n such that p = n= 2bottom;leftforAlgorithm3.1andAlgorithm3.4with A = I ,andright forAlgorithmsine-cosineandAlgorithmsine-cosinewith A = I Inthefollowingtests,westarttocheckthescalabilityofthecodeofAlgorithms3.1,3.2,3.3,andAlgorithms3.4,3.5,3.6with A = I ,forwell-conditioned cases.Weplotthecollectiveerrorin3.6.1byincreasingthesizeoftheproblem n fortheprincipalanglesbetween X 3 and Y 3 : Inourrsttwotests,weusethesame D asin[48].Thediagonalentriesof D arechosenasuniformlydistributedrandomnumbersontheinterval ; 1.Onthe toppartofFigure3.3,wex p = q =20andincreasethesizeoftheproblem n andplotthecollectiveerror,fortheprincipalanglesbetween X 3 and Y 3 ,againstthe value n= 2.TheleftsidegraphofFigure3.3presentsforAlgorithm3.1andtheright sidegraphpresentsforthesine-cosinebasedalgorithmin[48].Wegetthesimilar observationasin[48]thattheaverageerrorgrowsapproximatelytwotimeswitha tentimeincreaseintheproblemsizeinbothgraphs.ThebehaviorofAlgorithm3.1 62

PAGE 72

andsine-cosinebasedalgorithmisverysimilar.OnthebottompartofFigure3.3, weincrease p = q = n= 2.Weobservethattheaverageerrorgrowswiththesame paceastheproblemsizeinbothrightandleftgraphs.Algorithm3.4andsine-cosine basedalgorithmwith A = I behaveverysimilarlyinFigure3.3.Wealsotryother analogoustestswhichverifythesimilarbehaviorofthesealgorithms.Wealsotest Algorithms3.2,3.3andAlgorithms3.5,3.6with A = I .Weobtainsimilargraphsas thesedisplayedinFigure3.3,sosomegraphsarenotshownhere. Figure3.4: ErrorsintheprincipalanglesforAlgorithms3.1,3.2,andsine-cosine basedalgorithmwith D =diag )]TJ/F16 7.9701 Tf 6.587 0 Td [(16rand ;p and p =20. Inthesecondtest,wefocusontestingforsmallangles.Weset p = q =20and selecteverydiagonalelementof D intheform10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(16rand ;p ,whererand ; isagain auniformlydistributedrandomnumberontheinterval ; 1.Weincreasethesizeof theproblem n andplotthecollectiveerror,fortheprincipalanglesbetween X 3 and Y 3 .WecomparethecollectiveerrorsofournewAlgorithms3.1and3.2withthose ofthecosine-sinebasedalgorithminFigure3.4.Inthisgure,wegetgoodaccuracy witherrorslessthan6 10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(15 forthesethreemethods.Moreover,amongthesethree methods,Algorithm3.1isthemostaccuracyandAlgorithm3.2istheleastaccuracy. 63

PAGE 73

Inthethirdtest,weturnourattentiontotheerrorsforindividualPABS, j sin k )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ k j + j cos k )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos ~ k j ; for k =1 ;:::;p; insteadofthesumoferrorsin3.6.1.Wexasmall p = q and n =100andcompute theprincipalanglesbetween X 2 and Y 2 andbetween X 3 and Y 3 500times.In[48], theytestseveraldierentcombinationsofangleslessthan = 4.Oneoftheworse-case scenarioistaking D =diag )]TJ/F15 11.9552 Tf 5.48 -9.684 Td [(1 ; 0 : 5 ; 10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(11 ; 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(12 ; 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(13 ; 5 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(15 ; 2 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(15 ; 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(15 ; 10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(16 ; 0 : WerepeatthistestforourAlgorithms3.1,3.2andthesine-cosinebasedalgorithm, whichispresentedinFigure3.5.ThetopthreegraphsofFigure3.5showthedistributionsoftheerrorforindividualanglesbetween X 3 and Y 3 forAlgorithm3.1, Algorithm3.2andsine-cosinebasedalgorithmin500runs.InFigure3.5,smalleranglesarefurtherawayfromtheviewofreader.Again,weobservethatAlgorithm3.1 achievesthebeststabilityandAlgorithm3.2achievestheleaststabilityforsmall anglesamongthesethreealgorithms.ThreegraphsonthebottomofFigure3.5 demonstratetheperformanceofAlgorithm3.4,Algorithm3.5andsine-cosinebased algorithmwith A = I forthesameproblem. Surprisingly,theerrorsforsmallindividualanglesaregettingsmallerforAlgorithm3.4with A = I: Themainreasonisthefollowing.Weuse Q L 1 ,whichhas A -orthonormalcolumns,toapplyinAlgorithm3.4. Q L 1 shouldreturnasan n by k matrixintheexactarithmetic,butindoubleprecision Q L 1 returnsasan n by ^ k matrix,where ^ k
PAGE 74

Figure3.5: Errordistributionsforindividualangles:topthreeforAlgorithm3.1, Algorithm3.2,andAlgorithmsine-cosine;bottomthreeforAlgorithms3.4,3.5,and sine-cosinewith A = I: 65

PAGE 75

Figure3.6: ErrorsintheprincipalanglesforAlgorithms3.3and3.6. Figure3.6,wegethighaccuracyforbothalgorithms. Inthenexttest,wewillanalyzethestabilityofouralgorithmsforill-conditioned scalarproducts.Wetake A = A k =10 )]TJ/F19 7.9701 Tf 6.587 0 Td [(k I + H for k =1 ;:::; 16asin[48],where I istheidentitymatrixand H istheHilbertmatrixoftheordertwentywith h i;j = 1 = i + j )]TJ/F15 11.9552 Tf 12.314 0 Td [(1for i;j =1 ;:::; 20 : Wetake X 1 and Y 1 as G and F in[48],suchthat X 1 =eye20 ; 10and Y 1 isequaltothelasttencolumnsoftheVandermondematrix withelements y i;j = i 20 )]TJ/F19 7.9701 Tf 6.586 0 Td [(j for i;j =1 ;:::; 20 ; with Y 1 10 13 : WeapplytheLU factorizationto Y 1 withpartialpivotinginordertoreducetheconditionnumberof Y 1 .Therefore,wegetthelowertriangularpartmatrix Y 2 ,suchthat R Y 1 = R Y 2 As k increases,thesubspaces X and Y areinvariant,buttheconditionnumberof A increases,since A 10 k : Weplotthesinesofalltenprincipalanglesasfunctionof k inlogarithmicscale.Also,asin[48],weplottheerror,whichisdenedasfollows. error= k V H X AV X )]TJ/F18 11.9552 Tf 11.955 0 Td [(I k + k V H Y AV Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(I k + k diagcos )]TJ/F18 11.9552 Tf 11.956 0 Td [(V H X AV Y k ; where V X and V Y areprincipalvectorsofthepairofsubspaces. InFigure3.7,weobservethattheresultsofournewalgorithmsarequitesimilar asthoseofthesine-cosinebasedalgorithmin[48].Theerrorincreasesasthecondition numberof A increases.Furthermore,amongAlgorithms3.4,3.5,and3.6,theerror 66

PAGE 76

Figure3.7: Sineofanglesasfunctionsof k forAlgorithms3.4,3.5,and3.6top andleftonbottom;ErrorsasfunctionsofconditionnumberforAlgorithms3.4,3.5, and3.6rightonbottom. ofAlgorithm3.5isthesmallestforlarge k: Inparticular,for k =16and A 10 18 ourcodeforAlgorithm3.4returnszerosfor6smallestangles,whichleadstogaps occurringinFigure3.7forAlgorithm3.4.Thereasonisthesameaswehaveexplained intheprevioustest.Weloseaccuracyinthestepfor A -orthonormal.InFigure3.7 bottom,weobservethatAlgorithm3.6isnotstableforill-conditioned A: The reasonisthatthemodiedGram-Schmidtalgorithmwithrespecttothe A -based scalarproductisnotstableforill-conditionedmatrix. 3.7Discussion Inthissection,webrieydiscussalgorithmswhichwehavepresentedinthe previoussections,andgiveashortsummaryforthecomparisonforthesealgorithms 67

PAGE 77

andthesine-cosinebasedalgorithm.ThealgorithmstocomputePABSandPABSin the A -basedscalarproductaresummarizedinTables3.10. Table3.10: AlgorithmstocomputePABSandPABSinthe A -basedscalarproduct. Name Algorithm Name Algorithm A -basedscalarproduct Alg.3.1 X =orthupdate X 1 Y =orthupdate Y 1 SVDof[ XY ] Alg.3.4 X =orthupdate X 1 ; opts.A Y =orthupdate Y 1 ; opts.A L 1 =[ XY ] Q L 1 =orthupdate L 1 ; opts.A SVDof Q H L 1 AL 1 Alg.3.2 X =orthupdate X 1 Y =orthupdate Y 1 SVDof X H Y : U V H SVDof XU )]TJ/F11 9.9626 Tf 9.962 0 Td [(YV Alg.3.5 X =orthupdate X 1 ; opts.A Y =orthupdate Y 1 ; opts.A SVDof X H AY : U V H L 1 = XU )]TJ/F11 9.9626 Tf 9.962 0 Td [(YV Q L 1 =orthupdate L 1 ; opts.A SVDof Q H L 1 AL 1 Alg.3.3 R =qr[ X 1 Y 1 ] R Y =qr[ R 12 R 22 ] 0 SVDof 2 4 IR 12 R )]TJ/F7 6.9738 Tf 6.227 0 Td [(1 Y OR 22 R )]TJ/F7 6.9738 Tf 6.227 0 Td [(1 Y 3 5 Alg.3.6 R from A -orthogonalQRof[ X 1 Y 1 ] R Y from A -orthogonalQRof Y 1 SVDof 2 4 IR 12 R )]TJ/F7 6.9738 Tf 6.227 0 Td [(1 Y OR 22 R )]TJ/F7 6.9738 Tf 6.227 0 Td [(1 Y 3 5 Sine-cosine X =orthupdate X 1 Y =orthupdate Y 1 SVDof X H Y SVDof I )]TJ/F11 9.9626 Tf 9.963 0 Td [(XX H Y Sine-cosine X =orthupdate X 1 ; opts.A Y =orthupdate Y 1 ; opts.A SVDof X H AY L 1 = I )]TJ/F11 9.9626 Tf 9.963 0 Td [(XX H A Y Q L 1 =orthupdate L 1 ; opts.A SVDof Q H L 1 AL 1 WeremarkthatweusethefunctionORTHUPDATE.mwiththedefaultvalue, suchthatopts.fastis0,whichisisbackwardcompatiblewithMATLABbuilt-in functionorthtocomputetheorthonormalbases.Thefunctionorthupdate.mwith optionopts.Aisbackwardcompatiblewithortha.mfunctionin[48],whichismore expensivethan A -orthogonalQRfactorizationusingthefunctionmgsa.mbasedon 68

PAGE 78

ourMATLABtests. Next,wesummarizethecomparisonforalgorithmstocomputePABSinTable3.11andPABSinthe A -basedscalarproductinTable3.12. Table3.11: ComparisonforalgorithmstocomputePABS Name Advantage Disadvantage Alg.3.1 mostaccurate expensiveforverylarge n Alg.3.2 thecostisthesameasthecostof sine-cosinebasedalgorithm lessaccuracyforsmallanglescomparetoAlg.3.1andAlg.sine-cosine Alg.3.3 worthforsparsematrix Sine-cosine thecostisthesameasAlg.3.2 cannotcomputeprincipalvectors ingoodaccuracycorrespondingto PABSintheneighborhoodofthe threshold Table3.12: ComparisonforalgorithmstocomputePABSinthe A -basedscalar product. Name Advantage Disadvantage Alg.3.4 if A isavailableas R H R or K H K; PABScanbeobtainedinmosthigh accuracy. ifthefunctionorthloseaccuracy for[ XY ],thenthoseanglesreturn aszeros Alg.3.5 goodaccuracyforanill-conditioned matrix A Alg.3.6 cheap loseaccuracyforanill-conditioned matrix A andtheaccuracydepends onMSGalgorithm Sine-cosine goodaccuracyforanill-conditioned matrix A loseaccuracyforprincipalvectors correspondingPABSnearthreshold 69

PAGE 79

3.8Conclusions Inthischapter,wehavederivedseveralnewmethodstocomputeallPABSand PABSinthe A -basedscalarproductaccurately.Thealgorithmsarebasedoncomputingthesinesofhalfanglesusingthealgebraicandgeometricapproaches.Wehave alsodescribedanumericallyrobustandfastmethodtocomputePABSofsparse-type anddense-typematricesusingthebasisoftriangularmatricesfromQRfactorizations.Weconcludethisworkwithseveralnumericaltests.Thecodeisrobustand canprovideaccurateanglesnotonlyforlargescalematricesbutalsoforlargesparse matrices. Thereremainsanumberofproblemsassociatedwiththistechnique,suchasdeterminingorthonormalcolumninthe A -basedscalarproducttomakethefourthstep inAlgorithm3.5moreaccurate.Thoughalternativewaystocompute A -orthonormal basisareavailable,suchasavariantoftheGram-Schmidtalgorithm[72]orthemodiedGram-Schmidtasweuseforthe A -orthogonalQRfactorization,thosemethods donottwellforourpurpose. 70

PAGE 80

4.Majorizationinequalities Majorizationisatopicofmuchinterestinvariousareasofmathematics,e.g.,matrixtheory,numericalanalysis,combinatorics,statistics,andprobability.Majorizationisoneofthemostpowerfultechniquesthatcanbeusedtoderiveinequalitiesin aconciseway.Inparticular,majorizationrelationsamongeigenvaluesandsingular valuesofmatricesproduceavarietyofnorminequalitiesinmatrixtheory.Fora generaldiscussionofmajorization,seeMashall,OlkinandArnold[55],Bhatia[5], Ando[1],andHornandJohnson[36]. Inthischapter,wepresentweakmajorizationinequalitiesforsingularvaluesand eigenvaluesofHermitianmatrices.Weareconcernedwiththesingularvaluesfor thedierenceoftwoHermitianmatrices A )]TJ/F18 11.9552 Tf 12.566 0 Td [(B boundedbythesingularvaluesof AT )]TJ/F18 11.9552 Tf 12.318 0 Td [(TB and T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 12.318 0 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 ,where T ispositivedenite.Theextensionsofthese boundsappliedtoeigenvaluesareprovided.Wealsopresenttherelationsinvolving thedierenceofeigenvaluesof A and B withamatrix AT )]TJ/F18 11.9552 Tf 10.11 0 Td [(TB ,where T isinvertible. OurworkisinspiredbyBhatia,KittanehandLi[7],Bhatia,Davis,andKittaneh[6]. Also,inthischapterweconsidertheabsolutechangeintheprincipalangles betweensubspaces X and Y ; wheresubspacesareperturbedto ~ X and ~ Y .Weuseweak majorizationtomeasurethechange.Wealsoprovidesomebasicresultsinvolvingthe absolutechangeforthetangentsofPABSandpresentaperturbationanalysisfor sinesandcosinesofhalf-PABS. 4.1Denitionandpropertiesofmajorization Denition4.1.1Weakmajorizationandmajorization Let x # and y # 2 R n bethevectorsobtainedbyrearrangingthecoordinatesofvectors x and y inalgebraicallynonincreasingorder,denotedby x [1] ;:::;x [ n ] and y [1] ;:::;y [ n ] suchthat x [1] x [2] x [ n ] and y [1] y [2] y [ n ] : Wesaythat x isweakly majorizedby y ,usingthenotation x w y; if k X i =1 x [ i ] k X i =1 y [ i ] ; 1 k n: 71

PAGE 81

Ifinaddition P n i =1 x [ i ] = P n i =1 y [ i ] ; wesaythat x ismajorizedorstronglymajorized by y ,usingthenotation x y Forexample,wehavethefollowingstrongmajorization: 1 n ;:::; 1 n 1 n )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ;:::; 1 n )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 ; 0 [1 ; 0 ;:::; 0] : Moreover,wecanseethatallvectorswithsum1majorizethevectorwithuniform components,i.e., 1 n ; 1 n ;:::; 1 n Geometrically,for x;y 2 R n ,therelation x y holdsifandonlyif x isinthe convexhullofallvectorsobtainedbypermutingthecoordinatesof y ,i.e.,see[55]. InFigure4.1left,foraxedvector y weseethatallvectors x onthebluesegment aremajorizedby y ,suchthat[ y 1 + y 2 2 ; y 1 + y 2 2 ] x y .InFigure4.1right,for axedvector y weseethatallvectors x ontheareaboundedwithblueedgesare majorizedby y ,suchthat[ y 1 + y 2 + y 3 3 ; y 1 + y 2 + y 3 3 ; y 1 + y 2 + y 3 3 ] x y: Figure4.1: Geometryofmajorizationin2Dand3D. Weremarkthat x y means x i y i for i =1 ;:::;n .Therefore, x y implies x w y ,but x w y doesnotimply x y .Therelations and w are allreectiveandtransitive[3,5],butnotsymmetric,e.g., y x and x y do notimply x = y .Bythecharacterizationofmajorization,Polyaprovesin[65]that 72

PAGE 82

nondecreasingconvexfunctionspreservetheweakmajorization.Inotherwords,if y w x ,then f y w f x ,where f isanondecreasingconvexfunction.Weusethe notation f x =[ f x 1 ;f x 2 ;:::;f x n ] : Thefollowingtworesultsprovidewaystopreservemajorization.Thesetwobasic majorizationinequalitiesofvectorscanbeobtainedbystraightforwardalgebra.Let nonnegativevectors x;y;u; and v bedecreasingandofthesamesize.If x w y ,then x u w y u whichcanbefoundin[51].Notethat x u = x # u # =[ x 1 u 1 ;:::;x n u n ]. However,theconverseisnottrueingeneral.Moreover,if x w y and u w v ,then x u w y v .Theproofissimple. x w y implies x u w y u and u w v implies y u w y v ,sowehave x u w y v 4.2Someknownmajorizationresults Inthissection,webrieyreviewsomeexistingmajorizationinequalitiesbasedon singularvaluesandeigenvaluesofmatrices.Weprovidethesewellknownmajorization resultsastheyserveasafoundationforournewmajorizationinequalitiesforsingular values,eigenvalues,andtheabsolutechangeinPABS. Intherestofthisthesis, ;S ,and denotetheorderedvectors, i.e., = # ;S = S # and = # : Theorem4.2.1 [Lidskii-Mirsky-WielandtTheorem][5,55] If A and B areHermitian matrices,then A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B A )]TJ/F18 11.9552 Tf 11.956 0 Td [(B : Theorem4.2.2 [55,Fantheoremp.287] Foranysquarematrices A and B ,wehave S A + B w S A + S B : Theorem4.2.3 [5,Mirsky'sTheorem] Foranysquarematrices A and B; wehave j S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(S B j w S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B : 73

PAGE 83

Theorem4.2.4 [5,p.71,Lidskii'sTheorem] ForHermitianmatrices A and B; j A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B j w S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B : Lettheeigenvaluesof A and B beinnonincreasingorder,suchthat A =[ 1 A ;:::; n A ]and B =[ 1 B ;:::; n B ] : TherearetwoinequalitiesthatfollowfromTheorem4.2.4.OneiscalledWeyl's theorem: max i =1 ;:::;n j i A )]TJ/F18 11.9552 Tf 11.955 0 Td [( i B jk A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B k : TheotherseeStewartandSun[73]is p P n i =1 j i A )]TJ/F18 11.9552 Tf 11.955 0 Td [( i B j 2 k A )]TJ/F18 11.9552 Tf 11.616 0 Td [(B k F ; which isaspecialcaseoftheHoman-Wielandttheoremestablishedfornormalmatrices. Theresultsmentionedabovedescribetheeigenvaluesandthesingularvaluesof thedierenceorthesumofmatrices.Nextweturntothesingularvaluesofthe productofmatrices. Theorem4.2.5 [51] Forgeneralmatrices A and B ,wehave S AB w S A S B : Ifneeded,weaddzerostomatchthesizesofvectorsoneitherside. Theorem4.2.6 [5,p.94,TheoremIV.2.5] Let A and B besquarematrices.Then, forany t> 0 wehave S t AB w S t A S t B ; where S t A =[ s t 1 A ;:::;s t n A ] : WecanseethatTheorem4.2.6isageneralizationofTheorem4.2.5,if A and B aresquarematrices.However,Theorem4.2.5alsoworksforrectangularmatrices. 4.3Unitarilyinvariantnorms,symmetricgaugefunctions,and majorization 74

PAGE 84

Inthissection,wediscusstherelationshipbetweenunitarilyinvariantnorms, symmetricgaugefunctions,andweakmajorization. Denition4.3.1Unitarilyinvariantnorms Anorm jjjjjj onthespaceofmatricesiscalledunitarilyinvariant,if jjj UAV jjj = jjj A jjj ; forallunitarymatrices U and V Forexample,the2-norm,Ky-Fan k norms,Schatten p -norms,andFrobenius normareallunitarilyinvariantnorms. Ky-Fan k norms: k A k k = P k i =1 s i A ; where k =1 ;:::; min m;n : Schatten p -norms: k A k p = P min m;n i =1 s i A p 1 =p for p 1 : TheFrobeniusnormisaspecialcaseofaSchatten p -normfor p =2 : Denition4.3.2Symmetricgaugefunctions Afunction : R k R iscalledasymmetricgaugefunctionif isanorm, P = forany 2 R k andpermutationmatrix P Itisabsolute, j j = TherelationshipbetweenunitarilyinvariantnormsandsymmetricgaugefunctionsiswellinvestigatedbyvonNeumann,e.g.,see[35,p.438,Theorem7.4.24]. Heshowsthatifisasymmetricgaugefunction,thematrixfunctiondenedby S A isaunitarilyinvariantnorm.Conversely,foreveryunitarilyinvariantnorm, thereisacorrespondingsymmetricgaugefunctionsuchthat jjj A jjj = S A Consideringsingularvaluedecomposition,itisimmediatelyapparentthataunitarilyinvariantnormonamatrix A isafunctionofthesingularvaluesof A .Let 75

PAGE 85

theSVDof A be U V H ,then jjj A jjj = jjj U V H jjj = jjj jjj : Thefollowingtheoremprovidesanintimatelinkbetweenunitarilyinvariantnorminequalitiesandweak majorizationinequalities. Theorem4.3.3 [5] Let A and B besquarematrices.Then S A w S B ifand onlyif jjj A jjjjjj B jjj foreveryunitarilyinvariantnorm. Fromthistheorem,weseethattheinequalitiesofunitarilyinvariantnormsare equivalenttoweakmajorizationofsingularvaluesofmatrices.Forageneraldiscussionoftheconnectionbetweenunitarilyinvariantnorms,symmetricgaugefunctions, andmajorization,thereaderisreferredto[5]. Theorem4.3.4 [5,p.254] Let A and B besquarematricesand AB beaHermitian matrix.Then jjj AB jjjjjj re BA jjj ; foreveryunitarilyinvariantnorm,orequivalently, S AB w S re BA ; where re A denotes A + A H = 2 : Theorem4.3.5 [7] Let A and B besquarematricesand AB beanormalmatrix. Then jjj j AB j 1 = 2 pol jjjjjj j BA j 1 = 2 pol jjj ; foreveryunitarilyinvariantnorm,orequivalently, S j AB j 1 = 2 pol w S j BA j 1 = 2 pol ; where j A j pol = A H A 1 = 2 76

PAGE 86

Theorem4.3.6 [5,p.236VIII.3.9] Let A and B besquarematricessuchthat A = JD 1 J )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 and B = TD 2 T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ; where J and T areinvertiblematrices,and D 1 and D 2 arerealdiagonalmatrices.Then jjj diag A )]TJ/F15 11.9552 Tf 11.955 0 Td [(diag B jjj [ J T ] 1 = 2 jjj A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B jjj ; foreveryunitarilyinvariantnorm,orequivalently, j A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B j w [ J T ] 1 = 2 S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B ; where diag A denotesadiagonalmatrixwhosediagonalentriesarrangedindecreasingorderaretheeigenvaluesofA,and denotestheconditionnumber. Theorem4.3.7 [6] Let A and B beHermitianand T bepositivedenite.Then s min T jjj A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B jjjjjj AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB jjj ; foreveryunitarilyinvariantnorm,orequivalently, s min T S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B w S AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB ; where s min T denotesthesmallestsingularvalueof T 4.4Newmajorizationinequalitiesforsingularvaluesandeigenvalues Inthissection,weapplytheresultsofSections4.2and4.3toderiveweakmajorizationinequalitiesforthedierenceofsingularvaluesandeigenvaluesofmatrices. Lemma4.4.1 If A and B aresquarematrices,then S j AB j 1 = 2 pol w S 1 = 2 A S 1 = 2 B : Proof: First,weshowthat S j T j 1 = 2 pol = S 1 = 2 T forall T 2 C n n : Since j T j pol is semi-positivedeniteandHermitian,itfollowsthat 1 = 2 j T j pol = S 1 = 2 j T j pol .Also, itfollowsthat j T j 1 = 2 pol issemi-positivedeniteandHermitian.Therefore, S j T j 1 = 2 pol = 77

PAGE 87

j T j 1 = 2 pol = 1 = 2 j T j pol : Moreover,itiseasytocheckthat S j T j pol = S T : So, wehave S j T j 1 = 2 pol = 1 = 2 j T j pol = S 1 = 2 T : Setting T = AB; itfollowsthat S j AB j 1 = 2 pol = S 1 = 2 AB : ApplyingTheorem4.2.6for S 1 = 2 AB ,weobtaintheresult. Lemma4.4.2 ForasquarematrixA,wehave S re A w S A : Proof: FromTheorem4.2.2wehave S re A = S A + A H 2 w S A 2 + S A H 2 : Since S A = S A H ,wehave S re A w S A : ThefollowingtheoremisinspiredbyBhatia,KittanehandLi[7].Weusethe sametechniqueasBhatia,KittanehandLiusedin[7,Theorem2.2]toprovethe majorizationinequalityofsingularvaluesof AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB and T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : Theorem4.4.3 Let A and B beHermitianand T bepositivedenite.Then S 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B w S AT )]TJ/F18 11.9552 Tf 11.956 0 Td [(TB S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : Proof: Since A and B areHermitian,wehave A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B 2 pol = A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B 2 .Thus, S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B = S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B 2 1 = 2 pol = S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 T 1 = 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B 1 = 2 pol : ApplyingTheorem4.3.5,wehave S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B w S T 1 = 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B 2 T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 1 = 2 pol : 78

PAGE 88

UsingLemma4.4.1,itfollowsthat S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B w S T 1 = 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B 2 T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 1 = 2 pol w S 1 = 2 )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(T 1 = 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T 1 = 2 S 1 = 2 )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = 2 : Sincethenondecreasingconvexfunctionspreservetheweakmajorization,e.g., x 2 for x 0 ; wesquarebothsidestoget S 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B w S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(T 1 = 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T 1 = 2 S )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 2 : UsingTheorem4.3.4, S 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B w S re[ A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T ] S )]TJ/F15 11.9552 Tf 5.479 -9.683 Td [(re T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.956 0 Td [(B : .4.1 Since BT )]TJ/F18 11.9552 Tf 10.475 0 Td [(TB and BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 )]TJ/F18 11.9552 Tf 10.475 0 Td [(T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 B areskew-Hermitianmatrices,i.e., C H = )]TJ/F18 11.9552 Tf 9.299 0 Td [(C ,thus, re AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB =re[ A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T + BT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB ]=re[ A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B T ].4.2 and re )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 =re T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B )]TJ/F24 11.9552 Tf 11.955 9.683 Td [()]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 )]TJ/F18 11.9552 Tf 11.955 0 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 B =re T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B : .4.3 ByLemma4.4.2,weobtain S [re AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB ] w S AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB ; .4.4 S re )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 w S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : .4.5 Substituting4.4.2and4.4.3into4.4.1,togetherwith4.4.4and4.4.5,we obtaintheresultasdesired. Remark4.4.4 In [7,Theorem2.2] ,theauthorsprovetheinequalityforunitarily invariantnormsuchthat jjj A )]TJ/F18 11.9552 Tf 12.619 0 Td [(B jjj 2 jjj AT )]TJ/F18 11.9552 Tf 12.619 0 Td [(TB jjjjjj T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 12.618 0 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 jjj ,whichis equivalentto see[7,Proposition2.1] k X i =1 s i A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B 2 k X i =1 s i AT )]TJ/F18 11.9552 Tf 11.956 0 Td [(TB k X i =1 s i )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 79

PAGE 89

for k =1 ; 2 :::;n: However,theweakmajorizationinequalityinourTheorem4.4.3 impliesthat k X i =1 s 2 i A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B k X i =1 )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(s i AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB s i )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 for k =1 ; 2 :::;n: Lemma4.4.5 UndertheassumptionsofTheorem4.4.3,wehave S 2 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B w S 2 T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 S 2 AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB : Proof: Since T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.956 0 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 ,wehave S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 w S )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 S AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : ByTheorem4.4.3,thislemmaisproved. Lemma4.4.6 Let A and B beHermitianand T beinvertible.Wehave 1. s min T j A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B j w S AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB : 2. j A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B j 2 w S AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB S T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 : 3. j A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B j 2 w S 2 T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 S 2 AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB : Proof: LettheSVDof T be U V H ,where U and V are n by n unitarymatrices andisdiagonal.Since T isinvertible,itfollowsthatispositivedenite. AT )]TJ/F18 11.9552 Tf 11.956 0 Td [(TB = AU V H )]TJ/F18 11.9552 Tf 11.956 0 Td [(U V H B = U )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(U H AU )]TJ/F15 11.9552 Tf 11.956 0 Td [( V H BV V H : Sincethesingularvaluesareinvariantunderunitarytransforms,wehave S AT )]TJ/F18 11.9552 Tf 11.955 0 Td [(TB = S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(U H AU )]TJ/F15 11.9552 Tf 11.955 0 Td [( V H BV : .4.6 Taking A := U H AU B := V H BV ,and T :=inTheorem4.3.7,weobtain s min S U H AU )]TJ/F18 11.9552 Tf 11.955 0 Td [(V H BV w S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(U H AU )]TJ/F15 11.9552 Tf 11.955 0 Td [( V H BV : .4.7 80

PAGE 90

ApplyingTheorem4.2.4,wehave j A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B j = j U H AU )]TJ/F15 11.9552 Tf 11.955 0 Td [( V H BV j w S )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(U H AU )]TJ/F18 11.9552 Tf 11.955 0 Td [(V H BV : .4.8 Combining4.4.6,4.4.7,and4.4.8,weobtaintherststatement. Next,weprovethesecondstatement.Wehave T )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = V )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 U H A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BV )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 U H = V )]TJ/F15 11.9552 Tf 5.48 -9.684 Td [( )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 U H AU )]TJ/F18 11.9552 Tf 11.956 0 Td [(V H BV )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 U H : Therefore,wehave S T )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 A )]TJ/F18 11.9552 Tf 11.955 0 Td [(BT )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = S )]TJ/F15 11.9552 Tf 5.479 -9.684 Td [( )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 U H AU )]TJ/F18 11.9552 Tf 11.955 0 Td [(V H BV )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : ByTheorem4.4.3,wehave S 2 )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(U H AU )]TJ/F18 11.9552 Tf 11.956 0 Td [(V H BV w S )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(U H AU )]TJ/F15 11.9552 Tf 11.955 0 Td [( V H BV S )]TJ/F15 11.9552 Tf 5.48 -9.683 Td [( )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 U H AU )]TJ/F18 11.9552 Tf 11.955 0 Td [(V H BV )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : Byusing4.4.8,wegetthesecondargument.Similarly,applyingLemma4.4.5we obtainthethirdstatement. Remark4.4.7 ItisworthnotingthatTheorem4.2.4isonespecialcaseoftherst resultofLemma4.4.6.InLemma4.4.6,ifwetake T asanidentitymatrix,therst resultofLemma4.4.6becomes j A )]TJ/F15 11.9552 Tf 11.955 0 Td [( B j w S A )]TJ/F18 11.9552 Tf 11.955 0 Td [(B : 4.5Perturbationanalysisforprincipalangles Inthissection,wedealwiththeabsolutechangeintheprincipalanglesbetween thesubspaces X and Y ,wherethesubspace X isperturbedtoobtainasubspace ~ X .WeestablishsomeperturbationboundsforPABSusingmajorizationtechniques. First,westartwithareviewofinequalitiesfortheabsolutechangesofanglesbetween vectors.Forvectors x and y ,ifthevector x isperturbedtothenewvector~ x ,we havesee[50,66] j x;y )]TJ/F18 11.9552 Tf 11.955 0 Td [( ~ x;y j x; ~ x : Theperturbationboundforthecosinesofanglesis j cos x;y )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos ~ x;y j sin x; ~ x 81

PAGE 91

andtheperturbationboundthesinesofanglesis j sin x;y )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ x;y j sin x; ~ x : Forthesubspaces X and Y ,letthesubspace X beperturbedtothenewsubspace ~ X withdim X =dim ~ X =dim Y = p .Thereexistresultsformajorization inequalitiesforPABSsimilartotheanglesbetweenvectors,e.g.,see[50,66,82]. Theorem4.5.1 [66] j X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( ~ X ; Y j w X ; ~ X : Wenotethattheangles X ; Y ~ X ; Y ,and X ; ~ X areinnonincreasingorder. Theleftsideoftheabovemajorizationinequalityis j 1 X ; Y )]TJ/F18 11.9552 Tf 11.956 0 Td [( 1 ~ X ; Y j ;:::; j p X ; Y )]TJ/F18 11.9552 Tf 11.955 0 Td [( p ~ X ; Y j : Theorem4.5.1dealswiththeprincipalanglesthemselves.Thenexttwotheorems dealwiththesinesandcosinesoftheprincipalangles. Theorem4.5.2 [50] cos X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos ~ X ; Y w sin X ; ~ X : Theorem4.5.3 [50,82] sin X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ X ; Y w sin X ; ~ X : Remark4.5.4 Generally,wehavesimilarresultsforperturbationboundsforanglesinthe A -basedscalarproductasthoseforanglesinthestandardproduct. ByLemma3.4.7,weknowthattheprincipalanglesbetweenthesubspaces X and Y relativetothe A -basedscalarproductareequaltotheprincipalanglesbetween K X and K Y inthestandardproduct,where A = K H K: ByTheorem4.5.1,we have j K X ;K Y )]TJ/F15 11.9552 Tf 13.568 0 Td [( K ~ X ;K Y j w K X ; ~ K X ; whichimplies j A X ; Y )]TJ/F15 11.9552 Tf -422.702 -23.98 Td [( A ~ X ; Y j w A X ; ~ X : Therefore,weconcludethatiftheperturbationboundsfor theabsolutechangeofprincipalanglesinthestandardproducthold,thenwehave similarperturbationboundsfortheabsolutechangeofprincipalanglesinthe A -based scalarproduct.Forthisreason,inthefollowingweonlydealwiththeperturbation boundsfortheprincipalanglesinthestandardproduct. 82

PAGE 92

WhathappenswiththetangentsofPABS,ifonesubspaceisperturbed?Tosee whattoexpect,wederiveboundsforvectorsrst.Let x;y <= 2and ~ x;y < = 2.Bydirectcalculation, tan x;y )]TJ/F15 11.9552 Tf 11.955 0 Td [(tan ~ x;y = sin x;y cos ~ x;y )]TJ/F15 11.9552 Tf 11.956 0 Td [(cos x;y sin ~ x;y cos x;y cos ~ x;y = sin x;y )]TJ/F18 11.9552 Tf 11.955 0 Td [( ~ x;y cos x;y cos ~ x;y : Moreover, j sin x;y )]TJ/F18 11.9552 Tf 11.955 0 Td [( ~ x;y j =sin j x;y )]TJ/F18 11.9552 Tf 11.955 0 Td [( ~ x;y j sin x; ~ x : Noticethat x;y )]TJ/F18 11.9552 Tf 11.955 0 Td [( ~ x;y 2 )]TJ/F18 11.9552 Tf 9.299 0 Td [(= 2 ;= 2 : Thus, j tan x;y )]TJ/F15 11.9552 Tf 11.955 0 Td [(tan ~ x;y j sin x; ~ x cos x;y cos ~ x;y : .5.1 Inparticular,assumingthat x;y = 4and ~ x;y = 4,wehave cos x;y cos ~ x;y = cos x;y + ~ x;y +cos x;y )]TJ/F18 11.9552 Tf 11.955 0 Td [( ~ x;y 2 : Sincecosineisdecreasingin[0 ;= 2,weobtain j tan x;y )]TJ/F15 11.9552 Tf 11.955 0 Td [(tan ~ x;y j 2tan x; ~ x : .5.2 InthefollowingTheorem,wereplacesubspacesspannedbythevectors x y ,and ~ x withmultidimensionalsubspaces X Y ,and ~ X ,correspondingly.Weextendthe inequalitiesof4.5.1and4.5.2tothegeneralcase. Theorem4.5.5 If dim X =dim ~ X =dim Y X ; Y <= 2 and ~ X ; Y < = 2 ,then tan X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(tan ~ X ; Y w sin X ; ~ X cos max X ; Y cos max ~ X ; Y : 83

PAGE 93

Inaddition,if X ; Y = 4 and ~ X ; Y = 4 ,wehave tan X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(tan ~ X ; Y w 2sin X ; ~ X cos max X ; ~ X : Proof: ConsideringthepropertiesofthetangentsofPABS,e.g.,seeTable2.2,we havetan X ; Y =[ S + P Y ? P Y P X y ; 0 ;:::; 0],andsimilarly tan ~ X ; Y =[ S + P Y ? P Y P ~ X y ; 0 ;:::; 0] : Thedierencebetweentan X ; Y andtan ~ X ; Y canbewrittenas S P Y ? P Y P X y )]TJ/F18 11.9552 Tf 11.955 0 Td [(S P Y ? P Y P ~ X y exceptforsomezerosingularvalues. S P Y ? P Y P X y )]TJ/F18 11.9552 Tf 11.955 0 Td [(S P Y ? P Y P ~ X y w S P Y ? P Y P X y )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y ? P Y P ~ X y w S P Y ? S P Y P X y )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P ~ X y : TherstmajorizationinequalityisbasedonTheorem4.2.3.FromTheorem4.2.5, weobtainthesecondmajorizationinequality.Thedierencebetween P Y P X y and P Y P ~ X y canbetreatedasaperturbationforobliqueprojections[79]orasaperturbationofpseudoinverse[70].WeusethetechniqueofWedin[79].Wehave P Y P X y )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P ~ X y = P Y P X y I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P ~ X y )]TJ/F15 11.9552 Tf 11.956 0 Td [( I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P X y P Y P ~ X y : Weknowthat P Y P X y isanobliqueprojectoronto X along Y ? and I )]TJ/F15 11.9552 Tf 11.961 0 Td [( P Y P X y is anobliqueprojectoronto Y ? along X .Thus, P Y P X y = P Y P X y P Y and I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P X y = I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P X y I )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X : Similarly,wehave P Y P ~ X y = P ~ X P Y P ~ X y and I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P ~ X y = I )]TJ/F18 11.9552 Tf 11.955 0 Td [(P Y I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P ~ X y : Therefore, S )]TJ/F15 11.9552 Tf 5.48 -9.684 Td [( P Y P X y )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P ~ X y = S )]TJ/F21 11.9552 Tf 5.48 -9.684 Td [()]TJ/F15 11.9552 Tf 9.298 0 Td [( I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P X y I )]TJ/F18 11.9552 Tf 11.955 0 Td [(P X P ~ X P Y P ~ X y w S )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P X y S P X ? P ~ X S )]TJ/F15 11.9552 Tf 5.48 -9.684 Td [( P Y P ~ X y w I )]TJ/F15 11.9552 Tf 11.955 0 Td [( P Y P X y P Y P ~ X y S P X ? P ~ X : 84

PAGE 94

TherstmajorizationinequalityfollowsfromTheorem4.2.5.Thus, tan X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(tan ~ X ; Y w sin X ; ~ X cos max X ; Y cos max ~ X ; Y : Furthermore,let X ; Y = 4and ~ X ; Y = 4,bydirectcomputation,wehave cos max X ; Y cos max ~ X ; Y cos max X ; ~ X : InSection3,weproposeseveralalgorithmstocomputeprincipalanglesbasedon thesineofhalf-PABS.Nextwepresentperturbationboundsforthesineandcosine ofhalf-PABS,wheresubspacesareperturbed. Proposition4.5.6 cos 2 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos 2 1 2 ~ X ; Y w sin 1 2 X ; ~ X : sin 2 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin 2 1 2 ~ X ; Y w sin 1 2 X ; ~ X : Proof: Sincecos 2 = 2=+cos = 2andsin 2 = 2= )]TJ/F15 11.9552 Tf 12.21 0 Td [(cos = 2,byTheorem4.5.2itiseasytoobtainthefollowingtworesults, cos 2 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos 2 1 2 ~ X ; Y w 1 2 sin X ; ~ X and sin 2 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.956 0 Td [(sin 2 1 2 ~ X ; Y w 1 2 sin X ; ~ X : Moreover,sin X ; ~ X =2sin X ; ~ X = 2 cos X ; ~ X = 2 : ResultsinProposition4.5.6areweakmajorizationestimatesforthesquareof sinesorcosinesofhalfprincipalangles.Next,wedealwiththeperturbationboundfor sinesorcosinesofhalf-angles.Beforeweproceed,wepresentthefollowingtheorem, e.g.,[70],fromwhichtwonewresultsarederived. Theorem4.5.7 [70] Let X;Y 2 C n p with X H X = I and Y H Y = I .If 2 p n 85

PAGE 95

thereareunitarymatrices Q U and V suchthat QXU = p n )]TJ/F18 11.9552 Tf 11.955 0 Td [(p 2 6 4 I 0 3 7 5 and QYV = p p n )]TJ/F15 11.9552 Tf 11.955 0 Td [(2 p 2 6 6 6 6 4 )]TJ 0 -23.98 Td [( 0 3 7 7 7 7 5 ; .5.3 where )-278(=diag 1 ;:::; p and =diag 1 ;:::;! p satisfy 0 1 ::: p ; 1 ::: p 0; 2 k + 2 k =1 ;k =1 ;:::;p: Lemma4.5.8 Let 1 ::: p and ^ 1 ::: ^ p besinesofhalfoftheprincipal anglesbetweensubspaces X ; Y and ~ X ; Y ,correspondingly.Then,for k =1 ;:::;p j k )]TJ/F15 11.9552 Tf 12.57 0 Td [(^ k j sin 1 2 max X ; ~ X : .5.4 Proof: Withoutlossofgenerality,weassumethatthedimensionofthesubspaces X ; Y and ~ X islessthan n= 2,otherwisewecoulddealwiththecomplementarysubspacesof X ; Y and ~ X .Letthecolumnsof X and Y beorthonormalbasesforthe subspaces X and Y ,correspondingly.ApplyingTheorem4.5.7,wehave S XU )]TJ/F18 11.9552 Tf 11.955 0 Td [(YV = S I )]TJ/F15 11.9552 Tf 11.955 0 Td [()]TJ/F21 11.9552 Tf 21.511 0 Td [()]TJ/F15 11.9552 Tf 9.298 0 Td [(0 H ; where U;V; )-355(andareasdescribedinTheorem4.5.7and)-327(=cos X ; Y .Furthermore,byTheorem3.2.1 S XU )]TJ/F18 11.9552 Tf 12.856 0 Td [(YV =2sin X ; Y = 2.Inasimilarway, weobtain2sin ~ X ; Y = 2= S I )]TJ/F15 11.9552 Tf 12.685 3.022 Td [(^ )]TJ/F21 11.9552 Tf 21.511 0 Td [()]TJ/F15 11.9552 Tf 10.599 3.022 Td [(^ 0 H ,where ^ )-408(=cos ~ X ; Y : 86

PAGE 96

Therefore, 2sin 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(2sin 1 2 ~ X ; Y = S 0 B B B B @ 2 6 6 6 6 4 I )]TJ/F15 11.9552 Tf 11.955 0 Td [()]TJ/F21 11.9552 Tf -20.715 -23.98 Td [()]TJ/F15 11.9552 Tf 9.299 0 Td [( 0 3 7 7 7 7 5 1 C C C C A )]TJ/F18 11.9552 Tf 11.955 0 Td [(S 0 B B B B @ 2 6 6 6 6 4 I )]TJ/F15 11.9552 Tf 12.685 3.022 Td [(^ )]TJ/F21 11.9552 Tf -20.715 -23.98 Td [()]TJ/F15 11.9552 Tf 10.599 3.022 Td [(^ 0 3 7 7 7 7 5 1 C C C C A w S 0 B B B B @ ^ )]TJ/F21 11.9552 Tf 9.971 0 Td [()]TJ/F15 11.9552 Tf 11.955 0 Td [()]TJ -20.625 -20.958 Td [(^ )]TJ/F15 11.9552 Tf 11.955 0 Td [( 0 1 C C C C A = S r ^ )]TJ/F21 11.9552 Tf 9.971 0 Td [()]TJ/F15 11.9552 Tf 11.955 0 Td [()]TJ/F24 11.9552 Tf 7.315 13.271 Td [( 2 + ^ )]TJ/F15 11.9552 Tf 11.955 0 Td [( 2 : Wenotethat ^ )]TJ/F21 11.9552 Tf 9.971 0 Td [()]TJ/F15 11.9552 Tf 11.955 0 Td [()]TJ/F24 11.9552 Tf 7.314 13.271 Td [( 2 + ^ )]TJ/F15 11.9552 Tf 11.955 0 Td [( 2 isadiagonalmatrixwithdiagonalelements 2 )]TJ/F15 11.9552 Tf 11.955 0 Td [(2cos k X ; Y )]TJ/F18 11.9552 Tf 11.956 0 Td [( k ~ X ; Y for k =1 ;:::;p: Hence, 2sin 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(2sin 1 2 ~ X ; Y w 2sin 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( ~ X ; Y : AccordingtoTheorem4.5.1, max k =1 ;:::;p k X ; Y )]TJ/F18 11.9552 Tf 11.955 0 Td [( k ~ X ; Y max X ; ~ X : Therefore, 2sin 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.956 0 Td [(2sin 1 2 ~ X ; Y w 2sin 1 2 max X ; ~ X ; whichcompletestheproof. Lemma4.5.9 Let 1 ::: p and ^ 1 ::: ^ p becosinesofhalfoftheprincipal anglesbetweensubspaces X ; Y and ~ X ; Y ,correspondingly.Then,for k =1 ;:::;p j k )]TJ/F15 11.9552 Tf 12.875 0 Td [(^ k j sin 1 2 max X ; ~ X : .5.5 Proof: InthenotationoftheproofofLemma4.5.8,wehave S XU + YV =2cos X ; Y = 2 : TherestofproofissimilartotheproofinLemma4.5.8. 87

PAGE 97

Remark4.5.10 In [82] ,ZhangandLidevelopedmajorizationinequalitiesforconcavefunctionsofprincipalangles.Theystatethatif f :[0 ;= 2] [0 ; 1 isa nondecreasingconcavefunction,then f X ; Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(f ~ X ; Y w f X ; ~ X : Since sin x= 2 isanondecreasingconcavefunctionon [0 ;= 2] ,wehave sin 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin 1 2 ~ X ; Y w sin 1 2 X ; ~ X : AlthoughtheresultweobtaininLemma4.5.8isnotasstrongastheaboveweak majorizationinequalityofsinesofhalf-PABS,weuseadierenttechniquetoprove 4.5.4 .Furthermore,weusethesametechniquetoprovetheboundforthechangeof cosinesofhalf-PABSinLemma4.5.9. Proposition4.5.11 Let a =cos X ; Y = 2 )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos ~ X ; Y = 2 b =sin X ; Y = 2 )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ X ; Y = 2 : Then [ j a j ; j b j ] w p 2sin 1 2 X ; ~ X ; 0 ;:::; 0 : Proof: Letcolumnsofmatrices X ^ X and Y beorthonormalbasesforthesubspaces X ~ X and Y ,correspondingly.LettheSVDof X H ^ X be U V H .ByTheorem3.1.1 wehave p 2 cos 1 2 X ; Y ; sin 1 2 X ; Y = S [ XUY ] and p 2 cos 1 2 ~ X ; Y ; sin 1 2 ~ X ; Y = S [ ^ XVY ] : 88

PAGE 98

Wemayneedtoaddzerostothevectorsontherightside.Moreover,wehave S [ XUY ] )]TJ/F18 11.9552 Tf 11.956 0 Td [(S [ ^ XVY ] w S [ XU )]TJ/F15 11.9552 Tf 15.331 3.022 Td [(^ XV 0 ] =[ S XU )]TJ/F15 11.9552 Tf 15.332 3.022 Td [(^ XV ; 0 ;:::; 0] =[ S XUV H )]TJ/F15 11.9552 Tf 15.332 3.022 Td [(^ X ; 0 ;:::; 0] = 2sin 1 2 X ; ~ X ; 0 ;:::; 0 : FromTheorem4.2.3,weobtainweakmajorizationfortherstinequality.Thelast equalityisobtainedbyTheorem3.2.1.Let a =cos X ; Y = 2 )]TJ/F15 11.9552 Tf 12.17 0 Td [(cos ~ X ; Y = 2 and b =sin X ; Y = 2 )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ X ; Y = 2 : Thenwehave p 2[ j a j ; j b j ] w 2sin 1 2 X ; ~ X ; 0 ;:::; 0 ; whichcompletestheproof. Remark4.5.12 Let a =[ a 1 ;:::;a p ] and b =[ b 1 ;:::;b p ] : Fromthedenitionof majorization,wehave k X 1 j a i j # + k X 1 j b i j # p 2 k X 1 sin 1 2 # i X ; ~ X ; 1 k p; where a i =cos i X ; Y = 2 )]TJ/F15 11.9552 Tf 11.956 0 Td [(cos i ~ X ; Y = 2 ; b i =sin i X ; Y = 2 )]TJ/F15 11.9552 Tf 11.956 0 Td [(sin i ~ X ; Y = 2 : Moreover,itisworthnotingthattheboundforthesumoftheabsolutechangesofthe sinesofhalf-PABSandtheabsolutechangesofcosinesofhalf-PABSinProposition 4.5.11aresharp,sincetheconstantontherightsideofProposition4.5.11is p 2 and itwillbeatleast 2 bycombiningLemmas4.5.8and4.5.9. Allresultsfortheperturbationboundsforchangesofanglespresentedaboveuse theconditionthatoneofsubspacesisperturbed.Nowweassumebothsubspaces X and Y areperturbedtosubspaces ~ X and ~ Y ,correspondingly.Weobtainthefollowing lemmawhichcanbeeasilyobtainedfromTheorems4.5.1,4.5.2,and4.5.3. 89

PAGE 99

Corollary4.5.13 If dim X =dim ~ X =dim Y =dim ~ Y ,wehave 1. j X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( ~ X ; ~ Y j w X ; ~ X + Y ; ~ Y : 2. cos X ; Y )]TJ/F15 11.9552 Tf 11.956 0 Td [(cos ~ X ; ~ Y w sin X ; ~ X +sin Y ; ~ Y : 3. sin X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin ~ X ; ~ Y w sin X ; ~ X +sin Y ; ~ Y : Proof: Weprovetherststatement, j X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( ~ X ; ~ Y j = j X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( ~ X ; Y + ~ X ; Y )]TJ/F15 11.9552 Tf 11.956 0 Td [( ~ X ; ~ Y j j X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( ~ X ; Y j + j ~ X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( ~ X ; ~ Y j w X ; ~ X + Y ; ~ Y : Theprooffortheothertwoissimilar. Similarly,weboundthesquaresofsinesandcosinesofhalf-PABSasfollows. Corollary4.5.14 cos 2 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos 2 1 2 ~ X ; ~ Y w sin 1 2 X ; ~ X +sin 1 2 Y ; ~ Y : sin 2 1 2 X ; Y )]TJ/F15 11.9552 Tf 11.955 0 Td [(sin 2 1 2 ~ X ; ~ Y w sin 1 2 X ; ~ X +sin 1 2 Y ; ~ Y : Theorem4.5.15 Let 1 ::: p and ~ 1 ::: ~ p besinesofhalfoftheprincipal anglesbetweensubspaces X ; Y and ~ X ; ~ Y ,correspondingly.Then,for k =1 ;:::;p j k )]TJ/F15 11.9552 Tf 12.569 0 Td [(~ k j sin 1 2 max X ; ~ X +sin 1 2 max Y ; ~ Y : .5.6 Proof: First,byLemma4.5.8,for k =1 ;:::;p wehave j k )]TJ/F15 11.9552 Tf 12.57 0 Td [(^ k j sin 1 2 max X ; ~ X : Second,weapplyasimilarstatementtothesineofhalf-PABSbetweensubspaces ~ X ; Y and ^ X ; ~ Y ,correspondingly.Wehave j ^ k )]TJ/F15 11.9552 Tf 12.57 0 Td [(~ k j sin 1 2 max Y ; ~ Y : 90

PAGE 100

Usingthetriangleinequality,weobtaintheresultasdesired. Similarly,wecanestablishanestimateofthecosineofhalf-PABSwithrespect toperturbationsoftwosubspaces. Theorem4.5.16 Let 1 ::: p and ~ 1 ::: ~ p becosinesofhalfofthe principalanglesbetweensubspaces X ; Y and ~ X ; ~ Y ,correspondingly.Then,for k = 1 ;:::;p j k )]TJ/F15 11.9552 Tf 12.876 0 Td [(~ k j sin 1 2 max X ; ~ X +sin 1 2 max Y ; ~ Y : .5.7 Letusemphasizethatwecanobtainsimilarperturbationboundsforsinesor cosinesofhalf-PABSwithrespecttothe A -basedscalarproductasinLemmas4.5.8 and4.5.9,andTheorems4.5.15and4.5.16. Letusnowturnourattentiontoaperturbationanalysisintermsofmatrices X 1 and X 1 + X 1 thatgeneratesubspaces X and ~ X .Oftenasubspaceisdenedby thecolumnspaceofamatrix.Weareinterestedinestimatingthesensitivityofa columnspaceofamatrix.ThereareseveralresultsprovidedbySun[74],andGolub andZha[29]involvingtheperturbationanalysisfortheprincipalanglesofapairof matrices.Theboundsin[74]areimprovedin[29]withoutrestrictingthedimensions oftwosubspacestobeequal.Moreover,KnyazevandArgentati[48]generalizeresults of[29,74]tothe A -basedscalarproduct. Lemma4.5.17 [29,48,74] Let X and ~ X becolumn-spacesofmatrices X 1 and X 1 + X 1 ,correspondingly.Let A X 1 = s max A 1 = 2 X 1 =s min A 1 = 2 X 1 denotethe corresponding A -basedconditionnumberof X 1 .Then sin A max X ; ~ X A X 1 k A 1 = 2 X 1 k k A 1 = 2 X 1 k : Inparticular,for A = I ,wehave sin max X ; ~ X X 1 k X 1 k k X 1 k : 91

PAGE 101

Sincesin =2sin = 2cos = 2 2sin 2 = 2for 2 [0 ;= 2],ourtheoremsfor sinesandcosinesofhalf-PABStriviallyfollownext. Theorem4.5.18 UndertheassumptionsofTheorem4.5.15, j k )]TJ/F15 11.9552 Tf 12.57 0 Td [(~ k j 1 p 2 X 1 k X 1 k k X 1 k + 1 p 2 Y 1 k Y 1 k k Y 1 k ; for k =1 ;:::;p: Theorem4.5.19 UndertheassumptionsofTheorem4.5.16, j k )]TJ/F15 11.9552 Tf 12.876 0 Td [(~ k j 1 p 2 X 1 k X 1 k k X 1 k + 1 p 2 Y 1 k Y 1 k k Y 1 k ; for k =1 ;:::;p: Remark4.5.20 Thesetheoremsshowthattheaccuracyinvolvedincomputingthe sinesorcosinesforhalf-PABSdependsontheconditionnumberofmatrices X 1 and Y 1 .Thus,ill-conditionedmatrices X 1 and Y 1 mayproducelargeerrors.Sincecolumn scalingdoesnotchangethecolumnspaceofmatrix,columnscalingofmatrices X 1 and Y 1 mayhelptoimprovetheconditionnumber. Remark4.5.21 Wehavesimilarresultsforthesinesorcosinesofhalf-PABSinthe A -basedscalarproduct.Weusethenotation k and ~ k asinTheorem4.5.15and k and ~ k asinTheorem4.5.16,butwecompute k ~ k and k ~ k inthe A -basedscalar product. j k )]TJ/F15 11.9552 Tf 12.57 0 Td [(~ k j 1 p 2 A X 1 k A 1 = 2 X 1 k k A 1 = 2 X 1 k + 1 p 2 A Y 1 k A 1 = 2 Y 1 k k A 1 = 2 Y 1 k ; j k )]TJ/F15 11.9552 Tf 12.876 0 Td [(~ k j 1 p 2 A X 1 k A 1 = 2 X 1 k k A 1 = 2 X 1 k + 1 p 2 A Y 1 k A 1 = 2 Y 1 k k A 1 = 2 Y 1 k ; for k =1 ;:::;p: 4.6Conclusions Inthischapter,wehavederivedtheboundsforthesingularvaluesandtheeigenvaluesoftwoHermitianmatricesrelatedtoamatrixintermsofHermitianmatrices 92

PAGE 102

andaninvertiblematrix.Wehavepresentedaperturbationanalysisforsinesand cosinesofhalf-PABS.TheperturbationanalysisisbasedonPABSinthestandard product.ExtensionoftheresultstoPABSinthe A -basedscalarproductisstraightforward. 93

PAGE 103

5.BoundsfortheRayleighquotientandthespectrumofself-adjoint operators 3 5.1Introduction Let A be aboundedself-adjointoperator inarealorcomplexHilbertspace H weuseoperatorsinthischapterandwedealwithmatricesforother chapters .Foranonzerovector x ,theRayleighquotientRQisdenedby x = x;A = h x;Ax i h x;x i andthecorrespondingresidualvectorisdenotedby r x = r x;A = Ax )]TJ/F18 11.9552 Tf 12.416 0 Td [( x x; where h ; i isaninnerproduct,associatedwithanorm kk 2 = h ; i : Theacuteangle betweentwononzerovectors x and y isdenotedby f x;y g =arccos jh x;y ij k x kk y k : WeareinterestedintheRQvectorperturbationbounds.Specically,fornonzero vectors x and y ,wewanttoboundthefollowingquantity, j x )]TJ/F18 11.9552 Tf 12.106 0 Td [( y j ,intermsof f x;y g k r x k ,and k r y k : If x isaneigenvectorof A ,thentheRQof x isan exacteigenvalueof A and r x =0 : Inthiscase,theabsolutechangeoftheRQ j x )]TJ/F18 11.9552 Tf 11.178 0 Td [( y j becomestheabsoluteerrorintheeigenvalue x of A .TheRQisoften usedtoapproximatepointsofthespectrumspec A of A Knownboundsofthespectrumaretraditionallyclassieddependingontheterms thatappearinthebounds.Let 2 spec A beapproximatedby y ,i.e.,the absoluteapproximationerroris j )]TJ/F18 11.9552 Tf 11.154 0 Td [( y j : Boundsof j )]TJ/F18 11.9552 Tf 11.154 0 Td [( y j thatarebasedonthe normoftheresidual k r y k arecalledaposteriori"bounds,sincetheresidual r y anditsnormcantypicallybecomputedforthegivenvector y If isaneigenvaluewiththecorrespondingeigenvector x ,boundsof j )]TJ/F18 11.9552 Tf 12.205 0 Td [( y j thatrelyontheangle f x;y g arecalledapriori,"sincetheeigenvector x isusually 3 Thematerialofthischapterisbasedonourmanuscript,P.Zhu,M.E.Argentatiand A.V.Knyazev, BoundsfortheRayleighquotientandthespectrumofself-adjointoperators ,submittedtoSIMAX.Also,itisavailableathttp://arxiv.org/abs/1207.3240. 94

PAGE 104

notexplicitlyknown,andsomeaprioriinformationaboutitneedstobeusedtobound theangle f x;y g .InthecontextoftheFiniteElementMethoderrorbounds,where x and y arefunctions,suchinformationisusuallyassociatedwiththesmoothnessof thefunction x ;see,e.g.,[51].Finally,wecallaboundamixedtype,"ifitinvolves bothterms,theresidualnorm k r y k andtheangle f x;y g Sharpboundsofthespectrumapproximationerrorareveryimportantinthe theoryofnumericalsolutionofself-adjointeigenvalueproblems.Computableaposterioriboundsallowonetoobtaintwo-sidedapproximationstopointsofthespectrum.Aprioriandmixedboundsgiveoneanopportunitytodeterminethequality ofapproximationofapointofthespectrumbytheRQ. Werevisitthisclassicaltopicofresearchfortworeasons.Ourrstgoalisto carefullyexamineaposterioriandaprioriboundstogether,tounderstandwhat theyhaveincommonandwhytheyaresodierent.Second,weareinterestedin discoveringageneralframeworkforderivingboundsof j x )]TJ/F18 11.9552 Tf 12.858 0 Td [( y j forarbitrary vectors x and y whereknownboundsofthespectrumapproximationerrorbecome corollaries. Westart,inSection5.2,byreviewingthreeknownboundsofthespectrumapproximationerror,eachrepresentingapriori,aposteriori,andmixedtypes,correspondingly.InSection5.3,wederivenewidentitiesandboundsforthechangeinthe RQwithrespecttothechangeofthevectors,usinganorthogonalprojectoronthe subspacespan f x;y g : Theideaofsuchalow-dimensional"analysishasbeenfound fruitfulbefore,e.g.,in[46].Weobservethatinthecasedim H =2allthreebounds, whichwewanttoreinvent,turnintothesameidentity.InSection5.4,wederivethe aprioriandmixedtypeboundsfromthisidentity,usingasingleinequality,dierent foreachbound,onlyatthelaststep.Thisunifyingprooftechniqueallowsusto easilyspecifythecircumstanceswherethebondsaresharp.WededicateSection5.5 toaposterioribounds.Ourlow-dimensional"analysisnotonlyallowsustonda 95

PAGE 105

novelproofofawellknownaposterioribound,butalsogivesusanopportunityto improveit,andobtainseveralnewsharperresults.OurbonusinSection5.6touches thetopicofimprovingknownsin andtan errorboundsforeigenvectors. 5.2Shortreviewofsomeknownerrorboundsforeigenvalues Let y beanapproximationtoaneigenvector x oftheoperator A ,corresponding totheeigenvalue x of A: Anaprioriboundinvolvesaconstantandthesquareof thesineoftheanglebetweentheeigenvector x andthevector y ,see,e.g.,[49,53,67], j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j spec max A )]TJ/F15 11.9552 Tf 11.955 0 Td [(spec min A sin 2 f x;y g ; .2.1 wherewedenote = x ,andspec max A andspec min A denotethelargestandthe smallestpointsofthespectrumspec A of A ,correspondingly. Themixedtypeboundshownin[47,75]isintermsofthenormoftheresidual vectorandthetangentoftheanglebetweenvectors x and y ,i.e., j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j k r y k k y k tan f x;y g : .2.2 Thereisnoconstraintonthelocationoftheeigenvalue x relativeto y in5.2.1and5.2.2.Bounds5.2.1and5.2.2holdforanynonzerovector y and anyeigenvector x: Thenextboundwereviewrequireshavingsomeinformationabout thespectrumof A intheneighborhoodof y .Thisaposterioriboundinvolvesthe squareofthenormoftheresidualvectorandthegapbetween y andtherestof thespectrumof A .Letuschooserealnumbers and suchthat < y < and spec A ; = ; ,wherespec A representsthespectrumof A .ThentheTemple boundholds,see,e.g.,[32,44,76], )]TJ/F18 11.9552 Tf 11.955 0 Td [( y y )]TJ/F18 11.9552 Tf 11.955 0 Td [( k r y k 2 k y k 2 : .2.3 TheTemplebound5.2.3turnsintoanequalityifboth and areeigenvaluesof A andthevector y isalinearcombinationofthecorrespondingeigenvectors. 96

PAGE 106

Lettheonlypointofthespectrumspec A intheinterval a;b beaneigenvalue andletalso y 2 a;b ,then )]TJ/F15 11.9552 Tf 31.242 8.088 Td [(1 y )]TJ/F18 11.9552 Tf 11.955 0 Td [(a k r y k 2 k y k 2 y )]TJ/F18 11.9552 Tf 11.955 0 Td [( 1 b )]TJ/F18 11.9552 Tf 11.955 0 Td [( y k r y k 2 k y k 2 : .2.4 ThisisKato-Temple'sinequality[44,76],anditeasilyfollowsfrom5.2.3.Indeed, inthecasewhere islocatedtotherightof y thenwetake = a and = in5.2.3togetthelowerboundof5.2.4.Intheoppositecase,wetake = and = b toobtaintheupperboundof5.2.4. Ifwetake a = y )]TJ/F18 11.9552 Tf 11.081 0 Td [( and b = y + in5.2.4,where =min 2f spec A = g j )]TJ/F18 11.9552 Tf -422.702 -23.98 Td [( y j ,then5.2.4directlyimplies,see,e.g.,[64,68,73], j )]TJ/F18 11.9552 Tf 11.956 0 Td [( y j 1 k r y k 2 k y k 2 : .2.5 Bound5.2.5turnsintoanequalityif )]TJ/F18 11.9552 Tf 12.189 0 Td [( or + isaneigenvalueof A andthe vector y isalinearcombinationofthecorrespondingto )]TJ/F18 11.9552 Tf 12.397 0 Td [( or + eigenvector andaneigenvectorcorrespondingto Sinceamongallpointsofthespectrumof A ,theisolatedpoint 2 spec A is theclosestoneto y ,weevidentlyhavefrom5.2.5that j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j 2 j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j k r y k 2 k y k 2 ; whichgivestheKrylov-Weinsteinbound,see,e.g.,[13,64] j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j k r y k k y k : ThisderivationoftheKrylov-Weinsteinboundfrombound5.2.5,althoughtrivial, isnotapparentlyverywellknown.ItdemonstratesthattheKrylov-Weinsteinbound issimplyaparticularcaseof5.2.5,where j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j : Ourdiscussionaboveshowsthatbound5.2.3isthemostfundamentalaposterioribound,sinceotheraposterioribounds,reviewedhere,canbederivedfromit.The maingoaloftherestofthischapteristorevisitbounds5.2.1,5.2.2,and5.2.3. 97

PAGE 107

Weproposeanewunifyingapproachtoprovethem,analyzetheirsharpness,and derivesomenewsharperbounds. 5.3KeyidentitiesforRQ WepresentseveralkeyidentitiesfortheabsolutechangeintheRQ. Let A beaboundedself-adjointoperatoronarealorcomplexHilbertspace H .Let F denoteasubspaceof H and P F beanorthogonalprojectoron F .Let A F = P F A j F denotetherestrictionoftheoperator P F A tothesubspace F .For anonzerovector x 2F ,wedenote x;A F = h x;A F x i = h x;x i and r x;A F = A F x )]TJ/F18 11.9552 Tf 12.335 0 Td [( x;A F x 2F .Westartwithacoupleoftrivial,butextremelyimportant, forourapproach,lemmas. Lemma5.3.1 [2] If x 2F then x;A F = x Proof: Theorthogonalprojector P F isself-adjoint,so x;A F = h x;P F Ax i h x;x i = h x;Ax i h x;x i = x : Lemma5.3.2 If x 2F then r x;A F = P F r x : Proof: DirectlybythedenitionandLemma5.3.1,weobtain r x;A F = A F x )]TJ/F18 11.9552 Tf 11.955 0 Td [( x;A F x = P F Ax )]TJ/F18 11.9552 Tf 11.955 0 Td [( x x = P F Ax )]TJ/F18 11.9552 Tf 11.955 0 Td [( x x = P F r x : Intherestofthischapter,forgivenlinearlyindependentvectors x and y in H wealwaysdenethesubspace F as F =span f x;y g Corollary5.3.3 Ifoneofthevectors x or y isaneigenvectorof A ,itisalsoan eigenvectorof A F 98

PAGE 108

Proof: Let x beaneigenvectorof A ,then r x =0,so r x;A F = P F r x =0, whichmeansthat x isalsoaneigenvectorof A F Remark5.3.4 Itisimportanttorealizethattheoperator A F actsinatwodimensionalspace,thusmanystatementsconcerning A F canbeeasilyprovedbecauseof this.Here,wecollectsomebasicidentities,usingtheeigenvectorsof A F Letusdenote = max A F = u 1 and = min A F = u 2 ,where u 1 and u 2 areorthogonaleigenvectorsoftheoperator A F .Let P i with i =1 ; 2 denotethe orthogonalprojectorsonthesubspace span u i .Assuminganontrivialcase P i x 6 =0 for i =1 ; 2 ,weevidentlyhave f x;u i g = f x;P i x g .Now, x = h x;A F x i h x;x i = h P 1 x;A F P 1 x i + h P 2 x;A F P 2 x i h P 1 x;P 1 x i + h P 2 x;P 2 x i = h P 1 x;P 1 x i + h P 2 x;P 2 x i h P 1 x;P 1 x i + h P 2 x;P 2 x i since x = P 1 x + P 2 x and P 1 P 2 = P 1 A F P 2 =0 .Therefore, )]TJ/F18 11.9552 Tf 11.955 0 Td [( x = )]TJ/F18 11.9552 Tf 11.955 0 Td [( sin 2 f x;u 1 g and x )]TJ/F18 11.9552 Tf 11.956 0 Td [( = )]TJ/F18 11.9552 Tf 11.955 0 Td [( sin 2 f x;u 2 g : .3.1 Evidently,inatwo-dimensionalspace, cos f x;u 1 g =sin f x;u 2 g and cos f x;u 2 g =sin f x;u 1 g : .3.2 5.3.1Identitiesforthenormoftheresidual r x = Ax )]TJ/F18 11.9552 Tf 11.955 0 Td [( x x Ourmainidentityisinthefollowinglemma. Lemma5.3.5 [ max A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( x ][ x )]TJ/F18 11.9552 Tf 11.956 0 Td [( min A F ]= k P F r x k 2 k x k 2 : .3.3 Proof: Letusdenote = max A F and = min A F .ByLemmas5.3.1and5.3.2, identity5.3.3canbeequivalentlyrewrittenas [ )]TJ/F18 11.9552 Tf 11.955 0 Td [( x;A F ][ x;A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( ]= k r x;A F k 2 k x k 2 : 99

PAGE 109

Since h r x;A F ;x i =0,wehave k r x;A F k 2 = k A F x k 2 )]TJ/F18 11.9552 Tf 11.956 0 Td [( 2 x;A F k x k 2 andthus k r x;A F k 2 k x k 2 )]TJ/F15 11.9552 Tf 11.955 0 Td [( )]TJ/F18 11.9552 Tf 11.955 0 Td [( x;A F x;A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( = k A F x k 2 k x k 2 )]TJ/F15 11.9552 Tf 11.955 0 Td [( + x;A F + = h A F x )]TJ/F18 11.9552 Tf 11.955 0 Td [(x;A F x )]TJ/F18 11.9552 Tf 11.955 0 Td [(x i k x k 2 =0 ; where A F )]TJ/F18 11.9552 Tf 11.975 0 Td [( A F )]TJ/F18 11.9552 Tf 11.975 0 Td [( =0istheminimalpolynomialof A F ,sincedim F =2and and aretheeigenvaluesof A F : Remark5.3.6 Let x beaneigenvectorof A inLemma5.3.5.Thenoneofthetwo eigenvaluesof A F isequalto x : Proof: ThestatementfollowsdirectlyfromLemma5.3.1andCorollary5.3.3. Wealsomentionthefollowingidentity,relyingoneigenvectorsof A F Lemma5.3.7 Let u i with i =1 ; 2 denotetheeigenvectorsof A F .Forboth i =1 and 2 ,wehave 1 2 [ max A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A F ]sin f x;u i g = k P F r x k k x k : .3.4 Proof: ByLemma5.3.2,andusingthenotationofRemark5.3.4,identity5.3.4 canbeequivalentlyrewrittenas 1 2 [ )]TJ/F18 11.9552 Tf 11.956 0 Td [( ]sin f x;u i g = k r x;A F k k x k : ThestatementofthelemmafollowsfromLemma5.3.5andthetrigonometricidentity forthesineofthedoubleangle. 5.3.2IdentitiesfortheabsolutechangeintheRQ Hereweproveourmaintangent-andsine-basedidentities. Remark5.3.8 Let f x;y g <= 2 .Wenotethefollowingusefulidentities k r x k k x k cos f r x ;y g cos f x;y g = jh r x ;y ij jh x;y ij = jh P F r x ;y ij jh x;y ij ; 100

PAGE 110

jh P F r x ;y ij jh x;y ij = k P F r x k k x k cos f P F r x ;y g cos f x;y g ; and k P F r x k k x k cos f P F r x ;y g cos f x;y g = k P F r x k k x k tan f x;y g ; where cos f P F r x ;y g =sin f x;y g .Indeed, 0= h r x ;x i = h P F r x ;x i ,i.e., vectors P F r x 2F and x 2F areorthogonalin F ,andalso y 2F ; where dim F =2 : Letusnallyalsogiveanalternativesimpleidentity, jh r x ;y ij jh x;y ij = jh P y r x ;y ij jh x;y ij = k P y r x k k x k cos f P y r x ;y g cos f x;y g = k P y r x k k x k cos f x;y g ; where P y isanorthogonalprojectoronthesubspace span f y g and f P y r x ;y g =0 : Theorem5.3.9 Let f x;y g <= 2 andletusdenote = k P F r x k k x k k P F r y k k y k tan f x;y g : Wealwayshave )]TJ/F21 11.9552 Tf 10.405 1.793 Td [(j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j + : Now,letusassumethat h x;r y i = h x;y i or h r x ;y i = h x;y i isreal.Then j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j = + if h x;r y i = h r x ;y i 0 or h r x ;y i = h x;r y i 0 ,otherwise, j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j = )]TJ/F18 11.9552 Tf 7.084 1.794 Td [(: Proof: Byelementarycalculations,weobtain x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y h x;y i = h x x;y i)-222(h x; y y i + h x;Ay i)-222(h Ax;y i = h x;r y i)-222(h r x ;y i ; thus j x )]TJ/F18 11.9552 Tf 11.956 0 Td [( y j = jh x;r y i)-222(h r x ;y ij jh x;y ij : .3.5 101

PAGE 111

Inacomplexspace,theRQremainsreal,since A isHermitian,butthescalarproducts maynotbereal,soingeneralwehave,bythetriangleinequality,that jh x;r y ij)-222(jh r x ;y ij jh x;r y i)-222(h r x ;y ijjh x;r y ij + jh r x ;y ij : Unlessoneoftheratios h x;r y i = h x;y i or h r x ;y i = h x;y i isreal,theinequalities abovearestrict.Indeed,thedierenceoftheratiosisequalto x )]TJ/F18 11.9552 Tf 12.437 0 Td [( y ; i.e.,is alwaysreal.Thus,oneoftheratiosisrealifandonlyifoneofthescalarproducts h x;r y i and h r x ;y i isarealscalartimestheother. Theassumption f x;y g <= 2impliesthatatleastoneofthescalarproducts h x;r y i and h r x ;y i isnon-zero.Letitbe,e.g., h r x ;y i .If h x;r y i = h r x ;y i 0 thentheleftinequalityturnsintoanequality.If h x;r y i = h r x ;y i 0thenthe rightinequalityturnsintoanequality.Allothercases,possibleinacomplexspace, leadtostrictinequalitiesabove. ThestatementsofthetheoremnowfollowdirectlyfromRemark5.3.8. Lemma5.3.10 Forboth i =1 and 2 ,wehave j x )]TJ/F18 11.9552 Tf 11.758 0 Td [( y j =[ )]TJ/F18 11.9552 Tf 11.758 0 Td [( ]sin f x;u i g + f y;u i g j sin f x;u i g)]TJ/F27 11.9552 Tf 20.59 0 Td [( f y;u i g j : .3.6 Proof: Identities5.3.1and5.3.2ofRemark5.3.4imply j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j =[ )]TJ/F18 11.9552 Tf 11.955 0 Td [( ] cos 2 f x;u i g )]TJ/F15 11.9552 Tf 11.955 0 Td [(cos 2 f y;u i g ; whichleadsdirectlytoidentity5.3.6usingelementarytrigonometry. Theorem5.3.11 Letusfor i =1 or 2 denote =[ max A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A F ] j sin f x;u i g f y;u i g j sin f x;y g ; andlet C = h x;u 1 ih u 2 ;x ih u 1 ;y ih y;u 2 i .Wealwayshave )]TJ/F21 11.9552 Tf 10.405 1.793 Td [(j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j + : If C isacomplexquantitythenbothinequalitiesontheleftandrightarestrict.If C 0 C 0 thenwehaveequalityforthelowerupperbound. 102

PAGE 112

Proof: Identity5.3.6makesthestatementofthetheoremequivalentto j sin f x;u i g)]TJ/F27 11.9552 Tf 20.589 0 Td [( f y;u i g j sin f x;y g sin f x;u i g + f y;u i g : .3.7 Forsimplicityofnotation,letusassume,withoutlossofgenerality,thatallvectors involvedherearenormalized,i.e., k u 1 k = k u 2 k = k x k = k y k =1.Usingrepresentationsofthevectors x and y withrespecttotheorthonormalbasis u 1 and u 2 ofthe subspace S ,weobtainbydirectcomputation sin 2 f x;y g =cos 2 f x;u 1 g cos 2 f y;u 2 g +cos 2 f x;u 2 g cos 2 f y;u 1 g)]TJ/F15 11.9552 Tf 20.589 0 Td [(2 < C and j C j =cos f x;u 1 g cos f x;u 2 g cos f y;u 1 g cos f y;u 2 g Evidently, j C j< C j C j ,andif C 0 C 0then < C = j C j < C = j C j ,whichprovesbounds5.3.7andalsogivesconditionsfortheirsharpness. Remark5.3.12 AnalternativeproofofTheorem5.3.11isbasedonthefactthat anglesbetweensubspacesdescribeametric,see,e.g. [82] andreferencesthere,so j f x;u g)]TJ/F27 11.9552 Tf 20.589 0 Td [( f y;u gj f x;y gj f x;u g + f y;u gj : .3.8 Theorem5.3.11triviallyimpliesthebound j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j [ max A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A F ]sin f x;y g thatcanbefoundfortherealcasein[49]. Finally,weapplyourresultsabovetoanimportantspecialcase. Corollary5.3.13 Let x beaneigenvectorof A and x = .Then j )]TJ/F18 11.9552 Tf 11.956 0 Td [( y j =[ max A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A F ]sin 2 f x;y g : .3.9 If,inaddition f x;y g <= 2 ,then j )]TJ/F18 11.9552 Tf 11.956 0 Td [( y j = k P F r y k k y k tan f x;y g : .3.10 103

PAGE 113

Ifalso 6 = denotesaneigenvalueof A F ,then tan f x;y g = 1 j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j k P F r y k k y k : .3.11 Proof: Since x isaneigenvectorof A itisalsoaneigenvectorof A F ,withthesame eigenvalue x = ,byCorollary5.3.3.Thestatements5.3.9and5.3.10follow directlyfromTheorems5.3.9and5.3.11,correspondingly.Finally,identity5.3.11 comesfrom5.3.3and5.3.10. 5.4Derivingsomeknowneigenvalueerrorboundsfromouridentities Wenoweasilyderivetwoknownbounds,reviewedinSection5.2,fromtheresultsobtainedinSection5.3.First,sincespec min A min A F max A F spec max A ,from5.3.9weobtainaprioribound5.2.1, j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j =[ max A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A F ]sin 2 f x;y g [spec max A )]TJ/F15 11.9552 Tf 11.955 0 Td [(spec min A ]sin 2 f x;y g : Theinequalitybecomesanequalityifspec max A andspec min A areeigenvaluesof A F .Fordetails,seereference[49]. Thesecondknownresult,mixedbound5.2.2,followsfrom5.3.10, j )]TJ/F18 11.9552 Tf 11.956 0 Td [( y j = k P F r y k k y k tan f x;y g k r y k k y k tan f x;y g : Theinequalitywouldturnintoanequalityif P F r y = r y ,i.e.,if Ay 2 span f x;y g whichisequivalenttothesubspacespan f x;y g being A -invariant.Moredetailed informationaboutthequalityofthisboundisdeterminedbyRemark5.3.8where vectors x and y areswappedtogiveusthefollowingidentity k r y k cos f r y ;x g = k P F r y k cos f P F r y ;x g : 104

PAGE 114

Remark5.4.1 Ourassumptionthattheoperator A isboundedisnecessaryin bound 5.2.1 andcanbeusedinbound 5.2.2 toguaranteetheexistenceofthe vector Ay .OurresultsofSection5.3formallyspeakingwouldholdforanunbounded operator A andgivenvectors x and y assoonastheoperator A F canbecorrectly denedby A F = P F A j F ,where F =span f x;y g .Onecoulddoevenbetterthan that,infact,oneonlyneedstobeabletocorrectlydenethevalues x and y Ratherthanprovidingtechnicaldetails,weillustratesuchapossibilityusingthe followingexamplefromDavisandKahan [15] .Let =1 = 2 and H = l 2 : Wetake y =[1 ;; 2 ; ; n ; ] H 2 l 2 and A =diag ; )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 ; )]TJ/F16 7.9701 Tf 6.586 0 Td [(2 ; ; )]TJ/F19 7.9701 Tf 6.587 0 Td [(n ; .Weget y = 1+ eventhoughthevector Ay =[1 ; 1 ; 1 ; ] H hasaninnitenormin l 2 .Let ustake x =[1 ; 0 ; 0 ; ] H ,theeigenvectorcorrespondingtothelowesteigenvalue x =1 ,anddene w =[0 ;; 2 ; 3 ; ] H with w =+ = .Then A F canbe constructedusingitseigenvectors x and w andthecorrespondingeigenvalues x and w despitethefactthat Aw =[0 ; 1 ; 1 ; ] H doesnotexistin H = l 2 Wehavenowdemonstratedhowtwoknownbounds,aprioribound5.2.1and mixedbound5.2.2,canbederivedinasimpleanduniformmannerfromessentially twodimensionalidentities5.3.9and5.3.10.Thelastwellknownbounds,we reviewinSection5.2,theaposterioriTemplebound5.2.3anditsfollow-up5.2.5, canalsobederivedinasimilarwayfromtheirtwodimensionalprototype5.3.3. However,thederivationisnotsotrivial,sowededicateaseparatesectionforit, comingnext. 5.5Newaposterioriboundsforeigenvalues Forconvenience,letusremindthereaderoftheTemplebound5.2.3 )]TJ/F18 11.9552 Tf 11.955 0 Td [( y y )]TJ/F18 11.9552 Tf 11.955 0 Td [( k r y k 2 k y k 2 ; where < y < andspec A ; = ; ,anditstwodimensionalanalog5.3.3 [ max A F )]TJ/F18 11.9552 Tf 11.955 0 Td [( y ][ y )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A F ]= k P F r y k 2 k y k 2 : 105

PAGE 115

Ourprimarygoalistoderive5.2.3from5.3.3.Werstproveitforanimportant particularcase,wherewecanevenimprove5.2.3. Lemma5.5.1 Let = min A beanisolatedpointofthespectrumof A and > bethenearestlargerpointofthespectrumof A .Let X denotethecompleteeigenspace correspondingtotheeigenvalue : Foragivenvector y 6 =0 ,suchthat < y < wedene x = P X y ,where P X isanorthogonalprojectoronthesubspace X .Assuming that x 6 =0 ,wedenote F =span f x;y g .Then )]TJ/F18 11.9552 Tf 11.955 0 Td [( y y )]TJ/F18 11.9552 Tf 11.955 0 Td [( k P F r y k 2 k y k 2 ; .5.1 where P F isanorthogonalprojectoronthesubspace F Proof: Theassumption y > implies y= 2X since = min A ,so x 6 = y .The vector x isaneigenvectorof A F ,and x = min A = min A F .Let X ? denote theorthogonalcomplementtothesubspace X .Thevector w = y )]TJ/F18 11.9552 Tf 12.123 0 Td [(P X y 2FX ? isaneigenvectorcorrespondingtotheothereigenvalueof A F ,i.e., w = max A F Since X isthecompleteeigenspacecorrespondingtotheeigenvalue = min A ,the factthat w 2X ? guaranteesthat w .Therefore min A F = < y < w = max A F : Substituting max A F for max A F inidentity5.3.3givesthedesiredbound. TheTemplebound5.2.3followsfrombound5.5.1,using k P F r y kk r y k InRemark5.4.1wehavealreadydiscussedthathavingaboundbasedon k P F r y k ratherthanon k r y k maygivegreatadvantages. Ofcourse,Lemma5.5.1canbeeasilyreformulatedforthecasefortheopposite sideofthespectrumoftheoperator A: Returningourattentiontoourmaingoal,we canconstructsimpleexamplesshowingthatbound5.5.1doesnotholdifneither nor correspondstotheextremepointsofthespectrumoftheoperator A .The 106

PAGE 116

extensionofbound5.5.1tothegeneralcaseisdescribedbelow.Westartwitha technicallemma. Lemma5.5.2 Let y = 2 spec A foragivenvector y 6 =0 : Letusdenoteby U the invariantsubspaceof A associatedwithallofthespectrumof A largerthan y ,and dene V = U +span f y g .Then spec A V = v [ spec A U ,andthesmallestpoint ofthespectrum spec min A V < y isanisolatedpointofthespectrum spec A V | aneigenvalueofmultiplicityone. Proof: Since V = U +span f y g ,wehavedim )]TJ/F21 11.9552 Tf 5.479 -9.684 Td [(VU ? =1 : Wedeneanonzero vector v = I )]TJ/F18 11.9552 Tf 9.997 0 Td [(P U y 2VU ? ,andnoticethat V = U span f v g isanorthogonalsum. Since U isan A -invariantsubspace,itisalso A V -invariant.Thenthedecomposition V = U span f v g impliesthat v isaneigenvectoroftheoperator A V ,corresponding theeigenvalue v : Moreover,bythevariationalprinciplefortheRayleighquotient, v y < spec min A U ,accordingtothedenitionof U .Thus,wededucethat spec A V = v [ spec A U ,where v =spec min A V isanisolatedpoint.The multiplicityoftheeigenvalue v isone,because,again,dim )]TJ/F21 11.9552 Tf 5.479 -9.684 Td [(VU ? =1 : Wearenowpreparedtoproveourmainresultinthissection. Theorem5.5.3 Let < y < where and arethenearestpointsofthe spectrumof A to y .Let U betheinvariantsubspaceof A associatedwithallofthe spectrumof A largerthan y .Wedene x = I )]TJ/F18 11.9552 Tf 11.507 0 Td [(P U y and F =span f x;y g .Then )]TJ/F18 11.9552 Tf 11.955 0 Td [( y y )]TJ/F18 11.9552 Tf 11.955 0 Td [( k P F r y k 2 k y k 2 : .5.2 Proof: ApplyingLemma5.5.2andnoticingthatspec min A U = ,weobservethat theassumptionsofLemma5.5.1aresatisedif A V replaces A ,wherethescalar and theeigenvaluespec min A V arethenearestpointsofthespectrumoftheoperator A V to y;A V = y .Thus,byLemma5.5.1, )]TJ/F18 11.9552 Tf 11.955 0 Td [( y y )]TJ/F15 11.9552 Tf 11.955 0 Td [(spec min A V k P F r y;A V k 2 k y k 2 = k P F r y k 2 k y k 2 : 107

PAGE 117

Sincespec min A V < y ,weobtain5.5.2. Remark5.5.4 Substituting )]TJ/F18 11.9552 Tf 9.299 0 Td [(A for A inLemmas5.5.1and5.5.2,andusingsimilar arguments,weendupwiththesamesubspace F =span f P U y;y g =span f I )]TJ/F18 11.9552 Tf 10.013 0 Td [(P U y;y g andobtainexactlythesamebound. RepeatingtheargumentsofSection5.2withappropriatemodications,weobtainimprovedversionsoftheKato-Templeinequalityandotherbounds,reviewedin Section5.2,fromTheorem5.5.3.Specically,lettheonlypointofthespectrumin theinterval a;b beaneigenvalue ,andletalso y 2 a;b : Deningthesubspace F asinTheorem5.5.3,weget )]TJ/F15 11.9552 Tf 28.586 8.088 Td [(1 y )]TJ/F18 11.9552 Tf 11.955 0 Td [(a k P F r y k 2 k y k 2 y )]TJ/F18 11.9552 Tf 11.956 0 Td [( 1 b )]TJ/F18 11.9552 Tf 11.956 0 Td [( y k P F r y k 2 k y k 2 : Taking a = y )]TJ/F18 11.9552 Tf 11.955 0 Td [( and b = y + ; where =min 2f spec A = g j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j ,implies j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j 1 k P F r y k 2 k y k 2 ; which,initsturn,givesustheimprovedKrylov-Weinsteinbound, j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j k P F r y k k y k : Remark5.5.5 Wenotethat k P F r y kk P V r y kk r y k ,since FVH Thus,rst,theclassicalaposterioriboundsnowtriviallyfollowsfromourbounds. Second,ifthesmallestquantity k P F r y k maynotbeeasytoaccessinapractical situation,itcanbeboundedabovebyamuchsimplerexpression k P V r y k ,whichcan stillbedramaticallysmallercomparedtothestandardvalue k r y k usedintheTemple bound 5.2.3 .Asanalternativefor V ,wecanuse U ? +span f y gF 5.6Improvingknownerrorboundsforeigenvectors ThemaintopicofthisworkisRQbounds.However,havingthemachineryof identitiesalreadyconstructedinSection5.3,weneedonlyverylittleextraeortto 108

PAGE 118

revisitandimprovesomewellknownerrorboundsforeigenvectors,asabonus.The readermighthavenoticedacoupleofidentities,5.3.4and5.3.11,thathavenotyet beenused.Forconvenience,werepeatthemhere,changingthenotationin5.3.4, forthecasewhere F =span f x;y g ,thevector x isaneigenvectorof A and,thus, A F withtheeigenvalue = x ,thescalar 6 = denotestheothereigenvalueof A F ,and = f x;y g : sin = 2 j )]TJ/F18 11.9552 Tf 11.955 0 Td [( j k P F r y k k y k .6.1 and,if <= 2 ; tan = 1 j )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j k P F r y k k y k : .6.2 Inordertouseidentities5.6.1and5.6.2,oneneedstoboundthescalar Itiseasilypossibleinthespecialcase,where = min A or = max A .The latterchoiceisreducedtotheformeronebysubstituting A with )]TJ/F18 11.9552 Tf 9.299 0 Td [(A .Wecanusethe argumentsfromtheproofofLemma5.5.1tohandlethecase = min A .Letthe assumptionsofLemma5.5.1besatised,i.e.,let = min A beanisolatedpoint ofthespectrumof A and > bethenearestlargerpointofthespectrumof A Let X denotethecompleteeigenspacecorrespondingtotheeigenvalue : Foragiven vector y 6 =0,suchthat < y < ,wedene x = P X y ,where P X istheorthogonal projectortothesubspace X .Assumingthat x 6 =0,wedenote F =span f x;y g .Then = x = and ,directlybytheproofofLemma5.5.1.Substituting in5.6.1and5.6.2,weobtainthefollowingimprovementsofthecorresponding boundsfrom[15,64]: sin 2 )]TJ/F18 11.9552 Tf 11.955 0 Td [( k P F r y k k y k and,if <= 2 ; tan 1 )]TJ/F18 11.9552 Tf 11.955 0 Td [( y k P F r y k k y k : 109

PAGE 119

Remark5.6.1 Finally,wenotethattheresultsobtainedinthischaptercanbereformulatedtoapplytothegeneralizedeigenvalueproblem, Ax = Bx; where A isHermitianand B isHermitianandpositivedenite.Webrieydescribetheeigenvalue perturbationboundsforgeneralizedeigenvalueproblemasfollows.Letusrecallthe generalizedRayleighquotient GRQ ,associatedwith A and B ,foragivennonzero vector x x;A;B = h x;Ax i h x;Bx i : Thecorrespondingresidualisdenedas r x;A;B = Ax )]TJ/F18 11.9552 Tf 12.343 0 Td [( x;A;B Bx: Itiseasy toreducetheGRQwithapairofHermitianmatricestotheRQwithaHermitian matrix.Let B = G H G M = G )]TJ/F19 7.9701 Tf 6.587 0 Td [(H AG )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 and ~ x = Gx .Itfollows h x;Ax i h x;Bx i = Gx;G )]TJ/F19 7.9701 Tf 6.587 0 Td [(H AG )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Gx h Gx;Gx i = h ~ x;M ~ x i h ~ x; ~ x i : ThentheabsolutechangeintheGRQwithrespecttononzerovectors x y is j x;A;B )]TJ/F18 11.9552 Tf 11.955 0 Td [( y;A;B j = j ~ x;M )]TJ/F18 11.9552 Tf 11.955 0 Td [( ~ y;M j ; where ~ y = Gy: Inwhatfollows,itismoreconvenienttousethescalarproduct h ; i B andthecorrespondingnorm kk B : Let ~ F =span f ~ x; ~ y g .Bycalculation,wehavethe identities k P ~ x r ~ x;M k k ~ x k cos f ~ x; ~ y g = k P B x ~ r x k B k x k B cos B f x;y g = k P ~ F r ~ x;M k k ~ x k tan f ~ x; ~ y g = k P B F ~ r x k B k x k B tan B f x;y g ; where ~ r x = B )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 Ax )]TJ/F18 11.9552 Tf 11.699 0 Td [( x;A;B x ; P B x and P B F arethe B -orthogonalprojectorson thevector x andthesubspace F =span f x;y g ,correspondingly. B f x;y g denotes theanglebetween x and y inthe B -basedscalarproduct.Therstidentitycanbe obtainedbydirectcalculation.Thethirdidentityneedsextrawork.Themainstepis toshow P ~ F = GP B F G )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 .Wetakea B -orthonormalbasisfrom F denotedasamatrix V ,suchthat V H BV = I .Since ~ F =span f ~ x; ~ y g =span f Gx;Gy g ,wehave P ~ F = 110

PAGE 120

GV V H G H GV )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 V H G H = GV V H BV )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 V H G = GVV H G H = GVV H BG )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = GP B F G )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 : Consequently,severalresultsconcerningthechangeoftheGQRcanbeobtained byRemark5.3.8andTheorem5.3.9.Forexample,let B = k P B F ~ r x k B k x k B k P B F ~ r y k B k y k B tan B f x;y g : Wehave B )]TJ/F21 11.9552 Tf 10.406 2.292 Td [(j x;A;B )]TJ/F18 11.9552 Tf 11.955 0 Td [( y;A;B j B + : .6.3 Moreover,wehave k P ~ F r ~ x;M k k ~ x k tan f ~ x; ~ y g k r ~ x;M k k ~ x k tan f ~ x; ~ y g = k r x;A;B k B )]TJ/F17 5.9776 Tf 5.756 0 Td [(1 k x k B tan B f x;y g : ByRemark5.3.8andequality 5.6.3 ,weobtain B + k r x;A;B k B )]TJ/F17 5.9776 Tf 5.757 0 Td [(1 k x k B + k r y;A;B k B )]TJ/F17 5.9776 Tf 5.756 0 Td [(1 k y k B tan B f x;y g : .6.4 Furthermore,assumethat x isaneigenvectorof Ax = Bx; wehave j x;A;B )]TJ/F18 11.9552 Tf 11.955 0 Td [( y;A;B j = k P B F ~ r y k B k y k B tan B f x;y g k r y;A;B k B )]TJ/F17 5.9776 Tf 5.756 0 Td [(1 k y k B tan B f x;y g : 5.7Conclusions Notonlyhavewefoundanewunifyingprooftechnique,basedonRQidentities. Wehavealsomadeasignicantimprovement,addinganextraprojectorinfrontof theresidualintheboundsthatinvolvethenormoftheresidual. 111

PAGE 121

6.MajorizationboundsfortheabsolutechangeofRitzvaluesand eigenvalues Manyproblemsinscienceandengineeringleadtoeigenvalueproblemsformatrices.Inmostcases,eigenvaluescannotbecomputedexactly,sonumericalapproximationisrequired.Boundsforchangesofeigenvaluesofmatricesplayanimportant role. Let x and y beunitvectorsand f x;y g <= 2.Let P x and P y beorthogonal projectionsonvectors x and y ,correspondingly.WerecallthattheboundsinTheorem5.3.9fortheabsolutechangeintheRayleighquotientofHermitianmatrix A withrespecttovectors x and y are )]TJ/F21 11.9552 Tf 10.405 1.794 Td [(j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j + ; .0.1 where = 1 cos f x;y g abs k P y r x kk P x r y k andabs denotestheabsolute value.Theboundsin6.0.1aresharp.Specically,inarealspace6.0.1becomes anequality,i.e.,either j x )]TJ/F18 11.9552 Tf 11.579 0 Td [( y j = + or j x )]TJ/F18 11.9552 Tf 11.579 0 Td [( y j = )]TJ/F15 11.9552 Tf 7.085 1.793 Td [(.Furthermore,if x is aneigenvectorof A ,wealwayshavetheequalityalsosee[14] j x )]TJ/F18 11.9552 Tf 11.955 0 Td [( y j = 1 cos f x;y g k P x r y k ; .0.2 ineitheracomplexspaceorrealspace.Discussionofthesenovelresultsandproofs canbefoundinChapter5.Inthischapter,wereplace1Dsubspacesspannedbythe vectors x and y withmultidimensionalsubspaces X and Y ,andgeneralize6.0.1 and6.0.2tomultidimensionalsubspaces.In[61]onecanndageneralization of6.0.2,whichisaspecialcaseofournewresults.Wesummarizeourresultsinthe nextparagraph. TheRayleigh-RitzRRmethod[64,73]isaclassicalmethodforeigenvalue approximationofHermitianmatrices.WederiveerrorboundsfortheRRapproximationtoeigenvaluesofaHermitianmatrix A .Letthecolumnsofmatrices X and Y formorthonormalbasesforthesubspaces X and Y ,correspondingly.Wepresent 112

PAGE 122

perturbationboundsofthechangeofRitzvaluesexpressedintermsofprincipalanglesbetweensubspacesPABS,andtheresidualmatrices R X and R Y of A with respecttothematrices X and Y ,i.e, R X = AX )]TJ/F18 11.9552 Tf 11.762 0 Td [(XH X and R Y = AY )]TJ/F18 11.9552 Tf 11.762 0 Td [(YH Y with H X = X H AX and H Y = Y H AY .Inparticular,ifoneofthesubspaces X and Y is aninvariantsubspaceof A ,thenthechangesinRitzvaluesof A representabsolute eigenvalueapproximationerrors. Majorizationnaturallyextendstheinequalities.Weuseseveralmajorization resultsobtainedinSection4.4.Inthischapter,weobtainseveraldierentbounds fortheabsolutechangesofRitzvalues.Ourresultsgeneralizetheresultsobtainedin [47,61,75],sinceourboundsalsoholdforbothsubspaces X and Y ,notnecessarily A -invariant.Further,bytakingadvantageofmajorizationwepresentbetterbounds thanthosein[61,75]forthecasewhereonesubspaceis A -invariant. ArecentpaperbyKnyazevandArgentati[51]presentsapriorierrorboundsof Ritzvaluesandeigenvaluesintermsoftheprincipalanglesbetweensubspaces X and Y withsomeconstant.Aposterioriboundsofeigenvaluespresentedin[5,64,73] areintermsofthesingularvaluesoftheresidualmatrix.Otheraposterioribounds obtainedbyMathias[56]concerningtheabsolutechangeofeigenvaluesareinterms oftheresidualmatrixandthegap.OurnewboundstakeadvantageofPABSand theresidualmatrixandusemajorizationtechniques.Wethereforecallthesebounds majorization-typemixedbounds.Abriefdiscussionoftherelationshipbetweenthe mixedmajorizationboundsandclassicalboundsisincludedattheendofthischapter. Therestofthischapterisorganizedasfollows.WebrieydiscusstheRRmethod inSection6.1.Wereviewtheperturbationofeigenspacesandpresenttherelation oftheresidualmatrixtothetangentsofPABSinSection6.2.InSection6.3,we presentourmainresults|mixedboundsoftheabsolutechangesofRitzvaluesand eigenvaluesusingweakmajorization.Finally,webrieycompareournewbounds withtheknownaprioriboundsandaposterioribounds. 113

PAGE 123

6.1Rayleigh-Ritzmethod TheRRmethodisawidelyusedtechniqueforcomputingapproximationsto eigenvaluesandeigenvectorsofaHermitianmatrix A ,fromagivensubspace.BackgroundmaterialfortheRRmethodcanbefoundin[64,73].Let A beaHermitian matrixoforder n .Letthecolumnsofmatrix X 2 C n p formanorthonormalbasis forthesubspace X .ThematrixRayleighquotientassociatedwiththematrix X is denedas X = H X = X H AX: Theeigenvaluesof H X arecalledRitzvaluesof A correspondingto X .Let ;u beaneigenpairof X H AX .Theso-calledRitzpair ;Xu isastheapproximate eigenpairof A where Xu iscalledaRitzvector.Specially,letthesubspace X be A invariant,i.e., A XX .Inthiscase,wehave AX = XH X ; andthecolumnsof X areeigenvaluesof A .Theeigenvaluesof H X arealsothe eigenvaluesof A .Inotherwords,the p Ritzvaluesof A correspondingto X are exactlysome p eigenvaluesof A ,ifthesubspace X is A -invariant. TheRRapproximationsareoptimal,whichisdemonstratedbythefollowingtwo theorems. Theorem6.1.1 [64] Letthecolumnsof X 2 C n p formanorthonormalbasisfor thesubspace X .Let 1 H X p H X be p eigenvaluesof H X = X H AX with aHermitianmatrix A 2 C n n .Then i =max GX dim G = i min x 2G k x k =1 x H Ax for i =1 ;:::;p: 114

PAGE 124

Theorem6.1.2 [64,Theorem11.4.5] Letthecolumnsof X 2 C n p beorthonormal, A = A H ,and H X = X H AX .Foranymatrix B 2 C p p ,wehave k AX )]TJ/F18 11.9552 Tf 11.955 0 Td [(XH X kk AX )]TJ/F18 11.9552 Tf 11.955 0 Td [(XB k : Fromtheabovetheorem,theresidual H X hasaminimalproperty. 6.2Perturbationofeigenspaces AnimportantissueintheperturbationtheoryofHermitianmatricesistoinvestigatetheperturbationofeigenspaces.ThegeneralapproachisduetoDavisand Kahan[15],andthefollowingtheoremiscalledthesecondsintheorem. Theorem6.2.1 [15] Let A beHermitianandlet [ XX ? ] beunitarywith X 2 C n p suchthat [ XX ? ] H A [ XX ? ]=diag L 1 ;L 2 : Let Y 2 C n p haveorthonormalcolumns and R Y = AY )]TJ/F18 11.9552 Tf 12.815 0 Td [(YH Y with H Y = Y H AY: Moreover,assume H Y [ a;b ] and > 0 ,while L 2 R n [ a )]TJ/F18 11.9552 Tf 11.956 0 Td [(;b + ] .Thenforeveryunitarilyinvariantnorm jjj diagsin R X ; R Y jjj jjj R Y jjj orequivalently, sin R X ; R Y w S R Y ; .2.1 where S R Y denotesthevectorofsingularvaluesof R Y : Ifmoreinformationaboutthespectraof H Y and L 2 isavailable,wecanobtain asharperboundthan6.2.1.Thereaderisreferredto[80]fordetails. DavisandKahan[15]ndanotherimportanttheoremabouttheboundonthe tangentofprincipalanglesbetweenanexactspectralsubspaceandanapproximate subspaceofaHermitianmatrix,whichiscalledthetantheorem. Theorem6.2.2 [15] Inthenotationof sin Theorem,assume H Y [ a;b ] and > 0 ,while L 2 ;a )]TJ/F18 11.9552 Tf 12.366 0 Td [( ] or L 2 [ b + ; 1 .Thenforeveryunitarily 115

PAGE 125

invariantnorm jjj diagtan R X ; R Y jjj jjj R Y jjj ; orequivalently, tan R X ; R Y w S R Y : Thetantheoremrequiresmorerestrictiveconditionsontheeigenvaluesof A thanthesintheorem,sincethetantheoremisvalidonlyfortheRitzvaluesof A withrespectto Y aboveorbelowtheeigenvaluesof L 2 .Arecentpaper[59]shows thattheconditionsofthetantheoremcanberelaxed.Seethefollowingtheorem. Theorem6.2.3 [59] Inthenotationof sin Theorem,assume L 2 [ a;b ] and > 0 ,while H Y liesintheunionof ;a )]TJ/F18 11.9552 Tf 12.132 0 Td [( ] and [ b + ; 1 .Thenforevery unitarilyinvariantnorm jjj diagtan R X ; R Y jjj jjj R Y jjj ; orequivalently, tan R X ; R Y w S R Y : Analternativeproofofthecaseofthespectralnormisfoundin[54,58]. Anotherinterestingresultisthattheanglesbetweensubspaces R X and R AX arerelatedtothesingularvaluesoftheresidualmatrix. Theorem6.2.4 Letthecolumnsofmatrix X beorthonormaland H X = X H AX be invertiblewithaHermitianmatrix A .Then tan R X ; R AX w S R X S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H AX )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 : Moreover,if dim R X =1 usingvectornotation x insteadof X ,then tan x;Ax = k r x k j x j ; where r x = Ax )]TJ/F18 11.9552 Tf 11.955 0 Td [(xx H Ax 116

PAGE 126

Proof: Since H X isinvertible,matrix AX hasfullcolumnrank.FromCorollary2.2.9, tan R X ; R AX = S P X ? AX )]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(X H AX )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 : ApplyingTheorem4.2.5gives S P X ? AX )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H AX )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 w S P X ? AX S )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H AX )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : Since AX )]TJ/F18 11.9552 Tf 11.955 0 Td [(XH X = AX )]TJ/F18 11.9552 Tf 11.956 0 Td [(P X AX = P X ? AX ,weobtaintherstresult. Inparticular,fordim R X =1,wehave tan x;Ax = S P x ? Ax )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(x H Ax )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = k P x ? Ax )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(x H Ax )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 k = k r x k j x j ; whichcompletestheproof. Forthecasedim R X =1,Argentatihasprovedthisusingsinandcos in[2].Above,wetakeadvantageoftheformulaoftanderivedinChapter2to proveitdirectly. 6.3Majorization-typemixedbounds Inthissectionwederiveseveraldierentmajorization-typemixedboundsin termsofsingularvaluesofresidualmatrixandPABS.Oneofourmainresultsis containedinthefollowinglemma. Lemma6.3.1 Letthecolumnsofmatrices X and Y formtheorthonormalbases forthesubspaces X and Y with dim X =dim Y ,correspondingly.Let A bea Hermitianmatrixand X ; Y <= 2 .Then X H AX )]TJ/F15 11.9552 Tf 11.956 0 Td [( Y H AY w 1 cos max X ; Y f S P Y R X + S P X R Y g : .3.1 Inaddition,ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w 1 cos max X ; Y S P X R Y : .3.2 117

PAGE 127

Proof: Accordingtothehypothesis X ; Y <= 2,thesingularvaluesof X H Y arepositive,whichimpliesthatthematrix X H Y isinvertible.Weapplytherst statementofLemma4.4.6with A := X H AX B := Y H AY and T := X H Y .Thus, weget X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w 1 s min X H Y S )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H AX X H Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( X H Y Y H AY : .3.3 FromthedenitionofPABS,thesingularvaluesof X H Y arethecosinesofprincipal anglesbetweentwosubspaces X and Y .So,wehave s min X H Y =cos max X ; Y : .3.4 Additionally,theformula X H AX X H Y )]TJ/F15 11.9552 Tf 11.44 0 Td [( X H Y Y H AY ontherightsideof6.3.3 canbewrittenas X H AX X H Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( X H Y Y H AY = X H A I )]TJ/F18 11.9552 Tf 11.955 0 Td [(X ? X H ? Y )]TJ/F18 11.9552 Tf 11.955 0 Td [(X H I )]TJ/F18 11.9552 Tf 11.955 0 Td [(Y ? Y H ? AY = )]TJ/F18 11.9552 Tf 9.299 0 Td [(X H AX ? X H ? Y + X H Y ? Y H ? AY: .3.5 Further,thesingularvaluesareinvariantunderconjugatetransposeandorthonormal transforms.Thus, S X H AX ? X H ? Y = S Y H X ? X H ? AX = S P Y P X ? AX = S P Y R X : Similarly, S X H Y ? Y H ? AY = S P X R Y : FromTheorem4.2.2andequality6.3.5, weestablishthat S X H AX X H Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( X H Y Y H AY = S )]TJ/F18 11.9552 Tf 9.298 0 Td [(X H AX ? X H ? Y + X H Y ? Y H ? AY w S X H AX ? X H ? Y + S X H Y ? Y H ? AY = S P Y R X + S P X R Y : .3.6 Substituting6.3.6and6.3.4into6.3.3,weobtaintherstresultinLemma6.3.1 asexpected.Inaddition,ifthesubspace X is A -invariant,then R X =0which completestheproof. 118

PAGE 128

Whenthesubspace X is A -invariant,theeigenvaluesof X H AX areexactlysome eigenvaluesof A .Thus,thebounds6.3.2representtheapproximationerrorsfor eigenvaluesof A intheRRmethod.Analternativeproofof6.3.2canbefound in[61].TheboundsinLemma6.3.1aresharp.Forthecasedim X =dim Y =1, themajorizationinequalityin6.3.1couldbecomeanequality,seeChapter5.In particular,if X is A -invariantwithdim X =1,thentheweakmajorizationinequality in6.3.2becomesanequality,e.g.,see[14]andChapter5. Next,wedevelopseveraldierentmajorization-typemixedboundsintermsof residualmatricesusingorthogonalprojectionsonsubspacesandPABS. Lemma6.3.2 UndertheassumptionsofLemma6.3.1,wehave X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY 2 w f S P Y R X + S P X R Y g 2 cos 2 # X ; Y : Inaddition,ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY 2 w S 2 P X R Y cos 2 # X ; Y ; where S 2 P X R Y cos 2 # X ; Y = h s 2 max P X R Y cos 2 max X ; Y ;:::; s 2 min P X R Y cos 2 min X ; Y i : Proof: Since X ; Y <= 2,itfollowsthat X H Y isinvertible.Wereplace X H AX Y H AY and X H Y for A;B ,and T inthethirdresultofLemma4.4.6,suchthat X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY 2 w S 2 )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 S 2 )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H AX X H Y )]TJ/F15 11.9552 Tf 11.956 0 Td [( X H Y Y H AY : Nextwemainlydealwiththerightsideofthemajorizationinequalityabove.Based onthedenitionofPABS,weknowthat S X H Y =cos X ; Y .So S )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 = 1 cos # X ; Y : .3.7 From6.3.6,wealreadyhave S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H AX X H Y )]TJ/F15 11.9552 Tf 11.956 0 Td [( X H Y Y H AY w f S P Y R X + S P X R Y g : 119

PAGE 129

Sincenondecreasingconvexfunctionspreserveweakmajorization,wetakethefunction f x = x 2 fornonnegative x .Squaringbothsidesoftheweakmajorization inequalityabove,ityields S 2 )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H AX X H Y )]TJ/F15 11.9552 Tf 11.955 0 Td [( X H Y Y H AY w f S P Y R X + S P X R Y g 2 : Togetherwith6.3.7,thisprovestherststatementofLemma6.3.2.Ifthesubspace X is A -invariant,then R X =0,whichcompletestheproof. Lemma6.3.3 UndertheassumptionsofLemma6.3.1,wehave X H AX )]TJ/F15 11.9552 Tf 11.956 0 Td [( Y H AY w s cos min X ; Y cos max X ; Y f S P Y R X + S P X R Y g cos # X ; Y : Ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w s cos min X ; Y cos max X ; Y S P X R Y cos # X ; Y : Proof: X ; Y <= 2impliesthat X H Y isinvertible,so Y H AY issimilartothe matrix )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y Y H AY )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 .Thematrix A isHermitian,so X H AX and Y H AY are.Fromthespectraldecomposition,wehave X H AX = U 1 D 1 U H 1 ,where U 1 is unitaryand D 1 isdiagonal.Similarly, Y H AY = U 2 D 2 U H 2 ,where U 2 isunitaryand D 2 isdiagonal.Asaconsequence,wehave )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(Y H AY = )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y Y H AY )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(U 1 D 1 U H 1 )]TJ/F15 11.9552 Tf 11.955 0 Td [( )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y U 2 D 2 U H 2 )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : ApplyingTheorem4.3.6,weobtain j )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(Y H AY j w U 1 X H YU 2 1 = 2 S X H AX )]TJ/F24 11.9552 Tf 11.955 9.684 Td [()]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y Y H AY )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : Furthermore,theconditionnumberof U 1 is1andtheconditionnumberof X H YU 2 is equaltotheconditionnumberof X H Y ,i.e., U 1 =1and X H YU 2 = X H Y : 120

PAGE 130

Moreover,wehave X H AX )]TJ/F24 11.9552 Tf 11.955 9.683 Td [()]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(X H Y Y H AY )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 = X H AX )]TJ/F18 11.9552 Tf 5.48 -9.683 Td [(X H Y )]TJ/F24 11.9552 Tf 11.956 9.683 Td [()]TJ/F18 11.9552 Tf 5.479 -9.683 Td [(X H Y Y H AY )]TJ/F18 11.9552 Tf 12.453 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.587 0 Td [(1 : Consequently, )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(Y H AY w X H Y 1 = 2 S X H AX )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y )]TJ/F24 11.9552 Tf 11.955 9.684 Td [()]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(X H Y Y H AY S )]TJ/F18 11.9552 Tf 5.48 -9.684 Td [(X H Y )]TJ/F16 7.9701 Tf 6.586 0 Td [(1 : .3.8 Substituting6.3.6and6.3.7into6.3.8,weobtaintheresultasexpected. Remark6.3.4 TheboundsinLemmas6.3.1,6.3.2,and6.3.3lookquitedierent. Butinsomeparticularcases,theboundsintheselemmasarethesame.Itisworth notingthattheboundsfor max X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY arethesameinbothLemmas6.3.1and6.3.2.Theboundsfor max X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY inLemma6.3.3 areslightlyweakerthanthoseinLemmas6.3.1and6.3.2,sincethereisaconstant q cos min X ; Y cos max X ; Y 1 inLemma6.3.3.Ifwetake dim X =dim Y =1 ,thenthe boundsinthesethreelemmasarethesame. AlthoughtheboundsinLemma6.3.3areslightlyweakforthemaximumvalueof thevector v aswediscussabove,bytakingtheadvantageofweakmajorization,the boundsinLemma6.3.3maybesharperforthesumof X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY ,since theboundsinLemma6.3.3involvethevector 1 = cos # X ; Y ,whichisarranged innonincreasingorder.However,theboundsinLemmas6.3.1and6.3.2involve thetermsof 1 = cos max X ; Y and 1 = cos 2 # X ; Y ,correspondingly.Notethat 1 = cos # X ; Y [1 = cos max X ; Y ;:::; 1 = cos max X ; Y ] : Letushighlightthatwecannottakethesquarerootfrombothsidesoftheweak majorizationinequalitiesinLemma6.3.2.However,fromournumericaltest,we observethattheresultholdswithoutthesquareofbothsides.Inotherwords,the 121

PAGE 131

resultsinLemma6.3.3numericallyholdwithouttheconstant q cos min X ; Y cos max X ; Y onthe rightside.Wehavethefollowingconjecture. Conjecture6.3.5 UndertheassumptionsofLemma6.3.1, X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w f S P Y R X + S P X R Y g cos # X ; Y : Ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w S P X R Y cos # X ; Y : Ingeneral,theorthogonalprojectionsmaketheboundsinLemmas6.3.1,6.3.2, and6.3.3sharper.However,inpracticeitmaybediculttoevaluate S P X R Y or S P Y R X .Weprovidethefollowingusefulweakmajorizationinequalities. Lemma6.3.6 InthenotationofLemma6.3.1,wehave S P X R Y w S R Y sin X ; Y and S P Y R X w S R X sin X ; Y : Proof: Sincethesingularvaluesareinvariantunderunitarytransformsandthematrixconjugatetranspose,itfollowsthat S P X R Y = S X H P Y ? AY = S Y H AP Y ? X : AccordingtoTheorem4.2.5,wehave S Y H AP Y ? X w S )]TJ/F18 11.9552 Tf 5.479 -9.684 Td [(Y H AP Y ? S P Y ? X = S P Y ? AY S P Y ? X = S R Y S P Y ? X : Moreover,thesingularvaluesof P Y ? X coincidewiththesinesoftheprincipalangles between X and Y ,e.g.,see[9,48],sincedim X =dim Y : Inotherwords,wehave S P Y ? X =sin X ; Y .Consequently,wehave S P X R Y w S R Y sin X ; Y SincePABSaresymmetric,suchthat X ; Y = Y ; X ; andthetwosubspaces X and Y havethesamedimensions,itisclearthat S P X ? Y =sin X ; Y .Ina similarway,wecanestablishthat S P Y R X w S R X sin X ; Y : 122

PAGE 132

Theorem6.3.7 Letthecolumnsofthematrices X and Y formorthonormalbases forthesubspaces X and Y with dim X =dim Y ,correspondingly.Let A bea Hermitianmatrixand X ; Y <= 2 .Then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w 1 cos max X ; Y f S R X + S R Y g sin X ; Y : Inaddition,ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w 1 cos max X ; Y S R Y sin X ; Y : Proof: CombiningLemma6.3.1andLemma6.3.6yieldsthistheoremdirectly. Remark6.3.8 Forthecasewherethesubspace X is A -invariant,in [75,Theorem 3.3] theauthorobtains jjj X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY jjj k sin X ; Y k p 1 )-222(k sin X ; Y k 2 jjj R Y jjj ; whichimpliesthat j X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY j w tan max X ; Y S R Y : Ifwetake X and Y forthesubspaces and ~ H in [61,Theorem1] ,thenwehave cos max X ; Y j i X H AX )]TJ/F15 11.9552 Tf 11.956 0 Td [( i Y H AY j sin max X ; Y k R Y k : Sincewetakeadvantageofallsingularvaluesof R Y andallsinesofPABSbetween X and Y ,ourresultsarestrongerthanthosein [61,75] .Furthermore,ourbounds aremoregeneral,sincetheyalsoholdforbothsubspaces X and Y notnecessarily A -invariant. Theorem6.3.9 UndertheassumptionsofTheorem6.3.7,wehave X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY 2 w f S R X + S R Y g 2 tan 2 X ; Y : Inaddition,ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY 2 w S 2 R Y tan 2 X ; Y : 123

PAGE 133

Proof: CombiningLemmas6.3.2and6.3.6,weobtaintheresults. Lemmas6.3.3and6.3.6implythefollowingtheorem. Theorem6.3.10 UndertheassumptionsofTheorem6.3.7,wehave X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w s cos min X ; Y cos max X ; Y f S R X + S R Y g tan X ; Y : Ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w s cos min X ; Y cos max X ; Y S R Y tan X ; Y : Again,letusunderscorethataccordingtoournumericaltests,wecanremove theconstantfromtherighthandsideofweakmajorizationinequalitiesinTheorem 6.3.10.Wehavethefollowingconjecture. Conjecture6.3.11 UndertheassumptionsofTheorem6.3.7, X H AX )]TJ/F15 11.9552 Tf 11.956 0 Td [( Y H AY w f S R X + S R Y g tan X ; Y : Ifthesubspace X is A -invariant,then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w S R Y tan X ; Y : TheimplicationsoftheweakmajorizationinequalitiesinTheorems6.3.7,6.3.9, and6.3.10maynotbeobvioustoeveryreader.Letcomponentsofbothvectors X H AX and Y H AY bearrangedinnondecreasingorder,suchthat X H AX =[ 1 ; 2 ;:::; p ]and Y H AY =[ 1 ; 2 ;:::; p ] : Wediscussthecase wherethesubspace X is A -invariant.For k =1 ;:::;p ,theweakmajorizationinequalitiesofTheorem6.3.7directlyimply k X i =1 j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j # 1 cos max X ; Y k X i =1 sin # i X ; Y s # i R Y : Theorem6.3.9implies k X i =1 j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j 2 # k X i =1 tan 2 # i X ; Y s 2 # i R Y : 124

PAGE 134

Theorem6.3.10implies k X i =1 j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j # s cos min X ; Y cos max X ; Y k X i =1 tan # i X ; Y s # i R Y : Thesethreeinequalitiesmentionedabovearedierentfromeachother.Specically, for k =1,bothTheorems6.3.7and6.3.9imply max i =1 ;:::;p j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i jk R Y k tan max X ; Y : .3.9 Notethatcontinuousnondecreasingconvexfunctions f forrealvaluespreserve weakmajorizationinequalities[55],suchthat x w y implies f x w f y ,i.e., P n i =1 f x i w P n i =1 f y i : Forexample,ifwetake f t = t m with m 1 ; thenthe boundsinTheorem6.3.9imply k X i =1 j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j 2 m 1 = 2 m k X i =1 )]TJ/F15 11.9552 Tf 5.48 -9.684 Td [(tan 2 m i X ; Y s 2 m i R Y 1 = 2 m : Inparticular,setting k = p and m =1wehave p X i =1 j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j 2 1 = 2 tan max X ; Y p X i =1 s 2 i R Y 1 = 2 ; whichisequivalentto p X i =1 j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j 2 1 = 2 tan max X ; Y k R Y k F : Corollary6.3.12 Let A beHermitianandlet [ XX ? ] beunitarywith X 2 C n p suchthat [ XX ? ] H A [ XX ? ]=diag L 1 ;L 2 : Let Y 2 C n p haveorthonormalcolumns and R Y = AY )]TJ/F18 11.9552 Tf 12.815 0 Td [(YH Y with H Y = Y H AY: Moreover,assume H Y [ a;b ] and > 0 ,while L 2 R n [ a )]TJ/F18 11.9552 Tf 11.955 0 Td [(;b + ] .Then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w S 2 R Y cos max X ; Y : Proof: WeobtaintheresultdirectlybyusingTheorems6.2.1and6.3.7. 125

PAGE 135

Corollary6.3.13 InthenotationofCorollary6.3.12,assumethat H Y [ a;b ] and > 0 ,while L 2 ;a )]TJ/F18 11.9552 Tf 9.566 0 Td [( ] or L 2 [ b + ; 1 .Or,assume L 2 [ a;b ] and > 0 ,while H Y liesintheunionof ;a )]TJ/F18 11.9552 Tf 11.955 0 Td [( ] and [ b + ; 1 .Then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY 2 w S 4 R Y 2 : Proof: FromTheorems6.2.2and6.2.3,togetherwithTheorem6.3.9,weobtainthe resultdirectly. Remark6.3.14 WecanobtaintheresultdirectlyfromCorollary6.3.12,suchthat max i =1 ;:::;p j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j k R Y k 2 : Alternatively,thisresultcanbeobtainedbyTheorems6.2.2and6.2.3togetherwith inequality 6.3.9 6.4Discussion IfPABSaresmall,suchthattan max X ; Y 1,thenthemixedmajorization boundsinTheorems6.3.7and6.3.9aresharperthantheknownaposterioriboundsin termsoftheresidualmatrix.Forcompleteness,wereproducehereknownaposteriori majorizationboundsasfollows. Theorem6.4.1 [5,73,81] Let Y beanorthonormal n by p matrixandmatrix A be Hermitian.Thenthereexistasetofincices 1 i 1
PAGE 136

2inTheorem6.4.1cannotberemoved,whichisshownbyBhatiain[5,p.188].The implicationofthisfactisthatwecanchoose X inourbounds,suchthat max X ; Y = 4,tomakeourboundssharper.Next,wedescribesomesituationswhereprincipal anglesarelessthan = 4 : 1.[10,p.64]Let[ YY ? ]beunitary.Suppose max Y H AY < min Y H ? AY ? and X isthespacespannedbytheeigenvectorscorrespondingtothe p smallest eigenvaluesof A ,then max X ; Y <= 4. 2.[15,sinand8.2Theorem]Let A beHermitianandlet[ XX ? ]beunitary with X 2 C n p ,suchthat[ XX ? ] H A [ XX ? ]=diag L 1 ;L 2 : Let Y 2 C n p be withorthonormalcolumnsand H Y = Y H AY: Thereisa > 0,suchthat L 1 2 [ ; ]and L 2 2 R n [ )]TJ/F18 11.9552 Tf 10.997 0 Td [(; + ].Let H Y 2 [ )]TJ/F19 7.9701 Tf 12.287 4.707 Td [( 2 ; )]TJ/F19 7.9701 Tf 12.287 4.707 Td [( 2 ].Then, max X ; Y <= 4. 3.[25]Let A beaHermitianquasi-denitematrix,i.e., A = 2 6 4 HB H B )]TJ/F18 11.9552 Tf 9.298 0 Td [(G 3 7 5 ; .4.1 where H 2 C k k and G 2 C n )]TJ/F19 7.9701 Tf 6.587 0 Td [(k n )]TJ/F19 7.9701 Tf 6.586 0 Td [(k areHermitianpositivedenitematrices. Letthesubspace Y bespannedbytheeigenvectorsof A correspondingto p eigenvalueswhichhavethesamesign.Letthesubspace X bespannedby e 1 ;:::;e p and Z bespannedby e n )]TJ/F19 7.9701 Tf 6.586 0 Td [(p +1 ;:::;e n ,where e i isthecoordinatevector. Then,iftheeigenvaluescorrespondingtotheeigenspace Y arepositive,wehave max X ; Y <= 4 : Iftheeigenvaluescorrespondingtotheeigenspace Y are negative,wehave max Z ; Y <= 4 : Inthesesituations,wehave max i =1 ;:::;p j i )]TJ/F18 11.9552 Tf 11.956 0 Td [( i j < k R Y k and p X i =1 j i )]TJ/F18 11.9552 Tf 11.955 0 Td [( i j 2 1 = 2 < k R Y k F : 127

PAGE 137

Tosummarize,comparedtotheknownaposteriorbounds,theadvantagesofthe mixedmajorization-typeboundsinTheorem6.3.7aredescribedinthefollowing.The mixedmajorizationboundsaremoregeneralthantheknownaposterioribounds, becausetheaposterioriboundsholdonlyifonesubspaceis A -invariant.Wecan chooseanyinvariantsubspaceinthemajorization-typemixedbounds.Incontrast, intheknownaposteriorimajorizationboundsonlytheexistenceofthe A -invariant subspace X isstated.Finally,ifPABSaresmall,suchas max X ; Y = 4,thenthe majorization-typemixedboundsinTheorem6 : 3 : 7aresharper. Whatistheadvantageofthemajorization-typemixedboundscomparedtothe apriorimajorizationbounds?Forconvenience,weformulatehereknownapriori boundsasfollows. Theorem6.4.2 [51] Letthesubspaces X and Y havethesamedimensions,with orthonormalbasesgivenbythecolumnsofthematrices X and Y ,correspondingly. Let A = A H .Then X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w [ max A )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A ]sin X ; Y : Ifinadditiononeofthesubspacesis A -invariantthen X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w [ max A )]TJ/F18 11.9552 Tf 11.955 0 Td [( min A ]sin 2 X ; Y : Themainadvantageofthemajorization-typemixedboundsisinthecasewhere bothsubspaces X and Y approximatethesame A -invariantsubspace,sothatthe principalanglesbetweensubspaces X and Y aresmallandthesingularvaluesofthe residualmatricesaresmall. Forexample,inFigure6.1wechoosearandomHermitianmatrix A 2 C n n .Let thecolumnsof U beeigenvectorsof A .WeusetheMATLABnotation.Let E = orthrandnn,p.Weset X =orthU:,1:p+ c E and Y =orthU:,1:p+ c E where c =rand ; 1 10 )]TJ/F16 7.9701 Tf 6.586 0 Td [(3 +10 )]TJ/F16 7.9701 Tf 6.587 0 Td [(3 i for i =1 ;:::; 100.Inotherwords,wechoose 128

PAGE 138

theperturbedmatrix c E suchas k c E k islessthan0 : 1.Wecomparethelargest boundofabsolutechangeofRitzvaluesinthemixedtypeboundsinTheorem6.3.7 or6.3.9tothatintheaprioriboundsinTheorem6.4.2. Figure6.1: ComparisonoftheabsolutechangeofRitzvaluesforthemixedbounds andfortheaprioriboundswiththeexactabsolutechangeofRitzvaluesleft;The correspondingtangentofthelargestprincipalangleright. Figure6.2: ComparingthemixedtypeboundoftheabsolutechangeofRitzvalues withtheexactabsolutechangeofRitzvalues. Figure6.1leftdisplaysthecomparisonofboundsoftheabsolutechangesofRitz valuesformixedtypeboundsandtheaprioriboundswiththeexactabsolutechanges forRitzvalues.Figure6.1rightshowsthecorrespondingtangentofthemaximum 129

PAGE 139

anglebetween R X and R Y : FromFigure6.1,weseethatthemixedtypebounds aresmallerthantheaprioriboundsforthelargestboundoftheabsolutechange ofRitzvalues.Moreover,theaprioriboundsofRitzvalueperturbationincrease withthesamespeedas i increases.Incontrast,themixedtypeboundsofRitzvalue perturbationincreaseslightlyas i increases.Toseethismoreclearly,inthesametest thecomparisonofthemixedtypeboundfortheabsolutechangeofRitzvalueswith theexactvalueisshowninFigure6.2. 6.5Conclusions Inthischapter,wehavederivedseveralmajorization-typemixedboundsforthe absolutechangesofRitzvaluesandapproximationsofeigenvaluesofaHermitianmatrix.Ourresultsimprovetheexistingmixedboundsandextendtheresultsobtained inChapter5tomulti-dimensionalsubspaces. 130

PAGE 140

7.Conclusionsandfuturework Inthisthesis,wehaveobtainednewpropertiesofPABSrelatingthetangentsof PABStosingularvaluesofexplicitlyconstructedmatrices.Wearefurtherinterested inapplyingourmatrixformulationstoderivenovelmajorization-typeconvergence rateboundsofsubspaceiterations,nishingtheworkstartedin[52]. ConcerningthecomputationofPABS,wehavepresentedseveralnumericalalgorithmsbasedonthesinesofhalf-PABS.Fromourformulationsinvolvingthesines ofhalfangles,onecanalsodirectlycomputetheHausdometric.Noneofthealgorithmsisaclearwinner,witheachofthemhavingitsownadvantages.Wewouldlike toinvestmoretimeintolookingforanultimatealgorithmforcomputingthePABS, sinceitisveryimportantinmanyapplications.WealsowanttodevelopparallelalgorithmstocomputePABSinthefuture,andwriteaparallelcodetobeincludedinto ourBLOPEXBlockLocallyOptimalPreconditionedEigenvalueXolversparallel softwarelibrary. ForcomputingthePABSforsparsedata,iterativemethodsarepreferable,preservingthesparsity.Inthiswork,wehavenotconsideredusingiterativemethods. Onecouldevidentlytryaneigensolver,e.g.,thelocallyoptimalblockpreconditioned conjugategradientmethodLOBPCGtocomputethecosinesofPABSandthe principalvectors,reformulatingtheSVDasageneralizedeigenvalueproblem.We areinterestedtoinvestigateLOBPCG-likealgorithmsforthesinesofhalf-PABS. WehaveproposedaunifyingtechniquetoproveboundsfortheRayleighquotient andforthespectrumofself-adjointoperators.Newmixedmajorization-typebounds fortheabsolutechangeofRitzvaluesandeigenvalueshavebeenderived.Wehave formulatedthefollowingtwomajorizationinequalitiesasconjectures: X H AX )]TJ/F15 11.9552 Tf 11.955 0 Td [( Y H AY w f S P Y R X + S P X R Y g cos # X ; Y ; and X H AX )]TJ/F15 11.9552 Tf 11.956 0 Td [( Y H AY w f S R X + S R Y g tan X ; Y : 131

PAGE 141

Weplantoattemptprovingtheseelegantandpowerfulconjectures. 132

PAGE 142

APPENDIXA.MATLABcode TheMATLABcodeofsubspaceupdate.misasfollows: 1 function[theta,varargout]=subspaceupdateX,Y,varargin %SUBSPACEUPDATEFindprincipalanglesbetweensubspaces 3 % %ThisisbackwardcompatiblewiththeMATLABbuilt-in 5 %SUBSPACEfunction,exceptthatourfunctiongivesmuch %moreaccurateresultsfornearlyorthogonalvectors. 7 %IfXandYareunitvectors,thentheangleis %ACOSABSX *Yinexactarithmetic. 9 % %Ourfunctionhasmanymorefeatures,comparedtothebuilt-inoriginal. 11 %[THETA,U,V]=subspaceupdateX,Y %outputstheorthonormalizedprincipalvectors:U=X*AandV=Y*B. 13 %whereAandBaretheorthonormalizedcanonicalcoefficients. %TheprincipalvectorsUandVrevealthedirectionsofthelargest 15 %changebetweenXandY. % 17 %[THETA,U,V,A,B]=subspaceupdateX,Y %alsooutputstheorthonormalizedcanonicalcoefficientsAandB,which 19 %arep-by-dandq-by-dmatrices,whered=minrankX,rankYand %XandYaren-by-pandn-by-q,correspondingly. 21 % %ClasssupportforinputXandY: 23 %fullfloat:double,single, %sparsefloat:double 25 % %SUBSPACEUPDATEX,Y,OPTSspecifiesoptions: 27 %OPTS.fast:usefaster,butlessaccuratealgorithm[{0}|1|2] %OPTS.R:usemethodbasedonthetriangularuppermatrices[{0}|1] 29 %OPTS.A:useA-basedscalarproduct[matrixorfunctionhandle|{null}] %OPTS.number:findkprincipalangles,ifthenumberisk. 31 %OPTS.disp:diagnosticinformationdisplaylevel[{0}|1|2] % 33 %Thenumberofallprincipalanglesisequalto %minsizeorthupdateX,2,sizeorthupdateY,2 35 % %[THETA,U,V]=subspaceupdateX,Y 37 %outputsallorthonormalizedprincipalvectors:UforXandVforY, %wheresizeU=sizeorthXandsizeV=sizeorthY. 39 % %[THETA,U,V]=subspaceupdateX,Y,opts,ifopts.A=A. 41 %doesthesame,butintheA-basedscalarproductx *A*y,whereA %isHermitianandpositivedefinite.TheoptionalparameterAisa 43 %Hermitianandpositivedefinitematrix,oracorrespondingfunction. %WhenAisafunction,itmustacceptamatrixasanargument. 45 % %COSTHETAforcentereddataXandYwithzeromeangives 47 %canonicalcorrelations. % 49 %ThiscoderequiresORTHUPDATE.mandMGSA.m %Examples: 133

PAGE 143

51 %opts.fast=1;X=rand10,4;Y=randn10,6;theta=subspaceupdateX,Y,opts; %computes4anglesbetweenXandY,whileinaddition 53 %A=hilb10;opts.A=A;[theta,U,V]=subspaceupdateX,Y,opts; %computesanglesrelativetoAandcorrespondingvectorsUandV 55 % 57 ifnargout>5 error subspace:TooManyOutputs Toomanyoutputarguments. 59 end 61 ifnargin<2 error subspace:NeedAtLeastTwoArguments ,... 63 Subspaceneedsatleasttwoarguments ; end 65 if : isnumericX 67 error subspace:XNotNumeric ,... Xmustbeanumericmatrix. 69 end 71 if : isnumericY error subspace:YNotNumeric ,... 73 Ymustbeanumericmatrix. end 75 ifsizeX,1 6 = sizeY,1 77 error[ Therowdimension int2strsizeX,1... ofthematrixXisnotthesameas int2strsizeY,1... 79 therowdimensionofY ] end 81 %%%%%%%%%%%%%%setthedefaultvalue%%%%%%%%%%%%%%%%%%%%%%%%% 83 A=[];subFast=0;subR=0;subDisplay=0;subNumber=[]; 85 ifnargin>2 opts=varargin{1}; 87 if : isaopts, struct error subspace:OptionsNotStructure ,... 89 Optionsargumentmustbeastructure. end 91 ifisfieldopts, number subNumber=opts.number; 93 if : isscalarsubNumber error subspace:NonScalarSubNumber ,... 95 subNumbermustbeascalar. end 97 ifsubNumber 0 error subspace:InvalidNumberInput ,... 99 Invalidnumberinput ; end 101 end %ProcesstheinputA 103 ifisfieldopts, A A=opts.A; 105 ifisnumericA Amatrix=true; 107 else %BycheckingthefunctionAwithfcnchk,wecannowusedirect 109 %functionevaluationontheresult,withoutresortingtofeval A=fcnchkvarargin{1}; 111 Amatrix=false; end 134

PAGE 144

113 ifAmatrix [mA,nA]=sizeA; 115 n=sizeX,1; ifmA 6 = nA 117 error subspace:NonSquareMatrix ,... Amustbeasquarematrix. 119 end ifmA 6 = n 121 error subspace:AXSizeMismatch ,... [ Thesize int2strsizeA... 123 ofthematrixAdoesnotmatchwith int2strn... -thenumberofrowsofX ] 125 end if : isequalA,A ; 127 error subspace:NonHermitianMatrixn ,... AmustbeHermitian. 129 end 131 end end 133 ifisfieldopts, fast subFast=opts.fast; 135 if : issparseX&& : issparseY if : subFast==0||subFast==1||... 137 subFast==2 dispstr= Fastlevelopts.fastmustbe0,1or2. ; 139 error subspace:WrongDiagnosticLevel ,dispstr end 141 else if : subFast==0||subFast==1 143 dispstr= Fastlevelopts.fastmustbe0,or1. ; error subspace:WrongDiagnosticLevel ,dispstr 145 end end 147 end ifisfieldopts, R 149 subR=opts.R; if : subR==0||subR==1 151 dispstr= Fastlevelopts.Rmustbe0and1. ; error subspace:WrongDiagnosticLevel ,dispstr 153 end end 155 ifisfieldopts, disp subDisplay=opts.disp; 157 dispstr= Diagnosticlevelopts.dispmustbe0,1,or2. ; if : subDisplay==0||subDisplay==1||... 159 subDisplay==2 error subspace:WrongDiagnosticLevel ,dispstr 161 end end 163 end ifsubDisplay==1||subDisplay==2 165 ifisemptyA fprintf nComputetheprincipalanglesbasedonmethodusingn ; 167 ifsubFast==0&&subR==0 ifissparseX&&issparseY 169 disp thetriangluarmatrixRfromQRfactorization. ; else 171 dispstrcat theSVDofmatrix[QF,QG],where ,... QF=orthXandQF=orthY. 173 disp '' ; disp Thecomputationistheslowest,butprovidesmore ; 135

PAGE 145

175 disp reliableandaccuracy. ; end 177 elseifsubFast==1&&subR==0 ifissparseX&&issparseY 179 disp Computetheprincipalanglesbasedoncosine-based ; disp algorithmusingRfromCholeskyfactorization. ; 181 else disp thegeometryapproachbytheSVDofthedifferenceof 183 disp principalvectorsontwosubspaces. disp '' ; 185 disp Thecomputationisfaster. ; end 187 elseifsubFast==2&&subR==0 disp thecosine-basedalgorithm. ; 189 disp '' ; disp Thecomputationisthefastestpossibleandleast ; 191 disp accuracyforsmallangles. ; end 193 ifsubR==1 disp thetriangluaruppermatrixRfromQRfactorization. ; 195 end else 197 disp[ ComputetheprincipalanglesintheA-based ,... scalarproduct ]; 199 ifsubFast==0&&subR==0 dispstrcat theSVDofmatrixA^1/2[QF,QG],where ,... 201 QF^HAQF=IandQG^HAQG=I. ; disp '' ; 203 elseifsubFast==1&&subR==0 205 disp thegeometryapproachbytheSVDofA^1/2timethe disp differenceofprincipalvectorsontwosubspaces. 207 disp '' ; disp Thecomputationisfaster. ; 209 elseifsubFast==2&&subR==0 disp thecosine-basedalgorithmintheA-basedscalarproduct. 211 disp '' ; disp Thecomputationisthefastestpossibleandleast ,... 213 accuracyforsmallanlges. ; end 215 ifsubR==1 disp byusingthetriangularuppermatrixRfromA-orthgonal ; 217 disp QRfactorization. ; end 219 end end 221 ifisemptyA&& : issparseX|| : issparseY 223 ifissparseX X=fullX; 225 end 227 ifissparseY Y=fullY; 229 end %Columnscalingfirst 231 fori=1:sizeX,2, normi=normX:,i,inf; 233 %Adjustmentmakestolconsistentwithexperimentalresults ifnormi>eps^.981 235 X:,i=X:,i/normi; end 136

PAGE 146

237 end fori=1:sizeY,2, 239 normi=normY:,i,inf; %Adjustmentmakestolconsistentwithexperimentalresults 241 ifnormi>eps^.981 Y:,i=Y:,i/normi; 243 end end 245 %Usedefaultopts.fast=0inorthupdate %whichisthesameasORTHfunction 247 QF=orthupdateX;%ThiscanalsobedoneusingQR.m QG=orthupdateY;%withopts.fast=1 249 [n,p]=sizeQF; q=sizeQG,2; 251 k=minp,q; ifsubNumber>k 253 error subspace:InvalidNumberInput ,... Invalidnumberinput ; 255 end ifsubFast==0&&subR==0 257 index=p+q; newindex=index-k+1; 259 L=[QFQG]; ifindex n 261 [ : ,s,Vhat]=svdL,0; ifsizes,1==1 263 s=1/sqrt2*s1; else 265 s=1/sqrt2*diags; end 267 theta=flipud2*asinsnewindex:index; else 269 [Vhat,s]=svdL ; ifsizes,1==1 271 s=1/sqrt2*s1; else 273 s=1/sqrt2*diags; end 275 theta1=2*asinsnewindex:n; theta2=diagzerosindex-n; 277 theta=flipud[theta1;theta2]; end 279 Ys=fliplrVhat1:p,newindex:index.*sqrt2; Zs=fliplr-Vhatp+1:index,newindex:index.*sqrt2; 281 elseifsubFast==1&&subR==0 283 ifp q [Ys, : ,Zs]=svdQF *QG,0; 285 else [Zs, : ,Ys]=svdQG *QF,0; 287 end L=QF*Ys-QG*Zs; 289 [ : ,s,Vl]=svdL,0; ifsizes,1==1 291 s=0.5*s1; else 293 s=0.5*diags; end 295 theta=flipud2*asins; Ys=fliplrYs*Vl; 297 Zs=fliplrZs*Vl; 137

PAGE 147

299 elseifsubFast==2&&subR==0 [Ys,s,Zs]=svdQF *QG,0; 301 ifsizes,1==1 %makesuresiscolumnforoutput 303 s=s1; end 305 s=diags; theta=acoss; 307 end ifsubR==0 309 U=QF*Ys; V=QG*Zs; 311 end ifsubR==1 313 ifsubFast==1||subFast==2 error subspace:Option ,... 315 Optionsopts.fastshouldbe0 ; else 317 [theta,U,V]=subspace_generalX,Y; end 319 end ifsubR==0 321 ifisemptysubNumber varargout1={U:,1:k}; 323 varargout2={V:,1:k}; varargout3={Ys:,1:k}; 325 varargout4={Zs:,1:k}; 327 else K=subNumber; 329 theta=theta1:K,:; varargout1={U:,1:K}; 331 varargout2={V:,1:K}; varargout3={Ys:,1:K}; 333 varargout4={Zs:,1:K}; 335 end elseifsubR==1 337 ifisemptysubNumber varargout1={U:,1:k}; 339 varargout2={V:,1:k}; 341 else K=subNumber; 343 theta=theta1:K,:; varargout1={U:,1:K}; 345 varargout2={V:,1:K}; end 347 end end 349 ifisemptyA&&issparseX&&issparseY 351 %workforfullrankinputsXandY [n,p]=sizeX; 353 q=sizeY,2; k=minp,q; 355 ifsubNumber>k error subspace:InvalidNumberInput ,... 357 Invalidnumberinput ; end 359 ifsubFast==0 L1=[XY]; 138

PAGE 148

361 R=qrL1,0; R1=R1:p,p+1:p+q; 363 R2=Rp+1:end,p+1:p+q; a=minsizeL1; 365 L2=[R1;R2]; RY=qrL2,0; 367 L=[speyepR1/RY; sparsea-p,pR2/RY]; 369 index=p+q; newindex=index-k+1; 371 ifindex n 373 [ : ,s,Vhat]=svdfullL,0; ifsizes,1==1 375 s=1/sqrt2*s1; else 377 s=1/sqrt2*diags; end 379 theta=flipud2*asinsnewindex:index; Ys=fliplrVhat1:p,newindex:index.*sqrt2; 381 Zs=fliplr-Vhatp+1:index,newindex:index.*sqrt2; clearVhat 383 else 385 [Vhat,s]=svdfullL ,0; ifsizes,1==1 387 s=1/sqrt2*s1; else 389 s=1/sqrt2*diags; end 391 theta1=2*asinsnewindex:n; theta2=diagzerosindex-n; 393 theta=flipud[theta1;theta2]; Vhat1=Vhat:,newindex:n; 395 U1=fliplrVhat11:p,:.*sqrt2; V1=fliplr-Vhat1p+1:index,:.*sqrt2; 397 Vhat2=Vhat:,1:index-n; U2=Vhat21:p,:.*sqrt2; 399 V2=Vhat2p+1:q+p,:.*sqrt2; Ys=[U2U1]; 401 Zs=[V2V1]; clearVhatU1U2V1V2 403 end RX=R1:p,1:p; 405 elseifsubFast==1 XX=X *X; 407 RX=cholXX; YY=Y *Y; 409 RY=cholYY; L=X/RX *Y/RY; 411 [Ys,s,Zs]=svdfullL,k; ifsizes,1==1 413 s=s1; end 415 s=diags; %r=sums>1; 417 %s1:r,1=1; theta=acoss; 419 end 421 U=X/RX*Ys; V=Y/RY*Zs; 139

PAGE 149

423 ifsubDisp==2 ifisemptysubNumber 425 barx,costheta, g else 427 barx,costheta1:K,:, g end 429 title CanonicalCorrelations xlabel Dimension 431 end 433 ifisemptysubNumber varargout1={U:,1:k}; 435 varargout2={V:,1:k}; else 437 K=subNumber; theta=theta1:K,:; 439 varargout1={U:,1:K}; varargout2={V:,1:K}; 441 end 443 end 445 if : isemptyA&& : issparseX|| : issparseY ifissparseX 447 X=fullX; end 449 ifissparseY Y=fullY; 451 end optsorth.A=A; 453 [QX,AQX]=orthupdateX,optsorth; [QY,AQY]=orthupdateY,optsorth; 455 p=sizeQX,2; q=sizeQY,2; 457 k=minp,q; ifsubFast==0&&subR==0 459 index=p+q; newindex=index-k+1; 461 S=[XY]; %[a1,b1,p1]=luS; 463 %S=p1 *a1; QS=orthupdateS,optsorth; 465 m=sizeQS,2; L=QS *[AQXAQY]; 467 [ : ,s,Vhat]=svdL,0; ifsizes,1==1 469 s=1/sqrt2*s1; else 471 s=1/sqrt2*diags; end 473 theta1=2*asinsnewindex:m; theta2=zerosindex-m,1; 475 theta=flipud[theta1;theta2]; Ys=fliplrVhat1:p,newindex:index.*sqrt2; 477 Zs=fliplr-Vhatp+1:index,newindex:index.*sqrt2; 479 elseifsubFast==1&&subR==0 ifp q 481 [Ys, : ,Zs]=svdQX *AQY,0; else 483 [Zs, : ,Ys]=svdAQY *QX,0; end 140

PAGE 150

485 S=QX*Ys-QY*Zs; QS=orthupdateS,optsorth; 487 L=QS *A*S; b=minsizeL; 489 [ : ,s,Vl]=svdL,0; s=0.5*diags; 491 theta1=2*asins; b1=minp,q; 493 theta2=zerosb1-b,1; theta=flipud[theta1;theta2]; 495 Ys=fliplrYs*Vl; Zs=fliplrZs*Vl; 497 elseifsubFast==2&&subR==0 499 [Ys,s,Zs]=svdQX *AQY,0; ifsizes,1==1 501 s=s1; end 503 s=diags; theta=acoss; 505 end ifsubR==0 507 U=QX*Ys; V=QY*Zs; 509 end ifsubR==1 511 ifsubFast==1 error subspace:Option Opts.fastshouldbe0forA ; 513 else [theta,U,V]=subspace_generalX,Y,A; 515 end end 517 ifisemptysubNumber varargout1={U:,1:k}; 519 varargout2={V:,1:k}; else 521 K=subNumber; theta=theta1:K,:; 523 varargout1={U:,1:K}; varargout2={V:,1:K}; 525 end end 527 ifsubDisplay==2 ifisemptysubNumber 529 bar1:k,costheta, g else 531 bar1:K,costheta1:K,:, g end 533 ifisemptyA title CanonicalCorrelations 535 xlabel Dimension else 537 title CosinesofprincipalanglesintheA-basedscalarproduct xlabel Dimension 539 end end 541 end 141

PAGE 151

1 function[theta,U,V]=subspace_generalX,Y,A 3 %inputmatricesXandYhavefullrank %X=independentColumnupdateX; 5 %Y=independentColumnupdateY; [n,p]=sizeX; 7 q=sizeY,2; k=minp,q; 9 L1=[XY]; index=p+q; 11 newindex=index-k+1; 13 ifnargin<3 R=qrL1,0; 15 R=triuR; R1=R1:p,p+1:p+q; 17 ifp+q
PAGE 152

V=Y/RY*Zs; 63 else [ : ,R]=mgs_AA,L1; 65 R1=R1:p,p+1:p+q; R2=Rp+1:end,p+1:p+q; 67 [ : ,RY]=mgs_AA,Y; L=[eyepR1/RY; 69 zerosq,pR2/RY]; [ : ,s,Vhat]=svdL,0; 71 ifsizes,1==1 s=1/sqrt2*s1; 73 else s=1/sqrt2*diags; 75 end theta=flipud2*asinsnewindex:index; 77 Ys=fliplrVhat1:p,newindex:index.*sqrt2; Zs=fliplr-Vhatp+1:index,newindex:index.*sqrt2; 79 clearVhat RX=R1:p,1:p; 81 U=X/RX*Ys; V=Y/RY*Zs; 83 end BelowisalistofoptionsORTHPDATEtakes: opts.fast=0Backwardcompatiblewithbuilt-inMATLABfunctionORTH. opts.fast=1UsingQRfactorizationwithcompletepivoting. opts.fast=2UsingCholeskyfactorization. opts.AUse A -basedscalarproduct,where A ispositivedenite. opts.tolzeroCut-oforzerovectors. opts.tolCut-ofordependentvectors. opts.dispDiagnosticinformationdisplaylevel0 ; 1. 1 function[Q,varargout]=orthupdateX,varargin %ORTHUPDATEFindanorthonormalbasisfortherangeofX 3 %Q=ORTHUPDATEXreturnsanorthonormalbasisfortherangeofX. %ThatisQ *Q=I,thecolumnsofQspanthesamespaceasthecolumns 5 %ofX,andthenumberofcolumnsofQistherankofX.Backward %compatiblewithbuilt-inMATLABandOCTAVEcodesORTH,whichit 7 %isreplacing. % 9 %ClasssupportforinputX: %fullfloat:double,single, 11 %sparsefloat:double % 13 %[Q,R]=orthupdateXalsoreturnsRsuchthatQ=X*R. % 15 %[Q,R]=orthupdateX,optsspecifyoptions: %OPTS.tolzero:cut-offforzerovectors[float|{0}] 17 %OPTS.tol:cut-offfordependentvectors[float|{determinedbythecode}] %OPTS.A:A-basedscalarproduct[floatmatrixorfunctionhandle|{null}] 19 %OPTS.fast:usefaster,butlessaccuratealgorithm[{0}|1|2] %OPTS.disp:diagnosticinformationdisplaylevel[{0}|1] 21 % %Indefault,thedimensionoftherangespaceistakenasthenumberof 23 %singularvaluesofXgreaterthanopts.tol.Itiscomputedas 143

PAGE 153

%maxsizeX*maxsvd{X}*epsclassX; 25 % %Ifopts.Aispresent,Q=orthupdateX,optscomputesanorthonormalbasis 27 %QfortherangeofX,relativetothescalarproductusingapositive %definiteandselfadjointmatrixA.Thatis,Q *A*Q=I,thecolumnsof 29 %QspanthesamespaceascolumnsofX,andrankQ=rankX.Missing %opts.Aismathematicallyequivalenttoopts.A=identity. 31 % %[Q,AQ,R]=orthupdateX,optsalsogivesAQ=A*QandQ=X*R. 33 % %opts.A:eitheranmxmpositivedefiniteandselfadjointmatrixA 35 %oralinearoperatorA=Avthatispositivedefiniteselfadjoint; % 37 %Classsupportforinputopts.A: %fullfloat:double,single, 39 %sparsefloat:double %handletoafunctionmultiplyingA*X 41 % 43 ifnargout>3 error orth:TooManyOutputs Toomanyoutputarguments. 45 end 47 if : isnumericX error orth:XNotNumeric ,... 49 Xmustbeanumericmatrix. end 51 ifissparseX 53 warning orth:XSparse ,... SparseXmayresultinafill-ininQ 55 end 57 [m,n]=sizeX;%thefirstinputparameterisrequired 59 %Setthedefaults A=[];orthTolZero=[];orthTol=[];orthFast=0;orthDisplay=0; 61 ifnargin>1 63 opts=varargin{1}; if : isaopts, struct 65 error orth:OptionsNotStructure ,... Optionsargumentmustbeastructure. 67 end ifisfieldopts, tolzero 69 orthTolZero=opts.tolzero; if : isscalarorthTolZero 71 error orth:NonScalarOrthTolZero ,... orthTolZeromustbeascalar. 73 end if : orthTolZero>0 75 error orth:NonPositiveOrthTolZero ,... orthTolZeromustbepositiv. 77 end end 79 ifisfieldopts, tol orthTol=opts.tol; 81 if : isscalarorthTol error orth:NonScalarOrthTol ,... 83 orthTolmustbeascalar. end 85 if : orthTol>0 144

PAGE 154

error orth:orthTol ,... 87 orthTolmustbeapositivescalar. end 89 end %ProcesstheinputA 91 ifisfieldopts, A A=opts.A; 93 ifisnumericA Amatrix=true; 95 else %BycheckingthefunctionAwithfcnchk,wecannowusedirect 97 %functionevaluationontheresult,withoutresortingtofeval A=fcnchkvarargin{1}; 99 Amatrix=false; end 101 ifAmatrix [mA,nA]=sizeA; 103 ifmA 6 = nA error orth:NonSquareMatrix ,... 105 Amustbeasquarematrix. end 107 ifmA 6 = m error orth:AXSizeMismatch ,... 109 [ Thesize int2strsizeA... ofthematrixAdoesnotmatchwith int2strm... 111 -thenumberofrowsofX ] end 113 if : isequalA,A ; error orth:NonHermitianMatrixn ,... 115 AmustbeHermitian. end 117 end end 119 ifisfieldopts, fast orthFast=opts.fast; 121 dispstr= Fastlevelopts.fastmustbe0,1,or2. ; if : orthFast==0||orthFast==1||... 123 orthFast==2 error orth:WrongDiagnosticLevel ,dispstr 125 end end 127 ifisfieldopts, disp orthDisplay=opts.disp; 129 dispstr= Diagnosticlevelopts.dispmustbe0,1,or2. ; if : orthDisplay==0||orthDisplay==1||... 131 orthDisplay==2 error orth:WrongDiagnosticLevel ,dispstr 133 end end 135 end 137 %normalizationandcleaning,ifordered if : isemptyorthTolZero 139 normXc=maxabsX;%origianlisnormXc=maxabsX orthTolZeroI=normXc
PAGE 155

end 149 X=X/diagnornXc;%normalizingtherest clearnormXcorthTolZeroIdiagnornXc; 151 end iforthDisplay==1 153 ifisemptyA fprintf nAnorthonormalbasisfortherangeofinputmatrix.n ; 155 iforthFast==0 disp Themethodisbasedonsingularvaluedecomposition. 157 disp '' ; disp Thecomputationistheslowest,butprovidesmore ; 159 disp reliableandconsistentwithrankdetermination. ; elseiforthFast==1 161 disp ThemethodisbasedonQRdecomposition. disp Thecomputationisfaster,butlessaccurate. ; 163 elseiforthFast==2 disp ThemethodisbasedonCholeskyfactorization. 165 disp Thecomputationisthefastestpossibleandleastreliable. ; end 167 else fprintf nAnA-orthonoramlbasisQfortherangeofXn ; 169 fprintf Thatis:n ; fprintf ttQ '' *A*Q=I,n 171 end end 173 %Normalize,soORTHbelowwouldnotdeletewantedvectors fori=1:sizeX,2, 175 normXi=normX:,i,inf; %Adjustmentmakestolconsistentwithexperimentalresults 177 ifnormXi>eps^.981 X:,i=X:,i/normXi; 179 %Elseorthwilltakecareofthis end 181 end 183 ifisemptyA%standardscalarproduct iforthFast==0%themostrobustandslowest,SVD-based 185 ifissparseX X=fullX; 187 end [U,S,V]=svdX,0; 189 ifm>1 191 s=diagS; elseifm==1 193 s=S1; else 195 s=0; end 197 ifisemptyorthTol orthTol=maxm,n*maxs*epsclassX; 199 end r=sums>orthTol; 201 Q=U:,1:r; R=V:,1:r/S1:r,1:r; 203 elseiforthFast==1%faster,butlessaccurate,QR-based %%QRdecomposition 205 [Q,R,E]=qrX,0; %Determiner=effectiverank 207 ifisemptyorthTol orthTol=eps*normX, fro ; 209 end 146

PAGE 156

r=sumabsdiagR>orthTol; 211 r=r1;%fixforcasewhereRisvector. %UsefirstrcolumnsofQ. 213 ifr>0 Q=Q:,1:r; 215 %Cosmeticsignadjustment Q=-Q; 217 Q:,r=-Q:,r; else 219 Q=[]; end 221 A=eyen,n; R=A:,E*eyen,r/R1:r,1:r; 223 clearE; else%thefastestpossibleandleastreliable 225 XX=X *X; if : isrealXX 227 XX=XX+XX *0.5; end 229 [R,cholFlag]=cholXX; ifcholFlag 6 = 0 231 fprintf TheinputmatrixXisnotfullrank.n ; fprintf Pleasechooseotheroptionsforopts.fast:0or1.n ; 233 error orth:NonFullMatrix ,... TheinputmatrixXisnotfullrank. ; 235 end Q=X/R; 237 R=eyen,n/R; end 239 ifnargout>1 varargout1={R}; 241 end else 243 % %ComputeanA-orthonormalbasisAlgorithm: 245 % ifissparseX 247 X=fullX; end 249 [U,S,V]=svdX,0; ifm>1 251 s=diagS; elseifm==1 253 s=S1; else 255 s=0; end 257 ifisemptyorthTol orthTol=maxm,n*maxs*epsclassX; 259 end r=sums>orthTol; 261 X=U:,1:r; [m,n]=sizeX; 263 R1=V:,1:r/S1:r,1:r; %[m,n]=sizeX; 265 tol=maxm,n*eps; if : ischarA 267 AX=A*X; else 269 AX=fevalA,X; end 271 XAX=X *AX; 147

PAGE 157

273 XAX=0.5.*XAX +XAX; %[U,S,V]=svdXAX;clearV:%unused 275 [U,S, : ]=svdXAX; ifn>1 277 s=diagS; elseifn==1,s=S1; 279 elses=0; end 281 %Adjustmentmakestolconsistentwithexperimentalresults 283 threshold1=maxm,n*maxs*eps^1.1; r=sums>threshold1; 285 sr+1:sizes,1=1; S=diag1./sqrts,0; 287 R=U*S; X=X*U*S; 289 AX=AX*U*S; XAX=X *AX; 291 R=R1*R; %ChecksubspaceaErroragainsttolerance 293 subspaceaError=normestXAX1:r,1:r-eyer; %Checkinternalprecision,e.g.,80bitFPUregistersofP3/P4 295 precision_test=[1eps/1024-1]*[111] ; ifsubspaceaError1 315 s=diagS; elseifn==1,s=S1; 317 elses=0; end 319 threshold2=maxm,n*maxs*eps; r=sums>threshold2; 321 S=diag1./sqrts1:r,0; R1=U:,1:r*S1:r,1:r; 323 Q=X*U:,1:r*S1:r,1:r; varargout1={AX*U:,1:r*S1:r,1:r}; 325 varargout2={R*R1}; end 148

PAGE 158

TheMATLABcodeofmsga.misasfollows: 1 function[Q,R]=mgsaA,X 2 %MGSAFindA-orthogonalQRfactorization 3 %OurfunctionisbasedonModifiedGram-SchmidtrespecttoanA-based 4 %scalarproduct,whereAisapositivedefiniteandselfadjointmatrix. 5 % 6 %[Q,R]=mgsaA,XreturnsatriangularmatrixRandanA-orthogonalbasisQ, 7 %suchthatQ *A*Q=I. 8 % 9 10 [m,n]=sizeX; 11 Q=zerosm,n; 12 R=zerosn,n; 13 V=X; 14 fori=1:n 15 Ri,i=sqrtV:,i *A*V:,i; 16 Q:,i=V:,i/Ri,i; 17 forj=i+1:n 18 Ri,j=Q:,i *A*V:,j; 19 V:,j=V:,j-Ri,j*Q:,i; 20 end 21 end 149

PAGE 159

REFERENCES [1]T.Ando.Majorizationsandinequalitiesinmatrixtheory. LinearAlgebraAppl. 199:17{67,1994.doi:10.1016/0024-3795-7. [2]M.E.Argentati. PrincipalanglesbetweensubspacesasrelatedtoRayleighquotientandRaleigh{Ritzinequalitieswithapplicationstoeigenvalueaccuracyand aneigenvaluesolver .PhDthesis,UniversityofColoradoDenver,2003. [3]M.E.Argentati,A.V.Knyazev,C.C.Paige,andI.Panayotov.Boundson changesinRitzvaluesforaperturbedinvariantsubspaceofaHermitianmatrix. SIAMJ.MatrixAnal.Appl. ,30:548{559,2008.doi:10.1137/070684628. [4]Z.Bai,J.Demmel,J.Dongarra,A.Ruhe,andH.vanderVorst,editors. Templatesforthesolutionofalgebraiceigenvalueproblems ,volume11of Software,Environments,andTools .SocietyforIndustrialandAppliedMathematics SIAM,Philadelphia,PA,2000.URL http://web.eecs.utk.edu/ ~ dongarra/ etemplates/ .Apracticalguide. [5]R.Bhatia. Matrixanalysis ,volume169of GraduateTextsinMathematics Springer-Verlag,NewYork,1997.ISBN0-387-94846-5. [6]R.Bhatia,C.Davis,andF.Kittaneh.Someinequalitiesforcommutatorsand anapplicationtospectralvariation. AequationesMath. ,41:70{78,1991. doi:10.1007/BF02227441. [7]R.Bhatia,F.Kittaneh,andR.Li.Someinequalitiesforcommutatorsandan applicationtospectralvariation.II. LinearandMultilinearAlgebra ,43-3: 207{219,1997.doi:10.1080/03081089708818526. [8] A.Bjorck. Numericalmethodsforleastsquaresproblems .SocietyforIndustrial andAppliedMathematicsSIAM,Philadelphia,PA,1996.ISBN0-89871-360-9. doi:10.1137/1.9781611971484. [9] A.BjorckandG.Golub.Numericalmethodsforcomputinganglesbetween linearsubspaces. Math.Comp. ,27:579{594,1973.URL http://www.ams.org/ journals/mcom/1973-27-123/S0025-5718-1973-0348991-3/ [10]N.BosnerandZ.Drmac.SubspacegapresidualsforRayleigh-Ritzapproximations. SIAMJ.MatrixAnal.Appl. ,31:54{67,2009.doi:10.1137/070689425. [11]A.BottcherandI.M.Spitkovsky.Agentleguidetothebasicsoftwoprojectionstheory. LinearAlgebraAppl. ,432:1412{1459,2010.URL http: //www.sciencedirect.com/science/article/pii/S0024379509005722 150

PAGE 160

[12]T.F.Chan.Animprovedalgorithmforcomputingthesingularvaluedecomposition. ACMTrans.Math.Software ,8:72{83,1982.doi:10.1145/355984.355990. [13]F.Chatelin. SpectralApproximationofLinearOperators .Society forIndustrialandAppliedMathematicsSIAM,Philadelphia,PA,2011. doi:10.1137/1.9781611970678. [14]G.E.ChoandI.C.F.Ipsen.Ifamatrixhasonlyasingleeigenvalue,howsensitiveistheeigenvalue?TechnicalReportCRSC-TR97-20,CenterforResearchin ScienticComputation,DepartmentofMathematics,UniversityofNorthCarolina,1997.URL www4.ncsu.edu/ ~ ipsen/ps/tr97-20.ps [15]C.DavisandW.M.Kahan.Therotationofeigenvectorsbyaperturbation.III. SIAMJ.Numer.Anal. ,7:1{46,1970.doi:10.1137/0707001. [16]T.A.Davis.Algorithm915,SuiteSparseQR:multifrontalmultithreadedrankrevealingsparseQRfactorization. ACMTrans.Math.Software ,38:Art.8, 22,2011.doi:10.1145/2049662.2049670. [17]K.DeCockandB.DeMoor.Canonicalcorrelationsbetweeninputand outputprocessesoflinearstochasticmodels. InProceedingsoftheFifteenth InternationalSymposiumontheMathematicalTheoryofNetworksandSystemsMTNS2002 ,Aug12-16,2002.URL http://citeseerx.ist.psu.edu/ viewdoc/summary?doi=10.1.1.6.3374 [18]F.Deutsch.TheanglebetweensubspacesofaHilbertspace.In Approximationtheory,waveletsandapplicationsMaratea,1994 ,volume454of NATO Adv.Sci.Inst.Ser.CMath.Phys.Sci. ,pages107{130.KluwerAcad.Publ., Dordrecht,1995. [19]J.J.Dongarra,C.B.Moler,J.R.Bunch,andG.W.Stewart. LINPACKUsers' Guide .SocietyforIndustrialandAppliedMathematicsSIAM,Philadelphia, PA,1979.doi:10.1137/1.9781611971811. [20]Z.Drmac.OnrelativeresidualboundsfortheeigenvaluesofaHermitianmatrix. LinearAlgebraAppl. ,244:155{163,1996.doi:10.1016/0024-3795-5. [21]Z.Drmac.OnprincipalanglesbetweensubspacesofEuclideanspace. SIAMJ. MatrixAnal.Appl. ,22:173{194,2000.doi:10.1137/S0895479897320824. [22]A.EdelmanandB.D.Sutton.Thebeta-Jacobimatrixmodel,theCSdecomposition,andgeneralizedsingularvalueproblems. Found.Comput.Math. ,8: 259{285,2008.doi:10.1007/s10208-006-0215-9. [23]A.Galantai.Subspaces,anglesandpairsoforthogonalprojections. Linear MultilinearAlgebra ,56:227{260,2008.doi:10.1080/03081080600743338. [24]A.GalantaiandCs.J.Heged}us.Jordan'sprincipalanglesincomplexvector spaces. Numer.LinearAlgebraAppl. ,13:589{598,2006.doi:10.1002/nla.491. 151

PAGE 161

[25]A.George,Kh.Ikramov,andA.B.Kucherov.Somepropertiesofsymmetric quasi-denitematrices. SIAMJ.MatrixAnal.Appl. ,21:1318{1323electronic,2000.doi:10.1137/S0895479897329400. [26]J.R.Gilbert,C.Moler,andR.Schreiber.SparsematricesinMATLAB:designandimplementation. SIAMJ.MatrixAnal.Appl. ,13:333{356,1992. doi:10.1137/0613024. [27]R.Gittins. Canonicalanalysis:areviewwithapplicationsinecology .BiomathematicsBerlin.Springer-Verlag,1985.ISBN9783540136170.URL http: //books.google.com/books?id=0eMUAQAAIAAJ [28]G.H.GolubandC.F.VanLoan. Matrixcomputations .JohnsHopkinsStudies intheMathematicalSciences.JohnsHopkinsUniversityPress,Baltimore,MD, thirdedition,1996.ISBN0-8018-5413-X;0-8018-5414-8. [29]G.H.GolubandH.Zha.Perturbationanalysisofthecanonicalcorrelations ofmatrixpairs. LinearAlgebraAppl. ,210:3{28,1994.doi:10.1016/00243795-4. [30]G.H.GolubandH.Zha.Thecanonicalcorrelationsofmatrixpairsandtheir numericalcomputation.In LinearalgebraforsignalprocessingMinneapolis, MN,1992 ,volume69of IMAVol.Math.Appl. ,pages27{49.Springer,New York,1995. [31]P.R.Halmos.Twosubspaces. Trans.Amer.Math.Soc. ,144: 381{389,1969.URL http://www.ams.org/journals/tran/1969-144-00/ S0002-9947-1969-0251519-5/ [32]E.M.Harrell,II.GeneralizationsofTemple'sinequality. Proc.Amer. Math.Soc. ,69:271{276,1978.URL http://www.ams.org/journals/proc/ 1978-069-02/S0002-9939-1978-0487733-4/ [33]P.HeggernesandP.Matstoms.Findinggoodcolumnorderingsforsparseqr factorization.Technicalreport,InSecondSIAMConferenceonSparseMatrices, 1996.URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1. 17.408 [34]N.J.HighamandA.H.Al-Mohy.Computingmatrixfunctions. ActaNumer. 19:159{208,2010.doi:10.1017/S0962492910000036. [35]R.A.HornandC.R.Johnson. Matrixanalysis .CambridgeUniversityPress, Cambridge,1990.ISBN0-521-38632-2.Correctedreprintofthe1985original. [36]R.A.HornandC.R.Johnson. Topicsinmatrixanalysis .CambridgeUniversity Press,Cambridge,1994.ISBN0-521-46713-6.Correctedreprintofthe1991 original. 152

PAGE 162

[37]H.Hotelling.Relationsbetweentwosetsofvariates. Biometrika ,28:321{377, 1936.doi:10.1093/biomet/28.3-4.321. [38]I.C.F.IpsenandC.D.Meyer.Theanglebetweencomplementarysubspaces. Amer.Math.Monthly ,102:904{911,1995.doi:10.2307/2975268. [39]R.A.JohnsonandD.W.Wichern. Appliedmultivariatestatisticalanalysis PearsonPrenticeHall,UpperSaddleRiver,NJ,sixthedition,2007.ISBN9780-13-187715-3;0-13-187715-1. [40]C.Jordan.Essaisurlageometrieandimensions. Amer.Math.Monthly ,3: 103{174,1875. [41]A.Jujunashvili. Anglesbetweeninnitedimensionalsubspaces .PhDthesis, UniversityofColoradoDenver,2005. [42]W.M.Kahan.Thenearestorthogonalorunitarymatrix. Lecturenotes,private communication ,2011. [43]T.Kailath.Aviewofthreedecadesoflinearlteringtheory. IEEETrans. InformationTheory ,IT-20:146{181,1974.doi:10.1109/TIT.1974.1055174. [44]T.Kato.Ontheupperandlowerboundsofeigenvalues. J.Phys.Soc.Japan ,4: 334{339,1949.doi:10.1143/JPSJ.4.334. [45]T.Kim,O.Arandjelovic,andR.Cipolla.Boostedmanifoldprincipalangles forimageset-basedrecognition. PatternRecognition ,40:2475{2484,2007. doi:10.1016/j.patcog.2006.12.030. [46]A.V.Knyazev.SharpapriorierrorestimatesfortheRayleigh-Ritzmethodwith noassumptionsonxedsignorcompactness. Math.Notes ,38:998{1002,1985. doi:10.1007/BF01157020. [47]A.V.Knyazev.NewestimatesforRitzvectors. Math.Comp. ,66:985{995, 1997.doi:10.1090/S0025-5718-97-00855-7. [48]A.V.KnyazevandM.E.Argentati.Principalanglesbetweensubspacesinan A -basedscalarproduct:algorithmsandperturbationestimates. SIAMJ.Sci. Comput. ,23:2008{2040electronic,2002.doi:10.1137/S1064827500377332. [49]A.V.KnyazevandM.E.Argentati.OnproximityofRayleighquotientsfor dierentvectorsandRitzvaluesgeneratedbydierenttrialsubspaces. Linear AlgebraAppl. ,415:82{95,2006.doi:10.1016/j.laa.2005.06.032. [50]A.V.KnyazevandM.E.Argentati.Majorizationforchangesinanglesbetween subspaces,Ritzvalues,andgraphLaplacianspectra. SIAMJ.MatrixAnal. Appl. ,29:15{32electronic,2006/07.doi:10.1137/060649070. 153

PAGE 163

[51]A.V.KnyazevandM.E.Argentati.Rayleigh-Ritzmajorizationerrorbounds withapplicationstoFEM. SIAMJ.MatrixAnal.Appl. ,31:1521{1537,2009. doi:10.1137/08072574X. [52]A.V.KnyazevandM.E.Argentati.Rayleigh-ritzmajorizationerrorbounds withapplicationstoFEMandsubspaceiterations,2009.URL http://arxiv. org/abs/math/0701784 [53]A.V.Knyazev,A.Jujunashvili,andM.Argentati.AnglesbetweeninnitedimensionalsubspaceswithapplicationstotheRayleigh-Ritzandalternatingprojectorsmethods. J.Funct.Anal. ,259:1323{1345,2010. doi:10.1016/j.jfa.2010.05.018. [54]V.Kostrykin,K.A.Makarov,andA.K.Motovilov.Ontheexistenceofsolutions totheoperatorRiccatiequationandthetantheorem. IntegralEquations OperatorTheory ,51:121{140,2005.doi:10.1007/s00020-003-1248-6. [55]A.W.Marshall,I.Olkin,andB.C.Arnold. Inequalities:theoryofmajorization anditsapplications .SpringerSeriesinStatistics.Springer,NewYork,second edition,2011.ISBN978-0-387-40087-7.doi:10.1007/978-0-387-68276-1. [56]R.Mathias.QuadraticresidualboundsfortheHermitianeigenvalue problem. SIAMJ.MatrixAnal.Appl. ,19:541{550electronic,1998. doi:10.1137/S0895479896310536. [57]C.Moler.MATLABincorporatesLAPACK:Increasingthespeedandcapabilitiesofmatrixcomputation,2000.URL http://www.math.nps.navy.mil/ ~ art/ma3046/handouts/BLAS_Effects.pdf .MATLABNewsandNotes. [58]A.K.Motovilov.Commenton`thetantheoremwithrelaxedconditions',by Y.Nakatsukasa,2012.URL http://arxiv.org/abs/1204.4441 [59]Y.Nakatsukasa.Thetan theoremwithrelaxedconditions. LinearAlgebra Appl. ,436:1528{1534,2012.doi:10.1016/j.laa.2011.08.038. [60]M.Omladic.Spectraofthedierenceandproductofprojections. Proc.Amer. Math.Soc. ,99:317{318,1987.doi:10.2307/2046632. [61]E.Ovtchinnikov.ClusterrobusterrorestimatesfortheRayleigh-Ritzapproximation.II.Estimatesforeigenvalues. LinearAlgebraAppl. ,4151:188{209, 2006.doi:10.1016/j.laa.2005.06.041. [62]C.C.PaigeandM.A.Saunders.Towardsageneralizedsingularvaluedecomposition. SIAMJ.Numer.Anal. ,18:398{405,1981.doi:10.1137/0718026. [63]C.C.PaigeandM.Wei.HistoryandgeneralityoftheCSdecomposition. Linear AlgebraAppl. ,208/209:303{326,1994.doi:10.1016/0024-3795-4. 154

PAGE 164

[64]B.N.Parlett. Thesymmetriceigenvalueproblem ,volume20of ClassicsinAppliedMathematics .SocietyforIndustrialandAppliedMathematicsSIAM, Philadelphia,PA,1998.ISBN0-89871-402-8.doi:10.1137/1.9781611971163. Correctedreprintofthe1980original. [65]G.Polya.RemarkonWeyl'snoteInequalitiesbetweenthetwokindsofeigenvaluesofalineartransformation.". Proc.Nat.Acad.Sci.U.S.A. ,36:49{51, 1950.URL http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1063130/ [66]L.Qiu,Y.Zhang,andC.Li.UnitarilyinvariantmetricsontheGrassmannspace. SIAMJ.MatrixAnal.Appl. ,27:507{531electronic,2005. doi:10.1137/040607605. [67]A.Ruhe.Computationofeigenvaluesandeigenvectors.In SparsematrixtechniquesAdv.Course,TechnicalUniv.Denmark,Copenhagen,1976 pages130{184.LectureNotesinMath.,Vol.572.Springer,Berlin,1977. doi:10.1007/BFb0116617. [68]Y.Saad. Numericalmethodsforlargeeigenvalueproblems .Classicsinappliedmathematics.SocietyforIndustrialandAppliedMathematics,Philadelphie,2011.ISBN978-1-611-97072-2.URL http://opac.inria.fr/record= b1132864 [69]Y.Shapira. Matrix-basedmultigrid ,volume2of NumericalMethodsandAlgorithms .Springer,NewYork,secondedition,2008.ISBN978-0-387-49764-8. doi:10.1007/978-0-387-49765-5.Theoryandapplications. [70]G.W.Stewart.Ontheperturbationofpseudo-inverses,projectionsandlinear leastsquaresproblems. SIAMRev. ,19:634{662,1977.doi:10.1137/1019104. [71]G.W.Stewart. Matrixalgorithms.Vol.II .SocietyforIndustrialandAppliedMathematicsSIAM,Philadelphia,PA,2001.ISBN0-89871-503-2. doi:10.1137/1.9780898718058.Eigensystems. [72]G.W.Stewart.Onthenumericalanalysisofobliqueprojectors. SIAMJ.Matrix Anal.Appl. ,32:309{348,2011.doi:10.1137/100792093. [73]G.W.StewartandJ.Sun. Matrixperturbationtheory .ComputerScience andScienticComputing.AcademicPressInc.,Boston,MA,1990.ISBN012-670230-6. [74]J.Sun.Perturbationofanglesbetweenlinearsubspaces. J.Comput.Math. ,5 :58{61,1987.URL http://www.global-sci.org/jcm/volumes/v5n1/pdf/ 051-58.pdf [75]J.Sun.EigenvaluesofRayleighquotientmatrices. Numer.Math. ,59:603{ 614,1991.doi:10.1007/BF01385798. 155

PAGE 165

[76]G.Temple.ThetheoryofRayleigh'sprincipleasappliedtocontinuoussystems. Proc.Roy.Soc.London.Ser.A. ,119:276{293,1928.doi:10.1098/rspa.1928.0098. [77]L.N.TrefethenandD.Bau,III. Numericallinearalgebra .SocietyforIndustrial andAppliedMathematicsSIAM,Philadelphia,PA,1997.ISBN0-89871-361-7. [78]P.VanOverscheeandB.DeMoor. Subspaceidenticationforlinearsystems KluwerAcademicPublishers,Boston,MA,1996.ISBN0-7923-9717-7.Theory| implementation|applications,With1IBM-PCoppydisk.5inch;HD. [79]P.Wedin.Onanglesbetweensubspacesofanitedimensionalinnerproduct space.InBoKgstrmandAxelRuhe,editors, MatrixPencils ,volume973of LectureNotesinMathematics ,pages263{285.SpringerBerlinHeidelberg,1983. ISBN978-3-540-11983-8.doi:10.1007/BFb0062107. [80]J.Xie.AnoteontheDavis-Kahansin theorem. LinearAlgebraAppl. ,258: 129{135,1997.doi:10.1016/S0024-3795-8. [81]J.Xie.Residualboundsforunitarilyinvariantnormsonclusteredeigenvalues. LinearAlgebraAppl. ,252:107{113,1997.doi:10.1016/0024-3795-X. [82]Y.ZhangandL.Qiu.Fromsubadditiveinequalitiesofsingularvaluestotriangle inequalitiesofcanonicalangles. SIAMJ.MatrixAnal.Appl. ,31:1606{1620, 2009/10.doi:10.1137/09074841X. 156