A First Course in Linear Algebra

by by Robert A. Beezer

Copyright

© 2004 by Robert A. Beezer.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the appendix entitled “GNU Free Documentation License”.

The most recent version of this work can always be found at http://linear.ups.edu.

Next topic: Author Biography

Author Biography

Robert A. Beezer is a Professor of Mathematics at the University of Puget Sound, where he has been on the faculty since 1984. He received a B.S. in Mathematics (with an Emphasis in Computer Science) from the University of Santa Clara in 1978, a M.S. in Statistics from the University of Illinois at Urbana-Champaign in 1982 and a Ph.D. in Mathematics from the University of Illinois at Urbana-Champaign in 1984. He teaches calculus, linear algebra and abstract algebra regularly, while his research interests include the applications of linear algebra to graph theory. His professional website is at http://buzzard.ups.edu .

Previous topic: Copyright

Next topic: Edition

Edition

Version 2.30.
December 23, 2011.

Previous topic: Author Biography

Next topic: Publisher

Publisher

Robert A. Beezer
Department of Mathematics and Computer Science
University of Puget Sound
1500 North Warner
Tacoma, Washington 98416-1043
USA

Previous topic: Edition

Next topic: Dedication

Dedication

To my wife, Pat.

Previous topic: Publisher

Next topic: Theorems

Chapter 1. Theorems

Theorems

Section WILA

Section SSLE
Theorem EOPSS Equation Operations Preserve Solution Sets

Section RREF
Theorem REMES Row-Equivalent Matrices represent Equivalent Systems
Theorem REMEF Row-Equivalent Matrix in Echelon Form
Theorem RREFU Reduced Row-Echelon Form is Unique

Section TSS
Theorem RCLS Recognizing Consistency of a Linear System
Theorem ISRN Inconsistent Systems, r and n
Theorem CSRN Consistent Systems, r and n
Theorem FVCS Free Variables for Consistent Systems
Theorem PSSLS Possible Solution Sets for Linear Systems
Theorem CMVEI Consistent, More Variables than Equations, Infinite solutions

Section HSE
Theorem HSC Homogeneous Systems are Consistent
Theorem HMVEI Homogeneous, More Variables than Equations, Infinite solutions

Section NM
Theorem NMRRI Nonsingular Matrices Row Reduce to the Identity matrix
Theorem NMTNS Nonsingular Matrices have Trivial Null Spaces
Theorem NMUS Nonsingular Matrices and Unique Solutions
Theorem NME1 Nonsingular Matrix Equivalences, Round 1

Section VO
Theorem VSPCV Vector Space Properties of Column Vectors

Section LC
Theorem SLSLC Solutions to Linear Systems are Linear Combinations
Theorem VFSLS Vector Form of Solutions to Linear Systems
Theorem PSPHS Particular Solution Plus Homogeneous Solutions

Section SS
Theorem SSNS Spanning Sets for Null Spaces

Section LI
Theorem LIVHS Linearly Independent Vectors and Homogeneous Systems
Theorem LIVRN Linearly Independent Vectors, r and n
Theorem MVSLD More Vectors than Size implies Linear Dependence
Theorem NMLIC Nonsingular Matrices have Linearly Independent Columns
Theorem NME2 Nonsingular Matrix Equivalences, Round 2
Theorem BNS Basis for Null Spaces

Section LDS
Theorem DLDS Dependency in Linearly Dependent Sets
Theorem BS Basis of a Span

Section O
Theorem CRVA Conjugation Respects Vector Addition
Theorem CRSM Conjugation Respects Vector Scalar Multiplication
Theorem IPVA Inner Product and Vector Addition
Theorem IPSM Inner Product and Scalar Multiplication
Theorem IPAC Inner Product is Anti-Commutative
Theorem IPN Inner Products and Norms
Theorem PIP Positive Inner Products
Theorem OSLI Orthogonal Sets are Linearly Independent
Theorem GSP Gram-Schmidt Procedure

Section MO
Theorem VSPM Vector Space Properties of Matrices
Theorem SMS Symmetric Matrices are Square
Theorem TMA Transpose and Matrix Addition
Theorem TMSM Transpose and Matrix Scalar Multiplication
Theorem TT Transpose of a Transpose
Theorem CRMA Conjugation Respects Matrix Addition
Theorem CRMSM Conjugation Respects Matrix Scalar Multiplication
Theorem CCM Conjugate of the Conjugate of a Matrix
Theorem MCT Matrix Conjugation and Transposes
Theorem AMA Adjoint and Matrix Addition
Theorem AMSM Adjoint and Matrix Scalar Multiplication
Theorem AA Adjoint of an Adjoint

Section MM
Theorem SLEMM Systems of Linear Equations as Matrix Multiplication
Theorem EMMVP Equal Matrices and Matrix-Vector Products
Theorem EMP Entries of Matrix Products
Theorem MMZM Matrix Multiplication and the Zero Matrix
Theorem MMIM Matrix Multiplication and Identity Matrix
Theorem MMDAA Matrix Multiplication Distributes Across Addition
Theorem MMSMM Matrix Multiplication and Scalar Matrix Multiplication
Theorem MMA Matrix Multiplication is Associative
Theorem MMIP Matrix Multiplication and Inner Products
Theorem MMCC Matrix Multiplication and Complex Conjugation
Theorem MMT Matrix Multiplication and Transposes
Theorem MMAD Matrix Multiplication and Adjoints
Theorem AIP Adjoint and Inner Product
Theorem HMIP Hermitian Matrices and Inner Products

Section MISLE
Theorem TTMI Two-by-Two Matrix Inverse
Theorem CINM Computing the Inverse of a Nonsingular Matrix
Theorem MIU Matrix Inverse is Unique
Theorem SS Socks and Shoes
Theorem MIMI Matrix Inverse of a Matrix Inverse
Theorem MIT Matrix Inverse of a Transpose
Theorem MISM Matrix Inverse of a Scalar Multiple

Section MINM
Theorem NPNT Nonsingular Product has Nonsingular Terms
Theorem OSIS One-Sided Inverse is Sufficient
Theorem NI Nonsingularity is Invertibility
Theorem NME3 Nonsingular Matrix Equivalences, Round 3
Theorem SNCM Solution with Nonsingular Coefficient Matrix
Theorem UMI Unitary Matrices are Invertible
Theorem CUMOS Columns of Unitary Matrices are Orthonormal Sets
Theorem UMPIP Unitary Matrices Preserve Inner Products

Section CRS
Theorem CSCS Column Spaces and Consistent Systems
Theorem BCS Basis of the Column Space
Theorem CSNM Column Space of a Nonsingular Matrix
Theorem NME4 Nonsingular Matrix Equivalences, Round 4
Theorem REMRS Row-Equivalent Matrices have equal Row Spaces
Theorem BRS Basis for the Row Space
Theorem CSRST Column Space, Row Space, Transpose

Section FS
Theorem PEEF Properties of Extended Echelon Form
Theorem FS Four Subsets

Section VS
Theorem ZVU Zero Vector is Unique
Theorem AIU Additive Inverses are Unique
Theorem ZSSM Zero Scalar in Scalar Multiplication
Theorem ZVSM Zero Vector in Scalar Multiplication
Theorem AISM Additive Inverses from Scalar Multiplication
Theorem SMEZV Scalar Multiplication Equals the Zero Vector

Section S
Theorem TSS Testing Subsets for Subspaces
Theorem NSMS Null Space of a Matrix is a Subspace
Theorem SSS Span of a Set is a Subspace
Theorem CSMS Column Space of a Matrix is a Subspace
Theorem RSMS Row Space of a Matrix is a Subspace
Theorem LNSMS Left Null Space of a Matrix is a Subspace

Section LISS
Theorem VRRB Vector Representation Relative to a Basis

Section B
Theorem SUVB Standard Unit Vectors are a Basis
Theorem CNMB Columns of Nonsingular Matrix are a Basis
Theorem NME5 Nonsingular Matrix Equivalences, Round 5
Theorem COB Coordinates and Orthonormal Bases
Theorem UMCOB Unitary Matrices Convert Orthonormal Bases

Section D
Theorem SSLD Spanning Sets and Linear Dependence
Theorem BIS Bases have Identical Sizes
Theorem DCM Dimension of ℂ m
Theorem DP Dimension of P n
Theorem DM Dimension of M m n
Theorem CRN Computing Rank and Nullity
Theorem RPNC Rank Plus Nullity is Columns
Theorem RNNM Rank and Nullity of a Nonsingular Matrix
Theorem NME6 Nonsingular Matrix Equivalences, Round 6

Section PD
Theorem ELIS Extending Linearly Independent Sets
Theorem G Goldilocks
Theorem PSSD Proper Subspaces have Smaller Dimension
Theorem EDYES Equal Dimensions Yields Equal Subspaces
Theorem RMRT Rank of a Matrix is the Rank of the Transpose
Theorem DFS Dimensions of Four Subspaces
Theorem DSFB Direct Sum From a Basis
Theorem DSFOS Direct Sum From One Subspace
Theorem DSZV Direct Sums and Zero Vectors
Theorem DSZI Direct Sums and Zero Intersection
Theorem DSLI Direct Sums and Linear Independence
Theorem DSD Direct Sums and Dimension
Theorem RDS Repeated Direct Sums

Section DM
Theorem EMDRO Elementary Matrices Do Row Operations
Theorem EMN Elementary Matrices are Nonsingular
Theorem NMPEM Nonsingular Matrices are Products of Elementary Matrices
Theorem DMST Determinant of Matrices of Size Two
Theorem DER Determinant Expansion about Rows
Theorem DT Determinant of the Transpose
Theorem DEC Determinant Expansion about Columns

Section PDM
Theorem DZRC Determinant with Zero Row or Column
Theorem DRCS Determinant for Row or Column Swap
Theorem DRCM Determinant for Row or Column Multiples
Theorem DERC Determinant with Equal Rows or Columns
Theorem DRCMA Determinant for Row or Column Multiples and Addition
Theorem DIM Determinant of the Identity Matrix
Theorem DEM Determinants of Elementary Matrices
Theorem DEMMM Determinants, Elementary Matrices, Matrix Multiplication
Theorem SMZD Singular Matrices have Zero Determinants
Theorem NME7 Nonsingular Matrix Equivalences, Round 7
Theorem DRMM Determinant Respects Matrix Multiplication

Section EE
Theorem EMHE Every Matrix Has an Eigenvalue
Theorem EMRCP Eigenvalues of a Matrix are Roots of Characteristic Polynomials
Theorem EMS Eigenspace for a Matrix is a Subspace
Theorem EMNS Eigenspace of a Matrix is a Null Space

Section PEE
Theorem EDELI Eigenvectors with Distinct Eigenvalues are Linearly Independent
Theorem SMZE Singular Matrices have Zero Eigenvalues
Theorem NME8 Nonsingular Matrix Equivalences, Round 8
Theorem ESMM Eigenvalues of a Scalar Multiple of a Matrix
Theorem EOMP Eigenvalues Of Matrix Powers
Theorem EPM Eigenvalues of the Polynomial of a Matrix
Theorem EIM Eigenvalues of the Inverse of a Matrix
Theorem ETM Eigenvalues of the Transpose of a Matrix
Theorem ERMCP Eigenvalues of Real Matrices come in Conjugate Pairs
Theorem DCP Degree of the Characteristic Polynomial
Theorem NEM Number of Eigenvalues of a Matrix
Theorem ME Multiplicities of an Eigenvalue
Theorem MNEM Maximum Number of Eigenvalues of a Matrix
Theorem HMRE Hermitian Matrices have Real Eigenvalues
Theorem HMOE Hermitian Matrices have Orthogonal Eigenvectors

Section SD
Theorem SER Similarity is an Equivalence Relation
Theorem SMEE Similar Matrices have Equal Eigenvalues
Theorem DC Diagonalization Characterization
Theorem DMFE Diagonalizable Matrices have Full Eigenspaces
Theorem DED Distinct Eigenvalues implies Diagonalizable

Section LT
Theorem LTTZZ Linear Transformations Take Zero to Zero
Theorem MBLT Matrices Build Linear Transformations
Theorem MLTCV Matrix of a Linear Transformation, Column Vectors
Theorem LTLC Linear Transformations and Linear Combinations
Theorem LTDB Linear Transformation Defined on a Basis
Theorem SLTLT Sum of Linear Transformations is a Linear Transformation
Theorem MLTLT Multiple of a Linear Transformation is a Linear Transformation
Theorem VSLT Vector Space of Linear Transformations
Theorem CLTLT Composition of Linear Transformations is a Linear Transformation

Section ILT
Theorem KLTS Kernel of a Linear Transformation is a Subspace
Theorem KPI Kernel and Pre-Image
Theorem KILT Kernel of an Injective Linear Transformation
Theorem ILTLI Injective Linear Transformations and Linear Independence
Theorem ILTB Injective Linear Transformations and Bases
Theorem ILTD Injective Linear Transformations and Dimension
Theorem CILTI Composition of Injective Linear Transformations is Injective

Section SLT
Theorem RLTS Range of a Linear Transformation is a Subspace
Theorem RSLT Range of a Surjective Linear Transformation
Theorem SSRLT Spanning Set for Range of a Linear Transformation
Theorem RPI Range and Pre-Image
Theorem SLTB Surjective Linear Transformations and Bases
Theorem SLTD Surjective Linear Transformations and Dimension
Theorem CSLTS Composition of Surjective Linear Transformations is Surjective

Section IVLT
Theorem ILTLT Inverse of a Linear Transformation is a Linear Transformation
Theorem IILT Inverse of an Invertible Linear Transformation
Theorem ILTIS Invertible Linear Transformations are Injective and Surjective
Theorem CIVLT Composition of Invertible Linear Transformations
Theorem ICLT Inverse of a Composition of Linear Transformations
Theorem IVSED Isomorphic Vector Spaces have Equal Dimension
Theorem ROSLT Rank Of a Surjective Linear Transformation
Theorem NOILT Nullity Of an Injective Linear Transformation
Theorem RPNDD Rank Plus Nullity is Domain Dimension

Section VR
Theorem VRLT Vector Representation is a Linear Transformation
Theorem VRI Vector Representation is Injective
Theorem VRS Vector Representation is Surjective
Theorem VRILT Vector Representation is an Invertible Linear Transformation
Theorem CFDVS Characterization of Finite Dimensional Vector Spaces
Theorem IFDVS Isomorphism of Finite Dimensional Vector Spaces
Theorem CLI Coordinatization and Linear Independence
Theorem CSS Coordinatization and Spanning Sets

Section MR
Theorem FTMR Fundamental Theorem of Matrix Representation
Theorem MRSLT Matrix Representation of a Sum of Linear Transformations
Theorem MRMLT Matrix Representation of a Multiple of a Linear Transformation
Theorem MRCLT Matrix Representation of a Composition of Linear Transformations
Theorem KNSI Kernel and Null Space Isomorphism
Theorem RCSI Range and Column Space Isomorphism
Theorem IMR Invertible Matrix Representations
Theorem IMILT Invertible Matrices, Invertible Linear Transformation
Theorem NME9 Nonsingular Matrix Equivalences, Round 9

Section CB
Theorem CB Change-of-Basis
Theorem ICBM Inverse of Change-of-Basis Matrix
Theorem MRCB Matrix Representation and Change of Basis
Theorem SCB Similarity and Change of Basis
Theorem EER Eigenvalues, Eigenvectors, Representations

Section OD
Theorem PTMT Product of Triangular Matrices is Triangular
Theorem ITMT Inverse of a Triangular Matrix is Triangular
Theorem UTMR Upper Triangular Matrix Representation
Theorem OBUTR Orthonormal Basis for Upper Triangular Representation
Theorem OD Orthonormal Diagonalization
Theorem OBNM Orthonormal Bases and Normal Matrices

Section NLT
Theorem NJB Nilpotent Jordan Blocks
Theorem ENLT Eigenvalues of Nilpotent Linear Transformations
Theorem DNLT Diagonalizable Nilpotent Linear Transformations
Theorem KPLT Kernels of Powers of Linear Transformations
Theorem KPNLT Kernels of Powers of Nilpotent Linear Transformations
Theorem CFNLT Canonical Form for Nilpotent Linear Transformations

Section IS
Theorem EIS Eigenspaces are Invariant Subspaces
Theorem KPIS Kernels of Powers are Invariant Subspaces
Theorem GESIS Generalized Eigenspace is an Invariant Subspace
Theorem GEK Generalized Eigenspace as a Kernel
Theorem RGEN Restriction to Generalized Eigenspace is Nilpotent
Theorem MRRGE Matrix Representation of a Restriction to a Generalized Eigenspace

Section JCF
Theorem GESD Generalized Eigenspace Decomposition
Theorem DGES Dimension of Generalized Eigenspaces
Theorem JCFLT Jordan Canonical Form for a Linear Transformation
Theorem CHT Cayley-Hamilton Theorem

Section CNO
Theorem PCNA Properties of Complex Number Arithmetic
Theorem CCRA Complex Conjugation Respects Addition
Theorem CCRM Complex Conjugation Respects Multiplication
Theorem CCT Complex Conjugation Twice

Section SET

Section PT

Section F
Theorem FIMP Field of Integers Modulo a Prime

Section T
Theorem TL Trace is Linear
Theorem TSRM Trace is Symmetric with Respect to Multiplication
Theorem TIST Trace is Invariant Under Similarity Transformations
Theorem TSE Trace is the Sum of the Eigenvalues

Section HP
Theorem HPC Hadamard Product is Commutative
Theorem HPHID Hadamard Product with the Hadamard Identity
Theorem HPHI Hadamard Product with Hadamard Inverses
Theorem HPDAA Hadamard Product Distributes Across Addition
Theorem HPSMM Hadamard Product and Scalar Matrix Multiplication
Theorem DMHP Diagonalizable Matrices and the Hadamard Product
Theorem DMMP Diagonal Matrices and Matrix Products

Section VM
Theorem DVM Determinant of a Vandermonde Matrix
Theorem NVM Nonsingular Vandermonde Matrix

Section PSM
Theorem CPSM Creating Positive Semi-Definite Matrices
Theorem EPSM Eigenvalues of Positive Semi-definite Matrices

Section ROD
Theorem ROD Rank One Decomposition

Section TD
Theorem TD Triangular Decomposition
Theorem TDEE Triangular Decomposition, Entry by Entry

Section SVD
Theorem EEMAP Eigenvalues and Eigenvectors of Matrix-Adjoint Product
Theorem SVD Singular Value Decomposition

Section SR
Theorem PSMSR Positive Semi-Definite Matrices and Square Roots
Theorem EESR Eigenvalues and Eigenspaces of a Square Root
Theorem USR Unique Square Root

Section POD
Theorem PDM Polar Decomposition of a Matrix

Section CF
Theorem IP Interpolating Polynomial
Theorem LSMR Least Squares Minimizes Residuals

Section SAS

Previous topic: Dedication

Next topic: Notation

Chapter 2. Notation

Notation

M A   Matrix
MC A i j   Matrix Components
CV v   Column Vector
CVC v i   Column Vector Components
ZCV 0   Zero Column Vector
MRLS ℒ S A , b   Matrix Representation of a Linear System
AM A b   Augmented Matrix
RO R i ↔ R j , α R i , α R i + R j   Row Operations
RREFA r , D , F   Reduced Row-Echelon Form Analysis
NSM N A   Null Space of a Matrix
IM I m   Identity Matrix
VSCV ℂ m   Vector Space of Column Vectors
CVE u = v   Column Vector Equality
CVA u + v   Column Vector Addition
CVSM α u   Column Vector Scalar Multiplication
SSV S   Span of a Set of Vectors
CCCV u   Complex Conjugate of a Column Vector
IP u , v   Inner Product
NV v   Norm of a Vector
SUV e i   Standard Unit Vectors
VSM M m n   Vector Space of Matrices
ME A = B   Matrix Equality
MA A + B   Matrix Addition
MSM α A   Matrix Scalar Multiplication
ZM O   Zero Matrix
TM A t   Transpose of a Matrix
CCM A   Complex Conjugate of a Matrix
A A ∗   Adjoint
MVP A u   Matrix-Vector Product
MI A − 1   Matrix Inverse
CSM C A   Column Space of a Matrix
RSM ℛ A   Row Space of a Matrix
LNS ℒ A   Left Null Space
D dim V   Dimension
NOM n A   Nullity of a Matrix
ROM r A   Rank of a Matrix
DS V = U ⊕ W   Direct Sum
ELEM E i , j , E i α , E i , j α   Elementary Matrix
SM A i | j   SubMatrix
DM A , det A   Determinant of a Matrix
AME α A λ   Algebraic Multiplicity of an Eigenvalue
GME γ A λ   Geometric Multiplicity of an Eigenvalue
LT T : U → V   Linear Transformation
KLT K T   Kernel of a Linear Transformation
RLT ℛ T   Range of a Linear Transformation
ROLT r T   Rank of a Linear Transformation
NOLT n T   Nullity of a Linear Transformation
VR ρ B w   Vector Representation
MR M B , C T   Matrix Representation
JB J n λ   Jordan Block
GES G T λ   Generalized Eigenspace
LTR T | U   Linear Transformation Restriction
IE ι T λ   Index of an Eigenvalue
CNE α = β   Complex Number Equality
CNA α + β   Complex Number Addition
CNM α β   Complex Number Multiplication
CCN α   Conjugate of a Complex Number
SETM x ∈ S   Set Membership
SSET S ⊆ T   Subset
ES ∅   Empty Set
SE S = T   Set Equality
C S   Cardinality
SU S ∪ T   Set Union
SI S ∩ T   Set Intersection
SC S   Set Complement
T t A   Trace
HP A ∘ B   Hadamard Product
HID J m n   Hadamard Identity
HI A ̂   Hadamard Inverse
SRM A 1 ∕ 2   Square Root of a Matrix

Previous topic: Theorems

Next topic: Diagrams

Chapter 3. Diagrams

Diagrams

DTSLS  Decision Tree for Solving Linear Systems
CSRST  Column Space and Row Space Techniques
DLTA  Definition of Linear Transformation, Additive
DLTM  Definition of Linear Transformation, Multiplicative
GLT  General Linear Transformation
NILT  Non-Injective Linear Transformation
ILT  Injective Linear Transformation
FTMR  Fundamental Theorem of Matrix Representations
FTMRA  Fundamental Theorem of Matrix Representations (Alternate)
MRCLT  Matrix Representation and Composition of Linear Transformations

Previous topic: Notation

Next topic: Examples

Chapter 4. Examples

Examples

Section WILA
Example TMP Trail Mix Packaging

Section SSLE
Example STNE Solving two (nonlinear) equations
Example NSE Notation for a system of equations
Example TTS Three typical systems
Example US Three equations, one solution
Example IS Three equations, infinitely many solutions

Section RREF
Example AM A matrix
Example NSLE Notation for systems of linear equations
Example AMAA Augmented matrix for Archetype A
Example TREM Two row-equivalent matrices
Example USR Three equations, one solution, reprised
Example RREF A matrix in reduced row-echelon form
Example NRREF A matrix not in reduced row-echelon form
Example SAB Solutions for Archetype B
Example SAA Solutions for Archetype A
Example SAE Solutions for Archetype E

Section TSS
Example RREFN Reduced row-echelon form notation
Example ISSI Describing infinite solution sets, Archetype I
Example FDV Free and dependent variables
Example CFV Counting free variables
Example OSGMD One solution gives many, Archetype D

Section HSE
Example AHSAC Archetype C as a homogeneous system
Example HUSAB Homogeneous, unique solution, Archetype B
Example HISAA Homogeneous, infinite solutions, Archetype A
Example HISAD Homogeneous, infinite solutions, Archetype D
Example NSEAI Null space elements of Archetype I
Example CNS1 Computing a null space, #1
Example CNS2 Computing a null space, #2

Section NM
Example S A singular matrix, Archetype A
Example NM A nonsingular matrix, Archetype B
Example IM An identity matrix
Example SRR Singular matrix, row-reduced
Example NSR Nonsingular matrix, row-reduced
Example NSS Null space of a singular matrix
Example NSNM Null space of a nonsingular matrix

Section VO
Example VESE Vector equality for a system of equations
Example VA Addition of two vectors in ℂ 4
Example CVSM Scalar multiplication in ℂ 5

Section LC
Example TLC Two linear combinations in ℂ 6
Example ABLC Archetype B as a linear combination
Example AALC Archetype A as a linear combination
Example VFSAD Vector form of solutions for Archetype D
Example VFS Vector form of solutions
Example VFSAI Vector form of solutions for Archetype I
Example VFSAL Vector form of solutions for Archetype L
Example PSHS Particular solutions, homogeneous solutions, Archetype D

Section SS
Example ABS A basic span
Example SCAA Span of the columns of Archetype A
Example SCAB Span of the columns of Archetype B
Example SSNS Spanning set of a null space
Example NSDS Null space directly as a span
Example SCAD Span of the columns of Archetype D

Section LI
Example LDS Linearly dependent set in ℂ 5
Example LIS Linearly independent set in ℂ 5
Example LIHS Linearly independent, homogeneous system
Example LDHS Linearly dependent, homogeneous system
Example LDRN Linearly dependent, r < n
Example LLDS Large linearly dependent set in ℂ 4
Example LDCAA Linearly dependent columns in Archetype A
Example LICAB Linearly independent columns in Archetype B
Example LINSB Linear independence of null space basis
Example NSLIL Null space spanned by linearly independent set, Archetype L

Section LDS
Example RSC5 Reducing a span in ℂ 5
Example COV Casting out vectors
Example RSC4 Reducing a span in ℂ 4
Example RES Reworking elements of a span

Section O
Example CSIP Computing some inner products
Example CNSV Computing the norm of some vectors
Example TOV Two orthogonal vectors
Example SUVOS Standard Unit Vectors are an Orthogonal Set
Example AOS An orthogonal set
Example GSTV Gram-Schmidt of three vectors
Example ONTV Orthonormal set, three vectors
Example ONFV Orthonormal set, four vectors

Section MO
Example MA Addition of two matrices in M 2 3
Example MSM Scalar multiplication in M 3 2
Example TM Transpose of a 3 × 4 matrix
Example SYM A symmetric 5 × 5 matrix
Example CCM Complex conjugate of a matrix

Section MM
Example MTV A matrix times a vector
Example MNSLE Matrix notation for systems of linear equations
Example MBC Money’s best cities
Example PTM Product of two matrices
Example MMNC Matrix multiplication is not commutative
Example PTMEE Product of two matrices, entry-by-entry

Section MISLE
Example SABMI Solutions to Archetype B with a matrix inverse
Example MWIAA A matrix without an inverse, Archetype A
Example MI Matrix inverse
Example CMI Computing a matrix inverse
Example CMIAB Computing a matrix inverse, Archetype B

Section MINM
Example UM3 Unitary matrix of size 3
Example UPM Unitary permutation matrix
Example OSMC Orthonormal set from matrix columns

Section CRS
Example CSMCS Column space of a matrix and consistent systems
Example MCSM Membership in the column space of a matrix
Example CSTW Column space, two ways
Example CSOCD Column space, original columns, Archetype D
Example CSAA Column space of Archetype A
Example CSAB Column space of Archetype B
Example RSAI Row space of Archetype I
Example RSREM Row spaces of two row-equivalent matrices
Example IAS Improving a span
Example CSROI Column space from row operations, Archetype I

Section FS
Example LNS Left null space
Example CSANS Column space as null space
Example SEEF Submatrices of extended echelon form
Example FS1 Four subsets, #1
Example FS2 Four subsets, #2
Example FSAG Four subsets, Archetype G

Section VS
Example VSCV The vector space ℂ m
Example VSM The vector space of matrices, M m n
Example VSP The vector space of polynomials, P n
Example VSIS The vector space of infinite sequences
Example VSF The vector space of functions
Example VSS The singleton vector space
Example CVS The crazy vector space
Example PCVS Properties for the Crazy Vector Space

Section S
Example SC3 A subspace of ℂ 3
Example SP4 A subspace of P 4
Example NSC2Z A non-subspace in ℂ 2 , zero vector
Example NSC2A A non-subspace in ℂ 2 , additive closure
Example NSC2S A non-subspace in ℂ 2 , scalar multiplication closure
Example RSNS Recasting a subspace as a null space
Example LCM A linear combination of matrices
Example SSP Span of a set of polynomials
Example SM32 A subspace of M 3 2

Section LISS
Example LIP4 Linear independence in P 4
Example LIM32 Linear independence in M 3 2
Example LIC Linearly independent set in the crazy vector space
Example SSP4 Spanning set in P 4
Example SSM22 Spanning set in M 2 2
Example SSC Spanning set in the crazy vector space
Example AVR A vector representation

Section B
Example BP Bases for P n
Example BM A basis for the vector space of matrices
Example BSP4 A basis for a subspace of P 4
Example BSM22 A basis for a subspace of M 2 2
Example BC Basis for the crazy vector space
Example RSB Row space basis
Example RS Reducing a span
Example CABAK Columns as Basis, Archetype K
Example CROB4 Coordinatization relative to an orthonormal basis, ℂ 4
Example CROB3 Coordinatization relative to an orthonormal basis, ℂ 3

Section D
Example LDP4 Linearly dependent set in P 4
Example DSM22 Dimension of a subspace of M 2 2
Example DSP4 Dimension of a subspace of P 4
Example DC Dimension of the crazy vector space
Example VSPUD Vector space of polynomials with unbounded degree
Example RNM Rank and nullity of a matrix
Example RNSM Rank and nullity of a square matrix

Section PD
Example BPR Bases for P n , reprised
Example BDM22 Basis by dimension in M 2 2
Example SVP4 Sets of vectors in P 4
Example RRTI Rank, rank of transpose, Archetype I
Example SDS Simple direct sum

Section DM
Example EMRO Elementary matrices and row operations
Example SS Some submatrices
Example D33M Determinant of a 3 × 3 matrix
Example TCSD Two computations, same determinant
Example DUTM Determinant of an upper triangular matrix

Section PDM
Example DRO Determinant by row operations
Example ZNDAB Zero and nonzero determinant, Archetypes A and B

Section EE
Example SEE Some eigenvalues and eigenvectors
Example PM Polynomial of a matrix
Example CAEHW Computing an eigenvalue the hard way
Example CPMS3 Characteristic polynomial of a matrix, size 3
Example EMS3 Eigenvalues of a matrix, size 3
Example ESMS3 Eigenspaces of a matrix, size 3
Example EMMS4 Eigenvalue multiplicities, matrix of size 4
Example ESMS4 Eigenvalues, symmetric matrix of size 4
Example HMEM5 High multiplicity eigenvalues, matrix of size 5
Example CEMS6 Complex eigenvalues, matrix of size 6
Example DEMS5 Distinct eigenvalues, matrix of size 5

Section PEE
Example BDE Building desired eigenvalues

Section SD
Example SMS5 Similar matrices of size 5
Example SMS3 Similar matrices of size 3
Example EENS Equal eigenvalues, not similar
Example DAB Diagonalization of Archetype B
Example DMS3 Diagonalizing a matrix of size 3
Example NDMS4 A non-diagonalizable matrix of size 4
Example DEHD Distinct eigenvalues, hence diagonalizable
Example HPDM High power of a diagonalizable matrix
Example FSCF Fibonacci sequence, closed form

Section LT
Example ALT A linear transformation
Example NLT Not a linear transformation
Example LTPM Linear transformation, polynomials to matrices
Example LTPP Linear transformation, polynomials to polynomials
Example LTM Linear transformation from a matrix
Example MFLT Matrix from a linear transformation
Example MOLT Matrix of a linear transformation
Example LTDB1 Linear transformation defined on a basis
Example LTDB2 Linear transformation defined on a basis
Example LTDB3 Linear transformation defined on a basis
Example SPIAS Sample pre-images, Archetype S
Example STLT Sum of two linear transformations
Example SMLT Scalar multiple of a linear transformation
Example CTLT Composition of two linear transformations

Section ILT
Example NIAQ Not injective, Archetype Q
Example IAR Injective, Archetype R
Example IAV Injective, Archetype V
Example NKAO Nontrivial kernel, Archetype O
Example TKAP Trivial kernel, Archetype P
Example NIAQR Not injective, Archetype Q, revisited
Example NIAO Not injective, Archetype O
Example IAP Injective, Archetype P
Example NIDAU Not injective by dimension, Archetype U

Section SLT
Example NSAQ Not surjective, Archetype Q
Example SAR Surjective, Archetype R
Example SAV Surjective, Archetype V
Example RAO Range, Archetype O
Example FRAN Full range, Archetype N
Example NSAQR Not surjective, Archetype Q, revisited
Example NSAO Not surjective, Archetype O
Example SAN Surjective, Archetype N
Example BRLT A basis for the range of a linear transformation
Example NSDAT Not surjective by dimension, Archetype T

Section IVLT
Example AIVLT An invertible linear transformation
Example ANILT A non-invertible linear transformation
Example CIVLT Computing the Inverse of a Linear Transformations
Example IVSAV Isomorphic vector spaces, Archetype V

Section VR
Example VRC4 Vector representation in ℂ 4
Example VRP2 Vector representations in P 2
Example TIVS Two isomorphic vector spaces
Example CVSR Crazy vector space revealed
Example ASC A subspace characterized
Example MIVS Multiple isomorphic vector spaces
Example CP2 Coordinatizing in P 2
Example CM32 Coordinatization in M 3 2

Section MR
Example OLTTR One linear transformation, three representations
Example ALTMM A linear transformation as matrix multiplication
Example MPMR Matrix product of matrix representations
Example KVMR Kernel via matrix representation
Example RVMR Range via matrix representation
Example ILTVR Inverse of a linear transformation via a representation

Section CB
Example ELTBM Eigenvectors of linear transformation between matrices
Example ELTBP Eigenvectors of linear transformation between polynomials
Example CBP Change of basis with polynomials
Example CBCV Change of basis with column vectors
Example MRCM Matrix representations and change-of-basis matrices
Example MRBE Matrix representation with basis of eigenvectors
Example ELTT Eigenvectors of a linear transformation, twice
Example CELT Complex eigenvectors of a linear transformation

Section OD
Example ANM A normal matrix

Section NLT
Example NM64 Nilpotent matrix, size 6, index 4
Example NM62 Nilpotent matrix, size 6, index 2
Example JB4 Jordan block, size 4
Example NJB5 Nilpotent Jordan block, size 5
Example NM83 Nilpotent matrix, size 8, index 3
Example KPNLT Kernels of powers of a nilpotent linear transformation
Example CFNLT Canonical form for a nilpotent linear transformation

Section IS
Example TIS Two invariant subspaces
Example EIS Eigenspaces as invariant subspaces
Example ISJB Invariant subspaces and Jordan blocks
Example GE4 Generalized eigenspaces, dimension 4 domain
Example GE6 Generalized eigenspaces, dimension 6 domain
Example LTRGE Linear transformation restriction on generalized eigenspace
Example ISMR4 Invariant subspaces, matrix representation, dimension 4 domain
Example ISMR6 Invariant subspaces, matrix representation, dimension 6 domain
Example GENR6 Generalized eigenspaces and nilpotent restrictions, dimension 6 domain

Section JCF
Example JCF10 Jordan canonical form, size 10

Section CNO
Example ACN Arithmetic of complex numbers
Example CSCN Conjugate of some complex numbers
Example MSCN Modulus of some complex numbers

Section SET
Example SETM Set membership
Example SSET Subset
Example CS Cardinality and Size
Example SU Set union
Example SI Set intersection
Example SC Set complement

Section PT

Section F
Example IM11 Integers mod 11
Example VSIM5 Vector space over integers mod 5
Example SM2Z7 Symmetric matrices of size 2 over ℤ 7
Example FF8 Finite field of size 8

Section T

Section HP
Example HP Hadamard Product

Section VM
Example VM4 Vandermonde matrix of size 4

Section PSM

Section ROD
Example ROD2 Rank one decomposition, size 2
Example ROD4 Rank one decomposition, size 4

Section TD
Example TD4 Triangular decomposition, size 4
Example TDSSE Triangular decomposition solves a system of equations
Example TDEE6 Triangular decomposition, entry by entry, size 6

Section SVD

Section SR

Section POD

Section CF
Example PTFP Polynomial through five points

Section SAS
Example SS6W Sharing a secret 6 ways

Previous topic: Diagrams

Next topic: Preface

Preface

This textbook is designed to teach the university mathematics student the basics of linear algebra and the techniques of formal mathematics. There are no prerequisites other than ordinary algebra, but it is probably best used by a student who has the “mathematical maturity” of a sophomore or junior. The text has two goals: to teach the fundamental concepts and techniques of matrix algebra and abstract vector spaces, and to teach the techniques associated with understanding the definitions and theorems forming a coherent area of mathematics. So there is an emphasis on worked examples of nontrivial size and on proving theorems carefully.

This book is copyrighted. This means that governments have granted the author a monopoly — the exclusive right to control the making of copies and derivative works for many years (too many years in some cases). It also gives others limited rights, generally referred to as “fair use,” such as the right to quote sections in a review without seeking permission. However, the author licenses this book to anyone under the terms of the GNU Free Documentation License (GFDL), which gives you more rights than most copyrights (see Appendix GFDL). Loosely speaking, you may make as many copies as you like at no cost, and you may distribute these unmodified copies if you please. You may modify the book for your own use. The catch is that if you make modifications and you distribute the modified version, or make use of portions in excess of fair use in another work, then you must also license the new work with the GFDL. So the book has lots of inherent freedom, and no one is allowed to distribute a derivative work that restricts these freedoms. (See the license itself in the appendix for the exact details of the additional rights you have been given.)

Notice that initially most people are struck by the notion that this book is free (the French would say gratuit, at no cost). And it is. However, it is more important that the book has freedom (the French would say libert é, liberty). It will never go “out of print” nor will there ever be trivial updates designed only to frustrate the used book market. Those considering teaching a course with this book can examine it thoroughly in advance. Adding new exercises or new sections has been purposely made very easy, and the hope is that others will contribute these modifications back for incorporation into the book, for the benefit of all.

Depending on how you received your copy, you may want to check for the latest version (and other news) at http://linear.ups.edu/ .

Topics The first half of this text (through Chapter M) is basically a course in matrix algebra, though the foundation of some more advanced ideas is also being formed in these early sections. Vectors are presented exclusively as column vectors (since we also have the typographic freedom to avoid writing a column vector inline as the transpose of a row vector), and linear combinations are presented very early. Spans, null spaces, column spaces and row spaces are also presented early, simply as sets, saving most of their vector space properties for later, so they are familiar objects before being scrutinized carefully.

You cannot do everything early, so in particular matrix multiplication comes later than usual. However, with a definition built on linear combinations of column vectors, it should seem more natural than the more frequent definition using dot products of rows with columns. And this delay emphasizes that linear algebra is built upon vector addition and scalar multiplication. Of course, matrix inverses must wait for matrix multiplication, but this does not prevent nonsingular matrices from occurring sooner. Vector space properties are hinted at when vector and matrix operations are first defined, but the notion of a vector space is saved for a more axiomatic treatment later ( Chapter VS). Once bases and dimension have been explored in the context of vector spaces, linear transformations and their matrix representation follow. The goal of the book is to go as far as Jordan canonical form in the Core ( Part C), with less central topics collected in the Topics ( Part T). A third part contains contributed applications ( Part A), with notation and theorems integrated with the earlier two parts.

Linear algebra is an ideal subject for the novice mathematics student to learn how to develop a topic precisely, with all the rigor mathematics requires. Unfortunately, much of this rigor seems to have escaped the standard calculus curriculum, so for many university students this is their first exposure to careful definitions and theorems, and the expectation that they fully understand them, to say nothing of the expectation that they become proficient in formulating their own proofs. We have tried to make this text as helpful as possible with this transition. Every definition is stated carefully, set apart from the text. Likewise, every theorem is carefully stated, and almost every one has a complete proof. Theorems usually have just one conclusion, so they can be referenced precisely later. Definitions and theorems are cataloged in order of their appearance in the front of the book ( Definitions, Theorems), and alphabetical order in the index at the back. Along the way, there are discussions of some more important ideas relating to formulating proofs ( Proof Techniques), which is part advice and part logic.

Origin and History This book is the result of the confluence of several related events and trends.

However, much of my motivation for writing this book is captured by the sentiments expressed by H.M. Cundy and A.P. Rollet in their Preface to the First Edition of Mathematical Models (1952), especially the final sentence,

How To Use This Book Chapters, Theorems, etc. are not numbered in this book, but are instead referenced by acronyms. This means that Theorem XYZ will always be Theorem XYZ, no matter if new sections are added, or if an individual decides to remove certain other sections. Within sections, the subsections are acronyms that begin with the acronym of the section. So Subsection XYZ.AB is the subsection AB in Section XYZ. Acronyms are unique within their type, so for example there is just one Definition B, but there is also a Section B. At first, all the letters flying around may be confusing, but with time, you will begin to recognize the more important ones on sight. Furthermore, there are lists of theorems, examples, etc. in the front of the book, and an index that contains every acronym. If you are reading this in an electronic version (PDF or XML), you will see that all of the cross-references are hyperlinks, allowing you to click to a definition or example, and then use the back button to return. In printed versions, you must rely on the page numbers. However, note that page numbers are not permanent! Different editions, different margins, or different sized paper will affect what content is on each page. And in time, the addition of new material will affect the page numbering.

Chapter divisions are not critical to the organization of the book, as Sections are the main organizational unit. Sections are designed to be the subject of a single lecture or classroom session, though there is frequently more material than can be discussed and illustrated in a fifty-minute session. Consequently, the instructor will need to be selective about which topics to illustrate with other examples and which topics to leave to the student’s reading. Many of the examples are meant to be large, such as using five or six variables in a system of equations, so the instructor may just want to “walk” a class through these examples. The book has been written with the idea that some may work through it independently, so the hope is that students can learn some of the more mechanical ideas on their own.

The highest level division of the book is the three Parts: Core, Topics, Applications ( Part C, Part T, Part A). The Core is meant to carefully describe the basic ideas required of a first exposure to linear algebra. In the final sections of the Core, one should ask the question: which previous Sections could be removed without destroying the logical development of the subject? Hopefully, the answer is “none.” The goal of the book is to finish the Core with a very general representation of a linear transformation (Jordan canonical form, Section JCF). Of course, there will not be universal agreement on what should, or should not, constitute the Core, but the main idea is to limit it to about forty sections. Topics ( Part T) is meant to contain those subjects that are important in linear algebra, and which would make profitable detours from the Core for those interested in pursuing them. Applications ( Part A) should illustrate the power and widespread applicability of linear algebra to as many fields as possible. The Archetypes ( Appendix A) cover many of the computational aspects of systems of linear equations, matrices and linear transformations. The student should consult them often, and this is encouraged by exercises that simply suggest the right properties to examine at the right time. But what is more important, this a repository that contains enough variety to provide abundant examples of key theorems, while also providing counterexamples to hypotheses or converses of theorems. The summary table at the start of this appendix should be especially useful.

I require my students to read each Section prior to the day’s discussion on that section. For some students this is a novel idea, but at the end of the semester a few always report on the benefits, both for this course and other courses where they have adopted the habit. To make good on this requirement, each section contains three Reading Questions. These sometimes only require parroting back a key definition or theorem, or they require performing a small example of a key computation, or they ask for musings on key ideas or new relationships between old ideas. Answers are emailed to me the evening before the lecture. Given the flavor and purpose of these questions, including solutions seems foolish.

Every chapter of Part C ends with “Annotated Acronyms”, a short list of critical theorems or definitions from that chapter. There are a variety of reasons for any one of these to have been chosen, and reading the short paragraphs after some of these might provide insight into the possibilities. An end-of-chapter review might usefully incorporate a close reading of these lists.

Formulating interesting and effective exercises is as difficult, or more so, than building a narrative. But it is the place where a student really learns the material. As such, for the student’s benefit, complete solutions should be given. As the list of exercises expands, the amount with solutions should similarly expand. Exercises and their solutions are referenced with a section name, followed by a dot, then a letter (C,M, or T) and a number. The letter ‘C’ indicates a problem that is mostly computational in nature, while the letter ‘T’ indicates a problem that is more theoretical in nature. A problem with a letter ‘M’ is somewhere in between (middle, mid-level, median, middling), probably a mix of computation and applications of theorems. So Solution MO.T13 is a solution to an exercise in Section MO that is theoretical in nature. The number ‘13’ has no intrinsic meaning.

More on Freedom This book is freely-distributable under the terms of the GFDL, along with the underlying T EX  code from which the book is built. This arrangement provides many benefits unavailable with traditional texts.

Conclusion Foremost, I hope that students find their time spent with this book profitable. I hope that instructors find it flexible enough to fit the needs of their course. And I hope that everyone will send me their comments and suggestions, and also consider the myriad ways they can help (as listed on the book’s website at http://linear.ups.edu ).

Previous topic: Examples

Next topic: Acknowledgements

Acknowledgements

Many people have helped to make this book, and its freedoms, possible.

First, the time to create, edit and distribute the book has been provided implicitly and explicitly by the University of Puget Sound. A sabbatical leave Spring 2004 and a course release in Spring 2007 are two obvious examples of explicit support. The latter was provided by support from the Lind-VanEnkevort Fund. The university has also provided clerical support, computer hardware, network servers and bandwidth. Thanks to Dean Kris Bartanen and the chair of the Mathematics and Computer Science Department, Professor Martin Jackson, for their support, encouragement and flexibility.

My colleagues in the Mathematics and Computer Science Department have graciously taught our introductory linear algebra course using preliminary versions and have provided valuable suggestions that have improved the book immeasurably. Thanks to Professor Martin Jackson (v0.30), Professor David Scott (v0.70) and Professor Bryan Smith (v0.70, 0.80, v1.00).

University of Puget Sound librarians Lori Ricigliano, Elizabeth Knight and Jeanne Kimura provided valuable advice on production, and interesting conversations about copyrights.

Many aspects of the book have been influenced by insightful questions and creative suggestions from the students who have labored through the book in our courses. For example, the flashcards with theorems and definitions are a direct result of a student suggestion. I will single out a handful of students have been especially adept at finding and reporting mathematically significant typographical errors: Jake Linenthal, Christie Su, Kim Le, Sarah McQuate, Andy Zimmer, Travis Osborne, Andrew Tapay, Mark Shoemaker, Tasha Underhill, Tim Zitzer, Elizabeth Million, and Steve Canfield.

I have tried to be as original as possible in the organization and presentation of this beautiful subject. However, I have been influenced by many years of teaching from another excellent textbook, Introduction to Linear Algebra by L.W. Johnson, R.D. Reiss and J.T. Arnold. When I have needed inspiration for the correct approach to particularly important proofs, I have learned to eventually consult two other textbooks. Sheldon Axler’s Linear Algebra Done Right is a highly original exposition, while Ben Noble’s Applied Linear Algebra frequently strikes just the right note between rigor and intuition. Noble’s excellent book is highly recommended, even though its publication dates to 1969.

Conversion to various electronic formats have greatly depended on assistance from: Eitan Gurari, author of the powerful L A T EX  translator, tex4ht; Davide Cervone, author of jsMath; and Carl Witty, who advised and tested the Sony Reader format. Thanks to these individuals for their critical assistance.

General support and encouragement of free and affordable textbooks, in addition to specific promotion of this text, was provided by Nicole Allen, Textbook Advocate at Student Public Interest Research Groups. Nicole was an early consumer of this material, back when it looked more like lecture notes than a textbook.

Finally, in every possible case, the production and distribution of this book has been accomplished with open-source software. The range of individuals and projects is far too great to pretend to list them all. The book’s web site will someday maintain pointers to as many of these projects as possible.

Previous topic: Preface

Next topic: Part C  Core

Part I. Part C  Core

Chapter SLE  Systems of Linear Equations

Chapter V  Vectors

Chapter M  Matrices

Chapter VS  Vector Spaces

Chapter D  Determinants

Chapter E  Eigenvalues

Chapter LT  Linear Transformations

Chapter R  Representations

Appendix CN  Computation Notes

Appendix P  Preliminaries

Appendix A  Archetypes

Appendix GFDL  GNU Free Documentation License

Previous topic: Acknowledgements

Next topic: Chapter SLE  Systems of Linear Equations

Chapter 1. Chapter SLE  Systems of Linear Equations

We will motivate our study of linear algebra by studying solutions to systems of linear equations. While the focus of this chapter is on the practical matter of how to find, and describe, these solutions, we will also be setting ourselves up for more theoretical ideas that will appear later.

Section WILA  What is Linear Algebra?

Section SSLE  Solving Systems of Linear Equations

Section RREF  Reduced Row-Echelon Form

Section HSE  Homogeneous Systems of Equations

Section NM  Nonsingular Matrices

Parent topic: Part C  Core

Previous topic: Part C  Core

Next topic: Section WILA  What is Linear Algebra?

Section WILA  What is Linear Algebra?

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Parent topic: Chapter SLE  Systems of Linear Equations

Previous topic: Chapter SLE  Systems of Linear Equations

Next topic: Section SSLE  Solving Systems of Linear Equations

Subsection LA: “Linear” + “Algebra”

The subject of linear algebra can be partially explained by the meaning of the two terms comprising the title. “Linear” is a term you will appreciate better at the end of this course, and indeed, attaining this appreciation could be taken as one of the primary goals of this course. However for now, you can understand it to mean anything that is “straight” or “flat.” For example in the x y -plane you might be accustomed to describing straight lines (is there any other kind?) as the set of solutions to an equation of the form y = m x + b , where the slope m and the y -intercept b are constants that together describe the line. In multivariate calculus, you may have discussed planes. Living in three dimensions, with coordinates described by triples ( x , y , z ) , they can be described as the set of solutions to equations of the form a x + b y + c z = d , where a , b , c , d are constants that together determine the plane. While we might describe planes as “flat,” lines in three dimensions might be described as “straight.” From a multivariate calculus course you will recall that lines are sets of points described by equations such as x = 3 t − 4 , y = − 7 t + 2 , z = 9 t , where t is a parameter that can take on any value.

Another view of this notion of “flatness” is to recognize that the sets of points just described are solutions to equations of a relatively simple form. These equations involve addition and multiplication only. We will have a need for subtraction, and occasionally we will divide, but mostly you can describe “linear” equations as involving only addition and multiplication. Here are some examples of typical equations we will see in the next few sections:

2 x + 3 y − 4 z = 1 3 4 x 1 + 5 x 2 − x 3 + x 4 + x 5 = 0 9 a − 2 b + 7 c + 2 d = − 7

What we will not see are equations like:

x y + 5 y z = 1 3 x 1 + x 2 3 ∕ x 4 − x 3 x 4 x 5 2 = 0 tan ( a b ) + log ( c − d ) = − 7

The exception will be that we will on occasion need to take a square root.

You have probably heard the word “algebra” frequently in your mathematical preparation for this course. Most likely, you have spent a good ten to fifteen years learning the algebra of the real numbers, along with some introduction to the very similar algebra of complex numbers (see Section CNO). However, there are many new algebras to learn and use, and likely linear algebra will be your second algebra. Like learning a second language, the necessary adjustments can be challenging at times, but the rewards are many. And it will make learning your third and fourth algebras even easier. Perhaps you have heard of “groups” and “rings” (or maybe you have studied them already), which are excellent examples of other algebras with very interesting properties and applications. In any event, prepare yourself to learn a new algebra and realize that some of the old rules you used for the real numbers may no longer apply to this new algebra you will be learning!

The brief discussion above about lines and planes suggests that linear algebra has an inherently geometric nature, and this is true. Examples in two and three dimensions can be used to provide valuable insight into important concepts of this course. However, much of the power of linear algebra will be the ability to work with “flat” or “straight” objects in higher dimensions, without concerning ourselves with visualizing the situation. While much of our intuition will come from examples in two and three dimensions, we will maintain an algebraic approach to the subject, with the geometry being secondary. Others may wish to switch this emphasis around, and that can lead to a very fruitful and beneficial course, but here and now we are laying our bias bare.

Subsection AA: An Application

We conclude this section with a rather involved example that will highlight some of the power and techniques of linear algebra. Work through all of the details with pencil and paper, until you believe all the assertions made. However, in this introductory example, do not concern yourself with how some of the results are obtained or how you might be expected to solve a similar problem. We will come back to this example later and expose some of the techniques used and properties exploited. For now, use your background in mathematics to convince yourself that everything said here really is correct.

Example  TMP
Trail Mix Packaging
Suppose you are the production manager at a food-packaging plant and one of your product lines is trail mix, a healthy snack popular with hikers and backpackers, containing raisins, peanuts and hard-shelled chocolate pieces. By adjusting the mix of these three ingredients, you are able to sell three varieties of this item. The fancy version is sold in half-kilogram packages at outdoor supply stores and has more chocolate and fewer raisins, thus commanding a higher price. The standard version is sold in one kilogram packages in grocery stores and gas station mini-markets. Since the standard version has roughly equal amounts of each ingredient, it is not as expensive as the fancy version. Finally, a bulk version is sold in bins at grocery stores for consumers to load into plastic bags in amounts of their choosing. To appeal to the shoppers that like bulk items for their economy and healthfulness, this mix has many more raisins (at the expense of chocolate) and therefore sells for less.

Your production facilities have limited storage space and early each morning you are able to receive and store 380 kilograms of raisins, 500 kilograms of peanuts and 620 kilograms of chocolate pieces. As production manager, one of your most important duties is to decide how much of each version of trail mix to make every day. Clearly, you can have up to 1500 kilograms of raw ingredients available each day, so to be the most productive you will likely produce 1500 kilograms of trail mix each day. Also, you would prefer not to have any ingredients leftover each day, so that your final product is as fresh as possible and so that you can receive the maximum delivery the next morning. But how should these ingredients be allocated to the mixing of the bulk, standard and fancy versions?

First, we need a little more information about the mixes. Workers mix the ingredients in 15 kilogram batches, and each row of the table below gives a recipe for a 15 kilogram batch. There is some additional information on the costs of the ingredients and the price the manufacturer can charge for the different versions of the trail mix.

As production manager, it is important to realize that you only have three decisions to make — the amount of bulk mix to make, the amount of standard mix to make and the amount of fancy mix to make. Everything else is beyond your control or is handled by another department within the company. Principally, you are also limited by the amount of raw ingredients you can store each day. Let us denote the amount of each mix to produce each day, measured in kilograms, by the variable quantities b , s and f . Your production schedule can be described as values of b , s and f that do several things. First, we cannot make negative quantities of each mix, so

b ≥ 0 s ≥ 0 f ≥ 0

Second, if we want to consume all of our ingredients each day, the storage capacities lead to three (linear) equations, one for each ingredient,

7 1 5 b + 6 1 5 s + 2 1 5 f = 3 8 0  (raisins) 6 1 5 b + 4 1 5 s + 5 1 5 f = 5 0 0  (peanuts) 2 1 5 b + 5 1 5 s + 8 1 5 f = 6 2 0  (chocolate)

It happens that this system of three equations has just one solution. In other words, as production manager, your job is easy, since there is but one way to use up all of your raw ingredients making trail mix. This single solution is

b = 3 0 0   kg s = 3 0 0   kg f = 9 0 0   kg .

We do not yet have the tools to explain why this solution is the only one, but it should be simple for you to verify that this is indeed a solution. (Go ahead, we will wait.) Determining solutions such as this, and establishing that they are unique, will be the main motivation for our initial study of linear algebra.

So we have solved the problem of making sure that we make the best use of our limited storage space, and each day use up all of the raw ingredients that are shipped to us. Additionally, as production manager, you must report weekly to the CEO of the company, and you know he will be more interested in the profit derived from your decisions than in the actual production levels. So you compute,

3 0 0 ( 4 . 9 9 − 3 . 6 9 ) + 3 0 0 ( 5 . 5 0 − 3 . 8 6 ) + 9 0 0 ( 6 . 5 0 − 4 . 4 5 ) = 2 7 2 7 . 0 0

for a daily profit of $2,727 from this production schedule. The computation of the daily profit is also beyond our control, though it is definitely of interest, and it too looks like a “linear” computation.

As often happens, things do not stay the same for long, and now the marketing department has suggested that your company’s trail mix products standardize on every mix being one-third peanuts. Adjusting the peanut portion of each recipe by also adjusting the chocolate portion leads to revised recipes, and slightly different costs for the bulk and standard mixes, as given in the following table.

In a similar fashion as before, we desire values of b , s and f so that

b ≥ 0 s ≥ 0 f ≥ 0

and

7 1 5 b + 6 1 5 s + 2 1 5 f = 3 8 0  (raisins) 5 1 5 b + 5 1 5 s + 5 1 5 f = 5 0 0  (peanuts) 3 1 5 b + 4 1 5 s + 8 1 5 f = 6 2 0  (chocolate)

It now happens that this system of equations has infinitely many solutions, as we will now demonstrate. Let f remain a variable quantity. Then if we make f kilograms of the fancy mix, we will make 4 f − 3 3 0 0 kilograms of the bulk mix and − 5 f + 4 8 0 0 kilograms of the standard mix. Let us now verify that, for any choice of f , the values of b = 4 f − 3 3 0 0 and s = − 5 f + 4 8 0 0 will yield a production schedule that exhausts all of the day’s supply of raw ingredients (right now, do not be concerned about how you might derive expressions like these for b and s ). Grab your pencil and paper and play along.

7 1 5 ( 4 f − 3 3 0 0 ) + 6 1 5 ( − 5 f + 4 8 0 0 ) + 2 1 5 f = 0 f + 5 7 0 0 1 5 = 3 8 0 5 1 5 ( 4 f − 3 3 0 0 ) + 5 1 5 ( − 5 f + 4 8 0 0 ) + 5 1 5 f = 0 f + 7 5 0 0 1 5 = 5 0 0 3 1 5 ( 4 f − 3 3 0 0 ) + 4 1 5 ( − 5 f + 4 8 0 0 ) + 8 1 5 f = 0 f + 9 3 0 0 1 5 = 6 2 0

Convince yourself that these expressions for b and s allow us to vary f and obtain an infinite number of possibilities for solutions to the three equations that describe our storage capacities. As a practical matter, there really are not an infinite number of solutions, since we are unlikely to want to end the day with a fractional number of bags of fancy mix, so our allowable values of f should probably be integers. More importantly, we need to remember that we cannot make negative amounts of each mix! Where does this lead us? Positive quantities of the bulk mix requires that

b ≥ 0 ⇒ 4 f − 3 3 0 0 ≥ 0 ⇒ f ≥ 8 2 5

Similarly for the standard mix,

s ≥ 0 ⇒ − 5 f + 4 8 0 0 ≥ 0 ⇒ f ≤ 9 6 0

So, as production manager, you really have to choose a value of f from the finite set

8 2 5 , 8 2 6 , … , 9 6 0

leaving you with 136 choices, each of which will exhaust the day’s supply of raw ingredients. Pause now and think about which you would choose.

Recalling your weekly meeting with the CEO suggests that you might want to choose a production schedule that yields the biggest possible profit for the company. So you compute an expression for the profit based on your as yet undetermined decision for the value of f ,

( 4 f − 3 3 0 0 ) ( 4 . 9 9 − 3 . 7 0 ) + ( − 5 f + 4 8 0 0 ) ( 5 . 5 0 − 3 . 8 5 ) + ( f ) ( 6 . 5 0 − 4 . 4 5 ) = − 1 . 0 4 f + 3 6 6 3

Since f has a negative coefficient it would appear that mixing fancy mix is detrimental to your profit and should be avoided. So you will make the decision to set daily fancy mix production at f = 8 2 5 . This has the effect of setting b = 4 ( 8 2 5 ) − 3 3 0 0 = 0 and we stop producing bulk mix entirely. So the remainder of your daily production is standard mix at the level of s = − 5 ( 8 2 5 ) + 4 8 0 0 = 6 7 5 kilograms and the resulting daily profit is ( − 1 . 0 4 ) ( 8 2 5 ) + 3 6 6 3 = 2 8 0 5 . It is a pleasant surprise that daily profit has risen to $2,805, but this is not the most important part of the story. What is important here is that there are a large number of ways to produce trail mix that use all of the day’s worth of raw ingredients and you were able to easily choose the one that netted the largest profit. Notice too how all of the above computations look “linear.”

In the food industry, things do not stay the same for long, and now the sales department says that increased competition has led to the decision to stay competitive and charge just $5.25 for a kilogram of the standard mix, rather than the previous $5.50 per kilogram. This decision has no effect on the possibilities for the production schedule, but will affect the decision based on profit considerations. So you revisit just the profit computation, suitably adjusted for the new selling price of standard mix,

( 4 f − 3 3 0 0 ) ( 4 . 9 9 − 3 . 7 0 ) + ( − 5 f + 4 8 0 0 ) ( 5 . 2 5 − 3 . 8 5 ) + ( f ) ( 6 . 5 0 − 4 . 4 5 ) = 0 . 2 1 f + 2 4 6 3

Now it would appear that fancy mix is beneficial to the company’s profit since the value of f has a positive coefficient. So you take the decision to make as much fancy mix as possible, setting f = 9 6 0 . This leads to s = − 5 ( 9 6 0 ) + 4 8 0 0 = 0 and the increased competition has driven you out of the standard mix market all together. The remainder of production is therefore bulk mix at a daily level of b = 4 ( 9 6 0 ) − 3 3 0 0 = 5 4 0 kilograms and the resulting daily profit is 0 . 2 1 ( 9 6 0 ) + 2 4 6 3 = 2 6 6 4 . 6 0 . A daily profit of $2,664.60 is less than it used to be, but as production manager, you have made the best of a difficult situation and shown the sales department that the best course is to pull out of the highly competitive standard mix market completely. ⊠

This example is taken from a field of mathematics variously known by names such as operations research, systems science, or management science. More specifically, this is a perfect example of problems that are solved by the techniques of “linear programming.”

There is a lot going on under the hood in this example. The heart of the matter is the solution to systems of linear equations, which is the topic of the next few sections, and a recurrent theme throughout this course. We will return to this example on several occasions to reveal some of the reasons for its behavior.

Subsection READ: Reading Questions

Subsection EXC: Exercises

C10 In Example TMP the first table lists the cost (per kilogram) to manufacture each of the three varieties of trail mix (bulk, standard, fancy). For example, it costs $3.69 to make one kilogram of the bulk variety. Re-compute each of these three costs and notice that the computations are linear in character.  
Contributed by  Robert Beezer

M70 In Example TMP two different prices were considered for marketing standard mix with the revised recipes (one-third peanuts in each recipe). Selling standard mix at $5.50 resulted in selling the minimum amount of the fancy mix and no bulk mix. At $5.25 it was best for profits to sell the maximum amount of fancy mix and then sell no standard mix. Determine a selling price for standard mix that allows for maximum profits while still selling some of each type of mix.  
Contributed by  Robert Beezer Solution [ 26]

Subsection SOL: Solutions

M70 Contributed by  Robert Beezer Statement [ 25]
If the price of standard mix is set at $5.292, then the profit function has a zero coefficient on the variable quantity f . So, we can set f to be any integer quantity in 8 2 5 , 8 2 6 , … , 9 6 0 . All but the extreme values ( f = 8 2 5 , f = 9 6 0 ) will result in production levels where some of every mix is manufactured. No matter what value of f is chosen, the resulting profit will be the same, at $2,664.60.

Section SSLE  Solving Systems of Linear Equations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

We will motivate our study of linear algebra by considering the problem of solving several linear equations simultaneously. The word “solve” tends to get abused somewhat, as in “solve this problem.” When talking about equations we understand a more precise meaning: find all of the values of some variable quantities that make an equation, or several equations, true.

Parent topic: Chapter SLE  Systems of Linear Equations

Previous topic: Section WILA  What is Linear Algebra?

Next topic: Section RREF  Reduced Row-Echelon Form

Subsection SLE: Systems of Linear Equations

Example  STNE
Solving two (nonlinear) equations
Suppose we desire the simultaneous solutions of the two equations,

x 2 + y 2 = 1 − x + 3 y = 0

You can easily check by substitution that x = 3 2 , y = 1 2 and x = − 3 2 , y = − 1 2 are both solutions. We need to also convince ourselves that these are the only solutions. To see this, plot each equation on the x y -plane, which means to plot ( x , y ) pairs that make an individual equation true. In this case we get a circle centered at the origin with radius 1 and a straight line through the origin with slope 1 3 . The intersections of these two curves are our desired simultaneous solutions, and so we believe from our plot that the two solutions we know already are indeed the only ones. We like to write solutions as sets, so in this case we write the set of solutions as

S = 3 2 , 1 2 , − 3 2 , − 1 2

⊠

In order to discuss systems of linear equations carefully, we need a precise definition. And before we do that, we will introduce our periodic discussions about “Proof Techniques.” Linear algebra is an excellent setting for learning how to read, understand and formulate proofs. But this is a difficult step in your development as a mathematician, so we have included a series of short essays containing advice and explanations to help you along. These can be found back in Section PT of Appendix P, and we will reference them as they become appropriate. Be sure to head back to the appendix to read this as they are introduced. With a definition next, now is the time for the first of our proof techniques. Head back to Section PT of Appendix P and study Technique D. We’ll be right here when you get back. See you in a bit. 

Definition  SLE
System of Linear Equations
A system of linear equations is a collection of m equations in the variable quantities x 1 , x 2 , x 3 , … , x n of the form,

a 1 1 x 1 + a 1 2 x 2 + a 1 3 x 3 + ⋯ + a 1 n x n = b 1 a 2 1 x 1 + a 2 2 x 2 + a 2 3 x 3 + ⋯ + a 2 n x n = b 2 a 3 1 x 1 + a 3 2 x 2 + a 3 3 x 3 + ⋯ + a 3 n x n = b 3 ⋮ a m 1 x 1 + a m 2 x 2 + a m 3 x 3 + ⋯ + a m n x n = b m

where the values of a i j , b i and x j are from the set of complex numbers, ℂ . △

Don’t let the mention of the complex numbers, ℂ , rattle you. We will stick with real numbers exclusively for many more sections, and it will sometimes seem like we only work with integers! However, we want to leave the possibility of complex numbers open, and there will be occasions in subsequent sections where they are necessary. You can review the basic properties of complex numbers in Section CNO, but these facts will not be critical until we reach Section O.

Now we make the notion of a solution to a linear system precise.

Definition  SSLE
Solution of a System of Linear Equations
A solution of a system of linear equations in n variables, x 1 , x 2 , x 3 , … , x n (such as the system given in Definition SLE, is an ordered list of n complex numbers, s 1 , s 2 , s 3 , … , s n such that if we substitute s 1 for x 1 , s 2 for x 2 , s 3 for x 3 , …, s n for x n , then for every equation of the system the left side will equal the right side, i.e. each equation is true simultaneously. △

More typically, we will write a solution in a form like x 1 = 1 2 , x 2 = − 7 , x 3 = 2 to mean that s 1 = 1 2 , s 2 = − 7 , s 3 = 2 in the notation of Definition SSLE. To discuss all of the possible solutions to a system of linear equations, we now define the set of all solutions. (So Section SET is now applicable, and you may want to go and familiarize yourself with what is there.)

Definition  SSSLE
Solution Set of a System of Linear Equations
The solution set of a linear system of equations is the set which contains every solution to the system, and nothing more. △

Be aware that a solution set can be infinite, or there can be no solutions, in which case we write the solution set as the empty set, ∅ = ( Definition ES). Here is an example to illustrate using the notation introduced in Definition SLE and the notion of a solution ( Definition SSLE).

Example  NSE
Notation for a system of equations
Given the system of linear equations,

x 1 + 2 x 2 + x 4 = 7 x 1 + x 2 + x 3 − x 4 = 3 3 x 1 + x 2 + 5 x 3 − 7 x 4 = 1

we have n = 4 variables and m = 3 equations. Also,

a 1 1 = 1 a 1 2 = 2 a 1 3 = 0 a 1 4 = 1 b 1 = 7 a 2 1 = 1 a 2 2 = 1 a 2 3 = 1 a 2 4 = − 1 b 2 = 3 a 3 1 = 3 a 3 2 = 1 a 3 3 = 5 a 3 4 = − 7 b 3 = 1

Additionally, convince yourself that x 1 = − 2 , x 2 = 4 , x 3 = 2 , x 4 = 1 is one solution ( Definition SSLE), but it is not the only one! For example, another solution is x 1 = − 1 2 , x 2 = 1 1 , x 3 = 1 , x 4 = − 3 , and there are more to be found. So the solution set contains at least two elements. ⊠

We will often shorten the term “system of linear equations” to “system of equations” leaving the linear aspect implied. After all, this is a book about linear algebra.

Subsection PSS: Possibilities for Solution Sets

The next example illustrates the possibilities for the solution set of a system of linear equations. We will not be too formal here, and the necessary theorems to back up our claims will come in subsequent sections. So read for feeling and come back later to revisit this example.

Example  TTS
Three typical systems
Consider the system of two equations with two variables,

2 x 1 + 3 x 2 = 3 x 1 − x 2 = 4

If we plot the solutions to each of these equations separately on the x 1 x 2 -plane, we get two lines, one with negative slope, the other with positive slope. They have exactly one point in common, ( x 1 , x 2 ) = ( 3 , − 1 ) , which is the solution x 1 = 3 , x 2 = − 1 . From the geometry, we believe that this is the only solution to the system of equations, and so we say it is unique.

Now adjust the system with a different second equation,

2 x 1 + 3 x 2 = 3 4 x 1 + 6 x 2 = 6

A plot of the solutions to these equations individually results in two lines, one on top of the other! There are infinitely many pairs of points that make both equations true. We will learn shortly how to describe this infinite solution set precisely (see Example SAA, Theorem VFSLS). Notice now how the second equation is just a multiple of the first.

One more minor adjustment provides a third system of linear equations,

2 x 1 + 3 x 2 = 3 4 x 1 + 6 x 2 = 1 0

A plot now reveals two lines with identical slopes, i.e. parallel lines. They have no points in common, and so the system has a solution set that is empty, S = ∅ . ⊠

This example exhibits all of the typical behaviors of a system of equations. A subsequent theorem will tell us that every system of linear equations has a solution set that is empty, contains a single solution or contains infinitely many solutions (Theorem PSSLS). Example STNE yielded exactly two solutions, but this does not contradict the forthcoming theorem. The equations in Example STNE are not linear because they do not match the form of Definition SLE, and so we cannot apply Theorem PSSLS in this case.

Subsection ESEO: Equivalent Systems and Equation Operations

With all this talk about finding solution sets for systems of linear equations, you might be ready to begin learning how to find these solution sets yourself. We begin with our first definition that takes a common word and gives it a very precise meaning in the context of systems of linear equations.

Definition  ESYS
Equivalent Systems
Two systems of linear equations are equivalent if their solution sets are equal. △

Notice here that the two systems of equations could look very different (i.e. not be equal), but still have equal solution sets, and we would then call the systems equivalent. Two linear equations in two variables might be plotted as two lines that intersect in a single point. A different system, with three equations in two variables might have a plot that is three lines, all intersecting at a common point, with this common point identical to the intersection point for the first system. By our definition, we could then say these two very different looking systems of equations are equivalent, since they have identical solution sets. It is really like a weaker form of equality, where we allow the systems to be different in some respects, but we use the term equivalent to highlight the situation when their solution sets are equal.

With this definition, we can begin to describe our strategy for solving linear systems. Given a system of linear equations that looks difficult to solve, we would like to have an equivalent system that is easy to solve. Since the systems will have equal solution sets, we can solve the “easy” system and get the solution set to the “difficult” system. Here come the tools for making this strategy viable.

Definition  EO
Equation Operations
Given a system of linear equations, the following three operations will transform the system into a different one, and each operation is known as an equation operation.

△

These descriptions might seem a bit vague, but the proof or the examples that follow should make it clear what is meant by each. We will shortly prove a key theorem about equation operations and solutions to linear systems of equations. We are about to give a rather involved proof, so a discussion about just what a theorem really is would be timely. Head back and read Technique T.  In the theorem we are about to prove, the conclusion is that two systems are equivalent. By Definition ESYS this translates to requiring that solution sets be equal for the two systems. So we are being asked to show that two sets are equal. How do we do this? Well, there is a very standard technique, and we will use it repeatedly through the course. If you have not done so already, head to Section SET and familiarize yourself with sets, their operations, and especially the notion of set equality, Definition SE and the nearby discussion about its use.

Theorem  EOPSS
Equation Operations Preserve Solution Sets
If we apply one of the three equation operations of Definition EO to a system of linear equations ( Definition SLE), then the original system and the transformed system are equivalent. □

Proof   We take each equation operation in turn and show that the solution sets of the two systems are equal, using the definition of set equality ( Definition SE).

■

Theorem EOPSS is the necessary tool to complete our strategy for solving systems of equations. We will use equation operations to move from one system to another, all the while keeping the solution set the same. With the right sequence of operations, we will arrive at a simpler equation to solve. The next two examples illustrate this idea, while saving some of the details for later.

Example  US
Three equations, one solution
We solve the following system by a sequence of equation operations.

x 1 + 2 x 2 + 2 x 3 = 4 x 1 + 3 x 2 + 3 x 3 = 5 2 x 1 + 6 x 2 + 5 x 3 = 6   α = − 1  times equation 1, add to equation 2: x 1 + 2 x 2 + 2 x 3 = 4 0 x 1 + 1 x 2 + 1 x 3 = 1 2 x 1 + 6 x 2 + 5 x 3 = 6   α = − 2  times equation 1, add to equation 3: x 1 + 2 x 2 + 2 x 3 = 4 0 x 1 + 1 x 2 + 1 x 3 = 1 0 x 1 + 2 x 2 + 1 x 3 = − 2   α = − 2  times equation 2, add to equation 3: x 1 + 2 x 2 + 2 x 3 = 4 0 x 1 + 1 x 2 + 1 x 3 = 1 0 x 1 + 0 x 2 − 1 x 3 = − 4   α = − 1  times equation 3: x 1 + 2 x 2 + 2 x 3 = 4 0 x 1 + 1 x 2 + 1 x 3 = 1 0 x 1 + 0 x 2 + 1 x 3 = 4  which can be written more clearly as x 1 + 2 x 2 + 2 x 3 = 4 x 2 + x 3 = 1 x 3 = 4

This is now a very easy system of equations to solve. The third equation requires that x 3 = 4 to be true. Making this substitution into equation 2 we arrive at x 2 = − 3 , and finally, substituting these values of x 2 and x 3 into the first equation, we find that x 1 = 2 . Note too that this is the only solution to this final system of equations, since we were forced to choose these values to make the equations true. Since we performed equation operations on each system to obtain the next one in the list, all of the systems listed here are all equivalent to each other by Theorem EOPSS. Thus ( x 1 , x 2 , x 3 ) = ( 2 , − 3 , 4 ) is the unique solution to the original system of equations (and all of the other intermediate systems of equations listed as we transformed one into another). ⊠

Example  IS
Three equations, infinitely many solutions
The following system of equations made an appearance earlier in this section ( Example NSE), where we listed one of its solutions. Now, we will try to find all of the solutions to this system. Don’t concern yourself too much about why we choose this particular sequence of equation operations, just believe that the work we do is all correct.

x 1 + 2 x 2 + 0 x 3 + x 4 = 7 x 1 + x 2 + x 3 − x 4 = 3 3 x 1 + x 2 + 5 x 3 − 7 x 4 = 1   α = − 1  times equation 1, add to equation 2: x 1 + 2 x 2 + 0 x 3 + x 4 = 7 0 x 1 − x 2 + x 3 − 2 x 4 = − 4 3 x 1 + x 2 + 5 x 3 − 7 x 4 = 1   α = − 3  times equation 1, add to equation 3: x 1 + 2 x 2 + 0 x 3 + x 4 = 7 0 x 1 − x 2 + x 3 − 2 x 4 = − 4 0 x 1 − 5 x 2 + 5 x 3 − 1 0 x 4 = − 2 0   α = − 5  times equation 2, add to equation 3: x 1 + 2 x 2 + 0 x 3 + x 4 = 7 0 x 1 − x 2 + x 3 − 2 x 4 = − 4 0 x 1 + 0 x 2 + 0 x 3 + 0 x 4 = 0   α = − 1  times equation 2: x 1 + 2 x 2 + 0 x 3 + x 4 = 7 0 x 1 + x 2 − x 3 + 2 x 4 = 4 0 x 1 + 0 x 2 + 0 x 3 + 0 x 4 = 0   α = − 2  times equation 2, add to equation 1: x 1 + 0 x 2 + 2 x 3 − 3 x 4 = − 1 0 x 1 + x 2 − x 3 + 2 x 4 = 4 0 x 1 + 0 x 2 + 0 x 3 + 0 x 4 = 0  which can be written more clearly as x 1 + 2 x 3 − 3 x 4 = − 1 x 2 − x 3 + 2 x 4 = 4 0 = 0

What does the equation 0 = 0 mean? We can choose any values for x 1 , x 2 , x 3 , x 4 and this equation will be true, so we only need to consider further the first two equations, since the third is true no matter what. We can analyze the second equation without consideration of the variable x 1 . It would appear that there is considerable latitude in how we can choose x 2 , x 3 , x 4 and make this equation true. Let’s choose x 3 and x 4 to be anything we please, say x 3 = a and x 4 = b .

Now we can take these arbitrary values for x 3 and x 4 , substitute them in equation 1, to obtain

x 1 + 2 a − 3 b = − 1 x 1 = − 1 − 2 a + 3 b  Similarly, equation 2 becomes x 2 − a + 2 b = 4 x 2 = 4 + a − 2 b

So our arbitrary choices of values for x 3 and x 4 ( a and b ) translate into specific values of x 1 and x 2 . The lone solution given in Example NSE was obtained by choosing a = 2 and b = 1 . Now we can easily and quickly find many more (infinitely more). Suppose we choose a = 5 and b = − 2 , then we compute

x 1 = − 1 − 2 ( 5 ) + 3 ( − 2 ) = − 1 7 x 2 = 4 + 5 − 2 ( − 2 ) = 1 3

and you can verify that ( x 1 , x 2 , x 3 , x 4 ) = ( − 1 7 , 1 3 , 5 , − 2 ) makes all three equations true. The entire solution set is written as

S = ( − 1 − 2 a + 3 b , 4 + a − 2 b , a , b ) a ∈ ℂ , b ∈ ℂ

It would be instructive to finish off your study of this example by taking the general form of the solutions given in this set and substituting them into each of the three equations and verify that they are true in each case ( Exercise SSLE.M40). ⊠

In the next section we will describe how to use equation operations to systematically solve any system of linear equations. But first, read one of our more important pieces of advice about speaking and writing mathematics. See Technique L. 

Before attacking the exercises in this section, it will be helpful to read some advice on getting started on the construction of a proof. See Technique GS. 

Subsection READ: Reading Questions

Subsection EXC: Exercises

C10 Find a solution to the system in Example IS where x 3 = 6 and x 4 = 2 . Find two other solutions to the system. Find a solution where x 1 = − 1 7 and x 2 = 1 4 . How many possible answers are there to each of these questions?  
Contributed by  Robert Beezer

C20 Each archetype ( Appendix A) that is a system of equations begins by listing some specific solutions. Verify the specific solutions listed in the following archetypes by evaluating the system of equations with the solutions listed.
Archetype A
Archetype B
Archetype C
Archetype D
Archetype E
Archetype F
Archetype G
Archetype H
Archetype I
Archetype J  
Contributed by  Robert Beezer

C30 Find all solutions to the linear system:

x + y = 5 2 x − y = 3

Contributed by  Chris Black Solution [ 59]

C31 Find all solutions to the linear system:

3 x + 2 y = 1 x − y = 2 4 x + 2 y = 2

Contributed by  Chris Black

C32 Find all solutions to the linear system:

x + 2 y = 8 x − y = 2 x + y = 4

Contributed by  Chris Black

C33 Find all solutions to the linear system:

x + y − z = − 1 x − y − z = − 1 z = 2

Contributed by  Chris Black

C34 Find all solutions to the linear system:

x + y − z = − 5 x − y − z = − 3 x + y − z = 0

Contributed by  Chris Black

C50 A three-digit number has two properties. The tens-digit and the ones-digit add up to 5. If the number is written with the digits in the reverse order, and then subtracted from the original number, the result is 7 9 2 . Use a system of equations to find all of the three-digit numbers with these properties.  
Contributed by  Robert Beezer Solution [ 59]

C51 Find all of the six-digit numbers in which the first digit is one less than the second, the third digit is half the second, the fourth digit is three times the third and the last two digits form a number that equals the sum of the fourth and fifth. The sum of all the digits is 24. (From The MENSA Puzzle Calendar for January 9, 2006.)  
Contributed by  Robert Beezer Solution [ 61]

C52 Driving along, Terry notices that the last four digits on his car’s odometer are palindromic. A mile later, the last five digits are palindromic. After driving another mile, the middle four digits are palindromic. One more mile, and all six are palindromic. What was the odometer reading when Terry first looked at it? Form a linear system of equations that expresses the requirements of this puzzle. ( Car Talk Puzzler, National Public Radio, Week of January 21, 2008) (A car odometer displays six digits and a sequence is a palindrome if it reads the same left-to-right as right-to-left.)  
Contributed by  Robert Beezer Solution [ 63]

M10 Each sentence below has at least two meanings. Identify the source of the double meaning, and rewrite the sentence (at least twice) to clearly convey each meaning.

Contributed by  Robert Beezer Solution [ 63]

M11 Discuss the difference in meaning of each of the following three almost identical sentences, which all have the same grammatical structure. (These are due to Keith Devlin.)

Contributed by  Robert Beezer Solution [ 64]

M12 The following sentence, due to Noam Chomsky, has a correct grammatical structure, but is meaningless. Critique its faults. “Colorless green ideas sleep furiously.” (Chomsky, Noam. Syntactic Structures, The Hague/Paris: Mouton, 1957. p. 15.)  
Contributed by  Robert Beezer Solution [ 64]

M13 Read the following sentence and form a mental picture of the situation.

What assumptions did you make about the situation?  
Contributed by  Robert Beezer Solution [ 64]

M30 This problem appears in a middle-school mathematics textbook: Together Dan and Diane have $20. Together Diane and Donna have $15. How much do the three of them have in total? ( Transition Mathematics, Second Edition, Scott Foresman Addison Wesley, 1998. Problem 5–1.19.)  
Contributed by  David Beezer Solution [ 64]

M40 Solutions to the system in Example IS are given as

( x 1 , x 2 , x 3 , x 4 ) = ( − 1 − 2 a + 3 b , 4 + a − 2 b , a , b )

Evaluate the three equations of the original system with these expressions in a and b and verify that each equation is true, no matter what values are chosen for a and b .  
Contributed by  Robert Beezer

M70 We have seen in this section that systems of linear equations have limited possibilities for solution sets, and we will shortly prove Theorem PSSLS that describes these possibilities exactly. This exercise will show that if we relax the requirement that our equations be linear, then the possibilities expand greatly. Consider a system of two equations in the two variables x and y , where the departure from linearity involves simply squaring the variables.

x 2 − y 2 = 1 x 2 + y 2 = 4

After solving this system of non-linear equations, replace the second equation in turn by x 2 + 2 x + y 2 = 3 , x 2 + y 2 = 1 , x 2 − 4 x + y 2 = − 3 , − x 2 + y 2 = 1 and solve each resulting system of two equations in two variables. (This exercise includes suggestions from Don Kreher.)  
Contributed by  Robert Beezer Solution [ 65]

T10 Technique D asks you to formulate a definition of what it means for a whole number to be odd. What is your definition? (Don’t say “the opposite of even.”) Is 6 odd? Is 1 1 odd? Justify your answers by using your definition.  
Contributed by  Robert Beezer Solution [ 66]

T20 Explain why the second equation operation in Definition EO requires that the scalar be nonzero, while in the third equation operation this restriction on the scalar is not present.  
Contributed by  Robert Beezer Solution [ 66]

Subsection SOL: Solutions

C30 Contributed by  Chris Black Statement [ 51]
Solving each equation for y , we have the equivalent system

y = 5 − x y = 2 x − 3 .

Setting these expressions for y equal, we have the equation 5 − x = 2 x − 3 , which quickly leads to x = 8 3 . Substituting for x in the first equation, we have y = 5 − x = 5 − 8 3 = 7 3 . Thus, the solution is x = 8 3 , y = 7 3 .

C50 Contributed by  Robert Beezer Statement [ 54]
Let a be the hundreds digit, b the tens digit, and c the ones digit. Then the first condition says that b + c = 5 . The original number is 1 0 0 a + 1 0 b + c , while the reversed number is 1 0 0 c + 1 0 b + a . So the second condition is

7 9 2 = 1 0 0 a + 1 0 b + c − 1 0 0 c + 1 0 b + a = 9 9 a − 9 9 c

So we arrive at the system of equations

b + c = 5 9 9 a − 9 9 c = 7 9 2

Using equation operations, we arrive at the equivalent system

a − c = 8 b + c = 5

We can vary c and obtain infinitely many solutions. However, c must be a digit, restricting us to ten values (0 – 9). Furthermore, if c > 1 , then the first equation forces a > 9 , an impossibility. Setting c = 0 , yields 8 5 0 as a solution, and setting c = 1 yields 9 4 1 as another solution.

C51 Contributed by  Robert Beezer Statement [ 54]
Let a b c d e f denote any such six-digit number and convert each requirement in the problem statement into an equation.

a = b − 1 c = 1 2 b d = 3 c 1 0 ∗ e + f = d + e 2 4 = a + b + c + d + e + f

In a more standard form this becomes

a − b = − 1 − b + 2 c = 0 − 3 c + d = 0 − d + 9 e + f = 0 a + b + c + d + e + f = 2 4

Using equation operations (or the techniques of the upcoming Section RREF), this system can be converted to the equivalent system

a + 1 6 7 5 f = 5 b + 1 6 7 5 f = 6 c + 8 7 5 f = 3 d + 8 2 5 f = 9 e + 1 1 7 5 f = 1

Clearly, choosing f = 0 will yield the solution a b c d e = 5 6 3 9 1 0 . Furthermore, to have the variables result in single-digit numbers, none of the other choices for f ( 1 , 2 , … , 9 ) will yield a solution.

C52 Contributed by  Robert Beezer Statement [ 54]
198888 is one solution, and David Braithwaite found 199999 as another.

M10 Contributed by  Robert Beezer Statement [ 54]

M11 Contributed by  Robert Beezer Statement [ 55]
We know the dog belongs to the man, and the fountain belongs to the park. It is not clear if the telescope belongs to the man, the woman, or the park.

M12 Contributed by  Robert Beezer Statement [ 55]
In adjacent pairs the words are contradictory or inappropriate. Something cannot be both green and colorless, ideas do not have color, ideas do not sleep, and it is hard to sleep furiously.

M13 Contributed by  Robert Beezer Statement [ 55]
Did you assume that the baby and mother are human?
Did you assume that the baby is the child of the mother?
Did you assume that the mother picked up the baby as an attempt to stop the crying?

M30 Contributed by  Robert Beezer Statement [ 56]
If x , y and z represent the money held by Dan, Diane and Donna, then y = 1 5 − z and x = 2 0 − y = 2 0 − ( 1 5 − z ) = 5 + z . We can let z take on any value from 0 to 1 5 without any of the three amounts being negative, since presumably middle-schoolers are too young to assume debt.

Then the total capital held by the three is x + y + z = ( 5 + z ) + ( 1 5 − z ) + z = 2 0 + z . So their combined holdings can range anywhere from $20 (Donna is broke) to $35 (Donna is flush).

We will have more to say about this situation in Section TSS, and specifically Theorem CMVEI.

M70 Contributed by  Robert Beezer Statement [ 56]
The equation x 2 − y 2 = 1 has a solution set by itself that has the shape of a hyperbola when plotted. Four of the five different second equations have solution sets that are circles when plotted individually (the last is another hyperbola). Where the hyperbola and circles intersect are the solutions to the system of two equations. As the size and location of the circles vary, the number of intersections varies from four to one (in the order given). Teh last equation is a hyperbola that “opens” in the other direction. Sketching the relevant equations would be instructive, as was discussed in Example STNE.

The exact solution sets are (according to the choice of the second equation),

x 2 + y 2 = 4 : 5 2 , 3 2 , − 5 2 , 3 2 , 5 2 , − 3 2 , − 5 2 , − 3 2 x 2 + 2 x + y 2 = 3 : ( 1 , 0 ) , ( − 2 , 3 ) , ( − 2 , − 3 ) x 2 + y 2 = 1 : ( 1 , 0 ) , ( − 1 , 0 ) x 2 − 4 x + y 2 = − 3 : ( 1 , 0 ) − x 2 + y 2 = 1 :

T10 Contributed by  Robert Beezer Statement [ 57]
We can say that an integer is odd if when it is divided by 2 there is a remainder of 1. So 6 is not odd since 6 = 3 × 2 + 0 , while 1 1 is odd since 1 1 = 5 × 2 + 1 .

T20 Contributed by  Robert Beezer Statement [ 57]
Definition EO is engineered to make Theorem EOPSS true. If we were to allow a zero scalar to multiply an equation then that equation would be transformed to the equation 0 = 0 , which is true for any possible values of the variables. Any restrictions on the solution set imposed by the original equation would be lost.

However, in the third operation, it is allowed to choose a zero scalar, multiply an equation by this scalar and add the transformed equation to a second equation (leaving the first unchanged). The result? Nothing. The second equation is the same as it was before. So the theorem is true in this case, the two systems are equivalent. But in practice, this would be a silly thing to actually ever do! We still allow it though, in order to keep our theorem as general as possible.

Notice the location in the proof of Theorem EOPSS where the expression 1 α appears — this explains the prohibition on α = 0 in the second equation operation.

Section RREF  Reduced Row-Echelon Form

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

After solving a few systems of equations, you will recognize that it doesn’t matter so much what we call our variables, as opposed to what numbers act as their coefficients. A system in the variables x 1 , x 2 , x 3 would behave the same if we changed the names of the variables to a , b , c and kept all the constants the same and in the same places. In this section, we will isolate the key bits of information about a system of equations into something called a matrix, and then use this matrix to systematically solve the equations. Along the way we will obtain one of our most important and useful computational tools.

Parent topic: Chapter SLE  Systems of Linear Equations

Previous topic: Section SSLE  Solving Systems of Linear Equations

Next topic: Section HSE  Homogeneous Systems of Equations

Subsection MVNSE: Matrix and Vector Notation for Systems of Equations

Definition  M
Matrix
An m × n matrix is a rectangular layout of numbers from ℂ having m rows and n columns. We will use upper-case Latin letters from the start of the alphabet ( A , B , C , … ) to denote matrices and squared-off brackets to delimit the layout. Many use large parentheses instead of brackets — the distinction is not important. Rows of a matrix will be referenced starting at the top and working down (i.e. row 1 is at the top) and columns will be referenced starting from the left (i.e. column 1 is at the left). For a matrix A , the notation A i j will refer to the complex number in row i and column j of A .

(This definition contains Notation M.)

(This definition contains Notation MC.) △

Be careful with this notation for individual entries, since it is easy to think that A i j refers to the whole matrix. It does not. It is just a number, but is a convenient way to talk about the individual entries simultaneously. This notation will get a heavy workout once we get to Chapter M.

Example  AM
A matrix

B = − 1 2 5 3 1 0 − 6 1 − 4 2 2 − 2

is a matrix with m = 3 rows and n = 4 columns. We can say that B 2 , 3 = − 6 while B 3 , 4 = − 2 . ⊠

Some mathematical software is very particular about which types of numbers (integers, rationals, reals, complexes) you wish to work with.  See: Computation R.SAGE A calculator or computer language can be a convenient way to perform calculations with matrices. But first you have to enter the matrix.  See: Computation ME.MMA Computation ME.TI86 Computation ME.TI83 Computation ME.SAGE When we do equation operations on system of equations, the names of the variables really aren’t very important. x 1 , x 2 , x 3 , or a , b , c , or x , y , z , it really doesn’t matter. In this subsection we will describe some notation that will make it easier to describe linear systems, solve the systems and describe the solution sets. Here is a list of definitions, laden with notation.

Definition  CV
Column Vector
A column vector of size m is an ordered list of m numbers, which is written in order vertically, starting at the top and proceeding to the bottom. At times, we will refer to a column vector as simply a vector. Column vectors will be written in bold, usually with lower case Latin letter from the end of the alphabet such as u , v , w , x , y , z . Some books like to write vectors with arrows, such as u → . Writing by hand, some like to put arrows on top of the symbol, or a tilde underneath the symbol, as in u ∼ . To refer to the entry or component that is number i in the list that is the vector v we write v i .

(This definition contains Notation CV.)

(This definition contains Notation CVC.) △

Be careful with this notation. While the symbols v i might look somewhat substantial, as an object this represents just one component of a vector, which is just a single complex number.

Definition  ZCV
Zero Column Vector
The zero vector of size m is the column vector of size m where each entry is the number zero,

0 = 0 0 0 ⋮ 0

or defined much more compactly, 0 i = 0 for 1 ≤ i ≤ m .

(This definition contains Notation ZCV.) △

Definition  CM
Coefficient Matrix
For a system of linear equations,

a 1 1 x 1 + a 1 2 x 2 + a 1 3 x 3 + ⋯ + a 1 n x n = b 1 a 2 1 x 1 + a 2 2 x 2 + a 2 3 x 3 + ⋯ + a 2 n x n = b 2 a 3 1 x 1 + a 3 2 x 2 + a 3 3 x 3 + ⋯ + a 3 n x n = b 3 ⋮ a m 1 x 1 + a m 2 x 2 + a m 3 x 3 + ⋯ + a m n x n = b m

the coefficient matrix is the m × n matrix

A = a 1 1 a 1 2 a 1 3 … a 1 n a 2 1 a 2 2 a 2 3 … a 2 n a 3 1 a 3 2 a 3 3 … a 3 n ⋮ a m 1 a m 2 a m 3 … a m n

△

Definition  VOC
Vector of Constants
For a system of linear equations,

a 1 1 x 1 + a 1 2 x 2 + a 1 3 x 3 + ⋯ + a 1 n x n = b 1 a 2 1 x 1 + a 2 2 x 2 + a 2 3 x 3 + ⋯ + a 2 n x n = b 2 a 3 1 x 1 + a 3 2 x 2 + a 3 3 x 3 + ⋯ + a 3 n x n = b 3 ⋮ a m 1 x 1 + a m 2 x 2 + a m 3 x 3 + ⋯ + a m n x n = b m

the vector of constants is the column vector of size m

b = b 1 b 2 b 3 ⋮ b m

△

Definition  SOLV
Solution Vector
For a system of linear equations,

a 1 1 x 1 + a 1 2 x 2 + a 1 3 x 3 + ⋯ + a 1 n x n = b 1 a 2 1 x 1 + a 2 2 x 2 + a 2 3 x 3 + ⋯ + a 2 n x n = b 2 a 3 1 x 1 + a 3 2 x 2 + a 3 3 x 3 + ⋯ + a 3 n x n = b 3 ⋮ a m 1 x 1 + a m 2 x 2 + a m 3 x 3 + ⋯ + a m n x n = b m

the solution vector is the column vector of size n

x = x 1 x 2 x 3 ⋮ x n

△

The solution vector may do double-duty on occasion. It might refer to a list of variable quantities at one point, and subsequently refer to values of those variables that actually form a particular solution to that system.

Definition  MRLS
Matrix Representation of a Linear System
If A is the coefficient matrix of a system of linear equations and b is the vector of constants, then we will write ℒ S A , b as a shorthand expression for the system of linear equations, which we will refer to as the matrix representation of the linear system.

(This definition contains Notation MRLS.) △

Example  NSLE
Notation for systems of linear equations
The system of linear equations

2 x 1 + 4 x 2 − 3 x 3 + 5 x 4 + x 5 = 9 3 x 1 + x 2 + x 4 − 3 x 5 = 0 − 2 x 1 + 7 x 2 − 5 x 3 + 2 x 4 + 2 x 5 = − 3

has coefficient matrix

A = 2 4 − 3 5 1 3 1 0 1 − 3 − 2 7 − 5 2 2

and vector of constants

b = 9 0 − 3

and so will be referenced as ℒ S A , b . ⊠

Definition  AM
Augmented Matrix
Suppose we have a system of m equations in n variables, with coefficient matrix A and vector of constants b . Then the augmented matrix of the system of equations is the m × ( n + 1 ) matrix whose first n columns are the columns of A and whose last column (number n + 1 ) is the column vector b . This matrix will be written as A b .

(This definition contains Notation AM.) △

The augmented matrix represents all the important information in the system of equations, since the names of the variables have been ignored, and the only connection with the variables is the location of their coefficients in the matrix. It is important to realize that the augmented matrix is just that, a matrix, and not a system of equations. In particular, the augmented matrix does not have any “solutions,” though it will be useful for finding solutions to the system of equations that it is associated with. (Think about your objects, and review Technique L.) However, notice that an augmented matrix always belongs to some system of equations, and vice versa, so it is tempting to try and blur the distinction between the two. Here’s a quick example.

Example  AMAA
Augmented matrix for Archetype A
Archetype A is the following system of 3 equations in 3 variables.

x 1 − x 2 + 2 x 3 = 1 2 x 1 + x 2 + x 3 = 8 x 1 + x 2 = 5

Here is its augmented matrix.

1 − 1 2 1 2 1 1 8 1 1 0 5

⊠

Subsection RO: Row Operations

An augmented matrix for a system of equations will save us the tedium of continually writing down the names of the variables as we solve the system. It will also release us from any dependence on the actual names of the variables. We have seen how certain operations we can perform on equations ( Definition EO) will preserve their solutions ( Theorem EOPSS). The next two definitions and the following theorem carry over these ideas to augmented matrices.

Definition  RO
Row Operations
The following three operations will transform an m × n matrix into a different matrix of the same size, and each is known as a row operation.

We will use a symbolic shorthand to describe these row operations:

(This definition contains Notation RO.) △

Definition  REM
Row-Equivalent Matrices
Two matrices, A and B , are row-equivalent if one can be obtained from the other by a sequence of row operations. △

Example  TREM
Two row-equivalent matrices
The matrices

A = 2 − 1 3 4 5 2 − 2 3 1 1 0 6 B = 1 1 0 6 3 0 − 2 − 9 2 − 1 3 4

are row-equivalent as can be seen from

2 − 1 3 4 5 2 − 2 3 1 1 0 6 → R 1 ↔ R 3 1 1 0 6 5 2 − 2 3 2 − 1 3 4 → − 2 R 1 + R 2 1 1 0 6 3 0 − 2 − 9 2 − 1 3 4

We can also say that any pair of these three matrices are row-equivalent. ⊠

Notice that each of the three row operations is reversible ( Exercise RREF.T10), so we do not have to be careful about the distinction between “ A is row-equivalent to B ” and “ B is row-equivalent to A .” ( Exercise RREF.T11) The preceding definitions are designed to make the following theorem possible. It says that row-equivalent matrices represent systems of linear equations that have identical solution sets.

Theorem  REMES
Row-Equivalent Matrices represent Equivalent Systems
Suppose that A and B are row-equivalent augmented matrices. Then the systems of linear equations that they represent are equivalent systems. □

Proof   If we perform a single row operation on an augmented matrix, it will have the same effect as if we did the analogous equation operation on the corresponding system of equations. By exactly the same methods as we used in the proof of Theorem EOPSS we can see that each of these row operations will preserve the set of solutions for the corresponding system of equations. ■

So at this point, our strategy is to begin with a system of equations, represent it by an augmented matrix, perform row operations (which will preserve solutions for the corresponding systems) to get a “simpler” augmented matrix, convert back to a “simpler” system of equations and then solve that system, knowing that its solutions are those of the original system. Here’s a rehash of Example US as an exercise in using our new tools.

Example  USR
Three equations, one solution, reprised
We solve the following system using augmented matrices and row operations. This is the same system of equations solved in Example US using equation operations.

x 1 + 2 x 2 + 2 x 3 = 4 x 1 + 3 x 2 + 3 x 3 = 5 2 x 1 + 6 x 2 + 5 x 3 = 6

Form the augmented matrix,

A = 1 2 2 4 1 3 3 5 2 6 5 6

and apply row operations,

→ − 1 R 1 + R 2 1 2 2 4 0 1 1 1 2 6 5 6 → − 2 R 1 + R 3 1 2 2 4 0 1 1 1 0 2 1 − 2 → − 2 R 2 + R 3 1 2 2 4 0 1 1 1 0 0 − 1 − 4 → − 1 R 3 1 2 2 4 0 1 1 1 0 0 1 4

So the matrix

B = 1 2 2 4 0 1 1 1 0 0 1 4

is row equivalent to A and by Theorem REMES the system of equations below has the same solution set as the original system of equations.

x 1 + 2 x 2 + 2 x 3 = 4 x 2 + x 3 = 1 x 3 = 4

Solving this “simpler” system is straightforward and is identical to the process in Example US. ⊠

Subsection RREF: Reduced Row-Echelon Form

The preceding example amply illustrates the definitions and theorems we have seen so far. But it still leaves two questions unanswered. Exactly what is this “simpler” form for a matrix, and just how do we get it? Here’s the answer to the first question, a definition of reduced row-echelon form.

Definition  RREF
Reduced Row-Echelon Form
A matrix is in reduced row-echelon form if it meets all of the following conditions:

A row of only zero entries will be called a zero row and the leftmost nonzero entry of a nonzero row will be called a leading 1. The number of nonzero rows will be denoted by r .

A column containing a leading 1 will be called a pivot column. The set of column indices for all of the pivot columns will be denoted by D = d 1 , d 2 , d 3 , … , d r where d 1 < d 2 < d 3 < ⋯ < d r , while the columns that are not pivot columns will be denoted as F = f 1 , f 2 , f 3 , … , f n − r where f 1 < f 2 < f 3 < ⋯ < f n − r .

(This definition contains Notation RREFA.) △

The principal feature of reduced row-echelon form is the pattern of leading 1’s guaranteed by conditions (2) and (4), reminiscent of a flight of geese, or steps in a staircase, or water cascading down a mountain stream.

There are a number of new terms and notation introduced in this definition, which should make you suspect that this is an important definition. Given all there is to digest here, we will mostly save the use of D and F until Section TSS. However, one important point to make here is that all of these terms and notation apply to a matrix. Sometimes we will employ these terms and sets for an augmented matrix, and other times it might be a coefficient matrix. So always give some thought to exactly which type of matrix you are analyzing.

Example  RREF
A matrix in reduced row-echelon form
The matrix C is in reduced row-echelon form.

C = 1 − 3 0 6 0 0 − 5 9 0 0 0 0 1 0 3 − 7 0 0 0 0 0 1 7 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

This matrix has two zero rows and three leading 1’s. So r = 3 . Columns 1, 5, and 6 are pivot columns, so D = 1 , 5 , 6 and then F = 2 , 3 , 4 , 7 , 8 . ⊠

Example  NRREF
A matrix not in reduced row-echelon form
The matrix E is not in reduced row-echelon form, as it fails each of the four requirements once.

E = 1 0 − 3 0 6 0 7 − 5 9 0 0 0 5 0 1 0 3 − 7 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 − 4 2 0 0 0 0 0 0 1 7 3 0 0 0 0 0 0 0 0 0

⊠

Our next theorem has a “constructive” proof. Learn about the meaning of this term in Technique C. 

Theorem  REMEF
Row-Equivalent Matrix in Echelon Form
Suppose A is a matrix. Then there is a matrix B so that

□

Proof   Suppose that A has m rows and n columns. We will describe a process for converting A into B via row operations. This procedure is known as Gauss–Jordan elimination. Tracing through this procedure will be easier if you recognize that i refers to a row that is being converted, j refers to a column that is being converted, and r keeps track of the number of nonzero rows. Here we go.

The result of this procedure is that the matrix A is converted to a matrix in reduced row-echelon form, which we will refer to as B . We need to now prove this claim by showing that the converted matrix has the requisite properties of Definition RREF. First, the matrix is only converted through row operations (Step  6, Step  7, Step  8), so A and B are row-equivalent ( Definition REM).

It is a bit more work to be certain that B is in reduced row-echelon form. We claim that as we begin Step  2, the first j columns of the matrix are in reduced row-echelon form with r nonzero rows. Certainly this is true at the start when j = 0 , since the matrix has no columns and so vacuously meets the conditions of Definition RREF with r = 0 nonzero rows.

In Step  2 we increase j by 1 and begin to work with the next column. There are two possible outcomes for Step  3. Suppose that every entry of column j in rows r + 1 through m is zero. Then with no changes we recognize that the first j columns of the matrix has its first r rows still in reduced-row echelon form, with the final m − r rows still all zero.

Suppose instead that the entry in row i of column j is nonzero. Notice that since r + 1 ≤ i ≤ m , we know the first j − 1 entries of this row are all zero. Now, in Step  5 we increase r by 1, and then embark on building a new nonzero row. In Step  6 we swap row r and row i . In the first j columns, the first r − 1 rows remain in reduced row-echelon form after the swap. In Step  7 we multiply row r by a nonzero scalar, creating a 1 in the entry in column j of row i , and not changing any other rows. This new leading 1 is the first nonzero entry in its row, and is located to the right of all the leading 1’s in the preceding r − 1 rows. With Step  8 we insure that every entry in the column with this new leading 1 is now zero, as required for reduced row-echelon form. Also, rows r + 1 through m are now all zeros in the first j columns, so we now only have one new nonzero row, consistent with our increase of r by one. Furthermore, since the first j − 1 entries of row r are zero, the employment of the third row operation does not destroy any of the necessary features of rows 1 through r − 1 and rows r + 1 through m , in columns 1 through j − 1 .

So at this stage, the first j columns of the matrix are in reduced row-echelon form. When Step  2 finally increases j to n + 1 , then the procedure is completed and the full n columns of the matrix are in reduced row-echelon form, with the value of r correctly recording the number of nonzero rows. ■

The procedure given in the proof of Theorem REMEF can be more precisely described using a pseudo-code version of a computer program, as follows:

Notice that as a practical matter the “and” used in the conditional statement of the while statement should be of the “short-circuit” variety so that the array access that follows is not out-of-bounds.

So now we can put it all together. Begin with a system of linear equations ( Definition SLE), and represent the system by its augmented matrix ( Definition AM). Use row operations ( Definition RO) to convert this matrix into reduced row-echelon form ( Definition RREF), using the procedure outlined in the proof of Theorem REMEF. Theorem REMEF also tells us we can always accomplish this, and that the result is row-equivalent ( Definition REM) to the original augmented matrix. Since the matrix in reduced-row echelon form has the same solution set, we can analyze the row-reduced version instead of the original matrix, viewing it as the augmented matrix of a different system of equations. The beauty of augmented matrices in reduced row-echelon form is that the solution sets to their corresponding systems can be easily determined, as we will see in the next few examples and in the next section.

We will see through the course that almost every interesting property of a matrix can be discerned by looking at a row-equivalent matrix in reduced row-echelon form. For this reason it is important to know that the matrix B guaranteed to exist by Theorem REMEF is also unique.

Two proof techniques are applicable to the proof. First, head out and read two proof techniques: Technique CD and Technique U.    

Theorem  RREFU
Reduced Row-Echelon Form is Unique
Suppose that A is an m × n matrix and that B and C are m × n matrices that are row-equivalent to A and in reduced row-echelon form. Then B = C . □

Proof   We need to begin with no assumptions about any relationships between B and C , other than they are both in reduced row-echelon form, and they are both row-equivalent to A .

If B and C are both row-equivalent to A , then they are row-equivalent to each other. Repeated row operations on a matrix combine the rows with each other using operations that are linear, and are identical in each column. A key observation for this proof is that each individual row of B is linearly related to the rows of C . This relationship is different for each row of B , but once we fix a row, the relationship is the same across columns. More precisely, there are scalars δ i k , 1 ≤ i , k ≤ m such that for any 1 ≤ i ≤ m , 1 ≤ j ≤ n ,

B i j = ∑ k = 1 m δ i k C k j

You should read this as saying that an entry of row i of B (in column j ) is a linear function of the entries of all the rows of C that are also in column j , and the scalars ( δ i k ) depend on which row of B we are considering (the i subscript on δ i k ), but are the same for every column (no dependence on j in δ i k ). This idea may be complicated now, but will feel more familiar once we discuss “linear combinations” (Definition LCCV) and moreso when we discuss “row spaces” ( Definition RSM). For now, spend some time carefully working Exercise RREF.M40, which is designed to illustrate the origins of this expression. This completes our exploitation of the row-equivalence of B and C .

We now repeatedly exploit the fact that B and C are in reduced row-echelon form. Recall that a pivot column is all zeros, except a single one. More carefully, if R is a matrix in reduced row-echelon form, and d ℓ is the index of a pivot column, then R k d ℓ = 1 precisely when k = ℓ and is otherwise zero. Notice also that any entry of R that is both below the entry in row ℓ and to the left of column d ℓ is also zero (with below and left understood to include equality). In other words, look at examples of matrices in reduced row-echelon form and choose a leading 1 (with a box around it). The rest of the column is also zeros, and the lower left “quadrant” of the matrix that begins here is totally zeros.

Assuming no relationship about the form of B and C , let B have r nonzero rows and denote the pivot columns as D = d 1 , d 2 , d 3 , … , d r . For C let r ′ denote the number of nonzero rows and denote the pivot columns as D ′ = d ′ 1 , d ′ 2 , d ′ 3 , … , d ′ r ′ ( Notation RREFA). There are four steps in the proof, and the first three are about showing that B and C have the same number of pivot columns, in the same places. In other words, the “primed” symbols are a necessary fiction.

First Step. Suppose that d 1 < d 1 ′ . Then

1 = B 1 d 1    Definition RREF = ∑ k = 1 m δ 1 k C k d 1 = ∑ k = 1 m δ 1 k ( 0 ) d 1 < d 1 ′ = 0

The entries of C are all zero since they are left and below of the leading 1 in row 1 and column d 1 ′ of C . This is a contradiction, so we know that d 1 ≥ d 1 ′ . By an entirely similar argument, reversing the roles of B and C , we could conclude that d 1 ≤ d 1 ′ . Together this means that d 1 = d 1 ′ .

Second Step. Suppose that we have determined that d 1 = d 1 ′ , d 2 = d 2 ′ , d 3 = d 3 ′ , …, d p = d p ′ . Let’s now show that d p + 1 = d p + 1 ′ . Working towards a contradiction, suppose that d p + 1 < d p + 1 ′ . For 1 ≤ ℓ ≤ p ,

0 = B p + 1 , d ℓ   Definition RREF = ∑ k = 1 m δ p + 1 , k C k d ℓ = ∑ k = 1 m δ p + 1 , k C k d ℓ ′ = δ p + 1 , ℓ C ℓ d ℓ ′ + ∑ k = 1 k ≠ ℓ m δ p + 1 , k C k d ℓ ′   Property CACN = δ p + 1 , ℓ ( 1 ) + ∑ k = 1 k ≠ ℓ m δ p + 1 , k ( 0 )   Definition RREF = δ p + 1 , ℓ

Now,

1 = B p + 1 , d p + 1    Definition RREF = ∑ k = 1 m δ p + 1 , k C k d p + 1 = ∑ k = 1 p δ p + 1 , k C k d p + 1 + ∑ k = p + 1 m δ p + 1 , k C k d p + 1    Property AACN = ∑ k = 1 p ( 0 ) C k d p + 1 + ∑ k = p + 1 m δ p + 1 , k C k d p + 1 = ∑ k = p + 1 m δ p + 1 , k C k d p + 1 = ∑ k = p + 1 m δ p + 1 , k ( 0 ) d p + 1 < d p + 1 ′ = 0

This contradiction shows that d p + 1 ≥ d p + 1 ′ . By an entirely similar argument, we could conclude that d p + 1 ≤ d p + 1 ′ , and therefore d p + 1 = d p + 1 ′ .

Third Step. Now we establish that r = r ′ . Suppose that r ′ < r . By the arguments above, we know that d 1 = d 1 ′ , d 2 = d 2 ′ , d 3 = d 3 ′ , …, d r ′ = d r ′ ′ . For 1 ≤ ℓ ≤ r ′ < r ,

0 = B r d ℓ   Definition RREF = ∑ k = 1 m δ r k C k d ℓ = ∑ k = 1 r ′ δ r k C k d ℓ + ∑ k = r ′ + 1 m δ r k C k d ℓ   Property AACN = ∑ k = 1 r ′ δ r k C k d ℓ + ∑ k = r ′ + 1 m δ r k ( 0 )   Property AACN = ∑ k = 1 r ′ δ r k C k d ℓ = ∑ k = 1 r ′ δ r k C k d ℓ ′ = δ r ℓ C ℓ d ℓ ′ + ∑ k = 1 k ≠ ℓ r ′ δ r k C k d ℓ ′   Property CACN = δ r ℓ ( 1 ) + ∑ k = 1 k ≠ ℓ r ′ δ r k ( 0 )   Definition RREF = δ r ℓ

Now examine the entries of row r of B ,

B r j = ∑ k = 1 m δ r k C k j = ∑ k = 1 r ′ δ r k C k j + ∑ k = r ′ + 1 m δ r k C k j   Property CACN = ∑ k = 1 r ′ δ r k C k j + ∑ k = r ′ + 1 m δ r k ( 0 )   Definition RREF = ∑ k = 1 r ′ δ r k C k j = ∑ k = 1 r ′ ( 0 ) C k j = 0

So row r is a totally zero row, contradicting that this should be the bottommost nonzero row of B . So r ′ ≥ r . By an entirely similar argument, reversing the roles of B and C , we would conclude that r ′ ≤ r and therefore r = r ′ . Thus, combining the first three steps we can say that D = D ′ . In other words, B and C have the same pivot columns, in the same locations.

Fourth Step. In this final step, we will not argue by contradiction. Our intent is to determine the values of the δ i j . Notice that we can use the values of the d i interchangeably for B and C . Here we go,

1 = B i d i   Definition RREF = ∑ k = 1 m δ i k C k d i = δ i i C i d i + ∑ k = 1 k ≠ i m δ i k C k d i   Property CACN = δ i i ( 1 ) + ∑ k = 1 k ≠ i m δ i k ( 0 )   Definition RREF = δ i i

and for ℓ ≠ i

0 = B i d ℓ   Definition RREF = ∑ k = 1 m δ i k C k d ℓ = δ i ℓ C ℓ d ℓ + ∑ k = 1 k ≠ ℓ m δ i k C k d ℓ   Property CACN = δ i ℓ ( 1 ) + ∑ k = 1 k ≠ ℓ m δ i k ( 0 )   Definition RREF = δ i ℓ

Finally, having determined the values of the δ i j , we can show that B = C . For 1 ≤ i ≤ m , 1 ≤ j ≤ n ,

B i j = ∑ k = 1 m δ i k C k j = δ i i C i j + ∑ k = 1 k ≠ i m δ i k C k j   Property CACN = ( 1 ) C i j + ∑ k = 1 k ≠ i m ( 0 ) C k j = C i j

So B and C have equal values in every entry, and so are the same matrix. ■

We will now run through some examples of using these definitions and theorems to solve some systems of equations. From now on, when we have a matrix in reduced row-echelon form, we will mark the leading 1’s with a small box. In your work, you can box ’em, circle ’em or write ’em in a different color — just identify ’em somehow. This device will prove very useful later and is a very good habit to start developing right now.

Example  SAB
Solutions for Archetype B
Let’s find the solutions to the following system of equations,

− 7 x 1 − 6 x 2 − 1 2 x 3 = − 3 3 5 x 1 + 5 x 2 + 7 x 3 = 2 4 x 1 + 4 x 3 = 5

First, form the augmented matrix,

− 7 − 6 − 1 2 − 3 3 5 5 7 2 4 1 0 4 5

and work to reduced row-echelon form, first with j = 1 ,

→ R 1 ↔ R 3 1 0 4 5 5 5 7 2 4 − 7 − 6 − 1 2 − 3 3 → − 5 R 1 + R 2 1 0 4 5 0 5 − 1 3 − 1 − 7 − 6 − 1 2 − 3 3 → 7 R 1 + R 3 1 0 4 5 0 5 − 1 3 − 1 0 − 6 1 6 2  Now, with  j = 2 , → 1 5 R 2 1 0 4 5 0 1 − 1 3 5 − 1 5 0 − 6 1 6 2 → 6 R 2 + R 3 1 0 4 5 0 1 − 1 3 5 − 1 5 0 0 2 5 4 5  And finally, with  j = 3 , → 5 2 R 3 1 0 4 5 0 1 − 1 3 5 − 1 5 0 0 1 2 → 1 3 5 R 3 + R 2 1 0 4 5 0 1  0 5 0 0 1 2 → − 4 R 3 + R 1 1 0 0 − 3 0 1  0 5 0 0 1 2

This is now the augmented matrix of a very simple system of equations, namely x 1 = − 3 , x 2 = 5 , x 3 = 2 , which has an obvious solution. Furthermore, we can see that this is the only solution to this system, so we have determined the entire solution set,

S = − 3 5 2

You might compare this example with the procedure we used in Example US. ⊠

Archetypes A and B are meant to contrast each other in many respects. So let’s solve Archetype A now.

Example  SAA
Solutions for Archetype A
Let’s find the solutions to the following system of equations,

x 1 − x 2 + 2 x 3 = 1 2 x 1 + x 2 + x 3 = 8 x 1 + x 2 = 5

First, form the augmented matrix,

1 − 1 2 1 2 1 1 8 1 1 0 5

and work to reduced row-echelon form, first with j = 1 ,

→ − 2 R 1 + R 2 1 − 1 2 1 0 3 − 3 6 1 1 0 5 → − 1 R 1 + R 3 1 − 1 2 1 0 3 − 3 6 0 2 − 2 4  Now, with  j = 2 , → 1 3 R 2 1 − 1 2 1 0 1 − 1 2 0 2 − 2 4 → 1 R 2 + R 1 1 0 1 3 0 1 − 1 2 0 2 − 2 4 → − 2 R 2 + R 3 1 0 1 3 0 1  − 1 2 0 0 0 0

The system of equations represented by this augmented matrix needs to be considered a bit differently than that for Archetype B. First, the last row of the matrix is the equation 0 = 0 , which is always true, so it imposes no restrictions on our possible solutions and therefore we can safely ignore it as we analyze the other two equations. These equations are,

x 1 + x 3 = 3 x 2 − x 3 = 2 .

While this system is fairly easy to solve, it also appears to have a multitude of solutions. For example, choose x 3 = 1 and see that then x 1 = 2 and x 2 = 3 will together form a solution. Or choose x 3 = 0 , and then discover that x 1 = 3 and x 2 = 2 lead to a solution. Try it yourself: pick any value of x 3 you please, and figure out what x 1 and x 2 should be to make the first and second equations (respectively) true. We’ll wait while you do that. Because of this behavior, we say that x 3 is a “free” or “independent” variable. But why do we vary x 3 and not some other variable? For now, notice that the third column of the augmented matrix does not have any leading 1’s in its column. With this idea, we can rearrange the two equations, solving each for the variable that corresponds to the leading 1 in that row.

x 1 = 3 − x 3 x 2 = 2 + x 3

To write the set of solution vectors in set notation, we have

S = 3 − x 3 2 + x 3 x 3 x 3 ∈ ℂ

We’ll learn more in the next section about systems with infinitely many solutions and how to express their solution sets. Right now, you might look back at Example IS. ⊠

Example  SAE
Solutions for Archetype E
Let’s find the solutions to the following system of equations,

2 x 1 + x 2 + 7 x 3 − 7 x 4 = 2 − 3 x 1 + 4 x 2 − 5 x 3 − 6 x 4 = 3 x 1 + x 2 + 4 x 3 − 5 x 4 = 2

First, form the augmented matrix,

2 1 7 − 7 2 − 3 4 − 5 − 6 3 1 1 4 − 5 2

and work to reduced row-echelon form, first with j = 1 ,

→ R 1 ↔ R 3 1 1 4 − 5 2 − 3 4 − 5 − 6 3 2 1 7 − 7 2 → 3 R 1 + R 2 1 1 4 − 5 2 0 7 7 − 2 1 9 2 1 7 − 7 2 → − 2 R 1 + R 3 1 1 4 − 5 2 0 7 7 − 2 1 9 0 − 1 − 1 3 − 2  Now, with  j = 2 , → R 2 ↔ R 3 1 1 4 − 5 2 0 − 1 − 1 3 − 2 0 7 7 − 2 1 9 → − 1 R 2 1 1 4 − 5 2 0 1 1 − 3 2 0 7 7 − 2 1 9 → − 1 R 2 + R 1 1 0 3 − 2 0 0 1 1 − 3 2 0 7 7 − 2 1 9 → − 7 R 2 + R 3 1 0 3 − 2 0 0 1  1 − 3 2 0 0 0 0 − 5  And finally, with  j = 4 , → − 1 5 R 3 1 0 3 − 2 0 0 1  1 − 3 2 0 0 0 0 1 → − 2 R 3 + R 2 1 0 3 − 2 0 0 1  1 − 3 0 0 0 0 0 1

Let’s analyze the equations in the system represented by this augmented matrix. The third equation will read 0 = 1 . This is patently false, all the time. No choice of values for our variables will ever make it true. We’re done. Since we cannot even make the last equation true, we have no hope of making all of the equations simultaneously true. So this system has no solutions, and its solution set is the empty set, ∅ = ( Definition ES).

Notice that we could have reached this conclusion sooner. After performing the row operation − 7 R 2 + R 3 , we can see that the third equation reads 0 = − 5 , a false statement. Since the system represented by this matrix has no solutions, none of the systems represented has any solutions. However, for this example, we have chosen to bring the matrix fully to reduced row-echelon form for the practice. ⊠

These three examples ( Example SAB, Example SAA, Example SAE) illustrate the full range of possibilities for a system of linear equations — no solutions, one solution, or infinitely many solutions. In the next section we’ll examine these three scenarios more closely.

Definition  RR
Row-Reducing
To row-reduce the matrix A means to apply row operations to A and arrive at a row-equivalent matrix B in reduced row-echelon form. △

So the term row-reduce is used as a verb. Theorem REMEF tells us that this process will always be successful and Theorem RREFU tells us that the result will be unambiguous. Typically, the analysis of A will proceed by analyzing B and applying theorems whose hypotheses include the row-equivalence of A and B .

After some practice by hand, you will want to use your favorite computing device to do the computations required to bring a matrix to reduced row-echelon form (Exercise RREF.C30).  See: Computation RR.MMA Computation RR.TI86 Computation RR.TI83 Computation RR.SAGE

Subsection READ: Reading Questions

Subsection EXC: Exercises

C05 Each archetype below is a system of equations. Form the augmented matrix of the system of equations, convert the matrix to reduced row-echelon form by using equation operations and then describe the solution set of the original system of equations.
Archetype A
Archetype B
Archetype C
Archetype D
Archetype E
Archetype F
Archetype G
Archetype H
Archetype I
Archetype J  
Contributed by  Robert Beezer

For problems C10–C19, find all solutions to the system of linear equations. Use your favorite computing device to row-reduce the augmented matrices for the systems, and write the solutions as a set, using correct set notation.
C10

2 x 1 − 3 x 2 + x 3 + 7 x 4 = 1 4 2 x 1 + 8 x 2 − 4 x 3 + 5 x 4 = − 1 x 1 + 3 x 2 − 3 x 3 = 4 − 5 x 1 + 2 x 2 + 3 x 3 + 4 x 4 = − 1 9

Contributed by  Robert Beezer Solution [ 125]

C11

3 x 1 + 4 x 2 − x 3 + 2 x 4 = 6 x 1 − 2 x 2 + 3 x 3 + x 4 = 2 1 0 x 2 − 1 0 x 3 − x 4 = 1

Contributed by  Robert Beezer Solution [ 126]

C12

2 x 1 + 4 x 2 + 5 x 3 + 7 x 4 = − 2 6 x 1 + 2 x 2 + x 3 − x 4 = − 4 − 2 x 1 − 4 x 2 + x 3 + 1 1 x 4 = − 1 0

Contributed by  Robert Beezer Solution [ 126]

C13

x 1 + 2 x 2 + 8 x 3 − 7 x 4 = − 2 3 x 1 + 2 x 2 + 1 2 x 3 − 5 x 4 = 6 − x 1 + x 2 + x 3 − 5 x 4 = − 1 0

Contributed by  Robert Beezer Solution [ 127]

C14

2 x 1 + x 2 + 7 x 3 − 2 x 4 = 4 3 x 1 − 2 x 2 + 1 1 x 4 = 1 3 x 1 + x 2 + 5 x 3 − 3 x 4 = 1

Contributed by  Robert Beezer Solution [ 128]

C15

2 x 1 + 3 x 2 − x 3 − 9 x 4 = − 1 6 x 1 + 2 x 2 + x 3 = 0 − x 1 + 2 x 2 + 3 x 3 + 4 x 4 = 8

Contributed by  Robert Beezer Solution [ 130]

C16

2 x 1 + 3 x 2 + 1 9 x 3 − 4 x 4 = 2 x 1 + 2 x 2 + 1 2 x 3 − 3 x 4 = 1 − x 1 + 2 x 2 + 8 x 3 − 5 x 4 = 1

Contributed by  Robert Beezer Solution [ 131]

C17

− x 1 + 5 x 2 = − 8 − 2 x 1 + 5 x 2 + 5 x 3 + 2 x 4 = 9 − 3 x 1 − x 2 + 3 x 3 + x 4 = 3 7 x 1 + 6 x 2 + 5 x 3 + x 4 = 3 0

Contributed by  Robert Beezer Solution [ 132]

C18

x 1 + 2 x 2 − 4 x 3 − x 4 = 3 2 x 1 + 3 x 2 − 7 x 3 − x 5 = 3 3 x 1 + 2 x 3 − 2 x 4 + 3 x 5 = 2 2

Contributed by  Robert Beezer Solution [ 133]

C19

2 x 1 + x 2 = 6 − x 1 − x 2 = − 2 3 x 1 + 4 x 2 = 4 3 x 1 + 5 x 2 = 2

Contributed by  Robert Beezer Solution [ 135]

For problems C30–C33, row-reduce the matrix without the aid of a calculator, indicating the row operations you are using at each step using the notation of Definition RO.
C30

2 1 5 1 0 1 − 3 − 1 − 2 4 − 2 6 1 2

Contributed by  Robert Beezer Solution [ 136]

C31

1 2 − 4 − 3 − 1 − 3 − 2 1 − 7

Contributed by  Robert Beezer Solution [ 136]

C32

1 1 1 − 4 − 3 − 2 3 2 1

Contributed by  Robert Beezer Solution [ 137]

C33

1 2 − 1 − 1 2 4 − 1 4 − 1 − 2 3 5

Contributed by  Robert Beezer Solution [ 138]

M40 Consider the two 3 × 4 matrices below

B = 1 3 − 2 2 − 1 − 2 − 1 − 1 − 1 − 5 8 − 3 C = 1 2 1 2 1 1 4 0 − 1 − 1 − 4 1

(a) Row-reduce each matrix and determine that the reduced row-echelon forms of B and C are identical. From this argue that B and C are row-equivalent.

(b) In the proof of Theorem RREFU, we begin by arguing that entries of row-equivalent matrices are related by way of certain scalars and sums. In this example, we would write that entries of B from row i that are in column j are linearly related to the entries of C in column j from all three rows

B i j = δ i 1 C 1 j + δ i 2 C 2 j + δ i 3 C 3 j 1 ≤ j ≤ 4

For each 1 ≤ i ≤ 3 find the corresponding three scalars in this relationship. So your answer will be nine scalars, determined three at a time.  
Contributed by  Robert Beezer Solution [ 138]

M45 You keep a number of lizards, mice and peacocks as pets. There are a total of 108 legs and 30 tails in your menagerie. You have twice as many mice as lizards. How many of each creature do you have?  
Contributed by  Chris Black Solution [ 141]

M50 A parking lot has 66 vehicles (cars, trucks, motorcycles and bicycles) in it. There are four times as many cars as trucks. The total number of tires (4 per car or truck, 2 per motorcycle or bicycle) is 252. How many cars are there? How many bicycles?  
Contributed by  Robert Beezer Solution [ 142]

T10 Prove that each of the three row operations ( Definition RO) is reversible. More precisely, if the matrix B is obtained from A by application of a single row operation, show that there is a single row operation that will transform B back into A .  
Contributed by  Robert Beezer Solution [ 143]

T11 Suppose that A , B and C are m × n matrices. Use the definition of row-equivalence ( Definition REM) to prove the following three facts.

A relationship that satisfies these three properties is known as an equivalence relation, an important idea in the study of various algebras. This is a formal way of saying that a relationship behaves like equality, without requiring the relationship to be as strict as equality itself. We’ll see it again in Theorem SER.  
Contributed by  Robert Beezer

T12 Suppose that B is an m × n matrix in reduced row-echelon form. Build a new, likely smaller, k × ℓ matrix C as follows. Keep any collection of k adjacent rows, k ≤ m . From these rows, keep columns 1 through ℓ , ℓ ≤ n . Prove that C is in reduced row-echelon form.  
Contributed by  Robert Beezer

T13 Generalize Exercise RREF.T12 by just keeping any k rows, and not requiring the rows to be adjacent. Prove that any such matrix C is in reduced row-echelon form.  
Contributed by  Robert Beezer

Subsection SOL: Solutions

C10 Contributed by  Robert Beezer Statement [ 113]
The augmented matrix row-reduces to

1 0 0 0 1 0 1  0 0 − 3 0 0 1 0 − 4 0 0 0 1  1

This augmented matrix represents the linear system x 1 = 1 , x 2 = − 3 , x 3 = − 4 , x 4 = 1 , which clearly has only one possible solution. We can write this solution set then as

S = 1 − 3 − 4 1

C11 Contributed by  Robert Beezer Statement [ 114]
The augmented matrix row-reduces to

1 0 1 4 ∕ 5 0 0 1 − 1 − 1 ∕ 1 0 0 0 0 0 0 1

Row 3 represents the equation 0 = 1 , which is patently false, so the original system has no solutions. We can express the solution set as the empty set, ∅ = .

C12 Contributed by  Robert Beezer Statement [ 114]
The augmented matrix row-reduces to

1 2 0 − 4 2 0 0 1  3 − 6 0 0 0 0 0

In the spirit of Example SAA, we can express the infinitely many solutions of this system compactly with set notation. The key is to express certain variables in terms of others. More specifically, each pivot column number is the index of a variable that can be written in terms of the variables whose indices are non-pivot columns. Or saying the same thing: for each i in D , we can find an expression for x i in terms of the variables without their index in D . Here D = 1 , 3 , so

x 1 = 2 − 2 x 2 + 4 x 4 x 3 = − 6 − 3 x 4

As a set, we write the solutions precisely as

2 − 2 x 2 + 4 x 4 x 2 − 6 − 3 x 4 x 4 x 2 , x 4 ∈ ℂ

C13 Contributed by  Robert Beezer Statement [ 115]
The augmented matrix of the system of equations is

1 2 8 − 7 − 2 3 2 1 2 − 5 6 − 1 1 1 − 5 − 1 0

which row-reduces to

1 0 2 1 0 0 1  3 − 4 0 0 0 0 0 1

Row 3 represents the equation 0 = 1 , which is patently false, so the original system has no solutions. We can express the solution set as the empty set, ∅ = .

C14 Contributed by  Robert Beezer Statement [ 115]
The augmented matrix of the system of equations is

2 1 7 − 2 4 3 − 2 0 1 1 1 3 1 1 5 − 3 1

which row-reduces to

1 0 2 1 3 0 1  3 − 4 − 2 0 0 0 0 0

In the spirit of Example SAA, we can express the infinitely many solutions of this system compactly with set notation. The key is to express certain variables in terms of others. More specifically, each pivot column number is the index of a variable that can be written in terms of the variables whose indices are non-pivot columns. Or saying the same thing: for each i in D , we can find an expression for x i in terms of the variables without their index in D . Here D = 1 , 2 , so rearranging the equations represented by the two nonzero rows to gain expressions for the variables x 1 and x 2 yields the solution set,

S = 3 − 2 x 3 − x 4 − 2 − 3 x 3 + 4 x 4 x 3 x 4 x 3 , x 4 ∈ ℂ

C15 Contributed by  Robert Beezer Statement [ 116]
The augmented matrix of the system of equations is

2 3 − 1 − 9 − 1 6 1 2 1 0 0 − 1 2 3 4 8

which row-reduces to

1 0 0 2 3 0 1  0 − 3 − 5 0 0 1 4 7

In the spirit of Example SAA, we can express the infinitely many solutions of this system compactly with set notation. The key is to express certain variables in terms of others. More specifically, each pivot column number is the index of a variable that can be written in terms of the variables whose indices are non-pivot columns. Or saying the same thing: for each i in D , we can find an expression for x i in terms of the variables without their index in D . Here D = 1 , 2 , 3 , so rearranging the equations represented by the three nonzero rows to gain expressions for the variables x 1 , x 2 and x 3 yields the solution set,

S = 3 − 2 x 4 − 5 + 3 x 4 7 − 4 x 4 x 4 x 4 ∈ ℂ

C16 Contributed by  Robert Beezer Statement [ 116]
The augmented matrix of the system of equations is

2 3 1 9 − 4 2 1 2 1 2 − 3 1 − 1 2 8 − 5 1

which row-reduces to

1 0 2 1 0 0 1  5 − 2 0 0 0 0 0 1

Row 3 represents the equation 0 = 1 , which is patently false, so the original system has no solutions. We can express the solution set as the empty set, ∅ = .

C17 Contributed by  Robert Beezer Statement [ 117]
We row-reduce the augmented matrix of the system of equations,

− 1 5 0 0 − 8 − 2 5 5 2 9 − 3 − 1 3 1 3 7 6 5 1 3 0 →  RREF 1 0 0 0 3 0 1  0 0 − 1 0 0 1 0 2 0 0 0 1  5

This augmented matrix represents the linear system x 1 = 3 , x 2 = − 1 , x 3 = 2 , x 4 = 5 , which clearly has only one possible solution. We can write this solution set then as

S = 3 − 1 2 5

C18 Contributed by  Robert Beezer Statement [ 117]
We row-reduce the augmented matrix of the system of equations,

1 2 − 4 − 1 0 3 2 1 3 − 7 0 − 1 3 3 1 0 2 − 2 3 2 2 →  RREF 1 0 2 0 5 6 0 1  − 3 0 − 2 9 0 0 0 1 1 − 8

In the spirit of Example SAA, we can express the infinitely many solutions of this system compactly with set notation. The key is to express certain variables in terms of others. More specifically, each pivot column number is the index of a variable that can be written in terms of the variables whose indices are non-pivot columns. Or saying the same thing: for each i in D , we can find an expression for x i in terms of the variables without their index in D . Here D = 1 , 2 , 4 , so

x 1 + 2 x 3 + 5 x 5 = 6 → x 1 = 6 − 2 x 3 − 5 x 5 x 2 − 3 x 3 − 2 x 5 = 9 → x 2 = 9 + 3 x 3 + 2 x 5 x 4 + x 5 = − 8 → x 4 = − 8 − x 5

As a set, we write the solutions precisely as

S = 6 − 2 x 3 − 5 x 5 9 + 3 x 3 + 2 x 5 x 3 − 8 − x 5 x 5 x 3 , x 5 ∈ ℂ

C19 Contributed by  Robert Beezer Statement [ 118]
We form the augmented matrix of the system,

2 1 6 − 1 − 1 − 2 3 4 4 3 5 2

which row-reduces to

1 0 4 0 1  − 2 0 0 0 0 0 0

This augmented matrix represents the linear system x 1 = 4 , x 2 = − 2 , 0 = 0 , 0 = 0 , which clearly has only one possible solution. We can write this solution set then as

S = 4 − 2

C30 Contributed by  Robert Beezer Statement [ 119]

2 1 5 1 0 1 − 3 − 1 − 2 4 − 2 6 1 2 → R 1 ↔ R 2 1 − 3 − 1 − 2 2 1 5 1 0 4 − 2 6 1 2 → − 2 R 1 + R 2 1 − 3 − 1 − 2 0 7 7 1 4 4 − 2 6 1 2 → − 4 R 1 + R 3 1 − 3 − 1 − 2 0 7 7 1 4 0 1 0 1 0 2 0 → 1 7 R 2 1 − 3 − 1 − 2 0 1 1 2 0 1 0 1 0 2 0 → 3 R 2 + R 1 1 0 2 4 0 1 1 2 0 1 0 1 0 2 0 → − 1 0 R 2 + R 3 1 0 2 4 0 1  1 2 0 0 0 0

C31 Contributed by  Robert Beezer Statement [ 119]

1 2 − 4 − 3 − 1 − 3 − 2 1 − 7 → 3 R 1 + R 2 1 2 − 4 0 5 − 1 5 − 2 1 − 7 → 2 R 1 + R 3 1 2 − 4 0 5 − 1 5 0 5 − 1 5 → 1 5 R 2 1 2 − 4 0 1 − 3 0 5 − 1 5 → − 2 R 2 + R 1 1 0 2 0 1 − 3 0 5 − 1 5 → − 5 R 2 + R 3 1 0 2 0 1  − 3 0 0 0

C32 Contributed by  Robert Beezer Statement [ 120]
Following the algorithm of Theorem REMEF, and working to create pivot columns from left to right, we have

1 1 1 − 4 − 3 − 2 3 2 1 → 4 R 1 + R 2 1 1 1 0 1 2 3 2 1 → − 3 R 1 + R 3 1 1 1 0 1 2 0 − 1 − 2 → − 1 R 2 + R 1 1 0 − 1 0 1 2 0 − 1 − 2 → 1 R 2 + R 3 1 0 − 1 0 1  2 0 0 0

C33 Contributed by  Robert Beezer Statement [ 120]
Following the algorithm of Theorem REMEF, and working to create pivot columns from left to right, we have

1 2 − 1 − 1 2 4 − 1 4 − 1 − 2 3 5 → − 2 R 1 + R 2 1 2 − 1 − 1 0 0 1 6 − 1 − 2 3 5 → 1 R 1 + R 3 1 2 − 1 − 1 0 0 1 6 0 0 2 4 → 1 R 2 + R 1 1 2 0 5 0 0 1 6 0 0 2 4 → − 2 R 2 + R 3 1 2 0 5 0 0 1  6 0 0 0 − 8 → − 1 8 R 3 1 2 0 5 0 0 1  6 0 0 0 1 → − 6 R 3 + R 2 1 2 0 5 0 0 1  0 0 0 0 1 → − 5 R 3 + R 1 1 2 0 0 0 0 1  0 0 0 0 1

M40 Contributed by  Robert Beezer Statement [ 120]
(a) Let R be the common reduced row-echelon form of B and C . A sequence of row operations converts B to R and a second sequence of row operations converts C to R . If we “reverse” the second sequence’s order, and reverse each individual row operation (see Exercise RREF.T10) then we can begin with B , convert to R with the first sequence, and then convert to C with the reversed sequence. Satisfying Definition REM we can say B and C are row-equivalent matrices.

(b) We will work this carefully for the first row of B and just give the solution for the next two rows. For row 1 of B take i = 1 and we have

B 1 j = δ 1 1 C 1 j + δ 1 2 C 2 j + δ 1 3 C 3 j 1 ≤ j ≤ 4

If we substitute the four values for j we arrive at four linear equations in the three unknowns δ 1 1 , δ 1 2 , δ 1 3 ,

( j = 1 ) B 1 1 = δ 1 1 C 1 1 + δ 1 2 C 2 1 + δ 1 3 C 3 1 ⇒ 1 = δ 1 1 ( 1 ) + δ 1 2 ( 1 ) + δ 1 3 ( − 1 ) ( j = 2 ) B 1 2 = δ 1 1 C 1 2 + δ 1 2 C 2 2 + δ 1 3 C 3 2 ⇒ 3 = δ 1 1 ( 2 ) + δ 1 2 ( 1 ) + δ 1 3 ( − 1 ) ( j = 3 ) B 1 3 = δ 1 1 C 1 3 + δ 1 2 C 2 3 + δ 1 3 C 3 3 ⇒ − 2 = δ 1 1 ( 1 ) + δ 1 2 ( 4 ) + δ 1 3 ( − 4 ) ( j = 4 ) B 1 4 = δ 1 1 C 1 4 + δ 1 2 C 2 4 + δ 1 3 C 3 4 ⇒ 2 = δ 1 1 ( 2 ) + δ 1 2 ( 0 ) + δ 1 3 ( 1 )

We form the augmented matrix of this system and row-reduce to find the solutions,

1 1 − 1 1 2 1 − 1 3 1 4 − 4 − 2 2 0 1 2 →  RREF 1 0 0 2 0 1  0 − 3 0 0 1 − 2 0 0 0 0

So the unique solution is δ 1 1 = 2 , δ 1 2 = − 3 , δ 1 3 = − 2 . Entirely similar work will lead you to

δ 2 1 = − 1 δ 2 2 = 1 δ 2 3 = 1  and δ 3 1 = − 4 δ 3 2 = 8 δ 3 3 = 5

M45 Contributed by  Chris Black Statement [ 122]
Let l , m , p denote the number of lizards, mice and peacocks. Then the statements from the problem yield the equations:

4 l + 4 m + 2 p = 1 0 8 l + m + p = 3 0 2 l − m = 0

We form the augmented matrix for this system and row-reduce

4 4 2 1 0 8 1 1 1 3 0 2 − 1 0 0 →  RREF 1 0 0 8 0 1  0 1 6 0 0 1 6

From the row-reduced matrix, we see that we have an equivalent system l = 8 , m = 1 6 , and p = 6 , which means that you have 8 lizards, 16 mice and 6 peacocks.

M50 Contributed by  Robert Beezer Statement [ 122]
Let c , t , m , b denote the number of cars, trucks, motorcycles, and bicycles. Then the statements from the problem yield the equations:

c + t + m + b = 6 6 c − 4 t = 0 4 c + 4 t + 2 m + 2 b = 2 5 2

We form the augmented matrix for this system and row-reduce

1 1 1 1 6 6 1 − 4 0 0 0 4 4 2 2 2 5 2 →  RREF 1 0 0 0 4 8 0 1  0 0 1 2 0 0 1 1 6

The first row of the matrix represents the equation c = 4 8 , so there are 48 cars. The second row of the matrix represents the equation t = 1 2 , so there are 12 trucks. The third row of the matrix represents the equation m + b = 6 so there are anywhere from 0 to 6 bicycles. We can also say that b is a free variable, but the context of the problem limits it to 7 integer values since you cannot have a negative number of motorcycles.

T10 Contributed by  Robert Beezer Statement [ 122]
If we can reverse each row operation individually, then we can reverse a sequence of row operations. The operations that reverse each operation are listed below, using our shorthand notation. Notice how requiring the scalar α to be non-zero makes the second operation reversible.

R i ↔ R j R i ↔ R j α R i , α ≠ 0 1 α R i α R i + R j − α R i + R j

Section HSE  Homogeneous Systems of Equations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we specialize to systems of linear equations where every equation has a zero as its constant term. Along the way, we will begin to express more and more ideas in the language of matrices and begin a move away from writing out whole systems of equations. The ideas initiated in this section will carry through the remainder of the course.

Parent topic: Chapter SLE  Systems of Linear Equations

Previous topic: Section RREF  Reduced Row-Echelon Form

Next topic: Section NM  Nonsingular Matrices

Subsection SHS: Solutions of Homogeneous Systems

As usual, we begin with a definition.

Definition  HS
Homogeneous System
A system of linear equations, ℒ S A , b is homogeneous if the vector of constants is the zero vector, in other words, b = 0 . △

Example  AHSAC
Archetype C as a homogeneous system
For each archetype that is a system of equations, we have formulated a similar, yet different, homogeneous system of equations by replacing each equation’s constant term with a zero. To wit, for Archetype C, we can convert the original system of equations into the homogeneous system,

2 x 1 − 3 x 2 + x 3 − 6 x 4 = 0 4 x 1 + x 2 + 2 x 3 + 9 x 4 = 0 3 x 1 + x 2 + x 3 + 8 x 4 = 0

Can you quickly find a solution to this system without row-reducing the augmented matrix? ⊠

As you might have discovered by studying Example AHSAC, setting each variable to zero will always be a solution of a homogeneous system. This is the substance of the following theorem.

Theorem  HSC
Homogeneous Systems are Consistent
Suppose that a system of linear equations is homogeneous. Then the system is consistent. □

Proof   Set each variable of the system to zero. When substituting these values into each equation, the left-hand side evaluates to zero, no matter what the coefficients are. Since a homogeneous system has zero on the right-hand side of each equation as the constant term, each equation is true. With one demonstrated solution, we can call the system consistent. ■

Since this solution is so obvious, we now define it as the trivial solution.

Definition  TSHSE
Trivial Solution to Homogeneous Systems of Equations
Suppose a homogeneous system of linear equations has n variables. The solution x 1 = 0 , x 2 = 0 ,…, x n = 0 (i.e.  x = 0 ) is called the trivial solution. △

Here are three typical examples, which we will reference throughout this section. Work through the row operations as we bring each to reduced row-echelon form. Also notice what is similar in each example, and what differs.

Example  HUSAB
Homogeneous, unique solution, Archetype B
Archetype B can be converted to the homogeneous system,

− 1 1 x 1 + 2 x 2 − 1 4 x 3 = 0 2 3 x 1 − 6 x 2 + 3 3 x 3 = 0 1 4 x 1 − 2 x 2 + 1 7 x 3 = 0

whose augmented matrix row-reduces to

1 0 0 0 0 1  0 0 0 0 1 0

By Theorem HSC, the system is consistent, and so the computation n − r = 3 − 3 = 0 means the solution set contains just a single solution. Then, this lone solution must be the trivial solution. ⊠

Example  HISAA
Homogeneous, infinite solutions, Archetype A
Archetype A can be converted to the homogeneous system,

x 1 − x 2 + 2 x 3 = 0 2 x 1 + x 2 + x 3 = 0 x 1 + x 2 = 0

whose augmented matrix row-reduces to

1 0 1 0 0 1  − 1 0 0 0 0 0

By Theorem HSC, the system is consistent, and so the computation n − r = 3 − 2 = 1 means the solution set contains one free variable by Theorem FVCS, and hence has infinitely many solutions. We can describe this solution set using the free variable x 3 ,

S = x 1 x 2 x 3 x 1 = − x 3 , x 2 = x 3 = − x 3 x 3 x 3 x 3 ∈ ℂ

Geometrically, these are points in three dimensions that lie on a line through the origin. ⊠

Example  HISAD
Homogeneous, infinite solutions, Archetype D
Archetype D (and identically, Archetype E) can be converted to the homogeneous system,

2 x 1 + x 2 + 7 x 3 − 7 x 4 = 0 − 3 x 1 + 4 x 2 − 5 x 3 − 6 x 4 = 0 x 1 + x 2 + 4 x 3 − 5 x 4 = 0

whose augmented matrix row-reduces to

1 0 3 − 2 0 0 1  1 − 3 0 0 0 0 0 0

By Theorem HSC, the system is consistent, and so the computation n − r = 4 − 2 = 2 means the solution set contains two free variables by Theorem FVCS, and hence has infinitely many solutions. We can describe this solution set using the free variables x 3 and x 4 ,

S = x 1 x 2 x 3 x 4 x 1 = − 3 x 3 + 2 x 4 , x 2 = − x 3 + 3 x 4 = − 3 x 3 + 2 x 4 − x 3 + 3 x 4 x 3 x 4 x 3 , x 4 ∈ ℂ

⊠

After working through these examples, you might perform the same computations for the slightly larger example, Archetype J.

Notice that when we do row operations on the augmented matrix of a homogeneous system of linear equations the last column of the matrix is all zeros. Any one of the three allowable row operations will convert zeros to zeros and thus, the final column of the matrix in reduced row-echelon form will also be all zeros. So in this case, we may be as likely to reference only the coefficient matrix and presume that we remember that the final column begins with zeros, and after any number of row operations is still zero.

Example HISAD suggests the following theorem.

Theorem  HMVEI
Homogeneous, More Variables than Equations, Infinite solutions
Suppose that a homogeneous system of linear equations has m equations and n variables with n > m . Then the system has infinitely many solutions. □

Proof   We are assuming the system is homogeneous, so Theorem HSC says it is consistent. Then the hypothesis that n > m , together with Theorem CMVEI, gives infinitely many solutions. ■

Example HUSAB and Example HISAA are concerned with homogeneous systems where n = m and expose a fundamental distinction between the two examples. One has a unique solution, while the other has infinitely many. These are exactly the only two possibilities for a homogeneous system and illustrate that each is possible (unlike the case when n > m where Theorem HMVEI tells us that there is only one possibility for a homogeneous system).

Subsection NSM: Null Space of a Matrix

The set of solutions to a homogeneous system (which by Theorem HSC is never empty) is of enough interest to warrant its own name. However, we define it as a property of the coefficient matrix, not as a property of some system of equations.

Definition  NSM
Null Space of a Matrix
The null space of a matrix A , denoted N A , is the set of all the vectors that are solutions to the homogeneous system ℒ S A , 0 .

(This definition contains Notation NSM.) △

In the Archetypes ( Appendix A) each example that is a system of equations also has a corresponding homogeneous system of equations listed, and several sample solutions are given. These solutions will be elements of the null space of the coefficient matrix. We’ll look at one example.

Example  NSEAI
Null space elements of Archetype I
The write-up for Archetype I lists several solutions of the corresponding homogeneous system. Here are two, written as solution vectors. We can say that they are in the null space of the coefficient matrix for the system of equations in Archetype I.

x = 3 0 − 5 − 6 0 0 1 y = − 4 1 − 3 − 2 1 1 1

However, the vector

z = 1 0 0 0 0 0 2

is not in the null space, since it is not a solution to the homogeneous system. For example, it fails to even make the first equation true. ⊠

Here are two (prototypical) examples of the computation of the null space of a matrix.

Example  CNS1
Computing a null space, #1
Let’s compute the null space of

A = 2 − 1 7 − 3 − 8 1 0 2 4 9 2 2 − 2 − 1 8

which we write as N A . Translating Definition NSM, we simply desire to solve the homogeneous system ℒ S A , 0 . So we row-reduce the augmented matrix to obtain

1 0 2 0 1 0 0 1  − 3 0 4 0 0 0 0 1 2 0

The variables (of the homogeneous system) x 3 and x 5 are free (since columns 1, 2 and 4 are pivot columns), so we arrange the equations represented by the matrix in reduced row-echelon form to

x 1 = − 2 x 3 − x 5 x 2 = 3 x 3 − 4 x 5 x 4 = − 2 x 5

So we can write the infinite solution set as sets using column vectors,

N A = − 2 x 3 − x 5 3 x 3 − 4 x 5 x 3 − 2 x 5 x 5 x 3 , x 5 ∈ ℂ

⊠

Example  CNS2
Computing a null space, #2
Let’s compute the null space of

C = − 4 6 1 − 1 4 1 5 6 7 4 7 1

which we write as N C . Translating Definition NSM, we simply desire to solve the homogeneous system ℒ S C , 0 . So we row-reduce the augmented matrix to obtain

1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 0

There are no free variables in the homogeneous system represented by the row-reduced matrix, so there is only the trivial solution, the zero vector, 0 . So we can write the (trivial) solution set as

N C = 0 = 0 0 0

⊠

Subsection READ: Reading Questions

Subsection EXC: Exercises

C10 Each Archetype ( Appendix A) that is a system of equations has a corresponding homogeneous system with the same coefficient matrix. Compute the set of solutions for each. Notice that these solution sets are the null spaces of the coefficient matrices.
Archetype A
Archetype B
Archetype C
Archetype D/ Archetype E
Archetype F
Archetype G/ Archetype H
Archetype I
and Archetype J  
Contributed by  Robert Beezer

C20 Archetype K and Archetype L are simply 5 × 5 matrices (i.e. they are not systems of equations). Compute the null space of each matrix.  
Contributed by  Robert Beezer

For Exercises C21-C23, solve the given homogeneous linear system. Compare your results to the results of the corresponding exercise in Section TSS.
C21

x 1 + 4 x 2 + 3 x 3 − x 4 = 0 x 1 − x 2 + x 3 + 2 x 4 = 0 4 x 1 + x 2 + 6 x 3 + 5 x 4 = 0

Contributed by  Chris Black Solution [ 208]

C22

x 1 − 2 x 2 + x 3 − x 4 = 0 2 x 1 − 4 x 2 + x 3 + x 4 = 0 x 1 − 2 x 2 − 2 x 3 + 3 x 4 = 0

Contributed by  Chris Black Solution [ 209]

C23

x 1 − 2 x 2 + x 3 − x 4 = 0 x 1 + x 2 + x 3 − x 4 = 0 x 1 + x 3 − x 4 = 0

Contributed by  Chris Black Solution [ 210]

For Exercises C25-C27, solve the given homogeneous linear system. Compare your results to the results of the corresponding exercise in Section TSS.
C25

x 1 + 2 x 2 + 3 x 3 = 0 2 x 1 − x 2 + x 3 = 0 3 x 1 + x 2 + x 3 = 0 x 2 + 2 x 3 = 0

Contributed by  Chris Black Solution [ 211]

C26

x 1 + 2 x 2 + 3 x 3 = 0 2 x 1 − x 2 + x 3 = 0 3 x 1 + x 2 + x 3 = 0 5 x 2 + 2 x 3 = 0

Contributed by  Chris Black Solution [ 211]

C27

x 1 + 2 x 2 + 3 x 3 = 0 2 x 1 − x 2 + x 3 = 0 x 1 − 8 x 2 − 7 x 3 = 0 x 2 + x 3 = 0

Contributed by  Chris Black Solution [ 212]

C30 Compute the null space of the matrix A , N A .

A = 2 4 1 3 8 − 1 − 2 − 1 − 1 1 2 4 0 − 3 4 2 4 − 1 − 7 4

Contributed by  Robert Beezer Solution [ 213]

C31 Find the null space of the matrix B , N B .

B = − 6 4 − 3 6 6 2 − 1 1 0 − 1 − 3 2 − 1 8 3

Contributed by  Robert Beezer Solution [ 215]

M45 Without doing any computations, and without examining any solutions, say as much as possible about the form of the solution set for corresponding homogeneous system of equations of each archetype that is a system of equations.
Archetype A
Archetype B
Archetype C
Archetype D/ Archetype E
Archetype F
Archetype G/ Archetype H
Archetype I
Archetype J  
Contributed by  Robert Beezer

For Exercises M50–M52 say as much as possible about each system’s solution set. Be sure to make it clear which theorems you are using to reach your conclusions.
M50 A homogeneous system of 8 equations in 8 variables.  
Contributed by  Robert Beezer Solution [ 216]

M51 A homogeneous system of 8 equations in 9 variables.  
Contributed by  Robert Beezer Solution [ 216]

M52 A homogeneous system of 8 equations in 7 variables.  
Contributed by  Robert Beezer Solution [ 216]

T10 Prove or disprove: A system of linear equations is homogeneous if and only if the system has the zero vector as a solution.  
Contributed by  Martin Jackson Solution [ 216]

T12 Give an alternate proof of Theorem HSC that uses Theorem RCLS.  
Contributed by  Ivan Kessler

T20 Consider the homogeneous system of linear equations ℒ S A , 0 , and suppose that u = u 1 u 2 u 3 ⋮ u n is one solution to the system of equations. Prove that v = 4 u 1 4 u 2 4 u 3 ⋮ 4 u n is also a solution to ℒ S A , 0 .  
Contributed by  Robert Beezer Solution [ 217]

Subsection SOL: Solutions

C21 Contributed by  Chris Black Statement [ 201]
The augmented matrix for the given linear system and its row-reduced form are:

1 4 3 − 1 0 1 − 1 1 2 0 4 1 6 5 0 →  RREF 1 0 7 ∕ 5 7 ∕ 5 0 0 1 2 ∕ 5 − 3 ∕ 5 0 0 0 0 0 0 .

Thus, we see that the system is consistent (as predicted by Theorem HSC) and has an infinite number of solutions (as predicted by Theorem HMVEI). With suitable choices of x 3 and x 4 , each solution can be written as

− 7 5 x 3 − 7 5 x 4 − 2 5 x 3 + 3 5 x 4 x 3 x 4

C22 Contributed by  Chris Black Statement [ 202]
The augmented matrix for the given linear system and its row-reduced form are:

1 − 2 1 − 1 0 2 − 4 1 1 0 1 − 2 − 2 3 0 →  RREF 1 − 2 0 0 0 0 0 1  0 0 0 0 0 1 0 .

Thus, we see that the system is consistent (as predicted by Theorem HSC) and has an infinite number of solutions (as predicted by Theorem HMVEI). With a suitable choice of x 2 , each solution can be written as

2 x 2 x 2 0 0

C23 Contributed by  Chris Black Statement [ 202]
The augmented matrix for the given linear system and its row-reduced form are:

1 − 2 1 − 1 0 1 1 1 − 1 0 1 0 1 − 1 0 →  RREF 1 0 1 − 1 0 0 1  0 0 0 0 0 0 0 0 .

Thus, we see that the system is consistent (as predicted by Theorem HSC) and has an infinite number of solutions (as predicted by Theorem HMVEI). With suitable choices of x 3 and x 4 , each solution can be written as

− x 3 + x 4 0 x 3 x 4

C25 Contributed by  Chris Black Statement [ 203]
The augmented matrix for the given linear system and its row-reduced form are:

1 2 3 0 2 − 1 1 0 3 1 1 0 0 1 2 0 →  RREF 1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 0 .

An homogeneous system is always consistent ( Theorem HSC) and with n = r = 3 an application of Theorem FVCS yields zero free variables. Thus the only solution to the given system is the trivial solution, x = 0 .

C26 Contributed by  Chris Black Statement [ 203]
The augmented matrix for the given linear system and its row-reduced form are:

1 2 3 0 2 − 1 1 0 3 1 1 0 0 5 2 0 →  RREF 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 .

An homogeneous system is always consistent ( Theorem HSC) and with n = r = 3 an application of Theorem FVCS yields zero free variables. Thus the only solution to the given system is the trivial solution, x = 0 .

C27 Contributed by  Chris Black Statement [ 204]
The augmented matrix for the given linear system and its row-reduced form are:

1 2 3 0 2 − 1 1 0 1 − 8 − 7 0 0 1 1 0 →  RREF 1 0 1 0 0 1  1 0 0 0 0 0 0 0 0 0 .

An homogeneous system is always consistent ( Theorem HSC) and with n = 3 , r = 2 an application of Theorem FVCS yields one free variable. With a suitable choice of x 3 each solution can be written in the form

− x 3 − x 3 x 3

C30 Contributed by  Robert Beezer Statement [ 205]
Definition NSM tells us that the null space of A is the solution set to the homogeneous system ℒ S A , 0 . The augmented matrix of this system is

2 4 1 3 8 0 − 1 − 2 − 1 − 1 1 0 2 4 0 − 3 4 0 2 4 − 1 − 7 4 0

To solve the system, we row-reduce the augmented matrix and obtain,

1 2 0 0 5 0 0 0 1  0 − 8 0 0 0 0 1 2 0 0 0 0 0 0 0

This matrix represents a system with equations having three dependent variables ( x 1 , x 3 , and x 4 ) and two independent variables ( x 2 and x 5 ). These equations rearrange to

x 1 = − 2 x 2 − 5 x 5 x 3 = 8 x 5 x 4 = − 2 x 5

So we can write the solution set (which is the requested null space) as

N A = − 2 x 2 − 5 x 5 x 2 8 x 5 − 2 x 5 x 5 x 2 , x 5 ∈ ℂ

C31 Contributed by  Robert Beezer Statement [ 205]
We form the augmented matrix of the homogeneous system ℒ S B , 0 and row-reduce the matrix,

− 6 4 − 3 6 6 0 2 − 1 1 0 − 1 0 − 3 2 − 1 8 3 0 →  RREF 1 0 2 1 0 0 1  − 6 3 0 0 0 0 0 0

We knew ahead of time that this system would be consistent ( Theorem HSC), but we can now see there are n − r = 4 − 2 = 2 free variables, namely x 3 and x 4 (Theorem FVCS). Based on this analysis, we can rearrange the equations associated with each nonzero row of the reduced row-echelon form into an expression for the lone dependent variable as a function of the free variables. We arrive at the solution set to the homogeneous system, which is the null space of the matrix by Definition NSM,

N B = − 2 x 3 − x 4 6 x 3 − 3 x 4 x 3 x 4 x 3 , x 4 ∈ ℂ

M50 Contributed by  Robert Beezer Statement [ 206]
Since the system is homogeneous, we know it has the trivial solution ( Theorem HSC). We cannot say anymore based on the information provided, except to say that there is either a unique solution or infinitely many solutions (Theorem PSSLS). See Archetype A and Archetype B to understand the possibilities.

M51 Contributed by  Robert Beezer Statement [ 206]
Since there are more variables than equations, Theorem HMVEI applies and tells us that the solution set is infinite. From the proof of Theorem HSC we know that the zero vector is one solution.

M52 Contributed by  Robert Beezer Statement [ 206]
By Theorem HSC, we know the system is consistent because the zero vector is always a solution of a homogeneous system. There is no more that we can say, since both a unique solution and infinitely many solutions are possibilities.

T10 Contributed by  Robert Beezer Statement [ 206]
This is a true statement. A proof is:

( ⇒ ) Suppose we have a homogeneous system ℒ S A , 0 . Then by substituting the scalar zero for each variable, we arrive at true statements for each equation. So the zero vector is a solution. This is the content of Theorem HSC.

( ⇐ ) Suppose now that we have a generic (i.e. not necessarily homogeneous) system of equations, ℒ S A , b that has the zero vector as a solution. Upon substituting this solution into the system, we discover that each component of b must also be zero. So b = 0 .

T20 Contributed by  Robert Beezer Statement [ 206]
Suppose that a single equation from this system (the i -th one) has the form,

a i 1 x 1 + a i 2 x 2 + a i 3 x 3 + ⋯ + a i n x n = 0

Evaluate the left-hand side of this equation with the components of the proposed solution vector v ,

a i 1 4 u 1 + a i 2 4 u 2 + a i 3 4 u 3 + ⋯ + a i n 4 u n = 4 a i 1 u 1 + 4 a i 2 u 2 + 4 a i 3 u 3 + ⋯ + 4 a i n u n  Commutativity = 4 a i 1 u 1 + a i 2 u 2 + a i 3 u 3 + ⋯ + a i n u n  Distributivity = 4 ( 0 )   u  solution to  ℒ S A , 0 = 0

So v makes each equation true, and so is a solution to the system.

Notice that this result is not true if we change ℒ S A , 0 from a homogeneous system to a non-homogeneous system. Can you create an example of a (non-homogeneous) system with a solution u such that v is not a solution?

Section NM  Nonsingular Matrices

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we specialize and consider matrices with equal numbers of rows and columns, which when considered as coefficient matrices lead to systems with equal numbers of equations and variables. We will see in the second half of the course ( Chapter D, Chapter E Chapter LT, Chapter R) that these matrices are especially important.

Parent topic: Chapter SLE  Systems of Linear Equations

Previous topic: Section HSE  Homogeneous Systems of Equations

Next topic: Chapter V  Vectors

Subsection NM: Nonsingular Matrices

Our theorems will now establish connections between systems of equations (homogeneous or otherwise), augmented matrices representing those systems, coefficient matrices, constant vectors, the reduced row-echelon form of matrices (augmented and coefficient) and solution sets. Be very careful in your reading, writing and speaking about systems of equations, matrices and sets of vectors. A system of equations is not a matrix, a matrix is not a solution set, and a solution set is not a system of equations. Now would be a great time to review the discussion about speaking and writing mathematics in Technique L.

Definition  SQM
Square Matrix
A matrix with m rows and n columns is square if m = n . In this case, we say the matrix has size n . To emphasize the situation when a matrix is not square, we will call it rectangular. △

We can now present one of the central definitions of linear algebra.

Definition  NM
Nonsingular Matrix
Suppose A is a square matrix. Suppose further that the solution set to the homogeneous linear system of equations ℒ S A , 0 is 0 , i.e. the system has only the trivial solution. Then we say that A is a nonsingular matrix. Otherwise we say A is a singular matrix. △

We can investigate whether any square matrix is nonsingular or not, no matter if the matrix is derived somehow from a system of equations or if it is simply a matrix. The definition says that to perform this investigation we must construct a very specific system of equations (homogeneous, with the matrix as the coefficient matrix) and look at its solution set. We will have theorems in this section that connect nonsingular matrices with systems of equations, creating more opportunities for confusion. Convince yourself now of two observations, (1) we can decide nonsingularity for any square matrix, and (2) the determination of nonsingularity involves the solution set for a certain homogeneous system of equations.

Notice that it makes no sense to call a system of equations nonsingular (the term does not apply to a system of equations), nor does it make any sense to call a 5 × 7 matrix singular (the matrix is not square).

Example  S
A singular matrix, Archetype A
Example HISAA shows that the coefficient matrix derived from Archetype A, specifically the 3 × 3 matrix,

A = 1 − 1 2 2 1 1 1 1 0

is a singular matrix since there are nontrivial solutions to the homogeneous system ℒ S A , 0 . ⊠

Example  NM
A nonsingular matrix, Archetype B
Example HUSAB shows that the coefficient matrix derived from Archetype B, specifically the 3 × 3 matrix,

B = − 7 − 6 − 1 2 5 5 7 1 0 4

is a nonsingular matrix since the homogeneous system, ℒ S B , 0 , has only the trivial solution. ⊠

Notice that we will not discuss Example HISAD as being a singular or nonsingular coefficient matrix since the matrix is not square.

The next theorem combines with our main computational technique (row-reducing a matrix) to make it easy to recognize a nonsingular matrix. But first a definition.

Definition  IM
Identity Matrix
The m × m identity matrix, I m , is defined by

I m i j = 1 i = j 0 i ≠ j 1 ≤ i , j ≤ m

(This definition contains Notation IM.) △

Example  IM
An identity matrix
The 4 × 4 identity matrix is

I 4 = 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 .

⊠

Notice that an identity matrix is square, and in reduced row-echelon form. So in particular, if we were to arrive at the identity matrix while bringing a matrix to reduced row-echelon form, then it would have all of the diagonal entries circled as leading 1’s.

Theorem  NMRRI
Nonsingular Matrices Row Reduce to the Identity matrix
Suppose that A is a square matrix and B is a row-equivalent matrix in reduced row-echelon form. Then A is nonsingular if and only if B is the identity matrix. □

Proof   ( ⇐ ) Suppose B is the identity matrix. When the augmented matrix A 0 is row-reduced, the result is B 0 = I n 0 . The number of nonzero rows is equal to the number of variables in the linear system of equations ℒ S A , 0 , so n = r and Theorem FVCS gives n − r = 0 free variables. Thus, the homogeneous system ℒ S A , 0 has just one solution, which must be the trivial solution. This is exactly the definition of a nonsingular matrix.

( ⇒ ) If A is nonsingular, then the homogeneous system ℒ S A , 0 has a unique solution, and has no free variables in the description of the solution set. The homogeneous system is consistent ( Theorem HSC) so Theorem FVCS applies and tells us there are n − r free variables. Thus, n − r = 0 , and so n = r . So B has n pivot columns among its total of n columns. This is enough to force B to be the n × n identity matrix I n (see Exercise NM.T12). ■

Notice that since this theorem is an equivalence it will always allow us to determine if a matrix is either nonsingular or singular. Here are two examples of this, continuing our study of Archetype A and Archetype B.

Example  SRR
Singular matrix, row-reduced
The coefficient matrix for Archetype A is

A = 1 − 1 2 2 1 1 1 1 0

which when row-reduced becomes the row-equivalent matrix

B = 1 0 1 0 1  − 1 0 0 0 .

Since this matrix is not the 3 × 3 identity matrix, Theorem NMRRI tells us that A is a singular matrix. ⊠

Example  NSR
Nonsingular matrix, row-reduced
The coefficient matrix for Archetype B is

A = − 7 − 6 − 1 2 5 5 7 1 0 4

which when row-reduced becomes the row-equivalent matrix

B = 1 0 0 0 1  0 0 0 1 .

Since this matrix is the 3 × 3 identity matrix, Theorem NMRRI tells us that A is a nonsingular matrix. ⊠

Subsection NSNM: Null Space of a Nonsingular Matrix

Nonsingular matrices and their null spaces are intimately related, as the next two examples illustrate.

Example  NSS
Null space of a singular matrix
Given the coefficient matrix from Archetype A,

A = 1 − 1 2 2 1 1 1 1 0

the null space is the set of solutions to the homogeneous system of equations ℒ S A , 0 has a solution set and null space constructed in Example HISAA as

N A = − x 3 x 3 x 3 x 3 ∈ ℂ

⊠

Example  NSNM
Null space of a nonsingular matrix
Given the coefficient matrix from Archetype B,

A = − 7 − 6 − 1 2 5 5 7 1 0 4

the homogeneous system ℒ S A , 0 has a solution set constructed in Example HUSAB that contains only the trivial solution, so the null space has only a single element,

N A = 0 0 0

⊠

These two examples illustrate the next theorem, which is another equivalence.

Theorem  NMTNS
Nonsingular Matrices have Trivial Null Spaces
Suppose that A is a square matrix. Then A is nonsingular if and only if the null space of A , N A , contains only the zero vector, i.e.  N A = 0 . □

Proof   The null space of a square matrix, A , is equal to the set of solutions to the homogeneous system, ℒ S A , 0 . A matrix is nonsingular if and only if the set of solutions to the homogeneous system, ℒ S A , 0 , has only a trivial solution. These two observations may be chained together to construct the two proofs necessary for each half of this theorem. ■

The next theorem pulls a lot of big ideas together. Theorem NMUS tells us that we can learn much about solutions to a system of linear equations with a square coefficient matrix by just examining a similar homogeneous system.

Theorem  NMUS
Nonsingular Matrices and Unique Solutions
Suppose that A is a square matrix. A is a nonsingular matrix if and only if the system ℒ S A , b has a unique solution for every choice of the constant vector b . □

Proof   ( ⇐ ) The hypothesis for this half of the proof is that the system ℒ S A , b has a unique solution for every choice of the constant vector b . We will make a very specific choice for b : b = 0 . Then we know that the system ℒ S A , 0 has a unique solution. But this is precisely the definition of what it means for A to be nonsingular ( Definition NM). That almost seems too easy! Notice that we have not used the full power of our hypothesis, but there is nothing that says we must use a hypothesis to its fullest.

( ⇒ ) We assume that A is nonsingular of size n × n , so we know there is a sequence of row operations that will convert A into the identity matrix I n ( Theorem NMRRI). Form the augmented matrix A ′ = A b and apply this same sequence of row operations to A ′ . The result will be the matrix B ′ = I n c , which is in reduced row-echelon form with r = n . Then the augmented matrix B ′ represents the (extremely simple) system of equations x i = c i , 1 ≤ i ≤ n . The vector c is clearly a solution, so the system is consistent (Definition CS). With a consistent system, we use Theorem FVCS to count free variables. We find that there are n − r = n − n = 0 free variables, and so we therefore know that the solution is unique. (This half of the proof was suggested by Asa Scherer.) ■

This theorem helps to explain part of our interest in nonsingular matrices. If a matrix is nonsingular, then no matter what vector of constants we pair it with, using the matrix as the coefficient matrix will always yield a linear system of equations with a solution, and the solution is unique. To determine if a matrix has this property (non-singularity) it is enough to just solve one linear system, the homogeneous system with the matrix as coefficient matrix and the zero vector as the vector of constants (or any other vector of constants, see Exercise MM.T10).

Formulating the negation of the second part of this theorem is a good exercise. A singular matrix has the property that for some value of the vector b , the system ℒ S A , b does not have a unique solution (which means that it has no solution or infinitely many solutions). We will be able to say more about this case later (see the discussion following Theorem PSPHS). Square matrices that are nonsingular have a long list of interesting properties, which we will start to catalog in the following, recurring, theorem. Of course, singular matrices will then have all of the opposite properties. The following theorem is a list of equivalences. We want to understand just what is involved with understanding and proving a theorem that says several conditions are equivalent. So have a look at Technique ME before studying the first in this series of theorems. 

Theorem  NME1
Nonsingular Matrix Equivalences, Round 1
Suppose that A is a square matrix. The following are equivalent.

□

Proof   That A is nonsingular is equivalent to each of the subsequent statements by, in turn, Theorem NMRRI, Theorem NMTNS and Theorem NMUS. So the statement of this theorem is just a convenient way to organize all these results. ■

Finally, you may have wondered why we refer to a matrix as nonsingular when it creates systems of equations with single solutions ( Theorem NMUS)! I’ve wondered the same thing. We’ll have an opportunity to address this when we get to Theorem SMZD. Can you wait that long?

Subsection READ: Reading Questions

Subsection EXC: Exercises

In Exercises C30–C33 determine if the matrix is nonsingular or singular. Give reasons for your answer.
C30

− 3 1 2 8 2 0 3 4 1 2 7 − 4 5 − 1 2 0

Contributed by  Robert Beezer Solution [ 238]

C31

2 3 1 4 1 1 1 0 − 1 2 3 5 1 2 1 3

Contributed by  Robert Beezer Solution [ 238]

C32

9 3 2 4 5 − 6 1 3 4 1 3 − 5

Contributed by  Robert Beezer Solution [ 238]

C33

− 1 2 0 3 1 − 3 − 2 4 − 2 0 4 3 − 3 1 − 2 3

Contributed by  Robert Beezer Solution [ 239]

C40 Each of the archetypes below is a system of equations with a square coefficient matrix, or is itself a square matrix. Determine if these matrices are nonsingular, or singular. Comment on the null space of each matrix.
Archetype A
Archetype B
Archetype F
Archetype K
Archetype L  
Contributed by  Robert Beezer

C50 Find the null space of the matrix E below.

E = 2 1 − 1 − 9 2 2 − 6 − 6 1 2 − 8 0 − 1 2 − 1 2 1 2

Contributed by  Robert Beezer Solution [ 239]

M30 Let A be the coefficient matrix of the system of equations below. Is A nonsingular or singular? Explain what you could infer about the solution set for the system based only on what you have learned about A being singular or nonsingular.

− x 1 + 5 x 2 = − 8 − 2 x 1 + 5 x 2 + 5 x 3 + 2 x 4 = 9 − 3 x 1 − x 2 + 3 x 3 + x 4 = 3 7 x 1 + 6 x 2 + 5 x 3 + x 4 = 3 0

Contributed by  Robert Beezer Solution [ 240]

For Exercises M51–M52 say as much as possible about each system’s solution set. Be sure to make it clear which theorems you are using to reach your conclusions.
M51 6 equations in 6 variables, singular coefficient matrix.  
Contributed by  Robert Beezer Solution [ 241]

M52 A system with a nonsingular coefficient matrix, not homogeneous.  
Contributed by  Robert Beezer Solution [ 241]

T10 Suppose that A is a singular matrix, and B is a matrix in reduced row-echelon form that is row-equivalent to A . Prove that the last row of B is a zero row.  
Contributed by  Robert Beezer Solution [ 241]

T12 Suppose that A is a square matrix. Using the definition of reduced row-echelon form ( Definition RREF) carefully, give a proof of the following equivalence: Every column of A is a pivot column if and only if A is the identity matrix (Definition IM).  
Contributed by  Robert Beezer

T30 Suppose that A is a nonsingular matrix and A is row-equivalent to the matrix B . Prove that B is nonsingular.  
Contributed by  Robert Beezer Solution [ 242]

T90 Provide an alternative for the second half of the proof of Theorem NMUS, without appealing to properties of the reduced row-echelon form of the coefficient matrix. In other words, prove that if A is nonsingular, then ℒ S A , b has a unique solution for every choice of the constant vector b . Construct this proof without using Theorem REMEF or Theorem RREFU.  
Contributed by  Robert Beezer Solution [ 242]

Subsection SOL: Solutions

C30 Contributed by  Robert Beezer Statement [ 233]
The matrix row-reduces to

1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1

which is the 4 × 4 identity matrix. By Theorem NMRRI the original matrix must be nonsingular.

C31 Contributed by  Robert Beezer Statement [ 233]
Row-reducing the matrix yields,

1 0 0 − 2 0 1  0 3 0 0 1 − 1 0 0 0 0

Since this is not the 4 × 4 identity matrix, Theorem NMRRI tells us the matrix is singular.

C32 Contributed by  Robert Beezer Statement [ 233]
The matrix is not square, so neither term is applicable. See Definition NM, which is stated for just square matrices.

C33 Contributed by  Robert Beezer Statement [ 234]
Theorem NMRRI tells us we can answer this question by simply row-reducing the matrix. Doing this we obtain,

1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1

Since the reduced row-echelon form of the matrix is the 4 × 4 identity matrix I 4 , we know that B is nonsingular.

C50 Contributed by  Robert Beezer Statement [ 235]
We form the augmented matrix of the homogeneous system ℒ S E , 0 and row-reduce the matrix,

2 1 − 1 − 9 0 2 2 − 6 − 6 0 1 2 − 8 0 0 − 1 2 − 1 2 1 2 0 →  RREF 1 0 2 − 6 0 0 1  − 5 3 0 0 0 0 0 0 0 0 0 0 0

We knew ahead of time that this system would be consistent ( Theorem HSC), but we can now see there are n − r = 4 − 2 = 2 free variables, namely x 3 and x 4 since F = 3 , 4 , 5 (Theorem FVCS). Based on this analysis, we can rearrange the equations associated with each nonzero row of the reduced row-echelon form into an expression for the lone dependent variable as a function of the free variables. We arrive at the solution set to this homogeneous system, which is the null space of the matrix by Definition NSM,

N E = − 2 x 3 + 6 x 4 5 x 3 − 3 x 4 x 3 x 4 x 3 , x 4 ∈ ℂ

M30 Contributed by  Robert Beezer Statement [ 235]
We row-reduce the coefficient matrix of the system of equations,

− 1 5 0 0 − 2 5 5 2 − 3 − 1 3 1 7 6 5 1 →  RREF 1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1

Since the row-reduced version of the coefficient matrix is the 4 × 4 identity matrix, I 4 ( Definition IM by Theorem NMRRI, we know the coefficient matrix is nonsingular. According to Theorem NMUS we know that the system is guaranteed to have a unique solution, based only on the extra information that the coefficient matrix is nonsingular.

M51 Contributed by  Robert Beezer Statement [ 236]
Theorem NMRRI tells us that the coefficient matrix will not row-reduce to the identity matrix. So if we were to row-reduce the augmented matrix of this system of equations, we would not get a unique solution. So by Theorem PSSLS the remaining possibilities are no solutions, or infinitely many.

M52 Contributed by  Robert Beezer Statement [ 236]
Any system with a nonsingular coefficient matrix will have a unique solution by Theorem NMUS. If the system is not homogeneous, the solution cannot be the zero vector ( Exercise HSE.T10).

T10 Contributed by  Robert Beezer Statement [ 236]
Let n denote the size of the square matrix A . By Theorem NMRRI the hypothesis that A is singular implies that B is not the identity matrix I n . If B has n pivot columns, then it would have to be I n , so B must have fewer than n pivot columns. But the number of nonzero rows in B ( r ) is equal to the number of pivot columns as well. So the n rows of B have fewer than n nonzero rows, and B must contain at least one zero row. By Definition RREF, this row must be at the bottom of B .

T30 Contributed by  Robert Beezer Statement [ 236]
Since A and B are row-equivalent matrices, consideration of the three row operations ( Definition RO) will show that the augmented matrices, A 0 and B 0 , are also row-equivalent matrices. This says that the two homogeneous systems, ℒ S A , 0 and ℒ S B , 0 are equivalent systems. ℒ S A , 0 has only the zero vector as a solution ( Definition NM), thus ℒ S B , 0 has only the zero vector as a solution. Finally, by Definition NM, we see that B is nonsingular.

Form a similar theorem replacing “nonsingular” by “singular” in both the hypothesis and the conclusion. Prove this new theorem with an approach just like the one above, and/or employ the result about nonsingular matrices in a proof by contradiction.

T90 Contributed by  Robert Beezer Statement [ 237]
We assume A is nonsingular, and try to solve the system ℒ S A , b without making any assumptions about b . To do this we will begin by constructing a new homogeneous linear system of equations that looks very much like the original. Suppose A has size n (why must it be square?) and write the original system as,

a 1 1 x 1 + a 1 2 x 2 + a 1 3 x 3 + ⋯ + a 1 n x n = b 1 a 2 1 x 1 + a 2 2 x 2 + a 2 3 x 3 + ⋯ + a 2 n x n = b 2 a 3 1 x 1 + a 3 2 x 2 + a 3 3 x 3 + ⋯ + a 3 n x n = b 3 ⋮  ( ∗ ) a n 1 x 1 + a n 2 x 2 + a n 3 x 3 + ⋯ + a n n x n = b n

Form the new, homogeneous system in n equations with n + 1 variables, by adding a new variable y , whose coefficients are the negatives of the constant terms,

a 1 1 x 1 + a 1 2 x 2 + a 1 3 x 3 + ⋯ + a 1 n x n − b 1 y = 0 a 2 1 x 1 + a 2 2 x 2 + a 2 3 x 3 + ⋯ + a 2 n x n − b 2 y = 0 a 3 1 x 1 + a 3 2 x 2 + a 3 3 x 3 + ⋯ + a 3 n x n − b 3 y = 0 ⋮  ( ∗ ∗ ) a n 1 x 1 + a n 2 x 2 + a n 3 x 3 + ⋯ + a n n x n − b n y = 0

Since this is a homogeneous system with more variables than equations ( m = n + 1 > n ), Theorem HMVEI says that the system has infinitely many solutions. We will choose one of these solutions, any one of these solutions, so long as it is not the trivial solution. Write this solution as

x 1 = c 1 x 2 = c 2 x 3 = c 3 … x n = c n y = c n + 1

We know that at least one value of the c i is nonzero, but we will now show that in particular c n + 1 ≠ 0 . We do this using a proof by contradiction ( Technique CD). So suppose the c i form a solution as described, and in addition that c n + 1 = 0 . Then we can write the i -th equation of system ( ∗ ∗ ) as,

a i 1 c 1 + a i 2 c 2 + a i 3 c 3 + ⋯ + a i n c n − b i ( 0 ) = 0  which becomes a i 1 c 1 + a i 2 c 2 + a i 3 c 3 + ⋯ + a i n c n = 0

Since this is true for each i , we have that x 1 = c 1 , x 2 = c 2 , x 3 = c 3 , … , x n = c n is a solution to the homogeneous system ℒ S A , 0 formed with a nonsingular coefficient matrix. This means that the only possible solution is the trivial solution, so c 1 = 0 , c 2 = 0 , c 3 = 0 , … , c n = 0 . So, assuming simply that c n + 1 = 0 , we conclude that all of the c i are zero. But this contradicts our choice of the c i as not being the trivial solution to the system ( ∗ ∗ ) . So c n + 1 ≠ 0 .

We now propose and verify a solution to the original system ( ∗ ) . Set

x 1 = c 1 c n + 1 x 2 = c 2 c n + 1 x 3 = c 3 c n + 1 … x n = c n c n + 1

Notice how it was necessary that we know that c n + 1 ≠ 0 for this step to succeed. Now, evaluate the i -th equation of system ( ∗ ) with this proposed solution, and recognize in the third line that c 1 through c n + 1 appear as if they were substituted into the left-hand side of the i -th equation of system ( ∗ ∗ ) ,

a i 1 c 1 c n + 1 + a i 2 c 2 c n + 1 + a i 3 c 3 c n + 1 + ⋯ + a i n c n c n + 1 = 1 c n + 1 a i 1 c 1 + a i 2 c 2 + a i 3 c 3 + ⋯ + a i n c n = 1 c n + 1 a i 1 c 1 + a i 2 c 2 + a i 3 c 3 + ⋯ + a i n c n − b i c n + 1 + b i = 1 c n + 1 0 + b i = b i

Since this equation is true for every i , we have found a solution to system ( ∗ ) . To finish, we still need to establish that this solution is unique.

With one solution in hand, we will entertain the possibility of a second solution. So assume system ( ∗ ) has two solutions,

x 1 = d 1 x 2 = d 2 x 3 = d 3 … x n = d n x 1 = e 1 x 2 = e 2 x 3 = e 3 … x n = e n

Then,

a i 1 ( d 1 − e 1 ) + a i 2 ( d 2 − e 2 ) + a i 3 ( d 3 − e 3 ) + ⋯ + a i n ( d n − e n ) = a i 1 d 1 + a i 2 d 2 + a i 3 d 3 + ⋯ + a i n d n − a i 1 e 1 + a i 2 e 2 + a i 3 e 3 + ⋯ + a i n e n = b i − b i = 0

This is the i -th equation of the homogeneous system ℒ S A , 0 evaluated with x j = d j − e j , 1 ≤ j ≤ n . Since A is nonsingular, we must conclude that this solution is the trivial solution, and so 0 = d j − e j , 1 ≤ j ≤ n . That is, d j = e j for all j and the two solutions are identical, meaning any solution to ( ∗ ) is unique.

Notice that the proposed solution ( x i = c i c n + 1 ) appeared in this proof with no motivation whatsoever. This is just fine in a proof. A proof should convince you that a theorem is true. It is your job to read the proof and be convinced of every assertion. Questions like “Where did that come from?” or “How would I think of that?” have no bearing on the validity of the proof.

Annotated Acronyms SLE: Systems of Linear Equations

At the conclusion of each chapter you will find a section like this, reviewing selected definitions and theorems. There are many reasons for why a definition or theorem might be placed here. It might represent a key concept, it might be used frequently for computations, provide the critical step in many proofs, or it may deserve special comment.

These lists are not meant to be exhaustive, but should still be useful as part of reviewing each chapter. We will mention a few of these that you might eventually recognize on sight as being worth memorization. By that we mean that you can associate the acronym with a rough statement of the theorem — not that the exact details of the theorem need to be memorized. And it is certainly not our intent that everything on these lists is important enough to memorize.

Theorem RCLS
We will repeatedly appeal to this theorem to determine if a system of linear equations, does, or doesn’t, have a solution. This one we will see often enough that it is worth memorizing.

Theorem HMVEI
This theorem is the theoretical basis of several of our most important theorems. So keep an eye out for it, and its descendants, as you study other proofs. For example, Theorem HMVEI is critical to the proof of Theorem SSLD, Theorem SSLD is critical to the proof of Theorem G, Theorem G is critical to the proofs of the pair of similar theorems, Theorem ILTD and Theorem SLTD, while finally Theorem ILTD and Theorem SLTD are critical to the proof of an important result, Theorem IVSED. This chain of implications might not make much sense on a first reading, but come back later to see how some very important theorems build on the seemingly simple result that is Theorem HMVEI. Using the “find” feature in whatever software you use to read the electronic version of the text can be a fun way to explore these relationships.

Theorem NMRRI
This theorem gives us one of simplest ways, computationally, to recognize if a matrix is nonsingular, or singular. We will see this one often, in computational exercises especially.

Theorem NMUS
Nonsingular matrices will be an important topic going forward (witness the NMEx series of theorems). This is our first result along these lines, a useful theorem for other proofs, and also illustrates a more general concept from Chapter LT.

Chapter 2. Chapter V  Vectors

We have worked extensively in the last chapter with matrices, and some with vectors. In this chapter we will develop the properties of vectors, while preparing to study vector spaces ( Chapter VS). Initially we will depart from our study of systems of linear equations, but in Section LC we will forge a connection between linear combinations and systems of linear equations in Theorem SLSLC. This connection will allow us to understand systems of linear equations at a higher level, while consequently discussing them less frequently.

Section SS  Spanning Sets

Section O  Orthogonality

Parent topic: Part C  Core

Previous topic: Section NM  Nonsingular Matrices

Next topic: Section SS  Spanning Sets

Section SS  Spanning Sets

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we will describe a compact way to indicate the elements of an infinite set of vectors, making use of linear combinations. This will give us a convenient way to describe the elements of a set of solutions to a linear system, or the elements of the null space of a matrix, or many other sets of vectors.

Parent topic: Chapter V  Vectors

Previous topic: Chapter V  Vectors

Next topic: Section O  Orthogonality

Subsection SSV: Span of a Set of Vectors

In Example VFSAL we saw the solution set of a homogeneous system described as all possible linear combinations of two particular vectors. This happens to be a useful way to construct or describe infinite sets of vectors, so we encapsulate this idea in a definition.

Definition  SSCV
Span of a Set of Column Vectors
Given a set of vectors S = { u 1 , u 2 , u 3 , … , u p } , their span, S , is the set of all possible linear combinations of u 1 , u 2 , u 3 , … , u p . Symbolically,

S = α 1 u 1 + α 2 u 2 + α 3 u 3 + ⋯ + α p u p α i ∈ ℂ , 1 ≤ i ≤ p = ∑ i = 1 p α i u i α i ∈ ℂ , 1 ≤ i ≤ p

(This definition contains Notation SSV.) △

The span is just a set of vectors, though in all but one situation it is an infinite set. (Just when is it not infinite?) So we start with a finite collection of vectors S ( p of them to be precise), and use this finite set to describe an infinite set of vectors, S . Confusing the finite set S with the infinite set S is one of the most pervasive problems in understanding introductory linear algebra. We will see this construction repeatedly, so let’s work through some examples to get comfortable with it. The most obvious question about a set is if a particular item of the correct type is in the set, or not.

Example  ABS
A basic span
Consider the set of 5 vectors, S , from ℂ 4

S = 1 1 3 1 , 2 1 2 − 1 , 7 3 5 − 5 , 1 1 − 1 2 , − 1 0 9 0

and consider the infinite set of vectors S formed from all possible linear combinations of the elements of S . Here are four vectors we definitely know are elements of S , since we will construct them in accordance with Definition SSCV,

w = ( 2 ) 1 1 3 1 + ( 1 ) 2 1 2 − 1 + ( − 1 ) 7 3 5 − 5 + ( 2 ) 1 1 − 1 2 + ( 3 ) − 1 0 9 0 = − 4 2 2 8 1 0

x = ( 5 ) 1 1 3 1 + ( − 6 ) 2 1 2 − 1 + ( − 3 ) 7 3 5 − 5 + ( 4 ) 1 1 − 1 2 + ( 2 ) − 1 0 9 0 = − 2 6 − 6 2 3 4

y = ( 1 ) 1 1 3 1 + ( 0 ) 2 1 2 − 1 + ( 1 ) 7 3 5 − 5 + ( 0 ) 1 1 − 1 2 + ( 1 ) − 1 0 9 0 = 7 4 1 7 − 4

z = ( 0 ) 1 1 3 1 + ( 0 ) 2 1 2 − 1 + ( 0 ) 7 3 5 − 5 + ( 0 ) 1 1 − 1 2 + ( 0 ) − 1 0 9 0 = 0 0 0 0

The purpose of a set is to collect objects with some common property, and to exclude objects without that property. So the most fundamental question about a set is if a given object is an element of the set or not. Let’s learn more about S by investigating which vectors are elements of the set, and which are not.

First, is u = − 1 5 − 6 1 9 5 an element of S ? We are asking if there are scalars α 1 , α 2 , α 3 , α 4 , α 5 such that

α 1 1 1 3 1 + α 2 2 1 2 − 1 + α 3 7 3 5 − 5 + α 4 1 1 − 1 2 + α 5 − 1 0 9 0 = u = − 1 5 − 6 1 9 5

Applying Theorem SLSLC we recognize the search for these scalars as a solution to a linear system of equations with augmented matrix

1 2 7 1 − 1 − 1 5 1 1 3 1 0 − 6 3 2 5 − 1 9 1 9 1 − 1 − 5 2 0 5

which row-reduces to

1 0 − 1 0 3 1 0 0 1  4 0 − 1 − 9 0 0 0 1 − 2 − 7 0 0 0 0 0 0

At this point, we see that the system is consistent (Theorem RCLS), so we know there is a solution for the five scalars α 1 , α 2 , α 3 , α 4 , α 5 . This is enough evidence for us to say that u ∈ S . If we wished further evidence, we could compute an actual solution, say

α 1 = 2 α 2 = 1 α 3 = − 2 α 4 = − 3 α 5 = 2

This particular solution allows us to write

( 2 ) 1 1 3 1 + ( 1 ) 2 1 2 − 1 + ( − 2 ) 7 3 5 − 5 + ( − 3 ) 1 1 − 1 2 + ( 2 ) − 1 0 9 0 = u = − 1 5 − 6 1 9 5

making it even more obvious that u ∈ S .

Lets do it again. Is v = 3 1 2 − 1 an element of S ? We are asking if there are scalars α 1 , α 2 , α 3 , α 4 , α 5 such that

α 1 1 1 3 1 + α 2 2 1 2 − 1 + α 3 7 3 5 − 5 + α 4 1 1 − 1 2 + α 5 − 1 0 9 0 = v = 3 1 2 − 1

Applying Theorem SLSLC we recognize the search for these scalars as a solution to a linear system of equations with augmented matrix

1 2 7 1 − 1 3 1 1 3 1 0 1 3 2 5 − 1 9 2 1 − 1 − 5 2 0 − 1

which row-reduces to

1 0 − 1 0 3 0 0 1  4 0 − 1 0 0 0 0 1 − 2 0 0 0 0 0 0 1

At this point, we see that the system is inconsistent by Theorem RCLS, so we know there is not a solution for the five scalars α 1 , α 2 , α 3 , α 4 , α 5 . This is enough evidence for us to say that v ∉ S . End of story. ⊠

Example  SCAA
Span of the columns of Archetype A
Begin with the finite set of three vectors of size 3

S = { u 1 , u 2 , u 3 } = 1 2 1 , − 1 1 1 , 2 1 0

and consider the infinite set S . The vectors of S could have been chosen to be anything, but for reasons that will become clear later, we have chosen the three columns of the coefficient matrix in Archetype A. First, as an example, note that

v = ( 5 ) 1 2 1 + ( − 3 ) − 1 1 1 + ( 7 ) 2 1 0 = 2 2 1 4 2

is in S , since it is a linear combination of u 1 , u 2 , u 3 . We write this succinctly as v ∈ S . There is nothing magical about the scalars α 1 = 5 , α 2 = − 3 , α 3 = 7 , they could have been chosen to be anything. So repeat this part of the example yourself, using different values of α 1 , α 2 , α 3 . What happens if you choose all three scalars to be zero?

So we know how to quickly construct sample elements of the set S . A slightly different question arises when you are handed a vector of the correct size and asked if it is an element of S . For example, is w = 1 8 5 in S ? More succinctly, w ∈ S ?

To answer this question, we will look for scalars α 1 , α 2 , α 3 so that

α 1 u 1 + α 2 u 2 + α 3 u 3 = w  By  Theorem SLSLC  solutions to this vector equation are solutions to the system of equations α 1 − α 2 + 2 α 3 = 1 2 α 1 + α 2 + α 3 = 8 α 1 + α 2 = 5  Building the augmented matrix for this linear system, and row-reducing, gives 1 0 1 3 0 1  − 1 2 0 0 0 0

This system has infinitely many solutions (there’s a free variable in x 3 ), but all we need is one solution vector. The solution,

α 1 = 2 α 2 = 3 α 3 = 1

tells us that

( 2 ) u 1 + ( 3 ) u 2 + ( 1 ) u 3 = w

so we are convinced that w really is in S . Notice that there are an infinite number of ways to answer this question affirmatively. We could choose a different solution, this time choosing the free variable to be zero,

α 1 = 3 α 2 = 2 α 3 = 0

shows us that

( 3 ) u 1 + ( 2 ) u 2 + ( 0 ) u 3 = w

Verifying the arithmetic in this second solution will make it obvious that w is in this span. And of course, we now realize that there are an infinite number of ways to realize w as element of S . Let’s ask the same type of question again, but this time with y = 2 4 3 , i.e. is y ∈ S ?

So we’ll look for scalars α 1 , α 2 , α 3 so that

α 1 u 1 + α 2 u 2 + α 3 u 3 = y  By  Theorem SLSLC  solutions to this vector equation are the solutions to the system of equations α 1 − α 2 + 2 α 3 = 2 2 α 1 + α 2 + α 3 = 4 α 1 + α 2 = 3  Building the augmented matrix for this linear system, and row-reducing, gives 1 0 1 0 0 1  − 1 0 0 0 0 1

This system is inconsistent (there’s a leading 1 in the last column, Theorem RCLS), so there are no scalars α 1 , α 2 , α 3 that will create a linear combination of u 1 , u 2 , u 3 that equals y . More precisely, y ∉ S .

There are three things to observe in this example. (1) It is easy to construct vectors in S . (2) It is possible that some vectors are in S (e.g.  w ), while others are not (e.g.  y ). (3) Deciding if a given vector is in S leads to solving a linear system of equations and asking if the system is consistent.

With a computer program in hand to solve systems of linear equations, could you create a program to decide if a vector was, or wasn’t, in the span of a given set of vectors? Is this art or science?

This example was built on vectors from the columns of the coefficient matrix of Archetype A. Study the determination that v ∈ S and see if you can connect it with some of the other properties of Archetype A. ⊠

Having analyzed Archetype A in Example SCAA, we will of course subject Archetype B to a similar investigation.

Example  SCAB
Span of the columns of Archetype B
Begin with the finite set of three vectors of size 3 that are the columns of the coefficient matrix in Archetype B,

R = { v 1 , v 2 , v 3 } = − 7 5 1 , − 6 5 0 , − 1 2 7 4

and consider the infinite set R . First, as an example, note that

x = ( 2 ) − 7 5 1 + ( 4 ) − 6 5 0 + ( − 3 ) − 1 2 7 4 = − 2 9 − 1 0

is in R , since it is a linear combination of v 1 , v 2 , v 3 . In other words, x ∈ R . Try some different values of α 1 , α 2 , α 3 yourself, and see what vectors you can create as elements of R .

Now ask if a given vector is an element of R . For example, is z = − 3 3 2 4 5 in R ? Is z ∈ R ?

To answer this question, we will look for scalars α 1 , α 2 , α 3 so that

α 1 v 1 + α 2 v 2 + α 3 v 3 = z  By  Theorem SLSLC  solutions to this vector equation are the solutions to the system of equations − 7 α 1 − 6 α 2 − 1 2 α 3 = − 3 3 5 α 1 + 5 α 2 + 7 α 3 = 2 4 α 1 + 4 α 3 = 5  Building the augmented matrix for this linear system, and row-reducing, gives 1 0 0 − 3 0 1  0 5 0 0 1 2

This system has a unique solution,

α 1 = − 3 α 2 = 5 α 3 = 2

telling us that

( − 3 ) v 1 + ( 5 ) v 2 + ( 2 ) v 3 = z

so we are convinced that z really is in R . Notice that in this case we have only one way to answer the question affirmatively since the solution is unique.

Let’s ask about another vector, say is x = − 7 8 − 3 in R ? Is x ∈ R ?

We desire scalars α 1 , α 2 , α 3 so that

α 1 v 1 + α 2 v 2 + α 3 v 3 = x  By  Theorem SLSLC  solutions to this vector equation are the solutions to the system of equations − 7 α 1 − 6 α 2 − 1 2 α 3 = − 7 5 α 1 + 5 α 2 + 7 α 3 = 8 α 1 + 4 α 3 = − 3  Building the augmented matrix for this linear system, and row-reducing, gives 1 0 0 1 0 1  0 2 0 0 1 − 1

This system has a unique solution,

α 1 = 1 α 2 = 2 α 3 = − 1

telling us that

( 1 ) v 1 + ( 2 ) v 2 + ( − 1 ) v 3 = x

so we are convinced that x really is in R . Notice that in this case we again have only one way to answer the question affirmatively since the solution is again unique.

We could continue to test other vectors for membership in R , but there is no point. A question about membership in R inevitably leads to a system of three equations in the three variables α 1 , α 2 , α 3 with a coefficient matrix whose columns are the vectors v 1 , v 2 , v 3 . This particular coefficient matrix is nonsingular, so by Theorem NMUS, the system is guaranteed to have a solution. (This solution is unique, but that’s not critical here.) So no matter which vector we might have chosen for z , we would have been certain to discover that it was an element of R . Stated differently, every vector of size 3 is in R , or R = ℂ 3 .

Compare this example with Example SCAA, and see if you can connect z with some aspects of the write-up for Archetype B. ⊠

Subsection SSNS: Spanning Sets of Null Spaces

We saw in Example VFSAL that when a system of equations is homogeneous the solution set can be expressed in the form described by Theorem VFSLS where the vector c is the zero vector. We can essentially ignore this vector, so that the remainder of the typical expression for a solution looks like an arbitrary linear combination, where the scalars are the free variables and the vectors are u 1 , u 2 , u 3 , … , u n − r . Which sounds a lot like a span. This is the substance of the next theorem.

Theorem  SSNS
Spanning Sets for Null Spaces
Suppose that A is an m × n matrix, and B is a row-equivalent matrix in reduced row-echelon form with r nonzero rows. Let D = { d 1 , d 2 , d 3 , … , d r } be the column indices where B has leading 1’s (pivot columns) and F = { f 1 , f 2 , f 3 , … , f n − r } be the set of column indices where B does not have leading 1’s. Construct the n − r vectors z j , 1 ≤ j ≤ n − r of size n as

z j i = 1  if  i ∈ F ,  i = f j 0  if  i ∈ F ,  i ≠ f j − B k , f j  if  i ∈ D ,  i = d k

Then the null space of A is given by

N A = z 1 , z 2 , z 3 , … , z n − r

□

Proof   Consider the homogeneous system with A as a coefficient matrix, ℒ S A , 0 . Its set of solutions, S , is by Definition NSM, the null space of A , N A . Let B ′ denote the result of row-reducing the augmented matrix of this homogeneous system. Since the system is homogeneous, the final column of the augmented matrix will be all zeros, and after any number of row operations ( Definition RO), the column will still be all zeros. So B ′ has a final column that is totally zeros.

Now apply Theorem VFSLS to B ′ , after noting that our homogeneous system must be consistent ( Theorem HSC). The vector c has zeros for each entry that corresponds to an index in F . For entries that correspond to an index in D , the value is − B ′ k , n + 1 , but for B ′ any entry in the final column (index n + 1 ) is zero. So c = 0 . The vectors z j , 1 ≤ j ≤ n − r are identical to the vectors u j , 1 ≤ j ≤ n − r described in Theorem VFSLS. Putting it all together and applying Definition SSCV in the final step,

N A = S = c + α 1 u 1 + α 2 u 2 + α 3 u 3 + ⋯ + α n − r u n − r α 1 , α 2 , α 3 , … , α n − r ∈ ℂ = α 1 u 1 + α 2 u 2 + α 3 u 3 + ⋯ + α n − r u n − r α 1 , α 2 , α 3 , … , α n − r ∈ ℂ = z 1 , z 2 , z 3 , … , z n − r

■

Example  SSNS
Spanning set of a null space
Find a set of vectors, S , so that the null space of the matrix A below is the span of S , that is, S = N A .

A = 1 3 3 − 1 − 5 2 5 7 1 1 1 1 5 1 5 − 1 − 4 − 2 0 4

The null space of A is the set of all solutions to the homogeneous system ℒ S A , 0 . If we find the vector form of the solutions to this homogeneous system (Theorem VFSLS) then the vectors u j , 1 ≤ j ≤ n − r in the linear combination are exactly the vectors z j , 1 ≤ j ≤ n − r described in Theorem SSNS. So we can mimic Example VFSAL to arrive at these vectors (rather than being a slave to the formulas in the statement of the theorem).

Begin by row-reducing A . The result is

1 0 6 0 4 0 1  − 1 0 − 2 0 0 0 1 3 0 0 0 0 0

With D = 1 , 2 , 4 and F = 3 , 5 we recognize that x 3 and x 5 are free variables and we can express each nonzero row as an expression for the dependent variables x 1 , x 2 , x 4 (respectively) in the free variables x 3 and x 5 . With this we can write the vector form of a solution vector as

x 1 x 2 x 3 x 4 x 5 = − 6 x 3 − 4 x 5 x 3 + 2 x 5 x 3 − 3 x 5 x 5 = x 3 − 6 1 1 0 0 + x 5 − 4 2 0 − 3 1

Then in the notation of Theorem SSNS,

z 1 = − 6 1 1 0 0 z 2 = − 4 2 0 − 3 1

and

N A = z 1 , z 2 = − 6 1 1 0 0 , − 4 2 0 − 3 1

⊠

Example  NSDS
Null space directly as a span
Let’s express the null space of A as the span of a set of vectors, applying Theorem SSNS as economically as possible, without reference to the underlying homogeneous system of equations (in contrast to Example SSNS).

A = 2 1 5 1 5 1 1 1 3 1 6 − 1 − 1 1 − 1 0 4 − 3 − 3 2 − 4 − 4 − 7 0 3 − 1 5 2 2 3

Theorem SSNS creates vectors for the span by first row-reducing the matrix in question. The row-reduced version of A is

B = 1 0 2 0 − 1 2 0 1  1 0 3 − 1 0 0 0 1 4 − 2 0 0 0 0 0 0 0 0 0 0 0 0

We will mechanically follow the prescription of Theorem SSNS. Here we go, in two big steps.

First, the non-pivot columns have indices F = 3 , 5 , 6 , so we will construct the n − r = 6 − 3 = 3 vectors with a pattern of zeros and ones corresponding to the indices in F . This is the realization of the first two lines of the three-case definition of the vectors z j , 1 ≤ j ≤ n − r .

z 1 = 1 0 0 z 2 = 0 1 0 z 3 = 0 0 1

Each of these vectors arises due to the presence of a column that is not a pivot column. The remaining entries of each vector are the entries of the corresponding non-pivot column, negated, and distributed into the empty slots in order (these slots have indices in the set D and correspond to pivot columns). This is the realization of the third line of the three-case definition of the vectors z j , 1 ≤ j ≤ n − r .

z 1 = − 2 − 1 1 0 0 0 z 2 = 1 − 3 0 − 4 1 0 z 3 = − 2 1 0 2 0 1

So, by Theorem SSNS, we have

N A = z 1 , z 2 , z 3 = − 2 − 1 1 0 0 0 , 1 − 3 0 − 4 1 0 , − 2 1 0 2 0 1

We know that the null space of A is the solution set of the homogeneous system ℒ S A , 0 , but nowhere in this application of Theorem SSNS have we found occasion to reference the variables or equations of this system. These details are all buried in the proof of Theorem SSNS. ⊠

More advanced computational devices will compute the null space of a matrix.  See: Computation NS.MMA Here’s an example that will simultaneously exercise the span construction and Theorem SSNS, while also pointing the way to the next section.

Example  SCAD
Span of the columns of Archetype D
Begin with the set of four vectors of size 3

T = w 1 , w 2 , w 3 , w 4 = 2 − 3 1 , 1 4 1 , 7 − 5 4 , − 7 − 6 − 5

and consider the infinite set W = T . The vectors of T have been chosen as the four columns of the coefficient matrix in Archetype D. Check that the vector

z 2 = 2 3 0 1

is a solution to the homogeneous system ℒ S D , 0 (it is the vector z 2 provided by the description of the null space of the coefficient matrix D from Theorem SSNS). Applying Theorem SLSLC, we can write the linear combination,

2 w 1 + 3 w 2 + 0 w 3 + 1 w 4 = 0

which we can solve for w 4 ,

w 4 = ( − 2 ) w 1 + ( − 3 ) w 2 .

This equation says that whenever we encounter the vector w 4 , we can replace it with a specific linear combination of the vectors w 1 and w 2 . So using w 4 in the set T , along with w 1 and w 2 , is excessive. An example of what we mean here can be illustrated by the computation,

5 w 1 + ( − 4 ) w 2 + 6 w 3 + ( − 3 ) w 4 = 5 w 1 + ( − 4 ) w 2 + 6 w 3 + ( − 3 ) ( − 2 ) w 1 + ( − 3 ) w 2 = 5 w 1 + ( − 4 ) w 2 + 6 w 3 + 6 w 1 + 9 w 2 = 1 1 w 1 + 5 w 2 + 6 w 3

So what began as a linear combination of the vectors w 1 , w 2 , w 3 , w 4 has been reduced to a linear combination of the vectors w 1 , w 2 , w 3 . A careful proof using our definition of set equality ( Definition SE) would now allow us to conclude that this reduction is possible for any vector in W , so

W = w 1 , w 2 , w 3

So the span of our set of vectors, W , has not changed, but we have described it by the span of a set of three vectors, rather than four. Furthermore, we can achieve yet another, similar, reduction.

Check that the vector

z 1 = − 3 − 1 1 0

is a solution to the homogeneous system ℒ S D , 0 (it is the vector z 1 provided by the description of the null space of the coefficient matrix D from Theorem SSNS). Applying Theorem SLSLC, we can write the linear combination,

( − 3 ) w 1 + ( − 1 ) w 2 + 1 w 3 = 0

which we can solve for w 3 ,

w 3 = 3 w 1 + 1 w 2

This equation says that whenever we encounter the vector w 3 , we can replace it with a specific linear combination of the vectors w 1 and w 2 . So, as before, the vector w 3 is not needed in the description of W , provided we have w 1 and w 2 available. In particular, a careful proof (such as is done in Example RSC5) would show that

W = w 1 , w 2

So W began life as the span of a set of four vectors, and we have now shown (utilizing solutions to a homogeneous system) that W can also be described as the span of a set of just two vectors. Convince yourself that we cannot go any further. In other words, it is not possible to dismiss either w 1 or w 2 in a similar fashion and winnow the set down to just one vector.

What was it about the original set of four vectors that allowed us to declare certain vectors as surplus? And just which vectors were we able to dismiss? And why did we have to stop once we had two vectors remaining? The answers to these questions motivate “linear independence,” our next section and next definition, and so are worth considering carefully now. ⊠

It is possible to have your computational device crank out the vector form of the solution set to a linear system of equations.  See: Computation VFSS.MMA

Subsection READ: Reading Questions

Subsection EXC: Exercises

C22 For each archetype that is a system of equations, consider the corresponding homogeneous system of equations. Write elements of the solution set to these homogeneous systems in vector form, as guaranteed by Theorem VFSLS. Then write the null space of the coefficient matrix of each system as the span of a set of vectors, as described in Theorem SSNS.
Archetype A
Archetype B
Archetype C
Archetype D/ Archetype E
Archetype F
Archetype G/ Archetype H
Archetype I
Archetype J

Contributed by  Robert Beezer Solution [ 381]

C23 Archetype K and Archetype L are defined as matrices. Use Theorem SSNS directly to find a set S so that S is the null space of the matrix. Do not make any reference to the associated homogeneous system of equations in your solution.  
Contributed by  Robert Beezer Solution [ 381]

C40 Suppose that S = 2 − 1 3 4 , 3 2 − 2 1 . Let W = S and let x = 5 8 − 1 2 − 5 . Is x ∈ W ? If so, provide an explicit linear combination that demonstrates this.  
Contributed by  Robert Beezer Solution [ 381]

C41 Suppose that S = 2 − 1 3 4 , 3 2 − 2 1 . Let W = S and let y = 5 1 3 5 . Is y ∈ W ? If so, provide an explicit linear combination that demonstrates this.  
Contributed by  Robert Beezer Solution [ 382]

C42 Suppose R = 2 − 1 3 4 0 , 1 1 2 2 − 1 , 3 − 1 0 3 − 2 . Is y = 1 − 1 − 8 − 4 − 3 in R ?  
Contributed by  Robert Beezer Solution [ 384]

C43 Suppose R = 2 − 1 3 4 0 , 1 1 2 2 − 1 , 3 − 1 0 3 − 2 . Is z = 1 1 5 3 1 in R ?  
Contributed by  Robert Beezer Solution [ 385]

C44 Suppose that S = − 1 2 1 , 3 1 2 , 1 5 4 , − 6 5 1 . Let W = S and let y = − 5 3 0 . Is y ∈ W ? If so, provide an explicit linear combination that demonstrates this.  
Contributed by  Robert Beezer Solution [ 387]

C45 Suppose that S = − 1 2 1 , 3 1 2 , 1 5 4 , − 6 5 1 . Let W = S and let w = 2 1 3 . Is w ∈ W ? If so, provide an explicit linear combination that demonstrates this.  
Contributed by  Robert Beezer Solution [ 389]

C50 Let A be the matrix below.
(a) Find a set S so that N A = S .
(b) If z = 3 − 5 1 2 , then show directly that z ∈ N A .
(c) Write z as a linear combination of the vectors in S .

A = 2 3 1 4 1 2 1 3 − 1 0 1 1

Contributed by  Robert Beezer Solution [ 390]

C60 For the matrix A below, find a set of vectors S so that the span of S equals the null space of A , S = N A .

A = 1 1 6 − 8 1 − 2 0 1 − 2 1 − 6 7

Contributed by  Robert Beezer Solution [ 393]

M10 Consider the set of all size 2 vectors in the Cartesian plane ℝ 2 .

Contributed by  Chris Black Solution [ 394]

M11 Consider the set of all size 3 vectors in Cartesian 3-space ℝ 3 .

Contributed by  Chris Black Solution [ 395]

M12 Let u = 1 3 − 2 and v = 2 − 2 1 .

Contributed by  Chris Black Solution [ 396]

M20 In Example SCAD we began with the four columns of the coefficient matrix of Archetype D, and used these columns in a span construction. Then we methodically argued that we could remove the last column, then the third column, and create the same set by just doing a span construction with the first two columns. We claimed we could not go any further, and had removed as many vectors as possible. Provide a convincing argument for why a third vector cannot be removed.  
Contributed by  Robert Beezer

M21 In the spirit of Example SCAD, begin with the four columns of the coefficient matrix of Archetype C, and use these columns in a span construction to build the set S . Argue that S can be expressed as the span of just three of the columns of the coefficient matrix (saying exactly which three) and in the spirit of Exercise SS.M20 argue that no one of these three vectors can be removed and still have a span construction create S .  
Contributed by  Robert Beezer Solution [ 397]

T10 Suppose that v 1 , v 2 ∈ ℂ m . Prove that

v 1 , v 2 = v 1 , v 2 , 5 v 1 + 3 v 2

Contributed by  Robert Beezer Solution [ 398]

T20 Suppose that S is a set of vectors from ℂ m . Prove that the zero vector, 0 , is an element of S .  
Contributed by  Robert Beezer Solution [ 400]

T21 Suppose that S is a set of vectors from ℂ m and x , y ∈ S . Prove that x + y ∈ S .  
Contributed by  Robert Beezer

T22 Suppose that S is a set of vectors from ℂ m , α ∈ ℂ , and x ∈ S . Prove that α x ∈ S .  
Contributed by  Robert Beezer

Subsection SOL: Solutions

C22 Contributed by  Robert Beezer Statement [ 375]
The vector form of the solutions obtained in this manner will involve precisely the vectors described in Theorem SSNS as providing the null space of the coefficient matrix of the system as a span. These vectors occur in each archetype in a description of the null space. Studying Example VFSAL may be of some help.

C23 Contributed by  Robert Beezer Statement [ 375]
Study Example NSDS to understand the correct approach to this question. The solution for each is listed in the Archetypes ( Appendix A) themselves.

C40 Contributed by  Robert Beezer Statement [ 375]
Rephrasing the question, we want to know if there are scalars α 1 and α 2 such that

α 1 2 − 1 3 4 + α 2 3 2 − 2 1 = 5 8 − 1 2 − 5

Theorem SLSLC allows us to rephrase the question again as a quest for solutions to the system of four equations in two unknowns with an augmented matrix given by

2 3 5 − 1 2 8 3 − 2 − 1 2 4 1 − 5

This matrix row-reduces to

1 0 − 2 0 1  3 0 0 0 0 0 0

From the form of this matrix, we can see that α 1 = − 2 and α 2 = 3 is an affirmative answer to our question. More convincingly,

( − 2 ) 2 − 1 3 4 + ( 3 ) 3 2 − 2 1 = 5 8 − 1 2 − 5

C41 Contributed by  Robert Beezer Statement [ 375]
Rephrasing the question, we want to know if there are scalars α 1 and α 2 such that

α 1 2 − 1 3 4 + α 2 3 2 − 2 1 = 5 1 3 5

Theorem SLSLC allows us to rephrase the question again as a quest for solutions to the system of four equations in two unknowns with an augmented matrix given by

2 3 5 − 1 2 1 3 − 2 3 4 1 5

This matrix row-reduces to

1 0 0 0 1  0 0 0 1 0 0 0

With a leading 1 in the last column of this matrix (Theorem RCLS) we can see that the system of equations has no solution, so there are no values for α 1 and α 2 that will allow us to conclude that y is in W . So y ∉ W .

C42 Contributed by  Robert Beezer Statement [ 376]
Form a linear combination, with unknown scalars, of R that equals y ,

a 1 2 − 1 3 4 0 + a 2 1 1 2 2 − 1 + a 3 3 − 1 0 3 − 2 = 1 − 1 − 8 − 4 − 3

We want to know if there are values for the scalars that make the vector equation true since that is the definition of membership in R . By Theorem SLSLC any such values will also be solutions to the linear system represented by the augmented matrix,

2 1 3 1 − 1 1 − 1 − 1 3 2 0 − 8 4 2 3 − 4 0 − 1 − 2 − 3

Row-reducing the matrix yields,

1 0 0 − 2 0 1  0 − 1 0 0 1 2 0 0 0 0 0 0 0 0

From this we see that the system of equations is consistent (Theorem RCLS), and has a unique solution. This solution will provide a linear combination of the vectors in R that equals y . So y ∈ R .

C43 Contributed by  Robert Beezer Statement [ 376]
Form a linear combination, with unknown scalars, of R that equals z ,

a 1 2 − 1 3 4 0 + a 2 1 1 2 2 − 1 + a 3 3 − 1 0 3 − 2 = 1 1 5 3 1

We want to know if there are values for the scalars that make the vector equation true since that is the definition of membership in R . By Theorem SLSLC any such values will also be solutions to the linear system represented by the augmented matrix,

2 1 3 1 − 1 1 − 1 1 3 2 0 5 4 2 3 3 0 − 1 − 2 1

Row-reducing the matrix yields,

1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1 0 0 0 0

With a leading 1 in the last column, the system is inconsistent (Theorem RCLS), so there are no scalars a 1 , a 2 , a 3 that will create a linear combination of the vectors in R that equal z . So z ∉ R .

C44 Contributed by  Robert Beezer Statement [ 376]
Form a linear combination, with unknown scalars, of S that equals y ,

a 1 − 1 2 1 + a 2 3 1 2 + a 3 1 5 4 + a 4 − 6 5 1 = − 5 3 0

We want to know if there are values for the scalars that make the vector equation true since that is the definition of membership in S . By Theorem SLSLC any such values will also be solutions to the linear system represented by the augmented matrix,

− 1 3 1 − 6 − 5 2 1 5 5 3 1 2 4 1 0

Row-reducing the matrix yields,

1 0 2 3 2 0 1  1 − 1 − 1 0 0 0 0 0

From this we see that the system of equations is consistent (Theorem RCLS), and has a infinitely many solutions. Any solution will provide a linear combination of the vectors in R that equals y . So y ∈ S , for example,

( − 1 0 ) − 1 2 1 + ( − 2 ) 3 1 2 + ( 3 ) 1 5 4 + ( 2 ) − 6 5 1 = − 5 3 0

C45 Contributed by  Robert Beezer Statement [ 376]
Form a linear combination, with unknown scalars, of S that equals w ,

a 1 − 1 2 1 + a 2 3 1 2 + a 3 1 5 4 + a 4 − 6 5 1 = 2 1 3

We want to know if there are values for the scalars that make the vector equation true since that is the definition of membership in S . By Theorem SLSLC any such values will also be solutions to the linear system represented by the augmented matrix,

− 1 3 1 − 6 2 2 1 5 5 1 1 2 4 1 3

Row-reducing the matrix yields,

1 0 2 3 0 0 1  1 − 1 0 0 0 0 0 1

With a leading 1 in the last column, the system is inconsistent (Theorem RCLS), so there are no scalars a 1 , a 2 , a 3 , a 4 that will create a linear combination of the vectors in S that equal w . So w ∉ S .

C50 Contributed by  Robert Beezer Statement [ 376]
(a) Theorem SSNS provides formulas for a set S with this property, but first we must row-reduce A

A →  RREF 1 0 − 1 − 1 0 1  1 2 0 0 0 0

x 3 and x 4 would be the free variables in the homogeneous system ℒ S A , 0 and Theorem SSNS provides the set S = z 1 , z 2 where

z 1 = 1 − 1 1 0 z 2 = 1 − 2 0 1

(b) Simply employ the components of the vector z as the variables in the homogeneous system ℒ S A , 0 . The three equations of this system evaluate as follows,

2 ( 3 ) + 3 ( − 5 ) + 1 ( 1 ) + 4 ( 2 ) = 0 1 ( 3 ) + 2 ( − 5 ) + 1 ( 1 ) + 3 ( 2 ) = 0 − 1 ( 3 ) + 0 ( − 5 ) + 1 ( 1 ) + 1 ( 2 ) = 0

Since each result is zero, z qualifies for membership in N A .

(c) By Theorem SSNS we know this must be possible (that is the moral of this exercise). Find scalars α 1 and α 2 so that

α 1 z 1 + α 2 z 2 = α 1 1 − 1 1 0 + α 2 1 − 2 0 1 = 3 − 5 1 2 = z

Theorem SLSLC allows us to convert this question into a question about a system of four equations in two variables. The augmented matrix of this system row-reduces to

1 0 1 0 1  2 0 0 0 0 0 0

A solution is α 1 = 1 and α 2 = 2 . (Notice too that this solution is unique!)

C60 Contributed by  Robert Beezer Statement [ 377]
Theorem SSNS says that if we find the vector form of the solutions to the homogeneous system ℒ S A , 0 , then the fixed vectors (one per free variable) will have the desired property. Row-reduce A , viewing it as the augmented matrix of a homogeneous system with an invisible columns of zeros as the last column,

1 0 4 − 5 0 1  2 − 3 0 0 0 0

Moving to the vector form of the solutions (Theorem VFSLS), with free variables x 3 and x 4 , solutions to the consistent system (it is homogeneous, Theorem HSC) can be expressed as

x 1 x 2 x 3 x 4 = x 3 − 4 − 2 1 0 + x 4 5 3 0 1

Then with S given by

S = − 4 − 2 1 0 , 5 3 0 1

Theorem SSNS guarantees that

N A = S = − 4 − 2 1 0 , 5 3 0 1

M10 Contributed by  Chris Black Statement [ 378]

M11 Contributed by  Chris Black Statement [ 378]

M12 Contributed by  Chris Black Statement [ 378]

M21 Contributed by  Robert Beezer Statement [ 379]
If the columns of the coefficient matrix from Archetype C are named u 1 , u 2 , u 3 , u 4 then we can discover the equation

( − 2 ) u 1 + ( − 3 ) u 2 + u 3 + u 4 = 0

by building a homogeneous system of equations and viewing a solution to the system as scalars in a linear combination via Theorem SLSLC. This particular vector equation can be rearranged to read

u 4 = ( 2 ) u 1 + ( 3 ) u 2 + ( − 1 ) u 3

This can be interpreted to mean that u 4 is unnecessary in u 1 , u 2 , u 3 , u 4 , so that

u 1 , u 2 , u 3 , u 4 = u 1 , u 2 , u 3

If we try to repeat this process and find a linear combination of u 1 , u 2 , u 3 that equals the zero vector, we will fail. The required homogeneous system of equations (via Theorem SLSLC) has only a trivial solution, which will not provide the kind of equation we need to remove one of the three remaining vectors.

T10 Contributed by  Robert Beezer Statement [ 379]
This is an equality of sets, so Definition SE applies.

First show that X = v 1 , v 2 ⊆ v 1 , v 2 , 5 v 1 + 3 v 2 = Y .
Choose x ∈ X . Then x = a 1 v 1 + a 2 v 2 for some scalars a 1 and a 2 . Then,

x = a 1 v 1 + a 2 v 2 = a 1 v 1 + a 2 v 2 + 0 ( 5 v 1 + 3 v 2 )

which qualifies x for membership in Y , as it is a linear combination of v 1 , v 2 , 5 v 1 + 3 v 2 .

Now show the opposite inclusion, Y = v 1 , v 2 , 5 v 1 + 3 v 2 ⊆ v 1 , v 2 = X .
Choose y ∈ Y . Then there are scalars a 1 , a 2 , a 3 such that

y = a 1 v 1 + a 2 v 2 + a 3 ( 5 v 1 + 3 v 2 )

Rearranging, we obtain,

y = a 1 v 1 + a 2 v 2 + a 3 ( 5 v 1 + 3 v 2 ) = a 1 v 1 + a 2 v 2 + 5 a 3 v 1 + 3 a 3 v 2   Property DVAC = a 1 v 1 + 5 a 3 v 1 + a 2 v 2 + 3 a 3 v 2   Property CC = ( a 1 + 5 a 3 ) v 1 + ( a 2 + 3 a 3 ) v 2   Property DSAC

This is an expression for y as a linear combination of v 1 and v 2 , earning y membership in X . Since X is a subset of Y , and vice versa, we see that X = Y , as desired.

T20 Contributed by  Robert Beezer Statement [ 380]
No matter what the elements of the set S are, we can choose the scalars in a linear combination to all be zero. Suppose that S = v 1 , v 2 , v 3 , … , v p . Then compute

0 v 1 + 0 v 2 + 0 v 3 + ⋯ + 0 v p = 0 + 0 + 0 + ⋯ + 0 = 0

But what if we choose S to be the empty set? The convention is that the empty sum in Definition SSCV evaluates to “zero,” in this case this is the zero vector.

Section O  Orthogonality

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we define a couple more operations with vectors, and prove a few theorems. At first blush these definitions and results will not appear central to what follows, but we will make use of them at key points in the remainder of the course (such as Section MINM, Section OD). Because we have chosen to use ℂ as our set of scalars, this subsection is a bit more, uh, … complex than it would be for the real numbers. We’ll explain as we go along how things get easier for the real numbers ℝ . If you haven’t already, now would be a good time to review some of the basic properties of arithmetic with complex numbers described in Section CNO. With that done, we can extend the basics of complex number arithmetic to our study of vectors in ℂ m .

Parent topic: Chapter V  Vectors

Previous topic: Section SS  Spanning Sets

Next topic: Chapter M  Matrices

Subsection CAV: Complex Arithmetic and Vectors

We know how the addition and multiplication of complex numbers is employed in defining the operations for vectors in ℂ m (Definition CVA and Definition CVSM). We can also extend the idea of the conjugate to vectors.

Definition  CCCV
Complex Conjugate of a Column Vector
Suppose that u is a vector from ℂ m . Then the conjugate of the vector, u ¯ , is defined by

u ¯ i = u i ¯   1 ≤ i ≤ m

(This definition contains Notation CCCV.) △

With this definition we can show that the conjugate of a column vector behaves as we would expect with regard to vector addition and scalar multiplication.

Theorem  CRVA
Conjugation Respects Vector Addition
Suppose x and y are two vectors from ℂ m . Then

x + y ¯ = x ¯ + y ¯

□

Proof   For each 1 ≤ i ≤ m ,

x + y ¯ i = x + y i ¯   Definition CCCV = x i + y i ¯   Definition CVA = x i ¯ + y i ¯   Theorem CCRA = x ¯ i + y ¯ i   Definition CCCV = x ¯ + y ¯ i   Definition CVA

Then by Definition CVE we have x + y ¯ = x ¯ + y ¯ . ■

Theorem  CRSM
Conjugation Respects Vector Scalar Multiplication
Suppose x is a vector from ℂ m , and α ∈ ℂ is a scalar. Then

α x ¯ = α ¯ x ¯

□

Proof   For 1 ≤ i ≤ m ,

α x ¯ i = α x i ¯   Definition CCCV = α x i ¯   Definition CVSM = α ¯ x i ¯   Theorem CCRM = α ¯ x ¯ i   Definition CCCV = α ¯ x ¯ i   Definition CVSM

Then by Definition CVE we have α x ¯ = α ¯ x ¯ . ■

These two theorems together tell us how we can “push” complex conjugation through linear combinations.

Subsection IP: Inner products

Definition  IP
Inner Product
Given the vectors u , v ∈ ℂ m the inner product of u and v is the scalar quantity in ℂ ,

u , v = u 1 v 1 ¯ + u 2 v 2 ¯ + u 3 v 3 ¯ + ⋯ + u m v m ¯ = ∑ i = 1 m u i v i ¯

(This definition contains Notation IP.) △

This operation is a bit different in that we begin with two vectors but produce a scalar. Computing one is straightforward.

Example  CSIP
Computing some inner products
The scalar product of

u = 2 + 3 i 5 + 2 i − 3 + i  and v = 1 + 2 i − 4 + 5 i 0 + 5 i

is

u , v = ( 2 + 3 i ) ( 1 + 2 i ¯ ) + ( 5 + 2 i ) ( − 4 + 5 i ¯ ) + ( − 3 + i ) ( 0 + 5 i ¯ ) = ( 2 + 3 i ) ( 1 − 2 i ) + ( 5 + 2 i ) ( − 4 − 5 i ) + ( − 3 + i ) ( 0 − 5 i ) = ( 8 − i ) + ( − 1 0 − 3 3 i ) + ( 5 + 1 5 i ) = 3 − 1 9 i

The scalar product of

w = 2 4 − 3 2 8  and x = 3 1 0 − 1 − 2

is

w , x = 2 ( 3 ¯ ) + 4 ( 1 ¯ ) + ( − 3 ) ( 0 ¯ ) + 2 ( − 1 ¯ ) + 8 ( − 2 ¯ ) = 2 ( 3 ) + 4 ( 1 ) + ( − 3 ) 0 + 2 ( − 1 ) + 8 ( − 2 ) = − 8 .

⊠

In the case where the entries of our vectors are all real numbers (as in the second part of Example CSIP), the computation of the inner product may look familiar and be known to you as a dot product or scalar product. So you can view the inner product as a generalization of the scalar product to vectors from ℂ m (rather than ℝ m ).

Also, note that we have chosen to conjugate the entries of the second vector listed in the inner product, while many authors choose to conjugate entries from the first component. It really makes no difference which choice is made, it just requires that subsequent definitions and theorems are consistent with the choice. You can study the conclusion of Theorem IPAC as an explanation of the magnitude of the difference that results from this choice. But be careful as you read other treatments of the inner product or its use in applications, and be sure you know ahead of time which choice has been made.

There are several quick theorems we can now prove, and they will each be useful later.

Theorem  IPVA
Inner Product and Vector Addition
Suppose u , v , w ∈ ℂ m . Then

 1. u + v , w = u , w + v , w  2. u , v + w = u , v + u , w

□

Proof   The proofs of the two parts are very similar, with the second one requiring just a bit more effort due to the conjugation that occurs. We will prove part 2 and you can prove part 1 ( Exercise O.T10).

u , v + w = ∑ i = 1 m u i v + w i ¯   Definition IP = ∑ i = 1 m u i ( v i + w i ¯ )   Definition CVA = ∑ i = 1 m u i ( v i ¯ + w i ¯ )   Theorem CCRA = ∑ i = 1 m u i v i ¯ + u i w i ¯   Property DCN = ∑ i = 1 m u i v i ¯ + ∑ i = 1 m u i w i ¯   Property CACN = u , v + u , w   Definition IP

■

Theorem  IPSM
Inner Product and Scalar Multiplication
Suppose u , v ∈ ℂ m and α ∈ ℂ . Then

 1. α u , v = α u , v  2. u , α v = α ¯ u , v

□

Proof   The proofs of the two parts are very similar, with the second one requiring just a bit more effort due to the conjugation that occurs. We will prove part 2 and you can prove part 1 ( Exercise O.T11).

u , α v = ∑ i = 1 m u i α v i ¯   Definition IP = ∑ i = 1 m u i α v i ¯   Definition CVSM = ∑ i = 1 m u i α ¯ v i ¯   Theorem CCRM = ∑ i = 1 m α ¯ u i v i ¯   Property CMCN = α ¯ ∑ i = 1 m u i v i ¯   Property DCN = α ¯ u , v   Definition IP

■

Theorem  IPAC
Inner Product is Anti-Commutative
Suppose that u and v are vectors in ℂ m . Then u , v = v , u ¯ . □

Proof  

u , v = ∑ i = 1 m u i v i ¯   Definition IP = ∑ i = 1 m u i ¯ ¯ v i ¯   Theorem CCT = ∑ i = 1 m u i ¯ v i ¯   Theorem CCRM = ∑ i = 1 m u i ¯ v i ¯   Theorem CCRA = ∑ i = 1 m v i u i ¯ ¯   Property CMCN = v , u ¯   Definition IP

■

Subsection N: Norm

If treating linear algebra in a more geometric fashion, the length of a vector occurs naturally, and is what you would expect from its name. With complex numbers, we will define a similar function. Recall that if c is a complex number, then c denotes its modulus ( Definition MCN).

Definition  NV
Norm of a Vector
The norm of the vector u is the scalar quantity in ℂ

u = u 1 2 + u 2 2 + u 3 2 + ⋯ + u m 2 = ∑ i = 1 m u i 2

(This definition contains Notation NV.) △

Computing a norm is also easy to do.

Example  CNSV
Computing the norm of some vectors
The norm of

u = 3 + 2 i 1 − 6 i 2 + 4 i 2 + i

is

u = 3 + 2 i 2 + 1 − 6 i 2 + 2 + 4 i 2 + 2 + i 2 = 1 3 + 3 7 + 2 0 + 5 = 7 5 = 5 3 .

The norm of

v = 3 − 1 2 4 − 3

is

v = 3 2 + − 1 2 + 2 2 + 4 2 + − 3 2 = 3 2 + 1 2 + 2 2 + 4 2 + 3 2 = 3 9 .

⊠

Notice how the norm of a vector with real number entries is just the length of the vector. Inner products and norms are related by the following theorem.

Theorem  IPN
Inner Products and Norms
Suppose that u is a vector in ℂ m . Then u 2 = u , u . □

Proof  

u 2 = ∑ i = 1 m u i 2 2   Definition NV = ∑ i = 1 m u i 2 = ∑ i = 1 m u i u i ¯   Definition MCN = u , u   Definition IP

■

When our vectors have entries only from the real numbers Theorem IPN says that the dot product of a vector with itself is equal to the length of the vector squared.

Theorem  PIP
Positive Inner Products
Suppose that u is a vector in ℂ m . Then u , u ≥ 0 with equality if and only if u = 0 . □

Proof   From the proof of Theorem IPN we see that

u , u = u 1 2 + u 2 2 + u 3 2 + ⋯ + u m 2

Since each modulus is squared, every term is positive, and the sum must also be positive. (Notice that in general the inner product is a complex number and cannot be compared with zero, but in the special case of u , u the result is a real number.) The phrase, “with equality if and only if” means that we want to show that the statement u , u = 0 (i.e. with equality) is equivalent (“if and only if”) to the statement u = 0 .

If u = 0 , then it is a straightforward computation to see that u , u = 0 . In the other direction, assume that u , u = 0 . As before, u , u is a sum of moduli. So we have

0 = u , u = u 1 2 + u 2 2 + u 3 2 + ⋯ + u m 2

Now we have a sum of squares equaling zero, so each term must be zero. Then by similar logic, u i = 0 will imply that u i = 0 , since 0 + 0 i is the only complex number with zero modulus. Thus every entry of u is zero and so u = 0 , as desired. ■

Notice that Theorem PIP contains three implications:

u ∈ ℂ m ⇒ u , u ≥ 0 u = 0 ⇒ u , u = 0 u , u = 0 ⇒ u = 0

The results contained in Theorem PIP are summarized by saying “the inner product is positive definite.”

Subsection OV: Orthogonal Vectors

“Orthogonal” is a generalization of “perpendicular.” You may have used mutually perpendicular vectors in a physics class, or you may recall from a calculus class that perpendicular vectors have a zero dot product. We will now extend these ideas into the realm of higher dimensions and complex scalars.

Definition  OV
Orthogonal Vectors
A pair of vectors, u and v , from ℂ m are orthogonal if their inner product is zero, that is, u , v = 0 . △

Example  TOV
Two orthogonal vectors
The vectors

u = 2 + 3 i 4 − 2 i 1 + i 1 + i v = 1 − i 2 + 3 i 4 − 6 i 1

are orthogonal since

u , v = ( 2 + 3 i ) ( 1 + i ) + ( 4 − 2 i ) ( 2 − 3 i ) + ( 1 + i ) ( 4 + 6 i ) + ( 1 + i ) ( 1 ) = ( − 1 + 5 i ) + ( 2 − 1 6 i ) + ( − 2 + 1 0 i ) + ( 1 + i ) = 0 + 0 i .

⊠

We extend this definition to whole sets by requiring vectors to be pairwise orthogonal. Despite using the same word, careful thought about what objects you are using will eliminate any source of confusion.

Definition  OSV
Orthogonal Set of Vectors
Suppose that S = u 1 , u 2 , u 3 , … , u n is a set of vectors from ℂ m . Then S is an orthogonal set if every pair of different vectors from S is orthogonal, that is u i , u j = 0 whenever i ≠ j . △

We now define the prototypical orthogonal set, which we will reference repeatedly.

Definition  SUV
Standard Unit Vectors
Let e j ∈ ℂ m , 1 ≤ j ≤ m denote the column vectors defined by

e j i = 0  if  i ≠ j 1  if  i = j

Then the set

e 1 , e 2 , e 3 , … , e m = e j 1 ≤ j ≤ m

is the set of standard unit vectors in ℂ m .

(This definition contains Notation SUV.) △

Notice that e j is identical to column j of the m × m identity matrix I m ( Definition IM). This observation will often be useful. It is not hard to see that the set of standard unit vectors is an orthogonal set. We will reserve the notation e i for these vectors.

Example  SUVOS
Standard Unit Vectors are an Orthogonal Set
Compute the inner product of two distinct vectors from the set of standard unit vectors ( Definition SUV), say e i , e j , where i ≠ j ,

e i , e j = 0 0 ¯ + 0 0 ¯ + ⋯ + 1 0 ¯ + ⋯ + 0 0 ¯ + ⋯ + 0 1 ¯ + ⋯ + 0 0 ¯ + 0 0 ¯ = 0 ( 0 ) + 0 ( 0 ) + ⋯ + 1 ( 0 ) + ⋯ + 0 ( 1 ) + ⋯ + 0 ( 0 ) + 0 ( 0 ) = 0

So the set e 1 , e 2 , e 3 , … , e m is an orthogonal set. ⊠

Example  AOS
An orthogonal set
The set

x 1 , x 2 , x 3 , x 4 = 1 + i 1 1 − i i , 1 + 5 i 6 + 5 i − 7 − i 1 − 6 i , − 7 + 3 4 i − 8 − 2 3 i − 1 0 + 2 2 i 3 0 + 1 3 i , − 2 − 4 i 6 + i 4 + 3 i 6 − i

is an orthogonal set. Since the inner product is anti-commutative ( Theorem IPAC) we can test pairs of different vectors in any order. If the result is zero, then it will also be zero if the inner product is computed in the opposite order. This means there are six pairs of different vectors to use in an inner product computation. We’ll do two and you can practice your inner products on the other four.

x 1 , x 3 = ( 1 + i ) ( − 7 − 3 4 i ) + ( 1 ) ( − 8 + 2 3 i ) + ( 1 − i ) ( − 1 0 − 2 2 i ) + ( i ) ( 3 0 − 1 3 i ) = ( 2 7 − 4 1 i ) + ( − 8 + 2 3 i ) + ( − 3 2 − 1 2 i ) + ( 1 3 + 3 0 i ) = 0 + 0 i  and x 2 , x 4 = ( 1 + 5 i ) ( − 2 + 4 i ) + ( 6 + 5 i ) ( 6 − i ) + ( − 7 − i ) ( 4 − 3 i ) + ( 1 − 6 i ) ( 6 + i ) = ( − 2 2 − 6 i ) + ( 4 1 + 2 4 i ) + ( − 3 1 + 1 7 i ) + ( 1 2 − 3 5 i ) = 0 + 0 i

⊠

So far, this section has seen lots of definitions, and lots of theorems establishing un-surprising consequences of those definitions. But here is our first theorem that suggests that inner products and orthogonal vectors have some utility. It is also one of our first illustrations of how to arrive at linear independence as the conclusion of a theorem.

Theorem  OSLI
Orthogonal Sets are Linearly Independent
Suppose that S is an orthogonal set of nonzero vectors. Then S is linearly independent. □

Proof   Let S = u 1 , u 2 , u 3 , … , u n be an orthogonal set of nonzero vectors. To prove the linear independence of S , we can appeal to the definition (Definition LICV) and begin with an arbitrary relation of linear dependence (Definition RLDCV),

α 1 u 1 + α 2 u 2 + α 3 u 3 + ⋯ + α n u n = 0 .

Then, for every 1 ≤ i ≤ n , we have

α i = 1 u i , u i α i u i , u i   Theorem PIP = 1 u i , u i α 1 ( 0 ) + α 2 ( 0 ) + ⋯ + α i u i , u i + ⋯ + α n ( 0 )   Property ZCN = 1 u i , u i α 1 u 1 , u i + ⋯ + α i u i , u i + ⋯ + α n u n , u i   Definition OSV = 1 u i , u i α 1 u 1 , u i + α 2 u 2 , u i + ⋯ + α n u n , u i   Theorem IPSM = 1 u i , u i α 1 u 1 + α 2 u 2 + α 3 u 3 + ⋯ + α n u n , u i   Theorem IPVA = 1 u i , u i 0 , u i   Definition RLDCV = 1 u i , u i 0   Definition IP = 0   Property ZCN

So we conclude that α i = 0 for all 1 ≤ i ≤ n in any relation of linear dependence on S . But this says that S is a linearly independent set since the only way to form a relation of linear dependence is the trivial way (Definition LICV). Boom! ■

Subsection GSP: Gram-Schmidt Procedure

The Gram-Schmidt Procedure is really a theorem. It says that if we begin with a linearly independent set of p vectors, S , then we can do a number of calculations with these vectors and produce an orthogonal set of p vectors, T , so that S = T . Given the large number of computations involved, it is indeed a procedure to do all the necessary computations, and it is best employed on a computer. However, it also has value in proofs where we may on occasion wish to replace a linearly independent set by an orthogonal set.

This is our first occasion to use the technique of “mathematical induction” for a proof, a technique we will see again several times, especially in Chapter D. So study the simple example described in Technique I first. 

Theorem  GSP
Gram-Schmidt Procedure
Suppose that S = v 1 , v 2 , v 3 , … , v p is a linearly independent set of vectors in ℂ m . Define the vectors u i , 1 ≤ i ≤ p by

u i = v i − v i , u 1 u 1 , u 1 u 1 − v i , u 2 u 2 , u 2 u 2 − v i , u 3 u 3 , u 3 u 3 − ⋯ − v i , u i − 1 u i − 1 , u i − 1 u i − 1

Then if T = u 1 , u 2 , u 3 , … , u p , then T is an orthogonal set of non-zero vectors, and T = S . □

Proof   We will prove the result by using induction on p ( Technique I). To begin, we prove that T has the desired properties when p = 1 . In this case u 1 = v 1 and T = u 1 = v 1 = S . Because S and T are equal, S = T . Equally trivial, T is an orthogonal set. If u 1 = 0 , then S would be a linearly dependent set, a contradiction.

Suppose that the theorem is true for any set of p − 1 linearly independent vectors. Let S = v 1 , v 2 , v 3 , … , v p be a linearly independent set of p vectors. Then S ′ = v 1 , v 2 , v 3 , … , v p − 1 is also linearly independent. So we can apply the theorem to S ′ and construct the vectors T ′ = u 1 , u 2 , u 3 , … , u p − 1 . T ′ is therefore an orthogonal set of nonzero vectors and S ′ = T ′ . Define

u p = v p − v p , u 1 u 1 , u 1 u 1 − v p , u 2 u 2 , u 2 u 2 − v p , u 3 u 3 , u 3 u 3 − ⋯ − v p , u p − 1 u p − 1 , u p − 1 u p − 1

and let T = T ′ ∪ u p . We need to now show that T has several properties by building on what we know about T ′ . But first notice that the above equation has no problems with the denominators ( u i , u i ) being zero, since the u i are from T ′ , which is composed of nonzero vectors.

We show that T = S , by first establishing that T ⊆ S . Suppose x ∈ T , so

x = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a p u p

The term a p u p is a linear combination of vectors from T ′ and the vector v p , while the remaining terms are a linear combination of vectors from T ′ . Since T ′ = S ′ , any term that is a multiple of a vector from T ′ can be rewritten as a linear combination of vectors from S ′ . The remaining term a p v p is a multiple of a vector in S . So we see that x can be rewritten as a linear combination of vectors from S , i.e.  x ∈ S .

To show that S ⊆ T , begin with y ∈ S , so

y = a 1 v 1 + a 2 v 2 + a 3 v 3 + ⋯ + a p v p

Rearrange our defining equation for u p by solving for v p . Then the term a p v p is a multiple of a linear combination of elements of T . The remaining terms are a linear combination of v 1 , v 2 , v 3 , … , v p − 1 , hence an element of S ′ = T ′ . Thus these remaining terms can be written as a linear combination of the vectors in T ′ . So y is a linear combination of vectors from T , i.e.  y ∈ T .

The elements of T ′ are nonzero, but what about u p ? Suppose to the contrary that u p = 0 ,

0 = u p = v p − v p , u 1 u 1 , u 1 u 1 − v p , u 2 u 2 , u 2 u 2 − v p , u 3 u 3 , u 3 u 3 − ⋯ − v p , u p − 1 u p − 1 , u p − 1 u p − 1 v p = v p , u 1 u 1 , u 1 u 1 + v p , u 2 u 2 , u 2 u 2 + v p , u 3 u 3 , u 3 u 3 + ⋯ + v p , u p − 1 u p − 1 , u p − 1 u p − 1

Since S ′ = T ′ we can write the vectors u 1 , u 2 , u 3 , … , u p − 1 on the right side of this equation in terms of the vectors v 1 , v 2 , v 3 , … , v p − 1 and we then have the vector v p expressed as a linear combination of the other p − 1 vectors in S , implying that S is a linearly dependent set (Theorem DLDS), contrary to our lone hypothesis about S .

Finally, it is a simple matter to establish that T is an orthogonal set, though it will not appear so simple looking. Think about your objects as you work through the following — what is a vector and what is a scalar. Since T ′ is an orthogonal set by induction, most pairs of elements in T are already known to be orthogonal. We just need to test “new” inner products, between u p and u i , for 1 ≤ i ≤ p − 1 . Here we go, using summation notation,

u p , u i = v p − ∑ k = 1 p − 1 v p , u k u k , u k u k , u i = v p , u i − ∑ k = 1 p − 1 v p , u k u k , u k u k , u i   Theorem IPVA = v p , u i − ∑ k = 1 p − 1 v p , u k u k , u k u k , u i   Theorem IPVA = v p , u i − ∑ k = 1 p − 1 v p , u k u k , u k u k , u i   Theorem IPSM = v p , u i − v p , u i u i , u i u i , u i − ∑ k ≠ i v p , u k u k , u k ( 0 )  Induction Hypothesis = v p , u i − v p , u i − ∑ k ≠ i 0 = 0

■

Example  GSTV
Gram-Schmidt of three vectors
We will illustrate the Gram-Schmidt process with three vectors. Begin with the linearly independent (check this!) set

S = v 1 , v 2 , v 3 = 1 1 + i 1 , − i 1 1 + i , 0 i i

Then

u 1 = v 1 = 1 1 + i 1 u 2 = v 2 − v 2 , u 1 u 1 , u 1 u 1 = 1 4 − 2 − 3 i 1 − i 2 + 5 i u 3 = v 3 − v 3 , u 1 u 1 , u 1 u 1 − v 3 , u 2 u 2 , u 2 u 2 = 1 1 1 − 3 − i 1 + 3 i − 1 − i

and

T = u 1 , u 2 , u 3 = 1 1 + i 1 , 1 4 − 2 − 3 i 1 − i 2 + 5 i , 1 1 1 − 3 − i 1 + 3 i − 1 − i

is an orthogonal set (which you can check) of nonzero vectors and T = S (all by Theorem GSP). Of course, as a by-product of orthogonality, the set T is also linearly independent ( Theorem OSLI). ⊠

One final definition related to orthogonal vectors.

Definition  ONS
OrthoNormal Set
Suppose S = u 1 , u 2 , u 3 , … , u n is an orthogonal set of vectors such that u i = 1 for all 1 ≤ i ≤ n . Then S is an orthonormal set of vectors. △

Once you have an orthogonal set, it is easy to convert it to an orthonormal set — multiply each vector by the reciprocal of its norm, and the resulting vector will have norm 1. This scaling of each vector will not affect the orthogonality properties (apply Theorem IPSM).

Example  ONTV
Orthonormal set, three vectors
The set

T = u 1 , u 2 , u 3 = 1 1 + i 1 , 1 4 − 2 − 3 i 1 − i 2 + 5 i , 1 1 1 − 3 − i 1 + 3 i − 1 − i

from Example GSTV is an orthogonal set. We compute the norm of each vector,

u 1 = 2 u 2 = 1 2 1 1 u 3 = 2 1 1

Converting each vector to a norm of 1, yields an orthonormal set,

w 1 = 1 2 1 1 + i 1 w 2 = 1 1 2 1 1 1 4 − 2 − 3 i 1 − i 2 + 5 i = 1 2 1 1 − 2 − 3 i 1 − i 2 + 5 i w 3 = 1 2 1 1 1 1 1 − 3 − i 1 + 3 i − 1 − i = 1 2 2 − 3 − i 1 + 3 i − 1 − i

⊠

Example  ONFV
Orthonormal set, four vectors
As an exercise convert the linearly independent set

S = 1 + i 1 1 − i i , i 1 + i − 1 − i , i − i − 1 + i 1 , − 1 − i i 1 − 1

to an orthogonal set via the Gram-Schmidt Process ( Theorem GSP) and then scale the vectors to norm 1 to create an orthonormal set. You should get the same set you would if you scaled the orthogonal set of Example AOS to become an orthonormal set. ⊠

It is crazy to do all but the simplest and smallest instances of the Gram-Schmidt procedure by hand. Well, OK, maybe just once or twice to get a good understanding of Theorem GSP. After that, let a machine do the work for you. That’s what they are for.  See: Computation GSP.MMA

We will see orthonormal sets again in Subsection MINM.UM. They are intimately related to unitary matrices ( Definition UM) through Theorem CUMOS. Some of the utility of orthonormal sets is captured by Theorem COB in Subsection B.OBC. Orthonormal sets appear once again in Section OD where they are key in orthonormal diagonalization.

Subsection READ: Reading Questions

Subsection EXC: Exercises

C20 Complete Example AOS by verifying that the four remaining inner products are zero.  
Contributed by  Robert Beezer

C21 Verify that the set T created in Example GSTV by the Gram-Schmidt Procedure is an orthogonal set.  
Contributed by  Robert Beezer

M60 Suppose that u , v , w ⊆ ℂ n is an orthonormal set. Prove that u + v is not orthogonal to v + w .  
Contributed by  Manley Perkel

T10 Prove part 1 of the conclusion of Theorem IPVA.  
Contributed by  Robert Beezer

T11 Prove part 1 of the conclusion of Theorem IPSM.  
Contributed by  Robert Beezer

T20 Suppose that u , v , w ∈ ℂ n , α , β ∈ ℂ and u is orthogonal to both v and w . Prove that u is orthogonal to α v + β w .  
Contributed by  Robert Beezer Solution [ 542]

T30 Suppose that the set S in the hypothesis of Theorem GSP is not just linearly independent, but is also orthogonal. Prove that the set T created by the Gram-Schmidt procedure is equal to S . (Note that we are getting a stronger conclusion than T = S — the conclusion is that T = S .) In other words, it is pointless to apply the Gram-Schmidt procedure to a set that is already orthogonal.  
Contributed by  Steve Canfield

Subsection SOL: Solutions

T20 Contributed by  Robert Beezer Statement [ 540]
Vectors are orthogonal if their inner product is zero ( Definition OV), so we compute,

α v + β w , u = α v , u + β w , u   Theorem IPVA = α v , u + β w , u   Theorem IPSM = α 0 + β 0   Definition OV = 0

So by Definition OV, u and α v + β w are an orthogonal pair of vectors.

Annotated Acronyms V: Vectors

Theorem VSPCV
These are the fundamental rules for working with the addition, and scalar multiplication, of column vectors. We will see something very similar in the next chapter (Theorem VSPM) and then this will be generalized into what is arguably our most important definition, Definition VS.

Theorem SLSLC
Vector addition and scalar multiplication are the two fundamental operations on vectors, and linear combinations roll them both into one. Theorem SLSLC connects linear combinations with systems of equations. This one we will see often enough that it is worth memorizing.

Theorem PSPHS
This theorem is interesting in its own right, and sometimes the vaugeness surrounding the choice of z can seem mysterious. But we list it here because we will see an important theorem in Section ILT which will generalize this result ( Theorem KPI).

Theorem LIVRN
If you have a set of column vectors, this is the fastest computational approach to determine if the set is linearly independent. Make the vectors the columns of a matrix, row-reduce, compare r and n . That’s it — and you always get an answer. Put this one in your toolkit.

Theorem BNS
We will have several theorems (all listed in these “Annotated Acronyms” sections) whose conclusions will provide a linearly independent set of vectors whose span equals some set of interest (the null space here). While the notation in this theorem might appear gruesome, in practice it can become very routine to apply. So practice this one — we’ll be using it all through the book.

Theorem BS
As promised, another theorem that provides a linearly independent set of vectors whose span equals some set of interest (a span now). You can use this one to clean up any span.

Chapter 3. Chapter M  Matrices

We have made frequent use of matrices for solving systems of equations, and we have begun to investigate a few of their properties, such as the null space and nonsingularity. In this chapter, we will take a more systematic approach to the study of matrices.

Section MM  Matrix Multiplication

Section MISLE  Matrix Inverses and Systems of Linear Equations

Section MINM  Matrix Inverses and Nonsingular Matrices

Section CRS  Column and Row Spaces

Parent topic: Part C  Core

Previous topic: Section O  Orthogonality

Next topic: Section MM  Matrix Multiplication

Section MM  Matrix Multiplication

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

We know how to add vectors and how to multiply them by scalars. Together, these operations give us the possibility of making linear combinations. Similarly, we know how to add matrices and how to multiply matrices by scalars. In this section we mix all these ideas together and produce an operation known as “matrix multiplication.” This will lead to some results that are both surprising and central. We begin with a definition of how to multiply a vector by a matrix.

Parent topic: Chapter M  Matrices

Previous topic: Chapter M  Matrices

Next topic: Section MISLE  Matrix Inverses and Systems of Linear Equations

Subsection MVP: Matrix-Vector Product

We have repeatedly seen the importance of forming linear combinations of the columns of a matrix. As one example of this, the oft-used Theorem SLSLC, said that every solution to a system of linear equations gives rise to a linear combination of the column vectors of the coefficient matrix that equals the vector of constants. This theorem, and others, motivate the following central definition.

Definition  MVP
Matrix-Vector Product
Suppose A is an m × n matrix with columns A 1 , A 2 , A 3 , … , A n and u is a vector of size n . Then the matrix-vector product of A with u is the linear combination

A u = u 1 A 1 + u 2 A 2 + u 3 A 3 + ⋯ + u n A n

(This definition contains Notation MVP.) △

So, the matrix-vector product is yet another version of “multiplication,” at least in the sense that we have yet again overloaded juxtaposition of two symbols as our notation. Remember your objects, an m × n matrix times a vector of size n will create a vector of size m . So if A is rectangular, then the size of the vector changes. With all the linear combinations we have performed so far, this computation should now seem second nature.

Example  MTV
A matrix times a vector
Consider

A = 1 4 2 3 4 − 3 2 0 1 − 2 1 6 − 3 − 1 5 u = 2 1 − 2 3 − 1

Then

A u = 2 1 − 3 1 + 1 4 2 6 + ( − 2 ) 2 0 − 3 + 3 3 1 − 1 + ( − 1 ) 4 − 2 5 = 7 1 6 .

⊠

We can now represent systems of linear equations compactly with a matrix-vector product (Definition MVP) and column vector equality (Definition CVE). This finally yields a very popular alternative to our unconventional ℒ S A , b notation.

Theorem  SLEMM
Systems of Linear Equations as Matrix Multiplication
The set of solutions to the linear system ℒ S A , b equals the set of solutions for x in the vector equation A x = b . □

Proof   This theorem says that two sets (of solutions) are equal. So we need to show that one set of solutions is a subset of the other, and vice versa ( Definition SE). Let A 1 , A 2 , A 3 , … , A n be the columns of A . Both of these set inclusions then follow from the following chain of equivalences ( Technique E),

x   is a solution to  ℒ S A , b ⇔ x 1 A 1 + x 2 A 2 + x 3 A 3 + ⋯ + x n A n = b   Theorem SLSLC ⇔ x   is a solution to  A x = b   Definition MVP

■

Example  MNSLE
Matrix notation for systems of linear equations
Consider the system of linear equations from Example NSLE.

2 x 1 + 4 x 2 − 3 x 3 + 5 x 4 + x 5 = 9 3 x 1 + x 2 + x 4 − 3 x 5 = 0 − 2 x 1 + 7 x 2 − 5 x 3 + 2 x 4 + 2 x 5 = − 3

has coefficient matrix

A = 2 4 − 3 5 1 3 1 0 1 − 3 − 2 7 − 5 2 2

and vector of constants

b = 9 0 − 3

and so will be described compactly by the vector equation A x = b . ⊠

The matrix-vector product is a very natural computation. We have motivated it by its connections with systems of equations, but here is a another example.

Example  MBC
Money’s best cities
Every year Money magazine selects several cities in the United States as the “best” cities to live in, based on a wide array of statistics about each city. This is an example of how the editors of Money might arrive at a single number that consolidates the statistics about a city. We will analyze Los Angeles, Chicago and New York City, based on four criteria: average high temperature in July (Farenheit), number of colleges and universities in a 30-mile radius, number of toxic waste sites in the Superfund environmental clean-up program and a personal crime index based on FBI statistics (average = 100, smaller is safer). It should be apparent how to generalize the example to a greater number of cities and a greater number of statistics.

We begin by building a table of statistics. The rows will be labeled with the cities, and the columns with statistical categories. These values are from Money’s website in early 2005.

Conceivably these data might reside in a spreadsheet. Now we must combine the statistics for each city. We could accomplish this by weighting each category, scaling the values and summing them. The sizes of the weights would depend upon the numerical size of each statistic generally, but more importantly, they would reflect the editors opinions or beliefs about which statistics were most important to their readers. Is the crime index more important than the number of colleges and universities? Of course, there is no right answer to this question.

Suppose the editors finally decide on the following weights to employ: temperature, 0 . 2 3 ; colleges, 0 . 4 6 ; Superfund, − 0 . 0 5 ; crime, − 0 . 2 0 . Notice how negative weights are used for undesirable statistics. Then, for example, the editors would compute for Los Angeles,

( 0 . 2 3 ) ( 7 7 ) + ( 0 . 4 6 ) ( 2 8 ) + ( − 0 . 0 5 ) ( 9 3 ) + ( − 0 . 2 0 ) ( 2 5 4 ) = − 2 4 . 8 6

This computation might remind you of an inner product, but we will produce the computations for all of the cities as a matrix-vector product. Write the table of raw statistics as a matrix

T = 7 7 2 8 9 3 2 5 4 8 4 3 8 8 5 3 6 3 8 4 9 9 1 1 9 3

and the weights as a vector

w = 0 . 2 3 0 . 4 6 − 0 . 0 5 − 0 . 2 0

then the matrix-vector product ( Definition MVP) yields

T w = ( 0 . 2 3 ) 7 7 8 4 8 4 + ( 0 . 4 6 ) 2 8 3 8 9 9 + ( − 0 . 0 5 ) 9 3 8 5 1 + ( − 0 . 2 0 ) 2 5 4 3 6 3 1 9 3 = − 2 4 . 8 6 − 4 0 . 0 5 2 6 . 2 1

This vector contains a single number for each of the cities being studied, so the editors would rank New York best ( 2 6 . 2 1 ), Los Angeles next ( − 2 4 . 8 6 ), and Chicago third ( − 4 0 . 0 5 ). Of course, the mayor’s offices in Chicago and Los Angeles are free to counter with a different set of weights that cause their city to be ranked best. These alternative weights would be chosen to play to each cities’ strengths, and minimize their problem areas.

If a speadsheet were used to make these computations, a row of weights would be entered somewhere near the table of data and the formulas in the spreadsheet would effect a matrix-vector product. This example is meant to illustrate how “linear” computations (addition, multiplication) can be organized as a matrix-vector product.

Another example would be the matrix of numerical scores on examinations and exercises for students in a class. The rows would correspond to students and the columns to exams and assignments. The instructor could then assign weights to the different exams and assignments, and via a matrix-vector product, compute a single score for each student. ⊠

Later (much later) we will need the following theorem, which is really a technical lemma (see Technique LC). Since we are in a position to prove it now, we will. But you can safely skip it for the moment, if you promise to come back later to study the proof when the theorem is employed. At that point you will also be able to understand the comments in the paragraph following the proof.

Theorem  EMMVP
Equal Matrices and Matrix-Vector Products
Suppose that A and B are m × n matrices such that A x = B x for every x ∈ ℂ n . Then A = B . □

Proof   We are assuming A x = B x for all x ∈ ℂ n , so we can employ this equality for any choice of the vector x . However, we’ll limit our use of this equality to the standard unit vectors, e j , 1 ≤ j ≤ n ( Definition SUV). For all 1 ≤ j ≤ n , 1 ≤ i ≤ m ,

A i j = 0 A i 1 + ⋯ + 0 A i , j − 1 + 1 A i j + 0 A i , j + 1 + ⋯ + 0 A i n = A i 1 e j 1 + A i 2 e j 2 + A i 3 e j 3 + ⋯ + A i n e j n   Definition SUV = A e j i   Definition MVP = B e j i   Definition CVE = B i 1 e j 1 + B i 2 e j 2 + B i 3 e j 3 + ⋯ + B i n e j n   Definition MVP = 0 B i 1 + ⋯ + 0 B i , j − 1 + 1 B i j + 0 B i , j + 1 + ⋯ + 0 B i n   Definition SUV = B i j

So by Definition ME the matrices A and B are equal, as desired. ■

You might notice from studying the proof that the hypotheses of this theorem could be “weakened” (i.e. made less restrictive). We need only suppose the equality of the matrix-vector products for just the standard unit vectors ( Definition SUV) or any other spanning set (Definition TSVS) of ℂ n (Exercise LISS.T40). However, in practice, when we apply this theorem the stronger hypothesis will be in effect so this version of the theorem will suffice for our purposes. (If we changed the statement of the theorem to have the less restrictive hypothesis, then we would call the theorem “stronger.”)

Subsection MM: Matrix Multiplication

We now define how to multiply two matrices together. Stop for a minute and think about how you might define this new operation.

Many books would present this definition much earlier in the course. However, we have taken great care to delay it as long as possible and to present as many ideas as practical based mostly on the notion of linear combinations. Towards the conclusion of the course, or when you perhaps take a second course in linear algebra, you may be in a position to appreciate the reasons for this. For now, understand that matrix multiplication is a central definition and perhaps you will appreciate its importance more by having saved it for later.

Definition  MM
Matrix Multiplication
Suppose A is an m × n matrix and B is an n × p matrix with columns B 1 , B 2 , B 3 , … , B p . Then the matrix product of A with B is the m × p matrix where column i is the matrix-vector product A B i . Symbolically,

A B = A B 1 | B 2 | B 3 | … | B p = A B 1 | A B 2 | A B 3 | … | A B p .

△

Example  PTM
Product of two matrices
Set

A = 1 2 − 1 4 6 0 − 4 1 2 3 − 5 1 2 − 3 4 B = 1 6 2 1 − 1 4 3 2 1 1 2 3 6 4 − 1 2 1 − 2 3 0

Then

A B = A 1 − 1 1 6 1 A 6 4 1 4 − 2 A 2 3 2 − 1 3 A 1 2 3 2 0 = 2 8 1 7 2 0 1 0 2 0 − 1 3 − 3 − 1 − 1 8 − 4 4 1 2 − 3 .

⊠

Is this the definition of matrix multiplication you expected? Perhaps our previous operations for matrices caused you to think that we might multiply two matrices of the same size, entry-by-entry? Notice that our current definition uses matrices of different sizes (though the number of columns in the first must equal the number of rows in the second), and the result is of a third size. Notice too in the previous example that we cannot even consider the product B A , since the sizes of the two matrices in this order aren’t right.

But it gets weirder than that. Many of your old ideas about “multiplication” won’t apply to matrix multiplication, but some still will. So make no assumptions, and don’t do anything until you have a theorem that says you can. Even if the sizes are right, matrix multiplication is not commutative — order matters.

Example  MMNC
Matrix multiplication is not commutative
Set

A = 1 3 − 1 2 B = 4 0 5 1 .

Then we have two square, 2 × 2 matrices, so Definition MM allows us to multiply them in either order. We find

A B = 1 9 3 6 2 B A = 4 1 2 4 1 7

and A B ≠ B A . Not even close. It should not be hard for you to construct other pairs of matrices that do not commute (try a couple of 3 × 3 ’s). Can you find a pair of non-identical matrices that do commute? ⊠

Matrix multiplication is fundamental, so it is a natural procedure for any computational device.  See: Computation MM.MMA

Subsection MMEE: Matrix Multiplication, Entry-by-Entry

While certain “natural” properties of multiplication don’t hold, many more do. In the next subsection, we’ll state and prove the relevant theorems. But first, we need a theorem that provides an alternate means of multiplying two matrices. In many texts, this would be given as the definition of matrix multiplication. We prefer to turn it around and have the following formula as a consequence of our definition. It will prove useful for proofs of matrix equality, where we need to examine products of matrices, entry-by-entry.

Theorem  EMP
Entries of Matrix Products
Suppose A is an m × n matrix and B is an n × p matrix. Then for 1 ≤ i ≤ m , 1 ≤ j ≤ p , the individual entries of A B are given by

A B i j = A i 1 B 1 j + A i 2 B 2 j + A i 3 B 3 j + ⋯ + A i n B n j = ∑ k = 1 n A i k B k j

□

Proof   Denote the columns of A as the vectors A 1 , A 2 , A 3 , … , A n and the columns of B as the vectors B 1 , B 2 , B 3 , … , B p . Then for 1 ≤ i ≤ m , 1 ≤ j ≤ p ,

A B i j = A B j i   Definition MM = B j 1 A 1 + B j 2 A 2 + B j 3 A 3 + ⋯ + B j n A n i   Definition MVP = B j 1 A 1 i + B j 2 A 2 i + B j 3 A 3 i + ⋯ + B j n A n i   Definition CVA = B j 1 A 1 i + B j 2 A 2 i + B j 3 A 3 i + ⋯ + B j n A n i   Definition CVSM = B 1 j A i 1 + B 2 j A i 2 + B 3 j A i 3 + ⋯ + B n j A i n   Notation ME = A i 1 B 1 j + A i 2 B 2 j + A i 3 B 3 j + ⋯ + A i n B n j   Property CMCN = ∑ k = 1 n A i k B k j

■

Example  PTMEE
Product of two matrices, entry-by-entry
Consider again the two matrices from Example PTM

A = 1 2 − 1 4 6 0 − 4 1 2 3 − 5 1 2 − 3 4 B = 1 6 2 1 − 1 4 3 2 1 1 2 3 6 4 − 1 2 1 − 2 3 0

Then suppose we just wanted the entry of A B in the second row, third column:

A B 2 3 = A 2 1 B 1 3 + A 2 2 B 2 3 + A 2 3 B 3 3 + A 2 4 B 4 3 + A 2 5 B 5 3 = ( 0 ) ( 2 ) + ( − 4 ) ( 3 ) + ( 1 ) ( 2 ) + ( 2 ) ( − 1 ) + ( 3 ) ( 3 ) = − 3

Notice how there are 5 terms in the sum, since 5 is the common dimension of the two matrices (column count for A , row count for B ). In the conclusion of Theorem EMP, it would be the index k that would run from 1 to 5 in this computation. Here’s a bit more practice.

The entry of third row, first column:

A B 3 1 = A 3 1 B 1 1 + A 3 2 B 2 1 + A 3 3 B 3 1 + A 3 4 B 4 1 + A 3 5 B 5 1 = ( − 5 ) ( 1 ) + ( 1 ) ( − 1 ) + ( 2 ) ( 1 ) + ( − 3 ) ( 6 ) + ( 4 ) ( 1 ) = − 1 8

To get some more practice on your own, complete the computation of the other 10 entries of this product. Construct some other pairs of matrices (of compatible sizes) and compute their product two ways. First use Definition MM. Since linear combinations are straightforward for you now, this should be easy to do and to do correctly. Then do it again, using Theorem EMP. Since this process may take some practice, use your first computation to check your work. ⊠

Theorem EMP is the way many people compute matrix products by hand. It will also be very useful for the theorems we are going to prove shortly. However, the definition ( Definition MM) is frequently the most useful for its connections with deeper ideas like the null space and the upcoming column space.

Subsection PMM: Properties of Matrix Multiplication

In this subsection, we collect properties of matrix multiplication and its interaction with the zero matrix (Definition ZM), the identity matrix ( Definition IM), matrix addition (Definition MA), scalar matrix multiplication (Definition MSM), the inner product ( Definition IP), conjugation (Theorem MMCC), and the transpose (Definition TM). Whew! Here we go. These are great proofs to practice with, so try to concoct the proofs before reading them, they’ll get progressively more complicated as we go.

Theorem  MMZM
Matrix Multiplication and the Zero Matrix
Suppose A is an m × n matrix. Then
1. A O n × p = O m × p
2. O p × m A = O p × n □

Proof   We’ll prove (1) and leave (2) to you. Entry-by-entry, for 1 ≤ i ≤ m , 1 ≤ j ≤ p ,

A O n × p i j = ∑ k = 1 n A i k O n × p k j   Theorem EMP = ∑ k = 1 n A i k 0   Definition ZM = ∑ k = 1 n 0 = 0   Property ZCN = O m × p i j   Definition ZM

So by the definition of matrix equality (Definition ME), the matrices A O n × p and O m × p are equal. ■

Theorem  MMIM
Matrix Multiplication and Identity Matrix
Suppose A is an m × n matrix. Then
1. A I n = A
2. I m A = A □

Proof   Again, we’ll prove (1) and leave (2) to you. Entry-by-entry, For 1 ≤ i ≤ m , 1 ≤ j ≤ n ,

A I n i j = ∑ k = 1 n A i k I n k j   Theorem EMP = A i j I n j j + ∑ k = 1 k ≠ j n A i k I n k j   Property CACN = A i j ( 1 ) + ∑ k = 1 , k ≠ j n A i k ( 0 )   Definition IM = A i j + ∑ k = 1 , k ≠ j n 0 = A i j

So the matrices A and A I n are equal, entry-by-entry, and by the definition of matrix equality (Definition ME) we can say they are equal matrices. ■

It is this theorem that gives the identity matrix its name. It is a matrix that behaves with matrix multiplication like the scalar 1 does with scalar multiplication. To multiply by the identity matrix is to have no effect on the other matrix.

Theorem  MMDAA
Matrix Multiplication Distributes Across Addition
Suppose A is an m × n matrix and B and C are n × p matrices and D is a p × s matrix. Then
1. A ( B + C ) = A B + A C
2. ( B + C ) D = B D + C D □

Proof   We’ll do (1), you do (2). Entry-by-entry, for 1 ≤ i ≤ m , 1 ≤ j ≤ p ,

A ( B + C ) i j = ∑ k = 1 n A i k B + C k j   Theorem EMP = ∑ k = 1 n A i k ( B k j + C k j )   Definition MA = ∑ k = 1 n A i k B k j + A i k C k j   Property DCN = ∑ k = 1 n A i k B k j + ∑ k = 1 n A i k C k j   Property CACN = A B i j + A C i j   Theorem EMP = A B + A C i j   Definition MA

So the matrices A ( B + C ) and A B + A C are equal, entry-by-entry, and by the definition of matrix equality (Definition ME) we can say they are equal matrices. ■

Theorem  MMSMM
Matrix Multiplication and Scalar Matrix Multiplication
Suppose A is an m × n matrix and B is an n × p matrix. Let α be a scalar. Then α ( A B ) = ( α A ) B = A ( α B ) . □

Proof   These are equalities of matrices. We’ll do the first one, the second is similar and will be good practice for you. For 1 ≤ i ≤ m , 1 ≤ j ≤ p ,

α ( A B ) i j = α A B i j   Definition MSM = α ∑ k = 1 n A i k B k j   Theorem EMP = ∑ k = 1 n α A i k B k j   Property DCN = ∑ k = 1 n α A i k B k j   Definition MSM = ( α A ) B i j   Theorem EMP

So the matrices α ( A B ) and ( α A ) B are equal, entry-by-entry, and by the definition of matrix equality (Definition ME) we can say they are equal matrices. ■

Theorem  MMA
Matrix Multiplication is Associative
Suppose A is an m × n matrix, B is an n × p matrix and D is a p × s matrix. Then A ( B D ) = ( A B ) D . □

Proof   A matrix equality, so we’ll go entry-by-entry, no surprise there. For 1 ≤ i ≤ m , 1 ≤ j ≤ s ,

A ( B D ) i j = ∑ k = 1 n A i k B D k j   Theorem EMP = ∑ k = 1 n A i k ∑ ℓ = 1 p B k ℓ D ℓ j   Theorem EMP = ∑ k = 1 n ∑ ℓ = 1 p A i k B k ℓ D ℓ j   Property DCN  We can switch the order of the summation since these are finite sums, = ∑ ℓ = 1 p ∑ k = 1 n A i k B k ℓ D ℓ j   Property CACN  As  D ℓ j  does not depend on the index  k , we can use distributivity to move it outside of the inner sum, = ∑ ℓ = 1 p D ℓ j ∑ k = 1 n A i k B k ℓ   Property DCN = ∑ ℓ = 1 p D ℓ j A B i ℓ   Theorem EMP = ∑ ℓ = 1 p A B i ℓ D ℓ j   Property CMCN = ( A B ) D i j   Theorem EMP

So the matrices ( A B ) D and A ( B D ) are equal, entry-by-entry, and by the definition of matrix equality (Definition ME) we can say they are equal matrices. ■

The statement of our next theorem is technically inaccurate. If we upgrade the vectors u , v to matrices with a single column, then the expression u t v ¯ is a 1 × 1 matrix, though we will treat this small matrix as if it was simply the scalar quantity in its lone entry. When we apply Theorem MMIP there should not be any confusion.

Theorem  MMIP
Matrix Multiplication and Inner Products
If we consider the vectors u , v ∈ ℂ m as m × 1 matrices then

u , v = u t v ¯

□

Proof  

u , v = ∑ k = 1 m u k v k ¯   Definition IP = ∑ k = 1 m u k 1 v k 1 ¯  Column vectors as matrices = ∑ k = 1 m u t 1 k v k 1 ¯   Definition TM = ∑ k = 1 m u t 1 k v ¯ k 1   Definition CCCV = u t v ¯ 1 1   Theorem EMP

To finish we just blur the distinction between a 1 × 1 matrix ( u t v ¯ ) and its lone entry. ■

Theorem  MMCC
Matrix Multiplication and Complex Conjugation
Suppose A is an m × n matrix and B is an n × p matrix. Then A B ¯ = A ¯ B ¯ . □

Proof   To obtain this matrix equality, we will work entry-by-entry. For 1 ≤ i ≤ m , 1 ≤ j ≤ p ,

A B ¯ i j = A B i j ¯   Definition CCM = ∑ k = 1 n A i k B k j ¯   Theorem EMP = ∑ k = 1 n A i k B k j ¯   Theorem CCRA = ∑ k = 1 n A i k ¯ B k j ¯   Theorem CCRM = ∑ k = 1 n A ¯ i k B ¯ k j   Definition CCM = A ¯ B ¯ i j   Theorem EMP

So the matrices A B ¯ and A ¯ B ¯ are equal, entry-by-entry, and by the definition of matrix equality (Definition ME) we can say they are equal matrices. ■

Another theorem in this style, and its a good one. If you’ve been practicing with the previous proofs you should be able to do this one yourself.

Theorem  MMT
Matrix Multiplication and Transposes
Suppose A is an m × n matrix and B is an n × p matrix. Then ( A B ) t = B t A t . □

Proof   This theorem may be surprising but if we check the sizes of the matrices involved, then maybe it will not seem so far-fetched. First, A B has size m × p , so its transpose has size p × m . The product of B t with A t is a p × n matrix times an n × m matrix, also resulting in a p × m matrix. So at least our objects are compatible for equality (and would not be, in general, if we didn’t reverse the order of the matrix multiplication).

Here we go again, entry-by-entry. For 1 ≤ i ≤ m , 1 ≤ j ≤ p ,

( A B ) t j i = A B i j   Definition TM = ∑ k = 1 n A i k B k j   Theorem EMP = ∑ k = 1 n B k j A i k   Property CMCN = ∑ k = 1 n B t j k A t k i   Definition TM = B t A t j i   Theorem EMP

So the matrices ( A B ) t and B t A t are equal, entry-by-entry, and by the definition of matrix equality (Definition ME) we can say they are equal matrices. ■

This theorem seems odd at first glance, since we have to switch the order of A and B . But if we simply consider the sizes of the matrices involved, we can see that the switch is necessary for this reason alone. That the individual entries of the products then come along to be equal is a bonus.

As the adjoint of a matrix is a composition of a conjugate and a transpose, its interaction with matrix multiplication is similar to that of a transpose. Here’s the last of our long list of basic properties of matrix multiplication.

Theorem  MMAD
Matrix Multiplication and Adjoints
Suppose A is an m × n matrix and B is an n × p matrix. Then ( A B ) ∗ = B ∗ A ∗ . □

Proof  

( A B ) ∗ = A B ¯ t   Definition A = A ¯ B ¯ t   Theorem MMCC = B ¯ t A ¯ t   Theorem MMT = B ∗ A ∗   Definition A

■

Notice how none of these proofs above relied on writing out huge general matrices with lots of ellipses (“…”) and trying to formulate the equalities a whole matrix at a time. This messy business is a “proof technique” to be avoided at all costs. Notice too how the proof of Theorem MMAD does not use an entry-by-entry approach, but simply builds on previous results about matrix multiplication’s interaction with conjugation and transposes.

These theorems, along with Theorem VSPM and the other results in Section MO, give you the “rules” for how matrices interact with the various operations we have defined on matrices (addition, scalar multiplication, matrix multiplication, conjugation, transposes and adjoints). Use them and use them often. But don’t try to do anything with a matrix that you don’t have a rule for. Together, we would informally call all these operations, and the attendant theorems, “the algebra of matrices.” Notice, too, that every column vector is just a n × 1 matrix, so these theorems apply to column vectors also. Finally, these results, taken as a whole, may make us feel that the definition of matrix multiplication is not so unnatural.

Subsection HM: Hermitian Matrices

The adjoint of a matrix has a basic property when employed in a matrix-vector product as part of an inner product. At this point, you could even use the following result as a motivation for the definition of an adjoint.

Theorem  AIP
Adjoint and Inner Product
Suppose that A is an m × n matrix and x ∈ ℂ n , y ∈ ℂ m . Then A x , y = x , A ∗ y . □

Proof  

A x , y = A x t y ¯   Theorem MMIP = x t A t y ¯   Theorem MMT = x t A ¯ ¯ t y ¯   Theorem CCM = x t A ¯ t ¯ y ¯   Theorem MCT = x t A ∗ ¯ y ¯   Definition A = x t A ∗ y ¯   Theorem MMCC = x , A ∗ y   Theorem MMIP

■

Sometimes a matrix is equal to its adjoint (Definition A), and these matrices have interesting properties. One of the most common situations where this occurs is when a matrix has only real number entries. Then we are simply talking about symmetric matrices (Definition SYM), so you can view this as a generalization of a symmetric matrix.

Definition  HM
Hermitian Matrix
The square matrix A is Hermitian (or self-adjoint) if A = A ∗ . △

Again, the set of real matrices that are Hermitian is exactly the set of symmetric matrices. In Section PEE we will uncover some amazing properties of Hermitian matrices, so when you get there, run back here to remind yourself of this definition. Further properties will also appear in various sections of the Topics ( Part T). Right now we prove a fundamental result about Hermitian matrices, matrix vector products and inner products. As a characterization, this could be employed as a definition of a Hermitian matrix and some authors take this approach.

Theorem  HMIP
Hermitian Matrices and Inner Products
Suppose that A is a square matrix of size n . Then A is Hermitian if and only if A x , y = x , A y for all x , y ∈ ℂ n . □

Proof   ( ⇒ ) This is the “easy half” of the proof, and makes the rationale for a definition of Hermitian matrices most obvious. Assume A is Hermitian,

A x , y = x , A ∗ y   Theorem AIP = x , A y   Definition HM

( ⇐ ) This “half” will take a bit more work. Assume that A x , y = x , A y for all x , y ∈ ℂ n . Choose any x ∈ ℂ n . We want to show that A = A ∗ by establishing that A x = A ∗ x . With only this much motivation, consider the inner product,

A x − A ∗ x , A x − A ∗ x = A x − A ∗ x , A x − A x − A ∗ x , A ∗ x   Theorem IPVA = A x − A ∗ x , A x − A A x − A ∗ x , x   Theorem AIP = A x − A ∗ x , A x − A x − A ∗ x , A x  Hypothesis = 0   Property AICN

Because this first inner product equals zero, and has the same vector in each argument ( A x − A ∗ x ), Theorem PIP gives the conclusion that A x − A ∗ x = 0 . With A x = A ∗ x for all x ∈ ℂ n , Theorem EMMVP says A = A ∗ , which is the defining property of a Hermitian matrix ( Definition HM). ■

So, informally, Hermitian matrices are those that can be tossed around from one side of an inner product to the other with reckless abandon. We’ll see later what this buys us.

Subsection READ: Reading Questions

Subsection EXC: Exercises

C20 Compute the product of the two matrices below, A B . Do this using the definitions of the matrix-vector product ( Definition MVP) and the definition of matrix multiplication ( Definition MM).

A = 2 5 − 1 3 2 − 2 B = 1 5 − 3 4 2 0 2 − 3

Contributed by  Robert Beezer Solution [ 634]

C21 Compute the product A B of the two matrices below using both the definition of the matrix-vector product ( Definition MVP) and the definition of matrix multiplication ( Definition MM).

A = 1 3 2 − 1 2 1 0 1 0 B = 4 1 2 1 0 1 3 1 5

Contributed by  Chris Black Solution [ 634]

C22 Compute the product A B of the two matrices below using both the definition of the matrix-vector product ( Definition MVP) and the definition of matrix multiplication ( Definition MM).

A = 1 0 − 2 1 B = 2 3 4 6

Contributed by  Chris Black Solution [ 634]

C23 Compute the product A B of the two matrices below using both the definition of the matrix-vector product ( Definition MVP) and the definition of matrix multiplication ( Definition MM).

A = 3 1 2 4 6 5 1 2 B = − 3 1 4 2

Contributed by  Chris Black Solution [ 634]

C24 Compute the product A B of the two matrices below.

A = 1 2 3 − 2 0 1 − 2 − 1 1 1 3 1 B = 3 4 0 2

Contributed by  Chris Black Solution [ 635]

C25 Compute the product A B of the two matrices below.

A = 1 2 3 − 2 0 1 − 2 − 1 1 1 3 1 B = − 7 3 1 1

Contributed by  Chris Black Solution [ 635]

C26 Compute the product A B of the two matrices below using both the definition of the matrix-vector product ( Definition MVP) and the definition of matrix multiplication ( Definition MM).

A = 1 3 1 0 1 0 1 1 2 B = 2 − 5 − 1 0 1 0 − 1 2 1

Contributed by  Chris Black Solution [ 635]

C30 For the matrix A = 1 2 0 1 , find A 2 , A 3 , A 4 . Find a general formula for A n for any positive integer n .  
Contributed by  Chris Black Solution [ 635]

C31 For the matrix A = 1 − 1 0 1 , find A 2 , A 3 , A 4 . Find a general formula for A n for any positive integer n .  
Contributed by  Chris Black Solution [ 635]

C32 For the matrix A = 1 0 0 0 2 0 0 0 3 , find A 2 , A 3 , A 4 . Find a general formula for A n for any positive integer n .  
Contributed by  Chris Black Solution [ 635]

C33 For the matrix A = 0 1 2 0 0 1 0 0 0 , find A 2 , A 3 , A 4 . Find a general formula for A n for any positive integer n .  
Contributed by  Chris Black Solution [ 635]

T10 Suppose that A is a square matrix and there is a vector, b , such that ℒ S A , b has a unique solution. Prove that A is nonsingular. Give a direct proof (perhaps appealing to Theorem PSPHS) rather than just negating a sentence from the text discussing a similar situation.  
Contributed by  Robert Beezer Solution [ 636]

T20 Prove the second part of Theorem MMZM.  
Contributed by  Robert Beezer

T21 Prove the second part of Theorem MMIM.  
Contributed by  Robert Beezer

T22 Prove the second part of Theorem MMDAA.  
Contributed by  Robert Beezer

T23 Prove the second part of Theorem MMSMM.  
Contributed by  Robert Beezer Solution [ 636]

T31 Suppose that A is an m × n matrix and x , y ∈ N A . Prove that x + y ∈ N A .  
Contributed by  Robert Beezer

T32 Suppose that A is an m × n matrix, α ∈ ℂ , and x ∈ N A . Prove that α x ∈ N A .  
Contributed by  Robert Beezer

T40 Suppose that A is an m × n matrix and B is an n × p matrix. Prove that the null space of B is a subset of the null space of A B , that is N B ⊆ N A B . Provide an example where the opposite is false, in other words give an example where N A B ⊈ N B .  
Contributed by  Robert Beezer Solution [ 637]

T41 Suppose that A is an n × n nonsingular matrix and B is an n × p matrix. Prove that the null space of B is equal to the null space of A B , that is N B = N A B . (Compare with Exercise MM.T40.)  
Contributed by  Robert Beezer Solution [ 638]

T50 Suppose u and v are any two solutions of the linear system ℒ S A , b . Prove that u − v is an element of the null space of A , that is, u − v ∈ N A .  
Contributed by  Robert Beezer

T51 Give a new proof of Theorem PSPHS replacing applications of Theorem SLSLC with matrix-vector products ( Theorem SLEMM).  
Contributed by  Robert Beezer Solution [ 638]

T52 Suppose that x , y ∈ ℂ n , b ∈ ℂ m and A is an m × n matrix. If x , y and x + y are each a solution to the linear system ℒ S A , b , what interesting can you say about b ? Form an implication with the existence of the three solutions as the hypothesis and an interesting statement about ℒ S A , b as the conclusion, and then give a proof.  
Contributed by  Robert Beezer Solution [ 640]

Subsection SOL: Solutions

C20 Contributed by  Robert Beezer Statement [ 626]
By Definition MM,

A B = 2 5 − 1 3 2 − 2 1 2 2 5 − 1 3 2 − 2 5 0 2 5 − 1 3 2 − 2 − 3 2 2 5 − 1 3 2 − 2 4 − 2  Repeated applications of  Definition MVP  give = 1 2 − 1 2 + 2 5 3 − 2 5 2 − 1 2 + 0 5 3 − 2 − 3 2 − 1 2 + 2 5 3 − 2 4 2 − 1 2 + ( − 3 ) 5 3 − 2 = 1 2 1 0 4 − 7 5 − 5 9 − 1 3 − 2 1 0 − 1 0 1 4

C21 Contributed by  Chris Black Statement [ 626]
A B = 1 3 3 1 5 1 0 5 1 0 1 .

C22 Contributed by  Chris Black Statement [ 627]
A B = 2 3 0 0 .

C23 Contributed by  Chris Black Statement [ 627]
A B = − 5 5 1 0 1 0 2 1 6 5 5 .

C24 Contributed by  Chris Black Statement [ 628]
A B = 7 2 9 .

C25 Contributed by  Chris Black Statement [ 628]
A B = 0 0 0 .

C26 Contributed by  Chris Black Statement [ 629]
A B = 1 0 0 0 1 0 0 0 1 .

C30 Contributed by  Chris Black Statement [ 630]
A 2 = 1 4 0 1 , A 3 = 1 6 0 1 , A 4 = 1 8 0 1 . From this pattern, we see that A n = 1 2 n 0 1 .

C31 Contributed by  Chris Black Statement [ 630]
A 2 = 1 − 2 0 1 , A 3 = 1 − 3 0 1 , A 4 = 1 − 4 0 1 . From this pattern, we see that A n = 1 − n 0 1 .

C32 Contributed by  Chris Black Statement [ 630]
A 2 = 1 0 0 0 4 0 0 0 9 , A 3 = 1 0 0 0 8 0 0 0 2 7 , and A 4 = 1 0 0 0 1 6 0 0 0 8 1 . The pattern emerges, and we see that A n = 1 0 0 0 2 n 0 0 0 3 n .

C33 Contributed by  Chris Black Statement [ 630]
We quickly compute A 2 = 0 0 1 0 0 0 0 0 0 , and we then see that A 3 and all subsequent powers of A are the 3 × 3 zero matrix; that is, A n = O 3 , 3 for n ≥ 3 .

T10 Contributed by  Robert Beezer Statement [ 630]
Since ℒ S A , b has at least one solution, we can apply Theorem PSPHS. Because the solution is assumed to be unique, the null space of A must be trivial. Then Theorem NMTNS implies that A is nonsingular.

The converse of this statement is a trivial application of Theorem NMUS. That said, we could extend our NSMxx series of theorems with an added equivalence for nonsingularity, “Given a single vector of constants, b , the system ℒ S A , b has a unique solution.”

T23 Contributed by  Robert Beezer Statement [ 631]
We’ll run the proof entry-by-entry.

α ( A B ) i j = α A B i j   Definition MSM = α ∑ k = 1 n A i k B k j   Theorem EMP = ∑ k = 1 n α A i k B k j  Distributivity in  ℂ = ∑ k = 1 n A i k α B k j  Commutativity in  ℂ = ∑ k = 1 n A i k α B k j   Definition MSM = A ( α B ) i j   Theorem EMP

So the matrices α ( A B ) and A ( α B ) are equal, entry-by-entry, and by the definition of matrix equality (Definition ME) we can say they are equal matrices.

T40 Contributed by  Robert Beezer Statement [ 631]
To prove that one set is a subset of another, we start with an element of the smaller set and see if we can determine that it is a member of the larger set ( Definition SSET). Suppose x ∈ N B . Then we know that B x = 0 by Definition NSM. Consider

( A B ) x = A ( B x )   Theorem MMA = A 0  Hypothesis = 0   Theorem MMZM

This establishes that x ∈ N A B , so N B ⊆ N A B .

To show that the inclusion does not hold in the opposite direction, choose B to be any nonsingular matrix of size n . Then N B = 0 by Theorem NMTNS. Let A be the square zero matrix, O , of the same size. Then A B = O B = O by Theorem MMZM and therefore N A B = ℂ n , and is not a subset of N B = 0 .

T41 Contributed by  David Braithwaite Statement [ 632]
From the solution to Exercise MM.T40 we know that N B ⊆ N A B . So to establish the set equality ( Definition SE) we need to show that N A B ⊆ N B .

Suppose x ∈ N A B . Then we know that A B x = 0 by Definition NSM. Consider

0 = A B x   Definition NSM = A B x   Theorem MMA

So, B x ∈ N A . Because A is nonsingular, it has a trivial null space ( Theorem NMTNS) and we conclude that B x = 0 . This establishes that x ∈ N B , so N A B ⊆ N B and combined with the solution to Exercise MM.T40 we have N B = N A B when A is nonsingular.

T51 Contributed by  Robert Beezer Statement [ 632]
We will work with the vector equality representations of the relevant systems of equations, as described by Theorem SLEMM.

( ⇐ ) Suppose y = w + z and z ∈ N A . Then

A y = A ( w + z )  Substitution = A w + A z   Theorem MMDAA = b + 0 z ∈ N A = b   Property ZC

demonstrating that y is a solution.

( ⇒ ) Suppose y is a solution to ℒ S A , b . Then

A ( y − w ) = A y − A w   Theorem MMDAA = b − b   y , w  solutions to  A x = b = 0   Property AIC

which says that y − w ∈ N A . In other words, y − w = z for some vector z ∈ N A . Rewritten, this is y = w + z , as desired.

T52 Contributed by  Robert Beezer Statement [ 632]
ℒ S A , b must be homogeneous. To see this consider that

b = A x   Theorem SLEMM = A x + 0   Property ZC = A x + A y − A y   Property AIC = A x + y − A y   Theorem MMDAA = b − b   Theorem SLEMM = 0   Property AIC

By Definition HS we see that ℒ S A , b is homogeneous.

Section MISLE  Matrix Inverses and Systems of Linear Equations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

We begin with a familiar example, performed in a novel way.

Example  SABMI
Solutions to Archetype B with a matrix inverse
Archetype B is the system of m = 3 linear equations in n = 3 variables,

− 7 x 1 − 6 x 2 − 1 2 x 3 = − 3 3 5 x 1 + 5 x 2 + 7 x 3 = 2 4 x 1 + 4 x 3 = 5

By Theorem SLEMM we can represent this system of equations as

A x = b

where

A = − 7 − 6 − 1 2 5 5 7 1 0 4 x = x 1 x 2 x 3 b = − 3 3 2 4 5

We’ll pull a rabbit out of our hat and present the 3 × 3 matrix B ,

B = − 1 0 − 1 2 − 9 1 3 2 8 1 1 2 5 2 3 5 2

and note that

B A = − 1 0 − 1 2 − 9 1 3 2 8 1 1 2 5 2 3 5 2 − 7 − 6 − 1 2 5 5 7 1 0 4 = 1 0 0 0 1 0 0 0 1

Now apply this computation to the problem of solving the system of equations,

x = I 3 x   Theorem MMIM = ( B A ) x  Substitution = B ( A x )   Theorem MMA = B b  Substitution

So we have

x = B b = − 1 0 − 1 2 − 9 1 3 2 8 1 1 2 5 2 3 5 2 − 3 3 2 4 5 = − 3 5 2

So with the help and assistance of B we have been able to determine a solution to the system represented by A x = b through judicious use of matrix multiplication. We know by Theorem NMUS that since the coefficient matrix in this example is nonsingular, there would be a unique solution, no matter what the choice of b . The derivation above amplifies this result, since we were forced to conclude that x = B b and the solution couldn’t be anything else. You should notice that this argument would hold for any particular value of b . ⊠

The matrix B of the previous example is called the inverse of A . When A and B are combined via matrix multiplication, the result is the identity matrix, which can be inserted “in front” of x as the first step in finding the solution. This is entirely analogous to how we might solve a single linear equation like 3 x = 1 2 .

x = 1 x = 1 3 3 x = 1 3 3 x = 1 3 1 2 = 4

Here we have obtained a solution by employing the “multiplicative inverse” of 3 , 3 − 1 = 1 3 . This works fine for any scalar multiple of x , except for zero, since zero does not have a multiplicative inverse. Consider seperately the two linear equations,

0 x = 1 2 0 x = 0

The first has no solutions, while the second has infinitely many solutions. For matrices, it is all just a little more complicated. Some matrices have inverses, some do not. And when a matrix does have an inverse, just how would we compute it? In other words, just where did that matrix B in the last example come from? Are there other matrices that might have worked just as well?

Parent topic: Chapter M  Matrices

Previous topic: Section MM  Matrix Multiplication

Next topic: Section MINM  Matrix Inverses and Nonsingular Matrices

Subsection IM: Inverse of a Matrix

Definition  MI
Matrix Inverse
Suppose A and B are square matrices of size n such that A B = I n and B A = I n . Then A is invertible and B is the inverse of A . In this situation, we write B = A − 1 .

(This definition contains Notation MI.) △

Notice that if B is the inverse of A , then we can just as easily say A is the inverse of B , or A and B are inverses of each other.

Not every square matrix has an inverse. In Example SABMI the matrix B is the inverse the coefficient matrix of Archetype B. To see this it only remains to check that A B = I 3 . What about Archetype A? It is an example of a square matrix without an inverse.

Example  MWIAA
A matrix without an inverse, Archetype A
Consider the coefficient matrix from Archetype A,

A = 1 − 1 2 2 1 1 1 1 0

Suppose that A is invertible and does have an inverse, say B . Choose the vector of constants

b = 1 3 2

and consider the system of equations ℒ S A , b . Just as in Example SABMI, this vector equation would have the unique solution x = B b .

However, the system ℒ S A , b is inconsistent. Form the augmented matrix A b and row-reduce to

1 0 1 0 0 1  − 1 0 0 0 0 1

which allows to recognize the inconsistency by Theorem RCLS.

So the assumption of A ’s inverse leads to a logical inconsistency (the system can’t be both consistent and inconsistent), so our assumption is false. A is not invertible.

Its possible this example is less than satisfying. Just where did that particular choice of the vector b come from anyway? Stay tuned for an application of the future Theorem CSCS in Example CSAA. ⊠

Let’s look at one more matrix inverse before we embark on a more systematic study.

Example  MI
Matrix inverse
Consider the matrices,

A = 1 2 1 2 1 − 2 − 3 0 − 5 − 1 1 1 0 2 1 − 2 − 3 − 1 − 3 − 2 − 1 − 3 − 1 − 3 1 B = − 3 3 6 − 1 − 2 0 − 2 − 5 − 1 1 1 2 4 1 − 1 1 0 1 1 0 1 − 1 − 2 0 1

Then

A B = 1 2 1 2 1 − 2 − 3 0 − 5 − 1 1 1 0 2 1 − 2 − 3 − 1 − 3 − 2 − 1 − 3 − 1 − 3 1 − 3 3 6 − 1 − 2 0 − 2 − 5 − 1 1 1 2 4 1 − 1 1 0 1 1 0 1 − 1 − 2 0 1 = 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1  and B A = − 3 3 6 − 1 − 2 0 − 2 − 5 − 1 1 1 2 4 1 − 1 1 0 1 1 0 1 − 1 − 2 0 1 1 2 1 2 1 − 2 − 3 0 − 5 − 1 1 1 0 2 1 − 2 − 3 − 1 − 3 − 2 − 1 − 3 − 1 − 3 1 = 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1

so by Definition MI, we can say that A is invertible and write B = A − 1 . ⊠

We will now concern ourselves less with whether or not an inverse of a matrix exists, but instead with how you can find one when it does exist. In Section MINM we will have some theorems that allow us to more quickly and easily determine just when a matrix is invertible.

Subsection CIM: Computing the Inverse of a Matrix

We’ve seen that the matrices from Archetype B and Archetype K both have inverses, but these inverse matrices have just dropped from the sky. How would we compute an inverse? And just when is a matrix invertible, and when is it not? Writing a putative inverse with n 2 unknowns and solving the resultant n 2 equations is one approach. Applying this approach to 2 × 2 matrices can get us somewhere, so just for fun, let’s do it.

Theorem  TTMI
Two-by-Two Matrix Inverse
Suppose

A = a b c d

Then A is invertible if and only if a d − b c ≠ 0 . When A is invertible, then

A − 1 = 1 a d − b c d − b − c a

□

Proof   ( ⇐ ) Assume that a d − b c ≠ 0 . We will use the definition of the inverse of a matrix to establish that A has inverse ( Definition MI). Note that if a d − b c ≠ 0 then the displayed formula for A − 1 is legitimate since we are not dividing by zero). Using this proposed formula for the inverse of A , we compute

A A − 1 = a b c d 1 a d − b c d − b − c a = 1 a d − b c a d − b c 0 0 a d − b c = 1 0 0 1  and A − 1 A = 1 a d − b c d − b − c a a b c d = 1 a d − b c a d − b c 0 0 a d − b c = 1 0 0 1

By Definition MI this is sufficient to establish that A is invertible, and that the expression for A − 1 is correct.

( ⇒ ) Assume that A is invertible, and proceed with a proof by contradiction ( Technique CD), by assuming also that a d − b c = 0 . This translates to a d = b c . Let

B = e f g h

be a putative inverse of A . This means that

I 2 = A B = a b c d e f g h = a e + b g a f + b h c e + d g c f + d h

Working on the matrices on two ends of this equation, we will multiply the top row by c and the bottom row by a .

c 0 0 a = a c e + b c g a c f + b c h a c e + a d g a c f + a d h

We are assuming that a d = b c , so we can replace two occurrences of a d by b c in the bottom row of the right matrix.

c 0 0 a = a c e + b c g a c f + b c h a c e + b c g a c f + b c h

The matrix on the right now has two rows that are identical, and therefore the same must be true of the matrix on the left. Identical rows for the matrix on the left implies that a = 0 and c = 0 .

With this information, the product A B becomes

1 0 0 1 = I 2 = A B = a e + b g a f + b h c e + d g c f + d h = b g b h d g d h

So b g = d h = 1 and thus b , g , d , h are all nonzero. But then b h and d g (the “other corners”) must also be nonzero, so this is (finally) a contradiction. So our assumption was false and we see that a d − b c ≠ 0 whenever A has an inverse. ■

There are several ways one could try to prove this theorem, but there is a continual temptation to divide by one of the eight entries involved ( a through f ), but we can never be sure if these numbers are zero or not. This could lead to an analysis by cases, which is messy, messy, messy. Note how the above proof never divides, but always multiplies, and how zero/nonzero considerations are handled. Pay attention to the expression a d − b c , as we will see it again in a while ( Chapter D).

This theorem is cute, and it is nice to have a formula for the inverse, and a condition that tells us when we can use it. However, this approach becomes impractical for larger matrices, even though it is possible to demonstrate that, in theory, there is a general formula. (Think for a minute about extending this result to just 3 × 3 matrices. For starters, we need 18 letters!) Instead, we will work column-by-column. Let’s first work an example that will motivate the main theorem and remove some of the previous mystery.

Example  CMI
Computing a matrix inverse
Consider the matrix defined in Example MI as,

A = 1 2 1 2 1 − 2 − 3 0 − 5 − 1 1 1 0 2 1 − 2 − 3 − 1 − 3 − 2 − 1 − 3 − 1 − 3 1

For its inverse, we desire a matrix B so that A B = I 5 . Emphasizing the structure of the columns and employing the definition of matrix multiplication Definition MM,

A B = I 5 A [ B 1 | B 2 | B 3 | B 4 | B 5 ] = [ e 1 | e 2 | e 3 | e 4 | e 5 ] [ A B 1 | A B 2 | A B 3 | A B 4 | A B 5 ] = [ e 1 | e 2 | e 3 | e 4 | e 5 ] .

Equating the matrices column-by-column we have

A B 1 = e 1 A B 2 = e 2 A B 3 = e 3 A B 4 = e 4 A B 5 = e 5 .

Since the matrix B is what we are trying to compute, we can view each column, B i , as a column vector of unknowns. Then we have five systems of equations to solve, each with 5 equations in 5 variables. Notice that all 5 of these systems have the same coefficient matrix. We’ll now solve each system in turn,

 Row-reduce the augmented matrix of the linear system  ℒ S A , e 1 , 1 2 1 2 1 1 − 2 − 3 0 − 5 − 1 0 1 1 0 2 1 0 − 2 − 3 − 1 − 3 − 2 0 − 1 − 3 − 1 − 3 1 0 →  RREF 1 0 0 0 0 − 3 0 1  0 0 0 0 0 0 1 0 0 1 0 0 0 1  0 1 0 0 0 0 1 1  so B 1 = − 3 0 1 1 1  Row-reduce the augmented matrix of the linear system  ℒ S A , e 2 , 1 2 1 2 1 0 − 2 − 3 0 − 5 − 1 1 1 1 0 2 1 0 − 2 − 3 − 1 − 3 − 2 0 − 1 − 3 − 1 − 3 1 0 →  RREF 1 0 0 0 0 3 0 1  0 0 0 − 2 0 0 1 0 0 2 0 0 0 1  0 0 0 0 0 0 1 − 1  so B 2 = 3 − 2 2 0 − 1  Row-reduce the augmented matrix of the linear system  ℒ S A , e 3 , 1 2 1 2 1 0 − 2 − 3 0 − 5 − 1 0 1 1 0 2 1 1 − 2 − 3 − 1 − 3 − 2 0 − 1 − 3 − 1 − 3 1 0 →  RREF 1 0 0 0 0 6 0 1  0 0 0 − 5 0 0 1 0 0 4 0 0 0 1  0 1 0 0 0 0 1 − 2  so B 3 = 6 − 5 4 1 − 2  Row-reduce the augmented matrix of the linear system  ℒ S A , e 4 , 1 2 1 2 1 0 − 2 − 3 0 − 5 − 1 0 1 1 0 2 1 0 − 2 − 3 − 1 − 3 − 2 1 − 1 − 3 − 1 − 3 1 0 →  RREF 1 0 0 0 0 − 1 0 1  0 0 0 − 1 0 0 1 0 0 1 0 0 0 1  0 1 0 0 0 0 1 0  so B 4 = − 1 − 1 1 1 0  Row-reduce the augmented matrix of the linear system  ℒ S A , e 5 , 1 2 1 2 1 0 − 2 − 3 0 − 5 − 1 0 1 1 0 2 1 0 − 2 − 3 − 1 − 3 − 2 0 − 1 − 3 − 1 − 3 1 1 →  RREF 1 0 0 0 0 − 2 0 1  0 0 0 1 0 0 1 0 0 − 1 0 0 0 1  0 0 0 0 0 0 1 1  so B 5 = − 2 1 − 1 0 1

We can now collect our 5 solution vectors into the matrix B ,

B = [ B 1 | B 2 | B 3 | B 4 | B 5 ] = − 3 0 1 1 1 3 − 2 2 0 − 1 6 − 5 4 1 − 2 − 1 − 1 1 1 0 − 2 1 − 1 0 1 = − 3 3 6 − 1 − 2 0 − 2 − 5 − 1 1 1 2 4 1 − 1 1 0 1 1 0 1 − 1 − 2 0 1

By this method, we know that A B = I 5 . Check that B A = I 5 , and then we will know that we have the inverse of A . ⊠

Notice how the five systems of equations in the preceding example were all solved by exactly the same sequence of row operations. Wouldn’t it be nice to avoid this obvious duplication of effort? Our main theorem for this section follows, and it mimics this previous example, while also avoiding all the overhead.

Theorem  CINM
Computing the Inverse of a Nonsingular Matrix
Suppose A is a nonsingular square matrix of size n . Create the n × 2 n matrix M by placing the n × n identity matrix I n to the right of the matrix A . Let N be a matrix that is row-equivalent to M and in reduced row-echelon form. Finally, let J be the matrix formed from the final n columns of N . Then A J = I n . □

Proof   A is nonsingular, so by Theorem NMRRI there is a sequence of row operations that will convert A into I n . It is this same sequence of row operations that will convert M into N , since having the identity matrix in the first n columns of N is sufficient to guarantee that N is in reduced row-echelon form.

If we consider the systems of linear equations, ℒ S A , e i , 1 ≤ i ≤ n , we see that the aforementioned sequence of row operations will also bring the augmented matrix of each of these systems into reduced row-echelon form. Furthermore, the unique solution to ℒ S A , e i appears in column n + 1 of the row-reduced augmented matrix of the system and is identical to column n + i of N . Let N 1 , N 2 , N 3 , … , N 2 n denote the columns of N . So we find,

A J = A [ N n + 1 | N n + 2 | N n + 3 | … | N n + n ] = [ A N n + 1 | A N n + 2 | A N n + 3 | … | A N n + n ]   Definition MM = [ e 1 | e 2 | e 3 | … | e n ] = I n   Definition IM

as desired. ■

We have to be just a bit careful here about both what this theorem says and what it doesn’t say. If A is a nonsingular matrix, then we are guaranteed a matrix B such that A B = I n , and the proof gives us a process for constructing B . However, the definition of the inverse of a matrix ( Definition MI) requires that B A = I n also. So at this juncture we must compute the matrix product in the “opposite” order before we claim B as the inverse of A . However, we’ll soon see that this is always the case, in Theorem OSIS, so the title of this theorem is not inaccurate.

What if A is singular? At this point we only know that Theorem CINM cannot be applied. The question of A ’s inverse is still open. (But see Theorem NI in the next section.) We’ll finish by computing the inverse for the coefficient matrix of Archetype B, the one we just pulled from a hat in Example SABMI. There are more examples in the Archetypes ( Appendix A) to practice with, though notice that it is silly to ask for the inverse of a rectangular matrix (the sizes aren’t right) and not every square matrix has an inverse (remember Example MWIAA?).

Example  CMIAB
Computing a matrix inverse, Archetype B
Archetype B has a coefficient matrix given as

B = − 7 − 6 − 1 2 5 5 7 1 0 4  Exercising  Theorem CINM  we set M = − 7 − 6 − 1 2 1 0 0 5 5 7 0 1 0 1 0 4 0 0 1 .  which row reduces to N = 1 0 0 − 1 0 − 1 2 − 9 0 1 0 1 3 2 8 1 1 2 0 0 1 5 2 3 5 2 .  So B − 1 = − 1 0 − 1 2 − 9 1 3 2 8 1 1 2 5 2 3 5 2

once we check that B − 1 B = I 3 (the product in the opposite order is a consequence of the theorem). ⊠

While we can use a row-reducing procedure to compute any needed inverse, most computational devices have a built-in procedure to compute the inverse of a matrix straightaway.   See: Computation MI.MMA Computation MI.SAGE

Subsection PMI: Properties of Matrix Inverses

The inverse of a matrix enjoys some nice properties. We collect a few here. First, a matrix can have but one inverse.

Theorem  MIU
Matrix Inverse is Unique
Suppose the square matrix A has an inverse. Then A − 1 is unique. □

Proof   As described in Technique U, we will assume that A has two inverses. The hypothesis tells there is at least one. Suppose then that B and C are both inverses for A , so we know by Definition MI that A B = B A = I n and A C = C A = I n . Then we have,

B = B I n   Theorem MMIM = B ( A C )   Definition MI = ( B A ) C   Theorem MMA = I n C   Definition MI = C   Theorem MMIM

So we conclude that B and C are the same, and cannot be different. So any matrix that acts like an inverse, must be the inverse. ■

When most of us dress in the morning, we put on our socks first, followed by our shoes. In the evening we must then first remove our shoes, followed by our socks. Try to connect the conclusion of the following theorem with this everyday example.

Theorem  SS
Socks and Shoes
Suppose A and B are invertible matrices of size n . Then A B is an invertible matrix and ( A B ) − 1 = B − 1 A − 1 . □

Proof   At the risk of carrying our everyday analogies too far, the proof of this theorem is quite easy when we compare it to the workings of a dating service. We have a statement about the inverse of the matrix A B , which for all we know right now might not even exist. Suppose A B was to sign up for a dating service with two requirements for a compatible date. Upon multiplication on the left, and on the right, the result should be the identity matrix. In other words, A B ’s ideal date would be its inverse.

Now along comes the matrix B − 1 A − 1 (which we know exists because our hypothesis says both A and B are invertible and we can form the product of these two matrices), also looking for a date. Let’s see if B − 1 A − 1 is a good match for A B . First they meet at a non-committal neutral location, say a coffee shop, for quiet conversation:

( B − 1 A − 1 ) ( A B ) = B − 1 ( A − 1 A ) B   Theorem MMA = B − 1 I n B   Definition MI = B − 1 B   Theorem MMIM = I n   Definition MI  The first date having gone smoothly, a second, more serious, date is arranged, say dinner and a show: ( A B ) ( B − 1 A − 1 ) = A ( B B − 1 ) A − 1   Theorem MMA = A I n A − 1   Definition MI = A A − 1   Theorem MMIM = I n   Definition MI

So the matrix B − 1 A − 1 has met all of the requirements to be A B ’s inverse (date) and with the ensuing marriage proposal we can announce that ( A B ) − 1 = B − 1 A − 1 . ■

Theorem  MIMI
Matrix Inverse of a Matrix Inverse
Suppose A is an invertible matrix. Then A − 1 is invertible and ( A − 1 ) − 1 = A . □

Proof   As with the proof of Theorem SS, we examine if A is a suitable inverse for A − 1 (by definition, the opposite is true).

A A − 1 = I n   Definition MI  and A − 1 A = I n   Definition MI

The matrix A has met all the requirements to be the inverse of A − 1 , and so is invertible and we can write A = ( A − 1 ) − 1 . ■

Theorem  MIT
Matrix Inverse of a Transpose
Suppose A is an invertible matrix. Then A t is invertible and ( A t ) − 1 = ( A − 1 ) t . □

Proof   As with the proof of Theorem SS, we see if ( A − 1 ) t is a suitable inverse for A t . Apply Theorem MMT to see that

( A − 1 ) t A t = ( A A − 1 ) t   Theorem MMT = I n t   Definition MI = I n   Definition SYM  and A t ( A − 1 ) t = ( A − 1 A ) t   Theorem MMT = I n t   Definition MI = I n   Definition SYM

The matrix ( A − 1 ) t has met all the requirements to be the inverse of A t , and so is invertible and we can write ( A t ) − 1 = ( A − 1 ) t . ■

Theorem  MISM
Matrix Inverse of a Scalar Multiple
Suppose A is an invertible matrix and α is a nonzero scalar. Then α A − 1 = 1 α A − 1 and α A is invertible. □

Proof   As with the proof of Theorem SS, we see if 1 α A − 1 is a suitable inverse for α A .

1 α A − 1 α A = 1 α α A A − 1   Theorem MMSMM = 1 I n  Scalar multiplicative inverses = I n   Property OM  and α A 1 α A − 1 = α 1 α A − 1 A   Theorem MMSMM = 1 I n  Scalar multiplicative inverses = I n   Property OM

The matrix 1 α A − 1 has met all the requirements to be the inverse of α A , so we can write α A − 1 = 1 α A − 1 . ■

Notice that there are some likely theorems that are missing here. For example, it would be tempting to think that ( A + B ) − 1 = A − 1 + B − 1 , but this is false. Can you find a counterexample? (See Exercise MISLE.T10.)

Subsection READ: Reading Questions

Subsection EXC: Exercises

C16 If it exists, find the inverse of A = 1 0 1 1 1 1 2 − 1 1 , and check your answer.  
Contributed by  Chris Black Solution [ 681]

C17 If it exists, find the inverse of A = 2 − 1 1 1 2 1 3 1 2 , and check your answer.  
Contributed by  Chris Black Solution [ 681]

C18 If it exists, find the inverse of A = 1 3 1 1 2 1 2 2 1 , and check your answer.  
Contributed by  Chris Black Solution [ 681]

C19 If it exists, find the inverse of A = 1 3 1 0 2 1 2 2 1 , and check your answer.  
Contributed by  Chris Black Solution [ 681]

C21 Verify that B is the inverse of A .

A = 1 1 − 1 2 − 2 − 1 2 − 3 1 1 0 2 − 1 2 0 2 B = 4 2 0 − 1 8 4 − 1 − 1 − 1 0 1 0 − 6 − 3 1 1

Contributed by  Robert Beezer Solution [ 681]

C22 Recycle the matrices A and B from Exercise MISLE.C21 and set

c = 2 1 − 3 2 d = 1 1 1 1

Employ the matrix B to solve the two linear systems ℒ S A , c and ℒ S A , d .  
Contributed by  Robert Beezer Solution [ 681]

C23 If it exists, find the inverse of the 2 × 2 matrix

A = 7 3 5 2

and check your answer. (See Theorem TTMI.)  
Contributed by  Robert Beezer

C24 If it exists, find the inverse of the 2 × 2 matrix

A = 6 3 4 2

and check your answer. (See Theorem TTMI.)  
Contributed by  Robert Beezer

C25 At the conclusion of Example CMI, verify that B A = I 5 by computing the matrix product.  
Contributed by  Robert Beezer

C26 Let

D = 1 − 1 3 − 2 1 − 2 3 − 5 3 0 1 − 1 4 − 2 2 − 1 4 − 1 0 4 1 0 5 − 2 5

Compute the inverse of D , D − 1 , by forming the 5 × 1 0 matrix D I 5 and row-reducing ( Theorem CINM). Then use a calculator to compute D − 1 directly.  
Contributed by  Robert Beezer Solution [ 682]

C27 Let

E = 1 − 1 3 − 2 1 − 2 3 − 5 3 − 1 1 − 1 4 − 2 2 − 1 4 − 1 0 2 1 0 5 − 2 4

Compute the inverse of E , E − 1 , by forming the 5 × 1 0 matrix E I 5 and row-reducing ( Theorem CINM). Then use a calculator to compute E − 1 directly.  
Contributed by  Robert Beezer Solution [ 682]

C28 Let

C = 1 1 3 1 − 2 − 1 − 4 − 1 1 4 1 0 2 − 2 0 − 4 5

Compute the inverse of C , C − 1 , by forming the 4 × 8 matrix C I 4 and row-reducing ( Theorem CINM). Then use a calculator to compute C − 1 directly.  
Contributed by  Robert Beezer Solution [ 683]

C40 Find all solutions to the system of equations below, making use of the matrix inverse found in Exercise MISLE.C28.

x 1 + x 2 + 3 x 3 + x 4 = − 4 − 2 x 1 − x 2 − 4 x 3 − x 4 = 4 x 1 + 4 x 2 + 1 0 x 3 + 2 x 4 = − 2 0 − 2 x 1 − 4 x 3 + 5 x 4 = 9

Contributed by  Robert Beezer Solution [ 683]

C41 Use the inverse of a matrix to find all the solutions to the following system of equations.

x 1 + 2 x 2 − x 3 = − 3 2 x 1 + 5 x 2 − x 3 = − 4 − x 1 − 4 x 2 = 2

Contributed by  Robert Beezer Solution [ 684]

C42 Use a matrix inverse to solve the linear system of equations.

x 1 − x 2 + 2 x 3 = 5 x 1 − 2 x 3 = − 8 2 x 1 − x 2 − x 3 = − 6

Contributed by  Robert Beezer Solution [ 685]

T10 Construct an example to demonstrate that ( A + B ) − 1 = A − 1 + B − 1 is not true for all square matrices A and B of the same size.  
Contributed by  Robert Beezer Solution [ 687]

Subsection SOL: Solutions

C16 Contributed by  Chris Black Statement [ 674]
Answer: A − 1 = − 2 1 1 − 1 1 0 3 − 1 − 1 .

C17 Contributed by  Chris Black Statement [ 674]
The procedure we have for finding a matrix inverse fails for this matrix A since A does not row-reduce to I 3 . We suspect in this case that A is not invertible, although we do not yet know that concretely. (Stay tuned for upcoming revelations in Section MINM!)

C18 Contributed by  Chris Black Statement [ 674]
Answer: A − 1 = 0 − 1 1 1 − 1 0 − 2 4 − 1

C19 Contributed by  Chris Black Statement [ 674]
Answer: A − 1 = 0 − 1 ∕ 2 1 ∕ 2 1 − 1 ∕ 2 − 1 ∕ 2 − 2 2 1

C21 Contributed by  Robert Beezer Statement [ 674]
Check that both matrix products ( Definition MM) A B and B A equal the 4 × 4 identity matrix I 4 ( Definition IM).

C22 Contributed by  Robert Beezer Statement [ 675]
Represent each of the two systems by a vector equality, A x = c and A y = d . Then in the spirit of Example SABMI, solutions are given by

x = B c = 8 2 1 − 5 − 1 6 y = B d = 5 1 0 0 − 7

Notice how we could solve many more systems having A as the coefficient matrix, and how each such system has a unique solution. You might check your work by substituting the solutions back into the systems of equations, or forming the linear combinations of the columns of A suggested by Theorem SLSLC.

C26 Contributed by  Robert Beezer Statement [ 676]
The inverse of D is

D − 1 = − 7 − 6 − 3 2 1 − 7 − 4 2 2 − 1 − 5 − 2 3 1 − 1 − 6 − 3 1 1 0 4 2 − 2 − 1 1

C27 Contributed by  Robert Beezer Statement [ 677]
The matrix E has no inverse, though we do not yet have a theorem that allows us to reach this conclusion. However, when row-reducing the matrix E I 5 , the first 5 columns will not row-reduce to the 5 × 5 identity matrix, so we are a t a loss on how we might compute the inverse. When requesting that your calculator compute E − 1 , it should give some indication that E does not have an inverse.

C28 Contributed by  Robert Beezer Statement [ 678]
Employ Theorem CINM,

1 1 3 1 1 0 0 0 − 2 − 1 − 4 − 1 0 1 0 0 1 4 1 0 2 0 0 1 0 − 2 0 − 4 5 0 0 0 1 →  RREF 1 0 0 0 3 8 1 8 − 5 − 2 0 1  0 0 9 6 4 7 − 1 2 − 5 0 0 1 0 − 3 9 − 1 9 5 2 0 0 0 1  − 1 6 − 8 2 1

And therefore we see that C is nonsingular ( C row-reduces to the identity matrix, Theorem NMRRI) and by Theorem CINM,

C − 1 = 3 8 1 8 − 5 − 2 9 6 4 7 − 1 2 − 5 − 3 9 − 1 9 5 2 − 1 6 − 8 2 1

C40 Contributed by  Robert Beezer Statement [ 678]
View this system as ℒ S C , b , where C is the 4 × 4 matrix from Exercise MISLE.C28 and b = − 4 4 − 2 0 9 . Since C was seen to be nonsingular in Exercise MISLE.C28 Theorem SNCM says the solution, which is unique by Theorem NMUS, is given by

C − 1 b = 3 8 1 8 − 5 − 2 9 6 4 7 − 1 2 − 5 − 3 9 − 1 9 5 2 − 1 6 − 8 2 1 − 4 4 − 2 0 9 = 2 − 1 − 2 1

Notice that this solution can be easily checked in the original system of equations.

C41 Contributed by  Robert Beezer Statement [ 679]
The coefficient matrix of this system of equations is

A = 1 2 − 1 2 5 − 1 − 1 − 4 0

and the vector of constants is b = − 3 − 4 2 . So by Theorem SLEMM we can convert the system to the form A x = b . Row-reducing this matrix yields the identity matrix so by Theorem NMRRI we know A is nonsingular. This allows us to apply Theorem SNCM to find the unique solution as

x = A − 1 b = − 4 4 3 1 − 1 − 1 − 3 2 1 − 3 − 4 2 = 2 − 1 3

Remember, you can check this solution easily by evaluating the matrix-vector product A x ( Definition MVP).

C42 Contributed by  Robert Beezer Statement [ 679]
We can reformulate the linear system as a vector equality with a matrix-vector product via Theorem SLEMM. The system is then represented by A x = b where

A = 1 − 1 2 1 0 − 2 2 − 1 − 1 b = 5 − 8 − 6

According to Theorem SNCM, if A is nonsingular then the (unique) solution will be given by A − 1 b . We attempt the computation of A − 1 through Theorem CINM, or with our favorite computational device and obtain,

A − 1 = 2 3 − 2 3 5 − 4 1 1 − 1

So by Theorem NI, we know A is nonsingular, and so the unique solution is

A − 1 b = 2 3 − 2 3 5 − 4 1 1 − 1 5 − 8 − 6 = − 2 − 1 3

T10 Contributed by  Robert Beezer Statement [ 680]
For a large collection of small examples, let D be any 2 × 2 matrix that has an inverse ( Theorem TTMI can help you construct such a matrix, I 2 is a simple choice). Set A = D and B = ( − 1 ) D . While A − 1 and B − 1 both exist, what is A + B − 1 ?

For a large collection of examples of any size, consider A = B = I n . Can the proposed statement be salvaged to become a theorem?

Section MINM  Matrix Inverses and Nonsingular Matrices

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

We saw in Theorem CINM that if a square matrix A is nonsingular, then there is a matrix B so that A B = I n . In other words, B is halfway to being an inverse of A . We will see in this section that B automatically fulfills the second condition ( B A = I n ). Example MWIAA showed us that the coefficient matrix from Archetype A had no inverse. Not coincidentally, this coefficient matrix is singular. We’ll make all these connections precise now. Not many examples or definitions in this section, just theorems.

Parent topic: Chapter M  Matrices

Previous topic: Section MISLE  Matrix Inverses and Systems of Linear Equations

Next topic: Section CRS  Column and Row Spaces

Subsection NMI: Nonsingular Matrices are Invertible

We need a couple of technical results for starters. Some books would call these minor, but essential, results “lemmas.” We’ll just call ’em theorems.  See Technique LC for more on the distinction. 

The first of these technical results is interesting in that the hypothesis says something about a product of two square matrices and the conclusion then says the same thing about each individual matrix in the product. This result has an analogy in the algebra of complex numbers: suppose α , β ∈ ℂ , then α β ≠ 0 if and only if α ≠ 0 and β ≠ 0 . We can view this result as suggesting that the term “nonsingular” for matrices is like the term “nonzero” for scalars.

Theorem  NPNT
Nonsingular Product has Nonsingular Terms
Suppose that A and B are square matrices of size n . The product A B is nonsingular if and only if A and B are both nonsingular. □

Proof   ( ⇒ ) We’ll do this portion of the proof in two parts, each as a proof by contradiction ( Technique CD). Assume that A B is nonsingular. Establishing that B is nonsingular is the easier part, so we will do it first, but in reality, we will need to know that B is nonsingular when we prove that A is nonsingular.

You can also think of this proof as being a study of four possible conclusions in the table below. One of the four rows must happen (the list is exhaustive). In the proof we learn that the first three rows lead to contradictions, and so are impossible. That leaves the fourth row as a certainty, which is our desired conclusion.

Part 1. Suppose B is singular. Then there is a nonzero vector z that is a solution to ℒ S B , 0 . So

( A B ) z = A ( B z )   Theorem MMA = A 0   Theorem SLEMM = 0   Theorem MMZM

Because z is a nonzero solution to ℒ S A B , 0 , we conclude that A B is singular ( Definition NM). This is a contradiction, so B is nonsingular, as desired.

Part 2. Suppose A is singular. Then there is a nonzero vector y that is a solution to ℒ S A , 0 . Now consider the linear system ℒ S B , y . Since we know B is nonsingular from Case 1, the system has a unique solution ( Theorem NMUS), which we will denote as w . We first claim w is not the zero vector either. Assuming the opposite, suppose that w = 0 ( Technique CD). Then

y = B w   Theorem SLEMM = B 0  Hypothesis = 0   Theorem MMZM  contrary to  y  being nonzero. So  w ≠ 0 . The pieces are in place, so here we go, ( A B ) w = A ( B w )   Theorem MMA = A y   Theorem SLEMM = 0   Theorem SLEMM

So w is a nonzero solution to ℒ S A B , 0 , and thus we can say that A B is singular ( Definition NM). This is a contradiction, so A is nonsingular, as desired.

( ⇐ ) Now assume that both A and B are nonsingular. Suppose that x ∈ ℂ n is a solution to ℒ S A B , 0 . Then

0 = A B x   Theorem SLEMM = A B x   Theorem MMA

By Theorem SLEMM, B x is a solution to ℒ S A , 0 , and by the definition of a nonsingular matrix ( Definition NM), we conclude that B x = 0 . Now, by an entirely similar argument, the nonsingularity of B forces us to conclude that x = 0 . So the only solution to ℒ S A B , 0 is the zero vector and we conclude that A B is nonsingular by Definition NM. ■

This is a powerful result in the “forward” direction, because it allows us to begin with a hypothesis that something complicated (the matrix product A B ) has the property of being nonsingular, and we can then conclude that the simpler constituents ( A and B individually) then also have the property of being nonsingular. If we had thought that the matrix product was an artificial construction, results like this would make us begin to think twice.

The contrapositive of this result is equally interesting. It says that A or B (or both) is a singular matrix if and only if the product A B is singular. Notice how the negation of the theorem’s conclusion ( A and B both nonsingular) becomes the statement “at least one of A and B is singular.” (See Technique CP.)

Theorem  OSIS
One-Sided Inverse is Sufficient
Suppose A and B are square matrices of size n such that A B = I n . Then B A = I n . □

Proof   The matrix I n is nonsingular (since it row-reduces easily to I n , Theorem NMRRI). So A and B are nonsingular by Theorem NPNT, so in particular B is nonsingular. We can therefore apply Theorem CINM to assert the existence of a matrix C so that B C = I n . This application of Theorem CINM could be a bit confusing, mostly because of the names of the matrices involved. B is nonsingular, so there must be a “right-inverse” for B , and we’re calling it C .

Now

B A = ( B A ) I n   Theorem MMIM = ( B A ) ( B C )   Theorem CINM = B ( A B ) C   Theorem MMA = B I n C  Hypothesis = B C   Theorem MMIM = I n   Theorem CINM

which is the desired conclusion. ■

So Theorem OSIS tells us that if A is nonsingular, then the matrix B guaranteed by Theorem CINM will be both a “right-inverse” and a “left-inverse” for A , so A is invertible and A − 1 = B .

So if you have a nonsingular matrix, A , you can use the procedure described in Theorem CINM to find an inverse for A . If A is singular, then the procedure in Theorem CINM will fail as the first n columns of M will not row-reduce to the identity matrix. However, we can say a bit more. When A is singular, then A does not have an inverse (which is very different from saying that the procedure in Theorem CINM fails to find an inverse). This may feel like we are splitting hairs, but its important that we do not make unfounded assumptions. These observations motivate the next theorem.

Theorem  NI
Nonsingularity is Invertibility
Suppose that A is a square matrix. Then A is nonsingular if and only if A is invertible. □

Proof   ( ⇐ ) Since A is invertible, we can write I n = A A − 1 ( Definition MI). Notice that I n is nonsingular ( Theorem NSRRI) so Theorem NPNT implies that A (and A − 1 ) is nonsingular.

( ⇒ ) Suppose now that A is nonsingular. By Theorem CINM we find B so that A B = I n . Then Theorem OSIS tells us that B A = I n . So B is A ’s inverse, and by construction, A is invertible. ■

So for a square matrix, the properties of having an inverse and of having a trivial null space are one and the same. Can’t have one without the other.

Theorem  NME3
Nonsingular Matrix Equivalences, Round 3
Suppose that A is a square matrix of size n . The following are equivalent.

□

Proof   We can update our list of equivalences for nonsingular matrices (Theorem NME2) with the equivalent condition from Theorem NI. ■

In the case that A is a nonsingular coefficient matrix of a system of equations, the inverse allows us to very quickly compute the unique solution, for any vector of constants.

Theorem  SNCM
Solution with Nonsingular Coefficient Matrix
Suppose that A is nonsingular. Then the unique solution to ℒ S A , b is A − 1 b . □

Proof   By Theorem NMUS we know already that ℒ S A , b has a unique solution for every choice of b . We need to show that the expression stated is indeed a solution ( the solution). That’s easy, just “plug it in” to the corresponding vector equation representation ( Theorem SLEMM),

A A − 1 b = A A − 1 b   Theorem MMA = I n b   Definition MI = b   Theorem MMIM

Since A x = b is true when we substitute A − 1 b for x , A − 1 b is a (the!) solution to ℒ S A , b . ■

Subsection UM: Unitary Matrices

Recall that the adjoint of a matrix is A ∗ = A ¯ t (Definition A).

Definition  UM
Unitary Matrices
Suppose that U is a square matrix of size n such that U ∗ U = I n . Then we say U is unitary. △

This condition may seem rather far-fetched at first glance. Would there be any matrix that behaved this way? Well, yes, here’s one.

Example  UM3
Unitary matrix of size 3

U = 1 + i 5 3 + 2 i 5 5 2 + 2 i 2 2 1 − i 5 2 + 2 i 5 5 − 3 + i 2 2 i 5 3 − 5 i 5 5 − 2 2 2

The computations get a bit tiresome, but if you work your way through the computation of U ∗ U , you will arrive at the 3 × 3 identity matrix I 3 . ⊠

Unitary matrices do not have to look quite so gruesome. Here’s a larger one that is a bit more pleasing.

Example  UPM
Unitary permutation matrix
The matrix

P = 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 1 0 0

is unitary as can be easily checked. Notice that it is just a rearrangement of the columns of the 5 × 5 identity matrix, I 5 ( Definition IM).

An interesting exercise is to build another 5 × 5 unitary matrix, R , using a different rearrangement of the columns of I 5 . Then form the product P R . This will be another unitary matrix ( Exercise MINM.T10). If you were to build all 5 ! = 5 × 4 × 3 × 2 × 1 = 1 2 0 matrices of this type you would have a set that remains closed under matrix multiplication. It is an example of another algebraic structure known as a group since together the set and the one operation (matrix multiplication here) is closed, associative, has an identity ( I 5 ), and inverses ( Theorem UMI). Notice though that the operation in this group is not commutative! ⊠

If a matrix A has only real number entries (we say it is a real matrix) then the defining property of being unitary simplifies to A t A = I n . In this case we, and everybody else, calls the matrix orthogonal, so you may often encounter this term in your other reading when the complex numbers are not under consideration.

Unitary matrices have easily computed inverses. They also have columns that form orthonormal sets. Here are the theorems that show us that unitary matrices are not as strange as they might initially appear.

Theorem  UMI
Unitary Matrices are Invertible
Suppose that U is a unitary matrix of size n . Then U is nonsingular, and U − 1 = U ∗ . □

Proof   By Definition UM, we know that U ∗ U = I n . The matrix I n is nonsingular (since it row-reduces easily to I n , Theorem NMRRI). So by Theorem NPNT, U and U ∗ are both nonsingular matrices.

The equation U ∗ U = I n gets us halfway to an inverse of U , and Theorem OSIS tells us that then U U ∗ = I n also. So U and U ∗ are inverses of each other ( Definition MI). ■

Theorem  CUMOS
Columns of Unitary Matrices are Orthonormal Sets
Suppose that A is a square matrix of size n with columns S = A 1 , A 2 , A 3 , … , A n . Then A is a unitary matrix if and only if S is an orthonormal set. □

Proof   The proof revolves around recognizing that a typical entry of the product A ∗ A is an inner product of columns of A . Here are the details to support this claim.

A ∗ A i j = ∑ k = 1 n A ∗ i k A k j   Theorem EMP = ∑ k = 1 n A ¯ t i k A k j   Theorem EMP = ∑ k = 1 n A ¯ k i A k j   Definition TM = ∑ k = 1 n A k i ¯ A k j   Definition CCM = ∑ k = 1 n A k j A k i ¯   Property CMCN = ∑ k = 1 n A j k A i k ¯ = A j , A i   Definition IP

We now employ this equality in a chain of equivalences,

  S = A 1 , A 2 , A 3 , … , A n  is an orthonormal set ⇔ A j , A i = 0  if  i ≠ j 1  if  i = j   Definition ONS ⇔ A ∗ A i j = 0  if  i ≠ j 1  if  i = j ⇔ A ∗ A i j = I n i j , 1 ≤ i ≤ n , 1 ≤ j ≤ n   Definition IM ⇔ A ∗ A = I n   Definition ME ⇔   A  is a unitary matrix   Definition UM

■

Example  OSMC
Orthonormal set from matrix columns
The matrix

U = 1 + i 5 3 + 2 i 5 5 2 + 2 i 2 2 1 − i 5 2 + 2 i 5 5 − 3 + i 2 2 i 5 3 − 5 i 5 5 − 2 2 2

from Example UM3 is a unitary matrix. By Theorem CUMOS, its columns

1 + i 5 1 − i 5 i 5 , 3 + 2 i 5 5 2 + 2 i 5 5 3 − 5 i 5 5 , 2 + 2 i 2 2 − 3 + i 2 2 − 2 2 2

form an orthonormal set. You might find checking the six inner products of pairs of these vectors easier than doing the matrix product U ∗ U . Or, because the inner product is anti-commutative ( Theorem IPAC) you only need check three inner products (see Exercise MINM.T12). ⊠

When using vectors and matrices that only have real number entries, orthogonal matrices are those matrices with inverses that equal their transpose. Similarly, the inner product is the familiar dot product. Keep this special case in mind as you read the next theorem.

Theorem  UMPIP
Unitary Matrices Preserve Inner Products
Suppose that U is a unitary matrix of size n and u and v are two vectors from ℂ n . Then

U u , U v = u , v  and U v = v

□

Proof  

U u , U v = ( U u ) t U v ¯   Theorem MMIP = u t U t U v ¯   Theorem MMT = u t U t U ¯ v ¯   Theorem MMCC = u t U ¯ ¯ t U ¯ v ¯   Theorem CCT = u t U ¯ t ¯ U ¯ v ¯   Theorem MCT = u t U ¯ t U ¯ v ¯   Theorem MMCC = u t U ∗ U ¯ v ¯   Definition A = u t I n ¯ v ¯   Definition UM = u t I n v ¯   Definition IM = u t v ¯   Theorem MMIM = u , v   Theorem MMIP

The second conclusion is just a specialization of the first conclusion.

U v = U v 2 = U v , U v   Theorem IPN = v , v = v 2   Theorem IPN = v

■

Aside from the inherent interest in this theorem, it makes a bigger statement about unitary matrices. When we view vectors geometrically as directions or forces, then the norm equates to a notion of length. If we transform a vector by multiplication with a unitary matrix, then the length (norm) of that vector stays the same. If we consider column vectors with two or three slots containing only real numbers, then the inner product of two such vectors is just the dot product, and this quantity can be used to compute the angle between two vectors. When two vectors are multiplied (transformed) by the same unitary matrix, their dot product is unchanged and their individual lengths are unchanged. The results in the angle between the two vectors remaining unchanged.

A “unitary transformation” (matrix-vector products with unitary matrices) thus preserve geometrical relationships among vectors representing directions, forces, or other physical quantities. In the case of a two-slot vector with real entries, this is simply a rotation. These sorts of computations are exceedingly important in computer graphics such as games and real-time simulations, especially when increased realism is achieved by performing many such computations quickly. We will see unitary matrices again in subsequent sections (especially Theorem OD) and in each instance, consider the interpretation of the unitary matrix as a sort of geometry-preserving transformation. Some authors use the term isometry to highlight this behavior. We will speak loosely of a unitary matrix as being a sort of generalized rotation.

A final reminder: the terms “dot product,” “symmetric matrix” and “orthogonal matrix” used in reference to vectors or matrices with real number entries correspond to the terms “inner product,” “Hermitian matrix” and “unitary matrix” when we generalize to include complex number entries, so keep that in mind as you read elsewhere.

Subsection READ: Reading Questions

Subsection EXC: Exercises

C20 Let A = 1 2 1 0 1 1 1 0 2 and B = − 1 1 0 1 2 1 0 1 1 . Verify that A B is nonsingular.  
Contributed by  Chris Black

C40 Solve the system of equations below using the inverse of a matrix.

x 1 + x 2 + 3 x 3 + x 4 = 5 − 2 x 1 − x 2 − 4 x 3 − x 4 = − 7 x 1 + 4 x 2 + 1 0 x 3 + 2 x 4 = 9 − 2 x 1 − 4 x 3 + 5 x 4 = 9

Contributed by  Robert Beezer Solution [ 714]

M10 Find values of x , y z so that matrix A = 1 2 x 3 0 y 1 1 z is invertible.  
Contributed by  Chris Black Solution [ 715]

M11 Find values of x , y z so that matrix A = 1 x 1 1 y 4 0 z 5 is singular.  
Contributed by  Chris Black Solution [ 716]

M15 If A and B are n × n matrices, A is nonsingular, and B is singular, show directly that A B is singular, without using Theorem NPNT.  
Contributed by  Chris Black Solution [ 717]

M20 Construct an example of a 4 × 4 unitary matrix.  
Contributed by  Robert Beezer Solution [ 714]

M80 Matrix multiplication interacts nicely with many operations. But not always with transforming a matrix to reduced row-echelon form. Suppose that A is an m × n matrix and B is an n × p matrix. Let P be a matrix that is row-equivalent to A and in reduced row-echelon form, Q be a matrix that is row-equivalent to B and in reduced row-echelon form, and let R be a matrix that is row-equivalent to A B and in reduced row-echelon form. Is P Q = R ? (In other words, with nonstandard notation, is  rref ( A )  rref ( B ) =  rref ( A B ) ?)

Construct a counterexample to show that, in general, this statement is false. Then find a large class of matrices where if A and B are in the class, then the statement is true.  
Contributed by  Mark Hamrick Solution [ 717]

T10 Suppose that Q and P are unitary matrices of size n . Prove that Q P is a unitary matrix.  
Contributed by  Robert Beezer

T11 Prove that Hermitian matrices ( Definition HM) have real entries on the diagonal. More precisely, suppose that A is a Hermitian matrix of size n . Then A i i ∈ ℝ , 1 ≤ i ≤ n .  
Contributed by  Robert Beezer

T12 Suppose that we are checking if a square matrix of size n is unitary. Show that a straightforward application of Theorem CUMOS requires the computation of n 2 inner products when the matrix is unitary, and fewer when the matrix is not orthogonal. Then show that this maximum number of inner products can be reduced to 1 2 n ( n + 1 ) in light of Theorem IPAC.  
Contributed by  Robert Beezer

T25 The notation A k means a repeated matrix product between k copies of the square matrix A .
(a) Assume A is an n × n matrix where A 2 = O (which does not imply that A = O .) Prove that I n − A is invertible by showing that I n + A is an inverse of I n − A .
(b) Assume that A is an n × n matrix where A 3 = O . Prove that I n − A is invertible.
(c) Form a general theorem based on your observations from parts (a) and (b) and provide a proof.  
Contributed by  Manley Perkel

Subsection SOL: Solutions

C40 Contributed by  Robert Beezer Statement [ 710]
The coefficient matrix and vector of constants for the system are

1 1 3 1 − 2 − 1 − 4 − 1 1 4 1 0 2 − 2 0 − 4 5 b = 5 − 7 9 9

A − 1 can be computed by using a calculator, or by the method of Theorem CINM. Then Theorem SNCM says the unique solution is

A − 1 b = 3 8 1 8 − 5 − 2 9 6 4 7 − 1 2 − 5 − 3 9 − 1 9 5 2 − 1 6 − 8 2 1 5 − 7 9 9 = 1 − 2 1 3

M20 Contributed by  Robert Beezer Statement [ 711]
The 4 × 4 identity matrix, I 4 , would be one example ( Definition IM). Any of the 23 other rearrangements of the columns of I 4 would be a simple, but less trivial, example. See Example UPM.

M10 Contributed by  Chris Black Statement [ 710]
There are an infinite number of possible answers. We want to find a vector x y z so that the set

S = 1 3 1 , 2 0 1 , x y z

is a linearly independent set. We need a vector not in the span of the first two columns, which geometrically means that we need it to not be in the same plane as the first two columns of A . We can choose any values we want for x and y , and then choose a value of z that makes the three vectors independent.

I will (arbitrarily) choose x = 1 , y = 1 . Then, we have

A = 1 2 1 3 0 1 1 1 z →  RREF 1 0 2 z − 1 0 1  1 − z 0 0 4 − 6 z

which is invertible if and only if 4 − 6 z ≠ 0 . Thus, we can choose any value as long as z ≠ 2 3 , so we choose z = 0 , and we have found a matrix A = 1 2 1 3 0 1 1 1 0 that is invertible.

M11 Contributed by  Chris Black Statement [ 711]
There are an infinite number of possible answers. We need the set of vectors

S = 1 1 0 , x y z , 1 4 5

to be linearly dependent. One way to do this by inspection is to have x y z = 1 4 5 . Thus, if we let x = 1 , y = 4 , z = 5 , then the matrix A = 1 1 1 1 4 4 0 5 5 is singular.

M15 Contributed by  Chris Black Statement [ 711]
If B is singular, then there exists a vector x ≠ 0 so that x ∈ N B . Thus, B x = 0 , so A ( B x ) = ( A B ) x = 0 , so x ∈ N A B . Since the null space of A B is not trivial, A B is a nonsingular matrix.

M80 Contributed by  Robert Beezer Statement [ 711]
Take

A = 1 0 0 0 B = 0 0 1 0

Then A is already in reduced row-echelon form, and by swapping rows, B row-reduces to A . So the product of the row-echelon forms of A is A A = A ≠ O . However, the product A B is the 2 × 2 zero matrix, which is in reduced-echelon form, and not equal to A A . When you get there, Theorem PEEF or Theorem EMDRO might shed some light on why we would not expect this statement to be true in general.

If A and B are nonsingular, then A B is nonsingular ( Theorem NPNT), and all three matrices A , B and A B row-reduce to the identity matrix ( Theorem NMRRI). By Theorem MMIM, the desired relationship is true.

Section CRS  Column and Row Spaces

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Theorem SLSLC showed us that there is a natural correspondence between solutions to linear systems and linear combinations of the columns of the coefficient matrix. This idea motivates the following important definition.

Definition  CSM
Column Space of a Matrix
Suppose that A is an m × n matrix with columns A 1 , A 2 , A 3 , … , A n . Then the column space of A , written C A , is the subset of ℂ m containing all linear combinations of the columns of A ,

C A = A 1 , A 2 , A 3 , … , A n

(This definition contains Notation CSM.) △

Some authors refer to the column space of a matrix as the range, but we will reserve this term for use with linear transformations ( Definition RLT).

Parent topic: Chapter M  Matrices

Previous topic: Section MINM  Matrix Inverses and Nonsingular Matrices

Next topic: Chapter VS  Vector Spaces

Subsection CSSE: Column Spaces and Systems of Equations

Upon encountering any new set, the first question we ask is what objects are in the set, and which objects are not? Here’s an example of one way to answer this question, and it will motivate a theorem that will then answer the question precisely.

Example  CSMCS
Column space of a matrix and consistent systems
Archetype D and Archetype E are linear systems of equations, with an identical 3 × 4 coefficient matrix, which we call A here. However, Archetype D is consistent, while Archetype E is not. We can explain this difference by employing the column space of the matrix A .

The column vector of constants, b , in Archetype D is

b = 8 − 1 2 4

One solution to ℒ S A , b , as listed, is

x = 7 8 1 3

By Theorem SLSLC, we can summarize this solution as a linear combination of the columns of A that equals b ,

7 2 − 3 1 + 8 1 4 1 + 1 7 − 5 4 + 3 − 7 − 6 − 5 = 8 − 1 2 4 = b .

This equation says that b is a linear combination of the columns of A , and then by Definition CSM, we can say that b ∈ C A .

On the other hand, Archetype E is the linear system ℒ S A , c , where the vector of constants is

c = 2 3 2

and this system of equations is inconsistent. This means c ∉ C A , for if it were, then it would equal a linear combination of the columns of A and Theorem SLSLC would lead us to a solution of the system ℒ S A , c . ⊠

So if we fix the coefficient matrix, and vary the vector of constants, we can sometimes find consistent systems, and sometimes inconsistent systems. The vectors of constants that lead to consistent systems are exactly the elements of the column space. This is the content of the next theorem, and since it is an equivalence, it provides an alternate view of the column space.

Theorem  CSCS
Column Spaces and Consistent Systems
Suppose A is an m × n matrix and b is a vector of size m . Then b ∈ C A if and only if ℒ S A , b is consistent. □

Proof   ( ⇒ ) Suppose b ∈ C A . Then we can write b as some linear combination of the columns of A . By Theorem SLSLC we can use the scalars from this linear combination to form a solution to ℒ S A , b , so this system is consistent.

( ⇐ ) If ℒ S A , b is consistent, there is a solution that may be used with Theorem SLSLC to write b as a linear combination of the columns of A . This qualifies b for membership in C A . ■

This theorem tells us that asking if the system ℒ S A , b is consistent is exactly the same question as asking if b is in the column space of A . Or equivalently, it tells us that the column space of the matrix A is precisely those vectors of constants, b , that can be paired with A to create a system of linear equations ℒ S A , b that is consistent.

Employing Theorem SLEMM we can form the chain of equivalences

b ∈ C A ⇔ ℒ S A , b   is consistent ⇔ A x = b   for some  x

Thus, an alternative (and popular) definition of the column space of an m × n matrix A is

C A = y ∈ ℂ m y = A x   for some  x ∈ ℂ n = A x x ∈ ℂ n ⊆ ℂ m

We recognize this as saying create all the matrix vector products possible with the matrix A by letting x range over all of the possibilities. By Definition MVP we see that this means take all possible linear combinations of the columns of A — precisely the definition of the column space ( Definition CSM) we have chosen.

Notice how this formulation of the column space looks very much like the definition of the null space of a matrix ( Definition NSM), but for a rectangular matrix the column vectors of C A and N A have different sizes, so the sets are very different.

Given a vector b and a matrix A it is now very mechanical to test if b ∈ C A . Form the linear system ℒ S A , b , row-reduce the augmented matrix, A b , and test for consistency with Theorem RCLS. Here’s an example of this procedure.

Example  MCSM
Membership in the column space of a matrix
Consider the column space of the 3 × 4 matrix A ,

A = 3 2 1 − 4 − 1 1 − 2 3 2 − 4 6 − 8

We first show that v = 1 8 − 6 1 2 is in the column space of A , v ∈ C A . Theorem CSCS says we need only check the consistency of ℒ S A , v . Form the augmented matrix and row-reduce,

3 2 1 − 4 1 8 − 1 1 − 2 3 − 6 2 − 4 6 − 8 1 2 →  RREF 1 0 1 − 2 6 0 1  − 1 1 0 0 0 0 0 0

Without a leading 1 in the final column, Theorem RCLS tells us the system is consistent and therefore by Theorem CSCS, v ∈ C A .

If we wished to demonstrate explicitly that v is a linear combination of the columns of A , we can find a solution (any solution) of ℒ S A , v and use Theorem SLSLC to construct the desired linear combination. For example, set the free variables to x 3 = 2 and x 4 = 1 . Then a solution has x 2 = 1 and x 1 = 6 . Then by Theorem SLSLC,

v = 1 8 − 6 1 2 = 6 3 − 1 2 + 1 2 1 − 4 + 2 1 − 2 6 + 1 − 4 3 − 8

Now we show that w = 2 1 − 3 is not in the column space of A , w ∉ C A . Theorem CSCS says we need only check the consistency of ℒ S A , w . Form the augmented matrix and row-reduce,

3 2 1 − 4 2 − 1 1 − 2 3 1 2 − 4 6 − 8 − 3 →  RREF 1 0 1 − 2 0 0 1  − 1 1 0 0 0 0 0 1

With a leading 1 in the final column, Theorem RCLS tells us the system is inconsistent and therefore by Theorem CSCS, w ∉ C A . ⊠

Theorem CSCS completes a collection of three theorems, and one definition, that deserve comment. Many questions about spans, linear independence, null space, column spaces and similar objects can be converted to questions about systems of equations (homogeneous or not), which we understand well from our previous results, especially those in Chapter SLE. These previous results include theorems like Theorem RCLS which allows us to quickly decide consistency of a system, and Theorem BNS which allows us to describe solution sets for homogeneous systems compactly as the span of a linearly independent set of column vectors.

The table below lists these for definitions and theorems along with a brief reminder of the statement and an example of how the statement is used.

Subsection CSSOC: Column Space Spanned by Original Columns

So we have a foolproof, automated procedure for determining membership in C A . While this works just fine a vector at a time, we would like to have a more useful description of the set C A as a whole. The next example will preview the first of two fundamental results about the column space of a matrix.

Example  CSTW
Column space, two ways
Consider the 5 × 7 matrix A ,

2 4 1 − 1 1 4 4 1 2 1 0 2 4 7 0 0 1 4 1 8 7 1 2 − 1 2 1 9 6 − 2 − 4 1 3 − 1 − 2 − 2

According to the definition ( Definition CSM), the column space of A is

C A = 2 1 0 1 − 2 , 4 2 0 2 − 4 , 1 1 1 − 1 1 , − 1 0 4 2 3 , 1 2 1 1 − 1 , 4 4 8 9 − 2 , 4 7 7 6 − 2

While this is a concise description of an infinite set, we might be able to describe the span with fewer than seven vectors. This is the substance of Theorem BS. So we take these seven vectors and make them the columns of matrix, which is simply the original matrix A again. Now we row-reduce,

2 4 1 − 1 1 4 4 1 2 1 0 2 4 7 0 0 1 4 1 8 7 1 2 − 1 2 1 9 6 − 2 − 4 1 3 − 1 − 2 − 2 →  RREF 1 2 0 0 0 3 1 0 0 1  0 0 − 1 0 0 0 0 1 0 2 1 0 0 0 0 1  1 3 0 0 0 0 0 0 0

The pivot columns are D = 1 , 3 , 4 , 5 , so we can create the set

T = 2 1 0 1 − 2 , 1 1 1 − 1 1 , − 1 0 4 2 3 , 1 2 1 1 − 1

and know that C A = T and T is a linearly independent set of columns from the set of columns of A . ⊠

We will now formalize the previous example, which will make it trivial to determine a linearly independent set of vectors that will span the column space of a matrix, and is constituted of just columns of A .

Theorem  BCS
Basis of the Column Space
Suppose that A is an m × n matrix with columns A 1 , A 2 , A 3 , … , A n , and B is a row-equivalent matrix in reduced row-echelon form with r nonzero rows. Let D = { d 1 , d 2 , d 3 , … , d r } be the set of column indices where B has leading 1’s. Let T = A d 1 , A d 2 , A d 3 , … , A d r . Then

□

Proof   Definition CSM describes the column space as the span of the set of columns of A . Theorem BS tells us that we can reduce the set of vectors used in a span. If we apply Theorem BS to C A , we would collect the columns of A into a matrix (which would just be A again) and bring the matrix to reduced row-echelon form, which is the matrix B in the statement of the theorem. In this case, the conclusions of Theorem BS applied to A , B and C A are exactly the conclusions we desire. ■

This is a nice result since it gives us a handful of vectors that describe the entire column space (through the span), and we believe this set is as small as possible because we cannot create any more relations of linear dependence to trim it down further. Furthermore, we defined the column space ( Definition CSM) as all linear combinations of the columns of the matrix, and the elements of the set S are still columns of the matrix (we won’t be so lucky in the next two constructions of the column space).

Procedurally this theorem is extremely easy to apply. Row-reduce the original matrix, identify r columns with leading 1’s in this reduced matrix, and grab the corresponding columns of the original matrix. But it is still important to study the proof of Theorem BS and its motivation in Example COV which lie at the root of this theorem. We’ll trot through an example all the same.

Example  CSOCD
Column space, original columns, Archetype D
Let’s determine a compact expression for the entire column space of the coefficient matrix of the system of equations that is Archetype D. Notice that in Example CSMCS we were only determining if individual vectors were in the column space or not, now we are describing the entire column space.

To start with the application of Theorem BCS, call the coefficient matrix A

A = 2 1 7 − 7 − 3 4 − 5 − 6 1 1 4 − 5 .

and row-reduce it to reduced row-echelon form,

B = 1 0 3 − 2 0 1  1 − 3 0 0 0 0 .

There are leading 1’s in columns 1 and 2, so D = { 1 , 2 } . To construct a set that spans C A , just grab the columns of A indicated by the set D , so

C A = 2 − 3 1 , 1 4 1 .

That’s it.

In Example CSMCS we determined that the vector

c = 2 3 2

was not in the column space of A . Try to write c as a linear combination of the first two columns of A . What happens?

Also in Example CSMCS we determined that the vector

b = 8 − 1 2 4

was in the column space of A . Try to write b as a linear combination of the first two columns of A . What happens? Did you find a unique solution to this question? Hmmmm. ⊠

Subsection CSNM: Column Space of a Nonsingular Matrix

Let’s specialize to square matrices and contrast the column spaces of the coefficient matrices in Archetype A and Archetype B.

Example  CSAA
Column space of Archetype A
The coefficient matrix in Archetype A is

A = 1 − 1 2 2 1 1 1 1 0

which row-reduces to

1 0 1 0 1  − 1 0 0 0 .

Columns 1 and 2 have leading 1’s, so by Theorem BCS we can write

C A = A 1 , A 2 = 1 2 1 , − 1 1 1 .

We want to show in this example that C A ≠ ℂ 3 . So take, for example, the vector b = 1 3 2 . Thenthere is no solution to the system ℒ S A , b , or equivalently, it is not possible to write b as a linear combination of A 1 and A 2 . Try one of these two computations yourself. (Or try both!). Since b ∉ C A , the column space of A cannot be all of ℂ 3 . So by varying the vector of constants, it is possible to create inconsistent systems of equations with this coefficient matrix (the vector b being one such example).

In Example MWIAA we wished to show that the coefficient matrix from Archetype A was not invertible as a first example of a matrix without an inverse. Our device there was to find an inconsistent linear system with A as the coefficient matrix. The vector of constants in that example was b , deliberately chosen outside the column space of A . ⊠

Example  CSAB
Column space of Archetype B
The coefficient matrix in Archetype B, call it B here, is known to be nonsingular (see Example NM). By Theorem NMUS, the linear system ℒ S B , b has a (unique) solution for every choice of b . Theorem CSCS then says that b ∈ C B for all b ∈ ℂ 3 . Stated differently, there is no way to build an inconsistent system with the coefficient matrix B , but then we knew that already from Theorem NMUS. ⊠

Example CSAA and Example CSAB together motivate the following equivalence, which says that nonsingular matrices have column spaces that are as big as possible.

Theorem  CSNM
Column Space of a Nonsingular Matrix
Suppose A is a square matrix of size n . Then A is nonsingular if and only if C A = ℂ n . □

Proof   ( ⇒ ) Suppose A is nonsingular. We wish to establish the set equality C A = ℂ n . By Definition CSM, C A ⊆ ℂ n .

To show that ℂ n ⊆ C A choose b ∈ ℂ n . By Theorem NMUS, we know the linear system ℒ S A , b has a (unique) solution and therefore is consistent. Theorem CSCS then says that b ∈ C A . So by Definition SE, C A = ℂ n .

( ⇐ ) If e i is column i of the n × n identity matrix ( Definition SUV) and by hypothesis C A = ℂ n , then e i ∈ C A for 1 ≤ i ≤ n . By Theorem CSCS, the system ℒ S A , e i is consistent for 1 ≤ i ≤ n . Let b i denote any one particular solution to ℒ S A , e i , 1 ≤ i ≤ n .

Define the n × n matrix B = b 1 | b 2 | b 3 | … | b n . Then

A B = A b 1 | b 2 | b 3 | … | b n = [ A b 1 | A b 2 | A b 3 | … | A b n ]   Definition MM = e 1 | e 2 | e 3 | … | e n = I n   Definition SUV

So the matrix B is a “right-inverse” for A . By Theorem NMRRI, I n is a nonsingular matrix, so by Theorem NPNT both A and B are nonsingular. Thus, in particular, A is nonsingular. ( Travis Osborne contributed to this proof.) ■

With this equivalence for nonsingular matrices we can update our list, Theorem NME3.

Theorem  NME4
Nonsingular Matrix Equivalences, Round 4
Suppose that A is a square matrix of size n . The following are equivalent.

□

Proof   Since Theorem CSNM is an equivalence, we can add it to the list in Theorem NME3. ■

Subsection RSM: Row Space of a Matrix

The rows of a matrix can be viewed as vectors, since they are just lists of numbers, arranged horizontally. So we will transpose a matrix, turning rows into columns, so we can then manipulate rows as column vectors. As a result we will be able to make some new connections between row operations and solutions to systems of equations. OK, here is the second primary definition of this section.

Definition  RSM
Row Space of a Matrix
Suppose A is an m × n matrix. Then the row space of A , ℛ A , is the column space of A t , i.e.  ℛ A = C A t .

(This definition contains Notation RSM.) △

Informally, the row space is the set of all linear combinations of the rows of A . However, we write the rows as column vectors, thus the necessity of using the transpose to make the rows into columns. Additionally, with the row space defined in terms of the column space, all of the previous results of this section can be applied to row spaces.

Notice that if A is a rectangular m × n matrix, then C A ⊆ ℂ m , while ℛ A ⊆ ℂ n and the two sets are not comparable since they do not even hold objects of the same type. However, when A is square of size n , both C A and ℛ A are subsets of ℂ n , though usually the sets will not be equal (but see Exercise CRS.M20).

Example  RSAI
Row space of Archetype I
The coefficient matrix in Archetype I is

I = 1 4 0 − 1 0 7 − 9 2 8 − 1 3 9 − 1 3 7 0 0 2 − 3 − 4 1 2 − 8 − 1 − 4 2 4 8 − 3 1 3 7 .

To build the row space, we transpose the matrix,

I t = 1 2 0 − 1 4 8 0 − 4 0 − 1 2 2 − 1 3 − 3 4 0 9 − 4 8 7 − 1 3 1 2 − 3 1 − 9 7 − 8 3 7

Then the columns of this matrix are used in a span to build the row space,

ℛ I = C I t = 1 4 0 − 1 0 7 − 9 , 2 8 − 1 3 9 − 1 3 7 , 0 0 2 − 3 − 4 1 2 − 8 , − 1 − 4 2 4 8 − 3 1 3 7 .

However, we can use Theorem BCS to get a slightly better description. First, row-reduce I t ,

1 0 0 − 3 1 7 0 1 0 1 2 7 0 0 1 1 3 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .

Since there are leading 1’s in columns with indices D = 1 , 2 , 3 , the column space of I t can be spanned by just the first three columns of I t ,

ℛ I = C I t = 1 4 0 − 1 0 7 − 9 , 2 8 − 1 3 9 − 1 3 7 , 0 0 2 − 3 − 4 1 2 − 8 .

⊠

The row space would not be too interesting if it was simply the column space of the transpose. However, when we do row operations on a matrix we have no effect on the many linear combinations that can be formed with the rows of the matrix. This is stated more carefully in the following theorem.

Theorem  REMRS
Row-Equivalent Matrices have equal Row Spaces
Suppose A and B are row-equivalent matrices. Then ℛ A = ℛ B . □

Proof   Two matrices are row-equivalent ( Definition REM) if one can be obtained from another by a sequence of (possibly many) row operations. We will prove the theorem for two matrices that differ by a single row operation, and then this result can be applied repeatedly to get the full statement of the theorem. The row spaces of A and B are spans of the columns of their transposes. For each row operation we perform on a matrix, we can define an analogous operation on the columns. Perhaps we should call these column operations. Instead, we will still call them row operations, but we will apply them to the columns of the transposes.

Refer to the columns of A t and B t as A i and B i , 1 ≤ i ≤ m . The row operation that switches rows will just switch columns of the transposed matrices. This will have no effect on the possible linear combinations formed by the columns.

Suppose that B t is formed from A t by multiplying column A t by α ≠ 0 . In other words, B t = α A t , and B i = A i for all i ≠ t . We need to establish that two sets are equal, C A t = C B t . We will take a generic element of one and show that it is contained in the other.

β 1 B 1 + β 2 B 2 + β 3 B 3 + ⋯ + β t B t + ⋯ + β m B m = β 1 A 1 + β 2 A 2 + β 3 A 3 + ⋯ + β t α A t + ⋯ + β m A m = β 1 A 1 + β 2 A 2 + β 3 A 3 + ⋯ + α β t A t + ⋯ + β m A m

says that C B t ⊆ C A t . Similarly,

γ 1 A 1 + γ 2 A 2 + γ 3 A 3 + ⋯ + γ t A t + ⋯ + γ m A m = γ 1 A 1 + γ 2 A 2 + γ 3 A 3 + ⋯ + γ t α α A t + ⋯ + γ m A m = γ 1 A 1 + γ 2 A 2 + γ 3 A 3 + ⋯ + γ t α α A t + ⋯ + γ m A m = γ 1 B 1 + γ 2 B 2 + γ 3 B 3 + ⋯ + γ t α B t + ⋯ + γ m B m

says that C A t ⊆ C B t . So ℛ A = C A t = C B t = ℛ B when a single row operation of the second type is performed.

Suppose now that B t is formed from A t by replacing A t with α A s + A t for some α ∈ ℂ and s ≠ t . In other words, B t = α A s + A t , and B i = A i for i ≠ t .

β 1 B 1 + β 2 B 2 + β 3 B 3 + ⋯ + β s B s + ⋯ + β t B t + ⋯ + β m B m = β 1 A 1 + β 2 A 2 + β 3 A 3 + ⋯ + β s A s + ⋯ + β t α A s + A t + ⋯ + β m A m = β 1 A 1 + β 2 A 2 + β 3 A 3 + ⋯ + β s A s + ⋯ + β t α A s + β t A t + ⋯ + β m A m = β 1 A 1 + β 2 A 2 + β 3 A 3 + ⋯ + β s A s + β t α A s + ⋯ + β t A t + ⋯ + β m A m = β 1 A 1 + β 2 A 2 + β 3 A 3 + ⋯ + β s + β t α A s + ⋯ + β t A t + ⋯ + β m A m

says that C B t ⊆ C A t . Similarly,

γ 1 A 1 + γ 2 A 2 + γ 3 A 3 + ⋯ + γ s A s + ⋯ + γ t A t + ⋯ + γ m A m = γ 1 A 1 + γ 2 A 2 + γ 3 A 3 + ⋯ + γ s A s + ⋯ + − α γ t A s + α γ t A s + γ t A t + ⋯ + γ m A m = γ 1 A 1 + γ 2 A 2 + γ 3 A 3 + ⋯ + − α γ t A s + γ s A s + ⋯ + α γ t A s + γ t A t + ⋯ + γ m A m = γ 1 A 1 + γ 2 A 2 + γ 3 A 3 + ⋯ + − α γ t + γ s A s + ⋯ + γ t α A s + A t + ⋯ + γ m A m = γ 1 B 1 + γ 2 B 2 + γ 3 B 3 + ⋯ + − α γ t + γ s B s + ⋯ + γ t B t + ⋯ + γ m B m

says that C A t ⊆ C B t . So ℛ A = C A t = C B t = ℛ B when a single row operation of the third type is performed.

So the row space of a matrix is preserved by each row operation, and hence row spaces of row-equivalent matrices are equal sets. ■

Example  RSREM
Row spaces of two row-equivalent matrices
In Example TREM we saw that the matrices

A = 2 − 1 3 4 5 2 − 2 3 1 1 0 6 B = 1 1 0 6 3 0 − 2 − 9 2 − 1 3 4

are row-equivalent by demonstrating a sequence of two row operations that converted A into B . Applying Theorem REMRS we can say

ℛ A = 2 − 1 3 4 , 5 2 − 2 3 , 1 1 0 6 = 1 1 0 6 , 3 0 − 2 − 9 , 2 − 1 3 4 = ℛ B

⊠

Theorem REMRS is at its best when one of the row-equivalent matrices is in reduced row-echelon form. The vectors that correspond to the zero rows can be ignored. (Who needs the zero vector when building a span? See Exercise LI.T10.) The echelon pattern insures that the nonzero rows yield vectors that are linearly independent. Here’s the theorem.

Theorem  BRS
Basis for the Row Space
Suppose that A is a matrix and B is a row-equivalent matrix in reduced row-echelon form. Let S be the set of nonzero columns of B t . Then

□

Proof   From Theorem REMRS we know that ℛ A = ℛ B . If B has any zero rows, these correspond to columns of B t that are the zero vector. We can safely toss out the zero vector in the span construction, since it can be recreated from the nonzero vectors by a linear combination where all the scalars are zero. So ℛ A = S .

Suppose B has r nonzero rows and let D = d 1 , d 2 , d 3 , … , d r denote the column indices of B that have a leading one in them. Denote the r column vectors of B t , the vectors in S , as B 1 , B 2 , B 3 , … , B r . To show that S is linearly independent, start with a relation of linear dependence

α 1 B 1 + α 2 B 2 + α 3 B 3 + ⋯ + α r B r = 0

Now consider this vector equality in location d i . Since B is in reduced row-echelon form, the entries of column d i of B are all zero, except for a (leading) 1 in row i . Thus, in B t , row d i is all zeros, excepting a 1 in column i . So, for 1 ≤ i ≤ r ,

0 = 0 d i   Definition ZCV = α 1 B 1 + α 2 B 2 + α 3 B 3 + ⋯ + α r B r d i   Definition RLDCV = α 1 B 1 d i + α 2 B 2 d i + α 3 B 3 d i + ⋯ + α r B r d i +   Definition MA = α 1 B 1 d i + α 2 B 2 d i + α 3 B 3 d i + ⋯ + α r B r d i +   Definition MSM = α 1 ( 0 ) + α 2 ( 0 ) + α 3 ( 0 ) + ⋯ + α i ( 1 ) + ⋯ + α r ( 0 )   Definition RREF = α i

So we conclude that α i = 0 for all 1 ≤ i ≤ r , establishing the linear independence of S (Definition LICV). ■

Example  IAS
Improving a span
Suppose in the course of analyzing a matrix (its column space, its null space, its…) we encounter the following set of vectors, described by a span

X = 1 2 1 6 6 , 3 − 1 2 − 1 6 , 1 − 1 0 − 1 − 2 , − 3 2 − 3 6 − 1 0

Let A be the matrix whose rows are the vectors in X , so by design X = ℛ A ,

A = 1 2 1 6 6 3 − 1 2 − 1 6 1 − 1 0 − 1 − 2 − 3 2 − 3 6 − 1 0

Row-reduce A to form a row-equivalent matrix in reduced row-echelon form,

B = 1 0 0 2 − 1 0 1  0 3 1 0 0 1 − 2 5 0 0 0 0 0

Then Theorem BRS says we can grab the nonzero columns of B t and write

X = ℛ A = ℛ B = 1 0 0 2 − 1 , 0 1 0 3 1 , 0 0 1 − 2 5

These three vectors provide a much-improved description of X . There are fewer vectors, and the pattern of zeros and ones in the first three entries makes it easier to determine membership in X . And all we had to do was row-reduce the right matrix and toss out a zero row. Next to row operations themselves, this is probably the most powerful computational technique at your disposal as it quickly provides a much improved description of a span, any span. ⊠

Theorem BRS and the techniques of Example IAS will provide yet another description of the column space of a matrix. First we state a triviality as a theorem, so we can reference it later.

Theorem  CSRST
Column Space, Row Space, Transpose
Suppose A is a matrix. Then C A = ℛ A t . □

Proof  

C A = C A t t   Theorem TT = ℛ A t   Definition RSM

■

So to find another expression for the column space of a matrix, build its transpose, row-reduce it, toss out the zero rows, and convert the nonzero rows to column vectors to yield an improved set for the span construction. We’ll do Archetype I, then you do Archetype J.

Example  CSROI
Column space from row operations, Archetype I
To find the column space of the coefficient matrix of Archetype I, we proceed as follows. The matrix is

I = 1 4 0 − 1 0 7 − 9 2 8 − 1 3 9 − 1 3 7 0 0 2 − 3 − 4 1 2 − 8 − 1 − 4 2 4 8 − 3 1 3 7 .

The transpose is

1 2 0 − 1 4 8 0 − 4 0 − 1 2 2 − 1 3 − 3 4 0 9 − 4 8 7 − 1 3 1 2 − 3 1 − 9 7 − 8 3 7 .

Row-reduced this becomes,

1 0 0 − 3 1 7 0 1 0 1 2 7 0 0 1 1 3 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .

Now, using Theorem CSRST and Theorem BRS

C I = ℛ I t = 1 0 0 − 3 1 7 , 0 1 0 1 2 7 , 0 0 1 1 3 7 .

This is a very nice description of the column space. Fewer vectors than the 7 involved in the definition, and the pattern of the zeros and ones in the first 3 slots can be used to advantage. For example, Archetype I is presented as a consistent system of equations with a vector of constants

b = 3 9 1 4 .

Since ℒ S I , b is consistent, Theorem CSCS tells us that b ∈ C I . But we could see this quickly with the following computation, which really only involves any work in the 4th entry of the vectors as the scalars in the linear combination are dictated by the first three entries of b .

b = 3 9 1 4 = 3 1 0 0 − 3 1 7 + 9 0 1 0 1 2 7 + 1 0 0 1 1 3 7

Can you now rapidly construct several vectors, b , so that ℒ S I , b is consistent, and several more so that the system is inconsistent? ⊠

Subsection READ: Reading Questions

Subsection EXC: Exercises

C20 For parts (a), (b) and c , find a set of linearly independent vectors X so that C A = X , and a set of linearly independent vectors Y so that ℛ A = Y .

Contributed by  Chris Black

C30 Example CSOCD expresses the column space of the coefficient matrix from Archetype D (call the matrix A here) as the span of the first two columns of A . In Example CSMCS we determined that the vector

c = 2 3 2

was not in the column space of A and that the vector

b = 8 − 1 2 4

was in the column space of A . Attempt to write c and b as linear combinations of the two vectors in the span construction for the column space in Example CSOCD and record your observations.  
Contributed by  Robert Beezer Solution [ 769]

C31 For the matrix A below find a set of vectors T meeting the following requirements: (1) the span of T is the column space of A , that is, T = C A , (2) T is linearly independent, and (3) the elements of T are columns of A .

A = 2 1 4 − 1 2 1 − 1 5 1 1 − 1 2 − 7 0 1 2 − 1 8 − 1 2

Contributed by  Robert Beezer Solution [ 769]

C32 In Example CSAA, verify that the vector b is not in the column space of the coefficient matrix.  
Contributed by  Robert Beezer

C33 Find a linearly independent set S so that the span of S , S , is row space of the matrix B , and S is linearly independent.

B = 2 3 1 1 1 1 0 1 − 1 2 3 − 4

Contributed by  Robert Beezer Solution [ 770]

C34 For the 3 × 4 matrix A and the column vector y ∈ ℂ 4 given below, determine if y is in the row space of A . In other words, answer the question: y ∈ ℛ A ?

A = − 2 6 7 − 1 7 − 3 0 − 3 8 0 7 6 y = 2 1 3 − 2

Contributed by  Robert Beezer Solution [ 771]

C35 For the matrix A below, find two different linearly independent sets whose spans equal the column space of A , C A , such that
(a) the elements are each columns of A .
(b) the set is obtained by a procedure that is substantially different from the procedure you use in part (a).

A = 3 5 1 − 2 1 2 3 3 − 3 − 4 7 1 3

Contributed by  Robert Beezer Solution [ 772]

C40 The following archetypes are systems of equations. For each system, write the vector of constants as a linear combination of the vectors in the span construction for the column space provided by Theorem BCS (these vectors are listed for each of these archetypes).
Archetype A
Archetype B
Archetype C
Archetype D
Archetype E
Archetype F
Archetype G
Archetype H
Archetype I
Archetype J

Contributed by  Robert Beezer

C42 The following archetypes are either matrices or systems of equations with coefficient matrices. For each matrix, compute a set of column vectors such that (1) the vectors are columns of the matrix, (2) the set is linearly independent, and (3) the span of the set is the column space of the matrix. See Theorem BCS.
Archetype A
Archetype B
Archetype C
Archetype D/ Archetype E
Archetype F
Archetype G/ Archetype H
Archetype I
Archetype J
Archetype K
Archetype L

Contributed by  Robert Beezer

C50 The following archetypes are either matrices or systems of equations with coefficient matrices. For each matrix, compute a set of column vectors such that (1) the set is linearly independent, and (2) the span of the set is the row space of the matrix. See Theorem BRS.
Archetype A
Archetype B
Archetype C
Archetype D/ Archetype E
Archetype F
Archetype G/ Archetype H
Archetype I
Archetype J
Archetype K
Archetype L

Contributed by  Robert Beezer

C51 The following archetypes are either matrices or systems of equations with coefficient matrices. For each matrix, compute the column space as the span of a linearly independent set as follows: transpose the matrix, row-reduce, toss out zero rows, convert rows into column vectors. See Example CSROI.
Archetype A
Archetype B
Archetype C
Archetype D/ Archetype E
Archetype F
Archetype G/ Archetype H
Archetype I
Archetype J
Archetype K
Archetype L

Contributed by  Robert Beezer

C52 The following archetypes are systems of equations. For each different coefficient matrix build two new vectors of constants. The first should lead to a consistent system and the second should lead to an inconsistent system. Descriptions of the column space as spans of linearly independent sets of vectors with “nice patterns” of zeros and ones might be most useful and instructive in connection with this exercise. (See the end of Example CSROI.)
Archetype A
Archetype B
Archetype C
Archetype D/ Archetype E
Archetype F
Archetype G/ Archetype H
Archetype I
Archetype J

Contributed by  Robert Beezer

M10 For the matrix E below, find vectors b and c so that the system ℒ S E , b is consistent and ℒ S E , c is inconsistent.

E = − 2 1 1 0 3 − 1 0 2 4 1 1 6

Contributed by  Robert Beezer Solution [ 774]

M20 Usually the column space and null space of a matrix contain vectors of different sizes. For a square matrix, though, the vectors in these two sets are the same size. Usually the two sets will be different. Construct an example of a square matrix where the column space and null space are equal.  
Contributed by  Robert Beezer Solution [ 775]

M21 We have a variety of theorems about how to create column spaces and row spaces and they frequently involve row-reducing a matrix. Here is a procedure that some try to use to get a column space. Begin with an m × n matrix A and row-reduce to a matrix B with columns B 1 , B 2 , B 3 , … , B n . Then form the column space of A as

C A = B 1 , B 2 , B 3 , … , B n = C B

This is not not a legitimate procedure, and therefore is not a theorem. Construct an example to show that the procedure will not in general create the column space of A .  
Contributed by  Robert Beezer Solution [ 775]

T40 Suppose that A is an m × n matrix and B is an n × p matrix. Prove that the column space of A B is a subset of the column space of A , that is C A B ⊆ C A . Provide an example where the opposite is false, in other words give an example where C A ⊈ C A B . (Compare with Exercise MM.T40.)  
Contributed by  Robert Beezer Solution [ 775]

T41 Suppose that A is an m × n matrix and B is an n × n nonsingular matrix. Prove that the column space of A is equal to the column space of A B , that is C A = C A B . (Compare with Exercise MM.T41 and Exercise CRS.T40.)  
Contributed by  Robert Beezer Solution [ 776]

T45 Suppose that A is an m × n matrix and B is an n × m matrix where A B is a nonsingular matrix. Prove that
(1) N B = 0
(2) C B ∩ N A = 0
Discuss the case when m = n in connection with Theorem NPNT.  
Contributed by  Robert Beezer Solution [ 777]

Subsection SOL: Solutions

C30 Contributed by  Robert Beezer Statement [ 759]
In each case, begin with a vector equation where one side contains a linear combination of the two vectors from the span construction that gives the column space of A with unknowns for scalars, and then use Theorem SLSLC to set up a system of equations. For c , the corresponding system has no solution, as we would expect.

For b there is a solution, as we would expect. What is interesting is that the solution is unique. This is a consequence of the linear independence of the set of two vectors in the span construction. If we wrote b as a linear combination of all four columns of A , then there would be infinitely many ways to do this.

C31 Contributed by  Robert Beezer Statement [ 760]
Theorem BCS is the right tool for this problem. Row-reduce this matrix, identify the pivot columns and then grab the corresponding columns of A for the set T . The matrix A row-reduces to

1 0 3 0 0 0 1  − 2 0 0 0 0 0 1 0 0 0 0 0 1

So D = 1 , 2 , 4 , 5 and then

T = A 1 , A 2 , A 4 , A 5 = 2 1 − 1 2 , 1 − 1 2 − 1 , − 1 1 0 − 1 , 2 1 1 2

has the requested properties.

C33 Contributed by  Robert Beezer Statement [ 761]
Theorem BRS is the most direct route to a set with these properties. Row-reduce, toss zero rows, keep the others. You could also transpose the matrix, then look for the column space by row-reducing the transpose and applying Theorem BCS. We’ll do the former,

B →  RREF 1 0 − 1 2 0 1  1 − 1 0 0 0 0

So the set S is

S = 1 0 − 1 2 , 0 1 1 − 1

C34 Contributed by  Robert Beezer Statement [ 762]

y ∈ ℛ A ⇔ y ∈ C A t   Definition RSM ⇔ ℒ S A t , y   is consistent   Theorem CSCS

The augmented matrix A t y row reduces to

1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1

and with a leading 1 in the final column Theorem RCLS tells us the linear system is inconsistent and so y ∉ ℛ A .

C35 Contributed by  Robert Beezer Statement [ 762]
(a) By Theorem BCS we can row-reduce A , identify pivot columns with the set D , and “keep” those columns of A and we will have a set with the desired properties.

A →  RREF 1 0 − 1 3 − 1 9 0 1  8 1 1 0 0 0 0

So we have the set of pivot columns D = 1 , 2 and we “keep” the first two columns of A ,

3 1 − 3 , 5 2 − 4

(b) We can view the column space as the row space of the transpose ( Theorem CSRST). We can get a basis of the row space of a matrix quickly by bringing the matrix to reduced row-echelon form and keeping the nonzero rows as column vectors ( Theorem BRS). Here goes.

A t →  RREF 1 0 − 2 0 1  3 0 0 0 0 0 0

Taking the nonzero rows and tilting them up as columns gives us

1 0 − 2 , 0 1 3

An approach based on the matrix L from extended echelon form (Definition EEF) and Theorem FS will work as well as an alternative approach.

M10 Contributed by  Robert Beezer Statement [ 765]
Any vector from ℂ 3 will lead to a consistent system, and therefore there is no vector that will lead to an inconsistent system.

How do we convince ourselves of this? First, row-reduce E ,

E →  RREF 1 0 0 1 0 1  0 1 0 0 1 1

If we augment E with any vector of constants, and row-reduce the augmented matrix, we will never find a leading 1 in the final column, so by Theorem RCLS the system will always be consistent.

Said another way, the column space of E is all of ℂ 3 , C E = ℂ 3 . So by Theorem CSCS any vector of constants will create a consistent system (and none will create an inconsistent system).

M20 Contributed by  Robert Beezer Statement [ 766]
The 2 × 2 matrix

1 1 − 1 − 1

has C A = N A = 1 − 1 .

M21 Contributed by  Robert Beezer Statement [ 766]
Begin with a matrix A (of any size) that does not have any zero rows, but which when row-reduced to B yields at least one row of zeros. Such a matrix should be easy to construct (or find, like say from Archetype A).

C A will contain some vectors whose final slot (entry m ) is non-zero, however, every column vector from the matrix B will have a zero in slot m and so every vector in C B will also contain a zero in the final slot. This means that C A ≠ C B , since we have vectors in C A that cannot be elements of C B .

T40 Contributed by  Robert Beezer Statement [ 767]
Choose x ∈ C A B . Then by Theorem CSCS there is a vector w that is a solution to ℒ S A B , x . Define the vector y by y = B w . We’re set,

A y = A B w  Definition of  y = A B w   Theorem MMA = x   w  solution to  ℒ S A B , x

This says that ℒ S A , x is a consistent system, and by Theorem CSCS, we see that x ∈ C A and therefore C A B ⊆ C A .

For an example where C A ⊈ C A B choose A to be any nonzero matrix and choose B to be a zero matrix. Then C A ≠ 0 and C A B = C O = 0 .

T41 Contributed by  Robert Beezer Statement [ 767]
From the solution to Exercise CRS.T40 we know that C A B ⊆ C A . So to establish the set equality ( Definition SE) we need to show that C A ⊆ C A B .

Choose x ∈ C A . By Theorem CSCS the linear system ℒ S A , x is consistent, so let y be one such solution. Because B is nonsingular, and linear system using B as a coefficient matrix will have a solution ( Theorem NMUS). Let w be the unique solution to the linear system ℒ S B , y . All set, here we go,

A B w = A B w   Theorem MMA = A y   w  solution to  ℒ S B , y = x   y  solution to  ℒ S A , x

This says that the linear system ℒ S A B , x is consistent, so by Theorem CSCS, x ∈ C A B . So C A ⊆ C A B .

T45 Contributed by  Robert Beezer Statement [ 767]
First, 0 ∈ N B trivially. Now suppose that x ∈ N B . Then

A B x = A ( B x )   Theorem MMA = A 0 x ∈ N B = 0   Theorem MMZM

Since we have assumed A B is nonsingular, Definition NM implies that x = 0 .

Second, 0 ∈ C B and 0 ∈ N A trivially, and so the zero vector is in the intersection as well ( Definition SI). Now suppose that y ∈ C B ∩ N A . Because y ∈ C B , Theorem CSCS says the system ℒ S B , y is consistent. Let x ∈ ℂ n be one solution to this system. Then

A B x = A ( B x )   Theorem MMA = A y   x  solution to  ℒ S B , y = 0 y ∈ N A

Since we have assumed A B is nonsingular, Definition NM implies that x = 0 . Then y = B x = B 0 = 0 .

When A B is nonsingular and m = n we know that the first condition, N B = 0 , means that B is nonsingular ( Theorem NMTNS). Because B is nonsingular Theorem CSNM implies that C B = ℂ m . In order to have the second condition fulfilled, C B ∩ N A = 0 , we must realize that N A = 0 . However, a second application of Theorem NMTNS shows that A must be nonsingular. This reproduces Theorem NPNT.

Chapter 4. Chapter VS  Vector Spaces

We now have a computational toolkit in place and so we can begin our study of linear algebra in a more theoretical style.

Linear algebra is the study of two fundamental objects, vector spaces and linear transformations (see Chapter LT). This chapter will focus on the former. The power of mathematics is often derived from generalizing many different situations into one abstract formulation, and that is exactly what we will be doing throughout this chapter.

Parent topic: Part C  Core

Previous topic: Section CRS  Column and Row Spaces

Next topic: Chapter D  Determinants

Chapter 5. Chapter D  Determinants

The determinant is a function that takes a square matrix as an input and produces a scalar as an output. So unlike a vector space, it is not an algebraic structure. However, it has many beneficial properties for studying vector spaces, matrices and systems of equations, so it is hard to ignore (though some have tried). While the properties of a determinant can be very useful, they are also complicated to prove.

Section PDM  Properties of Determinants of Matrices

Parent topic: Part C  Core

Previous topic: Chapter VS  Vector Spaces

Next topic: Section PDM  Properties of Determinants of Matrices

Section PDM  Properties of Determinants of Matrices

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

We have seen how to compute the determinant of a matrix, and the incredible fact that we can perform expansion about any row or column to make this computation. In this largely theoretical section, we will state and prove several more intriguing properties about determinants. Our main goal will be the two results in Theorem SMZD and Theorem DRMM, but more specifically, we will see how the value of a determinant will allow us to gain insight into the various properties of a square matrix.

Parent topic: Chapter D  Determinants

Previous topic: Chapter D  Determinants

Next topic: Chapter E  Eigenvalues

Subsection DRO: Determinants and Row Operations

We start easy with a straightforward theorem whose proof presages the style of subsequent proofs in this subsection.

Theorem  DZRC
Determinant with Zero Row or Column
Suppose that A is a square matrix with a row where every entry is zero, or a column where every entry is zero. Then det A = 0 . □

Proof   Suppose that A is a square matrix of size n and row i has every entry equal to zero. We compute det A via expansion about row i .

det A = ∑ j = 1 n ( − 1 ) i + j A i j det A i | j   Theorem DER = ∑ j = 1 n ( − 1 ) i + j 0 det A i | j  Row  i  is zeros = ∑ j = 1 n 0 = 0

The proof for the case of a zero column is entirely similar, or could be derived from an application of Theorem DT employing the transpose of the matrix. ■

Theorem  DRCS
Determinant for Row or Column Swap
Suppose that A is a square matrix. Let B be the square matrix obtained from A by interchanging the location of two rows, or interchanging the location of two columns. Then det B = − det A . □

Proof   Begin with the special case where A is a square matrix of size n and we form B by swapping adjacent rows i and i + 1 for some 1 ≤ i ≤ n − 1 . Notice that the assumption about swapping adjacent rows means that B i + 1 | j = A i | j for all 1 ≤ j ≤ n , and B i + 1 , j = A i j for all 1 ≤ j ≤ n . We compute det B via expansion about row i + 1 .

det B = ∑ j = 1 n ( − 1 ) ( i + 1 ) + j B i + 1 , j det B i + 1 | j   Theorem DER = ∑ j = 1 n ( − 1 ) ( i + 1 ) + j A i j det A i | j  Hypothesis = ∑ j = 1 n ( − 1 ) 1 ( − 1 ) i + j A i j det A i | j = ( − 1 ) ∑ j = 1 n ( − 1 ) i + j A i j det A i | j = − det A   Theorem DER

So the result holds for the special case where we swap adjacent rows of the matrix. As any computer scientist knows, we can accomplish any rearrangement of an ordered list by swapping adjacent elements. This principle can be demonstrated by naïve sorting algorithms such as “bubble sort.” In any event, we don’t need to discuss every possible reordering, we just need to consider a swap of two rows, say rows s and t with 1 ≤ s < t ≤ n .

Begin with row s , and repeatedly swap it with each row just below it, including row t and stopping there. This will total t − s swaps. Now swap the former row t , which currently lives in row t − 1 , with each row above it, stopping when it becomes row s . This will total another t − s − 1 swaps. In this way, we create B through a sequence of 2 ( t − s ) − 1 swaps of adjacent rows, each of which adjusts det A by a multiplicative factor of − 1 . So

det B = ( − 1 ) 2 ( t − s ) − 1 det A = ( − 1 ) 2 t − s ( − 1 ) − 1 det A = − det A

as desired.

The proof for the case of swapping two columns is entirely similar, or could be derived from an application of Theorem DT employing the transpose of the matrix. ■

So Theorem DRCS tells us the effect of the first row operation ( Definition RO) on the determinant of a matrix. Here’s the effect of the second row operation.

Theorem  DRCM
Determinant for Row or Column Multiples
Suppose that A is a square matrix. Let B be the square matrix obtained from A by multiplying a single row by the scalar α , or by multiplying a single column by the scalar α . Then det B = α det A . □

Proof   Suppose that A is a square matrix of size n and we form the square matrix B by multiplying each entry of row i of A by α . Notice that the other rows of A and B are equal, so A i | j = B i | j , for all 1 ≤ j ≤ n . We compute det B via expansion about row i .

det B = ∑ j = 1 n ( − 1 ) i + j B i j det B i | j   Theorem DER = ∑ j = 1 n ( − 1 ) i + j B i j det A i | j  Hypothesis = ∑ j = 1 n ( − 1 ) i + j α A i j det A i | j  Hypothesis = α ∑ j = 1 n ( − 1 ) i + j A i j det A i | j = α det A   Theorem DER

The proof for the case of a multiple of a column is entirely similar, or could be derived from an application of Theorem DT employing the transpose of the matrix. ■

Let’s go for understanding the effect of all three row operations. But first we need an intermediate result, but it is an easy one.

Theorem  DERC
Determinant with Equal Rows or Columns
Suppose that A is a square matrix with two equal rows, or two equal columns. Then det A = 0 . □

Proof   Suppose that A is a square matrix of size n where the two rows s and t are equal. Form the matrix B by swapping rows s and t . Notice that as a consequence of our hypothesis, A = B . Then

det A = 1 2 det A + det A = 1 2 det A − det B   Theorem DRCS = 1 2 det A − det A  Hypothesis,  A = B = 1 2 ( 0 ) = 0

The proof for the case of two equal columns is entirely similar, or could be derived from an application of Theorem DT employing the transpose of the matrix. ■

Now explain the third row operation. Here we go.

Theorem  DRCMA
Determinant for Row or Column Multiples and Addition
Suppose that A is a square matrix. Let B be the square matrix obtained from A by multiplying a row by the scalar α and then adding it to another row, or by multiplying a column by the scalar α and then adding it to another column. Then det B = det A . □

Proof   Suppose that A is a square matrix of size n . Form the matrix B by multiplying row s by α and adding it to row t . Let C be the auxiliary matrix where we replace row t of A by row s of A . Notice that A t | j = B t | j = C t | j for all 1 ≤ j ≤ n . We compute the determinant of B by expansion about row t .

det B = ∑ j = 1 n ( − 1 ) t + j B t j det B t | j   Theorem DER = ∑ j = 1 n ( − 1 ) t + j α A s j + A t j det B t | j  Hypothesis = ∑ j = 1 n ( − 1 ) t + j α A s j det B t | j + ∑ j = 1 n ( − 1 ) t + j A t j det B t | j = α ∑ j = 1 n ( − 1 ) t + j A s j det B t | j + ∑ j = 1 n ( − 1 ) t + j A t j det B t | j = α ∑ j = 1 n ( − 1 ) t + j C t j det C t | j + ∑ j = 1 n ( − 1 ) t + j A t j det A t | j = α det C + det A   Theorem DER = α 0 + det A = det A   Theorem DERC

The proof for the case of adding a multiple of a column is entirely similar, or could be derived from an application of Theorem DT employing the transpose of the matrix. ■

Is this what you expected? We could argue that the third row operation is the most popular, and yet it has no effect whatsoever on the determinant of a matrix! We can exploit this, along with our understanding of the other two row operations, to provide another approach to computing a determinant. We’ll explain this in the context of an example.

Example  DRO
Determinant by row operations
Suppose we desire the determinant of the 4 × 4 matrix

A = 2 0 2 3 1 3 − 1 1 − 1 1 − 1 2 3 5 4 0

We will perform a sequence of row operations on this matrix, shooting for an upper triangular matrix, whose determinant will be simply the product of its diagonal entries. For each row operation, we will track the effect on the determinant via Theorem DRCS, Theorem DRCM, Theorem DRCMA.

→ R 1 ↔ R 2 A 1 = 1 3 − 1 1 2 0 2 3 − 1 1 − 1 2 3 5 4 0 det A = − det A 1   Theorem DRCS → − 2 R 1 + R 2 A 2 = 1 3 − 1 1 0 − 6 4 1 − 1 1 − 1 2 3 5 4 0 = − det A 2   Theorem DRCMA → 1 R 1 + R 3 A 3 = 1 3 − 1 1 0 − 6 4 1 0 4 − 2 3 3 5 4 0 = − det A 3   Theorem DRCMA → − 3 R 1 + R 4 A 4 = 1 3 − 1 1 0 − 6 4 1 0 4 − 2 3 0 − 4 7 − 3 = − det A 4   Theorem DRCMA → 1 R 3 + R 2 A 5 = 1 3 − 1 1 0 − 2 2 4 0 4 − 2 3 0 − 4 7 − 3 = − det A 5   Theorem DRCMA → − 1 2 R 2 A 6 = 1 3 − 1 1 0 1 − 1 − 2 0 4 − 2 3 0 − 4 7 − 3 = 2 det A 6   Theorem DRCM → − 4 R 2 + R 3 A 7 = 1 3 − 1 1 0 1 − 1 − 2 0 0 2 1 1 0 − 4 7 − 3 = 2 det A 7   Theorem DRCMA → 4 R 2 + R 4 A 8 = 1 3 − 1 1 0 1 − 1 − 2 0 0 2 1 1 0 0 3 − 1 1 = 2 det A 8   Theorem DRCMA → − 1 R 3 + R 4 A 9 = 1 3 − 1 1 0 1 − 1 − 2 0 0 2 1 1 0 0 1 − 2 2 = 2 det A 9   Theorem DRCMA → − 2 R 4 + R 3 A 1 0 = 1 3 − 1 1 0 1 − 1 − 2 0 0 0 5 5 0 0 1 − 2 2 = 2 det A 1 0   Theorem DRCMA → R 3 ↔ R 4 A 1 1 = 1 3 − 1 1 0 1 − 1 − 2 0 0 1 − 2 2 0 0 0 5 5 = − 2 det A 1 1   Theorem DRCS → 1 5 5 R 4 A 1 2 = 1 3 − 1 1 0 1 − 1 − 2 0 0 1 − 2 2 0 0 0 1 = − 1 1 0 det A 1 2   Theorem DRCM

The matrix A 1 2 is upper triangular, so expansion about the first column (repeatedly) will result in det A 1 2 = ( 1 ) ( 1 ) ( 1 ) ( 1 ) = 1 (see Example DUTM) and thus, det A = − 1 1 0 ( 1 ) = − 1 1 0 .

Notice that our sequence of row operations was somewhat ad hoc, such as the transformation to A 5 . We could have been even more methodical, and strictly followed the process that converts a matrix to reduced row-echelon form ( Theorem REMEF), eventually achieving the same numerical result with a final matrix that equaled the 4 × 4 identity matrix. Notice too that we could have stopped with A 8 , since at this point we could compute det A 8 by two expansions about first columns, followed by a simple determinant of a 2 × 2 matrix (Theorem DMST).

The beauty of this approach is that computationally we should already have written a procedure to convert matrices to reduced-row echelon form, so all we need to do is track the multiplicative changes to the determinant as the algorithm proceeds. Further, for a square matrix of size n this approach requires on the order of n 3 multiplications, while a recursive application of expansion about a row or column (Theorem DER, Theorem DEC) will require in the vicinity of ( n − 1 ) ( n ! ) multiplications. So even for very small matrices, a computational approach utilizing row operations will have superior run-time. Tracking, and controlling, the effects of round-off errors is another story, best saved for a numerical linear algebra course. ⊠

Subsection DROEM: Determinants, Row Operations, Elementary Matrices

As a final preparation for our two most important theorems about determinants, we prove a handful of facts about the interplay of row operations and matrix multiplication with elementary matrices with regard to the determinant. But first, a simple, but crucial, fact about the identity matrix.

Theorem  DIM
Determinant of the Identity Matrix
For every n ≥ 1 , det I n = 1 . □

Proof   It may be overkill, but this is a good situation to run through a proof by induction on n ( Technique I). Is the result true when n = 1 ? Yes,

det I 1 = I 1 1 1   Definition DM = 1   Definition IM

Now assume the theorem is true for the identity matrix of size n − 1 and investigate the determinant of the identity matrix of size n with expansion about row 1,

det I n = ∑ j = 1 n ( − 1 ) 1 + j I n 1 j det I n 1 | j   Definition DM = ( − 1 ) 1 + 1 I n 1 1 det I n 1 | 1 + ∑ j = 2 n ( − 1 ) 1 + j I n 1 j det I n 1 | j = 1 det I n − 1 + ∑ j = 2 n ( − 1 ) 1 + j 0 det I n 1 | j   Definition IM = 1 ( 1 ) + ∑ j = 2 n 0 = 1  Induction Hypothesis

■

Theorem  DEM
Determinants of Elementary Matrices
For the three possible versions of an elementary matrix (Definition ELEM) we have the determinants,

□

Proof   Swapping rows i and j of the identity matrix will create E i , j (Definition ELEM), so

det E i , j = − det I n   Theorem DRCS = − 1   Theorem DIM

Multiplying row i of the identity matrix by α will create E i α (Definition ELEM), so

det E i α = α det I n   Theorem DRCM = α ( 1 ) = α   Theorem DIM

Multiplying row i of the identity matrix by α and adding to row j will create E i , j α (Definition ELEM), so

det E i , j α = det I n   Theorem DRCMA = 1   Theorem DIM

■

Theorem  DEMMM
Determinants, Elementary Matrices, Matrix Multiplication
Suppose that A is a square matrix of size n and E is any elementary matrix of size n . Then

det E A = det E det A

□

Proof   The proof procedes in three parts, one for each type of elementary matrix, with each part very similar to the other two. First, let B be the matrix obtained from A by swapping rows i and j ,

det E i , j A = det B   Theorem EMDRO = − det A   Theorem DRCS = det E i , j det A   Theorem DEM

Second, let B be the matrix obtained from A by multiplying row i by α ,

det E i α A = det B   Theorem EMDRO = α det A   Theorem DRCM = det E i α det A   Theorem DEM

Third, let B be the matrix obtained from A by multiplying row i by α and adding to row j ,

det E i , j α A = det B   Theorem EMDRO = det A   Theorem DRCMA = det E i , j α det A   Theorem DEM

Since the desired result holds for each variety of elementary matrix individually, we are done. ■

Subsection DNMMM: Determinants, Nonsingular Matrices, Matrix Multiplication

If you asked someone with substantial experience working with matrices about the value of the determinant, they’d be likely to quote the following theorem as the first thing to come to mind.

Theorem  SMZD
Singular Matrices have Zero Determinants
Let A be a square matrix. Then A is singular if and only if det A = 0 . □

Proof   Rather than jumping into the two halves of the equivalence, we first establish a few items. Let B be the unique square matrix that is row-equivalent to A and in reduced row-echelon form ( Theorem REMEF, Theorem RREFU). For each of the row operations that converts B into A , there is an elementary matrix E i which effects the row operation by matrix multiplication (Theorem EMDRO). Repeated applications of Theorem EMDRO allow us to write

A = E s E s − 1 … E 2 E 1 B

Then

det A = det E s E s − 1 … E 2 E 1 B = det E s det E s − 1 … det E 2 det E 1 det B   Theorem DEMMM

From Theorem DEM we can infer that the determinant of an elementary matrix is never zero (note the ban on α = 0 for E i α in Definition ELEM). So the product on the right is composed of nonzero scalars, with the possible exception of det B . More precisely, we can argue that det A = 0 if and only if det B = 0 . With this established, we can take up the two halves of the equivalence.

( ⇒ ) If A is singular, then by Theorem NMRRI, B cannot be the identity matrix. Because (1) the number of pivot columns is equal to the number of nonzero rows, (2) not every column is a pivot column, and (3) B is square, we see that B must have a zero row. By Theorem DZRC the determinant of B is zero, and by the above, we conclude that the determinant of A is zero.

( ⇐ ) We will prove the contrapositive ( Technique CP). So assume A is nonsingular, then by Theorem NMRRI, B is the identity matrix and Theorem DIM tells us that det B = 1 ≠ 0 . With the argument above, we conclude that the determinant of A is nonzero as well. ■

For the case of 2 × 2 matrices you might compare the application of Theorem SMZD with the combination of the results stated in Theorem DMST and Theorem TTMI.

Example  ZNDAB
Zero and nonzero determinant, Archetypes A and B
The coefficient matrix in Archetype A has a zero determinant (check this!) while the coefficient matrix Archetype B has a nonzero determinant (check this, too). These matrices are singular and nonsingular, respectively. This is exactly what Theorem SMZD says, and continues our list of contrasts between these two archetypes. ⊠

Since Theorem SMZD is an equivalence ( Technique E) we can expand on our growing list of equivalences about nonsingular matrices. The addition of the condition det A ≠ 0 is one of the best motivations for learning about determinants.

Theorem  NME7
Nonsingular Matrix Equivalences, Round 7
Suppose that A is a square matrix of size n . The following are equivalent.

□

Proof   Theorem SMZD says A is singular if and only if det A = 0 . If we negate each of these statements, we arrive at two contrapositives that we can combine as the equivalence, A is nonsingular if and only if det A ≠ 0 . This allows us to add a new statement to the list found in Theorem NME6. ■

Computationally, row-reducing a matrix is the most efficient way to determine if a matrix is nonsingular, though the effect of using division in a computer can lead to round-off errors that confuse small quantities with critical zero quantities. Conceptually, the determinant may seem the most efficient way to determine if a matrix is nonsingular. The definition of a determinant uses just addition, subtraction and multiplication, so division is never a problem. And the final test is easy: is the determinant zero or not? However, the number of operations involved in computing a determinant by the definition very quickly becomes so excessive as to be impractical.

Now for the coup de gr â ce. We will generalize Theorem DEMMM to the case of any two square matrices. You may recall thinking that matrix multiplication was defined in a needlessly complicated manner. For sure, the definition of a determinant seems even stranger. (Though Theorem SMZD might be forcing you to reconsider.) Read the statement of the next theorem and contemplate how nicely matrix multiplication and determinants play with each other.

Theorem  DRMM
Determinant Respects Matrix Multiplication
Suppose that A and B are square matrices of the same size. Then det A B = det A det B . □

Proof   This proof is constructed in two cases. First, suppose that A is singular. Then det A = 0 by Theorem SMZD. By the contrapositive of Theorem NPNT, A B is singular as well. So by a second application of Theorem SMZD, det A B = 0 . Putting it all together

det A B = 0 = 0 det B = det A det B

as desired.

For the second case, suppose that A is nonsingular. By Theorem NMPEM there are elementary matrices E 1 , E 2 , E 3 , … , E s such that A = E 1 E 2 E 3 … E s . Then

det A B = det E 1 E 2 E 3 … E s B = det E 1 det E 2 det E 3 … det E s det B   Theorem DEMMM = det E 1 E 2 E 3 … E s det B   Theorem DEMMM = det A det B

■

It is amazing that matrix multiplication and the determinant interact this way. Might it also be true that det A + B = det A + det B ? (See Exercise PDM.M30.)

Subsection READ: Reading Questions

Subsection EXC: Exercises

C30 Each of the archetypes below is a system of equations with a square coefficient matrix, or is a square matrix itself. Compute the determinant of each matrix, noting how Theorem SMZD indicates when the matrix is singular or nonsingular.
Archetype A
Archetype B
Archetype F
Archetype K
Archetype L

Contributed by  Robert Beezer

M20 Construct a 3 × 3 nonsingular matrix and call it A . Then, for each entry of the matrix, compute the corresponding cofactor, and create a new 3 × 3 matrix full of these cofactors by placing the cofactor of an entry in the same location as the entry it was based on. Once complete, call this matrix C . Compute A C t . Any observations? Repeat with a new matrix, or perhaps with a 4 × 4 matrix.  
Contributed by  Robert Beezer Solution [ 1216]

M30 Construct an example to show that the following statement is not true for all square matrices A and B of the same size: det A + B = det A + det B .  
Contributed by  Robert Beezer

T10 Theorem NPNT says that if the product of square matrices A B is nonsingular, then the individual matrices A and B are nonsingular also. Construct a new proof of this result making use of theorems about determinants of matrices.  
Contributed by  Robert Beezer

T15 Use Theorem DRCM to prove Theorem DZRC as a corollary. (See Technique LC.)  
Contributed by  Robert Beezer

T20 Suppose that A is a square matrix of size n and α ∈ ℂ is a scalar. Prove that det α A = α n det A .  
Contributed by  Robert Beezer

T25 Employ Theorem DT to construct the second half of the proof of Theorem DRCM (the portion about a multiple of a column).  
Contributed by  Robert Beezer

Subsection SOL: Solutions

M20 Contributed by  Robert Beezer Statement [ 1214]
The result of these computations should be a matrix with the value of det A in the diagonal entries and zeros elsewhere. The suggestion of using a nonsingular matrix was partially so that it was obvious that the value of the determinant appears on the diagonal.

This result (which is true in general) provides a method for computing the inverse of a nonsingular matrix. Since A C t = det A I n , we can multiply by the reciprocal of the determinant (which is nonzero!) and the inverse of A (it exists!) to arrive at an expression for the matrix inverse:

A − 1 = 1 det A C t

Annotated Acronyms D: Determinants

Theorem EMDRO
The main purpose of elementary matrices is to provide a more formal foundation for row operations. With this theorem we can convert the notion of “doing a row operation” into the slightly more precise, and tractable, operation of matrix multiplication by an elementary matrix. The other big results in this chapter are made possible by this connection and our previous understanding of the behavior of matrix multiplication (such as results in Section MM).

Theorem DER
We define the determinant by expansion about the first row and then prove you can expand about any row (and with Theorem DEC, about any column). Amazing. If the determinant seems contrived, these results might begin to convince you that maybe something interesting is going on.

Theorem DRMM
Theorem EMDRO connects elementary matrices with matrix multiplication. Now we connect determinants with matrix multiplication. If you thought the definition of matrix multiplication (as exemplified by Theorem EMP) was as outlandish as the definition of the determinant, then no more. They seem to play together quite nicely.

Theorem SMZD
This theorem provides a simple test for nonsingularity, even though it is stated and titled as a theorem about singularity. It’ll be helpful, especially in concert with Theorem DRMM, in establishing upcoming results about nonsingular matrices or creating alternative proofs of earlier results. You might even use this theorem as an indicator of how often a matrix is singular. Create a square matrix at random — what are the odds it is singular? This theorem says the determinant has to be zero, which we might suspect is a rare occurrence. Of course, we have to be a lot more careful about words like “random,” “odds,” and “rare” if we want precise answers to this question.

Chapter 6. Chapter E  Eigenvalues

When we have a square matrix of size n , A , and we multiply it by a vector x from ℂ n to form the matrix-vector product ( Definition MVP), the result is another vector in ℂ n . So we can adopt a functional view of this computation — the act of multiplying by a square matrix is a function that converts one vector ( x ) into another one ( A x ) of the same size. For some vectors, this seemingly complicated computation is really no more complicated than scalar multiplication. The vectors vary according to the choice of A , so the question is to determine, for an individual choice of A , if there are any such vectors, and if so, which ones. It happens in a variety of situations that these vectors (and the scalars that go along with them) are of special interest.

We will be solving polynomial equations in this chapter, which raises the specter of roots that are complex numbers. This distinct possibility is our main reason for entertaining the complex numbers throughout the course. You might be moved to revisit Section CNO and Section O.

Section PEE  Properties of Eigenvalues and Eigenvectors

Parent topic: Part C  Core

Previous topic: Section PDM  Properties of Determinants of Matrices

Next topic: Section PEE  Properties of Eigenvalues and Eigenvectors

Section PEE  Properties of Eigenvalues and Eigenvectors

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

The previous section introduced eigenvalues and eigenvectors, and concentrated on their existence and determination. This section will be more about theorems, and the various properties eigenvalues and eigenvectors enjoy. Like a good 4 × 1 0 0   meter relay, we will lead-off with one of our better theorems and save the very best for the anchor leg.

Theorem  EDELI
Eigenvectors with Distinct Eigenvalues are Linearly Independent
Suppose that A is an n × n square matrix and S = x 1 , x 2 , x 3 , … , x p is a set of eigenvectors with eigenvalues λ 1 , λ 2 , λ 3 , … , λ p such that λ i ≠ λ j whenever i ≠ j . Then S is a linearly independent set. □

Proof   If p = 1 , then the set S = x 1 is linearly independent since eigenvectors are nonzero (Definition EEM), so assume for the remainder that p ≥ 2 .

We will prove this result by contradiction ( Technique CD). Suppose to the contrary that S is a linearly dependent set. Define S i = x 1 , x 2 , x 3 , … , x i and let k be an integer such that S k − 1 = x 1 , x 2 , x 3 , … , x k − 1 is linearly independent and S k = x 1 , x 2 , x 3 , … , x k is linearly dependent. We have to ask if there is even such an integer k ? First, since eigenvectors are nonzero, the set x 1 is linearly independent. Since we are assuming that S = S p is linearly dependent, there must be an integer k , 2 ≤ k ≤ p , where the sets S i transition from linear independence to linear dependence (and stay that way). In other words, x k is the vector with the smallest index that is a linear combination of just vectors with smaller indices.

Since x 1 , x 2 , x 3 , … , x k is linearly dependent there are scalars, a 1 , a 2 , a 3 , … , a k , some non-zero (Definition LI), so that

0 = a 1 x 1 + a 2 x 2 + a 3 x 3 + ⋯ + a k x k

Then,

0 = A − λ k I n 0   Theorem ZVSM = A − λ k I n a 1 x 1 + a 2 x 2 + a 3 x 3 + ⋯ + a k x k   Definition RLD = A − λ k I n a 1 x 1 + A − λ k I n a 2 x 2 + ⋯ + A − λ k I n a k x k   Theorem MMDAA = a 1 A − λ k I n x 1 + a 2 A − λ k I n x 2 + ⋯ + a k A − λ k I n x k   Theorem MMSMM = a 1 A x 1 − λ k I n x 1 + a 2 A x 2 − λ k I n x 2 + ⋯ + a k A x k − λ k I n x k   Theorem MMDAA = a 1 A x 1 − λ k x 1 + a 2 A x 2 − λ k x 2 + ⋯ + a k A x k − λ k x k   Theorem MMIM = a 1 λ 1 x 1 − λ k x 1 + a 2 λ 2 x 2 − λ k x 2 + ⋯ + a k λ k x k − λ k x k   Definition EEM = a 1 λ 1 − λ k x 1 + a 2 λ 2 − λ k x 2 + ⋯ + a k λ k − λ k x k   Theorem MMDAA = a 1 λ 1 − λ k x 1 + a 2 λ 2 − λ k x 2 + ⋯ + a k 0 x k   Property AICN = a 1 λ 1 − λ k x 1 + a 2 λ 2 − λ k x 2 + ⋯ + a k − 1 λ k − 1 − λ k x k − 1 + 0   Theorem ZSSM = a 1 λ 1 − λ k x 1 + a 2 λ 2 − λ k x 2 + ⋯ + a k − 1 λ k − 1 − λ k x k − 1   Property Z

This is a relation of linear dependence on the linearly independent set x 1 , x 2 , x 3 , … , x k − 1 , so the scalars must all be zero. That is, a i λ i − λ k = 0 for 1 ≤ i ≤ k − 1 . However, we have the hypothesis that the eigenvalues are distinct, so λ i ≠ λ k for 1 ≤ i ≤ k − 1 . Thus a i = 0 for 1 ≤ i ≤ k − 1 .

This reduces the original relation of linear dependence on x 1 , x 2 , x 3 , … , x k to the simpler equation a k x k = 0 . By Theorem SMEZV we conclude that a k = 0 or x k = 0 . Eigenvectors are never the zero vector (Definition EEM), so a k = 0 . So all of the scalars a i , 1 ≤ i ≤ k are zero, contradicting their introduction as the scalars creating a nontrivial relation of linear dependence on the set x 1 , x 2 , x 3 , … , x k . With a contradiction in hand, we conclude that S must be linearly independent. ■

There is a simple connection between the eigenvalues of a matrix and whether or not the matrix is nonsingular.

Theorem  SMZE
Singular Matrices have Zero Eigenvalues
Suppose A is a square matrix. Then A is singular if and only if λ = 0 is an eigenvalue of A . □

Proof   We have the following equivalences:

  A  is singular ⇔  there exists  x ≠ 0 ,  A x = 0   Definition NSM ⇔  there exists  x ≠ 0 ,  A x = 0 x   Theorem ZSSM ⇔   λ = 0  is an eigenvalue of  A   Definition EEM

■

With an equivalence about singular matrices we can update our list of equivalences about nonsingular matrices.

Theorem  NME8
Nonsingular Matrix Equivalences, Round 8
Suppose that A is a square matrix of size n . The following are equivalent.

□

Proof   The equivalence of the first and last statements is the contrapositive of Theorem SMZE, so we are able to improve on Theorem NME7. ■

Certain changes to a matrix change its eigenvalues in a predictable way.

Theorem  ESMM
Eigenvalues of a Scalar Multiple of a Matrix
Suppose A is a square matrix and λ is an eigenvalue of A . Then α λ is an eigenvalue of α A . □

Proof   Let x ≠ 0 be one eigenvector of A for λ . Then

α A x = α A x   Theorem MMSMM = α λ x   x  eigenvector of  A = α λ x   Property SMAC

So x ≠ 0 is an eigenvector of α A for the eigenvalue α λ . ■

Unfortunately, there are not parallel theorems about the sum or product of arbitrary matrices. But we can prove a similar result for powers of a matrix.

Theorem  EOMP
Eigenvalues Of Matrix Powers
Suppose A is a square matrix, λ is an eigenvalue of A , and s ≥ 0 is an integer. Then λ s is an eigenvalue of A s . □

Proof   Let x ≠ 0 be one eigenvector of A for λ . Suppose A has size n . Then we proceed by induction on s ( Technique I). First, for s = 0 ,

A s x = A 0 x = I n x = x   Theorem MMIM = 1 x   Property OC = λ 0 x = λ s x

so λ s is an eigenvalue of A s in this special case. If we assume the theorem is true for s , then we find

A s + 1 x = A s A x = A s λ x   x  eigenvector of  A  for  λ = λ A s x   Theorem MMSMM = λ λ s x  Induction hypothesis = λ λ s x   Property SMAC = λ s + 1 x

So x ≠ 0 is an eigenvector of A s + 1 for λ s + 1 , and induction tells us the theorem is true for all s ≥ 0 . ■

While we cannot prove that the sum of two arbitrary matrices behaves in any reasonable way with regard to eigenvalues, we can work with the sum of dissimilar powers of the same matrix. We have already seen two connections between eigenvalues and polynomials, in the proof of Theorem EMHE and the characteristic polynomial (Definition CP). Our next theorem strengthens this connection.

Theorem  EPM
Eigenvalues of the Polynomial of a Matrix
Suppose A is a square matrix and λ is an eigenvalue of A . Let q ( x ) be a polynomial in the variable x . Then q ( λ ) is an eigenvalue of the matrix q ( A ) . □

Proof   Let x ≠ 0 be one eigenvector of A for λ , and write q ( x ) = a 0 + a 1 x + a 2 x 2 + ⋯ + a m x m . Then

q ( A ) x = a 0 A 0 + a 1 A 1 + a 2 A 2 + ⋯ + a m A m x = ( a 0 A 0 ) x + ( a 1 A 1 ) x + ( a 2 A 2 ) x + ⋯ + ( a m A m ) x   Theorem MMDAA = a 0 ( A 0 x ) + a 1 ( A 1 x ) + a 2 ( A 2 x ) + ⋯ + a m ( A m x )   Theorem MMSMM = a 0 ( λ 0 x ) + a 1 ( λ 1 x ) + a 2 ( λ 2 x ) + ⋯ + a m ( λ m x )   Theorem EOMP = ( a 0 λ 0 ) x + ( a 1 λ 1 ) x + ( a 2 λ 2 ) x + ⋯ + ( a m λ m ) x   Property SMAC = a 0 λ 0 + a 1 λ 1 + a 2 λ 2 + ⋯ + a m λ m x   Property DSAC = q ( λ ) x

So x ≠ 0 is an eigenvector of q ( A ) for the eigenvalue q ( λ ) . ■

Example  BDE
Building desired eigenvalues
In Example ESMS4 the 4 × 4 symmetric matrix

C = 1 0 1 1 0 1 1 1 1 1 1 0 1 1 0 1

is shown to have the three eigenvalues λ = 3 , 1 , − 1 . Suppose we wanted a 4 × 4 matrix that has the three eigenvalues λ = 4 , 0 , − 2 . We can employ Theorem EPM by finding a polynomial that converts 3 to 4 , 1 to 0 , and − 1 to − 2 . Such a polynomial is called an interpolating polynomial, and in this example we can use

r ( x ) = 1 4 x 2 + x − 5 4

We will not discuss how to concoct this polynomial, but a text on numerical analysis should provide the details or see Section CF. For now, simply verify that r ( 3 ) = 4 , r ( 1 ) = 0 and r ( − 1 ) = − 2 .

Now compute

r ( C ) = 1 4 C 2 + C − 5 4 I 4 = 1 4 3 2 2 2 2 3 2 2 2 2 3 2 2 2 2 3 + 1 0 1 1 0 1 1 1 1 1 1 0 1 1 0 1 − 5 4 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 = 1 2 1 1 3 3 1 1 3 3 3 3 1 1 3 3 1 1

Theorem EPM tells us that if r ( x ) transforms the eigenvalues in the desired manner, then r ( C ) will have the desired eigenvalues. You can check this by computing the eigenvalues of r ( C ) directly. Furthermore, notice that the multiplicities are the same, and the eigenspaces of C and r ( C ) are identical. ⊠

Inverses and transposes also behave predictably with regard to their eigenvalues.

Theorem  EIM
Eigenvalues of the Inverse of a Matrix
Suppose A is a square nonsingular matrix and λ is an eigenvalue of A . Then 1 λ is an eigenvalue of the matrix A − 1 . □

Proof   Notice that since A is assumed nonsingular, A − 1 exists by Theorem NI, but more importantly, 1 λ does not involve division by zero since Theorem SMZE prohibits this possibility.

Let x ≠ 0 be one eigenvector of A for λ . Suppose A has size n . Then

A − 1 x = A − 1 ( 1 x )   Property OC = A − 1 ( 1 λ λ x )   Property MICN = 1 λ A − 1 ( λ x )   Theorem MMSMM = 1 λ A − 1 ( A x )   Definition EEM = 1 λ ( A − 1 A ) x   Theorem MMA = 1 λ I n x   Definition MI = 1 λ x   Theorem MMIM

So x ≠ 0 is an eigenvector of A − 1 for the eigenvalue 1 λ . ■

The theorems above have a similar style to them, a style you should consider using when confronted with a need to prove a theorem about eigenvalues and eigenvectors. So far we have been able to reserve the characteristic polynomial for strictly computational purposes. However, the next theorem, whose statement resembles the preceding theorems, has an easier proof if we employ the characteristic polynomial and results about determinants.

Theorem  ETM
Eigenvalues of the Transpose of a Matrix
Suppose A is a square matrix and λ is an eigenvalue of A . Then λ is an eigenvalue of the matrix A t . □

Proof   Suppose A has size n . Then

p A x = det A − x I n   Definition CP = det A − x I n t   Theorem DT = det A t − x I n t   Theorem TMA = det A t − x I n t   Theorem TMSM = det A t − x I n   Definition IM = p A t x   Definition CP

So A and A t have the same characteristic polynomial, and by Theorem EMRCP, their eigenvalues are identical and have equal algebraic multiplicities. Notice that what we have proved here is a bit stronger than the stated conclusion in the theorem. ■

If a matrix has only real entries, then the computation of the characteristic polynomial (Definition CP) will result in a polynomial with coefficients that are real numbers. Complex numbers could result as roots of this polynomial, but they are roots of quadratic factors with real coefficients, and as such, come in conjugate pairs. The next theorem proves this, and a bit more, without mentioning the characteristic polynomial.

Theorem  ERMCP
Eigenvalues of Real Matrices come in Conjugate Pairs
Suppose A is a square matrix with real entries and x is an eigenvector of A for the eigenvalue λ . Then x ¯ is an eigenvector of A for the eigenvalue λ ¯ . □

Proof  

A x ¯ = A ¯ x ¯   A  has real entries = A x ¯   Theorem MMCC = λ x ¯   x  eigenvector of  A = λ ¯ x ¯   Theorem CRSM

So x ¯ is an eigenvector of A for the eigenvalue λ ¯ . ■

This phenomenon is amply illustrated in Example CEMS6, where the four complex eigenvalues come in two pairs, and the two basis vectors of the eigenspaces are complex conjugates of each other. Theorem ERMCP can be a time-saver for computing eigenvalues and eigenvectors of real matrices with complex eigenvalues, since the conjugate eigenvalue and eigenspace can be inferred from the theorem rather than computed.

Parent topic: Chapter E  Eigenvalues

Previous topic: Chapter E  Eigenvalues

Next topic: Chapter LT  Linear Transformations

Subsection ME: Multiplicities of Eigenvalues

A polynomial of degree n will have exactly n roots. From this fact about polynomial equations we can say more about the algebraic multiplicities of eigenvalues.

Theorem  DCP
Degree of the Characteristic Polynomial
Suppose that A is a square matrix of size n . Then the characteristic polynomial of A , p A x , has degree n . □

Proof   We will prove a more general result by induction ( Technique I). Then the theorem will be true as a special case. We will carefully state this result as a proposition indexed by m , m ≥ 1 .

P ( m ) : Suppose that A is an m × m matrix whose entries are complex numbers or linear polynomials in the variable x of the form c − x , where c is a complex number. Suppose further that there are exactly k entries that contain x and that no row or column contains more than one such entry. Then, when k = m , det A is a polynomial in x of degree m , with leading coefficient ± 1 , and when k < m , det A is a polynomial in x of degree k or less.

Base Case: Suppose A is a 1 × 1 matrix. Then its determinant is equal to the lone entry (Definition DM). When k = m = 1 , the entry is of the form c − x , a polynomial in x of degree m = 1 with leading coefficient − 1 . When k < m , then k = 0 and the entry is simply a complex number, a polynomial of degree 0 ≤ k . So P ( 1 ) is true.

Induction Step: Assume P ( m ) is true, and that A is an ( m + 1 ) × ( m + 1 ) matrix with k entries of the form c − x . There are two cases to consider.

Suppose k = m + 1 . Then every row and every column will contain an entry of the form c − x . Suppose that for the first row, this entry is in column t . Compute the determinant of A by an expansion about this first row (Definition DM). The term associated with entry t of this row will be of the form

( c − x ) ( − 1 ) 1 + t det A 1 | t

The submatrix A 1 | t is an m × m matrix with k = m terms of the form c − x , no more than one per row or column. By the induction hypothesis, det A 1 | t will be a polynomial in x of degree m with coefficient ± 1 . So this entire term is then a polynomial of degree m + 1 with leading coefficient ± 1 .

The remaining terms (which constitute the sum that is the determinant of A ) are products of complex numbers from the first row with cofactors built from submatrices that lack the first row of A and lack some column of A , other than column t . As such, these submatrices are m × m matrices with k = m − 1 < m entries of the form c − x , no more than one per row or column. Applying the induction hypothesis, we see that these terms are polynomials in x of degree m − 1 or less. Adding the single term from the entry in column t with all these others, we see that det A is a polynomial in x of degree m + 1 and leading coefficient ± 1 .

The second case occurs when k < m + 1 . Now there is a row of A that does not contain an entry of the form c − x . We consider the determinant of A by expanding about this row (Theorem DER), whose entries are all complex numbers. The cofactors employed are built from submatrices that are m × m matrices with either k or k − 1 entries of the form c − x , no more than one per row or column. In either case, k ≤ m , and we can apply the induction hypothesis to see that the determinants computed for the cofactors are all polynomials of degree k or less. Summing these contributions to the determinant of A yields a polynomial in x of degree k or less, as desired.

Definition CP tells us that the characteristic polynomial of an n × n matrix is the determinant of a matrix having exactly n entries of the form c − x , no more than one per row or column. As such we can apply P ( n ) to see that the characteristic polynomial has degree n . ■

Theorem  NEM
Number of Eigenvalues of a Matrix
Suppose that A is a square matrix of size n with distinct eigenvalues λ 1 , λ 2 , λ 3 , … , λ k . Then

∑ i = 1 k α A λ i = n

□

Proof   By the definition of the algebraic multiplicity (Definition AME), we can factor the characteristic polynomial as

p A x = c ( x − λ 1 ) α A λ 1 ( x − λ 2 ) α A λ 2 ( x − λ 3 ) α A λ 3 ⋯ ( x − λ k ) α A λ k

where c is a nonzero constant. (We could prove that c = ( − 1 ) n , but we do not need that specificity right now. See Exercise PEE.T30) The left-hand side is a polynomial of degree n by Theorem DCP and the right-hand side is a polynomial of degree ∑ i = 1 k α A λ i . So the equality of the polynomials’ degrees gives the equality ∑ i = 1 k α A λ i = n . ■

Theorem  ME
Multiplicities of an Eigenvalue
Suppose that A is a square matrix of size n and λ is an eigenvalue. Then

1 ≤ γ A λ ≤ α A λ ≤ n

□

Proof   Since λ is an eigenvalue of A , there is an eigenvector of A for λ , x . Then x ∈ ℰ A λ , so γ A λ ≥ 1 , since we can extend x into a basis of ℰ A λ (Theorem ELIS).

To show that γ A λ ≤ α A λ is the most involved portion of this proof. To this end, let g = γ A λ and let x 1 , x 2 , x 3 , … , x g be a basis for the eigenspace of λ , ℰ A λ . Construct another n − g vectors, y 1 , y 2 , y 3 , … , y n − g , so that

x 1 , x 2 , x 3 , … , x g , y 1 , y 2 , y 3 , … , y n − g

is a basis of ℂ n . This can be done by repeated applications of Theorem ELIS. Finally, define a matrix S by

S = [ x 1 | x 2 | x 3 | … | x g | y 1 | y 2 | y 3 | … | y n − g ] = [ x 1 | x 2 | x 3 | … | x g | R ]

where R is an n × ( n − g ) matrix whose columns are y 1 , y 2 , y 3 , … , y n − g . The columns of S are linearly independent by design, so S is nonsingular (Theorem NMLIC) and therefore invertible ( Theorem NI). Then,

e 1 | e 2 | e 3 | … | e n = I n = S − 1 S = S − 1 [ x 1 | x 2 | x 3 | … | x g | R ] = [ S − 1 x 1 | S − 1 x 2 | S − 1 x 3 | … | S − 1 x g | S − 1 R ]

So

S − 1 x i = e i 1 ≤ i ≤ g ( ∗ )

Preparations in place, we compute the characteristic polynomial of A ,

p A x = det A − x I n   Definition CP = 1 det A − x I n   Property OCN = det I n det A − x I n   Definition DM = det S − 1 S det A − x I n   Definition MI = det S − 1 det S det A − x I n   Theorem DRMM = det S − 1 det A − x I n det S   Property CMCN = det S − 1 A − x I n S   Theorem DRMM = det S − 1 A S − S − 1 x I n S   Theorem MMDAA = det S − 1 A S − x S − 1 I n S   Theorem MMSMM = det S − 1 A S − x S − 1 S   Theorem MMIM = det S − 1 A S − x I n   Definition MI = p S − 1 A S x   Definition CP

What can we learn then about the matrix S − 1 A S ?

S − 1 A S = S − 1 A [ x 1 | x 2 | x 3 | … | x g | R ] = S − 1 [ A x 1 | A x 2 | A x 3 | … | A x g | A R ]   Definition MM = S − 1 [ λ x 1 | λ x 2 | λ x 3 | … | λ x g | A R ]   Definition EEM = [ S − 1 λ x 1 | S − 1 λ x 2 | S − 1 λ x 3 | … | S − 1 λ x g | S − 1 A R ]   Definition MM = [ λ S − 1 x 1 | λ S − 1 x 2 | λ S − 1 x 3 | … | λ S − 1 x g | S − 1 A R ]   Theorem MMSMM = [ λ e 1 | λ e 2 | λ e 3 | … | λ e g | S − 1 A R ]   S − 1 S = I n , (( ∗ ) above)

Now imagine computing the characteristic polynomial of A by computing the characteristic polynomial of S − 1 A S using the form just obtained. The first g columns of S − 1 A S are all zero, save for a λ on the diagonal. So if we compute the determinant by expanding about the first column, successively, we will get successive factors of ( λ − x ) . More precisely, let T be the square matrix of size n − g that is formed from the last n − g rows and last n − g columns of S − 1 A R . Then

p A x = p S − 1 A S x = ( λ − x ) g p T x .

This says that ( x − λ ) is a factor of the characteristic polynomial at least g times, so the algebraic multiplicity of λ as an eigenvalue of A is greater than or equal to g (Definition AME). In other words,

γ A λ = g ≤ α A λ

as desired.

Theorem NEM says that the sum of the algebraic multiplicities for all the eigenvalues of A is equal to n . Since the algebraic multiplicity is a positive quantity, no single algebraic multiplicity can exceed n without the sum of all of the algebraic multiplicities doing the same. ■

Theorem  MNEM
Maximum Number of Eigenvalues of a Matrix
Suppose that A is a square matrix of size n . Then A cannot have more than n distinct eigenvalues. □

Proof   Suppose that A has k distinct eigenvalues, λ 1 , λ 2 , λ 3 , … , λ k . Then

k = ∑ i = 1 k 1 ≤ ∑ i = 1 k α A λ i   Theorem ME = n   Theorem NEM

■

Subsection EHM: Eigenvalues of Hermitian Matrices

Recall that a matrix is Hermitian (or self-adjoint) if A = A ∗ ( Definition HM). In the case where A is a matrix whose entries are all real numbers, being Hermitian is identical to being symmetric (Definition SYM). Keep this in mind as you read the next two theorems. Their hypotheses could be changed to “suppose A is a real symmetric matrix.”

Theorem  HMRE
Hermitian Matrices have Real Eigenvalues
Suppose that A is a Hermitian matrix and λ is an eigenvalue of A . Then λ ∈ ℝ . □

Proof   Let x ≠ 0 be one eigenvector of A for the eigenvalue λ . Then by Theorem PIP we know x , x ≠ 0 . So

λ = 1 x , x λ x , x   Property MICN = 1 x , x λ x , x   Theorem IPSM = 1 x , x A x , x   Definition EEM = 1 x , x x , A x   Theorem HMIP = 1 x , x x , λ x   Definition EEM = 1 x , x λ ¯ x , x   Theorem IPSM = λ ¯   Property MICN

If a complex number is equal to its conjugate, then it has a complex part equal to zero, and therefore is a real number. ■

Notice the appealing symmetry to the justifications given for the steps of this proof. In the center is the ability to pitch a Hermitian matrix from one side of the inner product to the other.

Look back and compare Example ESMS4 and Example CEMS6. In Example CEMS6 the matrix has only real entries, yet the characteristic polynomial has roots that are complex numbers, and so the matrix has complex eigenvalues. However, in Example ESMS4, the matrix has only real entries, but is also symmetric, and hence Hermitian. So by Theorem HMRE, we were guaranteed eigenvalues that are real numbers.

In many physical problems, a matrix of interest will be real and symmetric, or Hermitian. Then if the eigenvalues are to represent physical quantities of interest, Theorem HMRE guarantees that these values will not be complex numbers.

The eigenvectors of a Hermitian matrix also enjoy a pleasing property that we will exploit later.

Theorem  HMOE
Hermitian Matrices have Orthogonal Eigenvectors
Suppose that A is a Hermitian matrix and x and y are two eigenvectors of A for different eigenvalues. Then x and y are orthogonal vectors. □

Proof   Let x be an eigenvector of A for λ and let y be an eigenvector of A for a different eigenvalue ρ . So we have λ − ρ ≠ 0 . Then

x , y = 1 λ − ρ λ − ρ x , y   Property MICN = 1 λ − ρ λ x , y − ρ x , y   Property MICN = 1 λ − ρ λ x , y − x , ρ ¯ y   Theorem IPSM = 1 λ − ρ λ x , y − x , ρ y   Theorem HMRE = 1 λ − ρ A x , y − x , A y   Definition EEM = 1 λ − ρ A x , y − A x , y   Theorem HMIP = 1 λ − ρ 0   Property AICN = 0

This equality says that x and y are orthogonal vectors ( Definition OV). ■

Notice again how the key step in this proof is the fundamental property of a Hermitian matrix ( Theorem HMIP) — the ability to swap A across the two arguments of the inner product. We’ll build on these results and continue to see some more interesting properties in Section OD.

Subsection READ: Reading Questions

Subsection EXC: Exercises

T10 Suppose that A is a square matrix. Prove that the constant term of the characteristic polynomial of A is equal to the determinant of A .  
Contributed by  Robert Beezer Solution [ 1322]

T20 Suppose that A is a square matrix. Prove that a single vector may not be an eigenvector of A for two different eigenvalues.  
Contributed by  Robert Beezer Solution [ 1323]

T22 Suppose that U is a unitary matrix with eigenvalue λ . Prove that λ had modulus 1, i.e.  λ = 1 . This says that all of the eigenvalues of a unitary matrix lie on the unit circle of the complex plane.  
Contributed by  Robert Beezer

T30 Theorem DCP tells us that the characteristic polynomial of a square matrix of size n has degree n . By suitably augmenting the proof of Theorem DCP prove that the coefficient of x n in the characteristic polynomial is ( − 1 ) n .  
Contributed by  Robert Beezer

T50 Theorem EIM says that if λ is an eigenvalue of the nonsingular matrix A , then 1 λ is an eigenvalue of A − 1 . Write an alternate proof of this theorem using the characteristic polynomial and without making reference to an eigenvector of A for λ .  
Contributed by  Robert Beezer Solution [ 1323]

Subsection SOL: Solutions

T10 Contributed by  Robert Beezer Statement [ 1320]
Suppose that the characteristic polynomial of A is

p A x = a 0 + a 1 x + a 2 x 2 + ⋯ + a n x n

Then

a 0 = a 0 + a 1 ( 0 ) + a 2 ( 0 ) 2 + ⋯ + a n ( 0 ) n = p A 0 = det A − 0 I n   Definition CP = det A

T20 Contributed by  Robert Beezer Statement [ 1320]
Suppose that the vector x ≠ 0 is an eigenvector of A for the two eigenvalues λ and ρ , where λ ≠ ρ . Then λ − ρ ≠ 0 , and we also have

0 = A x − A x   Property AIC = λ x − ρ x   Definition EEM = ( λ − ρ ) x   Property DSAC

By Theorem SMEZV, either λ − ρ = 0 or x = 0 , which are both contradictions.

T50 Contributed by  Robert Beezer Statement [ 1320]
Since λ is an eigenvalue of a nonsingular matrix, λ ≠ 0 ( Theorem SMZE). A is invertible ( Theorem NI), and so − λ A is invertible ( Theorem MISM). Thus − λ A is nonsingular ( Theorem NI) and det − λ A ≠ 0 ( Theorem SMZD).

p A − 1 1 λ = det A − 1 − 1 λ I n   Definition CP = 1 det A − 1 − 1 λ I n   Property OCN = 1 det − λ A det − λ A det A − 1 − 1 λ I n   Property MICN = 1 det − λ A det − λ A A − 1 − 1 λ I n   Theorem DRMM = 1 det − λ A det − λ A A − 1 − − λ A 1 λ I n   Theorem MMDAA = 1 det − λ A det − λ I n − − λ A 1 λ I n   Definition MI = 1 det − λ A det − λ I n + λ 1 λ A I n   Theorem MMSMM = 1 det − λ A det − λ I n + 1 A I n   Property MICN = 1 det − λ A det − λ I n + A I n   Property OCN = 1 det − λ A det − λ I n + A   Theorem MMIM = 1 det − λ A det A − λ I n   Property ACM = 1 det − λ A p A λ   Definition CP = 1 det − λ A 0   Theorem EMRCP = 0   Property ZCN

So 1 λ is a root of the characteristic polynomial of A − 1 and so is an eigenvalue of A − 1 . This proof is due to Sara Bucht.

Chapter 7. Chapter LT  Linear Transformations

In the next linear algebra course you take, the first lecture might be a reminder about what a vector space is (Definition VS), their ten properties, basic theorems and then some examples. The second lecture would likely be all about linear transformations. While it may seem we have waited a long time to present what must be a central topic, in truth we have already been working with linear transformations for some time.

Functions are important objects in the study of calculus, but have been absent from this course until now (well, not really, it just seems that way). In your study of more advanced mathematics it is nearly impossible to escape the use of functions — they are as fundamental as sets are.

Section ILT  Injective Linear Transformations

Section SLT  Surjective Linear Transformations

Section IVLT  Invertible Linear Transformations

Parent topic: Part C  Core

Previous topic: Section PEE  Properties of Eigenvalues and Eigenvectors

Next topic: Section ILT  Injective Linear Transformations

Section ILT  Injective Linear Transformations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Some linear transformations possess one, or both, of two key properties, which go by the names injective and surjective. We will see that they are closely related to ideas like linear independence and spanning, and subspaces like the null space and the column space. In this section we will define an injective linear transformation and analyze the resulting consequences. The next section will do the same for the surjective property. In the final section of this chapter we will see what happens when we have the two properties simultaneously.

As usual, we lead with a definition.

Definition  ILT
Injective Linear Transformation
Suppose T : U → V is a linear transformation. Then T is injective if whenever T x = T y , then x = y . △

Given an arbitrary function, it is possible for two different inputs to yield the same output (think about the function f ( x ) = x 2 and the inputs x = 3 and x = − 3 ). For an injective function, this never happens. If we have equal outputs ( T x = T y ) then we must have achieved those equal outputs by employing equal inputs ( x = y ). Some authors prefer the term one-to-one where we use injective, and we will sometimes refer to an injective linear transformation as an injection.

Parent topic: Chapter LT  Linear Transformations

Previous topic: Chapter LT  Linear Transformations

Next topic: Section SLT  Surjective Linear Transformations

Subsection EILT: Examples of Injective Linear Transformations

It is perhaps most instructive to examine a linear transformation that is not injective first.

Example  NIAQ
Not injective, Archetype Q
Archetype Q is the linear transformation

T : ℂ 5 → ℂ 5 , T x 1 x 2 x 3 x 4 x 5 = − 2 x 1 + 3 x 2 + 3 x 3 − 6 x 4 + 3 x 5 − 1 6 x 1 + 9 x 2 + 1 2 x 3 − 2 8 x 4 + 2 8 x 5 − 1 9 x 1 + 7 x 2 + 1 4 x 3 − 3 2 x 4 + 3 7 x 5 − 2 1 x 1 + 9 x 2 + 1 5 x 3 − 3 5 x 4 + 3 9 x 5 − 9 x 1 + 5 x 2 + 7 x 3 − 1 6 x 4 + 1 6 x 5

Notice that for

x = 1 3 − 1 2 4 y = 4 7 0 5 7  we have T 1 3 − 1 2 4 = 4 5 5 7 2 7 7 3 1 T 4 7 0 5 7 = 4 5 5 7 2 7 7 3 1

So we have two vectors from the domain, x ≠ y , yet T x = T y , in violation of Definition ILT. This is another example where you should not concern yourself with how x and y were selected, as this will be explained shortly. However, do understand why these two vectors provide enough evidence to conclude that T is not injective. ⊠

Here’s a cartoon of a non-injective linear transformation. Notice that the central feature of this cartoon is that T u = v = T w . Even though this happens again with some unnamed vectors, it only takes one occurrence to destroy the possibility of injectivity. Note also that the two vectors displayed in the bottom of V have no bearing, either way, on the injectivity of T .

To show that a linear transformation is not injective, it is enough to find a single pair of inputs that get sent to the identical output, as in Example NIAQ. However, to show that a linear transformation is injective we must establish that this coincidence of outputs never occurs. Here is an example that shows how to establish this.

Example  IAR
Injective, Archetype R
Archetype R is the linear transformation

T : ℂ 5 → ℂ 5 , T x 1 x 2 x 3 x 4 x 5 = − 6 5 x 1 + 1 2 8 x 2 + 1 0 x 3 − 2 6 2 x 4 + 4 0 x 5 3 6 x 1 − 7 3 x 2 − x 3 + 1 5 1 x 4 − 1 6 x 5 − 4 4 x 1 + 8 8 x 2 + 5 x 3 − 1 8 0 x 4 + 2 4 x 5 3 4 x 1 − 6 8 x 2 − 3 x 3 + 1 4 0 x 4 − 1 8 x 5 1 2 x 1 − 2 4 x 2 − x 3 + 4 9 x 4 − 5 x 5

To establish that R is injective we must begin with the assumption that T x = T y and somehow arrive from this at the conclusion that x = y . Here we go,

T x = T y T x 1 x 2 x 3 x 4 x 5 = T y 1 y 2 y 3 y 4 y 5 − 6 5 x 1 + 1 2 8 x 2 + 1 0 x 3 − 2 6 2 x 4 + 4 0 x 5 3 6 x 1 − 7 3 x 2 − x 3 + 1 5 1 x 4 − 1 6 x 5 − 4 4 x 1 + 8 8 x 2 + 5 x 3 − 1 8 0 x 4 + 2 4 x 5 3 4 x 1 − 6 8 x 2 − 3 x 3 + 1 4 0 x 4 − 1 8 x 5 1 2 x 1 − 2 4 x 2 − x 3 + 4 9 x 4 − 5 x 5 = − 6 5 y 1 + 1 2 8 y 2 + 1 0 y 3 − 2 6 2 y 4 + 4 0 y 5 3 6 y 1 − 7 3 y 2 − y 3 + 1 5 1 y 4 − 1 6 y 5 − 4 4 y 1 + 8 8 y 2 + 5 y 3 − 1 8 0 y 4 + 2 4 y 5 3 4 y 1 − 6 8 y 2 − 3 y 3 + 1 4 0 y 4 − 1 8 y 5 1 2 y 1 − 2 4 y 2 − y 3 + 4 9 y 4 − 5 y 5 − 6 5 x 1 + 1 2 8 x 2 + 1 0 x 3 − 2 6 2 x 4 + 4 0 x 5 3 6 x 1 − 7 3 x 2 − x 3 + 1 5 1 x 4 − 1 6 x 5 − 4 4 x 1 + 8 8 x 2 + 5 x 3 − 1 8 0 x 4 + 2 4 x 5 3 4 x 1 − 6 8 x 2 − 3 x 3 + 1 4 0 x 4 − 1 8 x 5 1 2 x 1 − 2 4 x 2 − x 3 + 4 9 x 4 − 5 x 5 − − 6 5 y 1 + 1 2 8 y 2 + 1 0 y 3 − 2 6 2 y 4 + 4 0 y 5 3 6 y 1 − 7 3 y 2 − y 3 + 1 5 1 y 4 − 1 6 y 5 − 4 4 y 1 + 8 8 y 2 + 5 y 3 − 1 8 0 y 4 + 2 4 y 5 3 4 y 1 − 6 8 y 2 − 3 y 3 + 1 4 0 y 4 − 1 8 y 5 1 2 y 1 − 2 4 y 2 − y 3 + 4 9 y 4 − 5 y 5 = 0 0 0 0 0 − 6 5 ( x 1 − y 1 ) + 1 2 8 ( x 2 − y 2 ) + 1 0 ( x 3 − y 3 ) − 2 6 2 ( x 4 − y 4 ) + 4 0 ( x 5 − y 5 ) 3 6 ( x 1 − y 1 ) − 7 3 ( x 2 − y 2 ) − ( x 3 − y 3 ) + 1 5 1 ( x 4 − y 4 ) − 1 6 ( x 5 − y 5 ) − 4 4 ( x 1 − y 1 ) + 8 8 ( x 2 − y 2 ) + 5 ( x 3 − y 3 ) − 1 8 0 ( x 4 − y 4 ) + 2 4 ( x 5 − y 5 ) 3 4 ( x 1 − y 1 ) − 6 8 ( x 2 − y 2 ) − 3 ( x 3 − y 3 ) + 1 4 0 ( x 4 − y 4 ) − 1 8 ( x 5 − y 5 ) 1 2 ( x 1 − y 1 ) − 2 4 ( x 2 − y 2 ) − ( x 3 − y 3 ) + 4 9 ( x 4 − y 4 ) − 5 ( x 5 − y 5 ) = 0 0 0 0 0 − 6 5 1 2 8 1 0 − 2 6 2 4 0 3 6 − 7 3 − 1 1 5 1 − 1 6 − 4 4 8 8 5 − 1 8 0 2 4 3 4 − 6 8 − 3 1 4 0 − 1 8 1 2 − 2 4 − 1 4 9 − 5 x 1 − y 1 x 2 − y 2 x 3 − y 3 x 4 − y 4 x 5 − y 5 = 0 0 0 0 0

Now we recognize that we have a homogeneous system of 5 equations in 5 variables (the terms x i − y i are the variables), so we row-reduce the coefficient matrix to

1 0 0 0 0 0 1  0 0 0 0 0 1 0 0 0 0 0 1  0 0 0 0 0 1

So the only solution is the trivial solution

x 1 − y 1 = 0 x 2 − y 2 = 0 x 3 − y 3 = 0 x 4 − y 4 = 0 x 5 − y 5 = 0

and we conclude that indeed x = y . By Definition ILT, T is injective. ⊠

Here’s the cartoon for an injective linear transformation. It is meant to suggest that we never have two inputs associated with a single output. Again, the two lonely vectors at the bottom of V have no bearing either way on the injectivity of T .

Let’s now examine an injective linear transformation between abstract vector spaces.

Example  IAV
Injective, Archetype V
Archetype V is defined by

T : P 3 → M 2 2 , T a + b x + c x 2 + d x 3 = a + b a − 2 c d b − d

To establish that the linear transformation is injective, begin by supposing that two polynomial inputs yield the same output matrix,

T a 1 + b 1 x + c 1 x 2 + d 1 x 3 = T a 2 + b 2 x + c 2 x 2 + d 2 x 3

Then

O = 0 0 0 0 = T a 1 + b 1 x + c 1 x 2 + d 1 x 3 − T a 2 + b 2 x + c 2 x 2 + d 2 x 3  Hypothesis = T ( a 1 + b 1 x + c 1 x 2 + d 1 x 3 ) − ( a 2 + b 2 x + c 2 x 2 + d 2 x 3 )   Definition LT = T ( a 1 − a 2 ) + ( b 1 − b 2 ) x + ( c 1 − c 2 ) x 2 + ( d 1 − d 2 ) x 3  Operations in  P 3 = ( a 1 − a 2 ) + ( b 1 − b 2 ) ( a 1 − a 2 ) − 2 ( c 1 − c 2 ) ( d 1 − d 2 ) ( b 1 − b 2 ) − ( d 1 − d 2 )  Definition of  T

This single matrix equality translates to the homogeneous system of equations in the variables a i − b i ,

( a 1 − a 2 ) + ( b 1 − b 2 ) = 0 ( a 1 − a 2 ) − 2 ( c 1 − c 2 ) = 0 ( d 1 − d 2 ) = 0 ( b 1 − b 2 ) − ( d 1 − d 2 ) = 0

This system of equations can be rewritten as the matrix equation

1 1 0 0 1 0 − 2 0 0 0 0 1 0 1 0 − 1 ( a 1 − a 2 ) ( b 1 − b 2 ) ( c 1 − c 2 ) ( d 1 − d 2 ) = 0 0 0 0

Since the coefficient matrix is nonsingular (check this) the only solution is trivial, i.e.

a 1 − a 2 = 0 b 1 − b 2 = 0 c 1 − c 2 = 0 d 1 − d 2 = 0  so that a 1 = a 2 b 1 = b 2 c 1 = c 2 d 1 = d 2

so the two inputs must be equal polynomials. By Definition ILT, T is injective. ⊠

Subsection KLT: Kernel of a Linear Transformation

For a linear transformation T : U → V , the kernel is a subset of the domain U . Informally, it is the set of all inputs that the transformation sends to the zero vector of the codomain. It will have some natural connections with the null space of a matrix, so we will keep the same notation, and if you think about your objects, then there should be little confusion. Here’s the careful definition.

Definition  KLT
Kernel of a Linear Transformation
Suppose T : U → V is a linear transformation. Then the kernel of T is the set

K T = u ∈ U T u = 0

(This definition contains Notation KLT.) △

Notice that the kernel of T is just the preimage of 0 , T − 1 0 (Definition PI). Here’s an example.

Example  NKAO
Nontrivial kernel, Archetype O
Archetype O is the linear transformation

T : ℂ 3 → ℂ 5 , T x 1 x 2 x 3 = − x 1 + x 2 − 3 x 3 − x 1 + 2 x 2 − 4 x 3 x 1 + x 2 + x 3 2 x 1 + 3 x 2 + x 3 x 1 + 2 x 3

To determine the elements of ℂ 3 in K T , find those vectors u such that T u = 0 , that is,

T u = 0 − u 1 + u 2 − 3 u 3 − u 1 + 2 u 2 − 4 u 3 u 1 + u 2 + u 3 2 u 1 + 3 u 2 + u 3 u 1 + 2 u 3 = 0 0 0 0 0

Vector equality (Definition CVE) leads us to a homogeneous system of 5 equations in the variables u i ,

− u 1 + u 2 − 3 u 3 = 0 − u 1 + 2 u 2 − 4 u 3 = 0 u 1 + u 2 + u 3 = 0 2 u 1 + 3 u 2 + u 3 = 0 u 1 + 2 u 3 = 0

Row-reducing the coefficient matrix gives

1 0 2 0 1  − 1 0 0 0 0 0 0 0 0 0

The kernel of T is the set of solutions to this homogeneous system of equations, which by Theorem BNS can be expressed as

K T = − 2 1 1

⊠

We know that the span of a set of vectors is always a subspace (Theorem SSS), so the kernel computed in Example NKAO is also a subspace. This is no accident, the kernel of a linear transformation is always a subspace.

Theorem  KLTS
Kernel of a Linear Transformation is a Subspace
Suppose that T : U → V is a linear transformation. Then the kernel of T , K T , is a subspace of U . □

Proof   We can apply the three-part test of Theorem TSS. First T 0 U = 0 V by Theorem LTTZZ, so 0 U ∈ K T and we know that the kernel is non-empty.

Suppose we assume that x , y ∈ K T . Is x + y ∈ K T ?

T x + y = T x + T y   Definition LT = 0 + 0 x , y ∈ K T = 0   Property Z

This qualifies x + y for membership in K T . So we have additive closure.

Suppose we assume that α ∈ ℂ and x ∈ K T . Is α x ∈ K T ?

T α x = α T x   Definition LT = α 0 x ∈ K T = 0   Theorem ZVSM

This qualifies α x for membership in K T . So we have scalar closure and Theorem TSS tells us that K T is a subspace of U . ■

Let’s compute another kernel, now that we know in advance that it will be a subspace.

Example  TKAP
Trivial kernel, Archetype P
Archetype P is the linear transformation

T : ℂ 3 → ℂ 5 , T x 1 x 2 x 3 = − x 1 + x 2 + x 3 − x 1 + 2 x 2 + 2 x 3 x 1 + x 2 + 3 x 3 2 x 1 + 3 x 2 + x 3 − 2 x 1 + x 2 + 3 x 3

To determine the elements of ℂ 3 in K T , find those vectors u such that T u = 0 , that is,

T u = 0 − u 1 + u 2 + u 3 − u 1 + 2 u 2 + 2 u 3 u 1 + u 2 + 3 u 3 2 u 1 + 3 u 2 + u 3 − 2 u 1 + u 2 + 3 u 3 = 0 0 0 0 0

Vector equality (Definition CVE) leads us to a homogeneous system of 5 equations in the variables u i ,

− u 1 + u 2 + u 3 = 0 − u 1 + 2 u 2 + 2 u 3 = 0 u 1 + u 2 + 3 u 3 = 0 2 u 1 + 3 u 2 + u 3 = 0 − 2 u 1 + u 2 + 3 u 3 = 0

Row-reducing the coefficient matrix gives

1 0 0 0 1  0 0 0 1 0 0 0 0 0 0

The kernel of T is the set of solutions to this homogeneous system of equations, which is simply the trivial solution u = 0 , so

K T = 0 =

⊠

Our next theorem says that if a preimage is a non-empty set then we can construct it by picking any one element and adding on elements of the kernel.

Theorem  KPI
Kernel and Pre-Image
Suppose T : U → V is a linear transformation and v ∈ V . If the preimage T − 1 v is non-empty, and u ∈ T − 1 v then

T − 1 v = u + z z ∈ K T = u + K T

□

Proof   Let M = u + z z ∈ K T . First, we show that M ⊆ T − 1 v . Suppose that w ∈ M , so w has the form w = u + z , where z ∈ K T . Then

T w = T u + z = T u + T z   Definition LT = v + 0 u ∈ T − 1 v , z ∈ K T = v   Property Z

which qualifies w for membership in the preimage of v , w ∈ T − 1 v .

For the opposite inclusion, suppose x ∈ T − 1 v . Then,

T x − u = T x − T u   Definition LT = v − v x , u ∈ T − 1 v = 0

This qualifies x − u for membership in the kernel of T , K T . So there is a vector z ∈ K T such that x − u = z . Rearranging this equation gives x = u + z and so x ∈ M . So T − 1 v ⊆ M and we see that M = T − 1 v , as desired. ■

This theorem, and its proof, should remind you very much of Theorem PSPHS. Additionally, you might go back and review Example SPIAS. Can you tell now which is the only preimage to be a subspace?

The next theorem is one we will cite frequently, as it characterizes injections by the size of the kernel.

Theorem  KILT
Kernel of an Injective Linear Transformation
Suppose that T : U → V is a linear transformation. Then T is injective if and only if the kernel of T is trivial, K T = 0 . □

Proof   ( ⇒ ) We assume T is injective and we need to establish that two sets are equal ( Definition SE). Since the kernel is a subspace ( Theorem KLTS), 0 ⊆ K T . To establish the opposite inclusion, suppose x ∈ K T .

T x = 0   Definition KLT = T 0   Theorem LTTZZ

We can apply Definition ILT to conclude that x = 0 . Therefore K T ⊆ 0 and by Definition SE, K T = 0 .

( ⇐ ) To establish that T is injective, appeal to Definition ILT and begin with the assumption that T x = T y . Then

T x − y = T x − T y   Definition LT = 0  Hypothesis

So x − y ∈ K T by Definition KLT and with the hypothesis that the kernel is trivial we conclude that x − y = 0 . Then

y = y + 0 = y + x − y = x

thus establishing that T is injective by Definition ILT. ■

Example  NIAQR
Not injective, Archetype Q, revisited
We are now in a position to revisit our first example in this section, Example NIAQ. In that example, we showed that Archetype Q is not injective by constructing two vectors, which when used to evaluate the linear transformation provided the same output, thus violating Definition ILT. Just where did those two vectors come from?

The key is the vector

z = 3 4 1 3 3

which you can check is an element of K T for Archetype Q. Choose a vector x at random, and then compute y = x + z (verify this computation back in Example NIAQ). Then

T y = T x + z = T x + T z   Definition LT = T x + 0 z ∈ K T = T x   Property Z

Whenever the kernel of a linear transformation is non-trivial, we can employ this device and conclude that the linear transformation is not injective. This is another way of viewing Theorem KILT. For an injective linear transformation, the kernel is trivial and our only choice for z is the zero vector, which will not help us create two different inputs for T that yield identical outputs. For every one of the archetypes that is not injective, there is an example presented of exactly this form. ⊠

Example  NIAO
Not injective, Archetype O
In Example NKAO the kernel of Archetype O was determined to be

− 2 1 1

a subspace of ℂ 3 with dimension 1. Since the kernel is not trivial, Theorem KILT tells us that T is not injective. ⊠

Example  IAP
Injective, Archetype P
In Example TKAP it was shown that the linear transformation in Archetype P has a trivial kernel. So by Theorem KILT, T is injective. ⊠

Subsection ILTLI: Injective Linear Transformations and Linear Independence

There is a connection between injective linear transformations and linearly independent sets that we will make precise in the next two theorems. However, more informally, we can get a feel for this connection when we think about how each property is defined. A set of vectors is linearly independent if the only relation of linear dependence is the trivial one. A linear transformation is injective if the only way two input vectors can produce the same output is in the trivial way, when both input vectors are equal.

Theorem  ILTLI
Injective Linear Transformations and Linear Independence
Suppose that T : U → V is an injective linear transformation and S = u 1 , u 2 , u 3 , … , u t is a linearly independent subset of U . Then R = T u 1 , T u 2 , T u 3 , … , T u t is a linearly independent subset of V . □

Proof   Begin with a relation of linear dependence on R (Definition RLD, Definition LI),

a 1 T u 1 + a 2 T u 2 + a 3 T u 3 + … + a t T u t = 0 T a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a t u t = 0   Theorem LTLC a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a t u t ∈ K T   Definition KLT a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a t u t ∈ 0   Theorem KILT a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a t u t = 0   Definition SET

Since this is a relation of linear dependence on the linearly independent set S , we can conclude that

a 1 = 0 a 2 = 0 a 3 = 0 … a t = 0

and this establishes that R is a linearly independent set. ■

Theorem  ILTB
Injective Linear Transformations and Bases
Suppose that T : U → V is a linear transformation and B = u 1 , u 2 , u 3 , … , u m is a basis of U . Then T is injective if and only if C = T u 1 , T u 2 , T u 3 , … , T u m is a linearly independent subset of V . □

Proof   ( ⇒ ) Assume T is injective. Since B is a basis, we know B is linearly independent (Definition B). Then Theorem ILTLI says that C is a linearly independent subset of V .

( ⇐ ) Assume that C is linearly independent. To establish that T is injective, we will show that the kernel of T is trivial ( Theorem KILT). Suppose that u ∈ K T . As an element of U , we can write u as a linear combination of the basis vectors in B (uniquely). So there are are scalars, a 1 , a 2 , a 3 , … , a m , such that

u = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a m u m

Then,

0 = T u   Definition KLT = T a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a m u m   Definition TSVS = a 1 T u 1 + a 2 T u 2 + a 3 T u 3 + ⋯ + a m T u m   Theorem LTLC

This is a relation of linear dependence (Definition RLD) on the linearly independent set C , so the scalars are all zero: a 1 = a 2 = a 3 = ⋯ = a m = 0 . Then

u = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a m u m = 0 u 1 + 0 u 2 + 0 u 3 + ⋯ + 0 u m   Theorem ZSSM = 0 + 0 + 0 + ⋯ + 0   Theorem ZSSM = 0   Property Z

Since u was chosen as an arbitrary vector from K T , we have K T = 0 and Theorem KILT tells us that T is injective. ■

Subsection ILTD: Injective Linear Transformations and Dimension

Theorem  ILTD
Injective Linear Transformations and Dimension
Suppose that T : U → V is an injective linear transformation. Then dim U ≤ dim V . □

Proof   Suppose to the contrary that m = dim U > dim V = t . Let B be a basis of U , which will then contain m vectors. Apply T to each element of B to form a set C that is a subset of V . By Theorem ILTB, C is linearly independent and therefore must contain m distinct vectors. So we have found a set of m linearly independent vectors in V , a vector space of dimension t , with m > t . However, this contradicts Theorem G, so our assumption is false and dim U ≤ dim V . ■

Example  NIDAU
Not injective by dimension, Archetype U
The linear transformation in Archetype U is

T : M 2 3 → ℂ 4 , T a b c d e f = a + 2 b + 1 2 c − 3 d + e + 6 f 2 a − b − c + d − 1 1 f a + b + 7 c + 2 d + e − 3 f a + 2 b + 1 2 c + 5 e − 5 f

Since dim M 2 3 = 6 > 4 = dim ℂ 4 , T cannot be injective for then T would violate Theorem ILTD. ⊠

Notice that the previous example made no use of the actual formula defining the function. Merely a comparison of the dimensions of the domain and codomain are enough to conclude that the linear transformation is not injective. Archetype M and Archetype N are two more examples of linear transformations that have “big” domains and “small” codomains, resulting in “collisions” of outputs and thus are non-injective linear transformations.

Subsection CILT: Composition of Injective Linear Transformations

In Subsection LT.NLTFO we saw how to combine linear transformations to build new linear transformations, specifically, how to build the composition of two linear transformations (Definition LTC). It will be useful later to know that the composition of injective linear transformations is again injective, so we prove that here.

Theorem  CILTI
Composition of Injective Linear Transformations is Injective
Suppose that T : U → V and S : V → W are injective linear transformations. Then ( S ∘ T ) : U → W is an injective linear transformation. □

Proof   That the composition is a linear transformation was established in Theorem CLTLT, so we need only establish that the composition is injective. Applying Definition ILT, choose x , y from U . Then if S ∘ T x = S ∘ T y ,

⇒ S T x = S T y   Definition LTC ⇒ T x = T y   Definition ILT  for  S ⇒ x = y   Definition ILT  for  T

■

Subsection READ: Reading Questions

Subsection EXC: Exercises

C10 Each archetype below is a linear transformation. Compute the kernel for each.
Archetype M
Archetype N
Archetype O
Archetype P
Archetype Q
Archetype R
Archetype S
Archetype T
Archetype U
Archetype V
Archetype W
Archetype X

Contributed by  Robert Beezer

C20 The linear transformation T : ℂ 4 → ℂ 3 is not injective. Find two inputs x , y ∈ ℂ 4 that yield the same output (that is T x = T y ).

T x 1 x 2 x 3 x 4 = 2 x 1 + x 2 + x 3 − x 1 + 3 x 2 + x 3 − x 4 3 x 1 + x 2 + 2 x 3 − 2 x 4

Contributed by  Robert Beezer Solution [ 1491]

C25 Define the linear transformation

T : ℂ 3 → ℂ 2 , T x 1 x 2 x 3 = 2 x 1 − x 2 + 5 x 3 − 4 x 1 + 2 x 2 − 1 0 x 3

Find a basis for the kernel of T , K T . Is T injective?  
Contributed by  Robert Beezer Solution [ 1493]

C26 Let A = 1 2 3 1 0 2 − 1 1 0 1 1 2 − 1 − 2 1 1 3 2 1 2 and let T : ℂ 5 → ℂ 4 be given by T x = A x . Is T injective? (Hint: No calculation is required.)

Contributed by  Chris Black Solution [ 1495]

C27 Let T : ℂ 3 → ℂ 3 be given by T x y z = 2 x + y + z x − y + 2 z x + 2 y − z . Find K T . Is T injective?  
Contributed by  Chris Black Solution [ 1495]

C28 Let A = 1 2 3 1 2 − 1 1 0 1 2 − 1 − 2 1 3 2 1 and let T : ℂ 4 → ℂ 4 be given by T x = A x . Find K T . Is T injective?  
Contributed by  Chris Black Solution [ 1496]

C29 Let A = 1 2 1 1 2 1 1 0 1 2 1 2 1 2 1 1 and let T : ℂ 4 → ℂ 4 be given by T x = A x . Find K T . Is T injective?  
Contributed by  Chris Black Solution [ 1496]

C30 Let T : M 2 , 2 → P 2 be given by T a b c d = ( a + b ) + ( a + c ) x + ( a + d ) x 2 . Is T injective? Find K T .  
Contributed by  Chris Black Solution [ 1497]

C31 Given that the linear transformation T : ℂ 3 → ℂ 3 , T x y z = 2 x + y 2 y + z x + 2 z is injective, show directly that T e 1 , T e 2 , T e 3 is a linearly independent set.  
Contributed by  Chris Black Solution [ 1498]

C32 Given that the linear transformation T : ℂ 2 → ℂ 3 , T x y = x + y 2 x + y x + 2 y is injective, show directly that T e 1 , T e 2 is a linearly independent set.  
Contributed by  Chris Black Solution [ 1498]

C33 Given that the linear transformation T : ℂ 3 → ℂ 5 , T x y z = 1 3 2 0 1 1 1 2 1 1 0 1 3 1 2 x y z is injective, show directly that T e 1 , T e 2 , T e 3 is a linearly independent set.  
Contributed by  Chris Black Solution [ 1499]

C40 Show that the linear transformation R is not injective by finding two different elements of the domain, x and y , such that R x = R y . ( S 2 2 is the vector space of symmetric 2 × 2 matrices.)

R : S 2 2 → P 1 R a b b c = ( 2 a − b + c ) + ( a + b + 2 c ) x

Contributed by  Robert Beezer Solution [ 1500]

M60 Suppose U and V are vector spaces. Define the function Z : U → V by T u = 0 V for every u ∈ U . Then by Exercise LT.M60, Z is a linear transformation. Formulate a condition on U that is equivalent to Z being an injective linear transformation. In other words, fill in the blank to complete the following statement (and then give a proof): Z is injective if and only if U is __________________. (See Exercise SLT.M60, Exercise IVLT.M60.)  
Contributed by  Robert Beezer

T10 Suppose T : U → V is a linear transformation. For which vectors v ∈ V is T − 1 v a subspace of U ?  
Contributed by  Robert Beezer

T15 Suppose that that T : U → V and S : V → W are linear transformations. Prove the following relationship between null spaces.

K T ⊆ K S ∘ T

Contributed by  Robert Beezer Solution [ 1502]

T20 Suppose that A is an m × n matrix. Define the linear transformation T by

T : ℂ n → ℂ m , T x = A x

Prove that the kernel of T equals the null space of A , K T = N A .  
Contributed by  Andy Zimmer Solution [ 1502]

Subsection SOL: Solutions

C20 Contributed by  Robert Beezer Statement [ 1485]
A linear transformation that is not injective will have a non-trivial kernel ( Theorem KILT), and this is the key to finding the desired inputs. We need one non-trivial element of the kernel, so suppose that z ∈ ℂ 4 is an element of the kernel,

0 0 0 = 0 = T z = 2 z 1 + z 2 + z 3 − z 1 + 3 z 2 + z 3 − z 4 3 z 1 + z 2 + 2 z 3 − 2 z 4

Vector equality Definition CVE leads to the homogeneous system of three equations in four variables,

2 z 1 + z 2 + z 3 = 0 − z 1 + 3 z 2 + z 3 − z 4 = 0 3 z 1 + z 2 + 2 z 3 − 2 z 4 = 0

The coefficient matrix of this system row-reduces as

2 1 1 0 − 1 3 1 − 1 3 1 2 − 2 →  RREF 1 0 0 1 0 1  0 1 0 0 1 − 3

From this we can find a solution (we only need one), that is an element of K T ,

z = − 1 − 1 3 1

Now, we choose a vector x at random and set y = x + z ,

x = 2 3 4 − 2 y = x + z = 2 3 4 − 2 + − 1 − 1 3 1 = 1 2 7 − 1

and you can check that

T x = 1 1 1 3 2 1 = T y

A quicker solution is to take two elements of the kernel (in this case, scalar multiples of z ) which both get sent to 0 by T . Quicker yet, take 0 and z as x and y , which also both get sent to 0 by T .

C25 Contributed by  Robert Beezer Statement [ 1485]
To find the kernel, we require all x ∈ ℂ 3 such that T x = 0 . This condition is

2 x 1 − x 2 + 5 x 3 − 4 x 1 + 2 x 2 − 1 0 x 3 = 0 0

This leads to a homogeneous system of two linear equations in three variables, whose coefficient matrix row-reduces to

1 − 1 2 5 2 0 0 0

With two free variables Theorem BNS yields the basis for the null space

− 5 2 0 1 , 1 2 1 0

With n T ≠ 0 , K T ≠ 0 , so Theorem KILT says T is not injective.

C26 Contributed by  Chris Black Statement [ 1486]
By Theorem ILTD, if a linear transformation T : U → V is injective, then dim ( U ) ≤ dim ( V ) . In this case, T : ℂ 5 → ℂ 4 , and 5 = dim ℂ 5 > dim ℂ 4 = 4 . Thus, T cannot possibly be injective.

C27 Contributed by  Chris Black Statement [ 1486]
If T x y z = 0 , then 2 x + y + z x − y + 2 z x + 2 y − z = 0 . Thus, we have the system

2 x + y + z = 0 x − y + 2 z = 0 x + 2 y − z = 0

. Thus, we are looking for the nullspace of the matrix A T = 2 1 1 1 − 1 2 1 2 − 1 . Since A T row-reduces to 1 0 1 0 1  − 1 0 0 0 , the kernel of T is all vectors where x = − z and y = z . Thus, K T = − 1 1 1 .

C28 Contributed by  Chris Black Statement [ 1486]
Since T is given by matrix multiplication, K T = N A . We have

1 2 3 1 2 − 1 1 0 1 2 − 1 − 2 1 3 2 1 →  RREF 1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1

The nullspace of A is 0 , so the kernel of T is also trivial: K T = 0 .

C29 Contributed by  Chris Black Statement [ 1487]
Since T is given by matrix multiplication, K T = N A . We have

1 2 1 1 2 1 1 0 1 2 1 2 1 2 1 1 →  RREF 1 0 1 ∕ 3 0 0 1 1 ∕ 3 0 0 0 0 1 0 0 0 0

Thus, a basis for the nullspace of A is − 1 − 1 3 0 , and the kernel is K T = − 1 − 1 3 0 . Since the kernel is nontrivial, this linear transformation is not injective.

C30 Contributed by  Chris Black Statement [ 1487]
We can see without computing that T is not injective, since the degree of M 2 , 2 is larger than the degree of P 2 . However, that doesn’t address the question of the kernel of T . We need to find all matrices a b c d so that ( a + b ) + ( a + c ) x + ( a + d ) x 2 = 0 . This means a + b = 0 , a + c = 0 , and a + d = 0 , or equivalently, b = d = c = − a . Thus, the kernel is a one-dimensional subspace of M 2 , 2 spanned by 1 − 1 − 1 − 1 . Symbolically, we have K T = 1 − 1 − 1 − 1 .

C31 Contributed by  Chris Black Statement [ 1487]
We have

T e 1 = 2 0 1 T e 2 = 1 2 0 T e 3 = 0 1 2

Let’s put these vectors into a matrix and row reduce to test their linear independence.

2 1 0 0 2 1 1 0 2 →  RREF 1 0 0 0 1  0 0 0 1

so the set of vectors T e 1 , T e 1 , T e 1 is linearly independent.

C32 Contributed by  Chris Black Statement [ 1487]
We have T e 1 = 1 2 1 and T e 2 = 1 1 2 . Putting these into a matrix as columns and row-reducing, we have

1 1 2 1 1 2 →  RREF 1 0 0 1 0 0

Thus, the set of vectors T e 1 , T e 2 is linearly independent.

C33 Contributed by  Chris Black Statement [ 1487]
We have

T e 1 = 1 0 1 1 3 T e 2 = 3 1 2 0 1 T e 3 = 2 1 1 1 2

Let’s row reduce the matrix of T to test linear independence.

1 3 2 0 1 1 1 2 1 1 0 1 3 1 2 →  RREF 1 0 0 0 1  0 0 0 1 0 0 0 0 0 0

so the set of vectors T e 1 , T e 2 , T e 3 is linearly independent.

C40 Contributed by  Robert Beezer Statement [ 1488]
We choose x to be any vector we like. A particularly cocky choice would be to choose x = 0 , but we will instead choose

x = 2 − 1 − 1 4

Then R x = 9 + 9 x . Now compute the kernel of R , which by Theorem KILT we expect to be nontrivial. Setting R a b b c equal to the zero vector, 0 = 0 + 0 x , and equating coefficients leads to a homogeneous system of equations. Row-reducing the coefficient matrix of this system will allow us to determine the values of a , b and c that create elements of the null space of R ,

2 − 1 1 1 1 2 →  RREF 1 0 1 0 1  1

We only need a single element of the null space of this coefficient matrix, so we will not compute a precise description of the whole null space. Instead, choose the free variable c = 2 . Then

z = − 2 − 2 − 2 2

is the corresponding element of the kernel. We compute the desired y as

y = x + z = 2 − 1 − 1 4 + − 2 − 2 − 2 2 = 0 − 3 − 3 6

Then check that R y = 9 + 9 x .

T15 Contributed by  Robert Beezer Statement [ 1489]
We are asked to prove that K T is a subset of K S ∘ T . Employing Definition SSET, choose x ∈ K T . Then we know that T x = 0 . So

S ∘ T x = S T x   Definition LTC = S 0 x ∈ K T = 0   Theorem LTTZZ

This qualifies x for membership in K S ∘ T .

T20 Contributed by  Andy Zimmer Statement [ 1489]
This is an equality of sets, so we want to establish two subset conditions ( Definition SE).

First, show N A ⊆ K T . Choose x ∈ N A . Check to see if x ∈ K T ,

T x = A x  Definition of  T = 0   x ∈ N A

So by Definition KLT, x ∈ K T and thus N A ⊆ K T .

Now, show K T ⊆ N A . Choose x ∈ K T . Check to see if x ∈ N A ,

A x = T x  Definition of  T = 0   x ∈ K T

So by Definition NSM, x ∈ N A and thus K T ⊆ N A .

Section SLT  Surjective Linear Transformations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

The companion to an injection is a surjection. Surjective linear transformations are closely related to spanning sets and ranges. So as you read this section reflect back on Section ILT and note the parallels and the contrasts. In the next section, Section IVLT, we will combine the two properties.

As usual, we lead with a definition.

Definition  SLT
Surjective Linear Transformation
Suppose T : U → V is a linear transformation. Then T is surjective if for every v ∈ V there exists a u ∈ U so that T u = v . △

Given an arbitrary function, it is possible for there to be an element of the codomain that is not an output of the function (think about the function y = f ( x ) = x 2 and the codomain element y = − 3 ). For a surjective function, this never happens. If we choose any element of the codomain ( v ∈ V ) then there must be an input from the domain ( u ∈ U ) which will create the output when used to evaluate the linear transformation ( T u = v ). Some authors prefer the term onto where we use surjective, and we will sometimes refer to a surjective linear transformation as a surjection.

Parent topic: Chapter LT  Linear Transformations

Previous topic: Section ILT  Injective Linear Transformations

Next topic: Section IVLT  Invertible Linear Transformations

Subsection ESLT: Examples of Surjective Linear Transformations

It is perhaps most instructive to examine a linear transformation that is not surjective first.

Example  NSAQ
Not surjective, Archetype Q
Archetype Q is the linear transformation

T : ℂ 5 → ℂ 5 , T x 1 x 2 x 3 x 4 x 5 = − 2 x 1 + 3 x 2 + 3 x 3 − 6 x 4 + 3 x 5 − 1 6 x 1 + 9 x 2 + 1 2 x 3 − 2 8 x 4 + 2 8 x 5 − 1 9 x 1 + 7 x 2 + 1 4 x 3 − 3 2 x 4 + 3 7 x 5 − 2 1 x 1 + 9 x 2 + 1 5 x 3 − 3 5 x 4 + 3 9 x 5 − 9 x 1 + 5 x 2 + 7 x 3 − 1 6 x 4 + 1 6 x 5

We will demonstrate that

v = − 1 2 3 − 1 4

is an unobtainable element of the codomain. Suppose to the contrary that u is an element of the domain such that T u = v . Then

− 1 2 3 − 1 4 = v = T u = T u 1 u 2 u 3 u 4 u 5 = − 2 u 1 + 3 u 2 + 3 u 3 − 6 u 4 + 3 u 5 − 1 6 u 1 + 9 u 2 + 1 2 u 3 − 2 8 u 4 + 2 8 u 5 − 1 9 u 1 + 7 u 2 + 1 4 u 3 − 3 2 u 4 + 3 7 u 5 − 2 1 u 1 + 9 u 2 + 1 5 u 3 − 3 5 u 4 + 3 9 u 5 − 9 u 1 + 5 u 2 + 7 u 3 − 1 6 u 4 + 1 6 u 5 = − 2 3 3 − 6 3 − 1 6 9 1 2 − 2 8 2 8 − 1 9 7 1 4 − 3 2 3 7 − 2 1 9 1 5 − 3 5 3 9 − 9 5 7 − 1 6 1 6 u 1 u 2 u 3 u 4 u 5

Now we recognize the appropriate input vector u as a solution to a linear system of equations. Form the augmented matrix of the system, and row-reduce to

1 0 0 0 − 1 0 0 1 0 0 − 4 3 0 0 0 1 0 − 1 3 0 0 0 0 1  − 1 0 0 0 0 0 0 1

With a leading 1 in the last column, Theorem RCLS tells us the system is inconsistent. From the absence of any solutions we conclude that no such vector u exists, and by Definition SLT, T is not surjective.

Again, do not concern yourself with how v was selected, as this will be explained shortly. However, do understand why this vector provides enough evidence to conclude that T is not surjective. ⊠

To show that a linear transformation is not surjective, it is enough to find a single element of the codomain that is never created by any input, as in Example NSAQ. However, to show that a linear transformation is surjective we must establish that every element of the codomain occurs as an output of the linear transformation for some appropriate input.

Example  SAR
Surjective, Archetype R
Archetype R is the linear transformation

T : ℂ 5 → ℂ 5 , T x 1 x 2 x 3 x 4 x 5 = − 6 5 x 1 + 1 2 8 x 2 + 1 0 x 3 − 2 6 2 x 4 + 4 0 x 5 3 6 x 1 − 7 3 x 2 − x 3 + 1 5 1 x 4 − 1 6 x 5 − 4 4 x 1 + 8 8 x 2 + 5 x 3 − 1 8 0 x 4 + 2 4 x 5 3 4 x 1 − 6 8 x 2 − 3 x 3 + 1 4 0 x 4 − 1 8 x 5 1 2 x 1 − 2 4 x 2 − x 3 + 4 9 x 4 − 5 x 5

To establish that R is surjective we must begin with a totally arbitrary element of the codomain, v and somehow find an input vector u such that T u = v . We desire,

T u = v − 6 5 u 1 + 1 2 8 u 2 + 1 0 u 3 − 2 6 2 u 4 + 4 0 u 5 3 6 u 1 − 7 3 u 2 − u 3 + 1 5 1 u 4 − 1 6 u 5 − 4 4 u 1 + 8 8 u 2 + 5 u 3 − 1 8 0 u 4 + 2 4 u 5 3 4 u 1 − 6 8 u 2 − 3 u 3 + 1 4 0 u 4 − 1 8 u 5 1 2 u 1 − 2 4 u 2 − u 3 + 4 9 u 4 − 5 u 5 = v 1 v 2 v 3 v 4 v 5 − 6 5 1 2 8 1 0 − 2 6 2 4 0 3 6 − 7 3 − 1 1 5 1 − 1 6 − 4 4 8 8 5 − 1 8 0 2 4 3 4 − 6 8 − 3 1 4 0 − 1 8 1 2 − 2 4 − 1 4 9 − 5 u 1 u 2 u 3 u 4 u 5 = v 1 v 2 v 3 v 4 v 5

We recognize this equation as a system of equations in the variables u i , but our vector of constants contains symbols. In general, we would have to row-reduce the augmented matrix by hand, due to the symbolic final column. However, in this particular example, the 5 × 5 coefficient matrix is nonsingular and so has an inverse ( Theorem NI, Definition MI).

− 6 5 1 2 8 1 0 − 2 6 2 4 0 3 6 − 7 3 − 1 1 5 1 − 1 6 − 4 4 8 8 5 − 1 8 0 2 4 3 4 − 6 8 − 3 1 4 0 − 1 8 1 2 − 2 4 − 1 4 9 − 5 − 1 = − 4 7 9 2 1 − 1 8 1 − 1 4 2 7 − 5 5 7 2 2 2 1 2 1 1 − 3 2 6 4 − 1 − 1 2 6 − 1 2 2 5 − 5 0 3 2 1 9 9 2 9 9 − 1 8 1 2 7 1 2 4

so we find that

u 1 u 2 u 3 u 4 u 5 = − 4 7 9 2 1 − 1 8 1 − 1 4 2 7 − 5 5 7 2 2 2 1 2 1 1 − 3 2 6 4 − 1 − 1 2 6 − 1 2 2 5 − 5 0 3 2 1 9 9 2 9 9 − 1 8 1 2 7 1 2 4 v 1 v 2 v 3 v 4 v 5 = − 4 7 v 1 + 9 2 v 2 + v 3 − 1 8 1 v 4 − 1 4 v 5 2 7 v 1 − 5 5 v 2 + 7 2 v 3 + 2 2 1 2 v 4 + 1 1 v 5 − 3 2 v 1 + 6 4 v 2 − v 3 − 1 2 6 v 4 − 1 2 v 5 2 5 v 1 − 5 0 v 2 + 3 2 v 3 + 1 9 9 2 v 4 + 9 v 5 9 v 1 − 1 8 v 2 + 1 2 v 3 + 7 1 2 v 4 + 4 v 5

This establishes that if we are given any output vector v , we can use its components in this final expression to formulate a vector u such that T u = v . So by Definition SLT we now know that T is surjective. You might try to verify this condition in its full generality (i.e. evaluate T with this final expression and see if you get v as the result), or test it more specifically for some numerical vector v (see Exercise SLT.C20). ⊠

Let’s now examine a surjective linear transformation between abstract vector spaces.

Example  SAV
Surjective, Archetype V
Archetype V is defined by

T : P 3 → M 2 2 , T a + b x + c x 2 + d x 3 = a + b a − 2 c d b − d

To establish that the linear transformation is surjective, begin by choosing an arbitrary output. In this example, we need to choose an arbitrary 2 × 2 matrix, say

v = x y z w

and we would like to find an input polynomial

u = a + b x + c x 2 + d x 3

so that T u = v . So we have,

x y z w = v = T u = T a + b x + c x 2 + d x 3 = a + b a − 2 c d b − d

Matrix equality leads us to the system of four equations in the four unknowns, x , y , z , w ,

a + b = x a − 2 c = y d = z b − d = w

which can be rewritten as a matrix equation,

1 1 0 0 1 0 − 2 0 0 0 0 1 0 1 0 − 1 a b c d = x y z w

The coefficient matrix is nonsingular, hence it has an inverse,

1 1 0 0 1 0 − 2 0 0 0 0 1 0 1 0 − 1 − 1 = 1 0 − 1 − 1 0 0 1 1 1 2 − 1 2 − 1 2 − 1 2 0 0 1 0

so we have

a b c d = 1 0 − 1 − 1 0 0 1 1 1 2 − 1 2 − 1 2 − 1 2 0 0 1 0 x y z w = x − z − w z + w 1 2 ( x − y − z − w ) z

So the input polynomial u = ( x − z − w ) + ( z + w ) x + 1 2 ( x − y − z − w ) x 2 + z x 3 will yield the output matrix v , no matter what form v takes. This means by Definition SLT that T is surjective. All the same, let’s do a concrete demonstration and evaluate T with u ,

T u = T ( x − z − w ) + ( z + w ) x + 1 2 ( x − y − z − w ) x 2 + z x 3 = ( x − z − w ) + ( z + w ) ( x − z − w ) − 2 ( 1 2 ( x − y − z − w ) ) z ( z + w ) − z = x y z w = v

⊠

Subsection RLT: Range of a Linear Transformation

For a linear transformation T : U → V , the range is a subset of the codomain V . Informally, it is the set of all outputs that the transformation creates when fed every possible input from the domain. It will have some natural connections with the column space of a matrix, so we will keep the same notation, and if you think about your objects, then there should be little confusion. Here’s the careful definition.

Definition  RLT
Range of a Linear Transformation
Suppose T : U → V is a linear transformation. Then the range of T is the set

ℛ T = T u u ∈ U

(This definition contains Notation RLT.) △

Example  RAO
Range, Archetype O
Archetype O is the linear transformation

T : ℂ 3 → ℂ 5 , T x 1 x 2 x 3 = − x 1 + x 2 − 3 x 3 − x 1 + 2 x 2 − 4 x 3 x 1 + x 2 + x 3 2 x 1 + 3 x 2 + x 3 x 1 + 2 x 3

To determine the elements of ℂ 5 in ℛ T , find those vectors v such that T u = v for some u ∈ ℂ 3 ,

v = T u = − u 1 + u 2 − 3 u 3 − u 1 + 2 u 2 − 4 u 3 u 1 + u 2 + u 3 2 u 1 + 3 u 2 + u 3 u 1 + 2 u 3 = − u 1 − u 1 u 1 2 u 1 u 1 + u 2 2 u 2 u 2 3 u 2 0 + − 3 u 3 − 4 u 3 u 3 u 3 2 u 3 = u 1 − 1 − 1 1 2 1 + u 2 1 2 1 3 0 + u 3 − 3 − 4 1 1 2

This says that every output of T ( v ) can be written as a linear combination of the three vectors

− 1 − 1 1 2 1 1 2 1 3 0 − 3 − 4 1 1 2

using the scalars u 1 , u 2 , u 3 . Furthermore, since u can be any element of ℂ 3 , every such linear combination is an output. This means that

ℛ T = − 1 − 1 1 2 1 , 1 2 1 3 0 , − 3 − 4 1 1 2

The three vectors in this spanning set for ℛ T form a linearly dependent set (check this!). So we can find a more economical presentation by any of the various methods from Section CRS and Section FS. We will place the vectors into a matrix as rows, row-reduce, toss out zero rows and appeal to Theorem BRS, so we can describe the range of T with a basis,

ℛ T = 1 0 − 3 − 7 − 2 , 0 1 2 5 1

⊠

We know that the span of a set of vectors is always a subspace (Theorem SSS), so the range computed in Example RAO is also a subspace. This is no accident, the range of a linear transformation is always a subspace.

Theorem  RLTS
Range of a Linear Transformation is a Subspace
Suppose that T : U → V is a linear transformation. Then the range of T , ℛ T , is a subspace of V . □

Proof   We can apply the three-part test of Theorem TSS. First, 0 U ∈ U and T 0 U = 0 V by Theorem LTTZZ, so 0 V ∈ ℛ T and we know that the range is non-empty.

Suppose we assume that x , y ∈ ℛ T . Is x + y ∈ ℛ T ? If x , y ∈ ℛ T then we know there are vectors w , z ∈ U such that T w = x and T z = y . Because U is a vector space, additive closure (Property AC) implies that w + z ∈ U . Then

T w + z = T w + T z   Definition LT = x + y  Definition of  w  and  z

So we have found an input, w + z , which when fed into T creates x + y as an output. This qualifies x + y for membership in ℛ T . So we have additive closure.

Suppose we assume that α ∈ ℂ and x ∈ ℛ T . Is α x ∈ ℛ T ? If x ∈ ℛ T , then there is a vector w ∈ U such that T w = x . Because U is a vector space, scalar closure implies that α w ∈ U . Then

T α w = α T w   Definition LT = α x  Definition of  w

So we have found an input ( α w ) which when fed into T creates α x as an output. This qualifies α x for membership in ℛ T . So we have scalar closure and Theorem TSS tells us that ℛ T is a subspace of V . ■

Let’s compute another range, now that we know in advance that it will be a subspace.

Example  FRAN
Full range, Archetype N
Archetype N is the linear transformation

T : ℂ 5 → ℂ 3 , T x 1 x 2 x 3 x 4 x 5 = 2 x 1 + x 2 + 3 x 3 − 4 x 4 + 5 x 5 x 1 − 2 x 2 + 3 x 3 − 9 x 4 + 3 x 5 3 x 1 + 4 x 3 − 6 x 4 + 5 x 5

To determine the elements of ℂ 3 in ℛ T , find those vectors v such that T u = v for some u ∈ ℂ 5 ,

v = T u = 2 u 1 + u 2 + 3 u 3 − 4 u 4 + 5 u 5 u 1 − 2 u 2 + 3 u 3 − 9 u 4 + 3 u 5 3 u 1 + 4 u 3 − 6 u 4 + 5 u 5 = 2 u 1 u 1 3 u 1 + u 2 − 2 u 2 0 + 3 u 3 3 u 3 4 u 3 + − 4 u 4 − 9 u 4 − 6 u 4 + 5 u 5 3 u 5 5 u 5 = u 1 2 1 3 + u 2 1 − 2 0 + u 3 3 3 4 + u 4 − 4 − 9 − 6 + u 5 5 3 5

This says that every output of T ( v ) can be written as a linear combination of the five vectors

2 1 3 1 − 2 0 3 3 4 − 4 − 9 − 6 5 3 5

using the scalars u 1 , u 2 , u 3 , u 4 , u 5 . Furthermore, since u can be any element of ℂ 5 , every such linear combination is an output. This means that

ℛ T = 2 1 3 , 1 − 2 0 , 3 3 4 , − 4 − 9 − 6 , 5 3 5

The five vectors in this spanning set for ℛ T form a linearly dependent set (Theorem MVSLD). So we can find a more economical presentation by any of the various methods from Section CRS and Section FS. We will place the vectors into a matrix as rows, row-reduce, toss out zero rows and appeal to Theorem BRS, so we can describe the range of T with a (nice) basis,

ℛ T = 1 0 0 , 0 1 0 , 0 0 1 = ℂ 3

⊠

In contrast to injective linear transformations having small (trivial) kernels ( Theorem KILT), surjective linear transformations have large ranges, as indicated in the next theorem.

Theorem  RSLT
Range of a Surjective Linear Transformation
Suppose that T : U → V is a linear transformation. Then T is surjective if and only if the range of T equals the codomain, ℛ T = V . □

Proof   ( ⇒ ) By Definition RLT, we know that ℛ T ⊆ V . To establish the reverse inclusion, assume v ∈ V . Then since T is surjective ( Definition SLT), there exists a vector u ∈ U so that T u = v . However, the existence of u gains v membership in ℛ T , so V ⊆ ℛ T . Thus, ℛ T = V .

( ⇐ ) To establish that T is surjective, choose v ∈ V . Since we are assuming that ℛ T = V , v ∈ ℛ T . This says there is a vector u ∈ U so that T u = v , i.e.  T is surjective. ■

Example  NSAQR
Not surjective, Archetype Q, revisited
We are now in a position to revisit our first example in this section, Example NSAQ. In that example, we showed that Archetype Q is not surjective by constructing a vector in the codomain where no element of the domain could be used to evaluate the linear transformation to create the output, thus violating Definition SLT. Just where did this vector come from?

The short answer is that the vector

v = − 1 2 3 − 1 4

was constructed to lie outside of the range of T . How was this accomplished? First, the range of T is given by

ℛ T = 1 0 0 0 1 , 0 1 0 0 − 1 , 0 0 1 0 − 1 , 0 0 0 1 2

Suppose an element of the range v ∗ has its first 4 components equal to − 1 , 2 , 3 , − 1 , in that order. Then to be an element of ℛ T , we would have

v ∗ = ( − 1 ) 1 0 0 0 1 + ( 2 ) 0 1 0 0 − 1 + ( 3 ) 0 0 1 0 − 1 + ( − 1 ) 0 0 0 1 2 = − 1 2 3 − 1 − 8

So the only vector in the range with these first four components specified, must have − 8 in the fifth component. To set the fifth component to any other value (say, 4) will result in a vector ( v in Example NSAQ) outside of the range. Any attempt to find an input for T that will produce v as an output will be doomed to failure.

Whenever the range of a linear transformation is not the whole codomain, we can employ this device and conclude that the linear transformation is not surjective. This is another way of viewing Theorem RSLT. For a surjective linear transformation, the range is all of the codomain and there is no choice for a vector v that lies in V , yet not in the range. For every one of the archetypes that is not surjective, there is an example presented of exactly this form. ⊠

Example  NSAO
Not surjective, Archetype O
In Example RAO the range of Archetype O was determined to be

ℛ T = 1 0 − 3 − 7 − 2 , 0 1 2 5 1

a subspace of dimension 2 in ℂ 5 . Since ℛ T ≠ ℂ 5 , Theorem RSLT says T is not surjective. ⊠

Example  SAN
Surjective, Archetype N
The range of Archetype N was computed in Example FRAN to be

ℛ T = 1 0 0 , 0 1 0 , 0 0 1

Since the basis for this subspace is the set of standard unit vectors for ℂ 3 (Theorem SUVB), we have ℛ T = ℂ 3 and by Theorem RSLT, T is surjective. ⊠

Subsection SSSLT: Spanning Sets and Surjective Linear Transformations

Just as injective linear transformations are allied with linear independence ( Theorem ILTLI, Theorem ILTB), surjective linear transformations are allied with spanning sets.

Theorem  SSRLT
Spanning Set for Range of a Linear Transformation
Suppose that T : U → V is a linear transformation and S = u 1 , u 2 , u 3 , … , u t spans U . Then

R = T u 1 , T u 2 , T u 3 , … , T u t

spans ℛ T . □

Proof   We need to establish that ℛ T = R , a set equality. First we establish that ℛ T ⊆ R . To this end, choose v ∈ ℛ T . Then there exists a vector u ∈ U , such that T u = v ( Definition RLT). Because S spans U there are scalars, a 1 , a 2 , a 3 , … , a t , such that

u = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a t u t

Then

v = T u   Definition RLT = T a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a t u t   Definition TSVS = a 1 T u 1 + a 2 T u 2 + a 3 T u 3 + … + a t T u t   Theorem LTLC

which establishes that v ∈ R (Definition SS). So ℛ T ⊆ R .

To establish the opposite inclusion, choose an element of the span of R , say v ∈ R . Then there are scalars b 1 , b 2 , b 3 , … , b t so that

v = b 1 T u 1 + b 2 T u 2 + b 3 T u 3 + ⋯ + b t T u t   Definition SS = T b 1 u 1 + b 2 u 2 + b 3 u 3 + ⋯ + b t u t   Theorem LTLC

This demonstrates that v is an output of the linear transformation T , so v ∈ ℛ T . Therefore R ⊆ ℛ T , so we have the set equality ℛ T = R ( Definition SE). In other words, R spans ℛ T (Definition TSVS). ■

Theorem SSRLT provides an easy way to begin the construction of a basis for the range of a linear transformation, since the construction of a spanning set requires simply evaluating the linear transformation on a spanning set of the domain. In practice the best choice for a spanning set of the domain would be as small as possible, in other words, a basis. The resulting spanning set for the codomain may not be linearly independent, so to find a basis for the range might require tossing out redundant vectors from the spanning set. Here’s an example.

Example  BRLT
A basis for the range of a linear transformation
Define the linear transformation T : M 2 2 → P 2 by

T a b c d = a + 2 b + 8 c + d + − 3 a + 2 b + 5 d x + a + b + 5 c x 2

A convenient spanning set for M 2 2 is the basis

S = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

So by Theorem SSRLT, a spanning set for ℛ T is

R = T 1 0 0 0 , T 0 1 0 0 , T 0 0 1 0 , T 0 0 0 1 = 1 − 3 x + x 2 , 2 + 2 x + x 2 , 8 + 5 x 2 , 1 + 5 x

The set R is not linearly independent, so if we desire a basis for ℛ T , we need to eliminate some redundant vectors. Two particular relations of linear dependence on R are

( − 2 ) ( 1 − 3 x + x 2 ) + ( − 3 ) ( 2 + 2 x + x 2 ) + ( 8 + 5 x 2 ) = 0 + 0 x + 0 x 2 = 0 ( 1 − 3 x + x 2 ) + ( − 1 ) ( 2 + 2 x + x 2 ) + ( 1 + 5 x ) = 0 + 0 x + 0 x 2 = 0

These, individually, allow us to remove 8 + 5 x 2 and 1 + 5 x from R with out destroying the property that R spans ℛ T . The two remaining vectors are linearly independent (check this!), so we can write

ℛ T = 1 − 3 x + x 2 , 2 + 2 x + x 2

and see that dim ℛ T = 2 . ⊠

Elements of the range are precisely those elements of the codomain with non-empty preimages.

Theorem  RPI
Range and Pre-Image
Suppose that T : U → V is a linear transformation. Then

v ∈ ℛ T   if and only if  T − 1 v ≠ ∅

□

Proof   ( ⇒ ) If v ∈ ℛ T , then there is a vector u ∈ U such that T u = v . This qualifies u for membership in T − 1 v , and thus the preimage of v is not empty.

( ⇐ ) Suppose the preimage of v is not empty, so we can choose a vector u ∈ U such that T u = v . Then v ∈ ℛ T . ■

Theorem  SLTB
Surjective Linear Transformations and Bases
Suppose that T : U → V is a linear transformation and B = u 1 , u 2 , u 3 , … , u m is a basis of U . Then T is surjective if and only if C = T u 1 , T u 2 , T u 3 , … , T u m is a spanning set for V . □

Proof   ( ⇒ ) Assume T is surjective. Since B is a basis, we know B is a spanning set of U (Definition B). Then Theorem SSRLT says that C spans ℛ T . But the hypothesis that T is surjective means V = ℛ T ( Theorem RSLT), so C spans V .

( ⇐ ) Assume that C spans V . To establish that T is surjective, we will show that every element of V is an output of T for some input ( Definition SLT). Suppose that v ∈ V . As an element of V , we can write v as a linear combination of the spanning set C . So there are are scalars, b 1 , b 2 , b 3 , … , b m , such that

v = b 1 T u 1 + b 2 T u 2 + b 3 T u 3 + ⋯ + b m T u m

Now define the vector u ∈ U by

u = b 1 u 1 + b 2 u 2 + b 3 u 3 + ⋯ + b m u m

Then

T u = T b 1 u 1 + b 2 u 2 + b 3 u 3 + ⋯ + b m u m = b 1 T u 1 + b 2 T u 2 + b 3 T u 3 + ⋯ + b m T u m   Theorem LTLC = v

So, given any choice of a vector v ∈ V , we can design an input u ∈ U to produce v as an output of T . Thus, by Definition SLT, T is surjective. ■

Subsection SLTD: Surjective Linear Transformations and Dimension

Theorem  SLTD
Surjective Linear Transformations and Dimension
Suppose that T : U → V is a surjective linear transformation. Then dim U ≥ dim V . □

Proof   Suppose to the contrary that m = dim U < dim V = t . Let B be a basis of U , which will then contain m vectors. Apply T to each element of B to form a set C that is a subset of V . By Theorem SLTB, C is spanning set of V with m or fewer vectors. So we have a set of m or fewer vectors that span V , a vector space of dimension t , with m < t . However, this contradicts Theorem G, so our assumption is false and dim U ≥ dim V . ■

Example  NSDAT
Not surjective by dimension, Archetype T
The linear transformation in Archetype T is

T : P 4 → P 5 , T p ( x ) = ( x − 2 ) p ( x )

Since dim P 4 = 5 < 6 = dim P 5 , T cannot be surjective for then it would violate Theorem SLTD. ⊠

Notice that the previous example made no use of the actual formula defining the function. Merely a comparison of the dimensions of the domain and codomain are enough to conclude that the linear transformation is not surjective. Archetype O and Archetype P are two more examples of linear transformations that have “small” domains and “big” codomains, resulting in an inability to create all possible outputs and thus they are non-surjective linear transformations.

Subsection CSLT: Composition of Surjective Linear Transformations

In Subsection LT.NLTFO we saw how to combine linear transformations to build new linear transformations, specifically, how to build the composition of two linear transformations (Definition LTC). It will be useful later to know that the composition of surjective linear transformations is again surjective, so we prove that here.

Theorem  CSLTS
Composition of Surjective Linear Transformations is Surjective
Suppose that T : U → V and S : V → W are surjective linear transformations. Then ( S ∘ T ) : U → W is a surjective linear transformation. □

Proof   That the composition is a linear transformation was established in Theorem CLTLT, so we need only establish that the composition is surjective. Applying Definition SLT, choose w ∈ W .

Because S is surjective, there must be a vector v ∈ V , such that S v = w . With the existence of v established, that T is surjective guarantees a vector u ∈ U such that T u = v . Now,

S ∘ T u = S T u   Definition LTC = S v  Definition of  u = w  Definition of  v

This establishes that any element of the codomain ( w ) can be created by evaluating S ∘ T with the right input ( u ). Thus, by Definition SLT, S ∘ T is surjective. ■

Subsection READ: Reading Questions

Subsection EXC: Exercises

C10 Each archetype below is a linear transformation. Compute the range for each.
Archetype M
Archetype N
Archetype O
Archetype P
Archetype Q
Archetype R
Archetype S
Archetype T
Archetype U
Archetype V
Archetype W
Archetype X

Contributed by  Robert Beezer

C20 Example SAR concludes with an expression for a vector u ∈ ℂ 5 that we believe will create the vector v ∈ ℂ 5 when used to evaluate T . That is, T u = v . Verify this assertion by actually evaluating T with u . If you don’t have the patience to push around all these symbols, try choosing a numerical instance of v , compute u , and then compute T u , which should result in v .  
Contributed by  Robert Beezer

C22 The linear transformation S : ℂ 4 → ℂ 3 is not surjective. Find an output w ∈ ℂ 3 that has an empty pre-image (that is S − 1 w = ∅ .)

S x 1 x 2 x 3 x 4 = 2 x 1 + x 2 + 3 x 3 − 4 x 4 x 1 + 3 x 2 + 4 x 3 + 3 x 4 − x 1 + 2 x 2 + x 3 + 7 x 4

Contributed by  Robert Beezer Solution [ 1547]

C23 Determine whether or not the following linear transformation T : ℂ 5 → P 3 is surjective:

T a b c d e = a + ( b + c ) x + ( c + d ) x 2 + ( d + e ) x 3

Contributed by  Chris Black Solution [ 1549]

C24 Determine whether or not the linear transformation T : P 3 → ℂ 5 below is surjective:

T a + b x + c x 2 + d x 3 = a + b b + c c + d a + c b + d .

Contributed by  Chris Black Solution [ 1551]

C25 Define the linear transformation

T : ℂ 3 → ℂ 2 , T x 1 x 2 x 3 = 2 x 1 − x 2 + 5 x 3 − 4 x 1 + 2 x 2 − 1 0 x 3

Find a basis for the range of T , ℛ T . Is T surjective?  
Contributed by  Robert Beezer Solution [ 1551]

C26 Let T : ℂ 3 → ℂ 3 be given by T a b c = a + b + 2 c 2 c a + b + c . Find a basis of ℛ T . Is T surjective?  
Contributed by  Chris Black Solution [ 1552]

C27 Let T : ℂ 3 → ℂ 4 be given by T a b c = a + b − c a − b + c − a + b + c a + b + c . Find a basis of ℛ T . Is T surjective?  
Contributed by  Chris Black Solution [ 1553]

C28 Let T : ℂ 4 → M 2 , 2 be given by T a b c d = a + b a + b + c a + b + c a + d . Find a basis of ℛ T . Is T surjective?  
Contributed by  Chris Black Solution [ 1554]

C29 Let T : P 2 → P 4 be given by T p ( x ) = x 2 p ( x ) . Find a basis of ℛ T . Is T surjective?  
Contributed by  Chris Black Solution [ 1555]

C30 Let T : P 4 → P 3 be given by T p ( x ) = p ′ ( x ) , where p ′ ( x ) is the derivative. Find a basis of ℛ T . Is T surjective?  
Contributed by  Chris Black Solution [ 1555]

C40 Show that the linear transformation T is not surjective by finding an element of the codomain, v , such that there is no vector u with T u = v .

T : ℂ 3 → ℂ 3 , T a b c = 2 a + 3 b − c 2 b − 2 c a − b + 2 c

Contributed by  Robert Beezer Solution [ 1556]

M60 Suppose U and V are vector spaces. Define the function Z : U → V by T u = 0 V for every u ∈ U . Then by Exercise LT.M60, Z is a linear transformation. Formulate a condition on V that is equivalent to Z being an surjective linear transformation. In other words, fill in the blank to complete the following statement (and then give a proof): Z is surjective if and only if V is __________________. (See Exercise ILT.M60, Exercise IVLT.M60.)  
Contributed by  Robert Beezer

T15 Suppose that that T : U → V and S : V → W are linear transformations. Prove the following relationship between ranges.

ℛ S ∘ T ⊆ ℛ S

Contributed by  Robert Beezer Solution [ 1557]

T20 Suppose that A is an m × n matrix. Define the linear transformation T by

T : ℂ n → ℂ m , T x = A x

Prove that the range of T equals the column space of A , ℛ T = C A .  
Contributed by  Andy Zimmer Solution [ 1558]

Subsection SOL: Solutions

C22 Contributed by  Robert Beezer Statement [ 1541]
To find an element of ℂ 3 with an empty pre-image, we will compute the range of the linear transformation ℛ S and then find an element outside of this set.

By Theorem SSRLT we can evaluate S with the elements of a spanning set of the domain and create a spanning set for the range.

S 1 0 0 0 = 2 1 − 1 S 0 1 0 0 = 1 3 2 S 0 0 1 0 = 3 4 1 S 0 0 0 1 = − 4 3 7

So

ℛ S = 2 1 − 1 , 1 3 2 , 3 4 1 , − 4 3 7

This spanning set is obviously linearly dependent, so we can reduce it to a basis for ℛ S using Theorem BRS, where the elements of the spanning set are placed as the rows of a matrix. The result is that

ℛ S = 1 0 − 1 , 0 1 1

Therefore, the unique vector in ℛ S with a first slot equal to 6 and a second slot equal to 15 will be the linear combination

6 1 0 − 1 + 1 5 0 1 1 = 6 1 5 9

So, any vector with first two components equal to 6 and 15, but with a third component different from 9, such as

w = 6 1 5 − 6 3

will not be an element of the range of S and will therefore have an empty pre-image. Another strategy on this problem is to guess. Almost any vector will lie outside the range of T , you have to be unlucky to randomly choose an element of the range. This is because the codomain has dimension 3, while the range is “much smaller” at a dimension of 2. You still need to check that your guess lies outside of the range, which generally will involve solving a system of equations that turns out to be inconsistent.

C23 Contributed by  Chris Black Statement [ 1542]
The linear transformation T is surjective if for any p ( x ) = α + β x + γ x 2 + δ x 3 , there is a vector u = a b c d e in ℂ 5 so that T u = p ( x ) . We need to be able to solve the system

a = α b + c = β c + d = γ d + e = δ

This system has an infinite number of solutions, one of which is a = α , b = β , c = 0 , d = γ and e = δ − γ , so that

T α β 0 γ δ − γ = α + ( β + 0 ) x + ( 0 + γ ) x 2 + ( γ + ( δ − γ ) ) x 3 = α + β x + γ x 2 + δ x 3 = p ( x ) .

Thus, T is surjective, since for every vector v ∈ P 3 , there exists a vector u ∈ ℂ 5 so that T u = v .

C24 Contributed by  Chris Black Statement [ 1542]
According to Theorem SLTD, if a linear transformation T : U → V is surjective, then dim U ≥ dim V . In this example, U = P 3 has dimension 4, and V = ℂ 5 has dimension 5, so T cannot be surjective. (There is no way T can “expand” the domain P 3 to fill the codomain ℂ 5 .)

C25 Contributed by  Robert Beezer Statement [ 1543]
To find the range of T , apply T to the elements of a spanning set for ℂ 3 as suggested in Theorem SSRLT. We will use the standard basis vectors (Theorem SUVB).

ℛ T = T e 1 , T e 2 , T e 3 = 2 − 4 , − 1 2 , 5 − 1 0

Each of these vectors is a scalar multiple of the others, so we can toss two of them in reducing the spanning set to a linearly independent set (or be more careful and apply Theorem BCS on a matrix with these three vectors as columns). The result is the basis of the range,

1 − 2

With r T ≠ 2 , ℛ T ≠ ℂ 2 , so Theorem RSLT says T is not surjective.

C26 Contributed by  Chris Black Statement [ 1543]
The range of T is

ℛ T = a + b + 2 c 2 c a + b + c a , b , c ∈ ℂ = a 1 0 1 + b 1 0 1 + c 2 2 1 a , b , c ∈ ℂ = 1 0 1 , 2 2 1

Since the vectors 1 0 1 and 2 2 1 are linearly independent (why?), a basis of ℛ T is 1 0 1 , 2 2 1 . Since the dimension of the range is 2 and the dimension of the codomain is 3, T is not surjective.

C27 Contributed by  Chris Black Statement [ 1544]
The range of T is

ℛ T = a + b − c a − b + c − a + b + c a + b + c a , b , c ∈ ℂ = a 1 1 − 1 1 + b 1 − 1 1 1 + c − 1 1 1 1 a , b , c ∈ ℂ = 1 1 − 1 1 , 1 − 1 1 1 , − 1 1 1 1

By row reduction (not shown), we can see that the set

1 1 − 1 1 , 1 − 1 1 1 , − 1 1 1 1

are linearly independent, so is a basis of ℛ T . Since the dimension of the range is 3 and the dimension of the codomain is 4, T is not surjective. (We should have anticipated that T was not surjective since the dimension of the domain is smaller than the dimension of the codomain.)

C28 Contributed by  Chris Black Statement [ 1544]
The range of T is

ℛ T = a + b a + b + c a + b + c a + d a , b , c , d ∈ ℂ = a 1 1 1 1 + b 1 1 1 0 + c 0 1 1 0 + d 0 0 0 1 a , b , c , d ∈ ℂ = 1 1 1 1 , 1 1 1 0 , 0 1 1 0 , 0 0 0 1 = 1 1 1 0 , 0 1 1 0 , 0 0 0 1 .

Can you explain the last equality above?

These three matrices are linearly independent, so a basis of ℛ T is 1 1 1 0 , 0 1 1 0 , 0 0 0 1 . Thus, T is not surjective, since the range has dimension 3 which is shy of dim M 2 , 2 = 4 . (Notice that the range is actually the subspace of symmetric 2 × 2 matrices in M 2 , 2 .)

C29 Contributed by  Chris Black Statement [ 1544]
If we transform the basis of P 2 , then Theorem SSRLT guarantees we will have a spanning set of ℛ T . A basis of P 2 is 1 , x , x 2 . If we transform the elements of this set, we get the set x 2 , x 3 , x 4 which is a spanning set for ℛ T . These three vectors are linearly independent, so x 2 , x 3 , x 4 is a basis of ℛ T .

C30 Contributed by  Chris Black Statement [ 1544]
If we transform the basis of P 4 , then Theorem SSRLT guarantees we will have a spanning set of ℛ T . A basis of P 4 is 1 , x , x 2 , x 3 , x 4 . If we transform the elements of this set, we get the set 0 , 1 , 2 x , 3 x 2 , 4 x 3 which is a spanning set for ℛ T . Reducing this to a linearly independent set, we find that { 1 , 2 x , 3 x 2 , 4 x 3 } is a basis of ℛ T . Since ℛ T and P 3 both have dimension 4, T is surjective.

C40 Contributed by  Robert Beezer Statement [ 1544]
We wish to find an output vector v that has no associated input. This is the same as requiring that there is no solution to the equality

v = T a b c = 2 a + 3 b − c 2 b − 2 c a − b + 2 c = a 2 0 1 + b 3 2 − 1 + c − 1 − 2 2

In other words, we would like to find an element of ℂ 3 not in the set

Y = 2 0 1 , 3 2 − 1 , − 1 − 2 2

If we make these vectors the rows of a matrix, and row-reduce, Theorem BRS provides an alternate description of Y ,

Y = 2 0 1 , 0 4 − 5

If we add these vectors together, and then change the third component of the result, we will create a vector that lies outside of Y , say v = 2 4 9 .

T15 Contributed by  Robert Beezer Statement [ 1545]
This question asks us to establish that one set ( ℛ S ∘ T ) is a subset of another ( ℛ S ). Choose an element in the “smaller” set, say w ∈ ℛ S ∘ T . Then we know that there is a vector u ∈ U such that

w = S ∘ T u = S T u

Now define v = T u , so that then

S v = S T u = w

This statement is sufficient to show that w ∈ ℛ S , so w is an element of the “larger” set, and ℛ S ∘ T ⊆ ℛ S .

T20 Contributed by  Andy Zimmer Statement [ 1546]
This is an equality of sets, so we want to establish two subset conditions ( Definition SE).

First, show C A ⊆ ℛ T . Choose y ∈ C A . Then by Definition CSM and Definition MVP there is a vector x ∈ ℂ n such that A x = y . Then

T x = A x  Definition of  T = y

This statement qualifies y as a member of ℛ T ( Definition RLT), so C A ⊆ ℛ T .

Now, show ℛ T ⊆ C A . Choose y ∈ ℛ T . Then by Definition RLT, there is a vector x in ℂ n such that T x = y . Then

A x = T x  Definition of  T = y

So by Definition CSM and Definition MVP, y qualifies for membership in C A and so ℛ T ⊆ C A .

Section IVLT  Invertible Linear Transformations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we will conclude our introduction to linear transformations by bringing together the twin properties of injectivity and surjectivity and consider linear transformations with both of these properties.

Parent topic: Chapter LT  Linear Transformations

Previous topic: Section SLT  Surjective Linear Transformations

Next topic: Chapter R  Representations

Subsection IVLT: Invertible Linear Transformations

One preliminary definition, and then we will have our main definition for this section.

Definition  IDLT
Identity Linear Transformation
The identity linear transformation on the vector space W is defined as

I W : W → W , I W w = w

△

Informally, I W is the “do-nothing” function. You should check that I W is really a linear transformation, as claimed, and then compute its kernel and range to see that it is both injective and surjective. All of these facts should be straightforward to verify ( Exercise IVLT.T05). With this in hand we can make our main definition.

Definition  IVLT
Invertible Linear Transformations
Suppose that T : U → V is a linear transformation. If there is a function S : V → U such that

S ∘ T = I U T ∘ S = I V

then T is invertible. In this case, we call S the inverse of T and write S = T − 1 . △

Informally, a linear transformation T is invertible if there is a companion linear transformation, S , which “undoes” the action of T . When the two linear transformations are applied consecutively (composition), in either order, the result is to have no real effect. It is entirely analogous to squaring a positive number and then taking its (positive) square root.

Here is an example of a linear transformation that is invertible. As usual at the beginning of a section, do not be concerned with where S came from, just understand how it illustrates Definition IVLT.

Example  AIVLT
An invertible linear transformation
Archetype V is the linear transformation

T : P 3 → M 2 2 , T a + b x + c x 2 + d x 3 = a + b a − 2 c d b − d

Define the function S : M 2 2 → P 3 defined by

S a b c d = ( a − c − d ) + ( c + d ) x + 1 2 ( a − b − c − d ) x 2 + c x 3

Then

T ∘ S a b c d = T S a b c d = T ( a − c − d ) + ( c + d ) x + 1 2 ( a − b − c − d ) x 2 + c x 3 = ( a − c − d ) + ( c + d ) ( a − c − d ) − 2 ( 1 2 ( a − b − c − d ) ) c ( c + d ) − c = a b c d = I M 2 2 a b c d  And S ∘ T a + b x + c x 2 + d x 3 = S T a + b x + c x 2 + d x 3 = S a + b a − 2 c d b − d = ( ( a + b ) − d − ( b − d ) ) + ( d + ( b − d ) ) x + 1 2 ( ( a + b ) − ( a − 2 c ) − d − ( b − d ) ) x 2 + ( d ) x 3 = a + b x + c x 2 + d x 3 = I P 3 a + b x + c x 2 + d x 3

For now, understand why these computations show that T is invertible, and that S = T − 1 . Maybe even be amazed by how S works so perfectly in concert with T ! We will see later just how to arrive at the correct form of S (when it is possible). ⊠

It can be as instructive to study a linear transformation that is not invertible.

Example  ANILT
A non-invertible linear transformation
Consider the linear transformation T : ℂ 3 → M 2 2 defined by

T a b c = a − b 2 a + 2 b + c 3 a + b + c − 2 a − 6 b − 2 c

Suppose we were to search for an inverse function S : M 2 2 → ℂ 3 .

First verify that the 2 × 2 matrix A = 5 3 8 2 is not in the range of T . This will amount to finding an input to T , a b c , such that

a − b = 5 2 a + 2 b + c = 3 3 a + b + c = 8 − 2 a − 6 b − 2 c = 2

As this system of equations is inconsistent, there is no input column vector, and A ∉ ℛ T . How should we define S A ? Note that

T S A = T ∘ S A = I M 2 2 A = A

So any definition we would provide for S A must then be a column vector that T sends to A and we would have A ∈ ℛ T , contrary to the definition of T . This is enough to see that there is no function S that will allow us to conclude that T is invertible, since we cannot provide a consistent definition for S A if we assume T is invertible.

Even though we now know that T is not invertible, let’s not leave this example just yet. Check that

T 1 − 2 4 = 3 2 5 2 = B T 0 − 3 8 = 3 2 5 2 = B

How would we define S B ?

S B = S T 1 − 2 4 = S ∘ T 1 − 2 4 = I ℂ 3 1 − 2 4 = 1 − 2 4  or S B = S T 0 − 3 8 = S ∘ T 0 − 3 8 = I ℂ 3 0 − 3 8 = 0 − 3 8

Which definition should we provide for S B ? Both are necessary. But then S is not a function. So we have a second reason to know that there is no function S that will allow us to conclude that T is invertible. It happens that there are infinitely many column vectors that S would have to take to B . Construct the kernel of T ,

K T = − 1 − 1 4

Now choose either of the two inputs used above for T and add to it a scalar multiple of the basis vector for the kernel of T . For example,

x = 1 − 2 4 + ( − 2 ) − 1 − 1 4 = 3 0 − 4

then verify that T x = B . Practice creating a few more inputs for T that would be sent to B , and see why it is hopeless to think that we could ever provide a reasonable definition for S B ! There is a “whole subspace’s worth” of values that S B would have to take on. ⊠

In Example ANILT you may have noticed that T is not surjective, since the matrix A was not in the range of T . And T is not injective since there are two different input column vectors that T sends to the matrix B . Linear transformations T that are not surjective lead to putative inverse functions S that are undefined on inputs outside of the range of T . Linear transformations T that are not injective lead to putative inverse functions S that are multiply-defined on each of their inputs. We will formalize these ideas in Theorem ILTIS.

But first notice in Definition IVLT that we only require the inverse (when it exists) to be a function. When it does exist, it too is a linear transformation.

Theorem  ILTLT
Inverse of a Linear Transformation is a Linear Transformation
Suppose that T : U → V is an invertible linear transformation. Then the function T − 1 : V → U is a linear transformation. □

Proof   We work through verifying Definition LT for T − 1 , using the fact that T is a linear transformation to obtain the second equality in each half of the proof. To this end, suppose x , y ∈ V and α ∈ ℂ .

T − 1 x + y = T − 1 T T − 1 x + T T − 1 y   Definition IVLT = T − 1 T T − 1 x + T − 1 y   Definition LT = T − 1 x + T − 1 y   Definition IVLT  Now check the second defining property of a linear transformation for T − 1 , T − 1 α x = T − 1 α T T − 1 x   Definition IVLT = T − 1 T α T − 1 x   Definition LT = α T − 1 x   Definition IVLT

■

So T − 1 fulfills the requirements of Definition LT and is therefore a linear transformation. So when T has an inverse, T − 1 is also a linear transformation. Additionally, T − 1 is invertible and its inverse is what you might expect.

Theorem  IILT
Inverse of an Invertible Linear Transformation
Suppose that T : U → V is an invertible linear transformation. Then T − 1 is an invertible linear transformation and T − 1 − 1 = T . □

Proof   Because T is invertible, Definition IVLT tells us there is a function T − 1 : V → U such that

T − 1 ∘ T = I U T ∘ T − 1 = I V

Additionally, Theorem ILTLT tells us that T − 1 is more than just a function, it is a linear transformation. Now view these two statements as properties of the linear transformation T − 1 . In light of Definition IVLT, they together say that T − 1 is invertible (let T play the role of S in the statement of the definition). Furthermore, the inverse of T − 1 is then T , i.e.  T − 1 − 1 = T . ■

Subsection IV: Invertibility

We now know what an inverse linear transformation is, but just which linear transformations have inverses? Here is a theorem we have been preparing for all chapter long.

Theorem  ILTIS
Invertible Linear Transformations are Injective and Surjective
Suppose T : U → V is a linear transformation. Then T is invertible if and only if T is injective and surjective. □

Proof   ( ⇒ ) Since T is presumed invertible, we can employ its inverse, T − 1 ( Definition IVLT). To see that T is injective, suppose x , y ∈ U and assume that T x = T y ,

x = I U x   Definition IDLT = T − 1 ∘ T x   Definition IVLT = T − 1 T x   Definition LTC = T − 1 T y   Definition ILT = T − 1 ∘ T y   Definition LTC = I U y   Definition IVLT = y   Definition IDLT   So by  Definition ILT   T  is injective. To check that  T  is surjective, suppose v ∈ V . Then  T − 1 v  is a vector in  U . Compute  T T − 1 v = T ∘ T − 1 v   Definition LTC = I V v   Definition IVLT = v   Definition IDLT

So there is an element from U , when used as an input to T (namely T − 1 v ) that produces the desired output, v , and hence T is surjective by Definition SLT.

( ⇐ ) Now assume that T is both injective and surjective. We will build a function S : V → U that will establish that T is invertible. To this end, choose any v ∈ V . Since T is surjective, Theorem RSLT says ℛ T = V , so we have v ∈ ℛ T . Theorem RPI says that the pre-image of v , T − 1 v , is nonempty. So we can choose a vector from the pre-image of v , say u . In other words, there exists u ∈ T − 1 v .

Since T − 1 v is non-empty, Theorem KPI then says that

T − 1 v = u + z z ∈ K T

However, because T is injective, by Theorem KILT the kernel is trivial, K T = 0 . So the pre-image is a set with just one element, T − 1 v = u . Now we can define S by S v = u . This is the key to this half of this proof. Normally the preimage of a vector from the codomain might be an empty set, or an infinite set. But surjectivity requires that the preimage not be empty, and then injectivity limits the preimage to a singleton. Since our choice of v was arbitrary, we know that every pre-image for T is a set with a single element. This allows us to construct S as a function. Now that it is defined, verifying that it is the inverse of T will be easy. Here we go.

Choose u ∈ U . Define v = T u . Then T − 1 v = u , so that S v = u and,

S ∘ T u = S T u = S v = u = I U u

and since our choice of u was arbitrary we have function equality, S ∘ T = I U .

Now choose v ∈ V . Define u to be the single vector in the set T − 1 v , in other words, u = S v . Then T u = v , so

T ∘ S v = T S v = T u = v = I V v

and since our choice of v was arbitrary we have function equality, T ∘ S = I V . ■

When a linear transformation is both injective and surjective, the pre-image of any element of the codomain is a set of size one (a “singleton”). This fact allowed us to construct the inverse linear transformation in one half of the proof of Theorem ILTIS (see Technique C). We can follow this approach to construct the inverse of a specific linear transformation, as the next example shows.

Example  CIVLT
Computing the Inverse of a Linear Transformations
Consider the linear transformation T : S 2 2 → P 2 defined by

T a b b c = a + b + c + − a + 2 c x + 2 a + 3 b + 6 c x 2

T is invertible, which you are able to verify, perhaps by determining that the kernel of T is empty and the range of T is all of P 2 . This will be easier once we have Theorem RPNDD, which appears later in this section.

By Theorem ILTIS we know T − 1 exists, and it will be critical shortly to realize that T − 1 is automatically known to be a linear transformation as well ( Theorem ILTLT). To determine the complete behavior of T − 1 : P 2 → S 2 2 we can simply determine its action on a basis for the domain, P 2 . This is the substance of Theorem LTDB, and an excellent example of its application. Choose any basis of P 2 , the simpler the better, such as B = 1 , x , x 2 . Values of T − 1 for these three basis elements will be the single elements of their preimages. In turn, we have

T − 1 1 : T a b b c = 1 + 0 x + 0 x 2 1 1 1 1 − 1 0 2 0 2 3 6 0 →  RREF 1 0 0 − 6 0 1 0 1 0 0 0 1 − 3  (preimage) T − 1 1 = − 6 1 0 1 0 − 3  (function) T − 1 1 = − 6 1 0 1 0 − 3 T − 1 x : T a b b c = 0 + 1 x + 0 x 2 1 1 1 0 − 1 0 2 1 2 3 6 0 →  RREF 1 0 0 − 3 0 1 0 4 0 0 1 − 1  (preimage) T − 1 x = − 3 4 4 − 1  (function) T − 1 x = − 3 4 4 − 1 T − 1 x 2 : T a b b c = 0 + 0 x + 1 x 2 1 1 1 0 − 1 0 2 0 2 3 6 1 →  RREF 1 0 0 2 0 1 0 − 3 0 0 1 1  (preimage) T − 1 x 2 = 2 − 3 − 3 1  (function) T − 1 x 2 = 2 − 3 − 3 1

Theorem LTDB says, informally, “it is enough to know what a linear transformation does to a basis.” Formally, we have the outputs of T − 1 for a basis, so by Theorem LTDB there is a unique linear transformation with these outputs. So we put this information to work. The key step here is that we can convert any element of P 2 into a linear combination of the elements of the basis B (Theorem VRRB). We are after a “formula” for the value of T − 1 on a generic element of P 2 , say p + q x + r x 2 .

T − 1 p + q x + r x 2 = T − 1 p ( 1 ) + q ( x ) + r ( x 2 )   Theorem VRRB = p T − 1 1 + q T − 1 x + r T − 1 x 2   Theorem LTLC = p − 6 1 0 1 0 − 3 + q − 3 4 4 − 1 + r 2 − 3 − 3 1 = − 6 p − 3 q + 2 r 1 0 p + 4 q − 3 r 1 0 p + 4 q − 3 r − 3 p − q + r

Notice how a linear combination in the domain of T − 1 has been translated into a linear combination in the codomain of T − 1 since we know T − 1 is a linear transformation by Theorem ILTLT.

Also, notice how the augmented matrices used to determine the three pre-images could be combined into one calculation of a matrix in extended echelon form, reminiscent of a procedure we know for computing the inverse of a matrix (see Example CMI). Hmmmm. ⊠

We will make frequent use of the characterization of invertible linear transformations provided by Theorem ILTIS. The next theorem is a good example of this, and we will use it often, too.

Theorem  CIVLT
Composition of Invertible Linear Transformations
Suppose that T : U → V and S : V → W are invertible linear transformations. Then the composition, S ∘ T : U → W is an invertible linear transformation. □

Proof   Since S and T are both linear transformations, S ∘ T is also a linear transformation by Theorem CLTLT. Since S and T are both invertible, Theorem ILTIS says that S and T are both injective and surjective. Then Theorem CILTI says S ∘ T is injective, and Theorem CSLTS says S ∘ T is surjective. Now apply the “other half” of Theorem ILTIS and conclude that S ∘ T is invertible. ■

When a composition is invertible, the inverse is easy to construct.

Theorem  ICLT
Inverse of a Composition of Linear Transformations
Suppose that T : U → V and S : V → W are invertible linear transformations. Then S ∘ T is invertible and S ∘ T − 1 = T − 1 ∘ S − 1 . □

Proof   Compute, for all w ∈ W

S ∘ T ∘ T − 1 ∘ S − 1 w = S T T − 1 S − 1 w = S I V S − 1 w   Definition IVLT = S S − 1 w   Definition IDLT = w   Definition IVLT = I W w   Definition IDLT   so  S ∘ T ∘ T − 1 ∘ S − 1 = I W  and also T − 1 ∘ S − 1 ∘ S ∘ T u = T − 1 S − 1 S T u = T − 1 I V T u   Definition IVLT = T − 1 T u   Definition IDLT = u   Definition IVLT = I U u   Definition IDLT

so T − 1 ∘ S − 1 ∘ S ∘ T = I U . By Definition IVLT, S ∘ T is invertible and S ∘ T − 1 = T − 1 ∘ S − 1 . ■

Notice that this theorem not only establishes what the inverse of S ∘ T is, it also duplicates the conclusion of Theorem CIVLT and also establishes the invertibility of S ∘ T . But somehow, the proof of Theorem CIVLT is nicer way to get this property.

Does Theorem ICLT remind you of the flavor of any theorem we have seen about matrices? (Hint: Think about getting dressed.) Hmmmm.

Subsection SI: Structure and Isomorphism

A vector space is defined (Definition VS) as a set of objects (“vectors”) endowed with a definition of vector addition ( + ) and a definition of scalar multiplication (written with juxtaposition). Many of our definitions about vector spaces involve linear combinations (Definition LC), such as the span of a set (Definition SS) and linear independence (Definition LI). Other definitions are built up from these ideas, such as bases (Definition B) and dimension (Definition D). The defining properties of a linear transformation require that a function “respect” the operations of the two vector spaces that are the domain and the codomain (Definition LT). Finally, an invertible linear transformation is one that can be “undone” — it has a companion that reverses its effect. In this subsection we are going to begin to roll all these ideas into one.

A vector space has “structure” derived from definitions of the two operations and the requirement that these operations interact in ways that satisfy the ten properties of Definition VS. When two different vector spaces have an invertible linear transformation defined between them, then we can translate questions about linear combinations (spans, linear independence, bases, dimension) from the first vector space to the second. The answers obtained in the second vector space can then be translated back, via the inverse linear transformation, and interpreted in the setting of the first vector space. We say that these invertible linear transformations “preserve structure.” And we say that the two vector spaces are “structurally the same.” The precise term is “isomorphic,” from Greek meaning “of the same form.” Let’s begin to try to understand this important concept.

Definition  IVS
Isomorphic Vector Spaces
Two vector spaces U and V are isomorphic if there exists an invertible linear transformation T with domain U and codomain V , T : U → V . In this case, we write U ≅ V , and the linear transformation T is known as an isomorphism between U and V . △

A few comments on this definition. First, be careful with your language ( Technique L). Two vector spaces are isomorphic, or not. It is a yes/no situation and the term only applies to a pair of vector spaces. Any invertible linear transformation can be called an isomorphism, it is a term that applies to functions. Second, a given pair of vector spaces there might be several different isomorphisms between the two vector spaces. But it only takes the existence of one to call the pair isomorphic. Third, U isomorphic to V , or V isomorphic to U ? Doesn’t matter, since the inverse linear transformation will provide the needed isomorphism in the “opposite” direction. Being “isomorphic to” is an equivalence relation on the set of all vector spaces (see Theorem SER for a reminder about equivalence relations).

Example  IVSAV
Isomorphic vector spaces, Archetype V
Archetype V is a linear transformation from P 3 to M 2 2 ,

T : P 3 → M 2 2 , T a + b x + c x 2 + d x 3 = a + b a − 2 c d b − d

Since it is injective and surjective, Theorem ILTIS tells us that it is an invertible linear transformation. By Definition IVS we say P 3 and M 2 2 are isomorphic.

At a basic level, the term “isomorphic” is nothing more than a codeword for the presence of an invertible linear transformation. However, it is also a description of a powerful idea, and this power only becomes apparent in the course of studying examples and related theorems. In this example, we are led to believe that there is nothing “structurally” different about P 3 and M 2 2 . In a certain sense they are the same. Not equal, but the same. One is as good as the other. One is just as interesting as the other.

Here is an extremely basic application of this idea. Suppose we want to compute the following linear combination of polynomials in P 3 ,

5 ( 2 + 3 x − 4 x 2 + 5 x 3 ) + ( − 3 ) ( 3 − 5 x + 3 x 2 + x 3 )

Rather than doing it straight-away (which is very easy), we will apply the transformation T to convert into a linear combination of matrices, and then compute in M 2 2 according to the definitions of the vector space operations there (Example VSM),

T 5 ( 2 + 3 x − 4 x 2 + 5 x 3 ) + ( − 3 ) ( 3 − 5 x + 3 x 2 + x 3 ) = 5 T 2 + 3 x − 4 x 2 + 5 x 3 + ( − 3 ) T 3 − 5 x + 3 x 2 + x 3   Theorem LTLC = 5 5 1 0 5 − 2 + ( − 3 ) − 2 − 3 1 − 6  Definition of  T = 3 1 5 9 2 2 8  Operations in  M 2 2

Now we will translate our answer back to P 3 by applying T − 1 , which we found in Example AIVLT,

T − 1 : M 2 2 → P 3 , T − 1 a b c d = ( a − c − d ) + ( c + d ) x + 1 2 ( a − b − c − d ) x 2 + c x 3

We compute,

T − 1 3 1 5 9 2 2 8 = 1 + 3 0 x − 2 9 x 2 + 2 2 x 3

which is, as expected, exactly what we would have computed for the original linear combination had we just used the definitions of the operations in P 3 (Example VSP). Notice this is meant only as an illustration and not a suggested route for doing this particular computation. ⊠

Checking the dimensions of two vector spaces can be a quick way to establish that they are not isomorphic. Here’s the theorem.

Theorem  IVSED
Isomorphic Vector Spaces have Equal Dimension
Suppose U and V are isomorphic vector spaces. Then dim U = dim V . □

Proof   If U and V are isomorphic, there is an invertible linear transformation T : U → V ( Definition IVS). T is injective by Theorem ILTIS and so by Theorem ILTD, dim U ≤ dim V . Similarly, T is surjective by Theorem ILTIS and so by Theorem SLTD, dim U ≥ dim V . The net effect of these two inequalities is that dim U = dim V . ■

The contrapositive of Theorem IVSED says that if U and V have different dimensions, then they are not isomorphic. Dimension is the simplest “structural” characteristic that will allow you to distinguish non-isomorphic vector spaces. For example P 6 is not isomorphic to M 3 4 since their dimensions (7 and 12, respectively) are not equal. With tools developed in Section VR we will be able to establish that the converse of Theorem IVSED is true. Think about that one for a moment.

Subsection RNLT: Rank and Nullity of a Linear Transformation

Just as a matrix has a rank and a nullity, so too do linear transformations. And just like the rank and nullity of a matrix are related (they sum to the number of columns, Theorem RPNC) the rank and nullity of a linear transformation are related. Here are the definitions and theorems, see the Archetypes ( Appendix A) for loads of examples.

Definition  ROLT
Rank Of a Linear Transformation
Suppose that T : U → V is a linear transformation. Then the rank of T , r T , is the dimension of the range of T ,

r T = dim ℛ T

(This definition contains Notation ROLT.) △

Definition  NOLT
Nullity Of a Linear Transformation
Suppose that T : U → V is a linear transformation. Then the nullity of T , n T , is the dimension of the kernel of T ,

n T = dim K T

(This definition contains Notation NOLT.) △

Here are two quick theorems.

Theorem  ROSLT
Rank Of a Surjective Linear Transformation
Suppose that T : U → V is a linear transformation. Then the rank of T is the dimension of V , r T = dim V , if and only if T is surjective. □

Proof   By Theorem RSLT, T is surjective if and only if ℛ T = V . Applying Definition ROLT, ℛ T = V if and only if r T = dim ℛ T = dim V . ■

Theorem  NOILT
Nullity Of an Injective Linear Transformation
Suppose that T : U → V is a linear transformation. Then the nullity of T is zero, n T = 0 , if and only if T is injective. □

Proof   By Theorem KILT, T is injective if and only if K T = 0 . Applying Definition NOLT, K T = 0 if and only if n T = 0 . ■

Just as injectivity and surjectivity come together in invertible linear transformations, there is a clear relationship between rank and nullity of a linear transformation. If one is big, the other is small.

Theorem  RPNDD
Rank Plus Nullity is Domain Dimension
Suppose that T : U → V is a linear transformation. Then

r T + n T = dim U

□

Proof   Let r = r T and s = n T . Suppose that R = v 1 , v 2 , v 3 , … , v r ⊆ V is a basis of the range of T , ℛ T , and S = u 1 , u 2 , u 3 , … , u s ⊆ U is a basis of the kernel of T , K T . Note that R and S are possibly empty, which means that some of the sums in this proof are “empty” and are equal to the zero vector.

Because the elements of R are all in the range of T , each must have a non-empty pre-image by Theorem RPI. Choose vectors w i ∈ U , 1 ≤ i ≤ r such that w i ∈ T − 1 v i . So T w i = v i , 1 ≤ i ≤ r . Consider the set

B = u 1 , u 2 , u 3 , … , u s , w 1 , w 2 , w 3 , … , w r

We claim that B is a basis for U .

To establish linear independence for B , begin with a relation of linear dependence on B . So suppose there are scalars a 1 , a 2 , a 3 , … , a s and b 1 , b 2 , b 3 , … , b r

0 = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a s u s + b 1 w 1 + b 2 w 2 + b 3 w 3 + ⋯ + b r w r

Then

0 = T 0   Theorem LTTZZ = T a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a s u s + b 1 w 1 + b 2 w 2 + b 3 w 3 + ⋯ + b r w r   Definition LI = a 1 T u 1 + a 2 T u 2 + a 3 T u 3 + ⋯ + a s T u s + b 1 T w 1 + b 2 T w 2 + b 3 T w 3 + ⋯ + b r T w r   Theorem LTLC = a 1 0 + a 2 0 + a 3 0 + ⋯ + a s 0 + b 1 T w 1 + b 2 T w 2 + b 3 T w 3 + ⋯ + b r T w r   Definition KLT = 0 + 0 + 0 + ⋯ + 0 + b 1 T w 1 + b 2 T w 2 + b 3 T w 3 + ⋯ + b r T w r   Theorem ZVSM = b 1 T w 1 + b 2 T w 2 + b 3 T w 3 + ⋯ + b r T w r   Property Z = b 1 v 1 + b 2 v 2 + b 3 v 3 + ⋯ + b r v r   Definition PI

This is a relation of linear dependence on R (Definition RLD), and since R is a linearly independent set (Definition LI), we see that b 1 = b 2 = b 3 = … = b r = 0 . Then the original relation of linear dependence on B becomes

0 = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a s u s + 0 w 1 + 0 w 2 + … + 0 w r = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a s u s + 0 + 0 + … + 0   Theorem ZSSM = a 1 u 1 + a 2 u 2 + a 3 u 3 + ⋯ + a s u s   Property Z

But this is again a relation of linear independence (Definition RLD), now on the set S . Since S is linearly independent (Definition LI), we have a 1 = a 2 = a 3 = … = a r = 0 . Since we now know that all the scalars in the relation of linear dependence on B must be zero, we have established the linear independence of S through Definition LI.

To now establish that B spans U , choose an arbitrary vector u ∈ U . Then T u ∈ R ( T ) , so there are scalars c 1 , c 2 , c 3 , … , c r such that

T u = c 1 v 1 + c 2 v 2 + c 3 v 3 + ⋯ + c r v r

Use the scalars c 1 , c 2 , c 3 , … , c r to define a vector y ∈ U ,

y = c 1 w 1 + c 2 w 2 + c 3 w 3 + ⋯ + c r w r

Then

T u − y = T u − T y   Theorem LTLC = T u − T c 1 w 1 + c 2 w 2 + c 3 w 3 + ⋯ + c r w r  Substitution = T u − c 1 T w 1 + c 2 T w 2 + ⋯ + c r T w r   Theorem LTLC = T u − c 1 v 1 + c 2 v 2 + c 3 v 3 + ⋯ + c r v r w i ∈ T − 1 v i = T u − T u  Substitution = 0   Property AI

So the vector u − y is sent to the zero vector by T and hence is an element of the kernel of T . As such it can be written as a linear combination of the basis vectors for K T , the elements of the set S . So there are scalars d 1 , d 2 , d 3 , … , d s such that

u − y = d 1 u 1 + d 2 u 2 + d 3 u 3 + ⋯ + d s u s

Then

u = u − y + y = d 1 u 1 + d 2 u 2 + d 3 u 3 + ⋯ + d s u s + c 1 w 1 + c 2 w 2 + c 3 w 3 + ⋯ + c r w r

This says that for any vector, u , from U , there exist scalars ( d 1 , d 2 , d 3 , … , d s , c 1 , c 2 , c 3 , … , c r ) that form u as a linear combination of the vectors in the set B . In other words, B spans U (Definition SS).

So B is a basis (Definition B) of U with s + r vectors, and thus

dim U = s + r = n T + r T

as desired. ■

Theorem RPNC said that the rank and nullity of a matrix sum to the number of columns of the matrix. This result is now an easy consequence of Theorem RPNDD when we consider the linear transformation T : ℂ n → ℂ m defined with the m × n matrix A by T x = A x . The range and kernel of T are identical to the column space and null space of the matrix A ( Exercise ILT.T20, Exercise SLT.T20), so the rank and nullity of the matrix A are identical to the rank and nullity of the linear transformation T . The dimension of the domain of T is the dimension of ℂ n , exactly the number of columns for the matrix A .

This theorem can be especially useful in determining basic properties of linear transformations. For example, suppose that T : ℂ 6 → ℂ 6 is a linear transformation and you are able to quickly establish that the kernel is trivial. Then n T = 0 . First this means that T is injective by Theorem NOILT. Also, Theorem RPNDD becomes

6 = dim ℂ 6 = r T + n T = r T + 0 = r T

So the rank of T is equal to the rank of the codomain, and by Theorem ROSLT we know T is surjective. Finally, we know T is invertible by Theorem ILTIS. So from the determination that the kernel is trivial, and consideration of various dimensions, the theorems of this section allow us to conclude the existence of an inverse linear transformation for T .

Similarly, Theorem RPNDD can be used to provide alternative proofs for Theorem ILTD, Theorem SLTD and Theorem IVSED. It would be an interesting exercise to construct these proofs.

It would be instructive to study the archetypes that are linear transformations and see how many of their properties can be deduced just from considering only the dimensions of the domain and codomain. Then add in just knowledge of either the nullity or rank, and so how much more you can learn about the linear transformation. The table preceding all of the archetypes ( Appendix A) could be a good place to start this analysis.

Subsection SLELT: Systems of Linear Equations and Linear Transformations

This subsection does not really belong in this section, or any other section, for that matter. It is just the right time to have a discussion about the connections between the central topic of linear algebra, linear transformations, and our motivating topic from Chapter SLE, systems of linear equations. We will discuss several theorems we have seen already, but we will also make some forward-looking statements that will be justified in Chapter R.

Archetype D and Archetype E are ideal examples to illustrate connections with linear transformations. Both have the same coefficient matrix,

D = 2 1 7 − 7 − 3 4 − 5 − 6 1 1 4 − 5

To apply the theory of linear transformations to these two archetypes, employ matrix multiplication ( Definition MM) and define the linear transformation,

T : ℂ 4 → ℂ 3 , T x = D x = x 1 2 − 3 1 + x 2 1 4 1 + x 3 7 − 5 4 + x 4 − 7 − 6 − 5

Theorem MBLT tells us that T is indeed a linear transformation. Archetype D asks for solutions to ℒ S D , b , where b = 8 − 1 2 − 4 . In the language of linear transformations this is equivalent to asking for T − 1 b . In the language of vectors and matrices it asks for a linear combination of the four columns of D that will equal b . One solution listed is w = 7 8 1 3 . With a non-empty preimage, Theorem KPI tells us that the complete solution set of the linear system is the preimage of b ,

w + K T = w + z z ∈ K T

The kernel of the linear transformation T is exactly the null space of the matrix D (see Exercise ILT.T20), so this approach to the solution set should be reminiscent of Theorem PSPHS. The kernel of the linear transformation is the preimage of the zero vector, exactly equal to the solution set of the homogeneous system ℒ S D , 0 . Since D has a null space of dimension two, every preimage (and in particular the preimage of b ) is as “big” as a subspace of dimension two (but is not a subspace).

Archetype E is identical to Archetype D but with a different vector of constants, d = 2 3 2 . We can use the same linear transformation T to discuss this system of equations since the coefficient matrix is identical. Now the set of solutions to ℒ S D , d is the pre-image of d , T − 1 d . However, the vector d is not in the range of the linear transformation (nor is it in the column space of the matrix, since these two sets are equal by Exercise SLT.T20). So the empty pre-image is equivalent to the inconsistency of the linear system.

These two archetypes each have three equations in four variables, so either the resulting linear systems are inconsistent, or they are consistent and application of Theorem CMVEI tells us that the system has infinitely many solutions. Considering these same parameters for the linear transformation, the dimension of the domain, ℂ 4 , is four, while the codomain, ℂ 3 , has dimension three. Then

n T = dim ℂ 4 − r T   Theorem RPNDD = 4 − dim ℛ T   Definition ROLT ≥ 4 − 3   ℛ T  subspace of  ℂ 3 = 1

So the kernel of T is nontrivial simply by considering the dimensions of the domain (number of variables) and the codomain (number of equations). Pre-images of elements of the codomain that are not in the range of T are empty (inconsistent systems). For elements of the codomain that are in the range of T (consistent systems), Theorem KPI tells us that the pre-images are built from the kernel, and with a non-trivial kernel, these pre-images are infinite (infinitely many solutions).

When do systems of equations have unique solutions? Consider the system of linear equations ℒ S C , f and the linear transformation S x = C x . If S has a trivial kernel, then pre-images will either be empty or be finite sets with single elements. Correspondingly, the coefficient matrix C will have a trivial null space and solution sets will either be empty (inconsistent) or contain a single solution (unique solution). Should the matrix be square and have a trivial null space then we recognize the matrix as being nonsingular. A square matrix means that the corresponding linear transformation, T , has equal-sized domain and codomain. With a nullity of zero, T is injective, and also Theorem RPNDD tells us that rank of T is equal to the dimension of the domain, which in turn is equal to the dimension of the codomain. In other words, T is surjective. Injective and surjective, and Theorem ILTIS tells us that T is invertible. Just as we can use the inverse of the coefficient matrix to find the unique solution of any linear system with a nonsingular coefficient matrix ( Theorem SNCM), we can use the inverse of the linear transformation to construct the unique element of any pre-image (proof of Theorem ILTIS).

The executive summary of this discussion is that to every coefficient matrix of a system of linear equations we can associate a natural linear transformation. Solution sets for systems with this coefficient matrix are preimages of elements of the codomain of the linear transformation. For every theorem about systems of linear equations there is an analogue about linear transformations. The theory of linear transformations provides all the tools to recreate the theory of solutions to linear systems of equations.

We will continue this adventure in Chapter R.

Subsection READ: Reading Questions

Subsection EXC: Exercises

C10 The archetypes below are linear transformations of the form T : U → V that are invertible. For each, the inverse linear transformation is given explicitly as part of the archetype’s description. Verify for each linear transformation that

T − 1 ∘ T = I U T ∘ T − 1 = I V

Archetype R,
Archetype V,
Archetype W  
Contributed by  Robert Beezer

C20 Determine if the linear transformation T : P 2 → M 2 2 is (a) injective, (b) surjective, (c) invertible.

T a + b x + c x 2 = a + 2 b − 2 c 2 a + 2 b − a + b − 4 c 3 a + 2 b + 2 c

Contributed by  Robert Beezer Solution [ 1611]

C21 Determine if the linear transformation S : P 3 → M 2 2 is (a) injective, (b) surjective, (c) invertible.

S a + b x + c x 2 + d x 3 = − a + 4 b + c + 2 d 4 a − b + 6 c − d a + 5 b − 2 c + 2 d a + 2 c + 5 d

Contributed by  Robert Beezer Solution [ 1612]

C25 For each linear transformation below: (a) Find the matrix representation of T , (b) Calculate n ( T ) , (c) Calculate r ( T ) , (d) Graph the image in either ℝ 2 or ℝ 3 as appropriate, (e) How many dimensions are lost?, and (f) How many dimensions are preserved?

Contributed by  Chris Black

C50 Consider the linear transformation S : M 1 2 → P 1 from the set of 1 × 2 matrices to the set of polynomials of degree at most 1, defined by

S a b = ( 3 a + b ) + ( 5 a + 2 b ) x

Prove that S is invertible. Then show that the linear transformation

R : P 1 → M 1 2 , R r + s x = ( 2 r − s ) ( − 5 r + 3 s )

is the inverse of S , that is S − 1 = R .  
Contributed by  Robert Beezer Solution [ 1615]

M30 The linear transformation S below is invertible. Find a formula for the inverse linear transformation, S − 1 .

S : P 1 → M 1 , 2 , S a + b x = 3 a + b 2 a + b

Contributed by  Robert Beezer Solution [ 1617]

M31 The linear transformation R : M 1 2 → M 2 1 is invertible. Determine a formula for the inverse linear transformation R − 1 : M 2 1 → M 1 2 .

R a b = a + 3 b 4 a + 1 1 b

Contributed by  Robert Beezer Solution [ 1619]

M50 Rework Example CIVLT, only in place of the basis B for P 2 , choose instead to use the basis C = 1 , 1 + x , 1 + x + x 2 . This will complicate writing a generic element of the domain of T − 1 as a linear combination of the basis elements, and the algebra will be a bit messier, but in the end you should obtain the same formula for T − 1 . The inverse linear transformation is what it is, and the choice of a particular basis should not influence the outcome.  
Contributed by  Robert Beezer

M60 Suppose U and V are vector spaces. Define the function Z : U → V by T u = 0 V for every u ∈ U . Then by Exercise LT.M60, Z is a linear transformation. Formulate a condition on U and V that is equivalent to Z being an invertible linear transformation. In other words, fill in the blank to complete the following statement (and then give a proof): Z is invertible if and only if U and V are ______________. (See Exercise ILT.M60, Exercise SLT.M60, Exercise MR.M60.)  
Contributed by  Robert Beezer

T05 Prove that the identity linear transformation ( Definition IDLT) is both injective and surjective, and hence invertible.  
Contributed by  Robert Beezer

T15 Suppose that T : U → V is a surjective linear transformation and dim U = dim V . Prove that T is injective.  
Contributed by  Robert Beezer Solution [ 1621]

T16 Suppose that T : U → V is an injective linear transformation and dim U = dim V . Prove that T is surjective.  
Contributed by  Robert Beezer

T30 Suppose that U and V are isomorphic vector spaces. Prove that there are infinitely many isomorphisms between U and V .  
Contributed by  Robert Beezer Solution [ 1621]

T40 Suppose T : U → V and S : V → W are linear transformations and dim U = dim V = dim W . Suppose that S ∘ T is invertible. Prove that S and T are individually invertible (this could be construed as a converse of Theorem CIVLT).  
Contributed by  Robert Beezer Solution [ 1622]

Subsection SOL: Solutions

C20 Contributed by  Robert Beezer Statement [ 1604]
(a) We will compute the kernel of T . Suppose that a + b x + c x 2 ∈ K T . Then

0 0 0 0 = T a + b x + c x 2 = a + 2 b − 2 c 2 a + 2 b − a + b − 4 c 3 a + 2 b + 2 c

and matrix equality ( Theorem ME) yields the homogeneous system of four equations in three variables,

a + 2 b − 2 c = 0 2 a + 2 b = 0 − a + b − 4 c = 0 3 a + 2 b + 2 c = 0

The coefficient matrix of this system row-reduces as

1 2 − 2 2 2 0 − 1 1 − 4 3 2 2 →  RREF 1 0 2 0 1  − 2 0 0 0 0 0 0

From the existence of non-trivial solutions to this system, we can infer non-zero polynomials in K T . By Theorem KILT we then know that T is not injective.

(b) Since 3 = dim P 2 < dim M 2 2 = 4 , by Theorem SLTD T is not surjective.

(c) Since T is not surjective, it is not invertible by Theorem ILTIS.

C21 Contributed by  Robert Beezer Statement [ 1605]
(a) To check injectivity, we compute the kernel of S . To this end, suppose that a + b x + c x 2 + d x 3 ∈ K S , so

0 0 0 0 = S a + b x + c x 2 + d x 3 = − a + 4 b + c + 2 d 4 a − b + 6 c − d a + 5 b − 2 c + 2 d a + 2 c + 5 d

this creates the homogeneous system of four equations in four variables,

− a + 4 b + c + 2 d = 0 4 a − b + 6 c − d = 0 a + 5 b − 2 c + 2 d = 0 a + 2 c + 5 d = 0

The coefficient matrix of this system row-reduces as,

− 1 4 1 2 4 − 1 6 − 1 1 5 − 2 2 1 0 2 5 →  RREF 1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1

We recognize the coefficient matrix as being nonsingular, so the only solution to the system is a = b = c = d = 0 , and the kernel of S is trivial, K S = 0 + 0 x + 0 x 2 + 0 x 3 . By Theorem KILT, we see that S is injective.

(b) We can establish that S is surjective by considering the rank and nullity of S .

r S = dim P 3 − n S   Theorem RPNDD = 4 − 0 = dim M 2 2

So, ℛ S is a subspace of M 2 2 ( Theorem RLTS) whose dimension equals that of M 2 2 . By Theorem EDYES, we gain the set equality ℛ S = M 2 2 . Theorem RSLT then implies that S is surjective.

(c) Since S is both injective and surjective, Theorem ILTIS says S is invertible.

C50 Contributed by  Robert Beezer Statement [ 1606]
Determine the kernel of S first. The condition that S a b = 0 becomes ( 3 a + b ) + ( 5 a + 2 b ) x = 0 + 0 x . Equating coefficients of these polynomials yields the system

3 a + b = 0 5 a + 2 b = 0

This homogeneous system has a nonsingular coefficient matrix, so the only solution is a = 0 , b = 0 and thus

K S = 0 0

By Theorem KILT, we know S is injective. With n S = 0 we employ Theorem RPNDD to find

r S = r S + 0 = r S + n S = dim M 1 2 = 2 = dim P 1

Since ℛ S ⊆ P 1 and dim ℛ S = dim P 1 , we can apply Theorem EDYES to obtain the set equality ℛ S = P 1 and therefore S is surjective.

One of the two defining conditions of an invertible linear transformation is ( Definition IVLT)

S ∘ R a + b x = S R a + b x = S ( 2 a − b ) ( − 5 a + 3 b ) = 3 ( 2 a − b ) + ( − 5 a + 3 b ) + 5 ( 2 a − b ) + 2 ( − 5 a + 3 b ) x = ( 6 a − 3 b ) + ( − 5 a + 3 b ) + ( 1 0 a − 5 b ) + ( − 1 0 a + 6 b ) x = a + b x = I P 1 a + b x

That R ∘ S a b = I M 1 2 a b is similar.

M30 Contributed by  Robert Beezer Statement [ 1607]
(Another approach to this solution would follow Example CIVLT.)

Suppose that S − 1 : M 1 , 2 → P 1 has a form given by

S − 1 z w = r z + s w + p z + q w x

where r , s , p , q are unknown scalars. Then

a + b x = S − 1 S a + b x = S − 1 3 a + b 2 a + b = r ( 3 a + b ) + s ( 2 a + b ) + p ( 3 a + b ) + q ( 2 a + b ) x = ( 3 r + 2 s ) a + ( r + s ) b + ( 3 p + 2 q ) a + ( p + q ) b x

Equating coefficients of these two polynomials, and then equating coefficients on a and b , gives rise to 4 equations in 4 variables,

3 r + 2 s = 1 r + s = 0 3 p + 2 q = 0 p + q = 1

This system has a unique solution: r = 1 , s = − 1 , p = − 2 , q = 3 . So the desired inverse linear transformation is

S − 1 z w = z − w + − 2 z + 3 w x

Notice that the system of 4 equations in 4 variables could be split into two systems, each with two equations in two variables (and identical coefficient matrices). After making this split, the solution might feel like computing the inverse of a matrix ( Theorem CINM). Hmmmm.

M31 Contributed by  Robert Beezer Statement [ 1608]
(Another approach to this solution would follow Example CIVLT.)

We are given that R is invertible. The inverse linear transformation can be formulated by considering the pre-image of a generic element of the codomain. With injectivity and surjectivity, we know that the pre-image of any element will be a set of size one — it is this lone element that will be the output of the inverse linear transformation.

Suppose that we set v = x y as a generic element of the codomain, M 2 1 . Then if r s = w ∈ R − 1 v ,

x y = v = R w = r + 3 s 4 r + 1 1 s

So we obtain the system of two equations in the two variables r and s ,

r + 3 s = x 4 r + 1 1 s = y

With a nonsingular coefficient matrix, we can solve the system using the inverse of the coefficient matrix,

r = − 1 1 x + 3 y s = 4 x − y

So we define,

R − 1 v = R − 1 x y = w = r s = − 1 1 x + 3 y 4 x − y

T15 Contributed by  Robert Beezer Statement [ 1609]
If T is surjective, then Theorem RSLT says ℛ T = V , so r T = dim V . In turn, the hypothesis gives r T = dim U . Then, using Theorem RPNDD,

n T = r T + n T − r T = dim U − dim U = 0

With a null space of zero dimension, K T = 0 , and by Theorem KILT we see that T is injective. T is both injective and surjective so by Theorem ILTIS, T is invertible.

T30 Contributed by  Robert Beezer Statement [ 1609]
Since U and V are isomorphic, there is at least one isomorphism between them ( Definition IVS), say T : U → V . As such, T is an invertible linear transformation.

For α ∈ ℂ define the linear transformation S : V → V by S v = α v . Convince yourself that when α ≠ 0 , S is an invertible linear transformation ( Definition IVLT). Then the composition, S ∘ T : U → V , is an invertible linear transformation by Theorem CIVLT. Once convinced that each non-zero value of α gives rise to a different functions for S ∘ T , then we have constructed infinitely many isomorphisms from U to V .

T40 Contributed by  Robert Beezer Statement [ 1610]
Since S ∘ T is invertible, by Theorem ILTIS S ∘ T is injective and therefore has a trivial kernel by Theorem KILT. Then

K T ⊆ K S ∘ T   Exercise ILT.T15 = 0   Theorem KILT

Since T has a trivial kernel, by Theorem KILT, T is injective. Also,

r T = dim U − n T   Theorem RPNDD = dim U − 0   Theorem NOILT = dim V  Hypothesis

Since ℛ T ⊆ V , Theorem EDYES gives ℛ T = V , so by Theorem RSLT, T is surjective. Finally, by Theorem ILTIS, T is invertible.

Since S ∘ T is invertible, by Theorem ILTIS S ∘ T is surjective and therefore has a full range by Theorem RSLT. Then

W = ℛ S ∘ T   Theorem RSLT ⊆ ℛ S   Exercise SLT.T15

Since ℛ S ⊆ W we have ℛ S = W and by Theorem RSLT, S is surjective. By an application of Theorem RPNDD similar to the first part of this solution, we see that S has a trivial kernel, is therefore injective ( Theorem KILT), and thus invertible ( Theorem ILTIS).

Annotated Acronyms LT: Linear Transformations

Theorem MBLT
You give me an m × n matrix and I’ll give you a linear transformation T : ℂ n → ℂ m . This is our first hint that there is some relationship between linear transformations and matrices.

Theorem MLTCV
You give me a linear transformation T : ℂ n → ℂ m and I’ll give you an m × n matrix. This is our second hint that there is some relationship between linear transformations and matrices. Generalizing this relationship to arbitrary vector spaces (i.e. not just ℂ n and ℂ m ) will be the most important idea of Chapter R.

Theorem LTLC
A simple idea, and as described in Exercise LT.T20, equivalent to the Definition LT. The statement is really just for convenience, as we’ll quote this one often.

Theorem LTDB
Another simple idea, but a powerful one. “It is enough to know what a linear transformation does to a basis.” At the outset of Chapter R, Theorem VRRB will help us define a very important function, and then Theorem LTDB will allow us to understand that this function is also a linear transformation.

Theorem KPI
The pre-image will be an important construction in this chapter, and this is one of the most important descriptions of the pre-image. It should remind you of Theorem PSPHS, which is described in Acronyms V. See Theorem RPI, which is also described below.

Theorem KILT
Kernels and injective linear transformations are intimately related. This result is the connection. Compare with Theorem RSLT below.

Theorem ILTB
Injective linear transformations and linear independence are intimately related. This result is the connection. Compare with Theorem SLTB below.

Theorem RSLT
Ranges and surjective linear transformations are intimately related. This result is the connection. Compare with Theorem KILT above.

Theorem SSRLT
This theorem provides the most direct way of forming the range of a linear transformation. The resulting spanning set might well be linearly dependent, and beg for some clean-up, but that doesn’t stop us from having very quickly formed a reasonable description of the range. If you find the determination of spanning sets or ranges difficult, this is one worth remembering. You can view this as the analogue of forming a column space by a direct application of Definition CSM.

Theorem SLTB
Surjective linear transformations and spanning sets are intimately related. This result is the connection. Compare with Theorem ILTB above.

Theorem RPI
This is the analogue of Theorem KPI. Membership in the range is equivalent to nonempty pre-images.

Theorem ILTIS
Injectivity and surjectivity are independent concepts. You can have one without the other. But when you have both, you get invertibility, a linear transformation that can be run “backwards.” This result might explain the entire structure of the four sections in this chapter.

Theorem RPNDD
This is the promised generalization of Theorem RPNC about matrices. So the number of columns of a matrix is the analogue of the dimension of the domain. This will become even more precise in Chapter R. For now, this can be a powerful result for determining dimensions of kernels and ranges, and consequently, the injectivity or surjectivity of linear transformations. Never underestimate a theorem that counts something.

Chapter 8. Chapter R  Representations

Previous work with linear transformations may have convinced you that we can convert most questions about linear transformations into questions about systems of equations or properties of subspaces of ℂ m . In this section we begin to make these vague notions precise. We have used the word “representation” prior, but it will get a heavy workout in this chapter. In many ways, everything we have studied so far was in preparation for this chapter.

Section CB  Change of Basis

Section OD  Orthonormal Diagonalization

Section NLT  Nilpotent Linear Transformations

Section IS  Invariant Subspaces

Section JCF  Jordan Canonical Form

Parent topic: Part C  Core

Previous topic: Section IVLT  Invertible Linear Transformations

Next topic: Section CB  Change of Basis

Section CB  Change of Basis

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

We have seen in Section MR that a linear transformation can be represented by a matrix, once we pick bases for the domain and codomain. How does the matrix representation change if we choose different bases? Which bases lead to especially nice representations? From the infinite possibilities, what is the best possible representation? This section will begin to answer these questions. But first we need to define eigenvalues for linear transformations and the change-of-basis matrix.

Parent topic: Chapter R  Representations

Previous topic: Chapter R  Representations

Next topic: Section OD  Orthonormal Diagonalization

Subsection EELT: Eigenvalues and Eigenvectors of Linear Transformations

We now define the notion of an eigenvalue and eigenvector of a linear transformation. It should not be too surprising, especially if you remind yourself of the close relationship between matrices and linear transformations.

Definition  EELT
Eigenvalue and Eigenvector of a Linear Transformation
Suppose that T : V → V is a linear transformation. Then a nonzero vector v ∈ V is an eigenvector of T for the eigenvalue λ if T v = λ v . △

We will see shortly the best method for computing the eigenvalues and eigenvectors of a linear transformation, but for now, here are some examples to verify that such things really do exist.

Example  ELTBM
Eigenvectors of linear transformation between matrices
Consider the linear transformation T : M 2 2 → M 2 2 defined by

T a b c d = − 1 7 a + 1 1 b + 8 c − 1 1 d − 5 7 a + 3 5 b + 2 4 c − 3 3 d − 1 4 a + 1 0 b + 6 c − 1 0 d − 4 1 a + 2 5 b + 1 6 c − 2 3 d

and the vectors

x 1 = 0 1 0 1 x 2 = 1 1 1 0 x 3 = 1 3 2 3 x 4 = 2 6 1 4

Then compute

T x 1 = T 0 1 0 1 = 0 2 0 2 = 2 x 1 T x 2 = T 1 1 1 0 = 2 2 2 0 = 2 x 2 T x 3 = T 1 3 2 3 = − 1 − 3 − 2 − 3 = ( − 1 ) x 3 T x 4 = T 2 6 1 4 = − 4 − 1 2 − 2 − 8 = ( − 2 ) x 4

So x 1 , x 2 , x 3 , x 4 are eigenvectors of T with eigenvalues (respectively) λ 1 = 2 , λ 2 = 2 , λ 3 = − 1 , λ 4 = − 2 . ⊠

Here’s another.

Example  ELTBP
Eigenvectors of linear transformation between polynomials
Consider the linear transformation R : P 2 → P 2 defined by

R a + b x + c x 2 = ( 1 5 a + 8 b − 4 c ) + ( − 1 2 a − 6 b + 3 c ) x + ( 2 4 a + 1 4 b − 7 c ) x 2

and the vectors

w 1 = 1 − x + x 2 w 2 = x + 2 x 2 w 3 = 1 + 4 x 2

Then compute

R w 1 = R 1 − x + x 2 = 3 − 3 x + 3 x 2 = 3 w 1 R w 2 = R x + 2 x 2 = 0 + 0 x + 0 x 2 = 0 w 2 R w 3 = R 1 + 4 x 2 = − 1 − 4 x 2 = ( − 1 ) w 3

So w 1 , w 2 , w 3 are eigenvectors of R with eigenvalues (respectively) λ 1 = 3 , λ 2 = 0 , λ 3 = − 1 . Notice how the eigenvalue λ 2 = 0 indicates that the eigenvector w 2 is a non-trivial element of the kernel of R , and therefore R is not injective ( Exercise CB.T15). ⊠

Of course, these examples are meant only to illustrate the definition of eigenvectors and eigenvalues for linear transformations, and therefore beg the question, “How would I find eigenvectors?” We’ll have an answer before we finish this section. We need one more construction first.

Subsection CBM: Change-of-Basis Matrix

Given a vector space, we know we can usually find many different bases for the vector space, some nice, some nasty. If we choose a single vector from this vector space, we can build many different representations of the vector by constructing the representations relative to different bases. How are these different representations related to each other? A change-of-basis matrix answers this question.

Definition  CBM
Change-of-Basis Matrix
Suppose that V is a vector space, and I V : V → V is the identity linear transformation on V . Let B = v 1 , v 2 , v 3 , … , v n and C be two bases of V . Then the change-of-basis matrix from B to C is the matrix representation of I V relative to B and C ,

C B , C = M B , C I V = ρ C I V v 1 ρ C I V v 2 ρ C I V v 3 … ρ C I V v n = ρ C v 1 ρ C v 2 ρ C v 3 … ρ C v n

△

Notice that this definition is primarily about a single vector space ( V ) and two bases of V ( B , C ). The linear transformation ( I V ) is necessary but not critical. As you might expect, this matrix has something to do with changing bases. Here is the theorem that gives the matrix its name (not the other way around).

Theorem  CB
Change-of-Basis
Suppose that v is a vector in the vector space V and B and C are bases of V . Then

ρ C v = C B , C ρ B v

□

Proof  

ρ C v = ρ C I V v   Definition IDLT = M B , C I V ρ B v   Theorem FTMR = C B , C ρ B v   Definition CBM

■

So the change-of-basis matrix can be used with matrix multiplication to convert a vector representation of a vector ( v ) relative to one basis ( ρ B v ) to a representation of the same vector relative to a second basis ( ρ C v ).

Theorem  ICBM
Inverse of Change-of-Basis Matrix
Suppose that V is a vector space, and B and C are bases of V . Then the change-of-basis matrix C B , C is nonsingular and

C B , C − 1 = C C , B

□

Proof   The linear transformation I V : V → V is invertible, and its inverse is itself, I V (check this!). So by Theorem IMR, the matrix M B , C I V = C B , C is invertible. Theorem NI says an invertible matrix is nonsingular.

Then

C B , C − 1 = M B , C I V − 1   Definition CBM = M C , B I V − 1    Theorem IMR = M C , B I V   Definition IDLT = C C , B   Definition CBM

■

Example  CBP
Change of basis with polynomials
The vector space P 4 (Example VSP) has two nice bases (Example BP),

B = 1 , x , x 2 , x 3 , x 4 C = 1 , 1 + x , 1 + x + x 2 , 1 + x + x 2 + x 3 , 1 + x + x 2 + x 3 + x 4

To build the change-of-basis matrix between B and C , we must first build a vector representation of each vector in B relative to C ,

ρ C 1 = ρ C ( 1 ) 1 = 1 0 0 0 0 ρ C x = ρ C ( − 1 ) 1 + ( 1 ) 1 + x = − 1 1 0 0 0 ρ C x 2 = ρ C ( − 1 ) 1 + x + ( 1 ) 1 + x + x 2 = 0 − 1 1 0 0 ρ C x 3 = ρ C ( − 1 ) 1 + x + x 2 + ( 1 ) 1 + x + x 2 + x 3 = 0 0 − 1 1 0 ρ C x 4 = ρ C ( − 1 ) 1 + x + x 2 + x 3 + ( 1 ) 1 + x + x 2 + x 3 + x 4 = 0 0 0 − 1 1

Then we package up these vectors as the columns of a matrix,

C B , C = 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1

Now, to illustrate Theorem CB, consider the vector u = 5 − 3 x + 2 x 2 + 8 x 3 − 3 x 4 . We can build the representation of u relative to B easily,

ρ B u = ρ B 5 − 3 x + 2 x 2 + 8 x 3 − 3 x 4 = 5 − 3 2 8 − 3

Applying Theorem CB, we obtain a second representation of u , but now relative to C ,

ρ C u = C B , C ρ B u   Theorem CB = 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 5 − 3 2 8 − 3 = 8 − 5 − 6 1 1 − 3   Definition MVP

We can check our work by unraveling this second representation,

u = ρ C − 1 ρ C u   Definition IVLT = ρ C − 1 8 − 5 − 6 1 1 − 3 = 8 ( 1 ) + ( − 5 ) ( 1 + x ) + ( − 6 ) ( 1 + x + x 2 ) + ( 1 1 ) ( 1 + x + x 2 + x 3 ) + ( − 3 ) ( 1 + x + x 2 + x 3 + x 4 )   Definition VR = 5 − 3 x + 2 x 2 + 8 x 3 − 3 x 4

The change-of-basis matrix from C to B is actually easier to build. Grab each vector in the basis C and form its representation relative to B

ρ B 1 = ρ B ( 1 ) 1 = 1 0 0 0 0 ρ B 1 + x = ρ B ( 1 ) 1 + ( 1 ) x = 1 1 0 0 0 ρ B 1 + x + x 2 = ρ B ( 1 ) 1 + ( 1 ) x + ( 1 ) x 2 = 1 1 1 0 0 ρ B 1 + x + x 2 + x 3 = ρ B ( 1 ) 1 + ( 1 ) x + ( 1 ) x 2 + ( 1 ) x 3 = 1 1 1 1 0 ρ B 1 + x + x 2 + x 3 + x 4 = ρ B ( 1 ) 1 + ( 1 ) x + ( 1 ) x 2 + ( 1 ) x 3 + ( 1 ) x 4 = 1 1 1 1 1

Then we package up these vectors as the columns of a matrix,

C C , B = 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 0 0 0 0 1

We formed two representations of the vector u above, so we can again provide a check on our computations by converting from the representation of u relative to C to the representation of u relative to B ,

ρ B u = C C , B ρ C u   Theorem CB = 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 0 0 0 0 1 8 − 5 − 6 1 1 − 3 = 5 − 3 2 8 − 3   Definition MVP

One more computation that is either a check on our work, or an illustration of a theorem. The two change-of-basis matrices, C B , C and C C , B , should be inverses of each other, according to Theorem ICBM. Here we go,

C B , C C C , B = 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 − 1 0 0 0 0 1 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 0 0 0 0 1 = 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1

⊠

The computations of the previous example are not meant to present any labor-saving devices, but instead are meant to illustrate the utility of the change-of-basis matrix. However, you might have noticed that C C , B was easier to compute than C B , C . If you needed C B , C , then you could first compute C C , B and then compute its inverse, which by Theorem ICBM, would equal C B , C .

Here’s another illustrative example. We have been concentrating on working with abstract vector spaces, but all of our theorems and techniques apply just as well to ℂ m , the vector space of column vectors. We only need to use more complicated bases than the standard unit vectors (Theorem SUVB) to make things interesting.

Example  CBCV
Change of basis with column vectors
For the vector space ℂ 4 we have the two bases,

B = 1 − 2 1 − 2 , − 1 3 1 1 , 2 − 3 3 − 4 , − 1 3 3 0 C = 1 − 6 − 4 − 1 , − 4 8 − 5 8 , − 5 1 3 − 2 9 , 3 − 7 3 − 6

The change-of-basis matrix from B to C requires writing each vector of B as a linear combination the vectors in C ,

ρ C 1 − 2 1 − 2 = ρ C ( 1 ) 1 − 6 − 4 − 1 + ( − 2 ) − 4 8 − 5 8 + ( 1 ) − 5 1 3 − 2 9 + ( − 1 ) 3 − 7 3 − 6 = 1 − 2 1 − 1 ρ C − 1 3 1 1 = ρ C ( 2 ) 1 − 6 − 4 − 1 + ( − 3 ) − 4 8 − 5 8 + ( 3 ) − 5 1 3 − 2 9 + ( 0 ) 3 − 7 3 − 6 = 2 − 3 3 0 ρ C 2 − 3 3 − 4 = ρ C ( 1 ) 1 − 6 − 4 − 1 + ( − 3 ) − 4 8 − 5 8 + ( 1 ) − 5 1 3 − 2 9 + ( − 2 ) 3 − 7 3 − 6 = 1 − 3 1 − 2 ρ C − 1 3 3 0 = ρ C ( 2 ) 1 − 6 − 4 − 1 + ( − 2 ) − 4 8 − 5 8 + ( 4 ) − 5 1 3 − 2 9 + ( 3 ) 3 − 7 3 − 6 = 2 − 2 4 3

Then we package these vectors up as the change-of-basis matrix,

C B , C = 1 2 1 2 − 2 − 3 − 3 − 2 1 3 1 4 − 1 0 − 2 3

Now consider a single (arbitrary) vector y = 2 6 − 3 4 . First,build the vector representation of y relative to B . This will require writing y as a linear combination of the vectors in B ,

ρ B y = ρ B 2 6 − 3 4 = ρ B ( − 2 1 ) 1 − 2 1 − 2 + ( 6 ) − 1 3 1 1 + ( 1 1 ) 2 − 3 3 − 4 + ( − 7 ) − 1 3 3 0 = − 2 1 6 1 1 − 7

Now, applying Theorem CB we can convert the representation of y relative to B into a representation relative to C ,

ρ C y = C B , C ρ B y   Theorem CB = 1 2 1 2 − 2 − 3 − 3 − 2 1 3 1 4 − 1 0 − 2 3 − 2 1 6 1 1 − 7 = − 1 2 5 − 2 0 − 2 2   Definition MVP

We could continue further with this example, perhaps by computing the representation of y relative to the basis C directly as a check on our work ( Exercise CB.C20). Or we could choose another vector to play the role of y and compute two different representations of this vector relative to the two bases B and C . ⊠

Subsection MRS: Matrix Representations and Similarity

Here is the main theorem of this section. It looks a bit involved at first glance, but the proof should make you realize it is not all that complicated. In any event, we are more interested in a special case.

Theorem  MRCB
Matrix Representation and Change of Basis
Suppose that T : U → V is a linear transformation, B and C are bases for U , and D and E are bases for V . Then

M B , D T = C E , D M C , E T C B , C

□

Proof  

C E , D M C , E T C B , C = M E , D I V M C , E T M B , C I U   Definition CBM = M E , D I V M B , E T ∘ I U   Theorem MRCLT = M E , D I V M B , E T   Definition IDLT = M B , D I V ∘ T   Theorem MRCLT = M B , D T   Definition IDLT

■

We will be most interested in a special case of this theorem ( Theorem SCB), but here’s an example that illustrates the full generality of Theorem MRCB.

Example  MRCM
Matrix representations and change-of-basis matrices
Begin with two vector spaces, S 2 , the subspace of M 2 2 containing all 2 × 2 symmetric matrices, and P 3 (Example VSP), the vector space of all polynomials of degree 3 or less. Then define the linear transformation Q : S 2 → P 3 by

Q a b b c = ( 5 a − 2 b + 6 c ) + ( 3 a − b + 2 c ) x + ( a + 3 b − c ) x 2 + ( − 4 a + 2 b + c ) x 3

Here are two bases for each vector space, one nice, one nasty. First for S 2 ,

B = 5 − 3 − 3 − 2 , 2 − 3 − 3 0 , 1 2 2 4 C = 1 0 0 0 , 0 1 1 0 , 0 0 0 1

and then for P 3 ,

D = 2 + x − 2 x 2 + 3 x 3 , − 1 − 2 x 2 + 3 x 3 , − 3 − x + x 3 , − x 2 + x 3 E = 1 , x , x 2 , x 3

We’ll begin with a matrix representation of Q relative to C and E . We first find vector representations of the elements of C relative to E ,

ρ E Q 1 0 0 0 = ρ E 5 + 3 x + x 2 − 4 x 3 = 5 3 1 − 4 ρ E Q 0 1 1 0 = ρ E − 2 − x + 3 x 2 + 2 x 3 = − 2 − 1 3 2 ρ E Q 0 0 0 1 = ρ E 6 + 2 x − x 2 + x 3 = 6 2 − 1 1

So

M C , E Q = 5 − 2 6 3 − 1 2 1 3 − 1 − 4 2 1

Now we construct two change-of-basis matrices. First, C B , C requires vector representations of the elements of B , relative to C . Since C is a nice basis, this is straightforward,

ρ C 5 − 3 − 3 − 2 = ρ C ( 5 ) 1 0 0 0 + ( − 3 ) 0 1 1 0 + ( − 2 ) 0 0 0 1 = 5 − 3 − 2 ρ C 2 − 3 − 3 0 = ρ C ( 2 ) 1 0 0 0 + ( − 3 ) 0 1 1 0 + ( 0 ) 0 0 0 1 = 2 − 3 0 ρ C 1 2 2 4 = ρ C ( 1 ) 1 0 0 0 + ( 2 ) 0 1 1 0 + ( 4 ) 0 0 0 1 = 1 2 4

So

C B , C = 5 2 1 − 3 − 3 2 − 2 0 4

The other change-of-basis matrix we’ll compute is C E , D . However, since E is a nice basis (and D is not) we’ll turn it around and instead compute C D , E and apply Theorem ICBM to use an inverse to compute C E , D .

ρ E 2 + x − 2 x 2 + 3 x 3 = ρ E ( 2 ) 1 + ( 1 ) x + ( − 2 ) x 2 + ( 3 ) x 3 = 2 1 − 2 3 ρ E − 1 − 2 x 2 + 3 x 3 = ρ E ( − 1 ) 1 + ( 0 ) x + ( − 2 ) x 2 + ( 3 ) x 3 = − 1 0 − 2 3 ρ E − 3 − x + x 3 = ρ E ( − 3 ) 1 + ( − 1 ) x + ( 0 ) x 2 + ( 1 ) x 3 = − 3 − 1 0 1 ρ E − x 2 + x 3 = ρ E ( 0 ) 1 + ( 0 ) x + ( − 1 ) x 2 + ( 1 ) x 3 = 0 0 − 1 1

So, we can package these column vectors up as a matrix to obtain C D , E and then,

C E , D = C D , E − 1   Theorem ICBM = 2 − 1 − 3 0 1 0 − 1 0 − 2 − 2 0 − 1 3 3 1 1 − 1 = 1 − 2 1 1 − 2 5 − 1 − 1 1 − 3 1 1 2 − 6 − 1 0

We are now in a position to apply Theorem MRCB. The matrix representation of Q relative to B and D can be obtained as follows,

M B , D Q = C E , D M C , E Q C B , C   Theorem MRCB = 1 − 2 1 1 − 2 5 − 1 − 1 1 − 3 1 1 2 − 6 − 1 0 5 − 2 6 3 − 1 2 1 3 − 1 − 4 2 1 5 2 1 − 3 − 3 2 − 2 0 4 = 1 − 2 1 1 − 2 5 − 1 − 1 1 − 3 1 1 2 − 6 − 1 0 1 9 1 6 2 5 1 4 9 9 − 2 − 7 3 − 2 8 − 1 4 4 = − 3 9 − 2 3 1 4 6 2 3 4 − 1 2 − 5 3 − 3 2 5 − 4 4 − 1 5 − 7

Now check our work by computing M B , D Q directly ( Exercise CB.C21). ⊠

Here is a special case of the previous theorem, where we choose U and V to be the same vector space, so the matrix representations and the change-of-basis matrices are all square of the same size.

Theorem  SCB
Similarity and Change of Basis
Suppose that T : V → V is a linear transformation and B and C are bases of V . Then

M B , B T = C B , C − 1 M C , C T C B , C

□

Proof   In the conclusion of Theorem MRCB, replace D by B , and replace E by C ,

M B , B T = C C , B M C , C T C B , C   Theorem MRCB = C B , C − 1 M C , C T C B , C   Theorem ICBM

■

This is the third surprise of this chapter. Theorem SCB considers the special case where a linear transformation has the same vector space for the domain and codomain ( V ). We build a matrix representation of T using the basis B simultaneously for both the domain and codomain ( M B , B T ), and then we build a second matrix representation of T , now using the basis C for both the domain and codomain ( M C , C T ). Then these two representations are related via a similarity transformation (Definition SIM) using a change-of-basis matrix ( C B , C )!

Example  MRBE
Matrix representation with basis of eigenvectors
We return to the linear transformation T : M 2 2 → M 2 2 of Example ELTBM defined by

T a b c d = − 1 7 a + 1 1 b + 8 c − 1 1 d − 5 7 a + 3 5 b + 2 4 c − 3 3 d − 1 4 a + 1 0 b + 6 c − 1 0 d − 4 1 a + 2 5 b + 1 6 c − 2 3 d

In Example ELTBM we showcased four eigenvectors of T . We will now put these four vectors in a set,

B = x 1 , x 2 , x 3 , x 4 = 0 1 0 1 , 1 1 1 0 , 1 3 2 3 , 2 6 1 4

Check that B is a basis of M 2 2 by first establishing the linear independence of B and then employing Theorem G to get the spanning property easily. Here is a second set of 2 × 2 matrices, which also forms a basis of M 2 2 (Example BM),

C = y 1 , y 2 , y 3 , y 4 = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

We can build two matrix representations of T , one relative to B and one relative to C . Each is easy, but for wildly different reasons. In our computation of the matrix representation relative to B we borrow some of our work in Example ELTBM. Here are the representations, then the explanation.

ρ B T x 1 = ρ B 2 x 1 = ρ B 2 x 1 + 0 x 2 + 0 x 3 + 0 x 4 = 2 0 0 0 ρ B T x 2 = ρ B 2 x 2 = ρ B 0 x 1 + 2 x 2 + 0 x 3 + 0 x 4 = 0 2 0 0 ρ B T x 3 = ρ B ( − 1 ) x 3 = ρ B 0 x 1 + 0 x 2 + ( − 1 ) x 3 + 0 x 4 = 0 0 − 1 0 ρ B T x 4 = ρ B ( − 2 ) x 4 = ρ B 0 x 1 + 0 x 2 + 0 x 3 + ( − 2 ) x 4 = 0 0 0 − 2

So the resulting representation is

M B , B T = 2 0 0 0 0 2 0 0 0 0 − 1 0 0 0 0 − 2

Very pretty. Now for the matrix representation relative to C first compute,

ρ C T y 1 = ρ C − 1 7 − 5 7 − 1 4 − 4 1 = ρ C ( − 1 7 ) 1 0 0 0 + ( − 5 7 ) 0 1 0 0 + ( − 1 4 ) 0 0 1 0 + ( − 4 1 ) 0 0 0 1 = − 1 7 − 5 7 − 1 4 − 4 1 ρ C T y 2 = ρ C 1 1 3 5 1 0 2 5 = ρ C 1 1 1 0 0 0 + 3 5 0 1 0 0 + 1 0 0 0 1 0 + 2 5 0 0 0 1 = 1 1 3 5 1 0 2 5 ρ C T y 3 = ρ C 8 2 4 6 1 6 = ρ C 8 1 0 0 0 + 2 4 0 1 0 0 + 6 0 0 1 0 + 1 6 0 0 0 1 = 8 2 4 6 1 6 ρ C T y 4 = ρ C − 1 1 − 3 3 − 1 0 − 2 3 = ρ C ( − 1 1 ) 1 0 0 0 + ( − 3 3 ) 0 1 0 0 + ( − 1 0 ) 0 0 1 0 + ( − 2 3 ) 0 0 0 1 = − 1 1 − 3 3 − 1 0 − 2 3

So the resulting representation is

M C , C T = − 1 7 1 1 8 − 1 1 − 5 7 3 5 2 4 − 3 3 − 1 4 1 0 6 − 1 0 − 4 1 2 5 1 6 − 2 3

Not quite as pretty. The purpose of this example is to illustrate Theorem SCB. This theorem says that the two matrix representations, M B , B T and M C , C T , of the one linear transformation, T , are related by a similarity transformation using the change-of-basis matrix C B , C . Lets compute this change-of-basis matrix. Notice that since C is such a nice basis, this is fairly straightforward,

ρ C x 1 = ρ C 0 1 0 1 = ρ C 0 1 0 0 0 + 1 0 1 0 0 + 0 0 0 1 0 + 1 0 0 0 1 = 0 1 0 1 ρ C x 2 = ρ C 1 1 1 0 = ρ C 1 1 0 0 0 + 1 0 1 0 0 + 1 0 0 1 0 + 0 0 0 0 1 = 1 1 1 0 ρ C x 3 = ρ C 1 3 2 3 = ρ C 1 1 0 0 0 + 3 0 1 0 0 + 2 0 0 1 0 + 3 0 0 0 1 = 1 3 2 3 ρ C x 4 = ρ C 2 6 1 4 = ρ C 2 1 0 0 0 + 6 0 1 0 0 + 1 0 0 1 0 + 4 0 0 0 1 = 2 6 1 4

So we have,

C B , C = 0 1 1 2 1 1 3 6 0 1 2 1 1 0 3 4

Now, according to Theorem SCB we can write,

M B , B T = C B , C − 1 M C , C T C B , C 2 0 0 0 0 2 0 0 0 0 − 1 0 0 0 0 − 2 = 0 1 1 2 1 1 3 6 0 1 2 1 1 0 3 4 − 1 − 1 7 1 1 8 − 1 1 − 5 7 3 5 2 4 − 3 3 − 1 4 1 0 6 − 1 0 − 4 1 2 5 1 6 − 2 3 0 1 1 2 1 1 3 6 0 1 2 1 1 0 3 4

This should look and feel exactly like the process for diagonalizing a matrix, as was described in Section SD. And it is. ⊠

We can now return to the question of computing an eigenvalue or eigenvector of a linear transformation. For a linear transformation of the form T : V → V , we know that representations relative to different bases are similar matrices. We also know that similar matrices have equal characteristic polynomials by Theorem SMEE. We will now show that eigenvalues of a linear transformation T are precisely the eigenvalues of any matrix representation of T . Since the choice of a different matrix representation leads to a similar matrix, there will be no “new” eigenvalues obtained from this second representation. Similarly, the change-of-basis matrix can be used to show that eigenvectors obtained from one matrix representation will be precisely those obtained from any other representation. So we can determine the eigenvalues and eigenvectors of a linear transformation by forming one matrix representation, using any basis we please, and analyzing the matrix in the manner of Chapter E.

Theorem  EER
Eigenvalues, Eigenvectors, Representations
Suppose that T : V → V is a linear transformation and B is a basis of V . Then v ∈ V is an eigenvector of T for the eigenvalue λ if and only if ρ B v is an eigenvector of M B , B T for the eigenvalue λ . □

Proof   ( ⇒ ) Assume that v ∈ V is an eigenvector of T for the eigenvalue λ . Then

M B , B T ρ B v = ρ B T v   Theorem FTMR = ρ B λ v   Definition EELT = λ ρ B v   Theorem VRLT

which by Definition EEM says that ρ B v is an eigenvector of the matrix M B , B T for the eigenvalue λ .

( ⇐ ) Assume that ρ B v is an eigenvector of M B , B T for the eigenvalue λ . Then

T v = ρ B − 1 ρ B T v   Definition IVLT = ρ B − 1 M B , B T ρ B v   Theorem FTMR = ρ B − 1 λ ρ B v   Definition EEM = λ ρ B − 1 ρ B v   Theorem ILTLT = λ v   Definition IVLT

which by Definition EELT says v is an eigenvector of T for the eigenvalue λ . ■

Subsection CELT: Computing Eigenvectors of Linear Transformations

Knowing that the eigenvalues of a linear transformation are the eigenvalues of any representation, no matter what the choice of the basis B might be, we could now unambiguously define items such as the characteristic polynomial of a linear transformation, rather than a matrix. We’ll say that again — eigenvalues, eigenvectors, and characteristic polynomials are intrinsic properties of a linear transformation, independent of the choice of a basis used to construct a matrix representation.

As a practical matter, how does one compute the eigenvalues and eigenvectors of a linear transformation of the form T : V → V ? Choose a nice basis B for V , one where the vector representations of the values of the linear transformations necessary for the matrix representation are easy to compute. Construct the matrix representation relative to this basis, and find the eigenvalues and eigenvectors of this matrix using the techniques of Chapter E. The resulting eigenvalues of the matrix are precisely the eigenvalues of the linear transformation. The eigenvectors of the matrix are column vectors that need to be converted to vectors in V through application of ρ B − 1 .

Now consider the case where the matrix representation of a linear transformation is diagonalizable. The n linearly independent eigenvectors that must exist for the matrix (Theorem DC) can be converted (via ρ B − 1 ) into eigenvectors of the linear transformation. A matrix representation of the linear transformation relative to a basis of eigenvectors will be a diagonal matrix — an especially nice representation! Though we did not know it at the time, the diagonalizations of Section SD were really finding especially pleasing matrix representations of linear transformations.

Here are some examples.

Example  ELTT
Eigenvectors of a linear transformation, twice
Consider the linear transformation S : M 2 2 → M 2 2 defined by

S a b c d = − b − c − 3 d − 1 4 a − 1 5 b − 1 3 c + d 1 8 a + 2 1 b + 1 9 c + 3 d − 6 a − 7 b − 7 c − 3 d

To find the eigenvalues and eigenvectors of S we will build a matrix representation and analyze the matrix. Since Theorem EER places no restriction on the choice of the basis B , we may as well use a basis that is easy to work with. So set

B = x 1 , x 2 , x 3 , x 4 = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

Then to build the matrix representation of S relative to B compute,

ρ B S x 1 = ρ B 0 − 1 4 1 8 − 6 = ρ B 0 x 1 + ( − 1 4 ) x 2 + 1 8 x 3 + ( − 6 ) x 4 = 0 − 1 4 1 8 − 6 ρ B S x 2 = ρ B − 1 − 1 5 2 1 − 7 = ρ B ( − 1 ) x 1 + ( − 1 5 ) x 2 + 2 1 x 3 + ( − 7 ) x 4 = − 1 − 1 5 2 1 − 7 ρ B S x 3 = ρ B − 1 − 1 3 1 9 − 7 = ρ B ( − 1 ) x 1 + ( − 1 3 ) x 2 + 1 9 x 3 + ( − 7 ) x 4 = − 1 − 1 3 1 9 − 7 ρ B S x 4 = ρ B − 3 1 3 − 3 = ρ B ( − 3 ) x 1 + 1 x 2 + 3 x 3 + ( − 3 ) x 4 = − 3 1 3 − 3

So by Definition MR we have

M = M B , B S = 0 − 1 − 1 − 3 − 1 4 − 1 5 − 1 3 1 1 8 2 1 1 9 3 − 6 − 7 − 7 − 3

Now compute eigenvalues and eigenvectors of the matrix representation of M with the techniques of Section EE. First the characteristic polynomial,

p M x = det M − x I 4 = x 4 − x 3 − 1 0 x 2 + 4 x + 2 4 = ( x − 3 ) ( x − 2 ) ( x + 2 ) 2

We could now make statements about the eigenvalues of M , but in light of Theorem EER we can refer to the eigenvalues of S and mildly abuse (or extend) our notation for multiplicities to write

α S 3 = 1 α S 2 = 1 α S − 2 = 2

Now compute the eigenvectors of M ,

λ = 3 M − 3 I 4 = − 3 − 1 − 1 − 3 − 1 4 − 1 8 − 1 3 1 1 8 2 1 1 6 3 − 6 − 7 − 7 − 6 →  RREF 1 0 0 1 0 1  0 − 3 0 0 1 3 0 0 0 0 ℰ M 3 = N M − 3 I 4 = − 1 3 − 3 1

λ = 2 M − 2 I 4 = − 2 − 1 − 1 − 3 − 1 4 − 1 7 − 1 3 1 1 8 2 1 1 7 3 − 6 − 7 − 7 − 5 →  RREF 1 0 0 2 0 1  0 − 4 0 0 1 3 0 0 0 0 ℰ M 2 = N M − 2 I 4 = − 2 4 − 3 1

λ = − 2 M − ( − 2 ) I 4 = 2 − 1 − 1 − 3 − 1 4 − 1 3 − 1 3 1 1 8 2 1 2 1 3 − 6 − 7 − 7 − 1 →  RREF 1 0 0 − 1 0 1  1 1 0 0 0 0 0 0 0 0 ℰ M − 2 = N M − ( − 2 ) I 4 = 0 − 1 1 0 , 1 − 1 0 1

According to Theorem EER the eigenvectors just listed as basis vectors for the eigenspaces of M are vector representations (relative to B ) of eigenvectors for S . So the application if the inverse function ρ B − 1 will convert these column vectors into elements of the vector space M 2 2 ( 2 × 2 matrices) that are eigenvectors of S . Since ρ B is an isomorphism (Theorem VRILT), so is ρ B − 1 . Applying the inverse function will then preserve linear independence and spanning properties, so with a sweeping application of the Coordinatization Principle and some extensions of our previous notation for eigenspaces and geometric multiplicities, we can write,

ρ B − 1 − 1 3 − 3 1 = ( − 1 ) x 1 + 3 x 2 + ( − 3 ) x 3 + 1 x 4 = − 1 3 − 3 1 ρ B − 1 − 2 4 − 3 1 = ( − 2 ) x 1 + 4 x 2 + ( − 3 ) x 3 + 1 x 4 = − 2 4 − 3 1 ρ B − 1 0 − 1 1 0 = 0 x 1 + ( − 1 ) x 2 + 1 x 3 + 0 x 4 = 0 − 1 1 0 ρ B − 1 1 − 1 0 1 = 1 x 1 + ( − 1 ) x 2 + 0 x 3 + 1 x 4 = 1 − 1 0 1

So

ℰ S 3 = − 1 3 − 3 1 ℰ S 2 = − 2 4 − 3 1 ℰ S − 2 = 0 − 1 1 0 , 1 − 1 0 1

with geometric multiplicities given by

γ S 3 = 1 γ S 2 = 1 γ S − 2 = 2

Suppose we now decided to build another matrix representation of S , only now relative to a linearly independent set of eigenvectors of S , such as

C = − 1 3 − 3 1 , − 2 4 − 3 1 , 0 − 1 1 0 , 1 − 1 0 1

At this point you should have computed enough matrix representations to predict that the result of representing S relative to C will be a diagonal matrix. Computing this representation is an example of how Theorem SCB generalizes the diagonalizations from Section SD. For the record, here is the diagonal representation,

M C , C S = 3 0 0 0 0 2 0 0 0 0 − 2 0 0 0 0 − 2

Our interest in this example is not necessarily building nice representations, but instead we want to demonstrate how eigenvalues and eigenvectors are an intrinsic property of a linear transformation, independent of any particular representation. To this end, we will repeat the foregoing, but replace B by another basis. We will make this basis different, but not extremely so,

D = y 1 , y 2 , y 3 , y 4 = 1 0 0 0 , 1 1 0 0 , 1 1 1 0 , 1 1 1 1

Then to build the matrix representation of S relative to D compute,

ρ D S y 1 = ρ D 0 − 1 4 1 8 − 6 = ρ D 1 4 y 1 + ( − 3 2 ) y 2 + 2 4 y 3 + ( − 6 ) y 4 = 1 4 − 3 2 2 4 − 6 ρ D S y 2 = ρ D − 1 − 2 9 3 9 − 1 3 = ρ D 2 8 y 1 + ( − 6 8 ) y 2 + 5 2 y 3 + ( − 1 3 ) y 4 = 2 8 − 6 8 5 2 − 1 3 ρ D S y 3 = ρ D − 2 − 4 2 5 8 − 2 0 = ρ D 4 0 y 1 + ( − 1 0 0 ) y 2 + 7 8 y 3 + ( − 2 0 ) y 4 = 4 0 − 1 0 0 7 8 − 2 0 ρ D S y 4 = ρ D − 5 − 4 1 6 1 − 2 3 = ρ D 3 6 y 1 + ( − 1 0 2 ) y 2 + 8 4 y 3 + ( − 2 3 ) y 4 = 3 6 − 1 0 2 8 4 − 2 3

So by Definition MR we have

N = M D , D S = 1 4 2 8 4 0 3 6 − 3 2 − 6 8 − 1 0 0 − 1 0 2 2 4 5 2 7 8 8 4 − 6 − 1 3 − 2 0 − 2 3

Now compute eigenvalues and eigenvectors of the matrix representation of N with the techniques of Section EE. First the characteristic polynomial,

p N x = det N − x I 4 = x 4 − x 3 − 1 0 x 2 + 4 x + 2 4 = ( x − 3 ) ( x − 2 ) ( x + 2 ) 2

Of course this is not news. We now know that M = M B , B S and N = M D , D S are similar matrices (Theorem SCB). But Theorem SMEE told us long ago that similar matrices have identical characteristic polynomials. Now compute eigenvectors for the matrix representation, which will be different than what we found for M ,

λ = 3 N − 3 I 4 = 1 1 2 8 4 0 3 6 − 3 2 − 7 1 − 1 0 0 − 1 0 2 2 4 5 2 7 5 8 4 − 6 − 1 3 − 2 0 − 2 6 →  RREF 1 0 0 4 0 1 0 − 6 0 0 1 4 0 0 0 0 ℰ N 3 = N N − 3 I 4 = − 4 6 − 4 1

λ = 2 N − 2 I 4 = 1 2 2 8 4 0 3 6 − 3 2 − 7 0 − 1 0 0 − 1 0 2 2 4 5 2 7 6 8 4 − 6 − 1 3 − 2 0 − 2 5 →  RREF 1 0 0 6 0 1 0 − 7 0 0 1 4 0 0 0 0 ℰ N 2 = N N − 2 I 4 = − 6 7 − 4 1

λ = − 2 N − ( − 2 ) I 4 = 1 6 2 8 4 0 3 6 − 3 2 − 6 6 − 1 0 0 − 1 0 2 2 4 5 2 8 0 8 4 − 6 − 1 3 − 2 0 − 2 1 →  RREF 1 0 − 1 − 3 0 1 2 3 0 0 0 0 0 0 0 0 ℰ N − 2 = N N − ( − 2 ) I 4 = 1 − 2 1 0 , 3 − 3 0 1

Employing Theorem EER we can apply ρ D − 1 to each of the basis vectors of the eigenspaces of N to obtain eigenvectors for S that also form bases for eigenspaces of S ,

ρ D − 1 − 4 6 − 4 1 = ( − 4 ) y 1 + 6 y 2 + ( − 4 ) y 3 + 1 y 4 = − 1 3 − 3 1 ρ D − 1 − 6 7 − 4 1 = ( − 6 ) y 1 + 7 y 2 + ( − 4 ) y 3 + 1 y 4 = − 2 4 − 3 1 ρ D − 1 1 − 2 1 0 = 1 y 1 + ( − 2 ) y 2 + 1 y 3 + 0 y 4 = 0 − 1 1 0 ρ D − 1 3 − 3 0 1 = 3 y 1 + ( − 3 ) y 2 + 0 y 3 + 1 y 4 = 1 − 2 1 1

The eigenspaces for the eigenvalues of algebraic multiplicity 1 are exactly as before,

ℰ S 3 = − 1 3 − 3 1 ℰ S 2 = − 2 4 − 3 1

However, the eigenspace for λ = − 2 would at first glance appear to be different. Here are the two eigenspaces for λ = − 2 , first the eigenspace obtained from M = M B , B S , then followed by the eigenspace obtained from M = M D , D S .

ℰ S − 2 = 0 − 1 1 0 , 1 − 1 0 1 ℰ S − 2 = 0 − 1 1 0 , 1 − 2 1 1

Subspaces generally have many bases, and that is the situation here. With a careful proof of set equality, you can show that these two eigenspaces are equal sets. The key observation to make such a proof go is that

1 − 2 1 1 = 0 − 1 1 0 + 1 − 1 0 1

which will establish that the second set is a subset of the first. With equal dimensions, Theorem EDYES will finish the task. So the eigenvalues of a linear transformation are independent of the matrix representation employed to compute them! ⊠

Another example, this time a bit larger and with complex eigenvalues.

Example  CELT
Complex eigenvectors of a linear transformation
Consider the linear transformation Q : P 4 → P 4 defined by

Q a + b x + c x 2 + d x 3 + e x 4 = ( − 4 6 a − 2 2 b + 1 3 c + 5 d + e ) + ( 1 1 7 a + 5 7 b − 3 2 c − 1 5 d − 4 e ) x + ( − 6 9 a − 2 9 b + 2 1 c − 7 e ) x 2 + ( 1 5 9 a + 7 3 b − 4 4 c − 1 3 d + 2 e ) x 3 + ( − 1 9 5 a − 8 7 b + 5 5 c + 1 0 d − 1 3 e ) x 4

Choose a simple basis to compute with, say

B = 1 , x , x 2 , x 3 , x 4

Then it should be apparent that the matrix representation of Q relative to B is

M = M B , B Q = − 4 6 − 2 2 1 3 5 1 1 1 7 5 7 − 3 2 − 1 5 − 4 − 6 9 − 2 9 2 1 0 − 7 1 5 9 7 3 − 4 4 − 1 3 2 − 1 9 5 − 8 7 5 5 1 0 − 1 3

Compute the characteristic polynomial, eigenvalues and eigenvectors according to the techniques of Section EE,

p Q x = − x 5 + 6 x 4 − x 3 − 8 8 x 2 + 2 5 2 x − 2 0 8 = − ( x − 2 ) 2 ( x + 4 ) x 2 − 6 x + 1 3 = − ( x − 2 ) 2 ( x + 4 ) x − ( 3 + 2 i ) x − ( 3 − 2 i )

α Q 2 = 2 α Q − 4 = 1 α Q 3 + 2 i = 1 α Q 3 − 2 i = 1

λ = 2 M − ( 2 ) I 5 = − 4 8 − 2 2 1 3 5 1 1 1 7 5 5 − 3 2 − 1 5 − 4 − 6 9 − 2 9 1 9 0 − 7 1 5 9 7 3 − 4 4 − 1 5 2 − 1 9 5 − 8 7 5 5 1 0 − 1 5 →  RREF 1 0 0 1 2 − 1 2 0 1 0 − 5 2 − 5 2 0 0 1 − 2 − 6 0 0 0 0 0 0 0 0 0 0 ℰ M 2 = N M − ( 2 ) I 5 = − 1 2 5 2 2 1 0 , 1 2 5 2 6 0 1 = − 1 5 4 2 0 , 1 5 1 2 0 2

λ = − 4 M − ( − 4 ) I 5 = − 4 2 − 2 2 1 3 5 1 1 1 7 6 1 − 3 2 − 1 5 − 4 − 6 9 − 2 9 2 5 0 − 7 1 5 9 7 3 − 4 4 − 9 2 − 1 9 5 − 8 7 5 5 1 0 − 9 →  RREF 1 0 0 0 1 0 1 0 0 − 3 0 0 1 0 − 1 0 0 0 1 − 2 0 0 0 0 0 ℰ M − 4 = N M − ( − 4 ) I 5 = − 1 3 1 2 1

λ = 3 + 2 i M − ( 3 + 2 i ) I 5 = − 4 9 − 2 i − 2 2 1 3 5 1 1 1 7 5 4 − 2 i − 3 2 − 1 5 − 4 − 6 9 − 2 9 1 8 − 2 i 0 − 7 1 5 9 7 3 − 4 4 − 1 6 − 2 i 2 − 1 9 5 − 8 7 5 5 1 0 − 1 6 − 2 i →  RREF 1 0 0 0 − 3 4 + i 4 0 1 0 0 7 4 − i 4 0 0 1 0 − 1 2 + i 2 0 0 0 1 7 4 − i 4 0 0 0 0 0 ℰ M 3 + 2 i = N M − ( 3 + 2 i ) I 5 = 3 4 − i 4 − 7 4 + i 4 1 2 − i 2 − 7 4 + i 4 1 = 3 − i − 7 + i 2 − 2 i − 7 + i 4

λ = 3 − 2 i M − ( 3 − 2 i ) I 5 = − 4 9 + 2 i − 2 2 1 3 5 1 1 1 7 5 4 + 2 i − 3 2 − 1 5 − 4 − 6 9 − 2 9 1 8 + 2 i 0 − 7 1 5 9 7 3 − 4 4 − 1 6 + 2 i 2 − 1 9 5 − 8 7 5 5 1 0 − 1 6 + 2 i →  RREF 1 0 0 0 − 3 4 − i 4 0 1 0 0 7 4 + i 4 0 0 1 0 − 1 2 − i 2 0 0 0 1 7 4 + i 4 0 0 0 0 0 ℰ M 3 − 2 i = N M − ( 3 − 2 i ) I 5 = 3 4 + i 4 − 7 4 − i 4 1 2 + i 2 − 7 4 − i 4 1 = 3 + i − 7 − i 2 + 2 i − 7 − i 4

M

P 4

ρ B − 1

ρ B − 1 − 1 5 4 2 0 = − 1 + 5 x + 4 x 2 + 2 x 3 ρ B − 1 1 5 1 2 0 2 = 1 + 5 x + 1 2 x 2 + 2 x 4 ρ B − 1 − 1 3 1 2 1 = − 1 + 3 x + x 2 + 2 x 3 + x 4 ρ B − 1 3 − i − 7 + i 2 − 2 i − 7 + i 4 = ( 3 − i ) + ( − 7 + i ) x + ( 2 − 2 i ) x 2 + ( − 7 + i ) x 3 + 4 x 4 ρ B − 1 3 + i − 7 − i 2 + 2 i − 7 − i 4 = ( 3 + i ) + ( − 7 − i ) x + ( 2 + 2 i ) x 2 + ( − 7 − i ) x 3 + 4 x 4

So we apply Theorem EER and the Coordinatization Principle to get the eigenspaces for Q ,

ℰ Q 2 = − 1 + 5 x + 4 x 2 + 2 x 3 , 1 + 5 x + 1 2 x 2 + 2 x 4 ℰ Q − 4 = − 1 + 3 x + x 2 + 2 x 3 + x 4 ℰ Q 3 + 2 i = ( 3 − i ) + ( − 7 + i ) x + ( 2 − 2 i ) x 2 + ( − 7 + i ) x 3 + 4 x 4 ℰ Q 3 − 2 i = ( 3 + i ) + ( − 7 − i ) x + ( 2 + 2 i ) x 2 + ( − 7 − i ) x 3 + 4 x 4

with geometric multiplicities

γ Q 2 = 2 γ Q − 4 = 1 γ Q 3 + 2 i = 1 γ Q 3 − 2 i = 1

⊠

Subsection READ: Reading Questions

Subsection EXC: Exercises

C20 In Example CBCV we computed the vector representation of y relative to C , ρ C y , as an example of Theorem CB. Compute this same representation directly. In other words, apply Definition VR rather than Theorem CB.  
Contributed by  Robert Beezer

C21 Perform a check on Example MRCM by computing M B , D Q directly. In other words, apply Definition MR rather than Theorem MRCB.  
Contributed by  Robert Beezer Solution [ 1819]

C30 Find a basis for the vector space P 3 composed of eigenvectors of the linear transformation T . Then find a matrix representation of T relative to this basis.

T : P 3 → P 3 , T a + b x + c x 2 + d x 3 = ( a + c + d ) + ( b + c + d ) x + ( a + b + c ) x 2 + ( a + b + d ) x 3

Contributed by  Robert Beezer Solution [ 1820]

C40 Let S 2 2 be the vector space of 2 × 2 symmetric matrices. Find a basis B for S 2 2 that yields a diagonal matrix representation of the linear transformation R .

R : S 2 2 → S 2 2 , R a b b c = − 5 a + 2 b − 3 c − 1 2 a + 5 b − 6 c − 1 2 a + 5 b − 6 c 6 a − 2 b + 4 c

Contributed by  Robert Beezer Solution [ 1822]

C41 Let S 2 2 be the vector space of 2 × 2 symmetric matrices. Find a basis for S 2 2 composed of eigenvectors of the linear transformation Q : S 2 2 → S 2 2 .

Q a b b c = 2 5 a + 1 8 b + 3 0 c − 1 6 a − 1 1 b − 2 0 c − 1 6 a − 1 1 b − 2 0 c − 1 1 a − 9 b − 1 2 c

Contributed by  Robert Beezer Solution [ 1825]

T10 Suppose that T : V → V is an invertible linear transformation with a nonzero eigenvalue λ . Prove that 1 λ is an eigenvalue of T − 1 .  
Contributed by  Robert Beezer Solution [ 1827]

T15 Suppose that V is a vector space and T : V → V is a linear transformation. Prove that T is injective if and only if λ = 0 is not an eigenvalue of T .  
Contributed by  Robert Beezer

Subsection SOL: Solutions

C21 Contributed by  Robert Beezer Statement [ 1816]
Apply Definition MR,

ρ D Q 5 − 3 − 3 − 2 = ρ D 1 9 + 1 4 x − 2 x 2 − 2 8 x 3 = ρ D ( − 3 9 ) ( 2 + x − 2 x 2 + 3 x 3 ) + 6 2 ( − 1 − 2 x 2 + 3 x 3 ) + ( − 5 3 ) ( − 3 − x + x 3 ) + ( − 4 4 ) ( − x 2 + x 3 ) = − 3 9 6 2 − 5 3 − 4 4 ρ D Q 2 − 3 − 3 0 = ρ D 1 6 + 9 x − 7 x 2 − 1 4 x 3 = ρ D ( − 2 3 ) ( 2 + x − 2 x 2 + 3 x 3 ) + ( 3 4 ) ( − 1 − 2 x 2 + 3 x 3 ) + ( − 3 2 ) ( − 3 − x + x 3 ) + ( − 1 5 ) ( − x 2 + x 3 ) = − 2 3 3 4 − 3 2 − 1 5 ρ D Q 1 2 2 4 = ρ D 2 5 + 9 x + 3 x 2 + 4 x 3 = ρ D ( 1 4 ) ( 2 + x − 2 x 2 + 3 x 3 ) + ( − 1 2 ) ( − 1 − 2 x 2 + 3 x 3 ) + 5 ( − 3 − x + x 3 ) + ( − 7 ) ( − x 2 + x 3 ) = 1 4 − 1 2 5 − 7

These three vectors are the columns of the matrix representation,

M B , D Q = − 3 9 − 2 3 1 4 6 2 3 4 − 1 2 − 5 3 − 3 2 5 − 4 4 − 1 5 − 7

which coincides with the result obtained in Example MRCM.

C30 Contributed by  Robert Beezer Statement [ 1816]
With the domain and codomain being identical, we will build a matrix representation using the same basis for both the domain and codomain. The eigenvalues of the matrix representation will be the eigenvalues of the linear transformation, and we can obtain the eigenvectors of the linear transformation by un-coordinatizing ( Theorem EER). Since the method does not depend on which basis we choose, we can choose a natural basis for ease of computation, say,

B = 1 , x , x 2 , x 3

The matrix representation is then,

M B , B T = 1 0 1 1 0 1 1 1 1 1 1 0 1 1 0 1

The eigenvalues and eigenvectors of this matrix were computed in Example ESMS4. A basis for ℂ 4 , composed of eigenvectors of the matrix representation is,

C = 1 1 1 1 , − 1 1 0 0 , 0 0 − 1 1 , − 1 − 1 1 1

Applying ρ B − 1 to each vector of this set, yields a basis of P 3 composed of eigenvectors of T ,

D = 1 + x + x 2 + x 3 , − 1 + x , − x 2 + x 3 , − 1 − x + x 2 + x 3

The matrix representation of T relative to the basis D will be a diagonal matrix with the corresponding eigenvalues along the diagonal, so in this case we get

M D , D T = 3 0 0 0 0 1 0 0 0 0 1 0 0 0 0 − 1

C40 Contributed by  Robert Beezer Statement [ 1816]
Begin with a matrix representation of R , any matrix representation, but use the same basis for both instances of S 2 2 . We’ll choose a basis that makes it easy to compute vector representations in S 2 2 .

B = 1 0 0 0 , 0 1 1 0 , 0 0 0 1

Then the resulting matrix representation of R (Definition MR) is

M B , B R = − 5 2 − 3 − 1 2 5 − 6 6 − 2 4

Now, compute the eigenvalues and eigenvectors of this matrix, with the goal of diagonalizing the matrix (Theorem DC),

λ = 2 ℰ M B , B R 2 = − 1 − 2 1 λ = 1 ℰ M B , B R 1 = − 1 0 2 , 1 3 0

The three vectors that occur as basis elements for these eigenspaces will together form a linearly independent set (check this!). So these column vectors may be employed in a matrix that will diagonalize the matrix representation. If we “un-coordinatize” these three column vectors relative to the basis B , we will find three linearly independent elements of S 2 2 that are eigenvectors of the linear transformation R ( Theorem EER). A matrix representation relative to this basis of eigenvectors will be diagonal, with the eigenvalues ( λ = 2 , 1 ) as the diagonal elements. Here we go,

ρ B − 1 − 1 − 2 1 = ( − 1 ) 1 0 0 0 + ( − 2 ) 0 1 1 0 + 1 0 0 0 1 = − 1 − 2 − 2 1 ρ B − 1 − 1 0 2 = ( − 1 ) 1 0 0 0 + 0 0 1 1 0 + 2 0 0 0 1 = − 1 0 0 2 ρ B − 1 1 3 0 = 1 1 0 0 0 + 3 0 1 1 0 + 0 0 0 0 1 = 1 3 3 0

So the requested basis of S 2 2 , yielding a diagonal matrix representation of R , is

− 1 − 2 − 2 1 − 1 0 0 2 , 1 3 3 0

C41 Contributed by  Robert Beezer Statement [ 1817]
Use a single basis for both the domain and codomain, since they are equal.

B = 1 0 0 0 , 0 1 1 0 , 0 0 0 1

The matrix representation of Q relative to B is

M = M B , B Q = 2 5 1 8 3 0 − 1 6 − 1 1 − 2 0 − 1 1 − 9 − 1 2

We can analyze this matrix with the techniques of Section EE and then apply Theorem EER. The eigenvalues of this matrix are λ = − 2 , 1 , 3 with eigenspaces

ℰ M − 2 = − 6 4 3 ℰ M 1 = − 2 1 1 ℰ M 3 = − 3 2 1

Because the three eigenvalues are distinct, the three basis vectors from the three eigenspaces for a linearly independent set ( Theorem EDELI). Theorem EER says we can uncoordinatize these eigenvectors to obtain eigenvectors of Q . By Theorem ILTLI the resulting set will remain linearly independent. Set

C = ρ B − 1 − 6 4 3 , ρ B − 1 − 2 1 1 , ρ B − 1 − 3 2 1 = − 6 4 4 3 , − 2 1 1 1 , − 3 2 2 1

Then C is a linearly independent set of size 3 in the vector space M 2 2 , which has dimension 3 as well. By Theorem G, C is a basis of M 2 2 .

T10 Contributed by  Robert Beezer Statement [ 1818]
Let v be an eigenvector of T for the eigenvalue λ . Then,

T − 1 v = 1 λ λ T − 1 v   λ ≠ 0 = 1 λ T − 1 λ v   Theorem ILTLT = 1 λ T − 1 T v   v  eigenvector of  T = 1 λ I V v   Definition IVLT = 1 λ v   Definition IDLT

which says that 1 λ is an eigenvalue of T − 1 with eigenvector v . Note that it is possible to prove that any eigenvalue of an invertible linear transformation is never zero. So the hypothesis that λ be nonzero is just a convenience for this problem.

Section OD  Orthonormal Diagonalization

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s s e c t i o n i s i n d r a f t f o r m
T h e o r e m s & d e f i n i t i o n s a r e c o m p l e t e, n e e d s e x a m p l e s

We have seen in Section SD that under the right conditions a square matrix is similar to a diagonal matrix. We recognize now, via Theorem SCB, that a similarity transformation is a change of basis on a matrix representation. So we can now discuss the choice of a basis used to build a matrix representation, and decide if some bases are better than others for this purpose. This will be the tone of this section. We will also see that every matrix has a reasonably useful matrix representation, and we will discover a new class of diagonalizable linear transformations. First we need some basic facts about triangular matrices.

Parent topic: Chapter R  Representations

Previous topic: Section CB  Change of Basis

Next topic: Section NLT  Nilpotent Linear Transformations

Subsection TM: Triangular Matrices

An upper, or lower, triangular matrix is exactly what it sounds like it should be, but here are the two relevant definitions.

Definition  UTM
Upper Triangular Matrix
The n × n square matrix A is upper triangular if A i j = 0 whenever i > j . △

Definition  LTM
Lower Triangular Matrix
The n × n square matrix A is lower triangular if A i j = 0 whenever i < j . △

Obviously, properties of a lower triangular matrices will have analogues for upper triangular matrices. Rather than stating two very similar theorems, we will say that matrices are “triangular of the same type” as a convenient shorthand to cover both possibilities and then give a proof for just one type.

Theorem  PTMT
Product of Triangular Matrices is Triangular
Suppose that A and B are square matrices of size n that are triangular of the same type. Then A B is also triangular of that type. □

Proof   We prove this for lower triangular matrices and leave the proof for upper triangular matrices to you. Suppose that A and B are both lower triangular. We need only establish that certain entries of the product A B are zero. Suppose that i < j , then

A B i j = ∑ k = 1 n A i k B k j   Theorem EMP = ∑ k = 1 j − 1 A i k B k j + ∑ k = j n A i k B k j   Property AACN = ∑ k = 1 j − 1 A i k 0 + ∑ k = j n A i k B k j   k < j ,  Definition LTM = ∑ k = 1 j − 1 A i k 0 + ∑ k = j n 0 B k j   i < j ≤ k ,  Definition LTM = ∑ k = 1 j − 1 0 + ∑ k = j n 0 = 0

Since A B i j = 0 whenever i < j , by Definition LTM, A B is lower triangular. ■

The inverse of a triangular matrix is triangular, of the same type.

Theorem  ITMT
Inverse of a Triangular Matrix is Triangular
Suppose that A is a nonsingular matrix of size n that is triangular. Then the inverse of A , A − 1 , is triangular of the same type. Furthermore, the diagonal entries of A − 1 are the reciprocals of the corresponding diagonal entries of A . More precisely, A − 1 i i = A i i − 1 . □

Proof   We give the proof for the case when A is lower triangular, and leave the case when A is upper triangular for you. Consider the process for computing the inverse of a matrix that is outlined in the proof of Theorem CINM. We augment A with the size n identity matrix, I n , and row-reduce the n × 2 n matrix to reduced row-echelon form via the algorithm in Theorem REMEF. The proof involves tracking the peculiarities of this process in the case of a lower triangular matrix. Let M = A I n .

First, none of the diagonal elements of A are zero. By repeated expansion about the first row, the determinant of a lower triangular matrix can be seen to be the product of the diagonal entries (Theorem DER). If just one of these diagonal elements was zero, then the determinant of A is zero and A is singular by Theorem SMZD. Slightly violating the exact algorithm for row reduction we can form a matrix, M ′ , that is row-equivalent to M , by multiplying row i by the nonzero scalar A i i − 1 , for 1 ≤ i ≤ n . This sets M ′ i i = 1 and M ′ i , n + 1 = A i i − 1 , and leaves every zero entry of M unchanged.

Let M j denote the matrix obtained form M ′ after converting column j to a pivot column. We can convert column j of M j − 1 into a pivot column with a set of n − j − 1 row operations of the form α R j + R k with j + 1 ≤ k ≤ n . The key observation here is that we add multiples of row j only to higher-numbered rows. This means that none of the entries in rows 1 through j − 1 is changed, and since row j has zeros in columns j + 1 through n , none of the entries in rows j + 1 through n is changed in columns j + 1 through n . The first n columns of M ′ form a lower triangular matrix with 1’s on the diagonal. In its conversion to the identity matrix through this sequence of row operations, it remains lower triangular with 1’s on the diagonal.

What happens in columns n + 1 through 2 n of M ′ ? These columns began in M as the identity matrix, and in M ′ each diagonal entry was scaled to a reciprocal of the corresponding diagonal entry of A . Notice that trivially, these final n columns of M ′ form a lower triangular matrix. Just as we argued for the first n columns, the row operations that convert M j − 1 into M j will preserve the lower triangular form in the final n columns and preserve the exact values of the diagonal entries. By Theorem CINM, the final n columns of M n is the inverse of A , and this matrix has the necessary properties advertised in the conclusion of this theorem. ■

Subsection UTMR: Upper Triangular Matrix Representation

Not every matrix is diagonalizable, but every linear transformation has a matrix representation that is an upper triangular matrix, and the basis that achieves this representation is especially pleasing. Here’s the theorem.

Theorem  UTMR
Upper Triangular Matrix Representation
Suppose that T : V → V is a linear transformation. Then there is a basis B for V such that the matrix representation of T relative to B , M B , B T , is an upper triangular matrix. Each diagonal entry is an eigenvalue of T , and if λ is an eigenvalue of T , then λ occurs α T λ times on the diagonal. □

Proof   We begin with a proof by induction ( Technique I) of the first statement in the conclusion of the theorem. We use induction on the dimension of V to show that if T : V → V is a linear transformation, then there is a basis B for V such that the matrix representation of T relative to B , M B , B T , is an upper triangular matrix.

To start suppose that dim V = 1 . Choose any nonzero vector v ∈ V and realize that V = v . Then we can determine T uniquely by T v = β v for some β ∈ ℂ (Theorem LTDB). This description of T also gives us a matrix representation relative to the basis B = v as the 1 × 1 matrix with lone entry equal to β . And this matrix representation is upper triangular ( Definition UTM).

For the induction step let dim V = m , and assume the theorem is true for every linear transformation defined on a vector space of dimension less than m . By Theorem EMHE (suitably converted to the setting of a linear transformation), T has at least one eigenvalue, and we denote this eigenvalue as λ . (We will remark later about how critical this step is.) We now consider properties of the linear transformation T − λ I V : V → V .

Let x be an eigenvector of T for λ . By definition x ≠ 0 . Then

T − λ I V x = T x − λ I V x   Theorem VSLT = T x − λ x   Definition IDLT = λ x − λ x   Definition EELT = 0   Property AI

So T − λ I V is not injective, as it has a nontrivial kernel ( Theorem KILT). With an application of Theorem RPNDD we bound the rank of T − λ I V ,

r T − λ I V = dim V − n T − λ I V ≤ m − 1

Define W to be the subspace of V that is the range of T − λ I V , W = ℛ T − λ I V . We define a new linear transformation S , on W ,

S : W → W S w = T w

This does not look we have accomplished much, since the action of S is identical to the action of T . For our purposes this will be a good thing. What is different is the domain and codomain. S is defined on W , a vector space with dimension less than m , and so is susceptible to our induction hypothesis. Verifying that S is really a linear transformation is almost entirely routine, with one exception. Employing T in our definition of S raises the possibility that the outputs of S will not be contained within W (but instead will lie inside V , but outside W ). To examine this possibility, suppose that w ∈ W .

S w = T w = T w + 0   Property Z = T w + λ I V w − λ I V w   Property AI = T w − λ I V w + λ I V w   Property AA = T w − λ I V w + λ w   Definition IDLT = T − λ I V w + λ w   Theorem VSLT

Since W is the range of T − λ I V , T − λ I V w ∈ W . And by Property SC, λ w ∈ W . Finally, applying Property AC we see by closure that the sum is in W and so we conclude that S w ∈ W . This argument convinces us that it is legitimate to define S as we did with W as the codomain.

S is a linear transformation defined on a vector space with dimension less than m , so we can apply the induction hypothesis and conclude that W has a basis, C = w 1 , w 2 , w 3 , … , w k , such that the matrix representation of S relative to C is an upper triangular matrix.

By Theorem DSFOS there exists a second subspace of V , which we will call U , so that V is a direct sum of W and U , V = W ⊕ U . Choose a basis D = u 1 , u 2 , u 3 , … , u ℓ for U . So m = k + ℓ by Theorem DSD, and B = C ∪ D is basis for V by Theorem DSLI and Theorem G. B is the basis we desire. What does a matrix representation of T look like, relative to B ?

Since the definition of T and S agree on W , the first k columns of M B , B T will have the upper triangular matrix representation of S in the first k rows. The remaining ℓ = m − k rows of these first k columns will be all zeros since the outputs of T on C are all contained in W . The situation for T on D is not quite as pretty, but it is close.

For 1 ≤ i ≤ ℓ , consider

ρ B T u i = ρ B T u i + 0   Property Z = ρ B T u i + λ I V u i − λ I V u i   Property AI = ρ B T u i − λ I V u i + λ I V u i   Property AA = ρ B T u i − λ I V u i + λ u i   Definition IDLT = ρ B T − λ I V u i + λ u i   Theorem VSLT = ρ B a 1 w 1 + a 2 w 2 + a 3 w 3 + ⋯ + a k w k + λ u i   Definition RLT = a 1 a 2 ⋮ a k 0 ⋮ 0 λ 0 ⋮ 0   Definition VR

In the penultimate step of this proof, we have rewritten an element of the range of T − λ I V as a linear combination of the basis vectors, C , for the range of T − λ I V , W , using the scalars a 1 , a 2 , a 3 , … , a k . If we incorporate these ℓ column vectors into the matrix representation M B , B T we find ℓ occurrences of λ on the diagonal, and any nonzero entries lying only in the first k rows. Together with the k × k upper triangular representation in the upper left-hand corner, the entire matrix representation is now clearly upper triangular. This completes the induction step, so for any linear transformation there is a basis that creates an upper triangular matrix representation.

We have one more statement in the conclusion of the theorem to verify. The eigenvalues of T , and their multiplicities, can be computed with the techniques of Chapter E relative to any matrix representation ( Theorem EER). We take this approach with our upper triangular matrix representation M B , B T . Let d i be the diagonal entry of M B , B T in row i and column i . Then the characteristic polynomial, computed as a determinant (Definition CP) with repeated expansions about the first column, is

p M B , B T x = d 1 − x d 2 − x d 3 − x ⋯ d m − x

The roots of the polynomial equation p M B , B T x = 0 are the eigenvalues of the linear transformation (Theorem EMRCP). So each diagonal entry is an eigenvalue, and is repeated on the diagonal exactly α T λ times (Definition AME). ■

A key step in this proof was the construction of the subspace W with dimension strictly less than that of V . This required an eigenvalue/eigenvector pair, which was guaranteed to us by Theorem EMHE. Digging deeper, the proof of Theorem EMHE requires that we can factor polynomials completely, into linear factors. This will not always happen if our set of scalars is the reals, ℝ . So this is our final explanation of our choice of the complex numbers, ℂ , as our set of scalars. In ℂ polynomials factor completely, so every matrix has at least one eigenvalue, and an inductive argument will get us to upper triangular matrix representations.

In the case of linear transformations defined on ℂ m , we can use the inner product ( Definition IP) profitably to fine-tune the basis that yields an upper triangular matrix representation. Recall that the adjoint of matrix A (Definition A) is written as A ∗ .

Theorem  OBUTR
Orthonormal Basis for Upper Triangular Representation
Suppose that A is a square matrix. Then there is a unitary matrix U , and an upper triangular matrix T , such that

U ∗ A U = T

and T has the eigenvalues of A as the entries of the diagonal. □

Proof   This theorem is a statement about matrices and similarity. We can convert it to a statement about linear transformations, matrix representations and bases ( Theorem SCB). Suppose that A is an n × n matrix, and define the linear transformation S : ℂ n → ℂ n by S x = A x . Then Theorem UTMR gives us a basis B = v 1 , v 2 , v 3 , … , v n for ℂ n such that a matrix representation of S relative to B , M B , B S , is upper triangular.

Now convert the basis B into an orthogonal basis, C , by an application of the Gram-Schmidt procedure ( Theorem GSP). This is a messy business computationally, but here we have an excellent illustration of the power of the Gram-Schmidt procedure. We need only be sure that B is linearly independent and spans ℂ n , and then we know that C is linearly independent, spans ℂ n and is also an orthogonal set. We will now consider the matrix representation of S relative to C (rather than B ). Write the new basis as C = y 1 , y 2 , y 3 , … , y n . The application of the Gram-Schmidt procedure creates each vector of C , say y j , as the difference of v j and a linear combination of y 1 , y 2 , y 3 , … , y j − 1 . We are not concerned here with the actual values of the scalars in this linear combination, so we will write

y j = v j − ∑ k = 1 j − 1 b j k y k

where the b j k are shorthand for the scalars. The equation above is in a form useful for creating the basis C from B . To better understand the relationship between B and C convert it to read

v j = y j + ∑ k = 1 j − 1 b j k y k

In this form, we recognize that the change-of-basis matrix C B , C = M B , C I ℂ n ( Definition CBM) is an upper triangular matrix. By Theorem SCB we have

M C , C S = C B , C M B , B S C B , C − 1

The inverse of an upper triangular matrix is upper triangular ( Theorem ITMT), and the product of two upper triangular matrices is again upper triangular ( Theorem PTMT). So M C , C S is an upper triangular matrix.

Now, multiply each vector of C by a nonzero scalar, so that the result has norm 1. In this way we create a new basis D which is an orthonormal set ( Definition ONS). Note that the change-of-basis matrix C C , D is a diagonal matrix with nonzero entries equal to the norms of the vectors in C .

Now we can convert our results into the language of matrices. Let E be the basis of ℂ n formed with the standard unit vectors ( Definition SUV). Then the matrix representation of S relative to E is simply A , A = M E , E S . The change-of-basis matrix C D , E has columns that are simply the vectors in D , the orthonormal basis. As such, Theorem CUMOS tells us that C D , E is a unitary matrix, and by Definition UM has an inverse equal to its adjoint. Write U = C D , E . We have

U ∗ A U = U − 1 A U   Theorem UMI = C D , E − 1 M E , E S C D , E = M D , D S   Theorem SCB = C C , D M C , C S C C , D − 1   Theorem SCB

The inverse of a diagonal matrix is also a diagonal matrix, and so this final expression is the product of three upper triangular matrices, and so is again upper triangular ( Theorem PTMT). Thus the desired upper triangular matrix, T , is the matrix representation of S relative to the orthonormal basis D , M D , D S . ■

Subsection NM: Normal Matrices

Normal matrices comprise a broad class of interesting matrices, many of which we have met already. But they are most interesting since they define exactly which matrices we can diagonalize via a unitary matrix. This is the upcoming Theorem OD. Here’s the definition.

Definition  NRML
Normal Matrix
The square matrix A is normal if A ∗ A = A A ∗ . △

So a normal matrix commutes with its adjoint. Part of the beauty of this definition is that it includes many other types of matrices. A diagonal matrix will commute with its adjoint, since the adjoint is again diagonal and the entries are just conjugates of the entries of the original diagonal matrix. A Hermitian (self-adjoint) matrix ( Definition HM) will trivially commute with its adjoint, since the two matrices are the same. A real, symmetric matrix is Hermitian, so these matrices are also normal. A unitary matrix ( Definition UM) has its adjoint as its inverse, and inverses commute ( Theorem OSIS), so unitary matrices are normal. Another class of normal matrices is the skew-symmetric matrices. However, these broad descriptions still do not capture all of the normal matrices, as the next example shows.

Example  ANM
A normal matrix
Let

A = 1 − 1 1 1

Then

1 − 1 1 1 1 1 − 1 1 = 2 0 0 2 = 1 1 − 1 1 1 − 1 1 1

so we see by Definition NRML that A is normal. However, A is not symmetric (hence, as a real matrix, not Hermitian), not unitary, and not skew-symmetric. ⊠

Subsection OD: Orthonormal Diagonalization

A diagonal matrix is very easy to work with in matrix multiplication (Example HPDM) and an orthonormal basis also has many advantages (Theorem COB). How about converting a matrix to a diagonal matrix through a similarity transformation using a unitary matrix (i.e. build a diagonal matrix representation with an orthonormal matrix)? That’d be fantastic! When can we do this? We can always accomplish this feat when the matrix is normal, and normal matrices are the only ones that behave this way. Here’s the theorem.

Theorem  OD
Orthonormal Diagonalization
Suppose that A is a square matrix. Then there is a unitary matrix U and a diagonal matrix D , with diagonal entries equal to the eigenvalues of A , such that U ∗ A U = D if and only if A is a normal matrix. □

Proof   ( ⇒ ) Suppose there is a unitary matrix U that diagonalizes A , resulting in D , i.e.  U ∗ A U = D . We check the normality of A ,

A ∗ A = I n A ∗ I n A I n   Theorem MMIM = U U ∗ A ∗ U U ∗ A U U ∗   Definition UM = U U ∗ A ∗ U D U ∗ = U U ∗ A ∗ U ∗ ∗ D U ∗   Theorem AA = U U ∗ A U ∗ D U ∗  Adjoint of a product = U D ∗ D U ∗ = U D ¯ t D U ∗   Definition A = U D ¯ D U ∗  Diagonal matrix = U D D ¯ U ∗   Property CMCN = U D D ¯ t U ∗  Diagonal matrix = U D D ∗ U ∗   Definition A = U D U ∗ A U ∗ U ∗ = U D U ∗ A ∗ U ∗ ∗ U ∗  Adjoint of a product = U D U ∗ A ∗ U U ∗   Theorem AA = U U ∗ A U U ∗ A ∗ U U ∗ = I n A I n A ∗ I n   Definition UM = A A ∗   Theorem MMIM

So by Definition NRML, A is a normal matrix.

( ⇐ ) For the converse, suppose that A is a normal matrix. Whether or not A is normal, Theorem OBUTR provides a unitary matrix U and an upper triangular matrix T , whose diagonal entries are the eigenvalues of A , and such that U ∗ A U = T . With the added condition that A is normal, we will determine that the entries of T above the diagonal must be all zero. Here we go. First we show that T is normal.

T ∗ T = U ∗ A U ∗ U ∗ A U = U ∗ A ∗ U ∗ ∗ U ∗ A U  Adjoint of a product = U ∗ A ∗ U U ∗ A U   Theorem AA = U ∗ A ∗ I n A U   Definition UM = U ∗ A ∗ A U   Theorem MMIM = U ∗ A A ∗ U   Definition NRML = U ∗ A I n A ∗ U   Theorem MMIM = U ∗ A U U ∗ A ∗ U   Definition UM = U ∗ A U U ∗ A ∗ U ∗ ∗   Theorem AA = U ∗ A U U ∗ A U ∗  Adjoint of a product = T T ∗

So by Definition NRML, T is a normal matrix.

We can translate the normality of T into the statement T T ∗ − T ∗ T = O . We now establish an equality we will use repeatedly. For 1 ≤ i ≤ n ,

0 = O i i   Definition ZM = T T ∗ − T ∗ T i i   Definition NRML = T T ∗ i i − T ∗ T i i   Definition MA = ∑ k = 1 n T i k T ∗ k i − ∑ k = 1 n T ∗ i k T k i   Theorem EMP = ∑ k = 1 n T i k T i k ¯ − ∑ k = 1 n T k i ¯ T k i   Definition A = ∑ k = i n T i k T i k ¯ − ∑ k = 1 i T k i ¯ T k i   Definition UTM = ∑ k = i n T i k 2 − ∑ k = 1 i T k i 2   Definition MCN

To conclude, we use the above equality repeatedly, beginning with i = 1 , and discover, row by row, that the entries above the diagonal of T are all zero. The key observation is that a sum of squares can only equal zero when each term of the sum is zero. For i = 1 we have

0 = ∑ k = 1 n T 1 k 2 − ∑ k = 1 1 T k 1 2 = ∑ k = 2 n T 1 k 2

which forces the conclusions

T 1 2 = 0 T 1 3 = 0 T 1 4 = 0 ⋯ T 1 n = 0

For i = 2 we use the same equality, but also incorporate the portion of the above conclusions that says T 1 2 = 0 ,

0 = ∑ k = 2 n T 2 k 2 − ∑ k = 1 2 T k 2 2 = ∑ k = 2 n T 2 k 2 − ∑ k = 2 2 T k 2 2 = ∑ k = 3 n T 2 k 2

which forces the conclusions

T 2 3 = 0 T 2 4 = 0 T 2 5 = 0 ⋯ T 2 n = 0

We can repeat this process for the subsequent values of i = 3 , 4 , 5 … , n − 1 . Notice that it is critical we do this in order, since we need to employ portions of each of the previous conclusions about rows having zero entries in order to successfully get the same conclusion for later rows. Eventually, we conclude that all of the nondiagonal entries of T are zero, so the extra assumption of normality forces T to be diagonal. ■

We can rearrange the conclusion of this theorem to read A = U D U ∗ . Recall that a unitary matrix can be viewed as a geometry-preserving transformation (isometry), or more loosely as a rotation of sorts. Then a matrix-vector product, A x , can be viewed instead as a sequence of three transformations. U ∗ is unitary, so is a rotation. Since D is diagonal, it just multiplies each entry of a vector by a scalar. Diagonal entries that are positive or negative, with absolute values bigger or smaller than 1 evoke descriptions like reflection, expansion and contraction. Generally we can say that D “stretches” a vector in each component. Final multiplication by U undoes (inverts) the rotation performed by U ∗ . So a normal matrix is a rotation-stretch-rotation transformation.

The orthonormal basis formed from the columns of U can be viewed as a system of mutually perpendicular axes. The rotation by U ∗ allows the transformation by A to be replaced by the simple transformation D along these axes, and then D brings the result back to the original coordinate system. For this reason Theorem OD is known as the Principal Axis Theorem.

The columns of the unitary matrix in Theorem OD create an especially nice basis for use with the normal matrix. We record this observation as a theorem.

Theorem  OBNM
Orthonormal Bases and Normal Matrices
Suppose that A is a normal matrix of size n . Then there is an orthonormal basis of ℂ n composed of eigenvectors of A . □

Proof   Let U be the unitary matrix promised by Theorem OD and let D be the resulting diagonal matrix. The desired set of vectors is formed by collecting the columns of U into a set. Theorem CUMOS says this set of columns is orthonormal. Since U is nonsingular ( Theorem UMI), Theorem CNMB says the set is a basis.

Since A is diagonalized by U , the diagonal entries of the matrix D are the eigenvalues of A . An argument exactly like the second half of the proof of Theorem DC shows that each vector of the basis is an eigenvector of A . ■

In a vague way Theorem OBNM is an improvement on Theorem HMOE which said that eigenvectors of a Hermitian matrix for different eigenvalues are always orthogonal. Hermitian matrices are normal and we see that we can find at least one basis where every pair of eigenvectors is orthogonal. Notice that this is not a generalization, since Theorem HMOE states a weak result which applies to many (but not all) pairs of eigenvectors, while Theorem OBNM is a seemingly stronger result, but only asserts that there is one collection of eigenvectors with the stronger property.

Section NLT  Nilpotent Linear Transformations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s s e c t i o n i s i n d r a f t f o r m
N e a r l y c o m p l e t e

We have seen that some matrices are diagonalizable and some are not. Some authors refer to a non-diagonalizable matrix as defective, but we will study them carefully anyway. Examples of such matrices include Example EMMS4, Example HMEM5, and Example CEMS6. Each of these matrices has at least one eigenvalue with geometric multiplicity strictly less than its algebraic multiplicity, and therefore Theorem DMFE tells us these matrices are not diagonalizable.

Given a square matrix A , it is likely similar to many, many other matrices. Of all these possibilities, which is the best? “Best” is a subjective term, but we might agree that a diagonal matrix is certainly a very nice choice. Unfortunately, as we have seen, this will not always be possible. What form of a matrix is “next-best”? Our goal, which will take us several sections to reach, is to show that every matrix is similar to a matrix that is “nearly-diagonal” ( Section JCF). More precisely, every matrix is similar to a matrix with elements on the diagonal, and zeros and ones on the diagonal just above the main diagonal (the “super diagonal”), with zeros everywhere else. In the language of equivalence relations (see Theorem SER), we are determining a systematic representative for each equivalence class. Such a representative for a set of similar matrices is called a canonical form.

We have just discussed the determination of a canonical form as a question about matrices. However, we know that every square matrix creates a natural linear transformation (Theorem MBLT) and every linear transformation with identical domain and codomain has a square matrix representation for each choice of a basis, with a change of basis creating a similarity transformation ( Theorem SCB). So we will state, and prove, theorems using the language of linear transformations on abstract vector spaces, while most of our examples will work with square matrices. You can, and should, mentally translate between the two settings frequently and easily.

Parent topic: Chapter R  Representations

Previous topic: Section OD  Orthonormal Diagonalization

Next topic: Section IS  Invariant Subspaces

Subsection NLT: Nilpotent Linear Transformations

We will discover that nilpotent linear transformations are the essential obstacle in a non-diagonalizable linear transformation. So we will study them carefully first, both as an object of inherent mathematical interest, but also as the object at the heart of the argument that leads to a pleasing canonical form for any linear transformation. Once we understand these linear transformations thoroughly, we will be able to easily analyze the structure of any linear transformation.

Definition  NLT
Nilpotent Linear Transformation
Suppose that T : V → V is a linear transformation such that there is an integer p > 0 such that T p v = 0 for every v ∈ V . The smallest p for which this condition is met is called the i n d e x of T . △

Of course, the linear transformation T defined by T v = 0 will qualify as nilpotent of index 1. But are there others?

Example  NM64
Nilpotent matrix, size 6, index 4
Recall that our definitions and theorems are being stated for linear transformations on abstract vector spaces, while our examples will work with square matrices (and use the same terms interchangeably). In this case, to demonstrate the existence of nontrivial nilpotent linear transformations, we desire a matrix such that some power of the matrix is the zero matrix. Consider

A = − 3 3 − 2 5 0 − 5 − 3 5 − 3 4 3 − 9 − 3 4 − 2 6 − 4 − 3 − 3 3 − 2 5 0 − 5 − 3 3 − 2 4 2 − 6 − 2 3 − 2 2 4 − 7  and compute powers of  A , A 2 = 1 − 2 1 0 − 3 4 0 − 2 1 1 − 3 4 3 0 0 − 3 0 0 1 − 2 1 0 − 3 4 0 − 2 1 1 − 3 4 − 1 − 2 1 2 − 3 4 A 3 = 1 0 0 − 1 0 0 1 0 0 − 1 0 0 0 0 0 0 0 0 1 0 0 − 1 0 0 1 0 0 − 1 0 0 1 0 0 − 1 0 0 A 4 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Thus we can say that A is nilpotent of index 4.

Because it will presage some upcoming theorems, we will record some extra information about the eigenvalues and eigenvectors of A here. A has just one eigenvalue, λ = 0 , with algebraic multiplicity 6 and geometric multiplicity 2 . The eigenspace for this eigenvalue is

ℰ A 0 = 2 2 5 2 1 0 , − 1 − 1 − 5 − 1 0 1

If there were degrees of singularity, we might say this matrix was very singular, since zero is an eigenvalue with maximum algebraic multiplicity ( Theorem SMZE, Theorem ME). Notice too that A is “far” from being diagonalizable (Theorem DMFE). ⊠

Another example.

Example  NM62
Nilpotent matrix, size 6, index 2
Consider the matrix

B = − 1 1 − 1 4 − 3 − 1 1 1 − 1 2 − 3 − 1 − 9 1 0 − 5 9 5 − 1 5 − 1 1 − 1 4 − 3 − 1 1 − 1 0 2 − 4 2 4 − 3 1 − 1 − 5 5  and compute the second power of  B , B 2 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

So B is nilpotent of index 2. Again, the only eigenvalue of B is zero, with algebraic multiplicity 6 . The geometric multiplicity of the eigenvalue is 3 , as seen in the eigenspace,

ℰ B 0 = 1 3 6 1 0 0 , 0 − 4 − 7 0 1 0 , 0 2 1 0 0 1

Again, Theorem DMFE tells us that B is far from being diagonalizable. ⊠

On a first encounter with the definition of a nilpotent matrix, you might wonder if such a thing was possible at all. That a high power of a nonzero object could be zero is so very different from our experience with scalars that it seems very unnatural. Hopefully the two previous examples were somewhat surprising. But we have seen that matrix algebra does not always behave the way we expect ( Example MMNC), and we also now recognize matrix products not just as arithmetic, but as function composition (Theorem MRCLT). We will now turn to some examples of nilpotent matrices which might be more transparent.

Definition  JB
Jordan Block
Given the scalar λ ∈ ℂ , the Jordan block J n λ is the n × n matrix defined by

J n λ i j = λ i = j 1 j = i + 1 0  otherwise

(This definition contains Notation JB.) △

Example  JB4
Jordan block, size 4
A simple example of a Jordan block,

J 4 5 = 5 1 0 0 0 5 1 0 0 0 5 1 0 0 0 5

⊠

We will return to general Jordan blocks later, but in this section we are just interested in Jordan blocks where λ = 0 . Here’s an example of why we are specializing in these matrices now.

Example  NJB5
Nilpotent Jordan block, size 5
Consider

J 5 0 = 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0  and compute powers, J 5 0 2 = 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 J 5 0 3 = 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 J 5 0 4 = 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 J 5 0 5 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

So J 5 0 is nilpotent of index 5. As before, we record some information about the eigenvalues and eigenvectors of this matrix. The only eigenvalue is zero, with algebraic multiplicity 5, the maximum possible ( Theorem ME). The geometric multiplicity of this eigenvalue is just 1, the minimum possible ( Theorem ME), as seen in the eigenspace,

ℰ J 5 0 0 = 1 0 0 0 0

There should not be any real surprises in this example. We can watch the ones in the powers of J 5 0 slowly march off to the upper-right hand corner of the powers. In some vague way, the eigenvalues and eigenvectors of this matrix are equally extreme. ⊠

We can form combinations of Jordan blocks to build a variety of nilpotent matrices. Simply place Jordan blocks on the diagonal of a matrix with zeros everywhere else, to create a block diagonal matrix.

Example  NM83
Nilpotent matrix, size 8, index 3
Consider the matrix

C = J 3 0 O O O J 3 0 O O O J 2 0 = 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0  and compute powers, C 2 = 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C 3 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

So C is nilpotent of index 3. You should notice how block diagonal matrices behave in products (much like diagonal matrices) and that it was the largest Jordan block that determined the index of this combination. All eight eigenvalues are zero, and each of the three Jordan blocks contributes one eigenvector to a basis for the eigenspace, resulting in zero having a geometric multiplicity of 3. ⊠

It would appear that nilpotent matrices only have zero as an eigenvalue, so the algebraic multiplicity will be the maximum possible. However, by creating block diagonal matrices with Jordan blocks on the diagonal you should be able to attain any desired geometric multiplicity for this lone eigenvalue. Likewise, the size of the largest Jordan block employed will determine the index of the matrix. So nilpotent matrices with various combinations of index and geometric multiplicities are easy to manufacture. The predictable properties of block diagonal matrices in matrix products and eigenvector computations, along with the next theorem, make this possible. You might find Example NJB5 a useful companion to this proof.

Theorem  NJB
Nilpotent Jordan Blocks
The Jordan block J n 0 is nilpotent of index n . □

Proof   While not phrased as an if-then statement, the statement in the theorem is understood to mean that if we have a specific matrix ( J n 0 ) then we need to establish it is nilpotent of a specified index. The first column of J n 0 is the zero vector, and the remaining n − 1 columns are the standard unit vectors e i , 1 ≤ i ≤ n − 1 ( Definition SUV), which are also the first n − 1 columns of the size n identity matrix I n . As shorthand, write J = J n 0 .

J = 0 e 1 e 2 e 3 … e n − 1

We will use the definition of matrix multiplication ( Definition MM), together with a proof by induction ( Technique I), to study the powers of J . Our claim is that

J k = 0 0 … 0 e 1 e 2 … e n − k

for 1 ≤ k ≤ n . For the base case, k = 1 , and the definition of J 1 = J n 0 establishes the claim. For the induction step, first note that J e 1 = 0 and J e i = e i − 1 for 2 ≤ i ≤ n . Then, assuming the claim is true for k , we examine the k + 1 case,

J k + 1 = J J k = J 0 0 … 0 e 1 e 2 … e n − k  Induction Hypothesis = J 0 J 0 … J 0 J e 1 J e 2 … J e n − k   Definition MM = 0 0 … 0 0 e 1 e 2 … e n − k − 1   Definition MVP = 0 0 … 0 e 1 e 2 … e n − ( k + 1 )

This concludes the induction. So J k has a nonzero entry (a one) in row n − k and column n , for 1 ≤ k ≤ n − 1 , and is therefore a nonzero matrix. However, J n = 0 0 … 0 = O . By Definition NLT, J is nilpotent of index n . ■

Subsection PNLT: Properties of Nilpotent Linear Transformations

In this subsection we collect some basic properties of nilpotent linear transformations. After studying the examples in the previous section, some of these will be no surprise.

Theorem  ENLT
Eigenvalues of Nilpotent Linear Transformations
Suppose that T : V → V is a nilpotent linear transformation and λ is an eigenvalue of T . Then λ = 0 . □

Proof   Let x be an eigenvector of T for the eigenvalue λ , and suppose that T is nilpotent with index p . Then

0 = T p x   Definition NLT = λ p x   Theorem EOMP

Because x is an eigenvector, it is nonzero, and therefore Theorem SMEZV tells us that λ p = 0 and so λ = 0 . ■

Paraphrasing, all of the eigenvalues of a nilpotent linear transformation are zero. So in particular, the characteristic polynomial of a nilpotent linear transformation, T , on a vector space of dimension n , is simply p T x = x n .

The next theorem is not critical for what follows, but it will explain our interest in nilpotent linear transformations. More specifically, it is the first step in backing up the assertion that nilpotent linear transformations are the essential obstacle in a non-diagonalizable linear transformation. While it is not obvious from the statement of the theorem, it says that a nilpotent linear transformation is not diagonalizable, unless it is trivially so.

Theorem  DNLT
Diagonalizable Nilpotent Linear Transformations
Suppose the linear transformation T : V → V is nilpotent. Then T is diagonalizable if and only T is the zero linear transformation. □

Proof   We start with the easy direction. Let n = dim V .

( ⇐ ) The linear transformation Z : V → V defined by Z v = 0 for all v ∈ V is nilpotent of index p = 1 and a matrix representation relative to any basis of V is the n × n zero matrix, O . Quite obviously, the zero matrix is a diagonal matrix (Definition DIM) and hence Z is diagonalizable (Definition DZM).

( ⇒ ) Assume now that T is diagonalizable, so γ T λ = α T λ for every eigenvalue λ (Theorem DMFE). By Theorem ENLT, T has only one eigenvalue (zero), which therefore must have algebraic multiplicity n ( Theorem NEM). So the geometric multiplicity of zero will be n as well, γ T 0 = n .

Let B be a basis for the eigenspace ℰ T 0 . Then B is a linearly independent subset of V of size n , and by Theorem G will be a basis for V . For any x ∈ B we have

T x = 0 x   Definition EM = 0   Theorem ZSSM

So T is identically zero on a basis for B , and since the action of a linear transformation on a basis determines all of the values of the linear transformation (Theorem LTDB), it is easy to see that T v = 0 for every v ∈ V . ■

So, other than one trivial case (the zero matrix), every nilpotent linear transformation is not diagonalizable. It remains to see what is so “essential” about this broad class of non-diagonalizable linear transformations. For this we now turn to a discussion of kernels of powers of nilpotent linear transformations, beginning with a result about general linear transformations that may not necessarily be nilpotent.

Theorem  KPLT
Kernels of Powers of Linear Transformations
Suppose T : V → V is a linear transformation, where dim V = n . Then there is an integer m , 0 ≤ m ≤ n , such that

0 = K T 0 ⊊ K T 1 ⊊ K T 2 ⊊ ⋯ ⊊ K T m = K T m + 1 = K T m + 2 = ⋯

□

Proof   There are several items to verify in the conclusion as stated. First, we show that K T k ⊆ K T k + 1 for any k . Choose z ∈ K T k . Then

T k + 1 z = T T k z   Definition LTC = T 0   Definition KLT = 0   Theorem LTTZZ

So by Definition KLT, z ∈ K T k + 1 and by Definition SSET we have K T k ⊆ K T k + 1 .

Second, we demonstrate the existence of a power m where consecutive powers result in equal kernels. A by-product will be the condition that m can be chosen so that m ≤ n . To the contrary, suppose that

0 = K T 0 ⊊ K T 1 ⊊ K T 2 ⊊ ⋯ ⊊ K T n − 1 ⊊ K T n ⊊ K T n + 1 ⊊ ⋯

Since K T k ⊊ K T k + 1 , Theorem PSSD implies that dim K T k + 1 ≥ dim K T k + 1 . Repeated application of this observation yields

dim K T n + 1 ≥ dim K T n + 1 ≥ dim K T n − 1 + 2 ⋮ ≥ dim K T 0 + ( n + 1 ) = dim 0 + n + 1 = n + 1

Thus, K T n + 1 has a basis of size at least n + 1 , which is a linearly independent set of size greater than n in the vector space V of dimension n . This contradicts Theorem G.

This contradiction yields the existence of an integer k such that K T k = K T k + 1 , so we can define m to be smallest such integer with this property. From the argument above about dimensions resulting from a strictly increasing chain of subspaces, it should be clear that m ≤ n .

It remains to show that once two consecutive kernels are equal, then all of the remaining kernels are equal. More formally, if K T m = K T m + 1 , then K T m = K T m + j for all j ≥ 1 . We will give a proof by induction on j ( Technique I). The base case ( j = 1 ) is precisely our defining property for m .

In the induction step, we assume that K T m = K T m + j and endeavor to show that K T m = K T m + j + 1 . At the outset of this proof we established that K T m ⊆ K T m + j + 1 . So Definition SE requires only that we establish the subset inclusion in the opposite direction. To wit, choose z ∈ K T m + j + 1 . Then

0 = T m + j + 1 z   Definition KLT = T m + j T z   Definition LTC = T m T z  Induction Hypothesis = T m + 1 z   Definition LTC = T m z  Base Case

So by Definition KLT, z ∈ K T m as desired. ■

We now specialize Theorem KPLT to the case of nilpotent linear transformations, which buys us just a bit more precision in the conclusion.

Theorem  KPNLT
Kernels of Powers of Nilpotent Linear Transformations
Suppose T : V → V is a nilpotent linear transformation with index p and dim V = n . Then 0 ≤ p ≤ n and

0 = K T 0 ⊊ K T 1 ⊊ K T 2 ⊊ ⋯ ⊊ K T p = K T p + 1 = ⋯ = V

□

Proof   Since T p = 0 it follows that T p + j = 0 for all j ≥ 0 and thus K T p + j = V for j ≥ 0 . So the value of m guaranteed by Theorem KPLT is at most p . The only remaining aspect of our conclusion that does not follow from Theorem KPLT is that m = p . To see this we must show that K T k ⊊ K T k + 1 for 0 ≤ k ≤ p − 1 . If K T k = K T k + 1 for some k < p , then K T k = K T p = V . This implies that T k = 0 , violating the fact that T has index p . So the smallest value of m is indeed p , and we learn that p < n . ■

The structure of the kernels of powers of nilpotent linear transformations will be crucial to what follows. But immediately we can see a practical benefit. Suppose we are confronted with the question of whether or not an n × n matrix, A , is nilpotent or not. If we don’t quickly find a low power that equals the zero matrix, when do we stop trying higher and higher powers? Theorem KPNLT gives us the answer: if we don’t see a zero matrix by the time we finish computing A n , then it is not going to ever happen. We’ll now take a look at one example of Theorem KPNLT in action.

Example  KPNLT
Kernels of powers of a nilpotent linear transformation
We will recycle the nilpotent matrix A of index 4 from Example NM64. We now know that would have only needed to look at the first 6 powers of A if the matrix had not been nilpotent. We list bases for the null spaces of the powers of A . (Notice how we are using null spaces for matrices interchangeably with kernels of linear transformations, see Theorem KNSI for justification.)

N A = N − 3 3 − 2 5 0 − 5 − 3 5 − 3 4 3 − 9 − 3 4 − 2 6 − 4 − 3 − 3 3 − 2 5 0 − 5 − 3 3 − 2 4 2 − 6 − 2 3 − 2 2 4 − 7 = 2 2 5 2 1 0 , − 1 − 1 − 5 − 1 0 1 N A 2 = N 1 − 2 1 0 − 3 4 0 − 2 1 1 − 3 4 3 0 0 − 3 0 0 1 − 2 1 0 − 3 4 0 − 2 1 1 − 3 4 − 1 − 2 1 2 − 3 4 = 0 1 2 0 0 0 , 2 1 0 2 0 0 , 0 − 3 0 0 2 0 , 0 2 0 0 0 1 N A 3 = N 1 0 0 − 1 0 0 1 0 0 − 1 0 0 0 0 0 0 0 0 1 0 0 − 1 0 0 1 0 0 − 1 0 0 1 0 0 − 1 0 0 = 0 1 0 0 0 0 , 0 0 1 0 0 0 , 1 0 0 1 0 0 , 0 0 0 0 1 0 , 0 0 0 0 0 1 N A 4 = N 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 1 0 0 0 0 0 , 0 1 0 0 0 0 , 0 0 1 0 0 0 , 0 0 0 1 0 0 , 0 0 0 0 1 0 , 0 0 0 0 0 1

With the exception of some convenience scaling of the basis vectors in N A 2 these are exactly the basis vectors described in Theorem BNS. We can see that the dimension of N A equals the geometric multiplicity of the zero eigenvalue. Why is this not an accident? We can see the dimensions of the kernels consistently increasing, and we can see that N A 4 = ℂ 6 . But Theorem KPNLT says a little more. Each successive kernel should be a superset of the previous one. We ought to be able to begin with a basis of N A and extend it to a basis of N A 2 . Then we should be able to extend a basis of N A 2 into a basis of N A 3 , all with repeated applications of Theorem ELIS. Verify the following,

N A = 2 2 5 2 1 0 , − 1 − 1 − 5 − 1 0 1 N A 2 = 2 2 5 2 1 0 , − 1 − 1 − 5 − 1 0 1 , 0 − 3 0 0 2 0 , 0 2 0 0 0 1 N A 3 = 2 2 5 2 1 0 , − 1 − 1 − 5 − 1 0 1 , 0 − 3 0 0 2 0 , 0 2 0 0 0 1 , 0 0 0 0 0 1 N A 4 = 2 2 5 2 1 0 , − 1 − 1 − 5 − 1 0 1 , 0 − 3 0 0 2 0 , 0 2 0 0 0 1 , 0 0 0 0 0 1 , 0 0 0 1 0 0

Do not be concerned at the moment about how these bases were constructed since we are not describing the applications of Theorem ELIS here. Do verify carefully for each alleged basis that, (1) it is a superset of the basis for the previous kernel, (2) the basis vectors really are members of the kernel of the right power of A , (3) the basis is a linearly independent set, (4) the size of the basis is equal to the size of the basis found previously for each kernel. With these verifications, Theorem G will tell us that we have successfully demonstrated what Theorem KPNLT guarantees. ⊠

Subsection CFNLT: Canonical Form for Nilpotent Linear Transformations

Our main purpose in this section is to find a basis so that a nilpotent linear transformation will have a pleasing, nearly-diagonal matrix representation. Of course, we will not have a definition for “pleasing,” nor for “nearly-diagonal.” But the short answer is that our preferred matrix representation will be built up from Jordan blocks, J n 0 . Here’s the theorem. You will find Example CFNLT helpful as you study this proof, since it uses the same notation, and is large enough to (barely) illustrate the full generality of the theorem (see ).

Theorem  CFNLT
Canonical Form for Nilpotent Linear Transformations
Suppose that T : V → V is a nilpotent linear transformation of index p . Then there is a basis for V so that the matrix representation, M B , B T , is block diagonal with each block being a Jordan block, J n 0 . The size of the largest block is the index p , and the total number of blocks is the nullity of T , n T . □

Proof   We will explicitly construct the desired basis, so the proof is constructive ( Technique C), and can be used in practice. As we begin, the basis vectors will not be in the proper order, but we will rearrange them at the end of the proof. For convenience, define n i = n T i , so for example, n 0 = 0 , n 1 = n T and n p = n T p = dim V . Define s i = n i − n i − 1 , for 1 ≤ i ≤ p , so we can think of s i as “how much bigger” K T i is than K T i − 1 . In particular, Theorem KPNLT implies that s i > 0 for 1 ≤ i ≤ p .

We are going to build a set of vectors z i , j , 1 ≤ i ≤ p , 1 ≤ j ≤ s i . Each z i , j will be an element of K T i and not an element of K T i − 1 . In total, we will obtain a linearly independent set of ∑ i = 1 p s i = ∑ i = 1 p n i − n i − 1 = n p − n 0 = dim V vectors that form a basis of V . We construct this set in pieces, starting at the “wrong” end. Our procedure will build a series of subspaces, Z i , each lying in between K T i − 1 and K T i , having bases z i , j , 1 ≤ j ≤ s i , and which together equal V as a direct sum. Now would be a good time to review the results on direct sums collected in Subsection PD.DS. OK, here we go.

We build the subspace Z p first (this is what we meant by “starting at the wrong end”). K T p − 1 is a proper subspace of K T p = V (Theorem KPNLT). Theorem DSFOS says that there is a subspace of V that will pair with the subspace K T p − 1 to form a direct sum of V . Call this subspace Z p , and choose vectors z p , j , 1 ≤ j ≤ s p as a basis of Z p , which we will denote as B p . Note that we have a fair amount of freedom in how to choose these first basis vectors. Several observations will be useful in the next step. First V = K T p − 1 ⊕ Z p . The basis B p = z p , 1 , z p , 2 , z p , 3 , … , z p , s p is linearly independent. For 1 ≤ j ≤ s p , z p , j ∈ K T p = V . Since the two subspaces of a direct sum have no nonzero vectors in common (Theorem DSZI), for 1 ≤ j ≤ s p , z p , j ∉ K T p − 1 . That was comparably easy.

If obtaining Z p was easy, getting Z p − 1 will be harder. We will repeat the next step p − 1 times, and so will do it carefully the first time. Eventually, Z p − 1 will have dimension s p − 1 . However, the first s p vectors of a basis are straightforward. Define z p − 1 , j = T z p , j , 1 ≤ j ≤ s p . Notice that we have no choice in creating these vectors, they are a consequence of our choices for z p , j . In retrospect (i.e. on a second reading of this proof), you will recognize this as the key step in realizing a matrix representation of a nilpotent linear transformation with Jordan blocks. We need to know that this set of vectors in linearly independent, so start with a relation of linear dependence (Definition RLD), and massage it,

0 = a 1 z p − 1 , 1 + a 2 z p − 1 , 2 + a 3 z p − 1 , 3 + ⋯ + a s p z p − 1 , s p = a 1 T z p , 1 + a 2 T z p , 2 + a 3 T z p , 3 + ⋯ + a s p T z p , s p = T a 1 z p , 1 + a 2 z p , 2 + a 3 z p , 3 + ⋯ + a s p z p , s p   Theorem LTLC

Define x = a 1 z p , 1 + a 2 z p , 2 + a 3 z p , 3 + ⋯ + a s p z p , s p . The statement just above means that x ∈ K T ⊆ K T p − 1 ( Definition KLT, Theorem KPNLT). As defined, x is a linear combination of the basis vectors B p , and therefore x ∈ Z p . Thus x ∈ K T p − 1 ∩ Z p ( Definition SI). Because V = K T p − 1 ⊕ Z p , Theorem DSZI tells us that x = 0 . Now we recognize the definition of x as a relation of linear dependence on the linearly independent set B p , and therefore a 1 = a 2 = ⋯ = a s p = 0 (Definition LI). This establishes the linear independence of z p − 1 , j , 1 ≤ j ≤ s p (Definition LI).

We also need to know where the vectors z p − 1 , j , 1 ≤ j ≤ s p live. First we demonstrate that they are members of K T p − 1 .

T p − 1 z p − 1 , j = T p − 1 T z p , j = T p z p , j = 0

So z p − 1 , j ∈ K T p − 1 , 1 ≤ j ≤ s p . However, we now show that these vectors are not elements of K T p − 2 . Suppose to the contrary ( Technique CD) that z p − 1 , j ∈ K T p − 2 . Then

0 = T p − 2 z p − 1 , j = T p − 2 T z p , j = T p − 1 z p , j

which contradicts the earlier statement that z p , j ∉ K T p − 1 . So z p − 1 , j ∉ K T p − 2 , 1 ≤ j ≤ s p .

Now choose a basis C p − 2 = u 1 , u 2 , u 3 , … , u n p − 2 for K T p − 2 . We want to extend this basis by adding in the z p − 1 , j to span a subspace of K T p − 1 . But first we want to know that this set is linearly independent. Let a k , 1 ≤ k ≤ n p − 2 and b j , 1 ≤ j ≤ s p be the scalars in a relation of linear dependence,

0 = a 1 u 1 + a 2 u 2 + ⋯ + a n p − 2 u n p − 2 + b 1 z p − 1 , 1 + b 2 z p − 1 , 2 + ⋯ + b s p z p − 1 , s p

Then,

0 = T p − 2 0 = T p − 2 a 1 u 1 + a 2 u 2 + ⋯ + a n p − 2 u n p − 2 + b 1 z p − 1 , 1 + b 2 z p − 1 , 2 + ⋯ + b s p z p − 1 , s p = a 1 T p − 2 u 1 + a 2 T p − 2 u 2 + ⋯ + a n p − 2 T p − 2 u n p − 2 + b 1 T p − 2 z p − 1 , 1 + b 2 T p − 2 z p − 1 , 2 + ⋯ + b s p T p − 2 z p − 1 , s p = a 1 0 + a 2 0 + ⋯ + a n p − 2 0 + b 1 T p − 2 z p − 1 , 1 + b 2 T p − 2 z p − 1 , 2 + ⋯ + b s p T p − 2 z p − 1 , s p = b 1 T p − 2 z p − 1 , 1 + b 2 T p − 2 z p − 1 , 2 + ⋯ + b s p T p − 2 z p − 1 , s p = b 1 T p − 2 T z p , 1 + b 2 T p − 2 T z p , 2 + ⋯ + b s p T p − 2 T z p , s p = b 1 T p − 1 z p , 1 + b 2 T p − 1 z p , 2 + ⋯ + b s p T p − 1 z p , s p = T p − 1 b 1 z p , 1 + b 2 z p , 2 + ⋯ + b s p z p , s p

Define y = b 1 z p , 1 + b 2 z p , 2 + ⋯ + b s p z p , s p . The statement just above means that y ∈ K T p − 1 ( Definition KLT). As defined, y is a linear combination of the basis vectors B p , and therefore y ∈ Z p . Thus y ∈ K T p − 1 ∩ Z p . Because V = K T p − 1 ⊕ Z p , Theorem DSZI tells us that y = 0 . Now we recognize the definition of y as a relation of linear dependence on the linearly independent set B p , and therefore b 1 = b 2 = ⋯ = b s p = 0 (Definition LI). Return to the full relation of linear dependence with both sets of scalars (the a i and b j ). Now that we know that b j = 0 for 1 ≤ j ≤ s p , this relation of linear dependence simplifies to a relation of linear dependence on just the basis C p − 1 . Therefore, a i = 0 , 1 ≤ a i ≤ n p − 1 and we have the desired linear independence.

Define a new subspace of K T p − 1 as

Q p − 1 = u 1 , u 2 , u 3 , … , u n p − 1 , z p − 1 , 1 , z p − 1 , 2 , z p − 1 , 3 , … , z p − 1 , s p

By Theorem DSFOS there exists a subspace of K T p − 1 which will pair with Q p − 1 to form a direct sum. Call this subspace R p − 1 , so by definition, K T p − 1 = Q p − 1 ⊕ R p − 1 . We are interested in the dimension of R p − 1 . Note first, that since the spanning set of Q p − 1 is linearly independent, dim Q p − 1 = n p − 2 + s p . Then

dim R p − 1 = dim K T p − 1 − dim Q p − 1   Theorem DSD = n p − 1 − n p − 2 + s p = n p − 1 − n p − 2 − s p = s p − 1 − s p

Notice that if s p − 1 = s p , then R p − 1 is trivial. Now choose a basis of R p − 1 , and denote these s p − 1 − s p vectors as z p − 1 , s p + 1 , z p − 1 , s p + 2 , z p − 1 , s p + 3 , …, z p − 1 , s p − 1 . This is another occassion to notice that we have some freedom in this choice.

We now have K T p − 1 = Q p − 1 ⊕ R p − 1 , and we have bases for each of the two subspaces. The union of these two bases will therefore be a linearly independent set in K T p − 1 with size

n p − 2 + s p + s p − 1 − s p = n p − 2 + s p − 1 = n p − 2 + n p − 1 − n p − 2 = n p − 1 = dim K T p − 1

So, by Theorem G, the following set is a basis of K T p − 1 ,

u 1 , u 2 , u 3 , … , u n p − 2 , z p − 1 , 1 , z p − 1 , 2 , … , z p − 1 , s p , z p − 1 , s p + 1 , z p − 1 , s p + 2 , … , z p − 1 , s p − 1

We built up this basis in three parts, we will now split it in half. Define the subspace Z p − 1 by

Z p − 1 = B p − 1 = z p − 1 , 1 , z p − 1 , 2 , … , z p − 1 , s p − 1

where we have implicitly denoted the basis as B p − 1 . Then Theorem DSFB allows us to split up the basis for K T p − 1 as C p − 1 ∪ B p − 1 and write

K T p − 1 = K T p − 2 ⊕ Z p − 1

Whew! This is a good place to recap what we have achieved. The vectors z i , j form bases for the subspaces Z i and right now

V = K T p − 1 ⊕ Z p = K T p − 2 ⊕ Z p − 1 ⊕ Z p

The key feature of this decomposition of V is that the first s p vectors in the basis for Z p − 1 are outputs of the linear transformation T using the basis vectors of Z p as inputs.

Now we want to further decompose K T p − 2 (into K T p − 3 and Z p − 2 ). The procedure is the same as above, so we will only sketch the key steps. Checking the details proceeds in the same manner as above. Technically, we could have set up the preceding as the induction step in a proof by induction ( Technique I), but this probably would make the proof harder to understand.

Hit each element of B p − 1 with T , to create vectors z p − 2 , j , 1 ≤ j ≤ s p − 1 . These vectors form a linearly independent set, and each is an element of K T p − 2 , but not an element of K T p − 3 . Grab a basis C p − 3 of K T p − 3 and tack on the newly-created vectors z p − 2 , j , 1 ≤ j ≤ s p − 1 . This expanded set is linearly independent, and we can define a subspace Q p − 2 using it as a basis. Theorem DSFOS gives us a subspace R p − 2 such that K T p − 2 = Q p − 2 ⊕ R p − 2 . Vectors z p − 2 , j , s p − 1 + 1 ≤ j ≤ s p − 2 are chosen as a basis for R p − 2 once the relevant dimensions have been verified. The union of C p − 3 and z p − 2 , j , 1 ≤ j ≤ s p − 2 then form a basis of K T p − 2 , which can be split into two parts to yield the decomposition

K T p − 2 = K T p − 3 ⊕ Z p − 2

Here Z p − 2 is the subspace of K T p − 2 with basis B p − 2 = z p − 2 , j 1 ≤ j ≤ s p − 2 . Finally,

V = K T p − 1 ⊕ Z p = K T p − 2 ⊕ Z p − 1 ⊕ Z p = K T p − 3 ⊕ Z p − 2 ⊕ Z p − 1 ⊕ Z p

Again, the key feature of this decomposition is that the first vectors in the basis of Z p − 2 are outputs of T using vectors from the basis Z p − 1 as inputs (and in turn, some of these inputs are outputs of T derived from inputs in Z p ).

Now assume we repeat this procedure until we decompose K T 2 into subspaces K T and Z 2 . Finally, decompose K T into subspaces K T 0 = K I n = 0 and Z 1 , so that we recognize the vectors z 1 , j , 1 ≤ j ≤ s 1 = n 1 as elements of K T . The set

B = B 1 ∪ B 2 ∪ B 3 ∪ ⋯ ∪ B p = z i , j 1 ≤ i ≤ p , 1 ≤ j ≤ s i

is linearly independent by Theorem DSLI and has size

∑ i = 1 p s i = ∑ i = 1 p n i − n i − 1 = n p − n 0 = dim V

So by Theorem G, B is a basis of V . We desire a matrix representation of T relative to B (Definition MR), but first we will reorder the elements of B . The following display lists the elements of B in the desired order, when read across the rows left-to-right in the usual way. Notice that we established the existence of these vectors column-by-column, and beginning on the right.

z 1 , 1 z 2 , 1 z 3 , 1 ⋯ z p , 1 z 1 , 2 z 2 , 2 z 3 , 2 ⋯ z p , 2 ⋮ ⋮ z 1 , s p z 2 , s p z 3 , s p ⋯ z p , s p z 1 , s p + 1 z 2 , s p + 1 z 3 , s p + 1 ⋯ ⋮ ⋮ z 1 , s 3 z 2 , s 3 z 3 , s 3 ⋮ z 1 , s 2 z 2 , s 2 ⋮ z 1 , s 1

It is difficult to layout this table with the notation we have been using, but it would not be especially useful to invent some notation to overcome the difficulty. (One approach would be to define something like the inverse of the nonincreasing function, i → s i .) Do notice that there are s 1 = n 1 rows and p columns. Column i is the basis B i . The vectors in the first column are elements of K T . Each row is the same length, or shorter, than the one above it. If we apply T to any vector in the table, other than those in the first column, the output is the preceding vector in the row.

Now contemplate the matrix representation of T relative to B as we read across the rows of the table above. In the first row, T z 1 , 1 = 0 , so the first column of the representation is the zero column. Next, T z 2 , 1 = z 1 , 1 , so the second column of the representation is a vector with a single one in the first entry, and zeros elsewhere. Next, T z 3 , 1 = z 2 , 1 , so column 3 of the representation is a zero, then a one, then all zeros. Continuing in this vein, we obtain the first p columns of the representation, which is the Jordan block J p 0 followed by rows of zeros.

When we apply T to the basis vectors of the second row, what happens? Applying T to the first vector, the result is the zero vector, so the representation gets a zero column. Applying T to the second vector in the row, the output is simply the first vector in that row, making the next column of the representation all zeros plus a lone one, sitting just above the diagonal. Continuing, we create a Jordan block, sitting on the diagonal of the matrix representation. It is not possible in general to state the size of this block, but since the second row is no longer than the first, it cannot have size larger than p .

Since there are as many rows as the dimension of K T , the representation contains as many Jordan blocks as the nullity of T , n T . Each successive block is smaller than the preceding one, with the first, and largest, having size p . The blocks are Jordan blocks since the basis vectors z i , j were often defined as the result of applying T to other elements of the basis already determined, and then we rearranged the basis into an order that placed outputs of T just before their inputs, excepting the start of each row, which was an element of K T . ■

The proof of Theorem CFNLT is constructive ( Technique C), so we can use it to create bases of nilpotent linear transformations with pleasing matrix representations. Recall that Theorem DNLT told us that nilpotent linear transformations are almost never diagonalizable, so this is progress. As we have hinted before, with a nice representation of nilpotent matrices, it will not be difficult to build up representations of other non-diagonalizable matrices. Here is the promised example which illustrates the previous theorem. It is a useful companion to your study of the proof of Theorem CFNLT.

Example  CFNLT
Canonical form for a nilpotent linear transformation
The 6 × 6 matrix, A , of Example NM64 is nilpotent of index p = 4 . If we define the linear transformation T : ℂ 6 → ℂ 6 by T x = A x , then T is nilpotent of index 4 and we can seek a basis of ℂ 6 that yields a matrix representation with Jordan blocks on the diagonal. The nullity of T is 2 , so from Theorem CFNLT we can expect the largest Jordan block to be J 4 0 , and there will be just two blocks. This only leaves enough room for the second block to have size 2.

We will recycle the bases for the null spaces of the powers of A from Example KPNLT rather than recomputing them here. We will also use the same notation used in the proof of Theorem CFNLT.

To begin, s 4 = n 4 − n 3 = 6 − 5 = 1 , so we need one vector of K T 4 = ℂ 6 , that is not in K T 3 , to be a basis for Z 4 . We have a lot of latitude in this choice, and we have not described any sure-fire method for constructing a vector outside of a subspace. Looking at the basis for K T 3 we see that if a vector is in this subspace, and has a nonzero value in the first entry, then it must also have a nonzero value in the fourth entry. So the vector

z 4 , 1 = 1 0 0 0 0 0

will not be an element of K T 3 (notice that many other choices could be made here, so our basis will not be unique). This completes the determination of Z p = Z 4 .

Next, s 3 = n 3 − n 2 = 5 − 4 = 1 , so we again need just a single basis vector for Z 3 . We start by evaluating T with each basis vector of Z 4 ,

z 3 , 1 = T z 4 , 1 = A z 4 , 1 = − 3 − 3 − 3 − 3 − 3 − 2

Since s 3 = s 4 , the subspace R 3 is trivial, and there is nothing left to do, z 3 , 1 is the lone basis vector of Z 3 .

Now s 2 = n 2 − n 1 = 4 − 2 = 2 , so the construction of Z 2 will not be as simple as the construction of Z 3 . We first apply T to the basis vector of Z 2 ,

z 2 , 1 = T z 3 , 1 = A z 3 , 1 = 1 0 3 1 0 − 1

The two basis vectors of K T 1 , together with z 2 , 1 , form a basis for Q 2 . Because dim K T 2 − dim Q 2 = 4 − 3 = 1 we need only find a single basis vector for R 2 . This vector must be an element of K T 2 , but not an element of Q 2 . Again, there is a variety of vectors that fit this description, and we have no precise algorithm for finding them. Since they are plentiful, they are not too hard to find. We add up the four basis vectors of K T 2 , ensuring an element of K T 2 . Then we check to see if the vector is a linear combination of three vectors: the two basis vectors of K T 1 and z 2 , 1 . Having passed the tests, we have chosen

z 2 , 2 = 2 1 2 2 2 1

Thus, Z 2 = z 2 , 1 , z 2 , 2 .

Lastly, s 1 = n 1 − n 0 = 2 − 0 = 2 . Since s 2 = s 1 , we again have a trivial R 1 and need only complete our basis by evaluating the basis vectors of Z 2 with T ,

z 1 , 1 = T z 2 , 1 = A z 2 , 1 = 1 1 0 1 1 1 z 1 , 2 = T z 2 , 2 = A z 2 , 2 = − 2 − 2 − 5 − 2 − 1 0

Now we reorder these vectors as the desired basis,

B = z 1 , 1 , z 2 , 1 , z 3 , 1 , z 4 , 1 , z 1 , 2 , z 2 , 2

We now apply Definition MR to build a matrix representation of T relative to B ,

ρ B T z 1 , 1 = ρ B A z 1 , 1 = ρ B 0 = 0 0 0 0 0 0 ρ B T z 2 , 1 = ρ B A z 2 , 1 = ρ B z 1 , 1 = 1 0 0 0 0 0 ρ B T z 3 , 1 = ρ B A z 3 , 1 = ρ B z 2 , 1 = 0 1 0 0 0 0 ρ B T z 4 , 1 = ρ B A z 4 , 1 = ρ B z 3 , 1 = 0 0 1 0 0 0 ρ B T z 1 , 2 = ρ B A z 1 , 2 = ρ B 0 = 0 0 0 0 0 0 ρ B T z 2 , 2 = ρ B A z 2 , 2 = ρ B z 1 , 2 = 0 0 0 0 1 0

Installing these vectors as the columns of the matrix representation we have

M B , B T = 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0

which is a block diagonal matrix with Jordan blocks J 4 0 and J 2 0 . If we constructed the matrix S having the vectors of B as columns, then Theorem SCB tells us that a similarity transformation with S relates the original matrix representation of T with the matrix representation consisting of Jordan blocks., i.e.  S − 1 A S = M B , B T . ⊠

Notice that constructing interesting examples of matrix representations requires domains with dimensions bigger than just two or three. Going forward we will see several more big examples.

Section IS  Invariant Subspaces

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s s e c t i o n i s i n d r a f t f o r m
N e a r l y c o m p l e t e

We have seen in Section NLT that nilpotent linear transformations are almost never diagonalizable ( Theorem DNLT), yet have matrix representations that are very nearly diagonal ( Theorem CFNLT). Our goal in this section, and the next ( Section JCF), is to obtain a matrix representation of any linear transformation that is very nearly diagonal. A key step in reaching this goal is an understanding of invariant subspaces, and a particular type of invariant subspace that contains vectors known as “generalized eigenvectors.”

Parent topic: Chapter R  Representations

Previous topic: Section NLT  Nilpotent Linear Transformations

Next topic: Section JCF  Jordan Canonical Form

Subsection IS: Invariant Subspaces

As is often the case, we start with a definition.

Definition  IS
Invariant Subspace
Suppose that T : V → V is a linear transformation and W is a subspace of V . Suppose further that T w ∈ W for every w ∈ W . Then W is an invariant subspace of V relative to T . △

We do not have any special notation for an invariant subspace, so it is important to recognize that an invariant subspace is always relative to both a superspace ( V ) and a linear transformation ( T ), which will sometimes not be mentioned, yet will be clear from the context. Note also that the linear transformation involved must have an equal domain and codomain — the definition would not make much sense if our outputs were not of the same type as our inputs.

As usual, we begin with an example that demonstrates the existence of invariant subspaces. We will return later to understand how this example was constructed, but for now, just understand how we check the existence of the invariant subspaces.

Example  TIS
Two invariant subspaces
Consider the linear transformation T : ℂ 4 → ℂ 4 defined by T x = A x where A is given by

A = − 8 6 − 1 5 9 − 8 1 4 − 1 0 1 8 1 1 3 0 3 − 8 2 − 1 1

Define (with zero motivation),

w 1 = − 7 − 2 3 0 w 2 = − 1 − 2 0 1

and set W = w 1 , w 2 . We verify that W is an invariant subspace of ℂ 4 with respect to T . By the definition of W , any vector chosen from W can be written as a linear combination of w 1 and w 2 . Suppose that w ∈ W , and then check the details of the following verification,

T w = T a 1 w 1 + a 2 w 2   Definition SS = a 1 T w 1 + a 2 T w 2   Theorem LTLC = a 1 − 1 − 2 0 1 + a 2 5 − 2 − 3 2 = a 1 w 2 + a 2 ( − 1 ) w 1 + 2 w 2 = ( − a 2 ) w 1 + ( a 1 + 2 a 2 ) w 2 ∈ W   Definition SS

So, by Definition IS, W is an invariant subspace of ℂ 4 relative to T . In an entirely similar manner we construct another invariant subspace of T .

With zero motivation, define

x 1 = − 3 − 1 1 0 x 2 = 0 − 1 0 1

and set X = x 1 , x 2 . We verify that X is an invariant subspace of ℂ 4 with respect to T . By the definition of X , any vector chosen from X can be written as a linear combination of x 1 and x 2 . Suppose that x ∈ X , and then check the details of the following verification,

T x = T b 1 x 1 + b 2 x 2   Definition SS = b 1 T x 1 + b 2 T x 2   Theorem LTLC = b 1 3 0 − 1 1 + b 2 3 4 − 1 − 3 = b 1 ( − 1 ) x 1 + x 2 + b 2 ( − 1 ) x 1 + ( − 3 ) x 2 = ( − b 1 − b 2 ) x 1 + ( b 1 − 3 b 2 ) x 2 ∈ X   Definition SS

So, by Definition IS, X is an invariant subspace of ℂ 4 relative to T .

There is a bit of magic in each of these verifications where the two outputs of T happen to equal linear combinations of the two inputs. But this is the essential nature of an invariant subspace. We’ll have a peek under the hood later, and it won’t look so magical after all.

As a hint of things to come, verify that B = w 1 , w 2 , x 1 , x 2 is a basis of ℂ 4 . Splitting this basis in half, Theorem DSFB, tells us that ℂ 4 = W ⊕ X . To see why a decomposition of a vector space into a direct sum of invariant subspaces might be interesting, construct the matrix representation of T relative to B , M B , B T . Hmmmmmm. ⊠

Example TIS is a bit mysterious at this stage. Do we know any other examples of invariant subspaces? Yes, as it turns out, we have already seen quite a few. We’ll give some examples now, and in more general situations, describe broad classes of invariant subspaces with theorems. First up is eigenspaces.

Theorem  EIS
Eigenspaces are Invariant Subspaces
Suppose that T : V → V is a linear transformation with eigenvalue λ and associated eigenspace ℰ T λ . Let W be any subspace of ℰ T λ . Then W is an invariant subspace of V relative to T . □

Proof   Choose w ∈ W . Then

T w = λ w   Definition EELT ∈ W   Property SC

So by Definition IS, W is an invariant subspace of V relative to T . ■

Theorem EIS is general enough to determine that an entire eigenspace is an invariant subspace, or that simply the span of a single eigenvector is an invariant subspace. It is not always the case that any subspace of an invariant subspace is again an invariant subspace, but eigenspaces do have this property. Here is an example of the theorem, which also allows us to very quickly build several several invariant (4x4, 2 evs, 1 2x2 jordan, 1 2x2 diag)

Example  EIS
Eigenspaces as invariant subspaces
Define the linear transformation S : M 2 2 → M 2 2 by

S a b c d = − 2 a + 1 9 b − 3 3 c + 2 1 d − 3 a + 1 6 b − 2 4 c + 1 5 d − 2 a + 9 b − 1 3 c + 9 d − a + 4 b − 6 c + 5 d

Build a matrix representation of S relative to the standard basis (Definition MR, Example BM) and compute eigenvalues and eigenspaces of S with the computational techniques of Chapter E in concert with Theorem EER. Then

ℰ S 1 = 4 3 2 1 ℰ S 2 = 6 3 1 0 , − 9 − 3 0 1

So by Theorem EIS, both ℰ S 1 and ℰ S 2 are invariant subspaces of M 2 2 relative to S . However, Theorem EIS provides even more invariant subspaces. Since ℰ S 1 has dimension 1, it has no interesting subspaces, however ℰ S 2 has dimension 2 and has a plethora of subspaces. For example, set

u = 2 6 3 1 0 + 3 − 9 − 3 0 1 = − 6 − 3 2 3

and define U = u . Then since U is a subspace of ℰ S 2 , Theorem EIS says that U is an invariant subspace of M 2 2 (or we could check this claim directly based simply on the fact that u is an eigenvector of S ). ⊠

For every linear transformation there are some obvious, trivial invariant subspaces. Suppose that T : V → V is a linear transformation. Then simply because T is a function (Definition LT), the subspace V is an invariant subspace of T . In only a minor twist on this theme, the range of T , ℛ T , is an invariant subspace of T by Definition RLT. Finally, Theorem LTTZZ provides the justification for claiming that 0 is an invariant subspace of T .

That the trivial subspace is always an invariant subspace is a special case of the next theorem. As an easy exercise before reading the next theorem, prove that the kernel of a linear transformation ( Definition KLT), K T , is an invariant subspace. We’ll wait.

Theorem  KPIS
Kernels of Powers are Invariant Subspaces
Suppose that T : V → V is a linear transformation. Then K T k is an invariant subspace of V . □

Proof   Suppose that z ∈ K T k . Then

T k T z = T k + 1 z   Definition LTC = T T k z   Definition LTC = T 0   Definition KLT = 0   Theorem LTTZZ

So by Definition KLT, we see that T z ∈ K T k . Thus K T k is an invariant subspace of V relative to T ( Definition IS). ■

Two interesting special cases of Theorem KPIS occur when choose k = 0 and k = 1 . Rather than give an example of this theorem, we will refer you back to Example KPNLT where we work with null spaces of the first four powers of a nilpotent matrix. By Theorem KPIS each of these null spaces is an invariant subspace of the associated linear transformation.

Here’s one more example of invariant subspaces we have encountered previously.

Example  ISJB
Invariant subspaces and Jordan blocks
Refer back to Example CFNLT. We decomposed the vector space ℂ 6 into a direct sum of the subspaces Z 1 , Z 2 , Z 3 , Z 4 . The union of the basis vectors for these subspaces is a basis of ℂ 6 , which we reordered prior to building a matrix representation of the linear transformation T . A principal reason for this reordering was to create invariant subspaces (though it was not obvious then).

Define

X 1 = z 1 , 1 , z 2 , 1 , z 3 , 1 , z 4 , 1 = 1 1 0 1 1 1 , 1 0 3 1 0 − 1 , − 3 − 3 − 3 − 3 − 3 − 2 , 1 0 0 0 0 0 X 2 = z 1 , 2 , z 2 , 2 = − 2 − 2 − 5 − 2 − 1 0 , 2 1 2 2 2 1

Recall from the proof of Theorem CFNLT or the computations in Example CFNLT that first elements of X 1 and X 2 are in the kernel of T , K T , and each element of X 1 and X 2 is the output of T when evaluated with the subsequent element of the set. This was by design, and it is this feature of these basis vectors that leads to the nearly diagonal matrix representation with Jordan blocks. However, we also recognize now that this property of these basis vectors allow us to conclude easily that X 1 and X 2 are invariant subspaces of ℂ 6 relative to T .

Furthermore, ℂ 6 = X 1 ⊕ X 2 (Theorem DSFB). So the domain of T is the direct sum of invariant subspaces and the resulting matrix representation has a block diagonal form. Hmmmmm. ⊠

Subsection GEE: Generalized Eigenvectors and Eigenspaces

We now define a new type of invariant subspace and explore its key properties. This generalization of eigenvalues and eigenspaces will allow us to move from diagonal matrix representations of diagonalizable matrices to nearly diagonal matrix representations of arbitrary matrices. Here are the definitions.

Definition  GEV
Generalized Eigenvector
Suppose that T : V → V is a linear transformation. Suppose further that for x ≠ 0 , T − λ I V k x = 0 for some k > 0 . Then x is a generalized eigenvector of T with eigenvalue λ . △

Definition  GES
Generalized Eigenspace
Suppose that T : V → V is a linear transformation. Define the generalized eigenspace of T for λ as

G T λ = x T − λ I V k x = 0   for some  k ≥ 0

(This definition contains Notation GES.) △

So the generalized eigenspace is composed of generalized eigenvectors, plus the zero vector. As the name implies, the generalized eigenspace is a subspace of V . But more topically, it is an invariant subspace of V relative to T .

Theorem  GESIS
Generalized Eigenspace is an Invariant Subspace
Suppose that T : V → V is a linear transformation. Then the generalized eigenspace G T λ is an invariant subspace of V relative to T . □

Proof   First we establish that G T λ is a subspace of V . First T − λ I V 1 0 = 0 by Theorem LTTZZ, so 0 ∈ G T λ .

Suppose that x , y ∈ G T λ . Then there are integers k , ℓ such that T − λ I V k x = 0 and T − λ I V ℓ y = 0 . Set m = k + ℓ ,

T − λ I V m x + y = T − λ I V m x + T − λ I V m y   Definition LT = T − λ I V k + ℓ x + T − λ I V k + ℓ y = T − λ I V ℓ T − λ I V k x + T − λ I V k T − λ I V ℓ y   Definition LTC = T − λ I V ℓ 0 + T − λ I V k 0   Definition GES = 0 + 0   Theorem LTTZZ = 0   Property Z

So x + y ∈ G T λ .

Suppose that x ∈ G T λ and α ∈ ℂ . Then there is an integer k such that T − λ I V k x = 0 .

T − λ I V k α x = α T − λ I V k x   Definition LT = α 0   Definition GES = 0   Theorem ZVSM

So α x ∈ G T λ . By Theorem TSS, G T λ is a subspace of V .

Now we show that G T λ is invariant relative to T . Suppose that x ∈ G T λ . Then by Definition GES there is an integer k such that T − λ I V k x = 0 . The following argument is due to Zoltan Toth.

T − λ I V k T x = T − λ I V k T x − 0   Property Z = T − λ I V k T x − λ 0   Theorem ZVSM = T − λ I V k T x − λ T − λ I V k x   Definition GES = T − λ I V k T x − T − λ I V k λ x   Definition LT = T − λ I V k T x − λ x   Definition LT = T − λ I V k T − λ I V x   Definition LTA = T − λ I V k + 1 x   Definition LTC = T − λ I V T − λ I V k x   Definition LTC = T − λ I V 0   Definition GES = 0   Theorem LTTZZ

This qualifies T x for membership in G T λ , so by Definition GES, G T λ is invariant relative to T . ■

Before we compute some generalized eigenspaces, we state and prove one theorem that will make it much easier to create a generalized eigenspace, since it will allow us to use tools we already know well, and will remove some the ambiguity of the clause “for some k ” in the definition.

Theorem  GEK
Generalized Eigenspace as a Kernel
Suppose that T : V → V is a linear transformation, dim V = n , and λ is an eigenvalue of T . Then G T λ = K T − λ I V n . □

Proof   The conclusion of this theorem is a set equality, so we will apply Definition SE by establishing two set inclusions. First, suppose that x ∈ G T λ . Then there is an integer k such that T − λ I V k x = 0 . This is equivalent to the statement that x ∈ K T − λ I V k . No matter what the value of k is, Theorem KPLT gives

x ∈ K T − λ I V k ⊆ K T − λ I V n

So, G T λ ⊆ K T − λ I V n . For the opposite inclusion, suppose y ∈ K T − λ I V n . Then T − λ I V n y = 0 , so y ∈ G T λ and thus K T − λ I V n ⊆ G T λ . By Definition SE we have the desired equality of sets. ■

Theorem GEK allows us to compute generalized eigenspaces as a single kernel (or null space of a matrix representation) with tools like Theorem KNSI and Theorem BNS. Also, we do not need to consider all possible powers k and can simply consider the case where k = n . It is worth noting that the “regular” eigenspace is a subspace of the generalized eigenspace since

ℰ T λ = K T − λ I V 1 ⊆ K T − λ I V n = G T λ

where the subset inclusion is a consequence of Theorem KPLT. Also, there is no such thing as a “generalized eigenvalue.” If λ is not an eigenvalue of T , then the kernel of T − λ I V is trivial and therefore subsequent powers of T − λ I V also have trivial kernels ( Theorem KPLT). So the generalized eigenspace of a scalar that is not already an eigenvalue would be trivial. Alright, we know enough now to compute some generalized eigenspaces. We will record some information about algebraic and geometric multiplicities of eigenvalues (Definition AME, Definition GME) as we go, since these observations will be of interest in light of some future theorems.

Example  GE4
Generalized eigenspaces, dimension 4 domain
In Example TIS we presented two invariant subspaces of ℂ 4 . There was some mystery about just how these were constructed, but we can now reveal that they are generalized eigenspaces. Example TIS featured T : ℂ 4 → ℂ 4 defined by T x = A x with A given by

A = − 8 6 − 1 5 9 − 8 1 4 − 1 0 1 8 1 1 3 0 3 − 8 2 − 1 1

A matrix representation of T relative to the standard basis ( Definition SUV) will equal A . So we can analyze A with the techniques of Chapter E. Doing so, we find two eigenvalues, λ = 1 , − 2 , with multiplicities,

α T 1 = 2 γ T 1 = 1 α T − 2 = 2 γ T − 2 = 1

To apply Theorem GEK we subtract each eigenvalue from the diagonal entries of A , raise the result to the power dim ℂ 4 = 4 , and compute a basis for the null space.

λ = − 2 A − ( − 2 ) I 4 4 = 6 4 8 − 1 2 1 5 7 2 9 − 1 2 1 5 − 3 2 4 4 8 6 − 4 8 6 4 8 6 − 4 0 5 7 2 9 − 4 8 6 7 2 9 2 9 7 − 4 8 6 4 0 5 − 4 8 6 →  RREF 1 0 3 0 0 1 1 1 0 0 0 0 0 0 0 0 G T − 2 = − 3 − 1 1 0 , 0 − 1 0 1 λ = 1 A − ( 1 ) I 4 4 = 8 1 − 4 0 5 − 8 1 − 7 2 9 − 1 0 8 − 1 8 9 − 3 7 8 − 4 8 6 − 2 7 1 3 5 2 7 2 4 3 1 3 5 5 4 3 5 1 2 4 3 →  RREF 1 0 7 3 1 0 1 2 3 2 0 0 0 0 0 0 0 0 G T 1 = − 7 − 2 3 0 , − 1 − 2 0 1

In Example TIS we concluded that these two invariant subspaces formed a direct sum of ℂ 4 , only at that time, they were called X and W . Now we can write

ℂ 4 = G T 1 ⊕ G T − 2

This is no accident. Notice that the dimension of each of these invariant subspaces is equal to the algebraic multiplicity of the associated eigenvalue. Not an accident either. (See the upcoming Theorem GESD.) ⊠

Example  GE6
Generalized eigenspaces, dimension 6 domain
Define the linear transformation S : ℂ 6 → ℂ 6 by S x = B x where

2 − 4 2 5 − 5 4 9 0 − 3 7 2 − 3 4 − 1 6 2 6 − 8 2 − 3 4 − 1 5 2 4 − 7 1 0 − 1 8 6 − 3 6 5 1 − 2 8 − 1 4 0 − 2 1 2 8 4 5 − 7 − 6 − 7 8 7

Then B will be the matrix representation of S relative to the standard basis ( Definition SUV) and we can use the techniques of Chapter E applied to B in order to find the eigenvalues of S .

α S 3 = 2 γ S 3 = 1 α S − 1 = 4 γ S − 1 = 2

To find the generalized eigenspaces of S we need to subtract an eigenvalue from the diagonal elements of B , raise the result to the power dim ℂ 6 = 6 and compute the null space. Here are the results for the two eigenvalues of S ,

λ = 3 B − 3 I 6 6 = 6 4 0 0 0 − 1 5 2 5 7 6 − 5 9 9 0 4 2 6 1 1 2 − 9 5 7 4 4 1 3 3 6 3 2 1 5 8 7 2 − 3 9 9 3 6 − 1 1 7 7 6 8 7 0 4 − 2 9 1 8 4 3 6 3 5 2 1 2 0 3 2 − 3 0 2 0 8 − 9 9 8 4 6 4 0 0 − 2 0 7 3 6 2 6 3 6 8 − 1 5 3 6 1 1 2 6 4 − 2 3 0 4 0 1 7 9 2 0 − 1 7 9 2 0 − 1 5 3 6 − 9 7 2 8 2 7 6 4 8 − 6 6 5 6 9 7 2 8 − 1 5 3 6 − 1 7 9 2 0 − 7 9 3 6 1 7 9 2 0 5 8 8 8 1 7 9 2 4 3 5 2 − 1 4 0 8 0 →  RREF 1 0 0 0 − 4 5 0 1 0 0 − 1 1 0 0 1 0 − 1 1 0 0 0 1 − 2 1 0 0 0 0 0 0 0 0 0 0 0 0 G S 3 = 4 1 1 2 1 0 , − 5 − 1 − 1 − 1 0 1 λ = − 1 B − ( − 1 ) I 6 6 = 6 1 4 4 − 1 6 3 8 4 1 8 4 3 2 − 3 6 8 6 4 5 7 3 4 4 − 1 8 4 3 2 4 0 9 6 − 8 1 9 2 4 0 9 6 − 1 6 3 8 4 2 4 5 7 6 − 4 0 9 6 4 0 9 6 − 8 1 9 2 4 0 9 6 − 1 6 3 8 4 2 4 5 7 6 − 4 0 9 6 1 8 4 3 2 − 3 2 7 6 8 6 1 4 4 − 6 1 4 4 0 9 0 1 1 2 − 6 1 4 4 1 4 3 3 6 − 2 4 5 7 6 2 0 4 8 − 4 5 0 5 6 6 5 5 3 6 − 2 0 4 8 1 0 2 4 0 − 1 6 3 8 4 − 2 0 4 8 − 2 8 6 7 2 4 0 9 6 0 2 0 4 8 →  RREF 1 0 − 5 2 − 4 5 0 1 − 3 3 − 5 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G S − 1 = 5 3 1 0 0 0 , − 2 − 3 0 1 0 0 , 4 5 0 0 1 0 , − 5 − 3 0 0 0 1

If we take the union of the two bases for these two invariant subspaces we obtain the set

C = v 1 , v 2 , v 3 , v 4 , v 5 , v 6 = 4 1 1 2 1 0 , − 5 − 1 − 1 − 1 0 1 , 5 3 1 0 0 0 , − 2 − 3 0 1 0 0 , 4 5 0 0 1 0 , − 5 − 3 0 0 0 1

You can check that this set is linearly independent (right now we have no guarantee this will happen). Once this is verified, we have a linearly independent set of size 6 inside a vector space of dimension 6, so by Theorem G, the set C is a basis for ℂ 6 . This is enough to apply Theorem DSFB and conclude that

ℂ 6 = G S 3 ⊕ G S − 1

This is no accident. Notice that the dimension of each of these invariant subspaces is equal to the algebraic multiplicity of the associated eigenvalue. Not an accident either. (See the upcoming Theorem GESD.) ⊠

Subsection RLT: Restrictions of Linear Transformations

Generalized eigenspaces will prove to be an important type of invariant subspace. A second reason for our interest in invariant subspaces is they provide us with another method for creating new linear transformations from old ones.

Definition  LTR
Linear Transformation Restriction
Suppose that T : V → V is a linear transformation, and U is an invariant subspace of V relative to T . Define the restriction of T to U by

T | U : U → U T | U u = T u

(This definition contains Notation LTR.) △

It might appear that this definition has not accomplished anything, as T | U would appear to take on exactly the same values as T . And this is true. However, T | U differs from T in the choice of domain and codomain. We tend to give little attention to the domain and codomain of functions, while their defining rules get the spotlight. But the restriction of a linear transformation is all about the choice of domain and codomain. We are restricting the rule of the function to a smaller subspace. Notice the importance of only using this construction with an invariant subspace, since otherwise we cannot be assured that the outputs of the function are even contained in the codomain. Maybe this observation should be the key step in the proof of a theorem saying that T | U is also a linear transformation, but we won’t bother.

Example  LTRGE
Linear transformation restriction on generalized eigenspace
In order to gain some experience with restrictions of linear transformations, we construct one and then also construct a matrix representation for the restriction. Furthermore, we will use a generalized eigenspace as the invariant subspace for the construction of the restriction.

Consider the linear transformation T : ℂ 5 → ℂ 5 defined by T x = A x , where

A = − 2 2 − 2 4 − 2 4 − 2 4 − 4 6 3 2 6 0 1 1 − 1 2 − 1 6 − 6 − 1 4 − 1 7 6 8 4 1 0 8 1 1 1 4 8 1 3 1 8

One of the eigenvalues of A is λ = 2 , with geometric multiplicity γ T 2 = 1 , and algebraic multiplicity α T 2 = 3 . We get the generalized eigenspace in the usual manner,

W = G T 2 = K T − 2 I ℂ 5 5 = − 2 1 1 0 0 , 0 − 1 0 1 0 , − 4 2 0 0 1 = w 1 , w 2 , w 3

By Theorem GESIS, we know W is invariant relative to T , so we can employ Definition LTR to form the restriction, T | W : W → W .

To better understand exactly what a restriction is (and isn’t), we’ll form a matrix representation of T | W . This will also be a skill we will use in subsequent examples. For a basis of W we will use C = w 1 , w 2 , w 3 . Notice that dim W = 3 , so our matrix representation will be a square matrix of size 3. Applying Definition MR, we compute

ρ C T w 1 = ρ C A w 1 = ρ C − 4 2 2 0 0 = ρ C 2 − 2 1 1 0 0 + 0 0 − 1 0 1 0 + 0 − 4 2 0 0 1 = 2 0 0 ρ C T w 2 = ρ C A w 2 = ρ C 0 − 2 2 2 − 1 = ρ C 2 − 2 1 1 0 0 + 2 0 − 1 0 1 0 + ( − 1 ) − 4 2 0 0 1 = 2 2 − 1 ρ C T w 3 = ρ C A w 3 = ρ C − 6 3 − 1 0 2 = ρ C ( − 1 ) − 2 1 1 0 0 + 0 0 − 1 0 1 0 + 2 − 4 2 0 0 1 = − 1 0 2

So the matrix representation of T | W relative to C is

M C , C T | W = 2 2 − 1 0 2 0 0 − 1 2

The question arises: how do we use a 3 × 3 matrix to compute with vectors from ℂ 5 ? To answer this question, consider the randomly chosen vector

w = − 4 4 4 − 2 − 1

First check that w ∈ G T 2 . There are two ways to do this, first verify that

T − 2 I ℂ 5 5 w = A − 2 I 5 5 w = 0

meeting Definition GES (with k = 5 ). Or, express w as a linear combination of the basis C for W , to wit, w = 4 w 1 − 2 w 2 − w 3 . Now compute T | W w directly using Definition LTR,

T | W w = T w = A w = − 1 0 9 5 − 4 0

It was necessary to verify that w ∈ G T 2 , and if we trust our work so far, then this output will also be an element of W , but it would be wise to check this anyway (using either of the methods we used for w ). We’ll wait.

Now we will repeat this sample computation, but instead using the matrix representation of T | W relative to C .

T | W w = ρ C − 1 M C , C T | W ρ C w   Theorem FTMR = ρ C − 1 M C , C T | W ρ C 4 w 1 − 2 w 2 − w 3 = ρ C − 1 2 2 − 1 0 2 0 0 − 1 2 4 − 2 − 1   Definition VR = ρ C − 1 5 − 4 0   Definition MVP = 5 w 1 − 4 w 2 + 0 w 3   Definition VR = 5 − 2 1 1 0 0 + ( − 4 ) 0 − 1 0 1 0 + 0 − 4 2 0 0 1 = − 1 0 9 5 − 4 0

which matches the previous computation. Notice how the “action” of T | W is accomplished by a 3 × 3 matrix multiplying a column vector of size 3. If you would like more practice with these sorts of computations, mimic the above using the other eigenvalue of T , which is λ = − 2 . The generalized eigenspace has dimension 2, so the matrix representation of the restriction to the generalized eigenspace will be a 2 × 2 matrix. ⊠

Suppose that T : V → V is a linear transformation and we can find a decomposition of V as a direct sum, say V = U 1 ⊕ U 2 ⊕ U 3 ⊕ ⋯ ⊕ U m where each U i is an invariant subspace of V relative to T . Then, for any v ∈ V there is a unique decomposition v = u 1 + u 2 + u 3 + ⋯ + u m with u i ∈ U i , 1 ≤ i ≤ m and furthermore

T v = T u 1 + u 2 + u 3 + ⋯ + u m   Definition DS = T u 1 + T u 2 + T u 3 + ⋯ + T u m   Theorem LTLC = T | U 1 u 1 + T | U 2 u 2 + T | U 3 u 3 + ⋯ + T | U m u m

So in a very real sense, we obtain a decomposition of the linear transformation T into the restrictions T | U i , 1 ≤ i ≤ m . If we wanted to be more careful, we could extend each restriction to a linear transformation defined on V by setting the output of T | U i to be the zero vector for inputs outside of U i . Then T would be exactly equal to the sum (Definition LTA) of these extended restrictions. However, the irony of extending our restrictions is more than we could handle right now.

Our real interest is in the matrix representation of a linear transformation when the domain decomposes as a direct sum of invariant subspaces. Consider forming a basis B of V as the union of bases B i from the individual U i , i.e.  B = ∪ i = 1 m B i . Now form the matrix representation of T relative to B . The result will be block diagonal, where each block is the matrix representation of a restriction T | U i relative to a basis B i , M B i , B i T | U i . Though we did not have the definitions to describe it then, this is exactly what was going on in the latter portion of the proof of Theorem CFNLT. Two examples should help to clarify these ideas.

Example  ISMR4
Invariant subspaces, matrix representation, dimension 4 domain
Example TIS and Example GE4 describe a basis of ℂ 4 which is derived from bases for two invariant subspaces (both generalized eigenspaces). In this example we will construct a matrix representation of the linear transformation T relative to this basis. Recycling the notation from Example TIS, we work with the basis,

B = w 1 , w 2 , x 1 , x 2 = − 7 − 2 3 0 , − 1 − 2 0 1 , − 3 − 1 1 0 , 0 − 1 0 1

Now we compute the matrix representation of T relative to B , borrowing some computations from Example TIS,

ρ B T w 1 = ρ B − 1 − 2 0 1 = ρ B ( 0 ) w 1 + ( 1 ) w 2 = 0 1 0 0 ρ B T w 2 = ρ B 5 − 2 − 3 2 = ρ B ( − 1 ) w 1 + ( 2 ) w 2 = − 1 2 0 0 ρ B T x 1 = ρ B 3 0 − 1 1 = ρ B ( − 1 ) x 1 + ( 1 ) x 2 = 0 0 − 1 1 ρ B T x 2 = ρ B 3 4 − 1 − 3 = ρ B ( − 1 ) x 1 + ( − 3 ) x 2 = 0 0 − 1 − 3

Applying Definition MR, we have

M B , B T = 0 − 1 0 0 1 2 0 0 0 0 − 1 − 1 0 0 1 − 3

The interesting feature of this representation is the two 2 × 2 blocks on the diagonal that arise from the decomposition of ℂ 4 into a direct sum (of generalized eigenspaces). Or maybe the interesting feature of this matrix is the two 2 × 2 submatrices in the “other” corners that are all zero. You decide. ⊠

Example  ISMR6
Invariant subspaces, matrix representation, dimension 6 domain
In Example GE6 we computed the generalized eigenspaces of the linear transformation S : ℂ 6 → ℂ 6 by S x = B x where

2 − 4 2 5 − 5 4 9 0 − 3 7 2 − 3 4 − 1 6 2 6 − 8 2 − 3 4 − 1 5 2 4 − 7 1 0 − 1 8 6 − 3 6 5 1 − 2 8 − 1 4 0 − 2 1 2 8 4 5 − 7 − 6 − 7 8 7

From this we found the basis

C = v 1 , v 2 , v 3 , v 4 , v 5 , v 6 = 4 1 1 2 1 0 , − 5 − 1 − 1 − 1 0 1 , 5 3 1 0 0 0 , − 2 − 3 0 1 0 0 , 4 5 0 0 1 0 , − 5 − 3 0 0 0 1

of ℂ 6 where v 1 , v 2 is a basis of G S 3 and v 3 , v 4 , v 5 , v 6 is a basis of G S − 1 . We can employ C in the construction of a matrix representation of S (Definition MR). Here are the computations,

ρ C S v 1 = ρ C 1 1 3 3 7 4 1 = ρ C 4 v 1 + 1 v 2 = 4 1 0 0 0 0 ρ C S v 2 = ρ C − 1 4 − 3 − 3 − 4 − 1 2 = ρ C ( − 1 ) v 1 + 2 v 2 = − 1 2 0 0 0 0 ρ C S v 3 = ρ C 2 3 5 5 2 − 2 − 2 = ρ C 5 v 3 + 2 v 4 + ( − 2 ) v 5 + ( − 2 ) v 6 = 0 0 5 2 − 2 − 2 ρ C S v 4 = ρ C − 4 6 − 1 1 − 1 0 − 2 5 4 = ρ C ( − 1 0 ) v 3 + ( − 2 ) v 4 + 5 v 5 + 4 v 6 = 0 0 − 1 0 − 2 5 4 ρ C S v 5 = ρ C 7 8 1 9 1 7 1 − 1 0 − 7 = ρ C 1 7 v 3 + 1 v 4 + ( − 1 0 ) v 5 + ( − 7 ) v 6 = 0 0 1 7 1 − 1 0 − 7 ρ C S v 6 = ρ C − 3 5 − 9 − 8 2 6 3 = ρ C ( − 8 ) v 3 + 2 v 4 + 6 v 5 + 3 v 6 = 0 0 − 8 2 6 3

These column vectors are the columns of the matrix representation, so we obtain

M C , C S = 4 − 1 0 0 0 0 1 2 0 0 0 0 0 0 5 − 1 0 1 7 − 8 0 0 2 − 2 1 2 0 0 − 2 5 − 1 0 6 0 0 − 2 4 − 7 3

As before, the key feature of this representation is the 2 × 2 and 4 × 4 blocks on the diagonal. We will discover in the final theorem of this section ( Theorem RGEN) that we already understand these blocks fairly well. For now, we recognize them as arising from generalized eigenspaces and suspect that their sizes are equal to the algebraic multiplicities of the eigenvalues. ⊠

The paragraph prior to these last two examples is worth repeating. A basis derived from a direct sum decomposition into invariant subspaces will provide a matrix representation of a linear transformation with a block diagonal form.

Diagonalizing a linear transformation is the most extreme example of decomposing a vector space into invariant subspaces. When a linear transformation is diagonalizable, then there is a basis composed of eigenvectors (Theorem DC). Each of these basis vectors can be used individually as the lone element of a spanning set for an invariant subspace ( Theorem EIS). So the domain decomposes into a direct sum of one-dimensional invariant subspaces (Theorem DSFB). The corresponding matrix representation is then block diagonal with all the blocks of size 1, i.e. the matrix is diagonal. Section NLT, Section IS and Section JCF are all devoted to generalizing this extreme situation when there are not enough eigenvectors available to make such a complete decomposition and arrive at such an elegant matrix representation.

One last theorem will roll up much of this section and Section NLT into one nice, neat package.

Theorem  RGEN
Restriction to Generalized Eigenspace is Nilpotent
Suppose T : V → V is a linear transformation with eigenvalue λ . Then the linear transformation T | G T λ − λ I G T λ is nilpotent. □

Proof   Notice first that every subspace of V is invariant with respect to I V , so I G T λ = I V | G T λ . Let n = dim V and choose v ∈ G T λ . Then

T | G T λ − λ I G T λ n v = T − λ I V n v   Definition LTR = 0   Theorem GEK

So by Definition NLT, T | G T λ − λ I G T λ is nilpotent. ■

The proof of Theorem RGEN indicates that the index of the nilpotent linear transformation is less than or equal to the dimension of V . In practice, it will be less than or equal to the dimension of the domain of the linear transformation, G T λ . In any event, the exact value of this index will be of some interest, so we define it now. Notice that this is a property of the eigenvalue λ , similar to the algebraic and geometric multiplicities (Definition AME, Definition GME).

Definition  IE
Index of an Eigenvalue
Suppose T : V → V is a linear transformation with eigenvalue λ . Then the index of λ , ι T λ , is the index of the nilpotent linear transformation T | G T λ − λ I G T λ .

(This definition contains Notation IE.) △

Example  GENR6
Generalized eigenspaces and nilpotent restrictions, dimension 6 domain
In Example GE6 we computed the generalized eigenspaces of the linear transformation S : ℂ 6 → ℂ 6 defined by S x = B x where

2 − 4 2 5 − 5 4 9 0 − 3 7 2 − 3 4 − 1 6 2 6 − 8 2 − 3 4 − 1 5 2 4 − 7 1 0 − 1 8 6 − 3 6 5 1 − 2 8 − 1 4 0 − 2 1 2 8 4 5 − 7 − 6 − 7 8 7

The generalized eigenspace, G S 3 , has dimension 2 , while G S − 1 , has dimension 4 . We’ll investigate each thoroughly in turn, with the intent being to illustrate Theorem RGEN. Much of our computations will be repeats of those done in Example ISMR6.

For U = G S 3 we compute a matrix representation of S | U using the basis found in Example GE6,

B = u 1 , u 2 = 4 1 1 2 1 0 , − 5 − 1 − 1 − 1 0 1

Since B has size 2, we obtain a 2 × 2 matrix representation (Definition MR) from

ρ B S | U u 1 = ρ B 1 1 3 3 7 4 1 = ρ B 4 u 1 + u 2 = 4 1 ρ B S | U u 2 = ρ B − 1 4 − 3 − 3 − 4 − 1 2 = ρ B ( − 1 ) u 1 + 2 u 2 = − 1 2

Thus

M = M U , U S | U = 4 − 1 1 2

Now we can illustrate Theorem RGEN with powers of the matrix representation (rather than the restriction itself),

M − 3 I 2 = 1 − 1 1 − 1 M − 3 I 2 2 = 0 0 0 0

So M − 3 I 2 is a nilpotent matrix of index 2 (meaning that S | U − 3 I U is a nilpotent linear transformation of index 2) and according to Definition IE we say ι S 3 = 2 .

For W = G S − 1 we compute a matrix representation of S | W using the basis found in Example GE6,

C = w 1 , w 2 , w 3 , w 4 = 5 3 1 0 0 0 , − 2 − 3 0 1 0 0 , 4 5 0 0 1 0 , − 5 − 3 0 0 0 1

Since C has size 4, we obtain a 4 × 4 matrix representation (Definition MR) from

ρ C S | W w 1 = ρ C 2 3 5 5 2 − 2 − 2 = ρ C 5 w 1 + 2 w 2 + ( − 2 ) w 3 + ( − 2 ) w 4 = 5 2 − 2 − 2 ρ C S | W w 2 = ρ C − 4 6 − 1 1 − 1 0 − 2 5 4 = ρ C ( − 1 0 ) w 1 + ( − 2 ) w 2 + 5 w 3 + 4 w 4 = − 1 0 − 2 5 4 ρ C S | W w 3 = ρ C 7 8 1 9 1 7 1 − 1 0 − 7 = ρ C 1 7 w 1 + w 2 + ( − 1 0 ) w 3 + ( − 7 ) w 4 = 1 7 1 − 1 0 − 7 ρ C S | W w 4 = ρ C − 3 5 − 9 − 8 2 6 3 = ρ C ( − 8 ) w 1 + 2 w 2 + 6 w 3 + 3 w 4 = − 8 2 6 3

Thus

N = M W , W S | W = 5 − 1 0 1 7 − 8 2 − 2 1 2 − 2 5 − 1 0 6 − 2 4 − 7 3

Now we can illustrate Theorem RGEN with powers of the matrix representation (rather than the restriction itself),

N − ( − 1 ) I 4 = 6 − 1 0 1 7 − 8 2 − 1 1 2 − 2 5 − 9 6 − 2 4 − 7 4 N − ( − 1 ) I 4 2 = − 2 3 − 5 2 4 − 6 1 0 − 4 4 − 6 1 0 − 4 2 − 3 5 − 2 N − ( − 1 ) I 4 3 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

So N − ( − 1 ) I 4 is a nilpotent matrix of index 3 (meaning that S | W − ( − 1 ) I W is a nilpotent linear transformation of index 3) and according to Definition IE we say ι S − 1 = 3 .

Notice that if we were to take the union of the two bases of the generalized eigenspaces, we would have a basis for ℂ 6 . Then a matrix representation of S relative to this basis would be the same block diagonal matrix we found in Example ISMR6, only we now understand each of these blocks as being very close to being a nilpotent matrix. ⊠

Invariant subspaces, and restrictions of linear transformations, are topics you will see again and again if you continue with further study of linear algebra. Our reasons for discussing them now is to arrive at a nice matrix representation of the restriction of a linear transformation to one of its generalized eigenspaces. Here’s the theorem.

Theorem  MRRGE
Matrix Representation of a Restriction to a Generalized Eigenspace
Suppose that T : V → V is a linear transformation with eigenvalue λ . Then there is a basis of the the generalized eigenspace G T λ such that the restriction T | G T λ : G T λ → G T λ has a matrix representation that is block diagonal where each block is a Jordan block of the form J n λ . □

Proof   Theorem RGEN tells us that T | G T λ − λ I G T λ is a nilpotent linear transformation. Theorem CFNLT tells us that a nilpotent linear transformation has a basis for its domain that yields a matrix representation that is block diagonal where the blocks are Jordan blocks of the form J n 0 . Let B be a basis of G T λ that yields such a matrix representation for T | G T λ − λ I G T λ .

By Definition LTA, we can write

T | G T λ = T | G T λ − λ I G T λ + λ I G T λ

The matrix representation of λ I G T λ relative to the basis B is then simply the diagonal matrix λ I m , where m = dim G T λ . By Theorem MRSLT we have the rather unwieldy expression,

M B , B T | G T λ = M B , B T | G T λ − λ I G T λ + λ I G T λ = M B , B T | G T λ − λ I G T λ + M B , B I G T λ

The first of these matrix representations has Jordan blocks with zero in every diagonal entry, while the second matrix representation has λ in every diagonal entry. The result of adding the two representations is to convert the Jordan blocks from the form J n 0 to the form J n λ . ■

Of course, Theorem CFNLT provides some extra information on the sizes of the Jordan blocks in a representation and we could carry over this information to Theorem MRRGE, but will save that for a subsequent application of this result.

Subsection EXC: Exercises

T10 Suppose that T : V → V is linear transformation, and p ( x ) is a polynomial. Then define the new linear transformation p ( T ) : V → V by interpreting the coefficients of the terms of the polynomial as scalar mutliples of linear transformations (Definition LTSM), addition of terms as the sum of linear transformations (Definition LTA), and powers as repeated composition of linear transformations (Definition LTC). Prove that T ∘ p ( T ) = p ( T ) ∘ T .

Use this observation to give a shorter argument for the proof of the invariance of the generalized eigenspace in Theorem GESIS.  
Contributed by  Robert Beezer

Section JCF  Jordan Canonical Form

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s s e c t i o n i s i n d r a f t f o r m
N e e d s e x a m p l e s n e a r b e g i n n i n g

We have seen in Section IS that generalized eigenspaces are invariant subspaces that in every instance have led to a direct sum decomposition of the domain of the associated linear transformation. This allows us to create a block diagonal matrix representation ( Example ISMR4, Example ISMR6). We also know from Theorem RGEN that the restriction of a linear transformation to a generalized eigenspace is almost a nilpotent linear transformation. Of course, we understand nilpotent linear transformations very well from Section NLT and we have carefully determined a nice matrix representation for them.

So here is the game plan for the final push. Prove that the domain of a linear transformation always decomposes into a direct sum of generalized eigenspaces. We have unravelled Theorem RGEN at Theorem MRRGE so that we can formulate the matrix representations of the restrictions on the generalized eigenspaces using our storehouse of results about nilpotent linear transformations. Arrive at a matrix representation of any linear transformation that is block diagonal with each block being a Jordan block.

Parent topic: Chapter R  Representations

Previous topic: Section IS  Invariant Subspaces

Next topic: Appendix CN  Computation Notes

Subsection GESD: Generalized Eigenspace Decomposition

In Theorem UTMR we were able to show that any linear transformation from V to V has an upper triangular matrix representation ( Definition UTM). We will now show that we can improve on the basis yielding this representation by massaging the basis so that the matrix representation is also block diagonal. The subspaces associated with each block will be generalized eigenspaces, so the most general result will be a decomposition of the domain of a linear transformation into a direct sum of generalized eigenspaces.

Theorem  GESD
Generalized Eigenspace Decomposition
Suppose that T : V → V is a linear transformation with distinct eigenvalues λ 1 , λ 2 , λ 3 , … , λ m . Then

V = G T λ 1 ⊕ G T λ 2 ⊕ G T λ 3 ⊕ ⋯ ⊕ G T λ m

□

Proof   Suppose that dim V = n and the n (not necessarily distinct) eigenvalues of T are ρ 1 , ρ 2 , ρ 3 , … , ρ n . We begin with a basis of V that yields an upper triangular matrix representation, as guaranteed by Theorem UTMR, B = x 1 , x 2 , x 3 , … , x n . Since the matrix representation is upper triangular, and the eigenvalues of the linear transformation are the diagonal elements we can choose this basis so that there are then scalars a i j , 1 ≤ j ≤ n , 1 ≤ i ≤ j − 1 such that

T x j = ∑ i = 1 j − 1 a i j x i + ρ j x j

We now define a new basis for V which is just a slight variation in the basis B . Choose any k and ℓ such that 1 ≤ k < ℓ ≤ n and ρ k ≠ ρ ℓ . Define the scalar α = a k l ∕ ρ ℓ − ρ k . The new basis is C = y 1 , y 2 , y 3 , … , y n where

y j = x j , j ≠ ℓ , 1 ≤ j ≤ n y ℓ = x ℓ + α x k

We now compute the values of the linear transformation T with inputs from C , noting carefully the changed scalars in the linear combinations of C describing the outputs. These changes will translate to minor changes in the matrix representation built using the basis C . There are three cases to consider, depending on which column of the matrix representation we are examining. First, assume j < ℓ . Then

T y j = T x j = ∑ i = 1 j − 1 a i j x i + ρ j x j = ∑ i = 1 j − 1 a i j y i + ρ j y j

That seems a bit pointless. The first ℓ − 1 columns of the matrix representations of T relative to B and C are identical. OK, if that was too easy, here’s the main act. Assume j = ℓ . Then

T y ℓ = T x ℓ + α x k = T x ℓ + α T x k = ∑ i = 1 ℓ − 1 a i ℓ x i + ρ ℓ x ℓ + α ∑ i = 1 k − 1 a i k x i + ρ k x k = ∑ i = 1 ℓ − 1 a i ℓ x i + ρ ℓ x ℓ + ∑ i = 1 k − 1 α a i k x i + α ρ k x k = ∑ i = 1 ℓ − 1 a i ℓ x i + ∑ i = 1 k − 1 α a i k x i + α ρ k x k + ρ ℓ x ℓ = ∑ i = 1 i ≠ k ℓ − 1 a i ℓ x i + ∑ i = 1 k − 1 α a i k x i + a k l x k + α ρ k x k + ρ ℓ x ℓ = ∑ i = 1 i ≠ k ℓ − 1 a i ℓ x i + ∑ i = 1 k − 1 α a i k x i + a k l x k + α ρ k x k − ρ ℓ α x k + ρ ℓ α x k + ρ ℓ x ℓ = ∑ i = 1 i ≠ k ℓ − 1 a i ℓ x i + ∑ i = 1 k − 1 α a i k x i + a k l + α ρ k − ρ ℓ α x k + ρ ℓ α x k + x ℓ = ∑ i = 1 i ≠ k ℓ − 1 a i ℓ x i + ∑ i = 1 k − 1 α a i k x i + a k l + α ρ k − ρ ℓ x k + ρ ℓ x ℓ + α x k = ∑ i = 1 i ≠ k ℓ − 1 a i ℓ y i + ∑ i = 1 k − 1 α a i k y i + a k l + α ρ k − ρ ℓ y k + ρ ℓ y ℓ

So how different are the matrix representations relative to B and C in column ℓ ? For i > k , the coefficient of y i is a i j , as in the representation relative to B . It is a different story for i ≤ k , where the coefficients of y i may be very different. We are especially interested in the coefficient of y k . In fact, this whole first part of this proof is about this particular entry of the matrix representation. The coefficient of y k is

a k l + α ρ k − ρ ℓ = a k l + a k l ρ ℓ − ρ k ρ k − ρ ℓ = a k l + ( − 1 ) a k l = 0

If the definition of α was a mystery, then no more. In the matrix representation of T relative to C , the entry in column ℓ , row k is a zero. Nice. The only price we pay is that other entries in column ℓ , specifically rows 1 through k − 1 , may also change in a way we can’t control.

One more case to consider. Assume j > ℓ . Then

T y j = T x j = ∑ i = 1 j − 1 a i j x i + ρ j x j = ∑ i = 1 i ≠ ℓ , k j − 1 a i j x i + a ℓ j x ℓ + a k j x k + ρ j x j = ∑ i = 1 i ≠ ℓ , k j − 1 a i j x i + a ℓ j x ℓ + α a ℓ j x k − α a ℓ j x k + a k j x k + ρ j x j = ∑ i = 1 i ≠ ℓ , k j − 1 a i j x i + a ℓ j x ℓ + α x k + a k j − α a ℓ j x k + ρ j x j = ∑ i = 1 i ≠ ℓ , k j − 1 a i j y i + a ℓ j y ℓ + a k j − α a ℓ j y k + ρ j y j

As before, we ask: how different are the matrix representations relative to B and C in column j ? Only y k has a coefficient different from the corresponding coefficient when the basis is B . So in the matrix representations, the only entries to change are in row k , for columns ℓ + 1 through n .

What have we accomplished? With a change of basis, we can place a zero in a desired entry (row k , column ℓ ) of the matrix representation, leaving most of the entries untouched. The only entries to possibly change are above the new zero entry, or to the right of the new zero entry. Suppose we repeat this procedure, starting by “zeroing out” the entry above the diagonal in the second column and first row. Then we move right to the third column, and zero out the element just above the diagonal in the second row. Next we zero out the element in the third column and first row. Then tackle the fourth column, work upwards from the diagonal, zeroing out elements as we go. Entries above, and to the right will repeatedly change, but newly created zeros will never get wrecked, since they are below, or just to the left of the entry we are working on. Similarly the values on the diagonal do not change either. This entire argument can be retooled in the language of change-of-basis matrices and similarity transformations, and this is the approach taken by Noble in his Applied Linear Algebra. It is interesting to concoct the change-of-basis matrix between the matrices B and C and compute the inverse.

Perhaps you have noticed that we have to be just a bit more careful than the previous paragraph suggests. The definition of α has a denominator that cannot be zero, which restricts our maneuvers to zeroing out entries in row k and column ℓ only when ρ k ≠ ρ ℓ . So we do not necessarily arrive at a diagonal matrix. More carefully we can write

T y j = ∑ i = 1 i : ρ i = ρ j j − 1 b i j y i + ρ j y j

where the b i j are our new coefficients after repeated changes, the y j are the new basis vectors, and the condition “ i : ρ i = ρ j ” means that we only have terms in the sum involving vectors whose final coefficients are identical diagonal values (the eigenvalues). Now reorder the basis vectors carefully. Group together vectors that have equal diagonal entries in the matrix representation, but within each group preserve the order of the precursor basis. This grouping will create a block diagonal structure for the matrix representation, while otherwise preserving the order of the basis will retain the upper triangular form of the representation. So we can arrive at a basis that yields a matrix representation that is upper triangular and block diagonal, with the diagonal entries of each block all equal to a common eigenvalue of the linear transformation.

More carefully, employing the distinct eigenvalues of T , λ i , 1 ≤ i ≤ m , we can assert there is a set of basis vectors for V , u i j , 1 ≤ i ≤ m , 1 ≤ j ≤ α T λ i , such that

T u i j = ∑ k = 1 j − 1 b i j k u i k + λ i u i j

So the subspace U i = u i j 1 ≤ j ≤ α T λ i , 1 ≤ i ≤ m is an invariant subspace of V relative to T and the restriction T | U i has an upper triangular matrix representation relative to the basis u i j 1 ≤ j ≤ α T λ i where the diagonal entries are all equal to λ i . Notice too that with this definition,

V = U 1 ⊕ U 2 ⊕ U 3 ⊕ ⋯ ⊕ U m

Whew. This is a good place to take a break, grab a cup of coffee, use the toilet, or go for a short stroll, before we show that U i is a subspace of the generalized eigenspace G T λ i . This will follow if we can prove that each of the basis vectors for U i is a generalized eigenvector of T for λ i ( Definition GEV). We need some power of T − λ i I V that takes u i j to the zero vector. We prove by induction on j ( Technique I) the claim that T − λ i I V j u i j = 0 . For j = 1 we have,

T − λ i I V u i 1 = T u i 1 − λ i I V u i 1 = T u i 1 − λ i u i 1 = λ i u i 1 − λ i u i 1 = 0

For the induction step, assume that if k < j , then T − λ i I V k takes u i k to the zero vector. Then

T − λ i I V j u i j = T − λ i I V j − 1 T − λ i I V u i j = T − λ i I V j − 1 T u i j − λ i I V u i j = T − λ i I V j − 1 T u i j − λ i u i j = T − λ i I V j − 1 ∑ k = 1 j − 1 b i j k u i k + λ i u i j − λ i u i j = T − λ i I V j − 1 ∑ k = 1 j − 1 b i j k u i k = ∑ k = 1 j − 1 b i j k T − λ i I V j − 1 u i k = ∑ k = 1 j − 1 b i j k T − λ i I V j − 1 − k T − λ i I V k u i k = ∑ k = 1 j − 1 b i j k T − λ i I V j − 1 − k 0 = ∑ k = 1 j − 1 b i j k 0 = 0

This completes the induction step. Since every vector of the spanning set for U i is an element of the subspace G T λ i , Property AC and Property SC allow us to conclude that U i ⊆ G T λ i . Then by Definition S, U i is a subspace of G T λ i . Notice that this inductive proof could be interpreted to say that every element of U i is a generalized eigenvector of T for λ i , and the algebraic multiplicity of λ i is a sufficiently high power to demonstrate this via the definition for each vector.

We are now prepared for our final argument in this long proof. We wish to establish that the dimension of the subspace G T λ i is the algebraic multiplicity of λ i . This will be enough to show that U i and G T λ i are equal, and will finally provide the desired direct sum decomposition.

We will prove by induction ( Technique I) the following claim. Suppose that T : V → V is a linear transformation and B is a basis for V that provides an upper triangular matrix representation of T . The number of times any eigenvalue λ occurs on the diagonal of the representation is greater than or equal to the dimension of the generalized eigenspace G T λ .

We will use the symbol m for the dimension of V so as to avoid confusion with our notation for the nullity. So dim V = m and our proof will proceed by induction on m . Use the notation # T ( λ ) to count the number of times λ occurs on the diagonal of a matrix representation of T . We want to show that

# T ( λ ) ≥ dim G T λ = dim K T − λ m   Theorem GEK = n T − λ m   Definition NOLT

For the base case, dim V = 1 . Every matrix representation of T is an upper triangular matrix with the lone eigenvalue of T , λ , as the single diagonal entry. So # T ( λ ) = 1 . The generalized eigenspace of λ is not trivial (since by Theorem GEK it equals the regular eigenspace), so it cannot be a subspace of dimension zero, and thus dim G T λ = 1 .

Now for the induction step, assume the claim is true for any linear transformation defined on a vector space with dimension m − 1 or less. Suppose that B = v 1 , v 2 , v 3 , … , v m is a basis for V that yields an upper triangular matrix representation for T with diagonal entries λ 1 , λ 2 , λ 3 , … , λ m . Then U = v 1 , v 2 , v 3 , … , v m − 1 is a subspace of V that is invariant relative to T . The restriction T | U : U → U is then a linear transformation defined on U , a vector space of dimension m − 1 . A matrix representation of T | U relative to the basis C = v 1 , v 2 , v 3 , … , v m − 1 will be an upper triangular matrix with diagonal entries λ 1 , λ 2 , λ 3 , … , λ m − 1 . We can therefore apply the induction hypothesis to T | U and its representation relative to C .

Suppose that λ is any eigenvalue of T . Then suppose that v ∈ K T − λ I V m . As an element of V , we can write v as a linear combination of the basis elements of B , or more compactly, there is a vector u ∈ U and a scalar α such that v = u + α v m . Then,

α λ m − λ m v m = α T − λ I V m v m   Theorem EOMP = 0 + α T − λ I V m v m   Property Z = − T − λ I V m u + T − λ I V m u + α T − λ I V m v m   Property AI = − T − λ I V m u + T − λ I V m u + α v m   Theorem LTLC = − T − λ I V m u + T − λ I V m v   Theorem LTLC = − T − λ I V m u + 0   Definition KLT = − T − λ I V m u   Property Z

The final expression in this string of equalities is an element of U since U is invariant relative to both T and I V . The expression at the beginning is a scalar multiple of v m , and as such cannot be a nonzero element of U without violating the linear independence of B . So

α λ m − λ m v m = 0

The vector v m is nonzero since B is linearly independent, so Theorem SMEZV tells us that α λ m − λ m = 0 . From the properties of scalar multiplication, we are confronted with two possibilities.

Our first case is that λ ≠ λ m . Notice then that λ occurs the same number of times along the diagonal in the representations of T | U and T . Now α = 0 and v = u + 0 v m = u . Since v was chosen as an arbitrary element of K T − λ I V m , Definition SSET says that K T − λ I V m ⊆ U . It is always the case that K T | U − λ I U m ⊆ K T − λ I V m . However, we can also see that in this case, the opposite set inclusion is true as well. By Definition SE we have K T | U − λ I U m = K T − λ I V m . Then

# T ( λ ) = # T | U ( λ ) ≥ dim G T | U λ  Induction Hypothesis = dim K T | U − λ I U m − 1   Theorem GEK = dim K T | U − λ I U m   Theorem KPLT = dim K T − λ I V m = dim G T λ   Theorem GEK

The second case is that λ = λ m . Notice then that λ occurs one more time along the diagonal in the representation of T compared to the representation of T | U . Then

T | U − λ I U m u = T − λ I V m u = T − λ I V m u + 0   Property Z = T − λ I V m u + α ( λ m − λ ) m v m   Theorem ZSSM = T − λ I V m u + α T − λ I V m v m   Theorem EOMP = T − λ I V m u + α v m   Theorem LTLC = T − λ I V m v = 0   Definition KLT

So u ∈ K T | U − λ I U m . The vector v was chosen as an arbitrary member of K T − λ I V m . From the expression v = u + α v m we can now see v also as an element of K T | U − λ I U m plus a scalar multiple of v m . This observation yields

dim K T − λ I V m ≤ dim K T | U − λ I U m + 1

Now count eigenvalues on the diagonal,

# T ( λ ) = # T | U ( λ ) + 1 ≥ dim G T | U λ + 1  Induction Hypothesis = dim K T | U − λ I U m − 1 + 1   Theorem GEK = dim K T | U − λ I U m + 1   Theorem KPLT ≥ dim K T − λ I V m = dim G T λ   Theorem GEK

In Theorem UTMR we constructed an upper triangular matrix representation of T where each eigenvalue occurred α T λ times on the diagonal. So

α T λ i = # T ( λ i )   Theorem UTMR ≥ dim G T λ i ≥ dim U i   Theorem PSSD = α T λ i   Theorem PSSD

Thus, dim G T λ i = α T λ i and by Theorem EDYES, U i = G T λ i and we can write

V = U 1 ⊕ U 2 ⊕ U 3 ⊕ ⋯ ⊕ U m = G T λ 1 ⊕ G T λ 2 ⊕ G T λ 3 ⊕ ⋯ ⊕ G T λ m

■

Besides a nice decomposition into invariant subspaces, this proof has a bonus for us.

Theorem  DGES
Dimension of Generalized Eigenspaces
Suppose T : V → V is a linear transformation with eigenvalue λ . Then the dimension of the generalized eigenspace for λ is the algebraic multiplicity of λ , dim G T λ i = α T λ i . □

Proof   At the very end of the proof of Theorem GESD we obtain the inequalities

α T λ i ≤ dim G T λ i ≤ α T λ i

which establishes the desired equality. ■

Subsection JCF: Jordan Canonical Form

Now we are in a position to define what we (and others) regard as an especially nice matrix representation. The word “canonical” has at its root, the word “canon,” which has various meanings. One is the set of laws established by a church council. Another is a set of writings that are authentic, important or representative. Here we take it to mean the accepted, or best, representative among a variety of choices. Every linear transformation admits a variety of representations, and we will declare one as the best. Hopefully you will agree.

Definition  JCF
Jordan Canonical Form
A square matrix is in Jordan canonical form if it meets the following requirements:

△

Theorem  JCFLT
Jordan Canonical Form for a Linear Transformation
Suppose T : V → V is a linear transformation. Then there is a basis B for V such that the matrix representation of T with the following properties:

□

Proof   This theorem is really just the consequence of applying to T , consecutively Theorem GESD, Theorem MRRGE and Theorem CFNLT.

Theorem GESD gives us a decomposition of V into generalized eigenspaces, one for each distinct eigenvalue. Since these generalized eigenspaces ar invariant relative to T , this provides a block diagonal matrix representation where each block is the matrix representation of the restriction of T to the generalized eigenspace.

Restricting T to a generalized eigenspace results in a “nearly nilpotent” linear transformation, as stated more precisely in Theorem RGEN. We unravel Theorem RGEN in the proof of Theorem MRRGE so that we can apply Theorem CFNLT about representations of nilpotent linear transformations.

We know the dimension of a generalized eigenspace is the algebraic multiplicity of the eigenvalue ( Theorem DGES), so the blocks associated with the generalized eigenspaces are square with a size equal to the algebraic multiplicity. In refining the basis for this block, and producing Jordan blocks the results of Theorem CFNLT apply. The total number of blocks will be the nullity of T | G T λ − λ I G T λ , which is the geometric multiplicity of λ as an eigenvalue of T (Definition GME). The largest of the Jordan blocks will have size equal to the index of the nilpotent linear transformation T | G T λ − λ I G T λ , which is exactly the definition of the index of the eigenvalue λ ( Definition IE). ■

Before we do some examples of this result, notice how close Jordan canonical form is to a diagonal matrix. Or, equivalently, notice how close we have come to diagonalizing a matrix (Definition DZM). We have a matrix representation which has diagonal entries that are the eigenvalues of a matrix. Each occurs on the diagonal as many times as the algebraic multiplicity. However, when the geometric multiplicity is strictly less than the algebraic multiplicity, we have some entries in the representation just above the diagonal (the “superdiagonal”). Furthermore, we have some idea how often this happens if we know the geometric multiplicity and the index of the eigenvalue.

We now recognize just how simple a diagonalizable linear transformation really is. For each eigenvalue, the generalized eigenspace is just the regular eigenspace, and it decomposes into a direct sum of one-dimensional subspaces, each spanned by a different eigenvector chosen from a basis of eigenvectors for the eigenspace.

Some authors create matrix representations of nilpotent linear transformations where the Jordan block has the ones just below the diagonal (the “subdiagonal”). No matter, it is really the same, just different. We have also defined Jordan canonical form to place blocks for the larger eigenvalues earlier, and for blocks with the same eigenvalue, we place the bigger ones earlier. This is fairly standard, but there is no reason we couldn’t order the blocks differently. It’d be the same, just different. The reason for choosing some ordering is to be assured that there is just one canonical matrix representation for each linear transformation.

Example  JCF10
Jordan canonical form, size 10
Suppose that T : ℂ 1 0 → ℂ 1 0 is the linear transformation defined by T x = A x where

A = − 6 9 − 7 − 5 5 1 2 − 2 2 1 4 8 2 1 − 3 5 − 3 − 1 2 7 − 1 2 9 1 1 2 8 − 9 8 6 0 − 1 4 2 5 − 1 3 − 4 − 2 6 − 7 9 − 7 − 5 0 1 3 − 2 3 1 3 2 2 4 0 − 1 0 − 1 − 3 − 2 3 − 4 − 2 − 3 3 2 1 2 9 − 1 1 5 5 − 5 − 1 3 − 3 − 2 4 3 − 6 4 4 3 3 − 4 3 2 1 − 5 9 − 5 1 − 9 0 2 0 0 2 2 − 4 4 2 4 − 4 4 − 5 − 4 − 1 6 − 1 1 4 1 1 0

We’ll find a basis for ℂ 1 0 that will yield a matrix representation of T in Jordan canonical form. First we find the eigenvalues, and their multiplicities, with the techniques of Chapter E.

λ = 2 α T 2 = 2 γ T 2 = 2 λ = 0 α T 0 = 3 γ T − 1 = 2 λ = − 1 α T − 1 = 5 γ T − 1 = 2

For each eigenvalue, we can compute a generalized eigenspace. By Theorem GESD we know that ℂ 1 0 will decompose into a direct sum of these eigenspaces, and we can restrict T to each part of this decomposition. At this stage we know that the Jordan canonical form will be block diagonal with blocks of size 2 , 3 and 5 , since the dimensions of the generalized eigenspaces are equal to the algebraic multiplicities of the eigenvalues ( Theorem DGES). The geometric multiplicities tell us how many Jordan blocks occupy each of the three larger blocks, but we will discuss this as we analyze each eigenvalue. We do not yet know the index of each eigenvalue (though we can easily infer it for λ = 2 ) and even if we did have this information, it only determines the size of the largest Jordan block (per eigenvalue). We will press ahead, considering each eigenvalue one at a time.

The eigenvalue λ = 2 has “full” geometric multiplicity, and is not an impediment to diagonalizing T . We will treat it in full generality anyway. First we compute the generalized eigenspace. Since Theorem GEK says that G T 2 = K T − 2 I ℂ 1 0 1 0 we can compute this generalized eigenspace as a null space derived from the matrix A ,

A − 2 I 1 0 1 0 →  RREF 1 0 0 0 0 0 0 0 − 2 − 1 0 1  0 0 0 0 0 0 − 1 − 1 0 0 1 0 0 0 0 0 1 2 0 0 0 1  0 0 0 0 − 1 − 2 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1  0 0 − 2 1 0 0 0 0 0 0 1 0 − 1 0 0 0 0 0 0 0 0 1  0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G T 2 = K A − 2 I 1 0 1 0 = 2 1 − 1 1 − 1 2 1 0 1 0 , 1 1 − 2 2 0 − 1 0 − 1 0 1

The restriction of T to G T 2 relative to the two basis vectors above has a matrix representation that is a 2 × 2 diagonal matrix with the eigenvalue λ = 2 as the diagonal entries. So these two vectors will be the first two vectors in our basis for ℂ 1 0 ,

v 1 = 2 1 − 1 1 − 1 2 1 0 1 0 v 2 = 1 1 − 2 2 0 − 1 0 − 1 0 1

Notice that it was not strictly necessary to compute the 10-th power of A − 2 I 1 0 . With α T 2 = γ T 2 the null space of the matrix A − 2 I 1 0 contains all of the generalized eigenvectors of T for the eigenvalue λ = 2 . But there was no harm in computing the 10-th power either. This discussion is equivalent to the observation that the linear transformation T | G T 2 : G T 2 → G T 2 is nilpotent of index 1 . In other words, ι T 2 = 1 .

The eigenvalue λ = 0 will not be quite as simple, since the geometric multiplicity is strictly less than the geometric multiplicity. As before, we first compute the generalized eigenspace. Since Theorem GEK says that G T 0 = K T − 0 I ℂ 1 0 1 0 we can compute this generalized eigenspace as a null space derived from the matrix A ,

A − 0 I 1 0 1 0 →  RREF 1 0 0 0 0 0 0 0 − 1 − 1 0 1  0 0 0 0 − 1 0 − 1 0 0 0 1 0 0 0 0 0 1 2 0 0 0 1  0 0 0 0 − 2 − 1 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1  − 1 0 − 1 2 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G T 0 = K A − 0 I 1 0 1 0 = 0 1 0 0 0 1 1 0 0 0 , 1 1 − 1 2 − 1 1 0 − 1 1 0 , 1 0 − 2 1 0 − 2 0 0 0 1 = F

So dim G T 0 = 3 = α T 0 , as expected. We will use these three basis vectors for the generalized eigenspace to construct a matrix representation of T | G T 0 , where F is being defined implicitly as the basis of G T 0 . We construct this representation as usual, applying Definition MR,

ρ F T | G T 0 0 1 0 0 0 1 1 0 0 0 = ρ F − 1 0 2 − 1 0 2 0 0 0 − 1 = ρ F ( − 1 ) 1 0 − 2 1 0 − 2 0 0 0 1 = 0 0 − 1 ρ F T | G T 0 1 1 − 1 2 − 1 1 0 − 1 1 0 = ρ F 1 0 − 2 1 0 − 2 0 0 0 1 = ρ F ( 1 ) 1 0 − 2 1 0 − 2 0 0 0 1 = 0 0 1 ρ F T | G T 0 1 0 − 2 1 0 − 2 0 0 0 1 = ρ F 0 0 0 0 0 0 0 0 0 0 = 0 0 0

So we have the matrix representation

M = M F , F T | G T 0 = 0 0 0 0 0 0 − 1 1 0

By Theorem RGEN we can obtain a nilpotent matrix from this matrix representation by subtracting the eigenvalue from the diagonal elements, and then we can apply Theorem CFNLT to M − ( 0 ) I 3 . First check that M − ( 0 ) I 3 2 = O , so we know that the index of M − ( 0 ) I 3 as a nilpotent matrix, and that therefore λ = 0 is an eigenvalue of T with index 2 , ι T 0 = 2 . To determine a basis of ℂ 3 that converts M − ( 0 ) I 3 to canonical form, we need the null spaces of the powers of M − ( 0 ) I 3 . For convenience, set N = M − ( 0 ) I 3 .

N N 1 = 1 1 0 , 0 0 1 N N 2 = 1 0 0 , 0 1 0 , 0 0 1 = ℂ 3

Then we choose a vector from N N 2 that is not an element of N N 1 . Any vector with unequal first two entries will fit the bill, say

z 2 , 1 = 1 0 0

where we are employing the notation in Theorem CFNLT. The next step is to multiply this vector by N to get part of the basis for N N 1 ,

z 1 , 1 = N z 2 , 1 = 0 0 0 0 0 0 − 1 1 0 1 0 0 = 0 0 − 1

We need a vector to pair with z 1 , 1 that will make a basis for the two-dimensional subspace N N 1 . Examining the basis for N N 1 we see that a vector with its first two entries equal will do the job.

z 1 , 2 = 1 1 0

Reordering, we find the basis,

C = z 1 , 1 , z 2 , 1 , z 1 , 2 = 0 0 − 1 , 1 0 0 , 1 1 0

From this basis, we can get a matrix representation of N (when viewed as a linear transformation) relative to the basis C for ℂ 3 ,

0 1 0 0 0 0 0 0 0 = J 2 0 O O J 1 0

Now we add back the eigenvalue λ = 0 to the representation of N to obtain a representation for M . Of course, with an eigenvalue of zero, the change is not apparent, so we won’t display the same matrix again. This is the second block of the Jordan canonical form for T . However, the three vectors in C will not suffice as basis vectors for the domain of T — they have the wrong size! The vectors in C are vectors in the domain of a linear transformation defined by the matrix M . But M was a matrix representation of T | G T 0 − 0 I G T 0 relative to the basis F for G T 0 . We need to “uncoordinatize” each of the basis vectors in C to produce a linear combination of vectors in F that will be an element of the generalized eigenspace G T 0 . These will be the next three vectors of our final answer, a basis for ℂ 1 0 that has a pleasing matrix representation.

v 3 = ρ F − 1 0 0 − 1 = 0 0 1 0 0 0 1 1 0 0 0 + 0 1 1 − 1 2 − 1 1 0 − 1 1 0 + ( − 1 ) 1 0 − 2 1 0 − 2 0 0 0 1 = − 1 0 2 − 1 0 2 0 0 0 − 1 v 4 = ρ F − 1 1 0 0 = 1 0 1 0 0 0 1 1 0 0 0 + 0 1 1 − 1 2 − 1 1 0 − 1 1 0 + 0 1 0 − 2 1 0 − 2 0 0 0 1 = 0 1 0 0 0 1 1 0 0 0 v 5 = ρ F − 1 1 1 0 = 1 0 1 0 0 0 1 1 0 0 0 + 1 1 1 − 1 2 − 1 1 0 − 1 1 0 + 0 1 0 − 2 1 0 − 2 0 0 0 1 = 1 2 − 1 2 − 1 2 1 − 1 1 0

Five down, five to go. Basis vectors, that is. λ = − 1 is the smallest eigenvalue, but it will require the most computation. First we compute the generalized eigenspace. Since Theorem GEK says that G T − 1 = K T − ( − 1 ) I ℂ 1 0 1 0 we can compute this generalized eigenspace as a null space derived from the matrix A ,

A − ( − 1 ) I 1 0 1 0 →  RREF 1 0 1 0 1 0 − 1 1 0 1 0 1  0 0 1 0 0 1 0 0 0 0 0 1 1 0 1 0 0 − 2 0 0 0 0 0 1  − 2 1 0 2 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G T − 1 = K A − ( − 1 ) I 1 0 1 0 = − 1 0 1 0 0 0 0 0 0 0 , − 1 − 1 0 − 1 1 0 0 0 0 0 , 1 0 0 − 1 0 2 1 0 0 0 , − 1 − 1 0 0 0 − 1 0 1 0 0 , − 1 0 0 2 0 − 2 0 0 0 1 = F

So dim G T − 1 = 5 = α T − 1 , as expected. We will use these five basis vectors for the generalized eigenspace to construct a matrix representation of T | G T − 1 , where F is being recycled and defined now implicitly as the basis of G T − 1 . We construct this representation as usual, applying Definition MR,

ρ F T | G T − 1 − 1 0 1 0 0 0 0 0 0 0 = ρ F − 1 0 0 0 0 − 2 − 2 0 0 − 1 = ρ F 0 − 1 0 1 0 0 0 0 0 0 0 + 0 − 1 − 1 0 − 1 1 0 0 0 0 0 + ( − 2 ) 1 0 0 − 1 0 2 1 0 0 0 + 0 − 1 − 1 0 0 0 − 1 0 1 0 0 + ( − 1 ) − 1 0 0 2 0 − 2 0 0 0 1 = 0 0 − 2 0 − 1 ρ F T | G T − 1 − 1 − 1 0 − 1 1 0 0 0 0 0 = ρ F 7 1 − 5 3 − 1 2 4 0 0 3 = ρ F ( − 5 ) − 1 0 1 0 0 0 0 0 0 0 + ( − 1 ) − 1 − 1 0 − 1 1 0 0 0 0 0 + 4 1 0 0 − 1 0 2 1 0 0 0 + 0 − 1 − 1 0 0 0 − 1 0 1 0 0 + 3 − 1 0 0 2 0 − 2 0 0 0 1 = − 5 − 1 4 0 3 ρ F T | G T − 1 1 0 0 − 1 0 2 1 0 0 0 = ρ F 1 0 − 1 1 0 0 1 0 0 1 = ρ F ( − 1 ) − 1 0 1 0 0 0 0 0 0 0 + 0 − 1 − 1 0 − 1 1 0 0 0 0 0 + 1 1 0 0 − 1 0 2 1 0 0 0 + 0 − 1 − 1 0 0 0 − 1 0 1 0 0 + 1 − 1 0 0 2 0 − 2 0 0 0 1 = − 1 0 1 0 1 ρ F T | G T − 1 − 1 − 1 0 0 0 − 1 0 1 0 0 = ρ F − 1 0 2 − 2 − 1 1 − 1 1 0 − 2 = ρ F 2 − 1 0 1 0 0 0 0 0 0 0 + ( − 1 ) − 1 − 1 0 − 1 1 0 0 0 0 0 + ( − 1 ) 1 0 0 − 1 0 2 1 0 0 0 + 1 − 1 − 1 0 0 0 − 1 0 1 0 0 + ( − 2 ) − 1 0 0 2 0 − 2 0 0 0 1 = 2 − 1 − 1 1 − 2 ρ F T | G T − 1 − 1 0 0 2 0 − 2 0 0 0 1 = ρ F − 7 − 1 6 − 5 − 1 − 2 − 6 2 0 − 6 = ρ F 6 − 1 0 1 0 0 0 0 0 0 0 + ( − 1 ) − 1 − 1 0 − 1 1 0 0 0 0 0 + ( − 6 ) 1 0 0 − 1 0 2 1 0 0 0 + 2 − 1 − 1 0 0 0 − 1 0 1 0 0 + ( − 6 ) − 1 0 0 2 0 − 2 0 0 0 1 = 6 − 1 − 6 2 − 6

So we have the matrix representation of the restriction of T (again recycling and redefining the matrix M )

M = M F , F T | G T − 1 = 0 − 5 − 1 2 6 0 − 1 0 − 1 − 1 − 2 4 1 − 1 − 6 0 0 0 1 2 − 1 3 1 − 2 − 6

By Theorem RGEN we can obtain a nilpotent matrix from this matrix representation by subtracting the eigenvalue from the diagonal elements, and then we can apply Theorem CFNLT to M − ( − 1 ) I 5 . First check that M − ( − 1 ) I 5 3 = O , so we know that the index of M − ( − 1 ) I 5 as a nilpotent matrix, and that therefore λ = − 1 is an eigenvalue of T with index 3 , ι T − 1 = 3 . To determine a basis of ℂ 5 that converts M − ( − 1 ) I 5 to canonical form, we need the null spaces of the powers of M − ( − 1 ) I 5 . Again, for convenience, set N = M − ( − 1 ) I 5 .

N N 1 = 1 0 1 0 0 , − 3 1 0 − 2 2 N N 2 = 3 1 0 0 0 , 1 0 1 0 0 , 0 0 0 1 0 , − 3 0 0 0 1 N N 3 = 1 0 0 0 0 , 0 1 0 0 0 , 0 0 1 0 0 , 0 0 0 1 0 , 0 0 0 0 1 = ℂ 5

Then we choose a vector from N N 3 that is not an element of N N 2 . The sum of the four basis vectors for N N 2 sum to a vector with all five entries equal to 1. We will mess with the first entry to create a vector not in N N 2 ,

z 3 , 1 = 0 1 1 1 1

where we are employing the notation in Theorem CFNLT. The next step is to multiply this vector by N to get a portion of the basis for N N 2 ,

z 2 , 1 = N z 3 , 1 = 1 − 5 − 1 2 6 0 0 0 − 1 − 1 − 2 4 2 − 1 − 6 0 0 0 2 2 − 1 3 1 − 2 − 5 0 1 1 1 1 = 2 − 2 − 1 4 − 3

We have a basis for the two-dimensional subspace N N 1 and we can add to that the vector z 2 , 1 and we have three of four basis vectors for N N 2 . These three vectors span the subspace we call Q 2 . We need a fourth vector outside of Q 2 to complete a basis of the four-dimensional subspace N N 2 . Check that the vector

z 2 , 2 = 3 1 3 1 1

is an element of N N 2 that lies outside of the subspace Q 2 . This vector was constructed by getting a nice basis for Q 2 and forming a linear combination of this basis that specifies three of the five entries of the result. Of the remaining two entries, one was changed to move the vector outside of Q 2 and this was followed by a change to the remaining entry to place the vector into N N 2 . The vector z 2 , 2 is the lone basis vector for the subspace we call R 2 .

The remaining two basis vectors are easy to come by. They are the result of applying N to each of the two most recently determined basis vectors,

z 1 , 1 = N z 2 , 1 = 3 − 1 0 2 − 2 z 1 , 2 = N z 2 , 2 = 3 − 2 − 3 4 − 4

Now we reorder these basis vectors, to arrive at the basis

C = z 1 , 1 , z 2 , 1 , z 3 , 1 , z 1 , 2 , z 2 , 2 = 3 − 1 0 2 − 2 , 2 − 2 − 1 4 − 3 , 0 1 1 1 1 , 3 − 2 − 3 4 − 4 , 3 1 3 1 1

A matrix representation of N relative to C is

0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 = J 3 0 O O J 2 0

To obtain a matrix representation of M , we add back in the matrix ( − 1 ) I 5 , placing the eigenvalue back along the diagonal, and slightly modifying the Jordan blocks,

− 1 1 0 0 0 0 − 1 1 0 0 0 0 − 1 0 0 0 0 0 − 1 1 0 0 0 0 − 1 = J 3 − 1 O O J 2 − 1

The basis C yields a pleasant matrix representation for the restriction of the linear transformation T − ( − 1 ) I to the generalized eigenspace G T − 1 . However, we must remember that these vectors in ℂ 5 are representations of vectors in ℂ 1 0 relative to the basis F . Each needs to be “un-coordinatized” before joining our final basis. Here we go,

v 6 = ρ F − 1 3 − 1 0 2 − 2 = 3 − 1 0 1 0 0 0 0 0 0 0 + ( − 1 ) − 1 − 1 0 − 1 1 0 0 0 0 0 + 0 1 0 0 − 1 0 2 1 0 0 0 + 2 − 1 − 1 0 0 0 − 1 0 1 0 0 + ( − 2 ) − 1 0 0 2 0 − 2 0 0 0 1 = − 2 − 1 3 − 3 − 1 2 0 2 0 − 2 v 7 = ρ F − 1 2 − 2 − 1 4 − 3 = 2 − 1 0 1 0 0 0 0 0 0 0 + ( − 2 ) − 1 − 1 0 − 1 1 0 0 0 0 0 + ( − 1 ) 1 0 0 − 1 0 2 1 0 0 0 + 4 − 1 − 1 0 0 0 − 1 0 1 0 0 + ( − 3 ) − 1 0 0 2 0 − 2 0 0 0 1 = − 2 − 2 2 − 3 − 2 0 − 1 4 0 − 3 v 8 = ρ F − 1 0 1 1 1 1 = 0 − 1 0 1 0 0 0 0 0 0 0 + 1 − 1 − 1 0 − 1 1 0 0 0 0 0 + 1 1 0 0 − 1 0 2 1 0 0 0 + 1 − 1 − 1 0 0 0 − 1 0 1 0 0 + 1 − 1 0 0 2 0 − 2 0 0 0 1 = − 2 − 2 0 0 1 − 1 1 1 0 1 v 9 = ρ F − 1 3 − 2 − 3 4 − 4 = 3 − 1 0 1 0 0 0 0 0 0 0 + ( − 2 ) − 1 − 1 0 − 1 1 0 0 0 0 0 + ( − 3 ) 1 0 0 − 1 0 2 1 0 0 0 + 4 − 1 − 1 0 0 0 − 1 0 1 0 0 + ( − 4 ) − 1 0 0 2 0 − 2 0 0 0 1 = − 4 − 2 3 − 3 − 2 − 2 − 3 4 0 − 4 v 1 0 = ρ F − 1 3 1 3 1 1 = 3 − 1 0 1 0 0 0 0 0 0 0 + 1 − 1 − 1 0 − 1 1 0 0 0 0 0 + 3 1 0 0 − 1 0 2 1 0 0 0 + 1 − 1 − 1 0 0 0 − 1 0 1 0 0 + 1 − 1 0 0 2 0 − 2 0 0 0 1 = − 3 − 2 3 − 2 1 3 3 1 0 1

To summarize, we list the entire basis B = v 1 , v 2 , v 3 , … , v 1 0 ,

v 1 = 2 1 − 1 1 − 1 2 1 0 1 0 v 2 = 1 1 − 2 2 0 − 1 0 − 1 0 1 v 3 = − 1 0 2 − 1 0 2 0 0 0 − 1 v 4 = 0 1 0 0 0 1 1 0 0 0 v 5 = 1 2 − 1 2 − 1 2 1 − 1 1 0 v 6 = − 2 − 1 3 − 3 − 1 2 0 2 0 − 2 v 7 = − 2 − 2 2 − 3 − 2 0 − 1 4 0 − 3 v 8 = − 2 − 2 0 0 1 − 1 1 1 0 1 v 9 = − 4 − 2 3 − 3 − 2 − 2 − 3 4 0 − 4 v 1 0 = − 3 − 2 3 − 2 1 3 3 1 0 1

The resulting matrix representation is

M B , B T = 2 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 − 1 1 0 0 0 0 0 0 0 0 0 − 1 1 0 0 0 0 0 0 0 0 0 − 1 0 0 0 0 0 0 0 0 0 0 − 1 1 0 0 0 0 0 0 0 0 0 − 1

If you are not inclined to check all of these computations, here are a few that should convince you of the amazing properties of the basis B . Compute the matrix-vector products A v i , 1 ≤ i ≤ 1 0 . In each case the result will be a vector of the form λ v i + δ v i − 1 , where λ is one of the eigenvalues (you should be able to predict ahead of time which one) and δ ∈ 0 , 1 .

Alternatively, if we can write inputs to the linear transformation T as linear combinations of the vectors in B (which we can do uniquely since B is a basis, Theorem VRRB), then the “action” of T is reduced to a matrix-vector product with the exceedingly simple matrix that is the Jordan canonical form. Wow! ⊠

Subsection CHT: Cayley-Hamilton Theorem

Jordan was a French mathematician who was active in the late 1800’s. Cayley and Hamilton were 19th-century contemporaries of Jordan from Britain. The theorem that bears their names is perhaps one of the most celebrated in basic linear algebra. While our result applies only to vector spaces and linear transformations with scalars from the set of complex numbers, ℂ , the result is equally true if we restrict our scalars to the real numbers, ℝ . It says that every matrix satisfies its own characteristic polynomial.

Theorem  CHT
Cayley-Hamilton Theorem
Suppose A is a square matrix with characteristic polynomial p A x . Then p A A = O . □

Proof   Suppose B and C are similar matrices via the matrix S , B = S − 1 C S , and q ( x ) is any polynomial. Then q B is similar to q C via S , q B = S − 1 q C S . (See Example HPDM for hints on how to convince yourself of this.)

By Theorem JCFLT and Theorem SCB we know A is similar to a matrix, J , in Jordan canonical form. Suppose λ 1 , λ 2 , λ 3 , … , λ m are the distinct eigenvalues of A (and are therefore the eigenvalues and diagonal entries of J ). Then by Theorem EMRCP and Definition AME, we can factor the characteristic polynomial as

p A x = x − λ 1 α A λ 1 x − λ 2 α A λ 2 x − λ 3 α A λ 3 ⋯ x − λ m α A λ m

On substituting the matrix J we have

p A J = J − λ 1 I α A λ 1 J − λ 2 I α A λ 2 J − λ 3 I α A λ 3 ⋯ J − λ m I α A λ m

The matrix J − λ k I will be block diagonal, and the block arising from the generalized eigenspace for λ k will have zeros along the diagonal. Suitably adjusted for matrices (rather than linear transformations), Theorem RGEN tells us this matrix is nilpotent. Since the size of this nilpotent matrix is equal to the algebraic multiplicity of λ k , the power J − λ k I α A λ k will be a zero matrix ( Theorem KPNLT) in the location of this block.

Repeating this argument for each of the m eigenvalues will place a zero block in some term of the product at every location on the diagonal. The entire product will then be zero blocks on the diagonal, and zero off the diagonal. In other words, it will be the zero matrix. Since A and J are similar, p A A = p A J = O . ■

Annotated Acronyms R: Representations

Definition VR
Matrix representations build on vector representations, so this is the definition that gets us started. A representation depends on the choice of a single basis for the vector space. Theorem VRRB is what tells us this idea might be useful.

Theorem VRILT
As an invertible linear transformation, vector representation allows us to translate, back and forth, between abstract vector spaces ( V ) and concrete vector spaces ( ℂ n ). This is key to all our notions of representations in this chapter.

Theorem CFDVS
Every vector space with finite dimension “looks like” a vector space of column vectors. Vector representation is the isomorphism that establishes that these vector spaces are isomorphic.

Definition MR
Building on the definition of a vector representation, we define a representation of a linear transformation, determined by a choice of two bases, one for the domain and one for the codomain. Notice that vectors are represented by columnar lists of scalars, while linear transformations are represented by rectangular tables of scalars. Building a matrix representation is as important a skill as row-reducing a matrix.

Theorem FTMR
Definition MR is not really very interesting until we have this theorem. The second form tells us that we can compute outputs of linear transformations via matrix multiplication, along with some bookkeeping for vector representations. Searching forward through the text on “FTMR” is an interesting exercise. You will find reference to this result buried inside many key proofs at critical points, and it also appears in numerous examples and solutions to exercises.

Theorem MRCLT
Turns out that matrix multiplication is really a very natural operation, it is just the chaining together (composition) of functions (linear transformations). Beautiful. Even if you don’t try to work the problem, study Solution MR.T80 for more insight.

Theorem KNSI
Kernels “are” null spaces. For this reason you’ll see these terms used interchangeably.

Theorem RCSI
Ranges “are” column spaces. For this reason you’ll see these terms used interchangeably.

Theorem IMR
Invertible linear transformations are represented by invertible (nonsingular) matrices.

Theorem NME9
The NMEx series has always been important, but we’ve held off saying so until now. This is the end of the line for this one, so it is a good time to contemplate all that it means.

Theorem SCB
Diagonalization back in Section SD was really a change of basis to achieve a diagonal matrix repesentation. Maybe we should be highlighting the more general Theorem MRCB here, but its overly technical description just isn’t as appealing. However, it will be important in some of the matrix decompostions in Chapter MD.

Theorem EER
This theorem, with the companion definition, Definition EELT, tells us that eigenvalues, and eigenvectors, are fundamentally a characteristic of linear transformations (not matrices). If you study matrix decompositions in Chapter MD you will come to appreciate that almost all of a matrix’s secrets can be unlocked with knowledge of the eigenvalues and eigenvectors.

Theorem OD
Can you imagine anything nicer than an orthonormal diagonalization? A basis of pairwise orthogonal, unit norm, eigenvectors that provide a diagonal representation for a matrix? Here we learn just when this can happen — precisely when a matrix is normal, which is a disarmingly simple property to define.

Theorem CFNLT
Nilpotent linear transformations are the fundamental obstacle to a matrix (or linear transformation) being diagonalizable. This specialized representation theorem is the fundamental expression of just how close we can come to surmounting the obstacle, i.e. how close we can come to a diagonal representation.

Theorem DGES
This theorem is a long time in coming, but perhaps it best explains our interest in generalized eigenspaces. When the dimension of a “regular” eigenspace (the geometic multiplicity) does not meet the algebraic multiplicity of the corresponding eigenvalue, then a matrix is not diagonalizable (Theorem DMFE). However, if we generalize the idea of an eigenspace ( Definition GES), then we arrive at invariant subspaces that together give a complete decomposition of the domain as a direct sum. And these subspaces have dimensions equal to the corresponding algebraic multiplicities.

Theorem JCFLT
If you can’t diagonalize, just how close can you come? This is an answer (there are others, like rational canonical form). “Canonicalism” is in the eye of the beholder. But this is a good place to conclude our study of a widely accepted canonical form that is possible for every matrix or linear transformation.

Chapter 9. Appendix CN  Computation Notes

Section MMA  Mathematica

Section TI86  Texas Instruments 86

Section TI83  Texas Instruments 83

Section SAGE  SAGE: Open Source Mathematics Software

Parent topic: Part C  Core

Previous topic: Section JCF  Jordan Canonical Form

Next topic: Section MMA  Mathematica

Section MMA  Mathematica

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Parent topic: Appendix CN  Computation Notes

Previous topic: Appendix CN  Computation Notes

Next topic: Section TI86  Texas Instruments 86

Computation Note ME.MMA: Matrix Entry

Matrices are input as lists of lists, since a list is a basic data structure in Mathematica. A matrix is a list of rows, with each row entered as a list. Mathematica uses braces ((  ,  )) to delimit lists. So the input

a = { { 1 , 2 , 3 , 4 } , { 5 , 6 , 7 , 8 } , { 9 , 1 0 , 1 1 , 1 2 } }

would create a 3 × 4 matrix named a that is equal to

1 2 3 4 5 6 7 8 9 1 0 1 1 1 2

To display a matrix named a “nicely” in Mathematica, type MatrixForm[a] , and the output will be displayed with rows and columns. If you just type a , then you will get a list of lists, like how you input the matrix in the first place.

Computation Note RR.MMA: Row Reduce

If a is the name of a matrix in Mathematica, then the command RowReduce[a] will output the reduced row-echelon form of the matrix.

Computation Note LS.MMA: Linear Solve

Mathematica will solve a linear system of equations using the LinearSolve[] command. The inputs are a matrix with the coefficients of the variables (but not the column of constants), and a list containing the constant terms of each equation. This will look a bit odd, since the lists in the matrix are rows, but the column of constants is also input as a list and so looks like a row rather than a column. The result will be a single solution (even if there are infinitely many), reported as a list, or the statement that there is no solution. When there are infinitely many, the single solution reported is exactly that solution used in the proof of Theorem RCLS, where the free variables are all set to zero, and the dependent variables come along with values from the final column of the row-reduced matrix.

As an example, Archetype A is

x 1 − x 2 + 2 x 3 = 1 2 x 1 + x 2 + x 3 = 8 x 1 + x 2 = 5

To ask Mathematica for a solution, enter

L i n e a r S o l v e [ { { 1 , − 1 , 2 } , { 2 , 1 , 1 } , { 1 , 1 , 0 } } , { 1 , 8 , 5 } ]

and you will get back the single solution

{ 3 , 2 , 0 }

We will see later how to coax Mathematica into giving us infinitely many solutions for this system ( Computation VFSS.MMA).

Computation Note VLC.MMA: Vector Linear Combinations

Contributed by  Robert Beezer
Vectors in Mathematica are represented as lists, written and displayed horizontally. For example, the vector

v = 1 2 3 4

would be entered and named via the command

v = { 1 , 2 , 3 , 4 }

Vector addition and scalar multiplication are then very natural. If u and v are two lists of equal length, then

2 u + ( − 3 ) v

will compute the correct vector and return it as a list. If u and v have different sizes, then Mathematica will complain about “objects of unequal length.”

Computation Note NS.MMA: Null Space

Given a matrix A , Mathematica will compute a set of column vectors whose span is the null space of the matrix with the NullSpace[] command. Perhaps not coincidentally, this set is exactly z j 1 ≤ j ≤ n − r . However, Mathematica prefers to output the vectors in the opposite order than one we have chosen. Here’s a small example.

Begin with the 3 × 4 matrix A , and its row-reduced version B ,

A = 1 2 − 1 0 3 4 1 − 2 − 1 1 − 5 3 →  RREF B = 1 0 3 − 2 0 1  − 2 1 0 0 0 0

We could extract entries from B to build the vectors z 1 and z 2 according to Theorem SSNS and describe N A as a span of the set z 1 , z 2 . Instead, if a has been set to A , then executing the command NullSpace[a] yields the list of lists (column vectors),

{ { 2 , − 1 , 0 , 1 } , { − 3 , 2 , 1 , 0 } }

Notice how our z 1 is second in the list. To “correct” this we can use a list-processing command from Mathematica, Reverse[] , as follows,

  Reverse[NullSpace[a]] 

and receive the output in our preferred order. Give it a try yourself.

Computation Note VFSS.MMA: Vector Form of Solution Set

Suppose that A is an m × n matrix and b ∈ ℂ m is a column vector. We might wish to find all of the solutions to the linear system ℒ S A , b . Mathematica’s LinearSolve[A,  b] will return at most one solution ( Computation LS.MMA). However, when the system is consistent, then this one solution reported is exactly the vector c , described in the statement of Theorem VFSLS.

The vectors u j , 1 ≤ j ≤ n − r of Theorem VFSLS are exactly the output of Mathematica’s NullSpace[] command, though Mathematica lists them in the opposite order from the order we have chosen. These are the same vectors listed as z j , 1 ≤ j ≤ n − r in Theorem SSNS. With c produced from the LinearSolve[] command, and the u j coming from the NullSpace[] command we can use Mathematica’s symbolic manipulation commands to create an expression that describes all of the solutions.

Begin with the system ℒ S A , b . Row-reduce A ( Computation RR.MMA) and identify the free variables by determining the non-pivot columns. Suppose, for the sake of argument, that we have the three free variables x 3 , x 7 and x 8 . Then the following command will build an expression for an arbitrary solution:

  LinearSolve[A,b]+{x8,x7,x3}.NullSpace[A] 

Be sure to include the “dot” right before the NullSpace[] command — it has the effect of creating a linear combination of the vectors in the null space, using scalars that are symbols reminiscent of the variables.

A concrete example should help here. Suppose we want a solution set for the linear system with coefficient matrix A and vector of constants b ,

A = 1 2 3 − 5 1 − 1 2 2 4 0 8 − 4 1 − 8 3 6 4 0 − 2 5 7 b = 8 1 − 5

If we were to apply Theorem VFSLS, we would extract the components of c and u j from the row-reduced version of the augmented matrix of the system (obtained with Mathematica, Computation RR.MMA),

1 2 0 4 − 2 0 − 5 2 0 0 1  − 3 1 0 3 1 0 0 0 0 0 1 2 − 3

Instead, we will use this augmented matrix in reduced row-echelon form only to identify the free variables. In this example, we locate the non-pivot columns and see that x 2 , x 4 , x 5 and x 7 are free. If we have set a to the coefficient matrix and b to the vector of constants, then we execute the Mathematica command,

  LinearSolve[a, b]+{x7, x5, x4, x2}.NullSpace[a] 

As output we obtain the column vector (list),

2 − 2   x2  − 4   x4  + 2   x5  + 5   x7   x2  1 + 3   x4  −  x5  − 3   x7   x4   x5  − 3 − 2  x7   x7 

Computation Note GSP.MMA: Gram-Schmidt Procedure

Mathematica has a built-in routine that will do the Gram-Schmidt procedure ( Theorem GSP). The input is a set of vectors, which must be linearly independent. This is written as a list, containing lists that are the vectors. Let a be such a list of lists, containing the vectors v i , 1 ≤ i ≤ p from the statement of the theorem. You will need to first load the right Mathematica package — execute <<LinearAlgebra‘Orthogonalization‘ to make this happen. Then execute GramSchmidt[a] . The output will be another list of lists containing the vectors u i , 1 ≤ i ≤ p from the statement of the theorem. Mathematica will complain if you do not provide a linearly independent set as input (try it!).

An example. Suppose our linearly independent set (check this!) is

S = − 1 4 1 0 3 , 0 3 0 3 − 3 , − 1 2 0 − 1 − 2 , − 1 − 2 − 3 1 4 , 1 6 − 1 4 6

The output of the GramSchmidt[] command will be the set,

T = − 1 3 3 4 3 3 1 3 3 0 1 3 , 1 1 2 1 5 2 3 1 2 1 5 − 1 1 2 1 5 3 3 5 4 − 5 3 2 , − 3 7 4 6 8 5 2 9 4 6 8 5 − 3 4 6 8 5 − 7 9 4 6 8 5 − 5 5 1 3 7 2 , − 3 3 7 2 1 2 0 4 2 3 − 3 7 6 1 2 0 4 2 3 − 1 7 6 3 6 1 2 0 4 2 3 3 3 7 6 1 2 0 4 2 3 5 0 1 2 0 4 2 3 , 2 3 8 7 9 2 6 3 8 7 9 − 4 4 3 8 7 9 − 2 3 3 8 7 9 1 8 7 9

Ugly, but true. At this stage, you might just as well be encouraged to think of the Gram-Schmidt procedure as a computational black box, linearly independent set in, orthogonal span-preserving set out.

To check that the output set is orthogonal, we can easily check the orthogonality of individual pairs of vectors. Suppose the output was set equal to b (say via b=GramSchmidt[a] ). We can extract the individual vectors of c as “parts” with syntax like c[[3]] , which would return the third vector in the set. When our vectors have only real number entries, we can accomplish an innerproduct with a “dot.” So, for example, you should discover that c[[3]].c[[5]] will return zero. Try it yourself with another pair of vectors.

Computation Note TM.MMA: Transpose of a Matrix

Contributed by  Robert Beezer
Suppose a is the name of a matrix stored in Mathematica. Then Transpose[a] will create the transpose of a .

Computation Note MM.MMA: Matrix Multiplication

If A and B are matrices defined in Mathematica, then A.B will return the product of the two matrices (notice the dot between the matrices). If A is a matrix and v is a vector, then A.v will return the vector that is the matrix-vector product of A and v . In every case the sizes of the matrices and vectors need to be correct.

Some examples:

{ { 1 , 2 } , { 3 , 4 } } . { { 5 , 6 , 7 } , { 8 , 9 , 1 0 } } = { { 2 1 , 2 4 , 2 7 } , { 4 7 , 5 4 , 6 1 } } { { 1 , 2 } , { 3 , 4 } } . { { 5 } , { 6 } } = { { 1 7 } , { 3 9 } } { { 1 , 2 } , { 3 , 4 } } . { 5 , 6 } = { 1 7 , 3 9 }

Understanding the difference between the last two examples will go a long way to explaining how some Mathematica constructs work.

Computation Note MI.MMA: Matrix Inverse

If A is a matrix defined in Mathematica, then Inverse[A] will return the inverse of A , should it exist. In the case where A does not have an inverse Mathematica will tell you the matrix is singular (see Theorem NI).

Section TI86  Texas Instruments 86

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Parent topic: Appendix CN  Computation Notes

Previous topic: Section MMA  Mathematica

Next topic: Section TI83  Texas Instruments 83

Computation Note ME.TI86: Matrix Entry

On the TI-86, press the MATRX key (Yellow-7) . Press the second menu key over, F2 , to bring up the EDIT screen. Give your matrix a name, one letter or many, then press ENTER . You can then change the size of the matrix (rows, then columns) and begin editing individual entries (which are initially zero). ENTER will move you from entry to entry, or the down arrow key will move you to the next row. A menu gives you extra options for editing.

Matrices may also be entered on the home screen as follows. Use brackets ([ , ]) to enclose rows with elements separated by commas. Group rows, in order, into a final set of brackets (with no commas between rows). This can then be stored in a name with the STO key. So, for example,

[ [ 1 , 2 , 3 , 4 ] [ 5 , 6 , 7 , 8 ] [ 9 , 1 0 , 1 1 , 1 2 ] ] → A

will create a matrix named A that is equal to

1 2 3 4 5 6 7 8 9 1 0 1 1 1 2

Computation Note RR.TI86: Row Reduce

If A is the name of a matrix stored in the TI-86, then the command rref A will return the reduced row-echelon form of the matrix. This command can also be found by pressing the MATRX key, then F4 for OPS , and finally, F5 for rref .

Note that this command will not work for a matrix with more rows than columns. (Ed. Not sure just why this is!) A work-around is to pad the matrix with extra columns of zeros until the matrix is square.

Computation Note VLC.TI86: Vector Linear Combinations

Contributed by  Robert Beezer
Vector operations on the TI-86 can be accessed via the VECTR key, which is Yellow-8 . The EDIT tool appears when the F2 key is pressed. After providing a name and giving a “dimension” (the size) then you can enter the individual entries, one at a time. Vectors can also be entered on the home screen using brackets ( [ , ] ). To create the vector

v = 1 2 3 4

use brackets and the store key ( STO ),

[ 1 , 2 , 3 , 4 ] → v

Vector addition and scalar multiplication are then very natural. If u and v are two vectors of equal size, then

2 ∗ u + ( − 3 ) ∗ v

will compute the correct vector and display the result as a vector.

Computation Note TM.TI86: Transpose of a Matrix

Contributed by  Eric Fickenscher
Suppose A is the name of a matrix stored in the TI-86. Use the command A T to transpose A . This command can be found by pressing the MATRX key, then F3 for MATH , then F2 for T .

Section TI83  Texas Instruments 83

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Parent topic: Appendix CN  Computation Notes

Previous topic: Section TI86  Texas Instruments 86

Next topic: Section SAGE  SAGE: Open Source Mathematics Software

Computation Note ME.TI83: Matrix Entry

Contributed by  Douglas Phelps
On the TI-83, press the MATRX key. Press the right arrow key twice so that EDIT is highlighted. Move the cursor down so that it is over the desired letter of the matrix and press ENTER . For example, let’s call our matrix B , so press the down arrow once and press ENTER . To enter a 2 × 3 matrix, press 2 ENTER 3 ENTER . To create the matrix

1 2 3 4 5 6

press 1 ENTER 2 ENTER 3 ENTER 4 ENTER 5 ENTER 6 ENTER .

Computation Note RR.TI83: Row Reduce

Contributed by  Douglas Phelps
Suppose B is the name of a matrix stored in the TI-83. Press the MATRX key. Press the right arrow key once so that MATH is highlighted. Press the down arrow eleven times so that rref ( is highlighted, then press ENTER . to choose the matrix B , press MATRX , then the down arrow once followed by ENTER . Supply a right parenthesis ( ) ) and press ENTER .

Note that this command will not work for a matrix with more rows than columns. (Ed. Not sure just why this is!) A work-around is to pad the matrix with extra columns of zeros until the matrix is square.

Computation Note VLC.TI83: Vector Linear Combinations

Contributed by  Douglas Phelps
Entering a vector on the TI-83 is the same process as entering a matrix. You press 4 ENTER 3 ENTER for a 4 × 3 matrix. Likewise, you press 4 ENTER 1 ENTER for a vector of size 4. To multiply a vector by 8, press the number 8, then press the MATRX key, then scroll down to the letter you named your vector (A, B, C, etc) and press ENTER .

To add vectors A and B for example, press the MATRX key, then ENTER . Then press the + key. Then press the MATRX key, then the down arrow once, then ENTER . [A] + [B] will appear on the screen. Press ENTER .

Section SAGE  SAGE: Open Source Mathematics Software

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Parent topic: Appendix CN  Computation Notes

Previous topic: Section TI83  Texas Instruments 83

Next topic: Appendix P  Preliminaries

Computation Note R.SAGE: Rings

Contributed by  Steve Canfield
SAGE uses different rings to denote the type of an object. The rings are as follows:

 ZZ: The set of integers  QQ: The set of rational numbers  RR: The real numbers  CC: The complex numbers

Most objects in SAGE will tell you which they are using with the base _ring() command. Keep this in mind, especially when row reducing or factoring. Here’s a quick example of where you might go wrong.

m = m a t r i x ( [ [ 2 , 3 ] , [ 4 , 7 ] ] ) m . b a s e  _ r i n g ( ) I n t e g e r R i n g m . e c h e l o n  _ f o r m ( ) 2 0 0 1

As you can clearly see, m isn’t even in reduced row-echelon form. This is because m is defined over the ZZ. You have to create matrices with the correct ring or you will get this type of odd result. This problem comes up in more places than just calculating the reduced row-echelon form, so unless you are specifically working with integers take note.

Computation Note ME.SAGE: Matrix Entry

Contributed by  Steve Canfield
A matrix in SAGE can be made a few ways. The first is simply to define the matrix as an array of rows. SAGE uses brackets (  ,  ) to delimit arrays. So the input

a = m a t r i x ( [ [ 1 , 2 , 3 , 4 ] , [ 5 , 6 , 7 , 8 ] , [ 9 , 1 0 , 1 1 , 1 2 ] ] )

would create a 3 × 4 matrix named a that is equal to

1 2 3 4 5 6 7 8 9 1 0 1 1 1 2

SAGE will guess what type of matrix you are working with based on the inputs. If all the entries are integers, you will get back an integer matrix. If your matrix contains an entry in the ℝ or ℂ space, the matrix will be of those types. This can cause problems as integers cannot become fractions, which is an issue when calculating reduced row-echelon form. We therefore recommend using the following construction to make your matrices,

a = m a t r i x ( Q Q , [ [ 1 , 2 , 3 , 4 ] , [ 5 , 6 , 7 , 8 ] , [ 9 , 1 0 , 1 1 , 1 2 ] ] )

This gives you a matrix over the rational numbers which will be sufficient for most of the course. If your matrix has entries that are complex numbers you would replace the QQ with CC .
To display a matrix named a , type a , and the output will be displayed with rows and columns. If you type latex(a) you will get L A T EX  code to display the matrix. Very handy.

Computation Note RR.SAGE: Row Reduce

Contributed by  Steve Canfield and Robert Beezer
Row-reducing a matrix is a simple operation in SAGE. However, because of Sage’s flexibility with different types of numbers (integers, rationals, reals, complexes), we need to be a bit more careful.

If a is a matrix entered in in SAGE (see Computation ME.SAGE) then a.echelon _form() will return a new matrix that is the reduced row-echelon form of a ( Definition RREF).

If your matrix has only integer entries (as is the case with many examples and exercises in this book), then row operations might introduce rational numbers (“fractions”). So when you enter your matrix, you need to tell SAGE that rational numbers are allowable in its calculations. This is the advice in Computation R.SAGE to use the ring QQ . As an illustration create

a = m a t r i x ( Q Q , [ [ 1 , 2 , 3 , 4 ] , [ 5 , 6 , 7 , 8 ] , [ 9 , 1 0 , 1 1 , 1 2 ] ] )

and issue the command a.echelon _form() . The result is

1 0 − 1 − 2 0 1 2 3 0 0 0 0

However, if we adjust the entry by neglecting to specify QQ , then SAGE assumes that we only want to work with integers, since every entry of the matrix is an integer. So as an experiment, enter

b = m a t r i x ( [ [ 1 , 2 , 3 , 4 ] , [ 5 , 6 , 7 , 8 ] , [ 9 , 1 0 , 1 1 , 1 2 ] ] )

and issue the command b.echelon _form() . The result is

1 2 3 4 0 4 8 1 2 0 0 0 0

You can now clearly see Sage’s reluctance to multiply row 2 by 1 4 .

The ring QQ will of course suffice if your matrix has rational numbers for entries. Decimal entries are another place to be careful. If an entry of your matrix is the real number 2 . 1 7 , you are free to enter it as the rational number 2 1 7 1 0 0 and keep the ring QQ in the specification of your matrix. If you want to consider your entries as real numbers, then you might as well just specify your ring as the complex numbers CC . This advice also applies if you have complex numbers as entries.

If you allow SAGE to work with real or complex numbers, then the problem of round-off error becomes relevant. Computer arithmetic with real numbers is, of necessity, subject to minor inaccuracies and errors. This becomes problematic when row-reducing a matrix. If a zero entry is computed instead as an extremely small number, such as 1 . 2 8 7 × 1 0 − 1 8 , then an incorrect sequence of row operations will follow (with further incorrect results). So if you use CC be on the lookout for these kinds of potential pitfalls.

So, in summary, remember to always specify the ring you will be using for your matrices, and most matrices can be handled with a choice of QQ or CC .

When you need to do significant scientific computing with SAGE, there are extra facilities that will help you work with these subtleties.

Finally, you can also use a command of the form a.echelonize() to replace a with its reduced row-echelon form.

Computation Note LS.SAGE: Linear Solve

SAGE can solve a variety of systems of equations with the solve( ) command, even when the equations are not linear (see Exercise SSLE.M70). But we can afford to specialize here to just linear systems. First, you must specify your variables in advance, so for example, var(’x1,x2,x3’) might precede a system with three equations. Equations are then written just as you might expect, except that equality is written as == , since computer programs have traditionally reserved = to assign values to variables. And remember to use a * to indicate that a coefficient multiplies a variable.

The example below illustrates the use of the command and the possibilities for results. Each system would be preceded by establishing the variables with the command var(’x,y’) . In the case of an infinite solution set, free variables are denoted as rx where x is an integer that increases throughout a session. The style of this description of a solution set is reminiscent of the style we used in Chapter SLE before we were accustomed to using linear combinations of vectors (Theorem VFSLS).

Notice how the output contains equations written a format that might be suitable as input for further use within SAGE.

Computation Note VLC.SAGE: Vector Linear Combinations

Contributed by  Robert Beezer
Vectors in SAGE are constructed from lists, and are displayed horizontally. For example, the vector

v = 1 2 3 4

would be entered and named via the command

v = v e c t o r ( Q Q , [ 1 , 2 , 3 , 4 ] )

See the notes about rings ( Computation R.SAGE) and matrix entry ( Computation ME.SAGE) for reminders about specifying the relevant ring.

Vector addition and scalar multiplication are then very natural. If u and v are two vectors of the same size, then

2 ∗ u + ( − 3 ) ∗ v

will compute the correct vector. The result can be assigned to a variable (which will then contain a vector), or be printed. If printed, it will be written horizontally with parentheses for grouping. If u and v have different sizes, then SAGE will complain about “unsupported operand(s).”

Computation Note MI.SAGE: Matrix Inverse

Contributed by  Steve Canfield
If a is a matrix defined in SAGE, then a.inverse() will return the inverse of a , should it exist. In the case where a does not have an inverse SAGE will tell you the matrix must be nonsingular (see Theorem NI).

Computation Note TM.SAGE: Transpose of a Matrix

Suppose a is the name of a matrix stored in SAGE. Then a.transpose() will return the transpose of a .

Computation Note E.SAGE: Eigenspaces

Contributed by  Steve Canfield
SAGE can compute eigenspaces and eigenvalues for you. If you have a matrix named a and you type

a . e i g e n s p a c e s ( )

you will get a listing of the eigenvalues and the eigenspace for each. Let’s do an example. Your output may be formatted slightly different from what we have here.

m = m a t r i x ( Q Q , [ [ − 1 3 , − 8 , − 4 ] , [ 1 2 , 7 , 4 ] , [ 2 4 , 1 6 , 7 ] ] ) m . e i g e n s p a c e s ( ) [ ( 3 , [ ( 1 , 2 ∕ 3 , 1 ∕ 3 ) ] ) , ( − 1 , [ ( 1 , 0 , 1 ∕ 2 ) , ( 0 , 1 , − 1 ∕ 2 ) ] ) ]

Whew, that looks like a mess. At the top level, eigenspaces() returns a dictionary whose keys are the eigenvalues. So in this case we have eigenvalues 3 and -1. Each eigenvalue has an array after it that forms the basis of the eigenspace. In our example, there is 1 vector for λ = 3 and 2 vectors for λ = − 1 . Finally, the vectors SAGE spits out may not be the nicest ones to work with. In particular, we might want to scale the vectors to get rid of fractions.

Chapter 10. Appendix P  Preliminaries

This appendix contains important ideas about complex numbers, sets, and the logic and techniques of forming proofs. It is not meant to be read straight through, but you should head here when you need to review these ideas.

We choose to expand the set of scalars from the real numbers, ℝ , to the set of complex numbers, ℂ . So basic operations with complex numbers (like addition and division) will be necessary. This can be safely postponed until your arrival in Section O, and a refresher before Chapter E would be a good idea as well.

Sets are extremely important in all of mathematics, but maybe you have not had much exposure to the basic operations. Check out Section SET. The text will send you here frequently as well. Visit often.

This book is as much about doing mathematics as it is about linear algebra. The “Proof Techniques” are vignettes about logic, types of theorems, structure of proofs, or just plain old-fashioned advice about how to do advanced mathematics. The text will frequently point to one of these techniques in advance of their first use, and for specific instructions there will be additional references. If you find constructing proofs difficult (we all did once), then head back here and browse through the advice for second or third readings.

Section CNO  Complex Number Operations

Section SET  Sets

Section PT  Proof Techniques

Parent topic: Part C  Core

Previous topic: Section SAGE  SAGE: Open Source Mathematics Software

Next topic: Section CNO  Complex Number Operations

Section CNO  Complex Number Operations

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we review of the basics of working with complex numbers.

Parent topic: Appendix P  Preliminaries

Previous topic: Appendix P  Preliminaries

Next topic: Section SET  Sets

Subsection CNA: Arithmetic with complex numbers

A complex number is a linear combination of 1 and i = − 1 , typically written in the form a + b i . Complex numbers can be added, subtracted, multiplied and divided, just like we are used to doing with real numbers, including the restriction on division by zero. We will not define these operations carefully, but instead illustrate with examples.

Example  ACN
Arithmetic of complex numbers

( 2 + 5 i ) + ( 6 − 4 i ) = ( 2 + 6 ) + ( 5 + ( − 4 ) ) i = 8 + i ( 2 + 5 i ) − ( 6 − 4 i ) = ( 2 − 6 ) + ( 5 − ( − 4 ) ) i = − 4 + 9 i ( 2 + 5 i ) ( 6 − 4 i ) = ( 2 ) ( 6 ) + ( 5 i ) ( 6 ) + ( 2 ) ( − 4 i ) + ( 5 i ) ( − 4 i ) = 1 2 + 3 0 i − 8 i − 2 0 i 2 = 1 2 + 2 2 i − 2 0 ( − 1 ) = 3 2 + 2 2 i  Division takes just a bit more care. We multiply the denominator by a complex number chosen to produce a real number and then we can produce a complex number as a result. 2 + 5 i 6 − 4 i = 2 + 5 i 6 − 4 i 6 + 4 i 6 + 4 i = − 8 + 3 8 i 5 2 = − 8 5 2 + 3 8 5 2 i = − 2 1 3 + 1 9 2 6 i

⊠

In this example, we used 6 + 4 i to convert the denominator in the fraction to a real number. This number is known as the conjugate, which we define in the next section. We will often exploit the basic properties of complex number addition, subtraction, multiplication and division, so we will carefully define the two basic operations, together with a definition of equality, and then collect nine basic properties in a theorem.

Definition  CNE
Complex Number Equality
The complex numbers α = a + b i and β = c + d i are equal, denoted α = β , if a = c and b = d .

(This definition contains Notation CNE.) △

Definition  CNA
Complex Number Addition
The sum of the complex numbers α = a + b i and β = c + d i , denoted α + β , is ( a + c ) + ( b + d ) i .

(This definition contains Notation CNA.) △

Definition  CNM
Complex Number Multiplication
The product of the complex numbers α = a + b i and β = c + d i , denoted α β , is ( a c − b d ) + ( a d + b c ) i .

(This definition contains Notation CNM.) △

Theorem  PCNA
Properties of Complex Number Arithmetic
The operations of addition and multiplication of complex numbers have the following properties.

□

Proof   We could derive each of these properties of complex numbers with a proof that builds on the identical properties of the real numbers. The only proof that might be at all interesting would be to show Property MICN since we would need to trot out a conjugate. For this property, and especially for the others, we might be tempted to construct proofs of the identical properties for the reals. This would take us way too far afield, so we will draw a line in the sand right here and just agree that these nine fundamental behaviors are true. OK?

Mostly we have stated these nine properties carefully so that we can make reference to them later in other proofs. So we will be linking back here often. ■

Subsection CCN: Conjugates of Complex Numbers

Definition  CCN
Conjugate of a Complex Number
The conjugate of the complex number α = a + b i ∈ ℂ is the complex number α ¯ = a − b i .

(This definition contains Notation CCN.) △

Example  CSCN
Conjugate of some complex numbers

2 + 3 i ¯ = 2 − 3 i 5 − 4 i ¯ = 5 + 4 i − 3 + 0 i ¯ = − 3 + 0 i 0 + 0 i ¯ = 0 + 0 i

⊠

Notice how the conjugate of a real number leaves the number unchanged. The conjugate enjoys some basic properties that are useful when we work with linear expressions involving addition and multiplication.

Theorem  CCRA
Complex Conjugation Respects Addition
Suppose that α and β are complex numbers. Then α + β ¯ = α ¯ + β ¯ . □

Proof   Let α = a + b i and β = r + s i . Then

α + β ¯ = ( a + r ) + ( b + s ) i ¯ = ( a + r ) − ( b + s ) i = ( a − b i ) + ( r − s i ) = α ¯ + β ¯

■

Theorem  CCRM
Complex Conjugation Respects Multiplication
Suppose that α and β are complex numbers. Then α β ¯ = α ¯ β ¯ . □

Proof   Let α = a + b i and β = r + s i . Then

α β ¯ = ( a r − b s ) + ( a s + b r ) i ¯ = ( a r − b s ) − ( a s + b r ) i = ( a r − ( − b ) ( − s ) ) + ( a ( − s ) + ( − b ) r ) i = ( a − b i ) ( r − s i ) = α ¯ β ¯

■

Theorem  CCT
Complex Conjugation Twice
Suppose that α is a complex number. Then α ¯ ¯ = α . □

Proof   Let α = a + b i . Then

α ¯ ¯ = a − b i ¯ = a − ( − b i ) = a + b i = α

■

Subsection MCN: Modulus of a Complex Number

We define one more operation with complex numbers that may be new to you.

Definition  MCN
Modulus of a Complex Number
The modulus of the complex number α = a + b i ∈ ℂ , is the nonnegative real number

α = α α ¯ = a 2 + b 2 .

△

Example  MSCN
Modulus of some complex numbers

2 + 3 i = 1 3 5 − 4 i = 4 1 − 3 + 0 i = 3 0 + 0 i = 0

⊠

The modulus can be interpreted as a version of the absolute value for complex numbers, as is suggested by the notation employed. You can see this in how − 3 = − 3 + 0 i = 3 . Notice too how the modulus of the complex zero, 0 + 0 i , has value 0 .

Section SET  Sets

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Definition  SET
Set
A set is an unordered collection of objects. If S is a set and x is an object that is in the set S , we write x ∈ S . If x is not in S , then we write x ∉ S . We refer to the objects in a set as its elements.

(This definition contains Notation SETM.) △

Hard to get much more basic than that. Notice that the objects in a set can be anything, and there is no notion of order among the elements of the set. A set can be finite as well as infinite. A set can contain other sets as its objects. At a primitive level, a set is just a way to break up some class of objects into two groupings: those objects in the set, and those objects not in the set.

Example  SETM
Set membership
From the set of all possible symbols, construct the following set of three symbols,

S = ■ , ♦ , ⋆

Then the statement ■ ∈ S is true, while the statement ▴ ∈ S is false. However, then the statement ▴ ∉ S is true. ⊠

A portion of a set is known as a subset. Notice how the following definition uses an implication (if whenever…then…). Note too how the definition of a subset relies on the definition of a set through the idea of set membership.

Definition  SSET
Subset
If S and T are two sets, then S is a subset of T , written S ⊆ T if whenever x ∈ S then x ∈ T .

(This definition contains Notation SSET.) △

If we want to disallow the possibility that S is the same as T , we use the notation S ⊂ T and we say that S is a proper subset of T . We’ll do an example, but first we’ll define a special set.

Definition  ES
Empty Set
The empty set is the set with no elements. Its is denoted by ∅ .

(This definition contains Notation ES.) △

Example  SSET
Subset
If S = ■ , ♦ , ⋆ , T = ⋆ , ♦ , R = ▴ , ⋆ , then

T ⊆ S R ⊈ T ∅ ⊆ S T ⊂ S S ⊆ S S ⊄ S

⊠

What does it mean for two sets to be equal? They must be the same. Well, that explanation is not really too helpful, is it? How about: If A ⊆ B and B ⊆ A , then A equals B . This gives us something to work with, if A is a subset of B , and vice versa, then they must really be the same set. We will now make the symbol “ = ” do double-duty and extend its use to statements like A = B , where A and B are sets. Here’s the definition, which we will reference often.

Definition  SE
Set Equality
Two sets, S and T , are equal, if S ⊆ T and T ⊆ S . In this case, we write S = T .

(This definition contains Notation SE.) △

Sets are typically written inside of braces, as , as we have seen above. However, when sets have more than a few elements, a description will typically have two components. The first is a description of the general type of objects contained in a set, while the second is some sort of restriction on the properties the objects have. Every object in the set must be of the type described in the first part and it must satisfy the restrictions in the second part. Conversely, any object of the proper type for the first part, that also meets the conditions of the second part, will be in the set. These two parts are set off from each other somehow, often with a vertical bar ( | ) or a colon (:).

I like to think of sets as clubs. The first part is some description of the type of people who might belong to the club, the basic objects. For example, a bicycle club would describe its members as being people who like to ride bicycles. The second part is like a membership committee, it restricts the people who are allowed in the club. Continuing with our bicycle club analogy, we might decide to limit ourselves to “serious” riders and only have members who can document having ridden 100 kilometers or more in a single day at least one time.

The restrictions on membership can migrate around some between the first and second part, and there may be several ways to describe the same set of objects. Here’s a more mathematical example, employing the set of all integers, ℤ , to describe the set of even integers.

E = x ∈ ℤ x   is an even number = x ∈ ℤ 2   divides  x   evenly = 2 k k ∈ ℤ

Notice how this set tells us that its objects are integer numbers (not, say, matrices or functions, for example) and just those that are even. So we can write that 1 0 ∈ E , while 1 7 ∉ E once we check the membership criteria. We also recognize the question

1 − 3 5 2 0 3 ∈ E  ?

as being simply ridiculous.

Parent topic: Appendix P  Preliminaries

Previous topic: Section CNO  Complex Number Operations

Next topic: Section PT  Proof Techniques

Subsection SC: Set Cardinality

On occasion, we will be interested in the number of elements in a finite set. Here’s the definition and the associated notation.

Definition  C
Cardinality
Suppose S is a finite set. Then the number of elements in S is called the cardinality or size of S , and is denoted S .

(This definition contains Notation C.) △

Example  CS
Cardinality and Size
If S = ♦ , ⋆ , ■ , then S = 3 . ⊠

Subsection SO: Set Operations

In this subsection we define and illustrate the three most common basic ways to manipulate sets to create other sets. Since much of linear algebra is about sets, we will use these often.

Definition  SU
Set Union
Suppose S and T are sets. Then the union of S and T , denoted S ∪ T , is the set whose elements are those that are elements of S or of T , or both. More formally,

x ∈ S ∪ T   if and only if  x ∈ S   or  x ∈ T

(This definition contains Notation SU.) △

Notice that the use of the word “or” in this definition is meant to be non-exclusive. That is, it allows for x to be an element of both S and T and still qualify for membership in S ∪ T .

Example  SU
Set union
If S = ♦ , ⋆ , ■ and T = ♦ , ⋆ , ▴ then S ∪ T = ♦ , ⋆ , ■ , ▴ . ⊠

Definition  SI
Set Intersection
Suppose S and T are sets. Then the intersection of S and T , denoted S ∩ T , is the set whose elements are only those that are elements of S and of T . More formally,

x ∈ S ∩ T   if and only if  x ∈ S   and  x ∈ T

(This definition contains Notation SI.) △

Example  SI
Set intersection
If S = ♦ , ⋆ , ■ and T = ♦ , ⋆ , ▴ then S ∩ T = ♦ , ⋆ . ⊠

The union and intersection of sets are operations that begin with two sets and produce a third, new, set. Our final operation is the set complement, which we usually think of as an operation that takes a single set and creates a second, new, set. However, if you study the definition carefully, you will see that it needs to be computed relative to some “universal” set.

Definition  SC
Set Complement
Suppose S is a set that is a subset of a universal set U . Then the complement of S , denoted S ¯ , is the set whose elements are those that are elements of U and not elements of S . More formally,

x ∈ S ¯   if and only if  x ∈ U   and  x ∉ S

(This definition contains Notation SC.) △

Notice that there is nothing at all special about the universal set. This is simply a term that suggests that U contains all of the possible objects we are considering. Often this set will be clear from the context, and we won’t think much about it, nor reference it in our notation. In other cases (rarely in our work in this course) the exact nature of the universal set must be made explicit, and reference to it will possibly be carried through in our choice of notation.

Example  SC
Set complement
If U = ♦ , ⋆ , ■ , ▴ and S = ♦ , ⋆ , ■ then S ¯ = ▴ . ⊠

There are many more natural operations that can be performed on sets, such as an exclusive-or and the symmetric difference. Many of these can be defined in terms of the union, intersection and complement. We will not have much need of them in this course, and so we will not give precise descriptions here in this preliminary section.

There is also an interesting variety of basic results that describe the interplay of these operations with each other. We mention just two as an example, these are known as DeMorgan’s Laws.

S ∪ T ¯ = S ¯ ∩ T ¯ S ∩ T ¯ = S ¯ ∪ T ¯

Besides having an appealing symmetry, we mention these two facts, since constructing the proofs of each is a useful exercise that will require a solid understanding of all but one of the definitions presented in this section. Give it a try.

Section PT  Proof Techniques

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we collect many short essays designed to help you understand how to read, understand and construct proofs. Some are very factual, while others consist of advice. They appear in the order that they are first needed (or advisable) in the text, and are meant to be self-contained. So you should not think of reading through this section in one sitting as you begin this course. But be sure to head back here for a first reading whenever the text suggests it. Also think about returning to browse at various points during the course, and especially as you struggle with becoming an accomplished mathematician who is comfortable with the difficult process of designing new proofs.

Parent topic: Appendix P  Preliminaries

Previous topic: Section SET  Sets

Next topic: Appendix A  Archetypes

Proof Technique D: Definitions

A definition is a made-up term, used as a kind of shortcut for some typically more complicated idea. For example, we say a whole number is even as a shortcut for saying that when we divide the number by two we get a remainder of zero. With a precise definition, we can answer certain questions unambiguously. For example, did you ever wonder if zero was an even number? Now the answer should be clear since we have a precise definition of what we mean by the term even.

A single term might have several possible definitions. For example, we could say that the whole number n is even if there is another whole number k such that n = 2 k . We say this is an equivalent definition since it categorizes even numbers the same way our first definition does.

Definitions are like two-way streets — we can use a definition to replace something rather complicated by its definition (if it fits) and we can replace a definition by its more complicated description. A definition is usually written as some form of an implication, such as “If something-nice-happens, then blatzo.” However, this also means that “If blatzo, then something-nice-happens,” even though this may not be formally stated. This is what we mean when we say a definition is a two-way street — it is really two implications, going in opposite “directions.”

Anybody (including you) can make up a definition, so long as it is unambiguous, but the real test of a definition’s utility is whether or not it is useful for describing interesting or frequent situations.

We will talk about theorems later (and especially equivalences). For now, be sure not to confuse the notion of a definition with that of a theorem.

In this book, we will display every new definition carefully set-off from the text, and the term being defined will be written thus: definition. Additionally, there is a full list of all the definitions, in order of their appearance located at the front of the book ( Definitions). Finally, the acronym for each definition can be found in the index ( Index). Definitions are critical to doing mathematics and proving theorems, so we’ve given you lots of ways to locate a definition should you forget its…uh, well, …definition.

Can you formulate a precise definition for what it means for a number to be odd? (Don’t just say it is the opposite of even. Act as if you don’t have a definition for even yet.) Can you formulate your definition a second, equivalent, way? Can you employ your definition to test an odd and an even number for “odd-ness”?

Proof Technique T: Theorems

Higher mathematics is about understanding theorems. Reading them, understanding them, applying them, proving them. Every theorem is a shortcut — we prove something in general, and then whenever we find a specific instance covered by the theorem we can immediately say that we know something else about the situation by applying the theorem. In many cases, this new information can be gained with much less effort than if we did not know the theorem.

The first step in understanding a theorem is to realize that the statement of every theorem can be rewritten using statements of the form “If something-happens, then something-else-happens.” The “something-happens” part is the hypothesis and the “something-else-happens” is the conclusion. To understand a theorem, it helps to rewrite its statement using this construction. To apply a theorem, we verify that “something-happens” in a particular instance and immediately conclude that “something-else-happens.” To prove a theorem, we must argue based on the assumption that the hypothesis is true, and arrive through the process of logic that the conclusion must then also be true.

Proof Technique L: Language

Mathematics is a language. It is a way to express complicated ideas clearly, precisely, and unambiguously. Because of this, it can be difficult to read. Read slowly, and have pencil and paper at hand. It will usually be necessary to read something several times. While reading can be difficult, it is even harder to speak mathematics, and so that is the topic of this technique.

“Natural” language, in the present case English, is fraught with ambiguity. Consider the possible meanings of the sentence: The fish is ready to eat. One fish, or two fish? Are the fish hungry, or will the fish be eaten? (See Exercise SSLE.M10, Exercise SSLE.M11, Exercise SSLE.M12, Exercise SSLE.M13.) In your daily interactions with others, give some thought to how many mis-understandings arise from the ambiguity of pronouns, modifiers and objects.

I am going to suggest a simple modification to the way you use language that will make it much, much easier to become proficient at speaking mathematics and eventually it will become second nature. Think of it as a training aid or practice drill you might use when learning to become skilled at a sport.

First, eliminate pronouns from your vocabulary when discussing linear algebra, in class or with your colleagues. Do not use: it, that, those, their or similar sources of confusion. This is the single easiest step you can take to make your oral expression of mathematics clearer to others, and in turn, it will greatly help your own understanding.

Now rid yourself of the word “thing” (or variants like “something”). When you are tempted to use this word realize that there is some object you want to discuss, and we likely have a definition for that object (see the discussion at Technique D). Always “think about your objects” and many aspects of the study of mathematics will get easier. Ask yourself: “Am I working with a set, a number, a function, an operation, a differential equation, or what?” Knowing what an object is will allow you to narrow down the procedures you may apply to it. If you have studied an object-oriented computer programming language, then you will already have experience identifying objects and thinking carefully about what procedures are allowed to be applied to them.

Third, eliminate the verb “works” (as in “the equation works”) from your vocabulary. This term is used as a substitute when we are not sure just what we are trying to accomplish. Usually we are trying to say that some object fulfills some condition. The condition might even have a definition associated with it, making it even easier to describe.

Last, speak slooooowly and thoughtfully as you try to get by without all these lazy words. It is hard at first, but you will get better with practice. Especially in class, when the pressure is on and all eyes are on you, don’t succumb to the temptation to use these weak words. Slow down, we’d all rather wait for a slow, well-formed question or answer than a fast, sloppy, incomprehensible one.

You will find the improvement in your ability to speak clearly about complicated ideas will greatly improve your ability to think clearly about complicated ideas. And I believe that you cannot think clearly about complicated ideas if you cannot formulate questions or answers clearly in the correct language. This is as applicable to the study of law, economics or philosophy as it is to the study of science or mathematics.

In this spirit, Dupont Hubert has contributed the following quotation, which is widely used in French mathematics courses (and which might be construed as the contrapositive of Technique CP)

which translates as

So when you come to class, check your pronouns at the door, along with other weak words. And when studying with friends, you might make a game of catching one another using pronouns, “thing,” or “works.” I know I’ll be calling you on it!

Proof Technique GS: Getting Started

“I don’t know how to get started!” is often the lament of the novice proof-builder. Here are a few pieces of advice.

Proof Technique C: Constructive Proofs

Conclusions of proofs come in a variety of types. Often a theorem will simply assert that something exists. The best way, but not the only way, to show something exists is to actually build it. Such a proof is called constructive. The thing to realize about constructive proofs is that the proof itself will contain a procedure that might be used computationally to construct the desired object. If the procedure is not too cumbersome, then the proof itself is as useful as the statement of the theorem.

Proof Technique E: Equivalences

When a theorem uses the phrase “if and only if” (or the abbreviation “iff”) it is a shorthand way of saying that two if-then statements are true. So if a theorem says “P if and only if Q,” then it is true that “if P, then Q” while it is also true that “if Q, then P.” For example, it may be a theorem that “I wear bright yellow knee-high plastic boots if and only if it is raining.” This means that I never forget to wear my super-duper yellow boots when it is raining and I wouldn’t be seen in such silly boots unless it was raining. You never have one without the other. I’ve got my boots on and it is raining or I don’t have my boots on and it is dry.

The upshot for proving such theorems is that it is like a 2-for-1 sale, we get to do two proofs. Assume P and conclude Q , then start over and assume Q and conclude P . For this reason, “if and only if” is sometimes abbreviated by ⇔ , while proofs indicate which of the two implications is being proved by prefacing each with ⇒ or ⇐ . A carefully written proof will remind the reader which statement is being used as the hypothesis, a quicker version will let the reader deduce it from the direction of the arrow. Tradition dictates we do the “easy” half first, but that’s hard for a student to know until you’ve finished doing both halves! Oh well, if you rewrite your proofs (a good habit), you can then choose to put the easy half first.

Theorems of this type are called “equivalences” or “characterizations,” and they are some of the most pleasing results in mathematics. They say that two objects, or two situations, are really the same. You don’t have one without the other, like rain and my yellow boots. The more different P and Q seem to be, the more pleasing it is to discover they are really equivalent. And if P describes a very mysterious solution or involves a tough computation, while Q is transparent or involves easy computations, then we’ve found a great shortcut for better understanding or faster computation. Remember that every theorem really is a shortcut in some form. You will also discover that if proving P ⇒ Q is very easy, then proving Q ⇒ P is likely to be proportionately harder. Sometimes the two halves are about equally hard. And in rare cases, you can string together a whole sequence of other equivalences to form the one you’re after and you don’t even need to do two halves. In this case, the argument of one half is just the argument of the other half, but in reverse.

One last thing about equivalences. If you see a statement of a theorem that says two things are “equivalent,” translate it first into an “if and only if” statement.

Proof Technique N: Negation

When we construct the contrapositive of a theorem ( Technique CP), we need to negate the two statements in the implication. And when we construct a proof by contradiction ( Technique CD), we need to negate the conclusion of the theorem. One way to construct a converse ( Technique CV) is to simultaneously negate the hypothesis and conclusion of an implication (but remember that this is not guaranteed to be a true statement). So we often have the need to negate statements, and in some situations it can be tricky.

If a statement says that a set is empty, then its negation is the statement that the set is nonempty. That’s straightforward. Suppose a statement says “something-happens” for all i , or every i , or any i . Then the negation is that “something-doesn’t-happen” for at least one value of i . If a statement says that there exists at least one “thing,” then the negation is the statement that there is no “thing.” If a statement says that a “thing” is unique, then the negation is that there is zero, or more than one, of the “thing.”

We are not covering all of the possibilities, but we wish to make the point that logical qualifiers like “there exists” or “for every” must be handled with care when negating statements. Studying the proofs which employ contradiction (as listed in Technique CD) is a good first step towards understanding the range of possibilities.

Proof Technique CP: Contrapositives

The contrapositive of an implication P ⇒ Q is the implication n o t ( Q ) ⇒ n o t ( P ) , where “not” means the logical negation, or opposite. An implication is true if and only if its contrapositive is true. In symbols, ( P ⇒ Q ) ⇔ ( n o t ( Q ) ⇒ n o t ( P ) ) is a theorem. Such statements about logic, that are always true, are known as tautologies.

For example, it is a theorem that “if a vehicle is a fire truck, then it has big tires and has a siren.” (Yes, I’m sure you can conjure up a counterexample, but play along with me anyway.) The contrapositive is “if a vehicle does not have big tires or does not have a siren, then it is not a fire truck.” Notice how the “and” became an “or” when we negated the conclusion of the original theorem.

It will frequently happen that it is easier to construct a proof of the contrapositive than of the original implication. If you are having difficulty formulating a proof of some implication, see if the contrapositive is easier for you. The trick is to construct the negation of complicated statements accurately. More on that later.

Proof Technique CV: Converses

The converse of the implication P ⇒ Q is the implication Q ⇒ P . There is no guarantee that the truth of these two statements are related. In particular, if an implication has been proven to be a theorem, then do not try to use its converse too, as if it were a theorem. Sometimes the converse is true (and we have an equivalence, see Technique E). But more likely the converse is false, especially if it wasn’t included in the statement of the original theorem.

For example, we have the theorem, “if a vehicle is a fire truck, then it is has big tires and has a siren.” The converse is false. The statement that “if a vehicle has big tires and a siren, then it is a fire truck” is false. A police vehicle for use on a sandy public beach would have big tires and a siren, yet is not equipped to fight fires.

We bring this up now, because Theorem CSRN has a tempting converse. Does this theorem say that if r < n , then the system is consistent? Definitely not, as Archetype E has r = 3 < 4 = n , yet is inconsistent. This example is then said to be a counterexample to the converse. Whenever you think a theorem that is an implication might actually be an equivalence, it is good to hunt around for a counterexample that shows the converse to be false (the archetypes, Appendix A, can be a good hunting ground).

Proof Technique CD: Contradiction

Another proof technique is known as “proof by contradiction” and it can be a powerful (and satisfying) approach. Simply put, suppose you wish to prove the implication, “If A , then B .” As usual, we assume that A is true, but we also make the additional assumption that B is false. If our original implication is true, then these twin assumptions should lead us to a logical inconsistency. In practice we assume the negation of B to be true (see Technique N). So we argue from the assumptions A and  not ( B ) looking for some obviously false conclusion such as 1 = 6 , or a set is simultaneously empty and nonempty, or a matrix is both nonsingular and singular.

You should be careful about formulating proofs that look like proofs by contradiction, but really aren’t. This happens when you assume A and  not ( B ) and proceed to give a “normal” and direct proof that B is true by only using the assumption that A is true. Your last step is to then claim that B is true and you then appeal to the assumption that  not ( B ) is true, thus getting the desired contradiction. Instead, you could have avoided the overhead of a proof by contradiction and just run with the direct proof. This stylistic flaw is known, quite graphically, as “setting up the strawman to knock him down.”

Here is a simple example of a proof by contradiction. There are direct proofs that are just about as easy, but this will demonstrate the point, while narrowly avoiding knocking down the straw man.

Theorem: If a and b are odd integers, then their product, a b , is odd.

Proof: To begin a proof by contradiction, assume the hypothesis, that a and b are odd. Also assume the negation of the conclusion, in this case, that a b is even. Then there are integers, j , k , ℓ so that a = 2 j + 1 , b = 2 k + 1 , a b = 2 ℓ . Then

0 = a b − a b = ( 2 j + 1 ) ( 2 k + 1 ) − ( 2 ℓ ) = 4 j k + 2 j + 2 k − 2 ℓ + 1 = 2 2 j k + j + k − ℓ + 1

Notice how we used both our hypothesis and the negation of the conclusion in the second line. Now divide the integer on each end of this string of equalities by 2. On the left we get a remainder of 0, while on the right we see that the remainder will be 1. Both remainders cannot be correct, so this is our desired contradiction. Thus, the conclusion (that a b is odd) is true.

Again, we do not offer this example as the best proof of this fact about even and odd numbers, but rather it is a simple illustration of a proof by contradiction. You can find examples of proofs by contradiction in Theorem RREFU, Theorem NMUS, Theorem NPNT, Theorem TTMI, Theorem GSP, Theorem ELIS, Theorem EDYES, Theorem EMHE, Theorem EDELI, and Theorem DMFE, in addition to several examples and solutions to exercises.

Proof Technique U: Uniqueness

A theorem will sometimes claim that some object, having some desirable property, is unique. In other words, there should be only one such object. To prove this, a standard technique is to assume there are two such objects and proceed to analyze the consequences. The end result may be a contradiction ( Technique CD), or the conclusion that the two allegedly different objects really are equal.

Proof Technique ME: Multiple Equivalences

A very specialized form of a theorem begins with the statement “The following are equivalent…,” which is then followed by a list of statements. Informally, this lead-in sometimes gets abbreviated by “TFAE.” This formulation means that any two of the statements on the list can be connected with an “if and only if” to form a theorem. So if the list has n statements then, there are n ( n − 1 ) 2 possible equivalences that can be constructed (and are claimed to be true).

Suppose a theorem of this form has statements denoted as A , B , C ,… Z . To prove the entire theorem, we can prove A ⇒ B , B ⇒ C , C ⇒ D ,…, Y ⇒ Z and finally, Z ⇒ A . This circular chain of n equivalences would allow us, logically, if not practically, to form any one of the n ( n − 1 ) 2 possible equivalences by chasing the equivalences around the circle as far as required.

Proof Technique PI: Proving Identities

Many theorems have conclusions that say two objects are equal. Perhaps one object is hard to compute or understand, while the other is easy to compute or understand. This would make for a pleasing theorem. Whether the result is pleasing or not, we take the same approach to formulate a proof. Sometimes we need to employ specialized notions of equality, such as Definition SE or Definition CVE, but in other cases we can string together a list of equalities.

The wrong way to prove an identity is to begin by writing it down and then beating on it until it reduces to an obvious identity. The first flaw is that you would be writing down the statement you wish to prove, as if you already believed it to be true. But more dangerous is the possibility that some of your maneuvers are not reversible. Here’s an example. Let’s prove that 3 = − 3 .

3 = − 3  (This is a bad start) 3 2 = ( − 3 ) 2  Square both sides 9 = 9 0 = 0  Subtract 9 from both sides

So because 0 = 0 is a true statement, does it follow that 3 = − 3 is a true statement? Nope. Of course, we didn’t really expect a legitimate proof of 3 = − 3 , but this attempt should illustrate the dangers of this (incorrect) approach.

What you have just seen in the proof of Theorem VSPCV, and what you will see consistently throughout this text, is proofs of the following form. To prove that A = D we write

A = B  Theorem, Definition or Hypothesis justifying  A = B = C  Theorem, Definition or Hypothesis justifying  B = C = D  Theorem, Definition or Hypothesis justifying  C = D

In your scratch work exploring possible approaches to proving a theorem you may massage a variety of expressions, sometimes making connections to various bits and pieces, while some parts get abandoned. Once you see a line of attack, rewrite your proof carefully mimicking this style.

Proof Technique DC: Decompositions

Much of your mathematical upbringing, especially once you began a study of algebra, revolved around simplifying expressions — combining like terms, obtaining common denominators so as to add fractions, factoring in order to solve polynomial equations. However, as often as not, we will do the opposite. Many theorems and techniques will revolve around taking some object and “decomposing” it into some combination of other objects, ostensibly in a more complicated fashion. When we say something can “be written as” something else, we mean that the one object can be decomposed into some combination of other objects. This may seem unnatural at first, but results of this type will give us insight into the structure of the original object by exposing its inner workings. An appropriate analogy might be stripping the wallboards away from the interior of a building to expose the structural members supporting the whole building.

Perhaps you have studied integral calculus, or a pre-calculus course, where you learned about partial fractions. This is a technique where a fraction of two polynomials is decomposed (written as, expressed as) a sum of simpler fractions. The purpose in calculus is to make finding an antiderivative simpler. For example, you can verify the truth of the expression

1 2 x 5 + 2 x 4 − 2 0 x 3 + 6 6 x 2 − 2 9 4 x + 3 0 8 x 6 + x 5 − 3 x 4 + 2 1 x 3 − 5 2 x 2 + 2 0 x − 4 8 = 5 x + 2 x 2 − x + 6 + 3 x − 7 x 2 + 1 + 3 x + 4 + 1 x − 2

In an early course in algebra, you might be expected to combine the four terms on the right over a common denominator to create the “simpler” expression on the left. Going the other way, the partial fraction technique would allow you to systematically decompose the fraction of polynomials on the left into the sum of the four (arguably) simpler fractions of polynomials on the right.

This is a major shift in thinking, so come back here often, especially when we say “can be written as”, or “can be expressed as,” or “can be decomposed as.”

Proof Technique I: Induction

“Induction” or “mathematical induction” is a framework for proving statements that are indexed by integers. In other words, suppose you have a statement to prove that is really multiple statements, one for n = 1 , another for n = 2 , a third for n = 3 , and so on. If there is enough similarity between the statements, then you can use a script (the framework) to prove them all at once.

For example, consider the theorem

Theorem 1 + 2 + 3 + ⋯ + n = n ( n + 1 ) 2 for n ≥ 1 .

This is shorthand for the many statements 1 = 1 ( 1 + 1 ) 2 , 1 + 2 = 2 ( 2 + 1 ) 2 , 1 + 2 + 3 = 3 ( 3 + 1 ) 2 , 1 + 2 + 3 + 4 = 4 ( 4 + 1 ) 2 , and so on. Forever. You can do the calculations in each of these statements and verify that all four are true. We might not be surprised to learn that the fifth statement is true as well (go ahead and check). However, do we think the theorem is true for n = 8 7 2 ? Or n = 1 , 2 3 4 , 5 2 9 ?

To see that these questions are not so ridiculous, consider the following example from Rotman’s Journey into Mathematics. The statement “ n 2 − n + 4 1 is prime” is true for integers 1 ≤ n ≤ 4 0 (check a few). However, when we check n = 4 1 we find 4 1 2 − 4 1 + 4 1 = 4 1 2 , which is not prime.

So how do we prove infinitely many statements all at once? More formally, lets denote our statements as P ( n ) . Then, if we can prove the two assertions

then it follows that P ( n ) is true for all n ≥ 1 . To understand this, I liken the process to climbing an infinitely long ladder with equally spaced rungs. Confronted with such a ladder, suppose I tell you that you are able to step up onto the first rung, and if you are on any particular rung, then you are capable of stepping up to the next rung. It follows that you can climb the ladder as far up as you wish. The first formal assertion above is akin to stepping onto the first rung, and the second formal assertion is akin to assuming that if you are on any one rung then you can always reach the next rung.

In practice, establishing that P ( 1 ) is true is called the “base case” and in most cases is straightforward. Establishing that P ( k ) ⇒ P ( k + 1 ) is referred to as the “induction step,” or in this book (and elsewhere) we will typically refer to the assumption of P ( k ) as the “induction hypothesis.” This is perhaps the most mysterious part of a proof by induction, since it looks like you are assuming ( P ( k ) ) what you are trying to prove ( P ( n ) ). Sometimes it is even worse, since as you get more comfortable with induction, we often don’t bother to use a different letter ( k ) for the index ( n ) in the induction step. Notice that the second formal assertion never says that P ( k ) is true, it simply says that if P ( k ) were true, what might logically follow. We can establish statements like “If I lived on the moon, then I could pole-vault over a bar 12 meters high.” This may be a true statement, but it does not say we live on the moon, and indeed we may never live there.

Enough generalities. Let’s work an example and prove the theorem above about sums of integers. Formally, our statement is P ( n ) : 1 + 2 + 3 + ⋯ + n = n ( n + 1 ) 2 .

Proof: Base Case. P ( 1 ) is the statement 1 = 1 ( 1 + 1 ) 2 , which we see simplifies to the true statement 1 = 1 .

Induction Step: We will assume P ( k ) is true, and will try to prove P ( k + 1 ) . Given what we want to accomplish, it is natural to begin by examining the sum of the first k + 1 integers.

1 + 2 + 3 + ⋯ + ( k + 1 ) = 1 + 2 + 3 + ⋯ + k + ( k + 1 ) = k ( k + 1 ) 2 + ( k + 1 )  Induction Hypothesis = k 2 + k 2 + 2 k + 2 2 = k 2 + 3 k + 2 2 = ( k + 1 ) ( k + 2 ) 2 = ( k + 1 ) ( ( k + 1 ) + 1 ) 2

We then recognize the two ends of this chain of equalities as P ( k + 1 ) . So, by mathematical induction, the theorem is true for all n .

How do you recognize when to use induction? The first clue is a statement that is really many statements, one for each integer. The second clue would be that you begin a more standard proof and you find yourself using words like “and so on” (as above!) or lots of ellipses (dots) to establish patterns that you are convinced continue on and on forever. However, there are many minor instances where induction might be warranted but we don’t bother.

Induction is important enough, and used often enough, that it appears in various variations. The base case sometimes begins with n = 0 , or perhaps an integer greater than n . Some formulate the induction step as P ( k − 1 ) ⇒ P ( k ) . There is also a “strong form” of induction where we assume all of P ( 1 ) , P ( 2 ) , P ( 3 ) , … P ( k ) as a hypothesis for showing the conclusion P ( k + 1 ) .
You can find examples of induction in the proofs of Theorem GSP, Theorem DER, Theorem DT, Theorem DIM, Theorem EOMP, Theorem DCP, and Theorem KPLT.

Proof Technique P: Practice

Here is a technique used by many practicing mathematicians when they are teaching themselves new mathematics. As they read a textbook, monograph or research article, they attempt to prove each new theorem themselves, before reading the proof. Often the proofs can be very difficult, so it is wise not to spend too much time on each. Maybe limit your losses and try each proof for 10 or 15 minutes. Even if the proof is not found, it is time well-spent. You become more familiar with the definitions involved, and the hypothesis and conclusion of the theorem. When you do work through the proof, it might make more sense, and you will gain added insight about just how to construct a proof.

Proof Technique LC: Lemmas and Corollaries

Theorems often go by different titles. Two of the most popular being “lemma” and “corollary.” Before we describe the fine distinctions, be aware that lemmas, corollaries, propositions, claims and facts are all just theorems. And every theorem can be rephrased as an “if-then” statement, or perhaps a pair of “if-then” statements expressed as an equivalence ( Technique E).

A lemma is a theorem that is not too interesting in its own right, but is important for proving other theorems. It might be a generalization or abstraction of a key step of several different proofs. For this reason you often hear the phrase “technical lemma” though some might argue that the adjective “technical” is redundant.

A corollary is a theorem that follows very easily from another theorem. For this reason, corollaries frequently do not have proofs. You are expected to easily and quickly see how a previous theorem implies the corollary.

A proposition or fact is really just a codeword for a theorem. A claim might be similar, but some authors like to use claims within a proof to organize key steps. In a similar manner, some long proofs are organized as a series of lemmas.

In order to not confuse the novice, we have just called all our theorems theorems. It is also an organizational convenience. With only theorems and definitions, the theoretical backbone of the course is laid bare in the two lists of Definitions and Theorems.  

Chapter 11. Appendix A  Archetypes

WordNet (an open-source lexical database) gives the following definition of “archetype”: something that serves as a model or a basis for making copies.

Our archetypes are typical examples of systems of equations, matrices and linear transformations. They have been designed to demonstrate the range of possibilities, allowing you to compare and contrast them. Several are of a size and complexity that is usually not presented in a textbook, but should do a better job of being “typical.”

We have made frequent reference to many of these throughout the text, such as the frequent comparisons between Archetype A and Archetype B. Some we have left for you to investigate, such as Archetype J, which parallels Archetype I.

How should you use the archetypes? First, consult the description of each one as it is mentioned in the text. See how other facts about the example might illuminate whatever property or construction is being described in the example. Second, each property has a short description that usually includes references to the relevant theorems. Perform the computations and understand the connections to the listed theorems. Third, each property has a small checkbox in front of it. Use the archetypes like a workbook and chart your progress by “checking-off” those properties that you understand.

The next page has a chart that summarizes some (but not all) of the properties described for each archetype. Notice that while there are several types of objects, there are fundamental connections between them. That some lines of the table do double-duty is meant to convey some of these connections. Consult this table when you wish to quickly find an example of a certain phenomenon.

Archetype A   

Archetype B   

Archetype C   

Archetype D   

Archetype E   

Archetype F   

Archetype G   

Archetype H   

Archetype I   

Archetype J   

Archetype K   

Archetype L   

Archetype M   

Archetype N   

Archetype O   

Archetype P   

Archetype Q   

Archetype R   

Archetype S   

Archetype T   

Archetype U   

Archetype V   

Archetype W   

Archetype X   

Parent topic: Part C  Core

Previous topic: Section PT  Proof Techniques

Next topic: Archetype A   

Archetype A   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Linear system of three equations, three unknowns. Singular coefficient matrix with dimension 1 null space. Integer eigenvalues and a degenerate eigenspace for coefficient matrix.

A system of linear equations ( Definition SLE):

x 1 − x 2 + 2 x 3 = 1 2 x 1 + x 2 + x 3 = 8 x 1 + x 2 = 5

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = 2 , x 2 = 3 , x 3 = 1
x 1 = 3 , x 2 = 2 , x 3 = 0

Augmented matrix of the linear system of equations ( Definition AM):

1 − 1 2 1 2 1 1 8 1 1 0 5

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 1 3 0 1  − 1 2 0 0 0 0

Analysis of the augmented matrix ( Notation RREFA):

r = 2 D = 1 , 2 F = 3 , 4

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 x 3 = 3 2 0 + x 3 − 1 1 1

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

x 1 − x 2 + 2 x 3 = 0 2 x 1 + x 2 + x 3 = 0 x 1 + x 2 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0
x 1 = − 1 , x 2 = 1 , x 3 = 1
x 1 = − 5 , x 2 = 5 , x 3 = 5

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 1 0 0 1  − 1 0 0 0 0 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 2 D = 1 , 2 F = 3 , 4

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

1 − 1 2 2 1 1 1 1 0

Matrix brought to reduced row-echelon form:

1 0 1 0 1  − 1 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 2 D = 1 , 2 F = 3

Matrix (coefficient matrix) is nonsingular or singular? ( Theorem NMRRI) at the same time, examine the size of the set F above.Notice that this property does not apply to matrices that are not square.
Singular.

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.
− 1 1 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
1 2 1 , − 1 1 1

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 − 2 3
− 3 0 1 , 2 1 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 − 1 3 , 0 1 2 3

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 1 , 0 1 − 1

Inverse matrix, if it exists. The inverse is not defined for matrices that are not square, and if the matrix is square, then the matrix must be nonsingular. ( Definition MI, Theorem NI)

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   3  Rank:   2  Nullity:   1

Determinant of the matrix, which is only defined for square matrices. The matrix is nonsingular if and only if the determinant is nonzero ( Theorem SMZD). (Product of all eigenvalues?)
 Determinant  = 0

Eigenvalues, and bases for eigenspaces. (Definition EEM,Definition EM)

λ = 0 ℰ A 0 = − 1 1 1 λ = 2 ℰ A 2 = 1 5 3

Geometric and algebraic multiplicities. (Definition GMEDefinition AME)

γ A 0 = 1 α A 0 = 2 γ A 2 = 1 α A 2 = 1

Diagonalizable? (Definition DZM)
No, γ A 0 ≠ α B 0 , Theorem DMFE.

Parent topic: Appendix A  Archetypes

Previous topic: Appendix A  Archetypes

Next topic: Archetype B   

Archetype B   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with three equations, three unknowns. Nonsingular coefficient matrix. Distinct integer eigenvalues for coefficient matrix.

A system of linear equations ( Definition SLE):

− 7 x 1 − 6 x 2 − 1 2 x 3 = − 3 3 5 x 1 + 5 x 2 + 7 x 3 = 2 4 x 1 + 4 x 3 = 5

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = − 3 , x 2 = 5 , x 3 = 2

Augmented matrix of the linear system of equations ( Definition AM):

− 7 − 6 − 1 2 − 3 3 5 5 7 2 4 1 0 4 5

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 0 − 3 0 1  0 5 0 0 1 2

Analysis of the augmented matrix ( Notation RREFA):

r = 3 D = 1 , 2 , 3 F = 4

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 x 3 = − 3 5 2

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

− 1 1 x 1 + 2 x 2 − 1 4 x 3 = 0 2 3 x 1 − 6 x 2 + 3 3 x 3 = 0 1 4 x 1 − 2 x 2 + 1 7 x 3 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 0 0 0 1  0 0 0 0 1 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 3 D = 1 , 2 , 3 F = 4

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

− 7 − 6 − 1 2 5 5 7 1 0 4

Matrix brought to reduced row-echelon form:

1 0 0 0 1  0 0 0 1

Analysis of the row-reduced matrix ( Notation RREFA):

r = 3 D = 1 , 2 , 3 F =

Matrix (coefficient matrix) is nonsingular or singular? ( Theorem NMRRI) at the same time, examine the size of the set F above.Notice that this property does not apply to matrices that are not square.
Nonsingular.

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
− 7 5 1 , − 6 5 0 , − 1 2 7 4

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L =
1 0 0 , 0 1 0 , 0 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 0 , 0 1 0 , 0 0 1

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 0 , 0 1 0 , 0 0 1

Inverse matrix, if it exists. The inverse is not defined for matrices that are not square, and if the matrix is square, then the matrix must be nonsingular. ( Definition MI, Theorem NI)
− 1 0 − 1 2 − 9 1 3 2 8 1 1 2 5 2 3 5 2

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   3  Rank:   3  Nullity:   0

Determinant of the matrix, which is only defined for square matrices. The matrix is nonsingular if and only if the determinant is nonzero ( Theorem SMZD). (Product of all eigenvalues?)
 Determinant  = − 2

Eigenvalues, and bases for eigenspaces. (Definition EEM,Definition EM)

λ = − 1 ℰ B − 1 = − 5 3 1 λ = 1 ℰ B 1 = − 3 2 1 λ = 2 ℰ B 2 = − 2 1 1

Geometric and algebraic multiplicities. (Definition GMEDefinition AME)

γ B − 1 = 1 α B − 1 = 1 γ B 1 = 1 α B 1 = 1 γ B 2 = 1 α B 2 = 1

Diagonalizable? (Definition DZM)
Yes, distinct eigenvalues, Theorem DED.

The diagonalization. (Theorem DC)

− 1 − 1 − 1 2 3 1 − 1 − 2 1 − 7 − 6 − 1 2 5 5 7 1 0 4 − 5 − 3 − 2 3 2 1 1 1 1 = − 1 0 0 0 1 0 0 0 2

Parent topic: Appendix A  Archetypes

Previous topic: Archetype A   

Next topic: Archetype C   

Archetype C   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with three equations, four variables. Consistent. Null space of coefficient matrix has dimension 1.

A system of linear equations ( Definition SLE):

2 x 1 − 3 x 2 + x 3 − 6 x 4 = − 7 4 x 1 + x 2 + 2 x 3 + 9 x 4 = − 7 3 x 1 + x 2 + x 3 + 8 x 4 = − 8

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = − 7 , x 2 = − 2 , x 3 = 7 , x 4 = 1
x 1 = − 1 , x 2 = 7 , x 3 = 4 , x 4 = − 2

Augmented matrix of the linear system of equations ( Definition AM):

2 − 3 1 − 6 − 7 4 1 2 9 − 7 3 1 1 8 − 8

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 0 2 − 5 0 1  0 3 1 0 0 1 − 1 6

Analysis of the augmented matrix ( Notation RREFA):

r = 3 D = 1 , 2 , 3 F = 4 , 5

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 x 3 x 4 = − 5 1 6 0 + x 4 − 2 − 3 1 1

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

2 x 1 − 3 x 2 + x 3 − 6 x 4 = 0 4 x 1 + x 2 + 2 x 3 + 9 x 4 = 0 3 x 1 + x 2 + x 3 + 8 x 4 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0 , x 4 = 0
x 1 = − 2 , x 2 = − 3 , x 3 = 1 , x 4 = 1
x 1 = − 4 , x 2 = − 6 , x 3 = 2 , x 4 = 2

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 0 2 0 0 1  0 3 0 0 0 1 − 1 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 3 D = 1 , 2 , 3 F = 4 , 5

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

2 − 3 1 − 6 4 1 2 9 3 1 1 8

Matrix brought to reduced row-echelon form:

1 0 0 2 0 1  0 3 0 0 1 − 1

Analysis of the row-reduced matrix ( Notation RREFA):

r = 3 D = 1 , 2 , 3 F = 4

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.
− 2 − 3 1 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
2 4 3 , − 3 1 1 , 1 2 1

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L =
1 0 0 , 0 1 0 , 0 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 0 , 0 1 0 , 0 0 1

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 0 2 , 0 1 0 3 , 0 0 1 − 1

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   4  Rank:   3  Nullity:   1

Parent topic: Appendix A  Archetypes

Previous topic: Archetype B   

Next topic: Archetype D   

Archetype D   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with three equations, four variables. Consistent. Null space of coefficient matrix has dimension 2. Coefficient matrix identical to that of Archetype E, vector of constants is different.

A system of linear equations ( Definition SLE):

2 x 1 + x 2 + 7 x 3 − 7 x 4 = 8 − 3 x 1 + 4 x 2 − 5 x 3 − 6 x 4 = − 1 2 x 1 + x 2 + 4 x 3 − 5 x 4 = 4

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 1 , x 3 = 2 , x 4 = 1
x 1 = 4 , x 2 = 0 , x 3 = 0 , x 4 = 0
x 1 = 7 , x 2 = 8 , x 3 = 1 , x 4 = 3

Augmented matrix of the linear system of equations ( Definition AM):

2 1 7 − 7 8 − 3 4 − 5 − 6 − 1 2 1 1 4 − 5 4

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 3 − 2 4 0 1  1 − 3 0 0 0 0 0 0

Analysis of the augmented matrix ( Notation RREFA):

r = 2 D = 1 , 2 F = 3 , 4 , 5

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 x 3 x 4 = 4 0 0 0 + x 3 − 3 − 1 1 0 + x 4 2 3 0 1

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

2 x 1 + x 2 + 7 x 3 − 7 x 4 = 0 − 3 x 1 + 4 x 2 − 5 x 3 − 6 x 4 = 0 x 1 + x 2 + 4 x 3 − 5 x 4 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0 , x 4 = 0
x 1 = − 3 , x 2 = − 1 , x 3 = 1 , x 4 = 0
x 1 = 2 , x 2 = 3 , x 3 = 0 , x 4 = 1
x 1 = − 1 , x 2 = 2 , x 3 = 1 , x 4 = 1

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 3 − 2 0 0 1  1 − 3 0 0 0 0 0 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 2 D = 1 , 2 F = 3 , 4 , 5

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

2 1 7 − 7 − 3 4 − 5 − 6 1 1 4 − 5

Matrix brought to reduced row-echelon form:

1 0 3 − 2 0 1  1 − 3 0 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 2 D = 1 , 2 F = 3 , 4

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.
− 3 − 1 1 0 , 2 3 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
2 − 3 1 , 1 4 1

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 1 7 − 1 1 7
1 1 7 0 1 , − 1 7 1 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 7 1 1 , 0 1 1 1 1

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 3 − 2 , 0 1 1 − 3

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   4  Rank:   2  Nullity:   2

Parent topic: Appendix A  Archetypes

Previous topic: Archetype C   

Next topic: Archetype E   

Archetype E   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with three equations, four variables. Inconsistent. Null space of coefficient matrix has dimension 2. Coefficient matrix identical to that of Archetype D, constant vector is different.

A system of linear equations ( Definition SLE):

2 x 1 + x 2 + 7 x 3 − 7 x 4 = 2 − 3 x 1 + 4 x 2 − 5 x 3 − 6 x 4 = 3 x 1 + x 2 + 4 x 3 − 5 x 4 = 2

Some solutions to the system of linear equations (not necessarily exhaustive):
None. (Why?)

Augmented matrix of the linear system of equations ( Definition AM):

2 1 7 − 7 2 − 3 4 − 5 − 6 3 1 1 4 − 5 2

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 3 − 2 0 0 1  1 − 3 0 0 0 0 0 1

Analysis of the augmented matrix ( Notation RREFA):

r = 3 D = 1 , 2 , 5 F = 3 , 4

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
Inconsistent system, no solutions exist.

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

2 x 1 + x 2 + 7 x 3 − 7 x 4 = 0 − 3 x 1 + 4 x 2 − 5 x 3 − 6 x 4 = 0 x 1 + x 2 + 4 x 3 − 5 x 4 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0 , x 4 = 0
x 1 = 4 , x 2 = 1 3 , x 3 = 2 , x 4 = 5

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 3 − 2 0 0 1  1 − 3 0 0 0 0 0 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 2 D = 1 , 2 F = 3 , 4 , 5

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

2 1 7 − 7 − 3 4 − 5 − 6 1 1 4 − 5

Matrix brought to reduced row-echelon form:

1 0 3 − 2 0 1  1 − 3 0 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 2 D = 1 , 2 F = 3 , 4

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.
− 3 − 1 1 0 , 2 3 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
2 − 3 1 , 1 4 1

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 1 7 − 1 1 7
1 1 7 0 1 , − 1 7 1 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 7 1 1 , 0 1 1 1 1

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 3 − 2 , 0 1 1 − 3

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   4  Rank:   2  Nullity:   2

Parent topic: Appendix A  Archetypes

Previous topic: Archetype D   

Next topic: Archetype F   

Archetype F   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with four equations, four variables. Nonsingular coefficient matrix. Integer eigenvalues, one has “high” multiplicity.

A system of linear equations ( Definition SLE):

3 3 x 1 − 1 6 x 2 + 1 0 x 3 − 2 x 4 = − 2 7 9 9 x 1 − 4 7 x 2 + 2 7 x 3 − 7 x 4 = − 7 7 7 8 x 1 − 3 6 x 2 + 1 7 x 3 − 6 x 4 = − 5 2 − 9 x 1 + 2 x 2 + 3 x 3 + 4 x 4 = 5

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = 1 , x 2 = 2 , x 3 = − 2 , x 4 = 4

Augmented matrix of the linear system of equations ( Definition AM):

3 3 − 1 6 1 0 − 2 − 2 7 9 9 − 4 7 2 7 − 7 − 7 7 7 8 − 3 6 1 7 − 6 − 5 2 − 9 2 3 4 5

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 0 0 1 0 1  0 0 2 0 0 1 0 − 2 0 0 0 1  4

Analysis of the augmented matrix ( Notation RREFA):

r = 4 D = 1 , 2 , 3 , 4 F = 5

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 x 3 x 4 = 1 2 − 2 4

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

3 3 x 1 − 1 6 x 2 + 1 0 x 3 − 2 x 4 = 0 9 9 x 1 − 4 7 x 2 + 2 7 x 3 − 7 x 4 = 0 7 8 x 1 − 3 6 x 2 + 1 7 x 3 − 6 x 4 = 0 − 9 x 1 + 2 x 2 + 3 x 3 + 4 x 4 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0 , x 4 = 0

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 0 0 0 0 1  0 0 0 0 0 1 0 0 0 0 0 1  0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 4 D = 1 , 2 , 3 , 4 F = 5

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

3 3 − 1 6 1 0 − 2 9 9 − 4 7 2 7 − 7 7 8 − 3 6 1 7 − 6 − 9 2 3 4

Matrix brought to reduced row-echelon form:

1 0 0 0 0 1  0 0 0 0 1 0 0 0 0 1

Analysis of the row-reduced matrix ( Notation RREFA):

r = 4 D = 1 , 2 , 3 , 4 F =

Matrix (coefficient matrix) is nonsingular or singular? ( Theorem NMRRI) at the same time, examine the size of the set F above.Notice that this property does not apply to matrices that are not square.
Nonsingular.

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
3 3 9 9 7 8 − 9 , − 1 6 − 4 7 − 3 6 2 , 1 0 2 7 1 7 3 , − 2 − 7 − 6 4

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L =
1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

Inverse matrix, if it exists. The inverse is not defined for matrices that are not square, and if the matrix is square, then the matrix must be nonsingular. ( Definition MI, Theorem NI)
− 8 6 3 3 8 3 − 1 1 3 7 3 − 1 2 9 2 8 6 3 − 1 7 2 3 1 6 − 1 3 6 − 2 1 − 4 5 2 2 9 3 − 5 2 1 3 6

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   4  Rank:   4  Nullity:   0

Determinant of the matrix, which is only defined for square matrices. The matrix is nonsingular if and only if the determinant is nonzero ( Theorem SMZD). (Product of all eigenvalues?)
 Determinant  = − 1 8

Eigenvalues, and bases for eigenspaces. (Definition EEM,Definition EM)

λ = − 1 ℰ F − 1 = 1 2 0 1 λ = 2 ℰ F 2 = 2 5 2 1 λ = 3 ℰ F 3 = 1 1 0 7 , 1 7 4 5 2 1 0

Geometric and algebraic multiplicities. (Definition GMEDefinition AME)

γ F − 1 = 1 α F − 1 = 1 γ F 2 = 1 α F 2 = 1 γ F 3 = 2 α F 3 = 2

Diagonalizable? (Definition DZM)
Yes, full eigenspaces, Theorem DMFE.

The diagonalization. (Theorem DC)

1 2 − 5 1 − 1 − 3 9 1 8 − 7 3 2 7 7 − 1 3 7 6 7 − 1 7 2 6 7 − 1 2 7 5 7 − 2 7 3 3 − 1 6 1 0 − 2 9 9 − 4 7 2 7 − 7 7 8 − 3 6 1 7 − 6 − 9 2 3 4 1 2 1 1 7 2 5 1 4 5 0 2 0 2 1 1 1 7 0 = − 1 0 0 0 0 2 0 0 0 0 3 0 0 0 0 3

Parent topic: Appendix A  Archetypes

Previous topic: Archetype E   

Next topic: Archetype G   

Archetype G   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with five equations, two variables. Consistent. Null space of coefficient matrix has dimension 0. Coefficient matrix identical to that of Archetype H, constant vector is different.

A system of linear equations ( Definition SLE):

2 x 1 + 3 x 2 = 6 − x 1 + 4 x 2 = − 1 4 3 x 1 + 1 0 x 2 = − 2 3 x 1 − x 2 = 2 0 6 x 1 + 9 x 2 = 1 8

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = 6 , x 2 = − 2

Augmented matrix of the linear system of equations ( Definition AM):

2 3 6 − 1 4 − 1 4 3 1 0 − 2 3 − 1 2 0 6 9 1 8

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 6 0 1  − 2 0 0 0 0 0 0 0 0 0

Analysis of the augmented matrix ( Notation RREFA):

r = 2 D = 1 , 2 F = 3

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 = 6 − 2

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

2 x 1 + 3 x 2 = 0 − x 1 + 4 x 2 = 0 3 x 1 + 1 0 x 2 = 0 3 x 1 − x 2 = 0 6 x 1 + 9 x 2 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 0 0 1  0 0 0 0 0 0 0 0 0 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 2 D = 1 , 2 F = 3

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

2 3 − 1 4 3 1 0 3 − 1 6 9

Matrix brought to reduced row-echelon form:

1 0 0 1 0 0 0 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 2 D = 1 , 2 F =

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
2 − 1 3 3 6 , 3 4 1 0 − 1 9

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 0 0 0 − 1 3 0 1 0 1 − 1 3 0 0 1 1 − 1
1 3 1 3 1 0 1 , 0 − 1 − 1 1 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 2 1 3 , 0 1 1 − 1 0

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 , 0 1

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   2  Rank:   2  Nullity:   0

Parent topic: Appendix A  Archetypes

Previous topic: Archetype F   

Next topic: Archetype H   

Archetype H   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with five equations, two variables. Inconsistent, overdetermined. Null space of coefficient matrix has dimension 0. Coefficient matrix identical to that of Archetype G, constant vector is different.

A system of linear equations ( Definition SLE):

2 x 1 + 3 x 2 = 5 − x 1 + 4 x 2 = 6 3 x 1 + 1 0 x 2 = 2 3 x 1 − x 2 = − 1 6 x 1 + 9 x 2 = 3

Some solutions to the system of linear equations (not necessarily exhaustive):
None. (Why?)

Augmented matrix of the linear system of equations ( Definition AM):

2 3 5 − 1 4 6 3 1 0 2 3 − 1 − 1 6 9 3

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 0 0 0 1  0 0 0 1 0 0 0 0 0 0

Analysis of the augmented matrix ( Notation RREFA):

r = 3 D = 1 , 2 , 3 F =

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
Inconsistent system, no solutions exist.

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

2 x 1 + 3 x 2 = 0 − x 1 + 4 x 2 = 0 3 x 1 + 1 0 x 2 = 0 3 x 1 − x 2 = 0 6 x 1 + 9 x 2 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 0 0 0 1  0 0 0 0 0 0 0 0 0 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 2 D = 1 , 2 F = 3

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

2 3 − 1 4 3 1 0 3 − 1 6 9

Matrix brought to reduced row-echelon form:

1 0 0 1 0 0 0 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 2 D = 1 , 2 F =

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
2 − 1 3 3 6 , 3 4 1 0 − 1 9

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L =
1 3 1 3 1 0 1 , 0 − 1 − 1 1 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 2 1 3 , 0 1 1 − 1 0

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 0 0 0 − 1 3 0 1 0 1 − 1 3 0 0 1 1 − 1
1 3 1 3 1 0 1 , 0 − 1 − 1 1 0

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 , 0 1

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   2  Rank:   2  Nullity:   0

Parent topic: Appendix A  Archetypes

Previous topic: Archetype G   

Next topic: Archetype I   

Archetype I   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with four equations, seven variables. Consistent. Null space of coefficient matrix has dimension 4.

A system of linear equations ( Definition SLE):

x 1 + 4 x 2 − x 4 + 7 x 6 − 9 x 7 = 3 2 x 1 + 8 x 2 − x 3 + 3 x 4 + 9 x 5 − 1 3 x 6 + 7 x 7 = 9 2 x 3 − 3 x 4 − 4 x 5 + 1 2 x 6 − 8 x 7 = 1 − x 1 − 4 x 2 + 2 x 3 + 4 x 4 + 8 x 5 − 3 1 x 6 + 3 7 x 7 = 4

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = − 2 5 , x 2 = 4 , x 3 = 2 2 , x 4 = 2 9 , x 5 = 1 , x 6 = 2 , x 7 = − 3
x 1 = − 7 , x 2 = 5 , x 3 = 7 , x 4 = 1 5 , x 5 = − 4 , x 6 = 2 , x 7 = 1
x 1 = 4 , x 2 = 0 , x 3 = 2 , x 4 = 1 , x 5 = 0 , x 6 = 0 , x 7 = 0

Augmented matrix of the linear system of equations ( Definition AM):

1 4 0 − 1 0 7 − 9 3 2 8 − 1 3 9 − 1 3 7 9 0 0 2 − 3 − 4 1 2 − 8 1 − 1 − 4 2 4 8 − 3 1 3 7 4

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 4 0 0 2 1 − 3 4 0 0 1  0 1 − 3 5 2 0 0 0 1 2 − 6 6 1 0 0 0 0 0 0 0 0

Analysis of the augmented matrix ( Notation RREFA):

r = 3 D = 1 , 3 , 4 F = 2 , 5 , 6 , 7 , 8

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 x 3 x 4 x 5 x 6 x 7 = 4 0 2 1 0 0 0 + x 2 − 4 1 0 0 0 0 0 + x 5 − 2 0 − 1 − 2 1 0 0 + x 6 − 1 0 3 6 0 1 0 + x 7 3 0 − 5 − 6 0 0 1

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

x 1 + 4 x 2 − x 4 + 7 x 6 − 9 x 7 = 0 2 x 1 + 8 x 2 − x 3 + 3 x 4 + 9 x 5 − 1 3 x 6 + 7 x 7 = 0 2 x 3 − 3 x 4 − 4 x 5 + 1 2 x 6 − 8 x 7 = 0 − x 1 − 4 x 2 + 2 x 3 + 4 x 4 + 8 x 5 − 3 1 x 6 + 3 7 x 7 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0 , x 4 = 0 , x 5 = 0 , x 6 = 0 , x 7 = 0
x 1 = 3 , x 2 = 0 , x 3 = − 5 , x 4 = − 6 , x 5 = 0 , x 6 = 0 , x 7 = 1
x 1 = − 1 , x 2 = 0 , x 3 = 3 , x 4 = 6 , x 5 = 0 , x 6 = 1 , x 7 = 0
x 1 = − 2 , x 2 = 0 , x 3 = − 1 , x 4 = − 2 , x 5 = 1 , x 6 = 0 , x 7 = 0
x 1 = − 4 , x 2 = 1 , x 3 = 0 , x 4 = 0 , x 5 = 0 , x 6 = 0 , x 7 = 0
x 1 = − 4 , x 2 = 1 , x 3 = − 3 , x 4 = − 2 , x 5 = 1 , x 6 = 1 , x 7 = 1

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 4 0 0 2 1 − 3 0 0 0 1  0 1 − 3 5 0 0 0 0 1 2 − 6 6 0 0 0 0 0 0 0 0 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 3 D = 1 , 3 , 4 F = 2 , 5 , 6 , 7 , 8

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

1 4 0 − 1 0 7 − 9 2 8 − 1 3 9 − 1 3 7 0 0 2 − 3 − 4 1 2 − 8 − 1 − 4 2 4 8 − 3 1 3 7

Matrix brought to reduced row-echelon form:

1 4 0 0 2 1 − 3 0 0 1  0 1 − 3 5 0 0 0 1 2 − 6 6 0 0 0 0 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 3 D = 1 , 3 , 4 F = 2 , 5 , 6 , 7

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.
− 4 1 0 0 0 0 0 , − 2 0 − 1 − 2 1 0 0 , − 1 0 3 6 0 1 0 , 3 0 − 5 − 6 0 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
1 2 0 − 1 , 0 − 1 2 2 , − 1 3 − 3 4

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 − 1 2 3 1 − 1 3 3 1 7 3 1
− 7 3 1 0 0 1 , 1 3 3 1 0 1 0 , 1 2 3 1 1 0 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 0 − 3 1 7 , 0 1 0 1 2 7 , 0 0 1 1 3 7

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 4 0 0 2 1 − 3 , 0 0 1 0 1 − 3 5 , 0 0 0 1 2 − 6 6

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   7  Rank:   3  Nullity:   4

Parent topic: Appendix A  Archetypes

Previous topic: Archetype H   

Next topic: Archetype J   

Archetype J   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  System with six equations, nine variables. Consistent. Null space of coefficient matrix has dimension 5.

A system of linear equations ( Definition SLE):

x 1 + 2 x 2 − 2 x 3 + 9 x 4 + 3 x 5 − 5 x 6 − 2 x 7 + x 8 + 2 7 x 9 = − 5 2 x 1 + 4 x 2 + 3 x 3 + 4 x 4 − x 5 + 4 x 6 + 1 0 x 7 + 2 x 8 − 2 3 x 9 = 1 8 x 1 + 2 x 2 + x 3 + 3 x 4 + x 5 + x 6 + 5 x 7 + 2 x 8 − 7 x 9 = 6 2 x 1 + 4 x 2 + 3 x 3 + 4 x 4 − 7 x 5 + 2 x 6 + 4 x 7 − 1 1 x 9 = 2 0 x 1 + 2 x 2 + 5 x 4 + 2 x 5 − 4 x 6 + 3 x 7 + 8 x 8 + 1 3 x 9 = − 4 − 3 x 1 − 6 x 2 − x 3 − 1 3 x 4 + 2 x 5 − 5 x 6 − 4 x 7 + 1 3 x 8 + 1 0 x 9 = − 2 9

Some solutions to the system of linear equations (not necessarily exhaustive):
x 1 = 6 , x 2 = 0 , x 3 = − 1 , x 4 = 0 , x 5 = − 1 , x 6 = 2 , x 7 = 0 , x 8 = 0 , x 9 = 0
x 1 = 4 , x 2 = 1 , x 3 = − 1 , x 4 = 0 , x 5 = − 1 , x 6 = 2 , x 7 = 0 , x 8 = 0 , x 9 = 0
x 1 = − 1 7 , x 2 = 7 , x 3 = 3 , x 4 = 2 , x 5 = − 1 , x 6 = 1 4 , x 7 = − 1 , x 8 = 3 , x 9 = 2
x 1 = − 1 1 , x 2 = − 6 , x 3 = 1 , x 4 = 5 , x 5 = − 4 , x 6 = 7 , x 7 = 3 , x 8 = 1 , x 9 = 1

Augmented matrix of the linear system of equations ( Definition AM):

1 2 − 2 9 3 − 5 − 2 1 2 7 − 5 2 4 3 4 − 1 4 1 0 2 − 2 3 1 8 1 2 1 3 1 1 5 2 − 7 6 2 4 3 4 − 7 2 4 0 − 1 1 2 0 1 2 0 5 2 − 4 3 8 1 3 − 4 − 3 − 6 − 1 − 1 3 2 − 5 − 4 1 3 1 0 − 2 9

Matrix in reduced row-echelon form, row-equivalent to augmented matrix:

1 2 0 5 0 0 1 − 2 3 6 0 0 1  − 2 0 0 3 5 − 6 − 1 0 0 0 0 1 0 1 1 − 1 − 1 0 0 0 0 0 1  0 − 2 − 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Analysis of the augmented matrix ( Notation RREFA):

r = 4 D = 1 , 3 , 5 , 6 F = 2 , 4 , 7 , 8 , 9 , 1 0

Vector form of the solution set to the system of equations (Theorem VFSLS). Notice the relationship between the free variables and the set F above. Also, notice the pattern of 0’s and 1’s in the entries of the vectors corresponding to elements of the set F for the larger examples.
x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 = 6 0 − 1 0 − 1 2 0 0 0 + x 2 − 2 1 0 0 0 0 0 0 0 + x 4 − 5 0 2 1 0 0 0 0 0 + x 7 − 1 0 − 3 0 − 1 0 1 0 0 + x 8 2 0 − 5 0 − 1 2 0 1 0 + x 9 − 3 0 6 0 1 3 0 0 1

Given a system of equations we can always build a new, related, homogeneous system ( Definition HS) by converting the constant terms to zeros and retaining the coefficients of the variables. Properties of this new system will have precise relationships with various properties of the original system.

x 1 + 2 x 2 − 2 x 3 + 9 x 4 + 3 x 5 − 5 x 6 − 2 x 7 + x 8 + 2 7 x 9 = 0 2 x 1 + 4 x 2 + 3 x 3 + 4 x 4 − x 5 + 4 x 6 + 1 0 x 7 + 2 x 8 − 2 3 x 9 = 0 x 1 + 2 x 2 + x 3 + 3 x 4 + x 5 + x 6 + 5 x 7 + 2 x 8 − 7 x 9 = 0 2 x 1 + 4 x 2 + 3 x 3 + 4 x 4 − 7 x 5 + 2 x 6 + 4 x 7 − 1 1 x 9 = 0 x 1 + 2 x 2 + + 5 x 4 + 2 x 5 − 4 x 6 + 3 x 7 + 8 x 8 + 1 3 x 9 = 0 − 3 x 1 − 6 x 2 − x 3 − 1 3 x 4 + 2 x 5 − 5 x 6 − 4 x 7 + 1 3 x 8 + 1 0 x 9 = 0

Some solutions to the associated homogenous system of linear equations (not necessarily exhaustive):
x 1 = 0 , x 2 = 0 , x 3 = 0 , x 4 = 0 , x 5 = 0 , x 6 = 0 , x 7 = 0 , x 8 = 0 , x 9 = 0
x 1 = − 2 , x 2 = 1 , x 3 = 0 , x 4 = 0 , x 5 = 0 , x 6 = 0 , x 7 = 0 , x 8 = 0 , x 9 = 0
x 1 = − 2 3 , x 2 = 7 , x 3 = 4 , x 4 = 2 , x 5 = 0 , x 6 = 1 2 , x 7 = − 1 , x 8 = 3 , x 9 = 2
x 1 = − 1 7 , x 2 = − 6 , x 3 = 2 , x 4 = 5 , x 5 = − 3 , x 6 = 5 , x 7 = 3 , x 8 = 1 , x 9 = 1

Form the augmented matrix of the homogenous linear system, and use row operations to convert to reduced row-echelon form. Notice how the entries of the final column remain zeros:

1 2 0 5 0 0 1 − 2 3 0 0 0 1  − 2 0 0 3 5 − 6 0 0 0 0 0 1 0 1 1 − 1 0 0 0 0 0 0 1  0 − 2 − 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Analysis of the augmented matrix for the homogenous system ( Notation RREFA). Notice the slight variation for the same analysis of the original system only when the original system was consistent:

r = 4 D = 1 , 3 , 5 , 6 F = 2 , 4 , 7 , 8 , 9 , 1 0

Coefficient matrix of original system of equations, and of associated homogenous system. This matrix will be the subject of further analysis, rather than the systems of equations.

1 2 − 2 9 3 − 5 − 2 1 2 7 2 4 3 4 − 1 4 1 0 2 − 2 3 1 2 1 3 1 1 5 2 − 7 2 4 3 4 − 7 2 4 0 − 1 1 1 2 0 5 2 − 4 3 8 1 3 − 3 − 6 − 1 − 1 3 2 − 5 − 4 1 3 1 0

Matrix brought to reduced row-echelon form:

1 2 0 5 0 0 1 − 2 3 0 0 1  − 2 0 0 3 5 − 6 0 0 0 0 1 0 1 1 − 1 0 0 0 0 0 1  0 − 2 − 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 4 D = 1 , 3 , 5 , 6 F = 2 , 4 , 7 , 8 , 9

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.
− 2 1 0 0 0 0 0 0 0 , − 5 0 2 1 0 0 0 0 0 , − 1 0 − 3 0 − 1 0 1 0 0 , 2 0 − 5 0 − 1 2 0 1 0 , − 3 0 6 0 1 3 0 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
1 2 1 2 1 − 3 , − 2 3 1 3 0 − 1 , 3 − 1 1 − 7 2 2 , − 5 4 1 2 − 4 − 5

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 0 1 8 6 1 3 1 5 1 1 3 1 − 1 8 8 1 3 1 7 7 1 3 1 0 1 − 2 7 2 1 3 1 − 4 5 1 3 1 5 8 1 3 1 − 1 4 1 3 1
− 7 7 1 3 1 1 4 1 3 1 0 0 0 1 , 1 8 8 1 3 1 − 5 8 1 3 1 0 0 1 0 , − 5 1 1 3 1 4 5 1 3 1 0 1 0 0 , − 1 8 6 1 3 1 2 7 2 1 3 1 1 0 0 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 0 0 − 1 − 2 9 7 , 0 1 0 0 − 1 1 2 − 9 4 7 , 0 0 1 0 1 0 2 2 , 0 0 0 1 3 2 3

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 2 0 5 0 0 1 − 2 3 , 0 0 1 − 2 0 0 3 5 − 6 , 0 0 0 0 1 0 1 1 − 1 , 0 0 0 0 0 1 0 − 2 − 3

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   9  Rank:   4  Nullity:   5

Parent topic: Appendix A  Archetypes

Previous topic: Archetype I   

Next topic: Archetype K   

Archetype K   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Square matrix of size 5. Nonsingular. 3 distinct eigenvalues, 2 of multiplicity 2.

A matrix:

1 0 1 8 2 4 2 4 − 1 2 1 2 − 2 − 6 0 − 1 8 − 3 0 − 2 1 − 2 3 − 3 0 3 9 2 7 3 0 3 6 3 7 − 3 0 1 8 2 4 3 0 3 0 − 2 0

Matrix brought to reduced row-echelon form:

1 0 0 0 0 0 1  0 0 0 0 0 1 0 0 0 0 0 1  0 0 0 0 0 1

Analysis of the row-reduced matrix ( Notation RREFA):

r = 5 D = 1 , 2 , 3 , 4 , 5 F =

Matrix (coefficient matrix) is nonsingular or singular? ( Theorem NMRRI) at the same time, examine the size of the set F above.Notice that this property does not apply to matrices that are not square.
Nonsingular.

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
1 0 1 2 − 3 0 2 7 1 8 , 1 8 − 2 − 2 1 3 0 2 4 , 2 4 − 6 − 2 3 3 6 3 0 , 2 4 0 − 3 0 3 7 3 0 , − 1 2 − 1 8 3 9 − 3 0 − 2 0

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L =
1 0 0 0 0 , 0 1 0 0 0 , 0 0 1 0 0 , 0 0 0 1 0 , 0 0 0 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 0 0 0 , 0 1 0 0 0 , 0 0 1 0 0 , 0 0 0 1 0 , 0 0 0 0 1

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 0 0 0 , 0 1 0 0 0 , 0 0 1 0 0 , 0 0 0 1 0 , 0 0 0 0 1

Inverse matrix, if it exists. The inverse is not defined for matrices that are not square, and if the matrix is square, then the matrix must be nonsingular. ( Definition MI, Theorem NI)
1 − 9 4 − 3 2 3 − 6 2 1 2 4 3 4 2 1 2 9 − 9 − 1 5 − 2 1 2 − 1 1 − 1 5 3 9 2 9 1 5 4 9 2 1 0 − 1 5 9 2 3 4 3 2 6 − 1 9 2

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   5  Rank:   5  Nullity:   0

Determinant of the matrix, which is only defined for square matrices. The matrix is nonsingular if and only if the determinant is nonzero ( Theorem SMZD). (Product of all eigenvalues?)
 Determinant  = 1 6

Eigenvalues, and bases for eigenspaces. (Definition EEM,Definition EM)

λ = − 2 ℰ K − 2 = 2 − 2 1 0 1 , − 1 2 − 2 1 0 λ = 1 ℰ K 1 = 4 − 1 0 7 0 2 , − 4 1 8 − 1 7 5 0 λ = 4 ℰ K 4 = 1 − 1 0 1 1

Geometric and algebraic multiplicities. (Definition GMEDefinition AME)

γ K − 2 = 2 α K − 2 = 2 γ K 1 = 2 α K 1 = 2 γ K 4 = 1 α K 4 = 1

Diagonalizable? (Definition DZM)
Yes, full eigenspaces, Theorem DMFE.

The diagonalization. (Theorem DC)

− 4 − 3 − 4 − 6 7 − 7 − 5 − 6 − 8 1 0 1 − 1 − 1 1 − 3 1 0 0 1 − 2 2 5 6 4 0 1 0 1 8 2 4 2 4 − 1 2 1 2 − 2 − 6 0 − 1 8 − 3 0 − 2 1 − 2 3 − 3 0 3 9 2 7 3 0 3 6 3 7 − 3 0 1 8 2 4 3 0 3 0 − 2 0 2 − 1 4 − 4 1 − 2 2 − 1 0 1 8 − 1 1 − 2 7 − 1 7 0 0 1 0 5 1 1 0 2 0 1 = − 2 0 0 0 0 0 − 2 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 4

Parent topic: Appendix A  Archetypes

Previous topic: Archetype J   

Next topic: Archetype L   

Archetype L   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Square matrix of size 5. Singular, nullity 2. 2 distinct eigenvalues, each of “high” multiplicity.

A matrix:

− 2 − 1 − 2 − 4 4 − 6 − 5 − 4 − 4 6 1 0 7 7 1 0 − 1 3 − 7 − 5 − 6 − 9 1 0 − 4 − 3 − 4 − 6 6

Matrix brought to reduced row-echelon form:

1 0 0 1 − 2 0 1  0 − 2 2 0 0 1 2 − 1 0 0 0 0 0 0 0 0 0 0

Analysis of the row-reduced matrix ( Notation RREFA):

r = 5 D = 1 , 2 , 3 F = 4 , 5

Matrix (coefficient matrix) is nonsingular or singular? ( Theorem NMRRI) at the same time, examine the size of the set F above.Notice that this property does not apply to matrices that are not square.
Singular.

This is the null space of the matrix. The set of vectors used in the span construction is a linearly independent set of column vectors that spans the null space of the matrix ( Theorem SSNS, Theorem BNS). Solve the homogenous system with this matrix as the coefficient matrix and write the solutions in vector form (Theorem VFSLS) to see these vectors arise.
− 1 2 − 2 1 0 , 2 − 2 1 0 1

Column space of the matrix, expressed as the span of a set of linearly independent vectors that are also columns of the matrix. These columns have indices that form the set D above. ( Theorem BCS)
− 2 − 6 1 0 − 7 − 4 , − 1 − 5 7 − 5 − 3 , − 2 − 4 7 − 6 − 4

The column space of the matrix, as it arises from the extended echelon form of the matrix. The matrix L is computed as described in Definition EEF. This is followed by the column space described by a set of linearly independent vectors that span the null space of L , computed as according to Theorem FS and Theorem BNS. When r = m , the matrix L has no rows and the column space is all of ℂ m .
L = 1 0 − 2 − 6 5 0 1 4 1 0 − 9
− 5 9 0 0 1 , 6 − 1 0 0 1 0 , 2 − 4 1 0 0

Column space of the matrix, expressed as the span of a set of linearly independent vectors. These vectors are computed by row-reducing the transpose of the matrix into reduced row-echelon form, tossing out the zero rows, and writing the remaining nonzero rows as column vectors. By Theorem CSRST and Theorem BRS, and in the style of Example CSROI, this yields a linearly independent set of vectors that span the column space.
1 0 0 9 4 5 2 , 0 1 0 5 4 3 2 , 0 0 1 1 2 1

Row space of the matrix, expressed as a span of a set of linearly independent vectors, obtained from the nonzero rows of the equivalent matrix in reduced row-echelon form. ( Theorem BRS)
1 0 0 1 − 2 , 0 1 0 − 2 2 , 0 0 1 2 − 1

Inverse matrix, if it exists. The inverse is not defined for matrices that are not square, and if the matrix is square, then the matrix must be nonsingular. ( Definition MI, Theorem NI)

Subspace dimensions associated with the matrix. (Definition NOM, Definition ROM) Verify Theorem RPNC

 Matrix columns:   5  Rank:   3  Nullity:   2

Determinant of the matrix, which is only defined for square matrices. The matrix is nonsingular if and only if the determinant is nonzero ( Theorem SMZD). (Product of all eigenvalues?)
 Determinant  = 0

Eigenvalues, and bases for eigenspaces. (Definition EEM,Definition EM)

λ = − 1 ℰ L − 1 = − 5 9 0 0 1 , 6 − 1 0 0 1 0 , 2 − 4 1 0 0 λ = 0 ℰ L 0 = 2 − 2 1 0 1 , − 1 2 − 2 1 0

Geometric and algebraic multiplicities. (Definition GMEDefinition AME)

γ L − 1 = 3 α L − 1 = 3 γ L 0 = 2 α L 0 = 2

Diagonalizable? (Definition DZM)
Yes, full eigenspaces, Theorem DMFE.

The diagonalization. (Theorem DC)

4 3 4 6 − 6 7 5 6 9 − 1 0 − 1 0 − 7 − 7 − 1 0 1 3 − 4 − 3 − 4 − 6 7 − 7 − 5 − 6 − 8 1 0 − 2 − 1 − 2 − 4 4 − 6 − 5 − 4 − 4 6 1 0 7 7 1 0 − 1 3 − 7 − 5 − 6 − 9 1 0 − 4 − 3 − 4 − 6 6 − 5 6 2 2 − 1 9 − 1 0 − 4 − 2 2 0 0 1 1 − 2 0 1 0 0 1 1 0 0 1 0 = − 1 0 0 0 0 0 − 1 0 0 0 0 0 − 1 0 0 0 0 0 0 0 0 0 0 0 0

Parent topic: Appendix A  Archetypes

Previous topic: Archetype K   

Next topic: Archetype M   

Archetype M   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Linear transformation with bigger domain than codomain, so it is guaranteed to not be injective. Happens to not be surjective.

A linear transformation: (Definition LT)

T : ℂ 5 → ℂ 3 , T x 1 x 2 x 3 x 4 x 5 = x 1 + 2 x 2 + 3 x 3 + 4 x 4 + 4 x 5 3 x 1 + x 2 + 4 x 3 − 3 x 4 + 7 x 5 x 1 − x 2 − 5 x 4 + x 5

A basis for the null space of the linear transformation: ( Definition KLT)

− 2 − 1 0 0 1 , 2 − 3 0 1 0 , − 1 − 1 1 0 0

Injective: No. ( Definition ILT)
Since the kernel is nontrivial Theorem KILT tells us that the linear transformation is not injective. Also, since the rank can not exceed 3, we are guaranteed to have a nullity of at least 2, just from checking dimensions of the domain and the codomain. In particular, verify that

T 1 2 − 1 4 5 = 3 8 2 4 − 1 6 T 0 − 3 0 5 6 = 3 8 2 4 − 1 6

This demonstration that T is not injective is constructed with the observation that

0 − 3 0 5 6 = 1 2 − 1 4 5 + − 1 − 5 1 1 1  and z = − 1 − 5 1 1 1 ∈ K T

so the vector z effectively “does nothing” in the evaluation of T .

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

1 3 1 , 2 1 − 1 , 3 4 0 , 4 − 3 − 5 , 4 7 1

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 − 4 5 , 0 1 3 5

Surjective: No. ( Definition SLT)
Notice that the range is not all of ℂ 3 since its dimension 2, not 3. In particular, verify that 3 4 5 ∉ ℛ T , by setting the output equal to this vector and seeing that the resulting system of linear equations has no solution, i.e. is inconsistent. So the preimage, T − 1 3 4 5 , is empty. This alone is sufficient to see that the linear transformation is not onto.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   5  Rank:   2  Nullity:   3

Invertible: No.
Not injective or surjective.

Matrix representation (Theorem MLTCV):

T : ℂ 5 → ℂ 3 , T x = A x , A = 1 2 3 4 4 3 1 4 − 3 7 1 − 1 0 − 5 1

Parent topic: Appendix A  Archetypes

Previous topic: Archetype L   

Next topic: Archetype N   

Archetype N   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Linear transformation with domain larger than its codomain, so it is guaranteed to not be injective. Happens to be onto.

A linear transformation: (Definition LT)

T : ℂ 5 → ℂ 3 , T x 1 x 2 x 3 x 4 x 5 = 2 x 1 + x 2 + 3 x 3 − 4 x 4 + 5 x 5 x 1 − 2 x 2 + 3 x 3 − 9 x 4 + 3 x 5 3 x 1 + 4 x 3 − 6 x 4 + 5 x 5

A basis for the null space of the linear transformation: ( Definition KLT)

1 − 1 − 2 0 1 , − 2 − 1 3 1 0

Injective: No. ( Definition ILT)
Since the kernel is nontrivial Theorem KILT tells us that the linear transformation is not injective. Also, since the rank can not exceed 3, we are guaranteed to have a nullity of at least 2, just from checking dimensions of the domain and the codomain. In particular, verify that

T − 3 1 − 2 − 3 1 = 6 1 9 6 T − 4 − 4 − 2 − 1 4 = 6 1 9 6

This demonstration that T is not injective is constructed with the observation that

− 4 − 4 − 2 − 1 4 = − 3 1 − 2 − 3 1 + − 1 − 5 0 2 3  and z = − 1 − 5 0 2 3 ∈ K T

so the vector z effectively “does nothing” in the evaluation of T .

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

2 1 3 , 1 − 2 0 , 3 3 4 , − 4 − 9 − 6 , 5 3 5

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 0 , 0 1 0 , 0 0 1

Surjective: Yes. ( Definition SLT)
Notice that the basis for the range above is the standard basis for ℂ 3 . So the range is all of ℂ 3 and thus the linear transformation is surjective.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   5  Rank:   3  Nullity:   2

Invertible: No.
Not surjective, and the relative sizes of the domain and codomain mean the linear transformation cannot be injective. ( Theorem ILTIS)

Matrix representation (Theorem MLTCV):

T : ℂ 5 → ℂ 3 , T x = A x , A = 2 1 3 − 4 5 1 − 2 3 − 9 3 3 0 4 − 6 5

Parent topic: Appendix A  Archetypes

Previous topic: Archetype M   

Next topic: Archetype O   

Archetype O   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Linear transformation with a domain smaller than the codomain, so it is guaranteed to not be onto. Happens to not be one-to-one.

A linear transformation: (Definition LT)

T : ℂ 3 → ℂ 5 , T x 1 x 2 x 3 = − x 1 + x 2 − 3 x 3 − x 1 + 2 x 2 − 4 x 3 x 1 + x 2 + x 3 2 x 1 + 3 x 2 + x 3 x 1 + 2 x 3

A basis for the null space of the linear transformation: ( Definition KLT)

− 2 1 1

Injective: No. ( Definition ILT)
Since the kernel is nontrivial Theorem KILT tells us that the linear transformation is not injective. Also, since the rank can not exceed 3, we are guaranteed to have a nullity of at least 2, just from checking dimensions of the domain and the codomain. In particular, verify that

T 5 − 1 3 = − 1 5 − 1 9 7 1 0 1 1 T 1 1 5 = − 1 5 − 1 9 7 1 0 1 1

This demonstration that T is not injective is constructed with the observation that

1 1 5 = 5 − 1 3 + − 4 2 2  and z = − 4 2 2 ∈ K T

so the vector z effectively “does nothing” in the evaluation of T .

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

− 1 − 1 1 2 1 , 1 2 1 3 0 , − 3 − 4 1 1 2

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 − 3 − 7 − 2 , 0 1 2 5 1

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   3  Rank:   2  Nullity:   1

Surjective: No. ( Definition SLT)
The dimension of the range is 2, and the codomain ( ℂ 5 ) has dimension 5. So the transformation is not onto. Notice too that since the domain ℂ 3 has dimension 3, it is impossible for the range to have a dimension greater than 3, and no matter what the actual definition of the function, it cannot possibly be onto.

To be more precise, verify that 2 3 1 1 1 ∉ ℛ T , by setting the output equal to this vector and seeing that the resulting system of linear equations has no solution, i.e. is inconsistent. So the preimage, T − 1 2 3 1 1 1 , is empty. This alone is sufficient to see that the linear transformation is not onto.

Invertible: No.
Not injective, and the relative dimensions of the domain and codomain prohibit any possibility of being surjective.

Matrix representation (Theorem MLTCV):

T : ℂ 3 → ℂ 5 , T x = A x , A = − 1 1 − 3 − 1 2 − 4 1 1 1 2 3 1 1 0 2

Parent topic: Appendix A  Archetypes

Previous topic: Archetype N   

Next topic: Archetype P   

Archetype P   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Linear transformation with a domain smaller that its codomain, so it is guaranteed to not be surjective. Happens to be injective.

A linear transformation: (Definition LT)

T : ℂ 3 → ℂ 5 , T x 1 x 2 x 3 = − x 1 + x 2 + x 3 − x 1 + 2 x 2 + 2 x 3 x 1 + x 2 + 3 x 3 2 x 1 + 3 x 2 + x 3 − 2 x 1 + x 2 + 3 x 3

A basis for the null space of the linear transformation: ( Definition KLT)

Injective: Yes. ( Definition ILT)
Since K T = 0 , Theorem KILT tells us that T is injective.

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

− 1 − 1 1 2 − 2 , 1 2 1 3 1 , 1 2 3 1 3

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 0 − 1 0 6 , 0 1 0 7 − 3 , 0 0 1 − 1 1

Surjective: No. ( Definition SLT)
The dimension of the range is 3, and the codomain ( ℂ 5 ) has dimension 5. So the transformation is not surjective. Notice too that since the domain ℂ 3 has dimension 3, it is impossible for the range to have a dimension greater than 3, and no matter what the actual definition of the function, it cannot possibly be surjective in this situation.

To be more precise, verify that 2 1 − 3 2 6 ∉ ℛ T , by setting the output equal to this vector and seeing that the resulting system of linear equations has no solution, i.e. is inconsistent. So the preimage, T − 1 2 1 − 3 2 6 , is empty. This alone is sufficient to see that the linear transformation is not onto.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   3  Rank:   3  Nullity:   0

Invertible: No.
The relative dimensions of the domain and codomain prohibit any possibility of being surjective, so apply Theorem ILTIS.

Matrix representation (Theorem MLTCV):

T : ℂ 3 → ℂ 5 , T x = A x , A = − 1 1 1 − 1 2 2 1 1 3 2 3 1 − 2 1 3

Parent topic: Appendix A  Archetypes

Previous topic: Archetype O   

Next topic: Archetype Q   

Archetype Q   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Linear transformation with equal-sized domain and codomain, so it has the potential to be invertible, but in this case is not. Neither injective nor surjective. Diagonalizable, though.

A linear transformation: (Definition LT)

T : ℂ 5 → ℂ 5 , T x 1 x 2 x 3 x 4 x 5 = − 2 x 1 + 3 x 2 + 3 x 3 − 6 x 4 + 3 x 5 − 1 6 x 1 + 9 x 2 + 1 2 x 3 − 2 8 x 4 + 2 8 x 5 − 1 9 x 1 + 7 x 2 + 1 4 x 3 − 3 2 x 4 + 3 7 x 5 − 2 1 x 1 + 9 x 2 + 1 5 x 3 − 3 5 x 4 + 3 9 x 5 − 9 x 1 + 5 x 2 + 7 x 3 − 1 6 x 4 + 1 6 x 5

A basis for the null space of the linear transformation: ( Definition KLT)

3 4 1 3 3

Injective: No. ( Definition ILT)
Since the kernel is nontrivial Theorem KILT tells us that the linear transformation is not injective. Also, since the rank can not exceed 3, we are guaranteed to have a nullity of at least 2, just from checking dimensions of the domain and the codomain. In particular, verify that

T 1 3 − 1 2 4 = 4 5 5 7 2 7 7 3 1 T 4 7 0 5 7 = 4 5 5 7 2 7 7 3 1

This demonstration that T is not injective is constructed with the observation that

4 7 0 5 7 = 1 3 − 1 2 4 + 3 4 1 3 3  and z = 3 4 1 3 3 ∈ K T

so the vector z effectively “does nothing” in the evaluation of T .

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

− 2 − 1 6 − 1 9 − 2 1 − 9 , 3 9 7 9 5 , 3 1 2 1 4 1 5 7 , − 6 − 2 8 − 3 2 − 3 5 − 1 6 , 3 2 8 3 7 3 9 1 6

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 0 0 1 , 0 1 0 0 − 1 , 0 0 1 0 − 1 , 0 0 0 1 2

Surjective: No. ( Definition SLT)
The dimension of the range is 4, and the codomain ( ℂ 5 ) has dimension 5. So ℛ T ≠ ℂ 5 and by Theorem RSLT the transformation is not surjective.

To be more precise, verify that − 1 2 3 − 1 4 ∉ ℛ T , by setting the output equal to this vector and seeing that the resulting system of linear equations has no solution, i.e. is inconsistent. So the preimage, T − 1 − 1 2 3 − 1 4 , is empty. This alone is sufficient to see that the linear transformation is not onto.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   5  Rank:   4  Nullity:   1

Invertible: No.
Neither injective nor surjective. Notice that since the domain and codomain have the same dimension, either the transformation is both onto and one-to-one (making it invertible) or else it is both not onto and not one-to-one (as in this case) by Theorem RPNDD.

Matrix representation (Theorem MLTCV):

T : ℂ 5 → ℂ 5 , T x = A x , A = − 2 3 3 − 6 3 − 1 6 9 1 2 − 2 8 2 8 − 1 9 7 1 4 − 3 2 3 7 − 2 1 9 1 5 − 3 5 3 9 − 9 5 7 − 1 6 1 6

Eigenvalues and eigenvectors ( Definition EELT, Theorem EER):

λ = − 1 ℰ T − 1 = 0 2 3 3 1 λ = 0 ℰ T 0 = 3 4 1 3 3 λ = 1 ℰ T 1 = 5 3 0 0 2 , − 3 1 0 2 0 , 1 − 1 2 0 0

Evaluate the linear transformation with each of these eigenvectors as an interesting check.

A diagonal matrix representation relative to a basis of eigenvectors, B .

B = 0 2 3 3 1 , 3 4 1 3 3 , 5 3 0 0 2 , − 3 1 0 2 0 , 1 − 1 2 0 0 M B , B T = − 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1

Parent topic: Appendix A  Archetypes

Previous topic: Archetype P   

Next topic: Archetype R   

Archetype R   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Linear transformation with equal-sized domain and codomain. Injective, surjective, invertible, diagonalizable, the works.

A linear transformation: (Definition LT)

T : ℂ 5 → ℂ 5 , T x 1 x 2 x 3 x 4 x 5 = − 6 5 x 1 + 1 2 8 x 2 + 1 0 x 3 − 2 6 2 x 4 + 4 0 x 5 3 6 x 1 − 7 3 x 2 − x 3 + 1 5 1 x 4 − 1 6 x 5 − 4 4 x 1 + 8 8 x 2 + 5 x 3 − 1 8 0 x 4 + 2 4 x 5 3 4 x 1 − 6 8 x 2 − 3 x 3 + 1 4 0 x 4 − 1 8 x 5 1 2 x 1 − 2 4 x 2 − x 3 + 4 9 x 4 − 5 x 5

A basis for the null space of the linear transformation: ( Definition KLT)

Injective: Yes. ( Definition ILT)
Since the kernel is trivial Theorem KILT tells us that the linear transformation is injective.

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

− 6 5 3 6 − 4 4 3 4 1 2 , 1 2 8 − 7 3 8 8 − 6 8 − 2 4 , 1 0 − 1 5 − 3 − 1 , − 2 6 2 1 5 1 − 1 8 0 1 4 0 4 9 , 4 0 − 1 6 2 4 − 1 8 − 5

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 0 0 0 , 0 1 0 0 0 , 0 0 1 0 0 , 0 0 0 1 0 , 0 0 0 0 1

Surjective: Yes. ( Definition SLT)
A basis for the range is the standard basis of ℂ 5 , so ℛ T = ℂ 5 and Theorem RSLT tells us T is surjective. Or, the dimension of the range is 5, and the codomain ( ℂ 5 ) has dimension 5. So the transformation is surjective.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   5  Rank:   5  Nullity:   0

Invertible: Yes.
Both injective and surjective ( Theorem ILTIS). Notice that since the domain and codomain have the same dimension, either the transformation is both injective and surjective (making it invertible, as in this case) or else it is both not injective and not surjective.

Matrix representation (Theorem MLTCV):

T : ℂ 5 → ℂ 5 , T x = A x , A = − 6 5 1 2 8 1 0 − 2 6 2 4 0 3 6 − 7 3 − 1 1 5 1 − 1 6 − 4 4 8 8 5 − 1 8 0 2 4 3 4 − 6 8 − 3 1 4 0 − 1 8 1 2 − 2 4 − 1 4 9 − 5

The inverse linear transformation ( Definition IVLT):

T − 1 : ℂ 5 → ℂ 5 , T − 1 x 1 x 2 x 3 x 4 x 5 = − 4 7 x 1 + 9 2 x 2 + x 3 − 1 8 1 x 4 − 1 4 x 5 2 7 x 1 − 5 5 x 2 + 7 2 x 3 + 2 2 1 2 x 4 + 1 1 x 5 − 3 2 x 1 + 6 4 x 2 − x 3 − 1 2 6 x 4 − 1 2 x 5 2 5 x 1 − 5 0 x 2 + 3 2 x 3 + 1 9 9 2 x 4 + 9 x 5 9 x 1 − 1 8 x 2 + 1 2 x 3 + 7 1 2 x 4 + 4 x 5

Verify that T T − 1 x = x and T T − 1 x = x , and notice that the representations of the transformation and its inverse are matrix inverses (Theorem IMR, Definition MI).

Eigenvalues and eigenvectors ( Definition EELT, Theorem EER):

λ = − 1 ℰ T − 1 = − 5 7 0 − 1 8 1 4 5 , 2 1 0 0 0 λ = 1 ℰ T 1 = − 1 0 − 5 − 6 0 1 , 2 3 1 1 0 λ = 2 ℰ T 2 = − 6 3 − 4 3 1

Evaluate the linear transformation with each of these eigenvectors as an interesting check.

A diagonal matrix representation relative to a basis of eigenvectors, B .

B = − 5 7 0 − 1 8 1 4 5 , 2 1 0 0 0 , − 1 0 − 5 − 6 0 1 , 2 3 1 1 0 , − 6 3 − 4 3 1 M B , B T = − 1 0 0 0 0 0 − 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 2

Parent topic: Appendix A  Archetypes

Previous topic: Archetype Q   

Next topic: Archetype S   

Archetype S   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Domain is column vectors, codomain is matrices. Domain is dimension 3 and codomain is dimension 4. Not injective, not surjective.

A linear transformation: (Definition LT)

T : ℂ 3 → M 2 2 , T a b c = a − b 2 a + 2 b + c 3 a + b + c − 2 a − 6 b − 2 c

A basis for the null space of the linear transformation: ( Definition KLT)

− 1 − 1 4

Injective: No. ( Definition ILT)
Since the kernel is nontrivial Theorem KILT tells us that the linear transformation is not injective. Also, since the rank can not exceed 3, we are guaranteed to have a nullity of at least 1, just from checking dimensions of the domain and the codomain. In particular, verify that

T 2 1 3 = 1 9 1 0 − 1 6 T 0 − 1 1 1 = 1 9 1 0 − 1 6

This demonstration that T is not injective is constructed with the observation that

0 − 1 1 1 = 2 1 3 + − 2 − 2 8  and z = − 2 − 2 8 ∈ K T

so the vector z effectively “does nothing” in the evaluation of T .

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

1 2 3 − 2 , − 1 2 1 − 6 , 0 1 1 − 2

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 1 2 , 0 1 1 − 2

Surjective: No. ( Definition SLT)
The dimension of the range is 2, and the codomain ( M 2 2 ) has dimension 4. So the transformation is not surjective. Notice too that since the domain ℂ 3 has dimension 3, it is impossible for the range to have a dimension greater than 3, and no matter what the actual definition of the function, it cannot possibly be surjective in this situation.

To be more precise, verify that 2 − 1 1 3 ∉ ℛ T , by setting the output of T equal to this matrix and seeing that the resulting system of linear equations has no solution, i.e. is inconsistent. So the preimage, T − 1 2 − 1 1 3 , is empty. This alone is sufficient to see that the linear transformation is not onto.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   3  Rank:   2  Nullity:   1

Invertible: No.
Not injective ( Theorem ILTIS), and the relative dimensions of the domain and codomain prohibit any possibility of being surjective.

Matrix representation (Definition MR):

B = 1 0 0 , 0 1 0 , 0 0 1 C = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1 M B , C T = 1 − 1 0 2 2 1 3 1 1 − 2 − 6 − 2

Parent topic: Appendix A  Archetypes

Previous topic: Archetype R   

Next topic: Archetype T   

Archetype T   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Domain and codomain are polynomials. Domain has dimension 5, while codomain has dimension 6. Is injective, can’t be surjective.

A linear transformation: (Definition LT)

T : P 4 → P 5 , T p ( x ) = ( x − 2 ) p ( x )

A basis for the null space of the linear transformation: ( Definition KLT)

Injective: Yes. ( Definition ILT)
Since the kernel is trivial Theorem KILT tells us that the linear transformation is injective.

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

x − 2 , x 2 − 2 x , x 3 − 2 x 2 , x 4 − 2 x 3 , x 5 − 2 x 4 , x 6 − 2 x 5

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

− 1 3 2 x 5 + 1 , − 1 1 6 x 5 + x , − 1 8 x 5 + x 2 , − 1 4 x 5 + x 3 , − 1 2 x 5 + x 4

Surjective: No. ( Definition SLT)
The dimension of the range is 5, and the codomain ( P 5 ) has dimension 6. So the transformation is not surjective. Notice too that since the domain P 4 has dimension 5, it is impossible for the range to have a dimension greater than 5, and no matter what the actual definition of the function, it cannot possibly be surjective in this situation.

To be more precise, verify that 1 + x + x 2 + x 3 + x 4 ∉ ℛ T , by setting the output equal to this vector and seeing that the resulting system of linear equations has no solution, i.e. is inconsistent. So the preimage, T − 1 1 + x + x 2 + x 3 + x 4 , is nonempty. This alone is sufficient to see that the linear transformation is not onto.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   5  Rank:   5  Nullity:   0

Invertible: No.
The relative dimensions of the domain and codomain prohibit any possibility of being surjective, so apply Theorem ILTIS.

Matrix representation (Definition MR):

B = 1 , x , x 2 , x 3 , x 4 C = 1 , x , x 2 , x 3 , x 4 , x 5 M B , C T = − 2 0 0 0 0 1 − 2 0 0 0 0 1 − 2 0 0 0 0 1 − 2 0 0 0 0 1 − 2 0 0 0 0 1

Parent topic: Appendix A  Archetypes

Previous topic: Archetype S   

Next topic: Archetype U   

Archetype U   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Domain is matrices, codomain is column vectors. Domain has dimension 6, while codomain has dimension 4. Can’t be injective, is surjective.

A linear transformation: (Definition LT)

T : M 2 3 → ℂ 4 , T a b c d e f = a + 2 b + 1 2 c − 3 d + e + 6 f 2 a − b − c + d − 1 1 f a + b + 7 c + 2 d + e − 3 f a + 2 b + 1 2 c + 5 e − 5 f

A basis for the null space of the linear transformation: ( Definition KLT)

3 − 4 0 1 2 1 , − 2 − 5 1 0 0 0

Injective: No. ( Definition ILT)
Since the kernel is nontrivial Theorem KILT tells us that the linear transformation is not injective. Also, since the rank can not exceed 4, we are guaranteed to have a nullity of at least 2, just from checking dimensions of the domain and the codomain. In particular, verify that

T 1 1 0 − 2 3 − 1 1 = − 7 − 1 4 − 1 − 1 3 T 5 − 3 − 1 5 3 3 = − 7 − 1 4 − 1 − 1 3

This demonstration that T is not injective is constructed with the observation that

5 − 3 − 1 5 3 3 = 1 1 0 − 2 3 − 1 1 + 4 − 1 3 1 2 4 2  and z = 4 − 1 3 1 2 4 2 ∈ K T

so the vector z effectively “does nothing” in the evaluation of T .

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

1 2 1 1 , 2 − 1 1 2 , 1 2 − 1 7 1 2 , − 3 1 2 0 , 1 0 1 5 , 6 − 1 1 − 3 − 5

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

Surjective: Yes. ( Definition SLT)
A basis for the range is the standard basis of ℂ 4 , so ℛ T = ℂ 4 and Theorem RSLT tells us T is surjective. Or, the dimension of the range is 4, and the codomain ( ℂ 4 ) has dimension 4. So the transformation is surjective.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   6  Rank:   4  Nullity:   2

Invertible: No.
The relative sizes of the domain and codomain mean the linear transformation cannot be injective. ( Theorem ILTIS)

Matrix representation (Definition MR):

B = 1 0 0 0 0 0 , 0 1 0 0 0 0 , 0 0 1 0 0 0 , 0 0 0 1 0 0 , 0 0 0 0 1 0 , 0 0 0 0 0 1 C = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1 M B , C T = 1 2 1 2 − 3 1 6 2 − 1 − 1 1 0 − 1 1 1 1 7 2 1 − 3 1 2 1 2 0 5 − 5

Parent topic: Appendix A  Archetypes

Previous topic: Archetype T   

Next topic: Archetype V   

Archetype V   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Domain is polynomials, codomain is matrices. Domain and codomain both have dimension 4. Injective, surjective, invertible. Square matrix representation, but domain and codomain are unequal, so no eigenvalue information.

A linear transformation: (Definition LT)

T : P 3 → M 2 2 , T a + b x + c x 2 + d x 3 = a + b a − 2 c d b − d

A basis for the null space of the linear transformation: ( Definition KLT)

Injective: Yes. ( Definition ILT)
Since the kernel is trivial Theorem KILT tells us that the linear transformation is injective.

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

1 1 0 0 , 1 0 0 1 , 0 − 2 0 0 , 0 0 1 − 1

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1

Surjective: Yes. ( Definition SLT)
A basis for the range is the standard basis of M 2 2 , so ℛ T = M 2 2 and Theorem RSLT tells us T is surjective. Or, the dimension of the range is 4, and the codomain ( M 2 2 ) has dimension 4. So the transformation is surjective.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   4  Rank:   4  Nullity:   0

Invertible: Yes.
Both injective and surjective ( Theorem ILTIS). Notice that since the domain and codomain have the same dimension, either the transformation is both injective and surjective (making it invertible, as in this case) or else it is both not injective and not surjective.

Matrix representation (Definition MR):

B = 1 , x , x 2 , x 3 C = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1 M B , C T = 1 1 0 0 1 0 − 2 0 0 0 0 1 0 1 0 − 1

Since invertible, the inverse linear transformation. ( Definition IVLT)

T − 1 : M 2 2 → P 3 , T − 1 a b c d = ( a − c − d ) + ( c + d ) x + 1 2 ( a − b − c − d ) x 2 + c x 3

Parent topic: Appendix A  Archetypes

Previous topic: Archetype U   

Next topic: Archetype W   

Archetype W   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Domain is polynomials, codomain is polynomials. Domain and codomain both have dimension 3. Injective, surjective, invertible, 3 distinct eigenvalues, diagonalizable.

A linear transformation: (Definition LT)

T : P 2 → P 2 , T a + b x + c x 2 = 1 9 a + 6 b − 4 c + − 2 4 a − 7 b + 4 c + 3 6 a + 1 2 b − 9 c

A basis for the null space of the linear transformation: ( Definition KLT)

Injective: Yes. ( Definition ILT)
Since the kernel is trivial Theorem KILT tells us that the linear transformation is injective.

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

1 9 − 2 4 x + 3 6 x 2 , 6 − 7 x + 1 2 x 2 , − 4 + 4 x − 9 x 2

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 , x , x 2

Surjective: Yes. ( Definition SLT)
A basis for the range is the standard basis of ℂ 5 , so ℛ T = ℂ 5 and Theorem RSLT tells us T is surjective. Or, the dimension of the range is 5, and the codomain ( ℂ 5 ) has dimension 5. So the transformation is surjective.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   3  Rank:   3  Nullity:   0

Invertible: Yes.
Both injective and surjective ( Theorem ILTIS). Notice that since the domain and codomain have the same dimension, either the transformation is both injective and surjective (making it invertible, as in this case) or else it is both not injective and not surjective.

Matrix representation (Definition MR):

B = 1 , x , x 2 C = 1 , x , x 2 M B , C T = 1 9 6 − 4 − 2 4 − 7 4 3 6 1 2 − 9

Since invertible, the inverse linear transformation. ( Definition IVLT)

T − 1 : P 2 → P 2 , T − 1 a + b x + c x 2 = ( − 5 a − 2 b + 4 3 c ) + ( 2 4 a + 9 b − 2 0 3 c ) x + ( 1 2 a + 4 b − 1 1 3 c ) x 2

Eigenvalues and eigenvectors ( Definition EELT, Theorem EER):

λ = − 1 ℰ T − 1 = 2 x + 3 x 2 λ = 1 ℰ T 1 = − 1 + 3 x λ = 3 ℰ T 3 = 1 − 2 x + x 2

Evaluate the linear transformation with each of these eigenvectors as an interesting check.

A diagonal matrix representation relative to a basis of eigenvectors, B .

B = 2 x + 3 x 2 , − 1 + 3 x , 1 − 2 x + x 2 M B , B T = − 1 0 0 0 1 0 0 0 3

Parent topic: Appendix A  Archetypes

Previous topic: Archetype V   

Next topic: Archetype X   

Archetype X   

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

Summary  Domain and codomain are square matrices. Domain and codomain both have dimension 4. Not injective, not surjective, not invertible, 3 distinct eigenvalues, diagonalizable.

A linear transformation: (Definition LT)

T : M 2 2 → M 2 2 , T a b c d = − 2 a + 1 5 b + 3 c + 2 7 d 1 0 b + 6 c + 1 8 d a − 5 b − 9 d − a − 4 b − 5 c − 8 d

A basis for the null space of the linear transformation: ( Definition KLT)

− 6 − 3 2 1

Injective: No. ( Definition ILT)
Since the kernel is nontrivial Theorem KILT tells us that the linear transformation is not injective. In particular, verify that

T − 2 0 1 − 4 = 1 1 5 7 8 − 3 8 − 3 5 T 4 3 − 1 3 = 1 1 5 7 8 − 3 8 − 3 5

This demonstration that T is not injective is constructed with the observation that

4 3 − 1 3 = − 2 0 1 − 4 + 6 3 − 2 − 1  and z = 6 3 − 2 − 1 ∈ K T

so the vector z effectively “does nothing” in the evaluation of T .

A basis for the range of the linear transformation: ( Definition RLT)
Evaluate the linear transformation on a standard basis to get a spanning set for the range ( Theorem SSRLT):

− 2 0 1 − 1 , 1 5 1 0 − 5 − 4 , 3 6 0 − 5 , 2 7 1 8 − 9 − 8

If the linear transformation is injective, then the set above is guaranteed to be linearly independent ( Theorem ILTLI). This spanning set may be converted to a “nice” basis, by making the vectors the rows of a matrix (perhaps after using a vector reperesentation), row-reducing, and retaining the nonzero rows ( Theorem BRS), and perhaps un-coordinatizing. A basis for the range is:

1 0 − 1 2 0 , 0 1 1 4 0 , 0 0 0 1

Surjective: No. ( Definition SLT)
The dimension of the range is 3, and the codomain ( M 2 2 ) has dimension 5. So ℛ T ≠ M 2 2 and by Theorem RSLT the transformation is not surjective.

To be more precise, verify that 2 4 3 1 ∉ ℛ T , by setting the output of T equal to this matrix and seeing that the resulting system of linear equations has no solution, i.e. is inconsistent. So the preimage, T − 1 2 4 3 1 , is empty. This alone is sufficient to see that the linear transformation is not onto.

Subspace dimensions associated with the linear transformation. Examine parallels with earlier results for matrices. Verify Theorem RPNDD.

 Domain dimension:   4  Rank:   3  Nullity:   1

Invertible: No.
Neither injective nor surjective ( Theorem ILTIS). Notice that since the domain and codomain have the same dimension, either the transformation is both injective and surjective or else it is both not injective and not surjective (making it not invertible, as in this case).

Matrix representation (Definition MR):

B = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1 C = 1 0 0 0 , 0 1 0 0 , 0 0 1 0 , 0 0 0 1 M B , C T = − 2 1 5 3 2 7 0 1 0 6 1 8 1 − 5 0 − 9 − 1 − 4 − 5 − 8

Eigenvalues and eigenvectors ( Definition EELT, Theorem EER):

λ = 0 ℰ T 0 = − 6 − 3 2 1 λ = 1 ℰ T 1 = − 7 − 2 3 0 , − 1 − 2 0 1 λ = 3 ℰ T 3 = − 3 − 2 1 1

Evaluate the linear transformation with each of these eigenvectors as an interesting check.

A diagonal matrix representation relative to a basis of eigenvectors, B .

B = − 6 − 3 2 1 , − 7 − 2 3 0 , − 1 − 2 0 1 , − 3 − 2 1 1 M B , B T = 0 0 0 0 0 1 0 0 0 0 3 0 0 0 0 3

Parent topic: Appendix A  Archetypes

Previous topic: Archetype W   

Next topic: Appendix GFDL  GNU Free Documentation License

Chapter 12. Appendix GFDL  GNU Free Documentation License

The purpose of this License is to make a manual, textbook, or other functional and useful document “free” in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of “copyleft”, which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The “Document”, below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as “you”. You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A “Modified Version” of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A “Secondary Section” is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The “Invariant Sections” are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The “Cover Texts” are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A “Transparent” copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not “Transparent” is called “Opaque”.

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The “Title Page” means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, “Title Page” means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text.

A section “Entitled XYZ” means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as “Acknowledgements”, “Dedications”, “Endorsements”, or “History”.) To “Preserve the Title” of such a section when you modify the Document means that it remains a section “Entitled XYZ” according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled “Endorsements”, provided it contains nothing but endorsements of your Modified Version by various parties–for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled “History” in the various original documents, forming one section Entitled “History”; likewise combine any sections Entitled “Acknowledgements”, and any sections Entitled “Dedications”. You must delete all sections Entitled “Endorsements”.

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an “aggregate” if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled “Acknowledgements”, “Dedications”, or “History”, the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License “or any later version” applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

To use this License in a document you have written, include a copy of the License in the document and put the following copyright and license notices just after the title page:

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.

Appendix GFDL  GNU Free Documentation License

Parent topic: Part C  Core

Previous topic: Archetype X   

Next topic: Appendix GFDL  GNU Free Documentation License

Parent topic: Appendix GFDL  GNU Free Documentation License

Previous topic: Appendix GFDL  GNU Free Documentation License

Next topic: Appendix GFDL  GNU Free Documentation License

Parent topic: Appendix GFDL  GNU Free Documentation License

Previous topic: Appendix GFDL  GNU Free Documentation License

Next topic: Part T  Topics

Part II. Part T  Topics

Section T  Trace

Section HP  Hadamard Product

Section VM  Vandermonde Matrix

Section PSM  Positive Semi-definite Matrices

Chapter MD  Matrix Decompositions

Previous topic: Appendix GFDL  GNU Free Documentation License

Next topic: Section T  Trace

Chapter 14. Section T  Trace

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

This section contributed by Andy Zimmer.

The matrix trace is a function that sends square matrices to scalars. In some ways it is reminiscent of the determinant. And like the determinant, it has many useful and surprising properties.

Definition  T
Trace
Suppose A is a square matrix of size n . Then the trace of A , t A , is the sum of the diagonal entries of A . Symbolically,

t A = ∑ i = 1 n A i i

(This definition contains Notation T.) △

The next three proofs make for excellent practice. In some books they would be left as exercises for the reader as they are all “trivial” in the sense they do not rely on anything but the definition of the matrix trace.

Theorem  TL
Trace is Linear
Suppose A and B are square matrices of size n . Then t A + B = t A + t B . Furthermore, if α ∈ ℂ , then t α A = α t A . □

Proof   These properties are exactly those required for a linear transformation. To prove these results we just manipulate sums,

t A + B = ∑ k = 1 n A + B i i   Definition T = ∑ i = 1 n A i i + B i i   Definition MA = ∑ i = 1 n A i i + ∑ i = 1 n B i i   Property CACN = t A + t B   Definition T

The second part is as straightforward as the first,

t α A = ∑ i = 1 n α A i i   Definition T = ∑ i = 1 n α A i i   Definition MSM = α ∑ i = 1 n A i i   Property DCN = α t A   Definition T

■

Theorem  TSRM
Trace is Symmetric with Respect to Multiplication
Suppose A and B are square matrices of size n . Then t A B = t B A . □

Proof  

t A B = ∑ k = 1 n A B k k   Definition T = ∑ k = 1 n ∑ ℓ = 1 n A k ℓ B ℓ k   Theorem EMP = ∑ ℓ = 1 n ∑ k = 1 n A k ℓ B ℓ k   Property CACN = ∑ ℓ = 1 n ∑ k = 1 n B ℓ k A k ℓ   Property CMCN = ∑ ℓ = 1 n B A ℓ ℓ   Theorem EMP = t B A   Definition T

■

Theorem  TIST
Trace is Invariant Under Similarity Transformations
Suppose A and S are square matrices of size n and S is invertible. Then t S − 1 A S = t A . □

Proof   Invariant means constant under some operation. In this case the operation is a similarity transformation. A lengthy exercise (but possibly a educational one) would be to prove this result without referencing Theorem TSRM. But here we will,

t S − 1 A S = t S − 1 A S   Theorem MMA = t S S − 1 A   Theorem TSRM = t S S − 1 A   Theorem MMA = t A   Definition MI

■

Now we could define the trace of a linear transformation as the trace of any matrix representation of the transformation. Would this definition be well-defined? That is, will two different representations of the same linear transformation always have the same trace? Why? (Think Theorem SCB.) We will now prove one of the most interesting and surprising results about the trace.

Theorem  TSE
Trace is the Sum of the Eigenvalues
Suppose that A is a square matrix of size n with distinct eigenvalues λ 1 , λ 2 , λ 3 , … , λ k . Then

t A = ∑ i = 1 k α A λ i λ i

□

Proof   It is amazing that the eigenvalues would have anything to do with the sum of the diagonal entries. Our proof will rely on double counting. We will demonstrate two different ways of counting the same thing therefore proving equality. Our object of interest is the coefficient of x n − 1 in the characteristic polynomial of A (Definition CP), which will be denoted α n − 1 . From the proof of Theorem NEM we have,

p A x = ( − 1 ) n ( x − λ 1 ) α A λ 1 ( x − λ 2 ) α A λ 2 ( x − λ 3 ) α A λ 3 ⋯ ( x − λ k ) α A λ k

First we want to prove that α n − 1 is equal to ( − 1 ) n + 1 ∑ i = 1 k α A λ i λ i and to do this we will use a straight forward counting argument. Induction can be used here as well (try it), but the intuitive approach is a much stronger technique. Let’s imagine creating each term one by one from the extended product. How do we do this? From each ( x − λ i ) we pick either a x or a λ i . But we are only interested in the terms that result in x to the power n − 1 . As ∑ i = 1 k α A λ i = n , we have n factors of the form ( x − λ i ) . Then to get terms with x n − 1 we need to pick x ’s in every ( x − λ i ) , except one. Since we have n linear factors there are n ways to do this, namely each eigenvalue represented as many times as it’s algebraic multiplicity. Now we have to take into account the sign of each term. As we pick n − 1 x ’s and one λ i (which has a negative sign in the linear factor) we get a factor of − 1 . Then we have to take into account the ( − 1 ) n in the characteristic polynomial. Thus α n − 1 is the sum of these terms,

α n − 1 = ( − 1 ) n + 1 ∑ i = 1 k α A λ i λ i

Now we will now show that α n − 1 is also equal to ( − 1 ) n − 1 t A . For this we will proceed by induction on the size of A . If A is a 1 × 1 square matrix then p A x = det A − x I n = A 1 1 − x and ( − 1 ) 1 − 1 t A = A 1 1 . With our base case in hand let’s assume A is a square matrix of size n . By Definition CP

p A x = det A − x I n = A − x I n 1 1 det ( A − x I n ) 1 | 1 − A − x I n 1 2 det ( A − x I n ) 1 | 2 + A − x I n 1 3 det ( A − x I n n ) 1 | 3 − ⋯ + ( − 1 ) n + 1 A − x I n 1 n det ( A − x I n ) 1 | n

First let’s consider the maximum degree of A − x I n 1 i det ( A − x I n ) 1 | i when i ≠ 1 . For polynomials, the degree of f , denoted d ( f ) , is the highest power of x in the expression f ( x ) . A well known result of this definition is: if f ( x ) = g ( x ) h ( x ) then d ( f ) = d ( g ) + d ( h ) (can you prove this?). Now A − x I n 1 i has degree zero when i ≠ 1 . Furthermore ( A − x I n ) 1 | i has n − 1 rows, one of which has all of its entries of degree zero, since column i is removed. The other n − 2 rows have one entry with degree one and the remainder of degree zero. Then by Exercise T.T30, the maximum degree of A − x I n 1 i det ( A − x I n ) 1 | i is n − 2 . So these terms will not affect the coefficient of x n − 1 . Now we are free to focus all of our attention on the term A − x I n 1 1 det ( A − x I n ) 1 | 1 . As A 1 | 1 is a ( n − 1 ) × ( n − 1 ) matrix the induction hypothesis tells us that det ( A − x I n ) 1 | 1 has a coefficient of ( − 1 ) n − 2 t A 1 | 1 for x n − 2 . We also note that the proof of Theorem NEM tells us that the leading coefficient of det ( A − x I n ) 1 | 1 is ( − 1 ) n − 1 . Then,

A − x I n 1 1 det ( A − x I n ) 1 | 1 = A 1 1 − x ( − 1 ) n − 1 x n − 1 + ( − 1 ) n − 2 t A 1 | 1 x n − 2 + …

Expanding the product shows α n − 1 (the coefficient of x n − 1 ) to be

α n − 1 = ( − 1 ) n − 1 A 1 1 + ( − 1 ) n − 1 t A 1 | 1 = ( − 1 ) n − 1 A 1 1 + ( − 1 ) n − 1 ∑ k = 1 n − 1 A 1 | 1 k k   Definition T = ( − 1 ) n − 1 A 1 1 + ∑ k = 1 n − 1 A 1 | 1 k k   Property DCN = ( − 1 ) n − 1 A 1 1 + ∑ k = 2 n A k k   Definition SM = ( − 1 ) n − 1 t A   Definition T

With two expressions for α n − 1 , we have our result,

t A = ( − 1 ) n + 1 ( − 1 ) n − 1 t A = ( − 1 ) n + 1 α n − 1 = ( − 1 ) n + 1 ( − 1 ) n + 1 ∑ i = 1 k α A λ i λ i = ∑ i = 1 k α A λ i λ i

■

Parent topic: Part T  Topics

Previous topic: Part T  Topics

Next topic: Section HP  Hadamard Product

Subsection EXC: Exercises

T10 Prove there are no square matrices A and B such that A B − B A = I n .  
Contributed by  Andy Zimmer

T12 Assume A is a square matrix of size n matrix. Prove t A = t A t .  
Contributed by  Andy Zimmer

T20 If T n = M ∈ M n n t M = 0 then prove T n is a subspace of M n n and determine it’s dimension.  
Contributed by  Andy Zimmer

T30 Assume A is a n × n matrix with polynomial entries. Define m d ( A , i ) to be the maximum degree of the entries in row i . Then d ( det A ) ≤ m d ( A , 1 ) + m d ( A , 2 ) + … + m d ( A , n ) . (Hint: If f ( x ) = h ( x ) + g ( x ) , then d ( f ) ≤  max { d ( h ) , d ( g ) } .)  
Contributed by  Andy Zimmer Solution [ 2363]

T40 If A is a square matrix, the matrix exponential is defined as

e A = ∑ i = 0 ∞ A i i !

Prove that det e A = e t A . (You might want to give some thought to the convergence of the infinite sum as well.)  
Contributed by  Andy Zimmer

Subsection SOL: Solutions

T30 Contributed by  Andy Zimmer Statement [ 2361]
We will proceed by induction. If A is a square matrix of size 1, then clearly d ( det A ) ≤ m d ( A , 1 ) . Now assume A is a square matrix of size n then by Theorem DER,

det A = ( − 1 ) 2 A 1 , 1 det A 1 | 1 + ( − 1 ) 3 A 1 , 2 det A 1 | 2 + ( − 1 ) 4 A 1 , 3 det A 1 | 3 + ⋯ + ( − 1 ) n + 1 A 1 , n det A 1 | n

Let’s consider the degree of term j , ( − 1 ) 1 + j A 1 , j det A 1 | j . By definition of the function m d , d ( A 1 , j ) ≤ m d ( A , j ) . We use our induction hypothesis to examine the other part of the product which tells us that

d det A 1 | j ≤ m d ( A 1 | j , 1 ) + m d ( A 1 | j , 2 ) + ⋯ + m d ( A 1 | j , n − 1 )

Furthermore by definition of A 1 | j (Definition SM) row i of matrix A contains all the entries of the corresponding row in A 1 | j then,

m d ( A 1 | j , 1 ) ≤ m d ( A , 1 ) m d ( A 1 | j , 2 ) ≤ m d ( A , 2 ) ⋮ m d ( A 1 | j , j − 1 ) ≤ m d ( A , j − 1 ) m d ( A 1 | j , j ) ≤ m d ( A , j + 1 ) ⋮ m d ( A 1 | j , n − 1 ) ≤ m d ( A , n )

So,

d det A 1 | j ≤ m d ( A 1 | j , 1 ) + m d ( A 1 | j , 2 ) + ⋯ + m d ( A 1 | j , n − 1 ) ≤ m d ( A , 1 ) + m d ( A , 2 ) + ⋯ + m d ( A , j − 1 ) + m d ( A , j + 1 ) + ⋯ + m d ( A , n − 1 )

Then using the property that if f ( x ) = g ( x ) h ( x ) then d ( f ) = d ( g ) + d ( h ) ,

d ( − 1 ) 1 + j A 1 , j det A 1 | j = d A 1 , j + d det A 1 | j ≤ m d ( A , j ) + m d ( A , 1 ) + m d ( A , 2 ) + ⋯ + m d ( A , j − 1 ) + m d ( A , j + 1 ) + ⋯ + m d ( A , n ) = m d ( A , 1 ) + m d ( A , 2 ) + ⋯ + m d ( A , n )

As j is arbitrary the degree of all terms in the determinant are so bounded. Finally using the fact that if f ( x ) = g ( x ) + h ( x ) then d ( f ) ≤  max { d ( h ) , d ( g ) } we have

d ( det A ) ≤ m d ( A , 1 ) + m d ( A , 2 ) + ⋯ + m d ( A , n )

Chapter 15. Section HP  Hadamard Product

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

This section is contributed by Elizabeth Million.

You may have once thought that the natural definition for matrix multiplication would be entrywise multiplication, much in the same way that a young child might say, “I writed my name.” The mistake is understandable, but it still makes us cringe. Unlike poor grammar, however, entrywise matrix multiplication has reason to be studied; it has nice properties in matrix analysis and additionally plays a role with relative gain arrays in chemical engineering, covariance matrices in probability and serves as an inertia preserver for Hermitian matrices in physics. Here we will only explore the properties of the Hadamard product in matrix analysis.

Definition  HP
Hadamard Product
Let A and B be m × n matrices. The Hadamard Product of A and B is defined by A ∘ B i j = A i j B i j for all 1 ≤ i ≤ m , 1 ≤ j ≤ n .

(This definition contains Notation HP.) △

As we can see, the Hadamard product is simply “entrywise multiplication”. Because of this, the Hadamard product inherits the same benefits (and restrictions) of multiplication in ℂ . Note also that both A and B need to be the same size, but not necessarily square. To avoid confusion, juxtaposition of matrices will imply the “usual” matrix multiplication, and we will use “ ∘ ” for the Hadamard product.

Example  HP
Hadamard Product
Consider

A = 1 0 6 3 π 5 B = 3 1 3 i 1 3 2 4

Then

A ∘ B = ( 1 ) ( 3 ) ( 0 ) ( 1 3 ) ( 6 ) ( i ) 3 ( 1 3 ) ( π ) ( 2 ) ( 5 ) ( 4 ) = 3 0 6 i 1 2 π 2 0 .

⊠

Now we will explore some basics properties of the Hadamard Product.

Theorem  HPC
Hadamard Product is Commutative
If A and B are m × n matrices then A ∘ B = B ∘ A . □

Proof   The proof follows directly from the fact that multiplication in ℂ is commutative. Let A and B be m × n matrices. Then

A ∘ B i j = A i j B i j   Definition HP = B i j A i j   Property CMCN = B ∘ A i j   Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. ■

Definition  HID
Hadamard Identity
The Hadamard identity is the m × n matrix J m n defined by J m n i j = 1 for all 1 ≤ i ≤ m , 1 ≤ j ≤ n .

(This definition contains Notation HID.) △

Theorem  HPHID
Hadamard Product with the Hadamard Identity
Suppose A is an m × n matrix. Then A ∘ J m n = J m n ∘ A = A . □

Proof  

A ∘ J m n i j = J m n ∘ A i j   Theorem HPC = J m n i j A i j   Definition HP = ( 1 ) A i j   Definition HID = A i j   Property OCN

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. ■

Definition  HI
Hadamard Inverse
Let A be an m × n matrix and suppose A i j ≠ 0 for all 1 ≤ i ≤ m , 1 ≤ j ≤ n . Then the Hadamard Inverse, A ̂ , is given by A ̂ i j = ( A i j ) − 1 for all 1 ≤ i ≤ m , 1 ≤ j ≤ n .

(This definition contains Notation HI.) △

Theorem  HPHI
Hadamard Product with Hadamard Inverses
Let A be an m × n matrix such that A i j ≠ 0 for all 1 ≤ i ≤ m , 1 ≤ j ≤ n . Then A ∘ A ̂ = A ̂ ∘ A = J m n . □

Proof  

A ∘ A ̂ i j = A ̂ ∘ A i j   Theorem HPC = A ̂ i j A i j   Definition HP = ( A i j ) − 1 A i j   Definition HI ,  A i j ≠ 0 = 1   Property MICN = J m n i j   Definition HID

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. ■

Since matrices have a different inverse and identity under the Hadamard product, we have used special notation to distinguish them from what we have been using with “normal” matrix multiplication. That is, compare “usual” matrix inverse, A − 1 , with the Hadamard inverse A ̂ , and the “usual” matrix identity, I n , with the Hadamard identity, J m n . The Hadamard identity matrix and the Hadamard inverse are both more limiting than helpful, so we will not explore their use further. One last fun fact for those of you who may be familiar with group theory: the set of m × n matrices with nonzero entries form an abelian (commutative) group under the Hadamard product (prove this!).

Theorem  HPDAA
Hadamard Product Distributes Across Addition
Suppose A , B and C are m × n matrices. Then C ∘ ( A + B ) = C ∘ A + C ∘ B . □

Proof  

C ∘ ( A + B ) i j = C i j A + B i j   Definition HP = C i j ( A i j + B i j )   Definition MA = C i j A i j + C i j B i j   Property DCN = C ∘ A i j + C ∘ B i j   Definition HP = C ∘ A + C ∘ B i j   Definition MA

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. ■

Theorem  HPSMM
Hadamard Product and Scalar Matrix Multiplication
Suppose α ∈ ℂ , and A and B are m × n matrices. Then α ( A ∘ B ) = ( α A ) ∘ B = A ∘ ( α B ) . □

Proof  

α A ∘ B i j = α A ∘ B i j   Definition MSM = α A i j B i j   Definition HP = α A i j B i j   Definition MSM = ( α A ) ∘ B i j   Definition HP = α A i j B i j   Definition MSM = A i j α B i j   Property CMCN = A i j α B i j   Definition MSM = A ∘ ( α B ) i j   Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. ■

Parent topic: Part T  Topics

Previous topic: Section T  Trace

Next topic: Section VM  Vandermonde Matrix

Subsection DMHP: Diagonal Matrices and the Hadamard Product

We can relate the Hadamard product with matrix multiplication by considering diagonal matrices, since A ∘ B = A B if and only if both A and B are diagonal (Citation!!!). For example, a simple calculation reveals that the Hadamard product relates the diagonal values of a diagonalizable matrix A with its eigenvalues:

Theorem  DMHP
Diagonalizable Matrices and the Hadamard Product
Let A be a diagonalizable matrix of size n with eigenvalues λ 1 , λ 2 , λ 3 , … , λ n . Let D be a diagonal matrix from the diagonalization of A , A = S D S − 1 , and d be a vector such that D i i = d i = λ i for all 1 ≤ i ≤ n . Then

A i i = S ∘ ( S − 1 ) t d i  for all  1 ≤ i ≤ n .

That is,

A 1 1 A 2 2 A 3 3 ⋮ A n n = S ∘ ( S − 1 ) t λ 1 λ 2 λ 3 ⋮ λ n

□

Proof  

S ∘ ( S − 1 ) t d i = ∑ k = 1 n S ∘ ( S − 1 ) t i k d k   Definition MVP = ∑ k = 1 n S ∘ ( S − 1 ) t i k λ k  Definition of  d = ∑ k = 1 n S i k ( S − 1 ) t i k λ k   Definition HP = ∑ k = 1 n S i k S − 1 k i λ k   Definition TM = ∑ k = 1 n S i k λ k S − 1 k i   Property CMCN = ∑ k = 1 n S i k D k k S − 1 k i  Definition of  D = ∑ j = 1 n ∑ k = 1 n S i k D k j S − 1 j i D k j = 0   for all  k ≠ j = ∑ j = 1 n S D i j S − 1 j i   Theorem EMP = S D S − 1 i i   Theorem EMP = A i i   Definition ME

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. ■

We obtain a similar result when we look at the singular value decomposition of square matrices (see exercises).

Theorem  DMMP
Diagonal Matrices and Matrix Products
Suppose A , B are m × n matrices, and D and E are diagonal matrices of size m and n , respectively. Then,

D ( A ∘ B ) E = ( D A E ) ∘ B = ( D A ) ∘ ( B E )

□

Proof  

D ( A ∘ B ) E i j = ∑ k = 1 m D i k ( A ∘ B ) E k j   Theorem EMP = ∑ k = 1 m ∑ l = 1 n D i k A ∘ B k l E l j   Theorem EMP = ∑ k = 1 m ∑ l = 1 n D i k A k l B k l E l j   Definition HP = ∑ k = 1 m D i k A k j B k j E j j E l j = 0   for all  l ≠ j = D i i A i j B i j E j j D i k = 0   for all  i ≠ k = D i i A i j E j j B i j   Property CMCN = D i i ( ∑ l = 1 n A i l E l j ) B i j E l j = 0   for all  l ≠ j = D i i A E i j B i j   Theorem EMP = ( ∑ k = 1 m D i k A E k j ) B i j D i k = 0   for all  i ≠ k = D A E i j B i j   Theorem EMP = ( D A E ) ∘ B i j   Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Also,

( D A E ) ∘ B i j = D A E i j B i j   Definition HP = ( ∑ k = 1 n D A i k E k j ) B i j   Theorem EMP = D A i j E j j B i j E k j = 0   for all  k ≠ j = D A i j B i j E j j   Property CMCN = D A i j ( ∑ k = 1 n B i k E k j ) E k j = 0   for all  k ≠ j = D A i j B E i j   Theorem EMP = ( D A ) ∘ ( B E ) i j   Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. ■

Subsection EXC: Exercises

T10 Prove that A ∘ B = A B if and only if both A and B are diagonal matrices.  
Contributed by  Elizabeth Million

T20 Suppose A , B are m × n matrices, and D and E are diagonal matrices of size m and n , respectively. Prove both parts of the following equality hold:

D ( A ∘ B ) E = ( A E ) ∘ ( D B ) = A ∘ ( D B E )

Contributed by  Elizabeth Million

T30 Let A be a square matrix of size n with singular values σ 1 , σ 2 , σ 3 , … , σ n . Let D be a diagonal matrix from the singular value decomposition of A , A = U D V ∗ ( Theorem SVD). Define the vector d by d i = D i i = σ i , 1 ≤ i ≤ n . Prove the following equality,

A i i = ( U ∘ V ¯ ) d i

Contributed by  Elizabeth Million

T40 Suppose A , B and C are m × n matrices. Prove that for all 1 ≤ i ≤ m ,

( A ∘ B ) C t i i = ( A ∘ C ) B t i i

Contributed by  Elizabeth Million

T50 Define the diagonal matrix D of size n with entries from a vector x ∈ ℂ n by

D i j = x i  if  i = j 0  otherwise

Furthermore, suppose A , B are m × n matrices. Prove that A D B t i i = ( A ∘ B ) x i for all 1 ≤ i ≤ m .  
Contributed by  Elizabeth Million

Chapter 16. Section VM  Vandermonde Matrix

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s

Alexandre-Théophile Vandermonde was a French mathematician in the 1700’s who was among the first to write about basic properties of the determinant (such as the effect of swapping two rows). However, the determinant that bears his name ( Theorem DVM) does not appear in any of his four published mathematical papers.

Definition  VM
Vandermonde Matrix
An square matrix of size n , A , is a Vandermonde matrix if there are scalars, x 1 , x 2 , x 3 , … , x n such that A i j = x i j − 1 , 1 ≤ i ≤ n , 1 ≤ j ≤ n . △

Example  VM4
Vandermonde matrix of size 4

A = 1 2 4 8 1 − 3 9 − 2 7 1 1 1 1 1 4 1 6 6 4

is a Vandermonde matrix since it meets the definition with x 1 = 2 , x 2 = − 3 , x 3 = 1 , x 4 = 4 . ⊠

Vandermonde matrices are not very interesting as numerical matrices, but instead appear more often in proofs and applications where the scalars x i are carried as symbols. Two such applications are in the sections on secret-sharing ( Section SAS) and curve-fitting ( Section CF). Principally, we would like to know when Vandermonde matrices are nonsingular, and the most convenient way to check this is by determining when the determinant is nonzero ( Theorem SMZD). As a bonus, the determinant of a Vandermonde matrix has an especially pleasing formula.

Theorem  DVM
Determinant of a Vandermonde Matrix
Suppose that A is a Vandermonde matrix of size n built with the scalars x 1 , x 2 , x 3 , … , x n . Then

det A = ∏ 1 ≤ i < j ≤ n x j − x i

□

Proof   The proof is by induction ( Technique I) on n , the size of the matrix. An empty product for a 1 × 1 matrix might make a good base case, but we’ll start at n = 2 instead. For a 2 × 2 Vandermonde matrix, we have

det A = 1 x 1 1 x 2 = x 2 − x 1 = ∏ 1 ≤ i < j ≤ 2 x j − x i

For the induction step we will perform row operations on A to obtain the determinant of A as multiple of the determinant of an ( n − 1 ) × ( n − 1 ) Vandermonde matrix. the notation in this theorem tens to obscure your intuition about the changes effected by various row and column manipulations. Construct a 4 × 4 Vandermonde matrix with four symbols as the scalars ( x 1 , x 2 , x 2 , x 4 , or perhaps a , b , c , d ) and play along with the example as you study the proof.

First we convert most of the first column to zeros. Subtract row n from each of the other n − 1 rows to form a matrix B . By Theorem DRCMA, B has the same determinant as A . The entries of B , in the first n − 1 rows, i.e. for 1 ≤ i ≤ n − 1 , 1 ≤ j ≤ n − 1 , are

B i j = x i j − 1 − x n j − 1 = x i − x n ∑ k = 0 j − 2 x i j − 2 − k x n k

As the elements of row i , 1 ≤ i ≤ n − 1 , have the common factor x i − x n , we form the new matrix C that differs from B by the removal of this factor from each of the first n − 1 rows. This will change the determinant, as we will track carefully in a moment. We also have a first column with zeros in each location, except row n , so we can use it for a column expansion computation of the determinant. We now know,

det A = det B   Theorem DRCMA = ( x 1 − x n ) ( x 2 − x n ) ⋯ ( x n − 1 − x n ) det C   Theorem DRCM = ( x 1 − x n ) ( x 2 − x n ) ⋯ ( x n − 1 − x n ) ( 1 ) ( − 1 ) n + 1 det C n − 1 | 1   Theorem DEC = ( x 1 − x n ) ( x 2 − x n ) ⋯ ( x n − 1 − x n ) ( − 1 ) n − 1 det C n − 1 | 1 = ( x n − x 1 ) ( x n − x 2 ) ⋯ ( x n − x n − 1 ) det C n − 1 | 1

For convenience, denote D = C n − 1 | 1 . Entries of this matrix are similar to those of B , but the factors used to build C are gone, and since the first column is gone, there is a slight re-indexing relative to the columns. For 1 ≤ i ≤ n − 1 , 1 ≤ j ≤ n − 1 ,

D i j = ∑ k = 0 j − 1 x i j − 1 − k x n k

We will perform many column operations on the matrix D , always of the type where we multiply a column by a scalar and add the result to another column. As such, Theorem DRCM insures that the determinant will remain constant. We will work column by column, left to right, to convert D into a Vandermonde matrix with scalars x 1 , x 2 , x 3 , … , x n − 1 . More precisely, we will build a sequence of matrices D = D 1 , D 2 , …, D n − 1 , where each obtainable from the previous by a sequence of determinant-preserving column operations and the first ℓ columns of D ℓ are the first ℓ columns of a Vandermonde matrix with scalars x 1 , x 2 , x 3 , … , x n − 1 . We could establish this claim by induction ( Technique I) on ℓ if we were to expand the claim to specify the exact values of the final n − 1 − ℓ columns as well. Since the claim is that matrices with certain properties exist, we will instead establish the claim by constructing the desired matrices one-by-one procedurally. The extension to an inductive proof should be clear, but not especially illuminating.

Set D 1 = D to begin, and note that the entries of the first column of D 1 are, for 1 ≤ i ≤ n − 1 ,

D 1 i 1 = ∑ k = 0 1 − 1 x i 1 − 1 − k x n k = 1 = x i 1 − 1

So the first column of D 1 has the properties we desire. We will use this column of all 1’s to remove the highest power of x n from each of the remaining columns and so build D 2 . Precisely, perform the n − 2 column operations where column 1 is multiplied by x n j − 1 and subtracted from column j , for 2 ≤ j ≤ n − 1 . Call the result D 2 , and examine its entries in columns 2 through n − 1 . For 1 ≤ i ≤ n − 1 , 2 ≤ j ≤ n − 1 ,

D 2 i j = − x n j − 1 D 1 i 1 + D 1 i j = − x n j − 1 ( 1 ) + ∑ k = 0 j − 1 x i j − 1 − k x n k = − x n j − 1 + x i j − 1 − ( j − 1 ) x n j − 1 + ∑ k = 0 j − 2 x i j − 1 − k x n k = ∑ k = 0 j − 2 x i j − 1 − k x n k

In particular, we examine column 2 of D 2 . For 1 ≤ i ≤ n − 1 ,

D 2 i 2 = ∑ k = 0 2 − 2 x i 2 − 1 − k x n k = x i 1 = x i 2 − 1

Now, form D 3 . Perform the n − 3 column operations where column 2 of D 2 is multiplied by x n j − 2 and subtracted from column j , for 3 ≤ j ≤ n − 1 . The result is D 3 , whose entries we now compute. For 1 ≤ i ≤ n − 1 ,

D 3 i j = − x n j − 2 D 2 i 2 + D 2 i j = − x n j − 2 x i 1 + ∑ k = 0 j − 2 x i j − 1 − k x n k = − x n j − 2 x i 1 + x i j − 1 − ( j − 2 ) x n j − 2 + ∑ k = 0 j − 3 x i j − 1 − k x n k = ∑ k = 0 j − 3 x i j − 1 − k x n k

Specifically, we examine column 3 of D 3 . For 1 ≤ i ≤ n − 1 ,

D 3 i 3 = ∑ k = 0 3 − 3 x i 3 − 1 − k x n k = x i 2 = x i 3 − 1

We could continue this procedure n − 4 more times, eventually totaling 1 2 n 2 − 3 n + 2 column operations, and arriving at D n − 1 , the Vandermonde matrix of size n − 1 built from the scalars x 1 , x 2 , x 3 , … , x n − 1 . Informally, we chop off the last term of every sum, until a single term is left in a column, and it is of the right form for the Vandermonde matrix. This desired column is then used in the next iteration to chop off some more final terms for columns to the right. Now we can apply our induction hypothesis to the determinant of D n − 1 and arrive at an expression for det A ,

det A = det C = ∏ k = 1 n − 1 x n − x k det D = ∏ k = 1 n − 1 x n − x k det D n − 1 = ∏ k = 1 n − 1 x n − x k ∏ 1 ≤ i < j ≤ n − 1 x j − x i = ∏ 1 ≤ i < j ≤ n x j − x i

which is the desired result. ■

Before we had Theorem DVM we could see that if two of the scalar values were equal, then the Vandermonde matrix would have two equal rows and hence be singular ( Theorem DERC, Theorem SMZD). But with this expression for the determinant, we can establish the converse.

Theorem  NVM
Nonsingular Vandermonde Matrix
A Vandermonde matrix of size n with scalars x 1 , x 2 , x 3 , … , x n is nonsingular if and only if the scalars are all different. □

Proof   Let A denote the Vandermonde matrix with scalars x 1 , x 2 , x 3 , … , x n . By Theorem SMZD, A is nonsingular if and only if the determinant of A is nonzero. The determinant is given by Theorem DVM, and this product is nonzero if and only if each term of the product is nonzero. This condition translates to x i − x j ≠ 0 whenever i ≠ j . In other words, the matrix is nonsingular if and only if the scalars are all different. ■

Parent topic: Part T  Topics

Previous topic: Section HP  Hadamard Product

Next topic: Section PSM  Positive Semi-definite Matrices

Chapter 17. Section PSM  Positive Semi-definite Matrices

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s
N e e d s N u m e r i c a l E x a m p l e s

Positive semi-definite matrices (and their cousins, positive definite matrices) are square matrices which in many ways behave like non-negative (respectively, positive) real numbers. Results given here are employed in the decompositions of Section SVD, Section SR and Section PD.

Parent topic: Part T  Topics

Previous topic: Section VM  Vandermonde Matrix

Next topic: Chapter MD  Matrix Decompositions

Subsection PSM: Positive Semi-Definite Matrices

Definition  PSM
Positive Semi-Definite Matrix
A square matrix A of size n is positive semi-definite if A is Hermitian and for all x ∈ ℂ n , A x , x ≥ 0 . △

For a definition of positive definite replace the inequality in the definition with a strict inequality, and exclude the zero vector from the vectors x required to meet the condition. Similar variations allow definitions of negative definite and negative semi-definite. Our first theorem in this section gives us an easy way to build positive semi-definite matrices.

Theorem  CPSM
Creating Positive Semi-Definite Matrices
Suppose that A is any m × n matrix. Then the matrices A ∗ A and A A ∗ are positive semi-definite matrices. □

Proof   We will give the proof for the first matrix, the proof for the second is entirely similar. First we check that A ∗ A is Hermitian,

A ∗ A ∗ = A ∗ A ∗ ∗   Theorem MMAD = A ∗ A   Theorem AA

so by Definition HM, the matrix A ∗ A is Hermitian. Second, for any x ∈ ℂ n ,

A ∗ A x , x = A x , A ∗ ∗ x   Theorem AIP = A x , A x   Theorem AA ≥ 0   Theorem PIP

which is the second criteria in the definition of a positive semi-definite matrix ( Definition PSM). ■

A statement very similar to the converse of this theorem is also true. Any positive semi-definite matrix can be realized as the product of a square matrix, B , with its adjoint, B ∗ . (See Exercise PSM.T20 after studying this entire section.) The matrices A ∗ A and A A ∗ will be important later when we define singular values ( Section SVD).

Positive semi-definite matrices can also be characterized by their eigenvalues, without any mention of inner products. This next result further reinforces the notion that positive semi-definite matrices behave like non-negative real numbers.

Theorem  EPSM
Eigenvalues of Positive Semi-definite Matrices
Suppose that A is a Hermitian matrix. Then A is positive semi-definite matrix if and only if whenever λ is an eigenvalue of A , then λ ≥ 0 . □

Proof   Notice first that since we are considering only Hermitian matrices in this theorem, it is always possible to compare eigenvalues with the real number zero, since eigenvalues of Hermitian matrices are all real numbers ( Theorem HMRE). Let n denote the size of A .

( ⇒ ) Let x ≠ 0 be an eigenvector of A for λ . Then by Theorem PIP we know x , x ≠ 0 . So

λ = 1 x , x λ x , x   Property MICN = 1 x , x λ x , x   Theorem IPSM = 1 x , x A x , x   Definition EEM

By Theorem PIP, x , x > 0 and by Definition PSM we have A x , x ≥ 0 . With λ expressed as the product of these two quantities, we have λ ≥ 0 .

( ⇐ ) Suppose now that λ 1 , λ 2 , λ 3 , … , λ n are the (not necessarily distinct) eigenvalues of the Hermitian matrix A , each of which is non-negative. Let B = x 1 , x 2 , x 3 , … , x n be a set of associated eigenvectors for these eigenvalues. Since a Hermitian matrix is normal ( Definition HM, Definition NM), Theorem OBNM allows us to choose this set of eigenvectors to also be an orthonormal basis of ℂ n . Choose any x ∈ ℂ n and let a 1 , a 2 , a 3 , … , a n be the scalars guaranteed by the spanning property of the basis B such that

x = a 1 x 1 + a 2 x 2 + a 3 x 3 + ⋯ + a n x n = ∑ i = 1 n a i x i

Since we have presumed A is Hermitian, we need only check the other defining property,

A x , x = A ∑ i = 1 n a i x i , ∑ j = 1 n a j x j   Definition TSVS = ∑ i = 1 n A a i x i , ∑ j = 1 n a j x j   Theorem MMDAA = ∑ i = 1 n a i A x i , ∑ j = 1 n a j x j   Theorem MMSMM = ∑ i = 1 n a i λ i x i , ∑ j = 1 n a j x j   Definition EEM = ∑ i = 1 n ∑ j = 1 n a i λ i x i , a j x j   Theorem IPVA = ∑ i = 1 n ∑ j = 1 n a i λ i a j ¯ x i , x j   Theorem IPSM = ∑ i = 1 n a i λ i a i ¯ x i , x i + ∑ i = 1 n ∑ j = 1 j ≠ i n a i λ i a j ¯ x i , x j   Property CACN = ∑ i = 1 n a i λ i a i ¯ ( 1 ) + ∑ i = 1 n ∑ j = 1 j ≠ i n a i λ i a j ¯ ( 0 )   Definition ONS = ∑ i = 1 n a i λ i a i ¯ = ∑ i = 1 n λ i a i 2   Definition MCN

With non-negative values for each eigenvalue λ i , 1 ≤ i ≤ n , and each modulus squared, it should be clear that this sum is non-negative. Which is exactly what is required by Definition PSM to establish that A is positive semi-definite. ■

As positive semi-definite matrices are defined to be Hermitian, they are then normal and subject to orthonormal diagonalization ( Theorem OD). Now consider the interpretation of orthonormal diagonalization as a rotation to principal axes, a stretch by a diagonal matrix and a rotation back ( Subsection OD.OD). For a positive semi-definite matrix, the diagonal matrix has diagonal entries that are the non-negative eigenvalues of the original positive semi-definite matrix. So the “stretching” along each axis is never a reflection.

Subsection EXC: Exercises

T20 Suppose that A is a positive semi-definite matrix of size n . Prove that there is a square matix B of size n such that A = B B ∗ .  
Contributed by  Robert Beezer

Chapter 18. Chapter MD  Matrix Decompositions

This chapter is about breaking up a matrix A into pieces that somehow combine to recreate A . Usually the pieces are again matrices, and usually they are then combined via matrix multiplication ( Definition MM). In some cases, the decomposition will be valid for any matrix, but often we might need extra conditions on A , such as being square ( Definition SQM), nonsingular ( Definition NM) or diagonalizable (Definition DZM) before we can guarantee the decomposition. If you are comfortable with topics like decomposing a solution vector into linear combinations (Subsection LC.VFSS) or decomposing vector spaces into direct sums (Subsection PD.DS), then we will be doing similar things in this chapter. If not, review these ideas and take another look at Technique DC on decompositions.

We have studied one matrix decomposition already, so we will review that here in this introduction, both as a way of previewing the topic in a familiar setting, but also since it does not deserve another section all of its own.

A diagonalizable matrix (Definition DZM) is defined to be a square matrix A such that there is an invertible matrix S and a diagonal matrix D where S − 1 A S = D . We can re-write this as A = S D S − 1 . Here we have a decomposition of A into three matrices, S , D and S − 1 , which recombine through matrix multiplication to recreate A . We also know that the diagonal entries of D are the eigenvalues of A . We cannot form this decomposition for just any matrix — A must be square and we know from Theorem DC that a matrix of size n is diagonalizable if and only if there is a basis for ℂ n composed entirely of eigenvectors of A , or by Theorem DMFE we know that A is diagonalizable if and only if each eigenvalue of A has a geometric multiplicity equal to its algebraic multiplicity. Some authors prefer to call this an eigen decomposition of A rather than a matrix diagonalization.

Another decomposition, which is similar in flavor to matrix diagonalization, is orthonormal diagonalization ( Theorem OD). Here we require the matrix A to be normal and we get the decomposition A = U D U ∗ , where D is a diagonal matrix with the eigenvalues of A on the diagonal, and U is unitary. The hypothesis that A is normal guarantees the decomposition and we get the extra information that U is unitary.

Each section of this chapter features a different matrix decomposition, with the exception of Section PSM, which presents background information on positive semi-definite matrices required for singular value decompositions, square roots and polar decompositions.

Section ROD  Rank One Decomposition

Section TD  Triangular Decomposition

Section SVD  Singular Value Decomposition

Section SR  Square Roots

Section POD  Polar Decomposition

Parent topic: Part T  Topics

Previous topic: Section PSM  Positive Semi-definite Matrices

Next topic: Section ROD  Rank One Decomposition

Section ROD  Rank One Decomposition

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s

Our first decomposition applies only to diagonalizable (Definition DZM) matrices, and yields a decomposition into a sum of very simple matrices.

Theorem  ROD
Rank One Decomposition
Suppose that A is a diagonalizable matrix of size n and rank r . Then there are r square matrices A 1 , A 2 , A 3 , … , A r , each of size n and rank 1 such that

A = A 1 + A 2 + A 3 + ⋯ + A r

Furthermore, if λ 1 , λ 2 , λ 3 , … , λ r are the nonzero eigenvalues of A , then there are two sets of r linearly independent vectors from ℂ n ,

X = x 1 , x 2 , x 3 , … , x r Y = y 1 , y 2 , y 3 , … , y r

such that A k = λ k x k y k t , 1 ≤ k ≤ r . □

Proof   The proof is constructive. Generally, we will diagonalize A , creating a nonsingular matrix S and a diagonal matrix D . Then we split up the diagonal matrix into a sum of matrices with a single nonzero entry (on the diagonal). This fundamentally creates the decomposition in the statement of the theorem, the remainder is just bookkeeping. The vectors in X and Y will result from the columns of S and the rows of S − 1 .

Let λ 1 , λ 2 , λ 3 , … , λ n be the eigenvalues of A (repeated according to their algebraic multiplicity). If A has rank r , then dim N A = n − r (Theorem RPNC). The null space of A is the eigenspace of the eigenvalue λ = 0 (Theorem EMNS), so it follows that the algebraic multiplicity of λ = 0 is n − r , α A 0 = n − r . Presume that the complete list of eigenvalues is ordered so that λ k = 0 for r + 1 ≤ k ≤ n .

Since A is hypothesized to be diagonalizable, there exists a diagonal matrix D and an invertible matrix S , such that D = S − 1 A S . We can rearrange tis equation to read, A = S D S − 1 . Also, the proof of Theorem DC says that the diagonal elements of D are the eigenvalues of A and we have the flexibility to assume they lie on the diagonal in the same order as we have specified above. Now, let X ∗ = x 1 , x 2 , x 3 , … , x n be the columns of S , and let Y ∗ = y 1 , y 2 , y 3 , … , y n be the rows of S − 1 converted to column vectors. With little motivation other than the statement of the theorem, define size n matrices A k , 1 ≤ k ≤ n by A k = λ k x k y k t . Finally, let D k be the size n matrix that is totally zero, other than having λ k in row k and column k .

With everything in place, we compute entry-by-entry,

A i j = S D S − 1 i j   Definition DZM = S ∑ k = 1 n D k S − 1 i j   Definition MA = S ∑ k = 1 n D k S − 1 i j   Theorem MMDAA = ∑ k = 1 n S D k S − 1 i j   Theorem MMDAA = ∑ k = 1 n S D k S − 1 i j   Definition MA = ∑ k = 1 n ∑ ℓ = 1 n S D k i ℓ S − 1 ℓ j   Theorem EMP = ∑ k = 1 n ∑ ℓ = 1 n ∑ p = 1 n S i p D k p ℓ S − 1 ℓ j   Theorem EMP = ∑ k = 1 n S i k D k k k S − 1 k j   D k p ℓ = 0  if  p ≠ k , or  ℓ ≠ k = ∑ k = 1 n S i k λ k S − 1 k j D k k k = λ k = ∑ k = 1 n λ k S i k S − 1 k j   Property CMCN = ∑ k = 1 n λ k x k i 1 y k t 1 j  Definition of  X ∗ ,  Y ∗ = ∑ k = 1 n λ k ∑ q = 1 1 x k i q y k t q j = ∑ k = 1 n λ k x k y k t i j   Theorem EMP = ∑ k = 1 n λ k x k y k t i j   Definition MSM = ∑ k = 1 n A k i j  Definition of  A k = ∑ k = 1 n A k i j   Definition MA

So by Definition ME we have the desired equality of matrices. The careful reader will have noted that A k = O , r + 1 ≤ k ≤ n , since λ k = 0 in these instances. To get the sets X and Y from X ∗ and Y ∗ , simply discard the last n − r vectors. We can safely ignore (or remove) A r + 1 , A r + 2 , … , A n from the summation just derived.

One last assertion to check. What is the rank of A k , 1 ≤ k ≤ r ? Every row of A k is a scalar multiple of y k t , row k of the nonsingular matrix S − 1 ( Theorem MIMI). As a row of a nonsingular matrix, y k t cannot be all zeros. In particular, row i of A k is obtained as a scalar multiple of y k t by the scalar α k x k i . We have restricted ourselves to the nonzero eigenvalues of A , and as S is nonsingular, some entry of x k is nonzero. This all implies that some row of A k will be nonzero. Now consider row-reducing A k . Swap the nonzero row up into row 1. Use scalar multiples of this row to zero out every other row. This leaves a single nonzero row in the reduced row-echelon form, so A k has rank one. ■

We record two observations that was not stated in our theorem above. First, the vectors in X , chosen as columns of S , are eigenvectors of A . Second, the product of two vectors from X and Y in the opposite order, by which we mean y i t x j , is the entry in row i and column j of the matrix product S − 1 S = I n ( Theorem EMP). In particular,

y i t x j = 1  if  i = j 0  if  i ≠ j

We give two computational examples. One small, one a bit bigger.

Example  ROD2
Rank one decomposition, size 2
Consider the 2 × 2 matrix,

A = − 1 6 − 6 4 5 1 7

By the techniques of Chapter E we find the eigenvalues and eigenspaces,

λ 1 = 2 ℰ A 2 = − 1 3 λ 2 = − 1 ℰ A − 1 = − 2 5

With n = 2 distinct eigenvalues, Theorem DED tells us that A is diagonalizable, and with no zero eigenvalues we see that A has full rank. Theorem DC says we can construct the nonsingular matrix S with eigenvectors of A as columns, so we have

S = − 1 − 2 3 5 S − 1 = 5 2 − 3 − 1

From these matrices we obtain the sets of vectors

X = − 1 3 , − 2 5 Y = 5 2 , − 3 − 1

And we have the matrices,

A 1 = 2 − 1 3 5 2 t = 2 − 5 − 2 1 5 6 = − 1 0 − 4 3 0 1 2 A 2 = ( − 1 ) − 2 5 − 3 − 1 t = ( − 1 ) 6 2 − 1 5 − 5 = − 6 − 2 1 5 5

And you can easily verify that A = A 1 + A 2 . ⊠

Here’s a slightly larger example, and the matrix does not have full rank.

Example  ROD4
Rank one decomposition, size 4
Consider the 4 × 4 matrix,

B = 3 4 1 8 − 1 − 6 − 4 4 − 2 4 − 1 9 3 6 1 8 − 3 − 6 3 6 1 8 − 6 − 3

By the techniques of Chapter E we find the eigenvalues and eigenvectors,

λ 1 = 3 ℰ B 3 = 1 − 2 1 − 1 , 1 − 1 1 2 λ 2 = − 2 ℰ B − 2 = − 1 2 0 0 λ 3 = 0 ℰ A 0 = 2 − 3 2 2

The algebraic and geometric multiplicities of each eigenvalue are equal, so Theorem DMFE tells us that A is diagonalizable. With a single zero eigenvalue we see that A has rank 4 − 1 = 3 . Theorem DC says we can construct the nonsingular matrix S with eigenvectors of A as columns, so we have

S = 1 1 − 1 2 − 2 − 1 2 − 3 1 1 0 2 − 1 2 0 2 S − 1 = 4 2 0 − 1 8 4 − 1 − 1 − 1 0 1 0 − 6 − 3 1 1

Since r = 3 , we need only collect three vectors from each of these matrices,

X = 1 − 2 1 − 1 , 1 − 1 1 2 , − 1 2 0 0 Y = 4 2 0 − 1 , 8 4 − 1 − 1 , − 1 0 1 0

And we obtain the matrices,

B 1 = 3 1 − 2 1 − 1 4 2 0 − 1 t = 3 4 2 0 − 1 − 8 − 4 0 2 4 2 0 − 1 − 4 − 2 0 1 = 1 2 6 0 − 3 − 2 4 − 1 2 0 6 1 2 6 0 − 3 − 1 2 − 6 0 3 B 2 = 3 1 − 1 1 2 8 4 − 1 − 1 t = 3 8 4 − 1 − 1 − 8 − 4 1 1 8 4 − 1 − 1 1 6 8 − 2 − 2 = 2 4 1 2 − 3 − 3 − 2 4 − 1 2 3 3 2 4 1 2 − 3 − 3 4 8 2 4 − 6 − 6 B 3 = ( − 2 ) − 1 2 0 0 − 1 0 1 0 t = ( − 2 ) 1 0 − 1 0 − 2 0 2 0 0 0 0 0 0 0 0 0 = − 2 0 2 0 4 0 − 4 0 0 0 0 0 0 0 0 0

Then we verify that

B = B 1 + B 2 + B 3 = 1 2 6 0 − 3 − 2 4 − 1 2 0 6 1 2 6 0 − 3 − 1 2 − 6 0 3 + 2 4 1 2 − 3 − 3 − 2 4 − 1 2 3 3 2 4 1 2 − 3 − 3 4 8 2 4 − 6 − 6 + − 2 0 2 0 4 0 − 4 0 0 0 0 0 0 0 0 0 = 3 4 1 8 − 1 − 6 − 4 4 − 2 4 − 1 9 3 6 1 8 − 3 − 6 3 6 1 8 − 6 − 3

⊠

Parent topic: Chapter MD  Matrix Decompositions

Previous topic: Chapter MD  Matrix Decompositions

Next topic: Section TD  Triangular Decomposition

Section TD  Triangular Decomposition

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s

Our next decomposition will break a square matrix into a product of two matrices, one lower triangular and the other upper triangular. So we will write A = L U , and hence many refer to this as LU decomposition. We will see that this decomposition is very easy to compute and that it has a direct application to solving systems of equations. Since this section is about triangular matrices you might want to review the definitions and a couple of basic theorems back in Subsection OD.TM.

Parent topic: Chapter MD  Matrix Decompositions

Previous topic: Section ROD  Rank One Decomposition

Next topic: Section SVD  Singular Value Decomposition

Subsection TD: Triangular Decomposition

With a slight condition on the nonsingularity of certain submatrices, we can split a matrix into a product of two triangular matrices.

Theorem  TD
Triangular Decomposition
Suppose A is a square matrix of size n . Let A k be the k × k matrix formed from A by taking the first k rows and the first k columns. Suppose that A k is nonsingular for all 1 ≤ k ≤ n . Then there is a lower triangular matrix L with all of its diagonal entries equal to 1 and an upper triangular matrix U such that A = L U . Furthermore, this decomposition is unique. □

Proof   We will row reduce A to a row-equivalent upper triangular matrix through a series of row operations, forming intermediate matrices A j ′ , 1 ≤ j ≤ n , that denote the state of the conversion after working on column j . First, the lone entry of A 1 is A 1 1 and this scalar must be nonzero if A 1 is nonsingular ( Theorem SMZD). We can use row operations Definition RO of the form α R 1 + R k , 2 ≤ k ≤ n , where α = − A 1 k ∕ A 1 1 to place zeros in the first column below the diagonal. The first two rows and columns of A 1 ′ are a 2 × 2 upper triangular matrix whose determinant is equal to the determinant of A 2 , since the matrices are row-equivalent through a sequence of row operations strictly of the third type ( Theorem DRCMA). As such the diagonal entries of this 2 × 2 submatrix of A 1 ′ are nonzero. We can employ this nonzero diagonal element with row operations of the form α R 2 + R k , 3 ≤ k ≤ n to place zeros below the diagonal in the second column. We can continue this process, column by column. The key observations are that our hypothesis on the nonsingularity of the A k will guarantee a nonzero diagonal entry for each column when we need it, that the row operations employed are always of the third type using a multiple of a row to transform another row with a greater row index, and that the final result will be a nonsingular upper triangular matrix. This is the desired matrix U .

Each row operation described in the previous paragraph can be accomplished with matrix multiplication by the appropriate elementary matrix (Theorem EMDRO). Since every row operation employed is adding a multiple of a row to a subsequent row these elementary matrices are of the form E j , k α with j < k . By Definition ELEM, these matrices are lower triangular with every diagonal entry equal to 1. We know that the product of two such matrices will again be lower triangular ( Theorem PTMT), but also, as you can also easily check using a proof with a style similar to one above, that the product maintains all 1’s on the diagonal. Let E 1 , E 2 , E 3 , … , E m denote the elementary matrices for this sequence of row operations. Then

U = E m E m − 1 … E 3 E 2 E 1 A = L ′ A

where L ′ is the product of the elementary matrices, and we know L ′ is lower triangular with all 1’s on the diagonal. Our desired matrix L is then L = L ′ − 1 . By Theorem ITMT, L is lower triangular with all 1’s on the diagonal and A = L U , as desired.

The process just described is deterministic. That is, the proof is constructive, with no freedom for each of us to walk through it differently. But could there be other matrices with the same properties as L and U that give such a decomposition of A . In other words, is the decomposition unique ( Technique U)? Suppose that we have two triangular decompositions, A = L 1 U 1 and A = L 2 U 2 . Since A is nonsingular, two applications of Theorem NPNT imply that L 1 , L 2 , U 1 , U 2 are all nonsingular. We have

L 2 − 1 L 1 = L 2 − 1 I n L 1   Theorem MMIM = L 2 − 1 A A − 1 L 1   Definition MI = L 2 − 1 L 2 U 2 L 1 U 1 − 1 L 1 = L 2 − 1 L 2 U 2 U 1 − 1 L 1 − 1 L 1   Theorem SS = I n U 2 U 1 − 1 I n   Definition MI = U 2 U 1 − 1   Theorem MMIM

Theorem ITMT tells us that L 2 − 1 is lower triangular and has 1’s as the diagonal entries. By Theorem PTMT, the product L 2 − 1 L 1 is again lower triangular, and it is simple to check (as before) that the diagonal entries of the product are again all 1’s. By the entirely similar process we can conclude that the product U 2 U 1 − 1 is upper triangular. Because these two products are equal, their common value is a matrix that is both lower triangular and upper triangular, with all 1’s on the diagonal. The only matrix meeting these three requirements is the identity matrix ( Definition IM). So, we have,

I n = L 2 − 1 L 1 ⇒ L 2 = L 1 I n = U 2 U 1 − 1 ⇒ U 1 = U 2

which establishes the uniqueness of the decomposition. ■

Studying the proofs of some previous theorems will perhaps give you an idea for an approach to computing a triangular decomposition. In the proof of Theorem CINM we augmented a nonsingular matrix with an identity matrix of the same size, and row-reduced until the original matrix became the identity matrix (as we knew in advance would happen, since we knew Theorem NMRRI). Theorem PEEF tells us about properties of extended echelon form, and in particular, that B = J A , where A is the matrix that begins on the left, and B is the reduced row-echelon form of A . The matrix J is the result on the right side of the augmented matrix, which is the result of applying the same row operations to the identity matrix. We should recognize now that J is just the product of the elementary matrices (Subsection DM.EM) that perform these row operations. Theorem ITMT used the extended echelon form to discern properties of the inverse of a triangular matrix. Theorem TD proves the existence of a triangular decomposition by applying specific row operations, and tracking the relevant elementary row operations. It is not a great leap to combine these observations into a computational procedure.

To find the triangular decomposition of A , augment A with the identity matrix of the same size and call this new 2 n × n matrix, M . Perform row operations on M that convert the first n columns to an upper triangular matrix. Do this using only row operations that add a scalar multiple of one row to another row with higher index (i.e. lower down). In this way, the last n columns of M will be converted into a lower triangular matrix with 1’s on the diagonal (since M has 1’s in these locations initially). We could think of this process as doing about half of the work required to compute the inverse of A . Take the first n columns of the row-equivalent version of M and call this matrix U . Take the final n columns of the row-equivalent version of M and call this matrix L ′ . Then by a proof employing elementary matrices, or a proof similar in spirit to the one used to prove Theorem PEEF, we arrive at a result similar to the second assertion of Theorem PEEF. Namely, U = L ′ A . Multiplication on the left, by the inverse of L ′ , will give us a decomposition of A (which we know to be unique). Ready? Lets try it.

Example  TD4
Triangular decomposition, size 4
In this example, we will illustrate the process for computing a triangular decomposition, as described in the previous paragraphs. Consider the nonsingular square matrix A of size 4,

A = − 2 6 − 8 7 − 4 1 6 − 1 4 1 5 − 6 2 2 − 2 3 2 6 − 6 2 6 − 1 8 1 7

We form M by augmenting A with the size 4 identity matrix I 4 . We will perform the allowed operations, column by column, only reporting intermediate results as we finish converting each column. It is easy to determine exactly which row operations we perform, since the final four columns contain a record of each such operation. We will not verify our hypotheses about the nonsingularity of the A k , since if we do not have these conditions, we will reach a stage where a diagonal entry is zero and we cannot create the row operations we need to zero out the bottom portion of the associated column. In other words, we can boldly proceed and the necessity of our hypotheses will become apparent.

M = − 2 6 − 8 7 1 0 0 0 − 4 1 6 − 1 4 1 5 0 1 0 0 − 6 2 2 − 2 3 2 6 0 0 1 0 − 6 2 6 − 1 8 1 7 0 0 0 1 → − 2 6 − 8 7 1 0 0 0 0 4 2 1 − 2 1 0 0 0 4 1 5 − 3 0 1 0 0 8 6 − 4 − 3 0 0 1 → − 2 6 − 8 7 1 0 0 0 0 4 2 1 − 2 1 0 0 0 0 − 1 4 − 1 − 1 1 0 0 0 2 − 6 1 − 2 0 1 → − 2 6 − 8 7 1 0 0 0 0 4 2 1 − 2 1 0 0 0 0 − 1 4 − 1 − 1 1 0 0 0 0 2 − 1 − 4 2 1

So at this point, we have U and L ′ ,

U = − 2 6 − 8 7 0 4 2 1 0 0 − 1 4 0 0 0 2 L ′ = 1 0 0 0 − 2 1 0 0 − 1 − 1 1 0 − 1 − 4 2 1

Then by whatever procedure we like (such as Theorem CINM), we find

L = L ′ − 1 = 1 0 0 0 2 1 0 0 3 1 1 0 3 2 − 2 1

It is instructive to verify that indeed L U = A . ⊠

Subsection TDSSE: Triangular Decomposition and Solving Systems of Equations

In this section we give an explanation of why you might be interested in a triangular decomposition for a matrix. Many of the computational problems in linear algebra revolve around solving large systems of equations, or nearly equivalently, finding inverses of large matrices. Suppose we have a system of equations with coefficient matrix A and vector of constants b , and suppose further that A has the triangular decomposition A = L U .

Let y be the solution to the linear system ℒ S L , b , so that by Theorem SLEMM, we have L y = b . Notice that since L is nonsingular, this solution is unique, and the form of L makes it trivial to solve the system. The first component of y is determined easily, and we can continue on through determining the components of y , without even ever dividing. Now, with y in hand, consider the linear system, ℒ S U , y . Let x be the unique solution to this system, so by Theorem SLEMM we have U x = y . Notice that a system of equations with U as a coefficient matrix is also straightforward to solve, though we will compute the bottom entries of x first, and we will need to divide. The upshot of all this is that x is a solution to ℒ S A , b , as we now show,

A x = L U x = L U x = L y = b

An application of Theorem SLEMM demonstrates that x is a solution to ℒ S A , b .

Example  TDSSE
Triangular decomposition solves a system of equations
Here we illustrate the previous discussion, recycling the decomposition found previously in Example TD4. Consider the linear system ℒ S A , b with

A = − 2 6 − 8 7 − 4 1 6 − 1 4 1 5 − 6 2 2 − 2 3 2 6 − 6 2 6 − 1 8 1 7 b = − 1 0 − 2 − 1 − 8

First we solve the system ℒ S L , b (see Example TD4 for L ),

y 1 = − 1 0 2 y 1 + y 2 = − 2 3 y 1 + y 2 + y 3 = − 1 3 y 1 + 2 y 2 − 2 y 3 + y 4 = − 8

Then

y 1 = − 1 0 y 2 = − 2 − 2 y 1 = − 2 − 2 ( − 1 0 ) = 1 8 y 3 = − 1 − 3 y 1 − y 2 = − 1 − 3 ( − 1 0 ) − 1 8 = 1 1 y 4 = − 8 − 3 y 1 − 2 y 2 + 2 y 3 = − 8 − 3 ( − 1 0 ) − 2 ( 1 8 ) + 2 ( 1 1 ) = 8

so

y = − 1 0 1 8 1 1 8

Then we solve the system ℒ S U , y (see Example TD4 for U ),

− 2 x 1 + 6 x 2 − 8 x 3 + 7 x 4 = − 1 0 4 x 2 + 2 x 3 + x 4 = 1 8 − x 3 + 4 x 4 = 1 1 2 x 4 = 8

Then

x 4 = 8 ∕ 2 = 4 x 3 = 1 1 − 4 x 4 ∕ ( − 1 ) = 1 1 − 4 ( 4 ) ∕ ( − 1 ) = 5 x 2 = 1 8 − 2 x 3 − x 4 ∕ 4 = 1 8 − 2 ( 5 ) − 4 ∕ 4 = 1 x 1 = − 1 0 − 6 x 2 + 8 x 3 − 7 x 4 ∕ ( − 2 ) = − 1 0 − 6 ( 1 ) + 8 ( 5 ) − 7 ( 4 ) ∕ ( − 2 ) = 2

And so

x = 4 5 1 2

is the solution to ℒ S U , y and consequently is the unique solution to ℒ S A , b , as you can easily verify. ⊠

Subsection CTD: Computing Triangular Decompositions

It would be a simple matter to adjust the algorithm for converting a matrix to reduced row-echelon form and obtain an algorithm to compute the triangular decomposition of the matrix, along the lines of Example TD4 and the discussion preceding this example. However, it is possible to obtain relatively simple formulas for the entries of the decomposition, and if computed in the proper order, an implementation will be straightforward. We will state the result as a theorem and then give an example of its use.

Theorem  TDEE
Triangular Decomposition, Entry by Entry
Suppose that A is a squarematrix of size n with a triangular decomposition A = L U , where L is lower triangular with diagonal entries all equal to 1, and U is upper triangular. Then

U i j = A i j − ∑ k = 1 i − 1 L i k U k j 1 ≤ i ≤ j ≤ n L i j = 1 U j j A i j − ∑ k = 1 j − 1 L i k U k j 1 ≤ j < i ≤ n

□

Proof   Consider a single scalar product of an entry of L with an entry of U of the form L i k U k j . By Definition LTM, if k > i then L i k = 0 , while Definition UTM, says that if k > j then U k j = 0 . So we can combine these two facts to assert that if k > min ( i , j ) , L i k U k j = 0 since at least one term of the product will be zero. Employing this observation,

A i j = ∑ k = 1 n L i k U k j   Theorem EMP = ∑ k = 1 min ( i , j ) L i k U k j

Now, assume that 1 ≤ i ≤ j ≤ n ,

U i j = A i j − A i j + U i j = A i j − ∑ k = 1 min ( i , j ) L i k U k j + U i j = A i j − ∑ k = 1 i L i k U k j + U i j = A i j − ∑ k = 1 i − 1 L i k U k j − L i i U i j + U i j = A i j − ∑ k = 1 i − 1 L i k U k j − U i j + U i j = A i j − ∑ k = 1 i − 1 L i k U k j

And for 1 ≤ j < i ≤ n ,

L i j = 1 U j j L i j U j j = 1 U j j A i j − A i j + L i j U j j = 1 U j j A i j − ∑ k = 1 min ( i , j ) L i k U k j + L i j U j j = 1 U j j A i j − ∑ k = 1 j L i k U k j + L i j U j j = 1 U j j A i j − ∑ k = 1 j − 1 L i k U k j − L i j U j j + L i j U j j = 1 U j j A i j − ∑ k = 1 j − 1 L i k U k j

■

At first glance, these formulas may look exceedingly complex. Upon closer examination, it looks even worse. We have expressions for entries of U that depend on other entries of U and also on entries of L . But then the formula for entries of L depend on entries from L and entries from U . Do these formula have circular dependencies? Or perhaps equivalently, how do we get started? The key is to be organized about the computations and employ these two (similar) formulas in a specific order. First compute the first row of L , followed by the first column of U . Then the second row of L , followed by the second column of U . And so on. In this way, all of the values required for each new entry will have already been computed previously.

Of course, the formula for entries of L require division by diagonal entries of U . These entries might be zero, but in this case A is nonsingular and does not have a triangular decomposition. So we need not check the hypothesis carefully and can launch into the arithmetic dictated by the formulas, confident that we will be reminded when a decomposition is not possible. Note that these formula give us all of the values that we need for the decomposition, since we require that L has 1’s on the diagonal. If we replace the 1’s on the diagonal of L by zeros, and add the matrix U , we get an n × n matrix containing all the information we need to resurrect the triangular decomposition. This is mostly a notational convenience, but it is a frequent way of presenting the information. We’ll employ it in the next example.

Example  TDEE6
Triangular decomposition, entry by entry, size 6
We illustrate the application of the formulas in Theorem TDEE for the 6 × 6 matrix A .

A = 3 3 − 3 − 2 − 1 0 − 6 − 4 5 2 4 2 9 9 − 7 − 7 0 1 − 6 − 1 0 8 1 0 − 1 − 7 6 4 − 9 − 2 − 1 0 1 9 3 − 1 2 − 3 − 2 1 − 2

Using the notational convenience of packaging the two triangular matrices into one matrix, and using the ordering of the computations mentioned above, we display the results after computing a single row and column of each of the two triangular matrices.

3 3 − 3 − 2 − 1 0 − 2 3 − 2 2 3 3 3 − 3 − 2 − 1 0 − 2 2 − 1 − 2 2 2 3 0 − 2 − 2 2 − 1 3 − 3 3 3 − 3 − 2 − 1 0 − 2 2 − 1 − 2 2 2 3 0 2 − 1 3 1 − 2 − 2 0 2 − 1 − 2 3 − 3 − 3 3 3 − 3 − 2 − 1 0 − 2 2 − 1 − 2 2 2 3 0 2 − 1 3 1 − 2 − 2 0 2 1 − 3 2 − 1 − 2 − 1 3 − 3 − 3 − 3 3 3 − 3 − 2 − 1 0 − 2 2 − 1 − 2 2 2 3 0 2 − 1 3 1 − 2 − 2 0 2 1 − 3 2 − 1 − 2 − 1 1 2 3 − 3 − 3 − 3 0 3 3 − 3 − 2 − 1 0 − 2 2 − 1 − 2 2 2 3 0 2 − 1 3 1 − 2 − 2 0 2 1 − 3 2 − 1 − 2 − 1 1 2 3 − 3 − 3 − 3 0 − 2

Splitting out the pieces of this matrix, we have the decomposition,

L = 1 0 0 0 0 0 − 2 1 0 0 0 0 3 0 1 0 0 0 − 2 − 2 0 1 0 0 2 − 1 − 2 − 1 1 0 3 − 3 − 3 − 3 0 1 U = 3 3 − 3 − 2 − 1 0 0 2 − 1 − 2 2 2 0 0 2 − 1 3 1 0 0 0 2 1 − 3 0 0 0 0 1 2 0 0 0 0 0 − 2

⊠

The hypotheses of Theorem TD can be weakened slightly to include matrices where not every A k is nonsingular. The introduces a rearrangement of the rows and columns of A to force as many as possible of the smaller submatrices to be nonsingular. Then permutation matrices also enter into the decomposition. We will not present the details here, but instead suggest consulting a more advanced text on matrix analysis.

Section SVD  Singular Value Decomposition

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s
N e e d s N u m e r i c a l E x a m p l e s

The singular value decomposition is one of the more useful ways to represent any matrix, even rectangular ones. We can also view the singular values of a (rectangular) matrix as analogues of the eigenvalues of a square matrix. Our definitions and theorems in this section rely heavily on the properties of the matrix-adjoint products ( A ∗ A and A A ∗ ), which we first met in Theorem CPSM. We start by examining some of the basic properties of these two matrices. Now would be a good time to review the basic facts about positive semi-definite matrices in Section PSM.

Parent topic: Chapter MD  Matrix Decompositions

Previous topic: Section TD  Triangular Decomposition

Next topic: Section SR  Square Roots

Subsection MAP: Matrix-Adjoint Product

Theorem  EEMAP
Eigenvalues and Eigenvectors of Matrix-Adjoint Product
Suppose that A is an m × n matrix and A ∗ A has rank r . Let λ 1 , λ 2 , λ 3 , … , λ p be the nonzero distinct eigenvalues of A ∗ A and let ρ 1 , ρ 2 , ρ 3 , … , ρ q be the nonzero distinct eigenvalues of A A ∗ . Then,

□

Proof   Suppose that x ∈ ℂ n is any eigenvector of A ∗ A for a nonzero eigenvalue λ . We will show that A x is an eigenvector of A A ∗ for the same eigenvalue, λ . First, we ascertain that A x is not the zero vector.

A x , A x = A x , A ∗ ∗ x   Theorem AA = A ∗ A x , x   Theorem AIP = λ x , x   Definition EEM = λ x , x   Theorem IPSM

Since x is an eigenvector, x ≠ 0 , and by Theorem PIP, x , x ≠ 0 . As λ was assumed to be nonzero, we see that A x , A x ≠ 0 . Again, Theorem PIP tells us that A x ≠ 0 .

Much of the sequel turns on the following simple computation. If you ever wonder what all the fuss is about adjoints, Hermitian matrices, square roots, and singular values, return to this brief computation, as it holds the key. There is much more to do in this proof, but after this it is mostly bookkeeping. Here we go. We check that A x functions as an eigenvector of A A ∗ for the eigenvalue λ ,

A A ∗ A x = A A ∗ A x   Theorem MMA = A λ x   Definition EEM = λ A x   Theorem MMSMM

That’s it. If x is an eigenvector of A ∗ A (for a nonzero eigenvalue), then A x is an eigenvector for A A ∗ for the same eigenvalue. Let’s see what this buys us.

A ∗ A and A A ∗ are Hermitian matrices ( Definition HM), and hence are normal ( Definition NRML). This provides the existence of orthonormal bases of eigenvectors for each matrix by Theorem OBNM. Also, since each matrix is diagonalizable (Definition DZM) by Theorem OD we can interchange algebraic and geometric multiplicities by Theorem DMFE.

Our first step is to establish that an eigenvalue λ has the same geometric multiplicity for both A ∗ A and A A ∗ . Suppose x 1 , x 2 , x 3 , … , x s is an orthonormal basis of eigenvectors of A ∗ A for the eigenspace ℰ A ∗ A λ . Then for 1 ≤ i < j ≤ s , note

A x i , A x j = A x i , A ∗ ∗ x j   Theorem AA = A ∗ A x i , x j   Theorem AIP = λ x i , x j   Definition EEM = λ x i , x j   Theorem IPSM = λ ( 0 )   Definition ONS = 0   Property ZCN

Then the set E = A x 1 , A x 2 , A x 3 , … , A x s is an orthogonal set of nonzero eigenvectors of A A ∗ for the eigenvalue λ . By Theorem OSLI, the set E is linearly independent and so the geometric multiplicity of λ as an eigenvalue of A A ∗ is s or greater. We have

α A ∗ A λ = γ A ∗ A λ ≤ γ A A ∗ λ = α A A ∗ λ

This inequality applies to any matrix, so long as the eigenvalue is nonzero. We now apply it to the matrix A ∗ ,

α A A ∗ λ = α A ∗ ∗ A ∗ λ ≤ α A ∗ A ∗ ∗ λ = α A ∗ A λ

So for a nonzero eigenvalue, its algebraic multiplicities as an eigenvalue of A ∗ A and A A ∗ are equal. This is enough to establish that p = q and the eigenvalues can be ordered such that λ i = ρ i for 1 ≤ i ≤ p .

For any matrix B , the null space is identical to the eigenspace of the zero eigenvalue, N B = ℰ B 0 , and thus the nullity of the matrix is equal to the geometric multiplicity of the zero eigenvalue. With this, we can examine the ranks of A ∗ A and A A ∗ .

r A ∗ A = n − n A ∗ A   Theorem RPNC = α A ∗ A 0 + ∑ i = 1 p α A ∗ A λ i − n A ∗ A   Theorem NEM = α A ∗ A 0 + ∑ i = 1 p α A ∗ A λ i − γ A ∗ A 0   Definition GME = α A ∗ A 0 + ∑ i = 1 p α A ∗ A λ i − α A ∗ A 0   Theorem DMFE = ∑ i = 1 p α A ∗ A λ i = ∑ i = 1 p α A A ∗ λ i = α A A ∗ 0 + ∑ i = 1 p α A A ∗ λ i − α A A ∗ 0 = α A A ∗ 0 + ∑ i = 1 p α A A ∗ λ i − γ A A ∗ 0   Theorem DMFE = α A A ∗ 0 + ∑ i = 1 p α A A ∗ λ i − n A A ∗   Definition GME = m − n A A ∗   Theorem NEM = r A A ∗   Theorem RPNC

When A is rectangular, the square matrices A ∗ A and A A ∗ have different sizes. With equal algebraic and geometric multiplicities for their common nonzero eigenvalues, the difference in their sizes is manifest in different algebraic multiplicities for the zero eigenvalue and different nullities. Specifically,

n A ∗ A = n − r n A A ∗ = m − r

Suppose that x 1 , x 2 , x 3 , … , x n is an orthonormal basis of ℂ n composed of eigenvectors of A ∗ A and ordered so that x i , r + 1 ≤ i ≤ n are eigenvectors of A A ∗ for the zero eigenvalue. Denote the associated nonzero eigenvalues of A ∗ A for these eigenvectors by δ i , 1 ≤ i ≤ r . Then define

y i = 1 δ i A x i 1 ≤ i ≤ r

Let y r + 1 , y r + 2 , y r + 2 , … , y m be an orthonormal basis for the eigenspace ℰ A A ∗ 0 , whose existence is guaranteed by Theorem GSP. As scalar multiples of demonstrated eigenvectors of A A ∗ , y i , 1 ≤ i ≤ r are also eigenvectors of A A ∗ , and y i , r + 1 ≤ i ≤ n have been chosen as eigenvectors of A A ∗ . These eigenvectors also have norm 1, as we now show. For 1 ≤ i ≤ r ,

y i = 1 δ i A x i = 1 δ i A x i , 1 δ i A x i   Theorem IPN = 1 δ i 1 δ i ¯ A x i , A x i   Theorem IPSM = 1 δ i 1 δ i A x i , A x i   Theorem HMRE = 1 δ i A x i , A x i = 1 δ i A x i , A ∗ ∗ x i   Theorem AA = 1 δ i A ∗ A x i , x i   Theorem AIP = 1 δ i δ i x i , x i   Definition EEM = 1 δ i δ i x i , x i   Theorem IPSM = 1 δ i δ i ( 1 )   Definition ONS = 1

For r + 1 ≤ i ≤ n , the y i have been chosen to have norm 1.

Finally we check orthogonality. Consider two eigenvectors y i and y j with 1 ≤ i < j ≤ m . If these two vectors have different eigenvalues, then Theorem HMOE establishes that the two eigenvectors are orthogonal. If the two eigenvectors have a zero eigenvalue, then they are orthogonal by the choice of the orthonormal basis of ℰ A A ∗ 0 . If the two eigenvectors have identical, nonzero, eigenvalues, then

y i , y j = 1 δ i A x i , 1 δ j A x j = 1 δ i 1 δ j ¯ A x i , A x j   Theorem IPSM = 1 δ i δ j A x i , A x j   Theorem HMRE = 1 δ i δ j A x i , A ∗ ∗ x j   Theorem AA = 1 δ i δ j A ∗ A x i , x j   Theorem AIP = 1 δ i δ j δ i x i , x j   Definition EEM = δ i δ i δ j x i , x j   Theorem IPSM = δ i δ i δ j ( 0 )   Definition ONS = 0

So y 1 , y 2 , y 3 , … , y m is an orthonormal set of eigenvectors for A A ∗ . The critical relationship between these two orthonormal bases is present by design. For 1 ≤ i ≤ r ,

A x i = δ i 1 δ i A x i = δ i y i

For r + 1 ≤ i ≤ n we have

A x i , A x i = A x i , A ∗ ∗ x i   Theorem AA = A ∗ A x i , x i   Theorem AIP = 0 , x i   Definition EEM = 0   Definition IP

So by Theorem PIP, A x i = 0 . ■

Subsection SVD: Singular Value Decomposition

The square roots of the eigenvalues of A ∗ A (or almost equivalently, A A ∗ !) are known as the singular values of A . Here is the definition.

Definition  SV
Singular Values
Suppose A is an m × n matrix. If the eigenvalues of A ∗ A are δ 1 , δ 2 , δ 3 , … , δ n , then the singular values of A are δ 1 , δ 2 , δ 3 , … , δ n . △

Theorem EEMAP is a total setup for the singular value decomposition. This remarkable theorem says that any matrix can be broken into a product of three matrices. Two are square, and unitary. In light of Theorem UMPIP, we can view these matrices as transforming vectors or coordinates in a rotational fashion. The middle matrix of this decomposition is rectangular, but is as close to being diagonal as a rectangular matrix can be. Viewed as a transformation, this matrix effects, reflections, contractions or expansions along axes — it stretches vectors. So any matrix, viewed as a transformation is the product of a rotation, a stretch and a rotation.

The singular value theorem can also be viewed as an application of our most general statement about matrix representations of linear transformations relative to different bases. Theorem MRCB concerns linear transformations T : U → V where U and V are possibly different vector spaces. When U and V have different dimensions, the resulting matrix representation will be rectangular. In Section CB we quickly specialized to the case where U = V and the matrix representations are square with one of our most central results, Theorem SCB. Theorem SVD is an application of the full generality of Theorem MRCB where the relevant bases are now orthonormal sets.

Theorem  SVD
Singular Value Decomposition
Suppose A is an m × n matrix of rank r with nonzero singular values s 1 , s 2 , s 3 , … , s r . Then A = U D V ∗ where U is a unitary matrix of size m , V is a unitary matrix of size n and D is an m × n matrix given by

D i j = s i  if  1 ≤ i = j ≤ r 0  otherwise

□

Proof   Let x 1 , x 2 , x 3 , … , x n and y 1 , y 2 , y 3 , … , y m be the orthonormal bases described by the conclusion of Theorem EEMAP. Define U to be the m × m matrix whose columns are y i , 1 ≤ i ≤ m , and define V to be the n × n matrix whose columns are x i , 1 ≤ i ≤ n . With orthonormal sets of columns, by Theorem CUMOS both U and V are unitary matrices.

Then for 1 ≤ i ≤ m , 1 ≤ j ≤ n ,

A V i j = A x j i   Definition MM = δ j y j i   Theorem EEMAP = s j y j i   Definition SV = y j i s j   Definition CVSM = U i j D j j = ∑ k = 1 m U i k D k j = U D i j   Theorem EMP

So by Theorem ME, A V = U D and thus

A = A I n = A V V ∗ = U D V ∗

■

Section SR  Square Roots

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s
N e e d s N u m e r i c a l E x a m p l e s

With all our results about Hermitian matrices, their eigenvalues and their diagonalizations, it will be a nearly trivial matter to now construct a “square root” of a positive semi-definite matrix. We will describe the square root of a matrix A as a matrix S such that A = S 2 . In general, a matrix A might have many such square roots. But with a few results in hand we will be able to impose an extra condition on S that will make a unique S such that A = S 2 . At that point we can define the square root of A formally.

Parent topic: Chapter MD  Matrix Decompositions

Previous topic: Section SVD  Singular Value Decomposition

Next topic: Section POD  Polar Decomposition

Subsection SRM: Square Root of a Matrix

Theorem  PSMSR
Positive Semi-Definite Matrices and Square Roots
Suppose A is a square matrix. There is a positive semi-definite matrix S such that A = S 2 if and only if A is positive semi-definite. □

Proof   Let n denote the size of A .

( ⇐ ) Suppose that A is positive semi-definite. Since A is Hermitian ( Definition PSM) we know A is normal ( Definition NRML) and so by Theorem OD there is a unitary matrix U and a diagonal matrix D , whose diagonal entries are the eigenvalues of A , such that D = U ∗ A U . The eigenvalues of A are all non-negative ( Theorem EPSM), which allows us to define a diagonal matrix E whose diagonal entries are the positive square roots of the eigenvalues of A , in the same order as they appear in D . More precisely, define E to be the diagonal matrix with non-negative diagonal entries such that E 2 = D . Set S = U E U ∗ , and compute

S 2 = U E U ∗ U E U ∗ = U E I n E U ∗   Definition UM = U E E U ∗   Theorem MMIM = U D U ∗ = U U ∗ A U U ∗   Theorem OD = I n A I n   Definition UM = A   Theorem MMIM

We need to first verify that S is Hermitian.

S ∗ = U E U ∗ ∗ = U E U ∗ ∗ = U ∗ ∗ E ∗ U ∗   Theorem MMAD = U E ∗ U ∗   Theorem AA = U E ¯ t U ∗   Definition A = U E t U ∗   Theorem HMRE = U E U ∗  Diagonal matrix = S

And finally, we want to check the use of S in an inner product. Notice that E is Hermitian since it is a diagonal matrix with real entries. Furthermore, as a diagonal matrix, the eigenvalues of E are precisely the diagonal entries, and since these were chosen to be positive, an application of Theorem EPSM tells us that E is positive semi-definite. Now, for any x ∈ ℂ n ,

S x , x = U E U ∗ x , x = E U ∗ x , U ∗ x   Theorem AIP = E U ∗ x , U ∗ x ≥ 0   Definition PSM

So, according to Definition PSM, S is positive semi-definite.

( ⇒ ) Assume that A = S 2 , with S positive semi-definite. Then S is Hermitian, and we check that A is Hermitian.

A ∗ = S S ∗ = S ∗ S ∗   Theorem MMAD = S S   Definition HM = A

Now for the use of A in an inner product. For any x ∈ ℂ n ,

A x , x = S 2 x , x = S x , S ∗ x   Theorem AIP = S x , S x   Definition HM ≥ 0   Theorem PIP

So by Definition PSM, A is positive semi-definite. ■

There is a very close relationship between the eigenvalues and eigenspaces of a positive semi-definite matrix and its positive semi-definite square root. The next theorem is interesting in its own right, but is also an important technical step in some other important results, such as the upcoming uniqueness of the square root ( Theorem USR).

Theorem  EESR
Eigenvalues and Eigenspaces of a Square Root
Suppose that A is a positive semi-definite matrix and S is a positive semi-definite matrix such that A = S 2 . If λ 1 , λ 2 , λ 3 , … , λ p are the distinct eigenvalues of A , then the distinct eigenvalues of S are λ 1 , λ 2 , λ 3 , … , λ p , and ℰ S λ i = ℰ A λ i for 1 ≤ i ≤ p . □

Proof   Let x be an eigenvector of S for an eigenvalue ρ . Then, in the style of Theorem EPM,

A x = S 2 x = S S x = S ρ x = ρ S x = ρ 2 x

so ρ 2 is an eigenvalue of A and must equal some λ i . Furthermore, because S is positive semi-definite, Theorem EPSM tells us that ρ ≥ 0 . The impact for us here is that we cannot have two different eigenvalues of S whose squares equal the same eigenvalue of A , so we can pair each eigenvalue of S with a different eigenvalue of A , equal to its square. (A good exercise is to track through the rest of this proof in the situation where S is not assumed to be positive semi-definite and we do not have this condition on the eigenvalues. Where does the proof then break down?) Let ρ i , 1 ≤ i ≤ q denote the q distinct eigenvalues of S . The discussion above implies that we can order the eigenvalues of A and S so that λ i = ρ i 2 for 1 ≤ i ≤ q . Notice that at this point we know that q ≤ p , though we will be showing that q = p .

Additionally, the equation above tells us that every eigenvector of S for ρ i is again an eigenvector of A for ρ i 2 . So for 1 ≤ i ≤ q , the relevant eigenspaces are related by

ℰ S λ i = ℰ S ρ i ⊆ ℰ A ρ i 2 = ℰ A λ i

So the eigenspaces of S are subsets of the eigenspaces of A , for the related eigenvalues. However, we will be showing that these sets are indeed equal to each other.

Both A and S are positive semi-definite, hence Hermitian and therefore normal. Theorem OD then tells us that each is diagonalizable (Definition DZM). Then Theorem DMFE says that the algebraic multiplicity and geometric multiplicity of each eigenvalue are equal. Then, if we let n denote the size of A ,

n = ∑ i = 1 q α S λ i   Theorem NEM = ∑ i = 1 q γ S λ i   Theorem DMFE = ∑ i = 1 q dim ℰ S λ i   Definition GME ≤ ∑ i = 1 q dim ℰ A λ i   Theorem PSSD ≤ ∑ i = 1 p dim ℰ A λ i   Definition D = ∑ i = 1 p γ A λ i   Definition GME = ∑ i = 1 p α A λ i   Theorem DMFE = n   Theorem NEM

With equal values at the two ends of this chain of equalities and inequalities, we know that the two inequalities are forced to actually be equalities. In particular, the second inequality implies that p = q and the first, in conjunction with Theorem EDYES, implies that ℰ S λ i = ℰ A λ i for 1 ≤ i ≤ p . ■

Notice that we defined the singular values of a matrix A as the square roots of the eigenvalues of A ∗ A ( Definition SV). With Theorem EESR in hand we recognize the singular values of A as simply the eigenvalues of A ∗ A 1 ∕ 2 . Indeed, many authors take this as the definition of singular values, since it is equivalent to our definition. We have chosen not to wait for a discussion of square roots before making a definition of singular values, allowing us to present the singular value decomposition ( Theorem SVD) all the sooner.

In the first half of the proof of Theorem PSMSR we could have chosen the matrix E (which was the essential component of the desired matrix S ) in a variety of ways. Any collection of diagonal entries of E could be replaced by their negatives and we would maintain the property that E 2 = D . However, if we decide to enforce the entries of E as non-negative quantities then E is positive semi-definite, and then S follows along as a positive semi-definite matrix. We now show that of all the possible square roots of a positive semi-definite matrix, only one is itself again positive semi-definite. In other words, the S of Theorem PSMSR is unique.

Theorem  USR
Unique Square Root
Suppose A is a positive semi-definite matrix. Then there is a unique positive semi-definite matrix S such that A = S 2 . □

Proof   Theorem PSMSR gives us the existence of at least one positive semi-definite matrix S such that A = S 2 . As usual, we will assume that S 1 and S 2 are positive semi-definite matrices such that A = S 1 2 = S 2 2 ( Technique U).

As A is diagonalizable, there is a basis of ℂ n composed entirely of eigenvectors of A (Theorem DC), say B = x 1 , x 2 , x 3 , … , x n . Let δ 1 , δ 2 , δ 3 , … , δ n denote the associated eigenvalues. Theorem EESR allows to conclude that ℰ A δ i = ℰ S 1 δ i = ℰ S 2 δ i . So S 1 x i = δ i x i = S 2 x i for 1 ≤ i ≤ n .

Choose any x ∈ ℂ n . The spanning property of B allows us to conclude the existence of a set of scalars, a 1 , a 2 , a 3 , … , a n , yielding x as a linear combination of the vectors in B . So,

S 1 x = S 1 ∑ i = 1 n a i x i = ∑ i = 1 n a i S 1 x i = ∑ i = 1 n a i δ i x i = ∑ i = 1 n a i S 2 x i = S 2 ∑ i = 1 n a i x i = S 2 x

Since S 1 and S 2 have the same action on every vector, Theorem EMMVP yields the conclusion that S 1 = S 2 . ■

With a criteria that distinguishes one square root from all the rest (positive semi-definiteness) we can now define the square root of a positive semi-definite matrix.

Definition  SRM
Square Root of a Matrix
Suppose A is a positive semi-definite matrix and S is the positive semi-definite matrix such that S 2 = S S = A . Then S is the square root of A and we write S = A 1 ∕ 2 .

(This definition contains Notation SRM.) △

Section POD  Polar Decomposition

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s
N e e d s N u m e r i c a l E x a m p l e s

The polar decomposition of a matrix writes any matrix as the product of a unitary matrix ( Definition UM)and a positive semi-definite matrix ( Definition PSM). It takes its name from a special way to write complex numbers. If you’ve had a basic course in complex analysis, the next paragraph will help explain the name. If the next paragraph makes no sense to you, there’s no harm in skipping it.

Any complex number z ∈ ℂ can be written as z = r e i θ where r is a positive number (computed as a square root of a function of the real amd imaginary parts of z ) and θ is an angle of rotation that converts 1 to the complex number e i θ = cos ( θ ) + i sin ( θ ) . The polar form of a square matrix is a product of a positive semi-definite matrix that is a square root of a function of the matrix together with a unitary matrix, which can be viewed as achieving a rotation ( Theorem UMPIP).

OK, enough preliminaries. We have all the tools in place to jump straight to our main theorem.

Theorem  PDM
Polar Decomposition of a Matrix
Suppose that A is a square matrix. Then there is a unitary matrix U such that A = A A ∗ 1 ∕ 2 U . □

Proof   This theorem only claims the existence of a unitary matrix U that does a certain job. We will manufacture U and check that it meets the requirements.

Suppose A has size n and rank r . We begin by applying Theorem EEMAP to A . Let B = x 1 , x 2 , x 3 , … , x n be the orthonormal basis of ℂ n composed of eigenvectors for A ∗ A , and let C = y 1 , y 2 , y 3 , … , y n be the orthonormal basis of ℂ n composed of eigenvectors for A A ∗ . We have A x i = δ i x i , 1 ≤ i ≤ r , and A x i = 0 , r + 1 ≤ i ≤ n , where δ i , 1 ≤ i ≤ r are the distinct nonzero eigenvalues of A ∗ A .

Define T : ℂ n → ℂ n to be the unique linear transformation such that T x i = y i , 1 ≤ i ≤ n , as guaranteed by Theorem LTDB. Let E be the basis of standard unit vectors for ℂ n ( Definition SUV), and define U to be the matrix representation (Definition MR) of T with respect to E , more carefully U = M E , E T . This is the matrix we are after. Notice that

U x i = M E , E T ρ E x i   Definition VR = ρ E T x i   Theorem FTMR = ρ E y i   Theorem FTMR = y i   Definition VR

Since B and C are orthonormal bases, and C is the result of multiplying the vectors of B by U , we conclude that U is unitary by Theorem UMCOB. So once again, Theorem EEMAP is a big part of the setup for a decomposition.

Let x ∈ ℂ n be any vector. Since B is a basis of ℂ n , there are scalars a 1 , a 2 , a 3 , … , a n expressing x as a linear combination of the vectors in B . then

A A ∗ 1 ∕ 2 U x = A A ∗ 1 ∕ 2 U ∑ i = 1 n a i x i   Definition B = ∑ i = 1 n A A ∗ 1 ∕ 2 U a i x i   Theorem MMDAA = ∑ i = 1 n a i A A ∗ 1 ∕ 2 U x i   Theorem MMSMM = ∑ i = 1 n a i A A ∗ 1 ∕ 2 y i = ∑ i = 1 r a i A A ∗ 1 ∕ 2 y i + ∑ i = r + 1 n a i A A ∗ 1 ∕ 2 y i   Property AAC = ∑ i = 1 r a i δ i y i + ∑ i = r + 1 n a i ( 0 ) y i   Theorem EESR = ∑ i = 1 r a i δ i y i + ∑ i = r + 1 n a i 0   Theorem ZSSM = ∑ i = 1 r a i A x i + ∑ i = r + 1 n a i A x i   Theorem EEMAP = ∑ i = 1 n a i A x i   Property AAC = ∑ i = 1 n A a i x i   Theorem MMSMM = A ∑ i = 1 n a i x i   Theorem MMDAA = A x

So by Theorem EMMVP we have the matrix equality A A ∗ 1 ∕ 2 U = A . ■

Parent topic: Chapter MD  Matrix Decompositions

Previous topic: Section SR  Square Roots

Next topic: Part A  Applications

Part III. Part A  Applications

Section CF  Curve Fitting

Section SAS  Sharing A Secret

Previous topic: Section POD  Polar Decomposition

Next topic: Section CF  Curve Fitting

Chapter 19. Section CF  Curve Fitting

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s I n c o m p l e t e

Given two points in the plane, there is a unique line through them. Given three points in the plane, and not in a line, there is a unique parabola through them. Given four points in the plane, there is a unique polynomial, of degree 3 or less, passing through them. And so on. We can prove this result, and give a procedure for finding the polynomial with the help of Vandermonde matrices ( Section VM).

Theorem  IP
Interpolating Polynomial
Suppose ( x i , y i ) 1 ≤ i ≤ n + 1 is a set of n + 1 points in the plane where the x -coordinates are all different. Then there is a unique polynomial of degree n or less, p ( x ) , such that p ( x i ) = y i , 1 ≤ i ≤ n + 1 . □

Proof   Write p ( x ) = a 0 + a 1 x + a 2 x 2 + ⋯ + a n x n . To meet the conclusion of the theorem, we desire,

y i = p ( x i ) = a 0 + a 1 x i + a 2 x i 2 + ⋯ + a n x i n 1 ≤ i ≤ n + 1

This is a system of n + 1 linear equations in the n + 1 variables a 0 , a 1 , a 2 , … , a n . The vector of constants in this system is the vector containing the y -coordinates of the points. More importantly, the coefficient matrix is a Vandermonde matrix ( Definition VM) built from the x -coordinates x 1 , x 2 , x 3 , … , x n + 1 . Since we have required that these scalars all be different, Theorem NVM tells us that the coefficient matrix is nonsingular and Theorem NMUS says the solution for the coefficients of the polynomial exists, and is unique. As a practical matter, Theorem SNCM provides an expression for the solution. ■

Example  PTFP
Polynomial through five points
Suppose we have the following 5 points in the plane and we wish to pass a degree 4 polynomial through them.

The required system of equations has a coefficient matrix that is the Vandermonde matrix where row i is successive powers of x i

A = 1 − 3 9 − 2 7 8 1 1 − 1 1 − 1 1 1 2 4 8 1 6 1 3 9 2 7 8 1 1 6 3 6 2 1 6 1 2 9 6

Theorem NMUS provides a solution as

a 0 a 1 a 2 a 3 a 4 = A − 1 2 7 6 1 6 3 1 1 4 4 2 3 1 9 = − 1 1 5 9 1 4 9 1 0 − 1 2 1 4 2 0 − 3 7 3 4 − 1 3 1 8 4 5 1 0 8 − 1 5 6 − 1 4 1 7 7 2 − 1 1 7 5 6 − 1 5 4 1 2 1 − 1 1 2 1 1 8 − 1 7 5 6 1 5 4 0 − 1 1 6 8 1 6 0 − 1 7 2 1 7 5 6 2 7 6 1 6 3 1 1 4 4 2 3 1 9 = 3 − 4 5 − 2 2

So the polynomial is p ( x ) = 3 − 4 x + 5 x 2 − 2 x 3 + 2 x 4 . ⊠

The unique polynomial passing through a set of points is known as the interpolating polynomial and it has many uses. Unfortunately, when confronted with data from an experiment the situation may not be so simple or clear cut. Read on.

Parent topic: Part A  Applications

Previous topic: Part A  Applications

Next topic: Section SAS  Sharing A Secret

Subsection DF: Data Fitting

Suppose that we have n real variables, x 1 , x 2 , x 3 , … , x n , that we can measure in an experiment. We believe that these variables combine, in a linear fashion, to equal another real variable, y . In other words, we have reason to believe from our understanding of the experiment, that

y = a 1 x 1 + a 2 x 2 + a 3 x 3 + ⋯ + a n x n

where the scalars a 1 , a 2 , a 3 , … , a n are not known to us, but are instead desirable. We would call this our model of the situation. Then we run the experiment m times, collecting sets of values for the variables of the experiment. For run number k we might denote these values as y k , x k 1 , x k 2 , x k 3 , …, x k n . If we substitute these values into the model equation, we get m linear equations in the unknown coefficients a 1 , a 2 , a 3 , … , a n . If m = n , then we have a square coefficient matrix of the system which might happen to be nonsingular and there would be a unique solution.

However, more likely m > n (the more data we collect, the greater our confidence in the results) and the resulting system is inconsistent. It may be that our model is only an approximate understanding of the relationship between the x i and y , or our measurements are not completely accurate. Still we would like to understand the situation we are studying, and would like some best answer for a 1 , a 2 , a 3 , … , a n .

Let y denote the vector with y i = y i , 1 ≤ i ≤ m , let a denote the vector with a j = a j , 1 ≤ j ≤ n , and let X denote the m × n matrix with X i j = x i j , 1 ≤ i ≤ m , 1 ≤ j ≤ n . Then the model equation, evaluated with each run of the experiment, translates to X a = y . With the presumption that this system has no solution, we can try to minimize the difference between the two side of the equation y − X a . As a vector, it is hard to imagine what the minimum might be, so we instead minimize the square of its norm

S = y − X a t y − X a

To keep the logical flow accurate, we will define the minimizing value and then give the proof that it behaves as desired.

Definition  LSS
Least Squares Solution
Given the equation X a = y , where X is an m × n matrix of rank n , the least squares solution for a is X t X − 1 X t y . △

Theorem  LSMR
Least Squares Minimizes Residuals
Suppose that X is an m × n matrix of rank n . The least squares solution of X a = y , a ′ = X t X − 1 X t y , minimizes the expression

S = y − X a t y − X a

□

Proof   We begin by finding the critical points of S . In preparation, let X j denote column j of X , for 1 ≤ j ≤ n and compute partial derivatives with respect to a j , 1 ≤ j ≤ n . A matrix product of the form x t y is a sum of products, so a derivative is a sum of applications of the product rule,

∂ ∂ a j S = ∂ ∂ a j y − X a t y − X a = ∑ i = 1 m ∂ ∂ a j y − X a i y − X a i + y − X a i ∂ ∂ a j y − X a i = 2 ∑ i = 1 m ∂ ∂ a j y − X a i y − X a i = 2 ∑ i = 1 m ∂ ∂ a j y i − ∑ k = 1 n X i k a k y − X a i = 2 ∑ i = 1 m − X i j y − X a i = − 2 X j t y − X a

The first partial derivatives will allow us to find critical points, while second partial derivatives will be needed to confirm that a critical point will yield a minimum. Return to the next-to-last expression for the first partial derivative of S ,

∂ ∂ a ℓ a j S = ∂ ∂ a ℓ 2 ∑ i = 1 m − X i j y − X a i = − 2 ∑ i = 1 m ∂ ∂ a ℓ X i j y − X a i = − 2 ∑ i = 1 m X i j ∂ ∂ a ℓ y i − ∑ k = 1 n X i k a k = − 2 ∑ i = 1 m X i j − X i ℓ = 2 ∑ i = 1 m X i j X i ℓ = 2 ∑ i = 1 m X t j i X i ℓ = 2 X t X j ℓ

For 1 ≤ j ≤ n , set ∂ ∂ a j S = 0 . This results in the n scalar equations

X j t X a = X j t y 1 ≤ j ≤ n

These n vector equations can be summarized in the single vector equation,

X t X a = X t y

X t X is an n × n matrix and since we have assumed that X has rank n , X t X will also have rank n . Since X t X is invertible, we have a critical point at

a ′ = X t X − 1 X t y

Is this lone critical point really a minimum? The matrix of second partial derivatives is constant, and a positive multiple of X t X . Theorem CPSM tells us that this matrix is positive semi-definite. In an advanced course on multivariable calculus, it is shown that a minimum occurs exactly where the matrix of second partial derivatives is positive semi-definite. You may have seen this in the two-variable case, where a check on the positive semi-definiteness is disguised with a determinant of the 2 × 2 matrix of second partial derivatives. ■

Subsection EXC: Exercises

T20 Theorem IP constructs a unique polynomial through a set of n + 1 points in the plane, ( x i , y i ) 1 ≤ i ≤ n + 1 , where the x -coordinates are all different. Prove that the expression below is the same polynomial and include an explanation of the necessity of the hypothesis that the x -coordinates are all different.

p ( x ) = ∑ i = 1 n + 1 y i ∏ j = 1 j ≠ i n + 1 x − x j x i − x j

This is known as the Lagrange form of the interpolating polynomial.  
Contributed by  Robert Beezer

Chapter 20. Section SAS  Sharing A Secret

From A First Course in Linear Algebra
Version 2.30
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

T h i s S e c t i o n i s a D r a f t, S u b j e c t t o C h a n g e s

In this section we will see how to use solutions to systems of equations to share a secret among a group of people. We will be able to break a secret up into, say 10 pieces, so as to distribute the secret among 10 people. But rather than requiring all 10 people to collaborate on restoring the secret, we can design the split so that any smaller group, of say just 4 of these people, can collaborate and restore the secret. The numbers 10 and 4 here are arbitrary, we can choose them to be anything.

Suppose we have a secret, S . This could be the combination to a lock, a password on an account, or a recipe for chocolate chip cookies. If the secret is text, we will assume that the characters have been translated into integers (say with the ASCII code), and these numbers have been rolled up into one grand positive integer (perhaps by concatenating binary strings for the ASCII code numbers, and interpreting the longer string as one big base 2 integer). So we will assume S is some positive integer.

Suppose you wish to give parts of your secret to n people, and you wish to require that any group of m (or more) of these people should be able to combine their parts and recover the secret. Perhaps you are President and CEO of a small company and only you know the password that authorizes large transfers of money among the company’s bank accounts. If you were to die or become incapacitated, it would perhaps hamper the company’s ability to function if they couldn’t quickly rearrange their assets, especially since they are also without a CEO. So you might wish to give this secret to six of your trusted Vice-Presidents. But you don’t trust them that much and you certainly don’t want any one of these people to be able to access the company’s accounts all by themselves without anybody else in the company knowing about it. Simultaneously, you know that in an emergency, it might not be possible to get all six Vice-Presidents together and maybe even one or two of them have met the same unfortunate fate you did. So you would like any group of three Vice-Presidents to be able to combine their parts and recover S . So you would choose n = 6 and m = 3 .

We will describe the split, with no motivation. The explanation of how the secret recovery is handled will explain our choices here. Choose a large prime number, p , bigger than any possible secret. For a single number in a combination lock, p could be small. For a one-page recipe, p would need to be huge. All of our subsequent arithmetic will be modulo p , so consult Subsection F.FF for a brief description of how we do linear algebra when our field is ℤ p . Build a polynomial, r ( x ) , of degree m − 1 as follows. Set the constant term to S , and choose the other m − 1 coefficients at random from ℤ p . The quality of your random generator will ultimately affect the quality of how hidden your secret remains.

Compute the pairs ( i , r ( i ) ) , 1 ≤ i ≤ n . To person i , of the n persons you will give a part of your secret, present the pair ( i , r ( i ) ) , and instruct them to keep this secret, for all 1 ≤ i ≤ n . They could perhaps encrypt their pairs with AES (Advanced Encryption Standard) using a password known only to them individually. Or you could do this for each of them in advance and tell them the chose password orally, in private. At any rate, each person gets a pair of integers, an input to the polynomial, and the output of evaluating the polynomial, and they keep this information secret. They do not know the polynomial itself, and certainly not the constant term S , so the secret is still safe.

Now suppose that m of these people get together, in the event you are unable to act, or perhaps without your permission. Suppose they pool all of their pairs, or even just turn them over to one member of the group. What do they now know collectively? Suppose that

r ( x ) = a 0 + a 1 x + a 2 x 2 + ⋯ + a m − 1 x m − 1

where, of course, a 0 = S is the secret. A single pair, ( i , r ( i ) ) , results in a linear equation whose unknowns are the m coefficients of r ( x ) . With m pairs revealed, we now have m equations in m variables. Furthermore, the coefficient matrix of this system is a Vandermonde matrix ( Definition VM). With our inputs to the polynomial all different (we used 1 , 2 , 3 , … , n ), the Vandermonde matrix is nonsingular ( Theorem NVM). Thus by Theorem NMUS there is a unique solution for the coefficients of r ( x ) . We only desire the constant term — the other coefficients (the randomly chosen ones) are of no interest, they were used to mask the secret as it was split into parts.

A few practical considerations. If certain individuals in your group are more important, or more trustworthy, you can give them more than one part. You could split a secret into 30 parts, giving 5 Vice-Presidents each 4 parts and give 10 department heads each 1 part. Then you might require 12 parts to be present. This way three Vice-Presidents could recover the secret, or 4 department heads could stand-in for a Vice-President. Furthermore, the 10 department heads could not recover the secret without having at least one Vice-President present.

The inputs do not have to be consecutive integers, starting at 1. Any set of different integers will suffice. Why make it any easier for an attacker? Mix it up and choose the inputs randomly as well, just keep them different.

Why do all this arithmetic over ℤ p ? If we worked with polynomials having real number coefficients, properties of polynomials as continuous functions might give an attacker the ability to compute the secret with a reasonable amount of computing time. For example, the magnitude of the output is going to dominated by the term of r ( x ) having degree m − 1 . Suppose an attacker had a few of the pairs, but not a full set of m of them. Or even worse, suppose some group of fewer than m of your trusted acquaintances were to conspire against you. It might be possible to guess a limited range of values for the coefficient of the largest term. With a limited range of values here, the next term might fall to a similar analysis. And so on. However, modular arithmetic is in some ways very unpredictable looking and as high powers “wrap-around” this sort of analysis will be frustrated. And we know it is no harder to do linear algebra in ℤ p than in ℂ .

OK, here’s a non-trivial example.

Example  SS6W
Sharing a secret 6 ways
Let’s return to the CEO and his six Vice-Presidents. Suppose the password for the company’s accounts is a sequence of 5 two-digit numbers, which we will concatenate into a 10-digit number, in this case S = 0 6 0 3 7 2 5 9 6 2 . For a prime p we choose the 11-digit prime number p = 2 2 8 0 1 7 6 1 3 7 9 . From the requirement that m = 3 Vice-Presidents are needed to recover the secret, we need a second-degree polynomial and so need two more coefficients, which we will construct at random between 1 and p . The resulting polynomial is

r ( x ) = 6 0 3 7 2 5 9 6 2 + 2 2 5 6 1 9 8 2 9 1 9 x + 8 8 4 4 0 8 8 3 3 8 x 2

We will now build six pairs of inputs and outputs, where we will choose the inputs at random (not allowing duplicates) and we do all our arithmetic modulo p ,

The two numbers of each row of the table are then given to the indicated Vice-President. Done. The secret has been split six ways, and any three VP’s can jointly recover the secret.

Let’s test the recovery process, especially since it contains the relevant linear algebra. Suppose we write the unknown polynomial as r ( x ) = a 0 + a 1 x + a 2 x 2 and the VP’s for Finance, Marketing and Legal all get together to recover the secret. The equations we arrive at are,

 Finance 7 2 0 5 6 9 9 6 5 4 = r ( 2 0 2 2 0 4 0 6 0 4 6 ) = a 0 + a 1 ( 2 0 2 2 0 4 0 6 0 4 6 ) + a 2 ( 2 0 2 2 0 4 0 6 0 4 6 ) 2 = a 0 + 2 0 2 2 0 4 0 6 0 4 6 a 1 + 7 7 9 3 5 9 6 2 1 5 a 2  Marketing 1 8 5 0 3 1 5 8 0 7 9 = r ( 1 3 7 4 7 1 2 7 9 5 7 ) = a 0 + a 1 ( 1 3 7 4 7 1 2 7 9 5 7 ) + a 2 ( 1 3 7 4 7 1 2 7 9 5 7 ) 2 = a 0 + 1 3 7 4 7 1 2 7 9 5 7 a 1 + 1 8 8 4 0 3 0 1 3 7 0 a 2  Legal 1 4 0 6 0 7 0 5 9 9 9 = r ( 1 5 8 3 5 1 2 0 3 1 9 ) = a 0 + a 1 ( 1 5 8 3 5 1 2 0 3 1 9 ) + a 2 ( 1 5 8 3 5 1 2 0 3 1 9 ) 2 = a 0 + 1 5 8 3 5 1 2 0 3 1 9 a 1 + 8 8 7 4 4 1 2 9 9 9 a 2

So they have a linear system, ℒ S A , b with

A = 1 2 0 2 2 0 4 0 6 0 4 6 7 7 9 3 5 9 6 2 1 5 1 1 3 7 4 7 1 2 7 9 5 7 1 8 8 4 0 3 0 1 3 7 0 1 1 5 8 3 5 1 2 0 3 1 9 8 8 7 4 4 1 2 9 9 9 b = 7 2 0 5 6 9 9 6 5 4 1 8 5 0 3 1 5 8 0 7 9 1 4 0 6 0 7 0 5 9 9 9

With a Vandermonde matrix as the coefficient matrix, they know there is a solution, and it is unique. By Theorem SNCM (or through row-reducing the augmented matrix) they arrive at the solution,

A − 1 b = 5 7 1 6 9 0 0 8 7 9 9 2 3 4 4 3 7 6 4 6 7 8 5 0 4 2 2 8 5 5 2 0 9 5 2 2 0 0 7 4 7 1 6 4 5 2 5 9 5 9 2 2 8 1 9 8 7 2 6 0 8 9 1 7 2 8 6 9 4 3 7 9 6 1 8 0 1 8 2 4 1 5 9 7 1 0 2 9 8 3 3 7 3 6 5 7 2 0 5 6 9 9 6 5 4 1 8 5 0 3 1 5 8 0 7 9 1 4 0 6 0 7 0 5 9 9 9 = 6 0 3 7 2 5 9 6 2 2 2 5 6 1 9 8 2 9 1 9 8 8 4 4 0 8 8 3 3 8

So the CEO’s password is the secret S = a 0 = 6 0 3 7 2 5 9 6 2 = 0 6 0 3 7 2 5 9 6 2 (as expected). ⊠

Parent topic: Part A  Applications

Previous topic: Section CF  Curve Fitting

Next topic: Contributors

Chapter 21. Contributors

Contributors

Beezer, David. Belarmine Preparatory School, Tacoma
Beezer, Robert. University of Puget Sound http://buzzard.ups.edu/
Black, Chris.
Braithwaite, David. Chicago, Illinois
Bucht, Sara. University of Puget Sound
Canfield, Steve. University of Puget Sound
Hubert, Dupont. Créteil, France
Fellez, Sarah. University of Puget Sound
Fickenscher, Eric. University of Puget Sound
Jackson, Martin. University of Puget Sound http://www.math.ups.edu/˜martinj
Kessler, Ivan. University of Puget Sound
Kreher, Don. Michigan Technological University http://www.math.mtu.edu/˜kreher/
Hamrick, Mark. St. Louis University
Linenthal, Jacob. University of Puget Sound
Million, Elizabeth. University of Puget Sound
Osborne, Travis. University of Puget Sound
Riegsecker, Joe. Middlebury, Indiana joepye (at) pobox (dot) com
Perkel, Manley. University of Puget Sound
Phelps, Douglas. University of Puget Sound
Shoemaker, Mark. University of Puget Sound
Toth, Zoltan. http://zoli.web.elte.hu
Zimmer, Andy. University of Puget Sound

Previous topic: Section SAS  Sharing A Secret

Next topic: Definitions

Chapter 22. Definitions

Definitions

Section WILA

Section SSLE
Definition SLE System of Linear Equations
Definition SSLE Solution of a System of Linear Equations
Definition SSSLE Solution Set of a System of Linear Equations
Definition ESYS Equivalent Systems
Definition EO Equation Operations

Section RREF
Definition M Matrix
Definition CV Column Vector
Definition ZCV Zero Column Vector
Definition CM Coefficient Matrix
Definition VOC Vector of Constants
Definition SOLV Solution Vector
Definition MRLS Matrix Representation of a Linear System
Definition AM Augmented Matrix
Definition RO Row Operations
Definition REM Row-Equivalent Matrices
Definition RREF Reduced Row-Echelon Form
Definition RR Row-Reducing

Section TSS
Definition CS Consistent System
Definition IDV Independent and Dependent Variables

Section HSE
Definition HS Homogeneous System
Definition TSHSE Trivial Solution to Homogeneous Systems of Equations
Definition NSM Null Space of a Matrix

Section NM
Definition SQM Square Matrix
Definition NM Nonsingular Matrix
Definition IM Identity Matrix

Section VO
Definition VSCV Vector Space of Column Vectors
Definition CVE Column Vector Equality
Definition CVA Column Vector Addition
Definition CVSM Column Vector Scalar Multiplication

Section LC
Definition LCCV Linear Combination of Column Vectors

Section SS
Definition SSCV Span of a Set of Column Vectors

Section LI
Definition RLDCV Relation of Linear Dependence for Column Vectors
Definition LICV Linear Independence of Column Vectors

Section LDS

Section O
Definition CCCV Complex Conjugate of a Column Vector
Definition IP Inner Product
Definition NV Norm of a Vector
Definition OV Orthogonal Vectors
Definition OSV Orthogonal Set of Vectors
Definition SUV Standard Unit Vectors
Definition ONS OrthoNormal Set

Section MO
Definition VSM Vector Space of m × n Matrices
Definition ME Matrix Equality
Definition MA Matrix Addition
Definition MSM Matrix Scalar Multiplication
Definition ZM Zero Matrix
Definition TM Transpose of a Matrix
Definition SYM Symmetric Matrix
Definition CCM Complex Conjugate of a Matrix
Definition A Adjoint

Section MM
Definition MVP Matrix-Vector Product
Definition MM Matrix Multiplication
Definition HM Hermitian Matrix

Section MISLE
Definition MI Matrix Inverse

Section MINM
Definition UM Unitary Matrices

Section CRS
Definition CSM Column Space of a Matrix
Definition RSM Row Space of a Matrix

Section FS
Definition LNS Left Null Space
Definition EEF Extended Echelon Form

Section VS
Definition VS Vector Space

Section S
Definition S Subspace
Definition TS Trivial Subspaces
Definition LC Linear Combination
Definition SS Span of a Set

Section LISS
Definition RLD Relation of Linear Dependence
Definition LI Linear Independence
Definition TSVS To Span a Vector Space

Section B
Definition B Basis

Section D
Definition D Dimension
Definition NOM Nullity Of a Matrix
Definition ROM Rank Of a Matrix

Section PD
Definition DS Direct Sum

Section DM
Definition ELEM Elementary Matrices
Definition SM SubMatrix
Definition DM Determinant of a Matrix

Section PDM

Section EE
Definition EEM Eigenvalues and Eigenvectors of a Matrix
Definition CP Characteristic Polynomial
Definition EM Eigenspace of a Matrix
Definition AME Algebraic Multiplicity of an Eigenvalue
Definition GME Geometric Multiplicity of an Eigenvalue

Section PEE

Section SD
Definition SIM Similar Matrices
Definition DIM Diagonal Matrix
Definition DZM Diagonalizable Matrix

Section LT
Definition LT Linear Transformation
Definition PI Pre-Image
Definition LTA Linear Transformation Addition
Definition LTSM Linear Transformation Scalar Multiplication
Definition LTC Linear Transformation Composition

Section ILT
Definition ILT Injective Linear Transformation
Definition KLT Kernel of a Linear Transformation

Section SLT
Definition SLT Surjective Linear Transformation
Definition RLT Range of a Linear Transformation

Section IVLT
Definition IDLT Identity Linear Transformation
Definition IVLT Invertible Linear Transformations
Definition IVS Isomorphic Vector Spaces
Definition ROLT Rank Of a Linear Transformation
Definition NOLT Nullity Of a Linear Transformation

Section VR
Definition VR Vector Representation

Section MR
Definition MR Matrix Representation

Section CB
Definition EELT Eigenvalue and Eigenvector of a Linear Transformation
Definition CBM Change-of-Basis Matrix

Section OD
Definition UTM Upper Triangular Matrix
Definition LTM Lower Triangular Matrix
Definition NRML Normal Matrix

Section NLT
Definition NLT Nilpotent Linear Transformation
Definition JB Jordan Block

Section IS
Definition IS Invariant Subspace
Definition GEV Generalized Eigenvector
Definition GES Generalized Eigenspace
Definition LTR Linear Transformation Restriction
Definition IE Index of an Eigenvalue

Section JCF
Definition JCF Jordan Canonical Form

Section CNO
Definition CNE Complex Number Equality
Definition CNA Complex Number Addition
Definition CNM Complex Number Multiplication
Definition CCN Conjugate of a Complex Number
Definition MCN Modulus of a Complex Number

Section SET
Definition SET Set
Definition SSET Subset
Definition ES Empty Set
Definition SE Set Equality
Definition C Cardinality
Definition SU Set Union
Definition SI Set Intersection
Definition SC Set Complement

Section PT

Section F
Definition F Field
Definition IMP Integers Modulo a Prime

Section T
Definition T Trace

Section HP
Definition HP Hadamard Product
Definition HID Hadamard Identity
Definition HI Hadamard Inverse

Section VM
Definition VM Vandermonde Matrix

Section PSM
Definition PSM Positive Semi-Definite Matrix

Section ROD

Section TD

Section SVD
Definition SV Singular Values

Section SR
Definition SRM Square Root of a Matrix

Section POD

Section CF
Definition LSS Least Squares Solution

Section SAS

Previous topic: Contributors