Some Notes to Supplement the Lectures in Mathematics 700

1 The Uniqueness of the Row Reduced Echelon Form

Let A and B be two matrices of the same size and assume that they are row equivalent. Then there are elementary matrices E_{1}, …, E_{k} so that B = E_{k}E_{k−1} ⋯ E_{1} ⋯ E_{k}A. Then as all the matrices E_{1}, …, E_{k} are invertible so is the product P := E_{k}E_{k−1} ⋯ E_{1}. Therefore if A and B are row equivalent matrices there is an invertible matrix P so that B = P A. If we write B\overline{∼}A to mean there is an invertible matrix P so that B = P A. Then this is an equivalence relation. (For letting P = I shows A\overline{∼}A. If B = P A, then A = P^{−1}B so A\overline{∼}B implies B\overline{∼}A. If B = P A and C = QB then C = QP A so A\overline{∼}B and B\overline{∼}C implies A\overline{∼}B. Of course we know from class work that this equivalence relation in just the same as row equivalence, but I don’t want to use this fact in the following proof as then we can use the argument here to give an new proof of this. A virtue of the proof here is that it gives an example of how to use block matrices to prove things. It has several vices, such as being more computational than conceptual.

Theorem 1.1 If R and R′ are row reduced echelon matrices of the same size so that for some invertible matrix P there holds R′ = P R, then R = R′. In particular if R and R′ are row equivalent then they are equal.

Proof: Assume that R and R′ are both m × n. We will use induction and assume that the result holds for row reduced echelon matrices with fewer than m rows and show that it holds when they have m rows. We write

R = [ R_{1} ; ⋮ ; R_{m} ] ,  R′ = [ R′_{1} ; ⋮ ; R′_{m} ] ,  P = [ \begin{matrix}p_{11} & ⋯ & p_{1m} \\ ⋮ & ... & ⋮ \\ p_{m1} & ⋯ & p_{mm} \end{matrix} ]

where R_{1}, …, R_{m} are the rows of R and R′_{1}, …, R′_{m} are the rows of R. Then R′ = P R implies R′_{i} = ∑_{j=1}^{m} p_{ij}R_{j}. That is the rows of R′ are linear combinations of the rows of R. By symmetry of the equivalence relation \overline{∼} this also implies that the rows of R are linear combinations of the rows of R′. If R = 0 then also R′ = P R = 0 and we are done. So we can assume R ≠ 0. But the first row of a nonzero row reduced echelon matrix is nonzero. Assume the leading one of the first row of R_{1} has occurs in the k-th place. Then the rows of R′ are all linear combinations of the rows of R the leading one of the first row of R′ also occurs in the k-th place. We now write R and R′ in block form:

R = [ \begin{matrix}0 & ⋯ & 1 & R_{12} \\ 0 & ⋯ & 0 \\ ⋮ & ... & ⋮ & R_{22} \\ 0 & ⋯ & 0 \end{matrix} ] , R′ = [ \begin{matrix}0 & ⋯ & 1 & R′_{12} \\ 0 & ⋯ & 0 \\ ⋮ & ... & ⋮ & R′_{22} \\ 0 & ⋯ & 0 \end{matrix} ]

where R_{12} and R′_{12} are of size 1 × (n − k) and R_{22} and R′_{22} are (m − 1) × (n − k). Also write P in block from

P = [ \begin{matrix}p_{11} & P_{12} \\ P_{21} & P_{22} \end{matrix} ] = [ \begin{matrix}p_{11} & p_{12} & ⋯ & p_{1m} \\ p_{21} \\ ⋮ & & P_{22} \\ p_{m1} \end{matrix} ] where P_{12} = [p_{12}, …, p_{1m}], P_{21} = [ p_{21} ; ⋮ ; p_{m1}, ]

P_{22} is (m − 1) × (m − 1) and P_{11} = p_{11} is 1 × 1 (that is just a scalar). Now writing out P R = R′ in block form

P R = [ \begin{matrix}p_{11} & P_{12} \\ P_{21} & P_{22} \end{matrix} ] [ \begin{matrix}0 & ⋯ & 1 & R_{12} \\ 0 & ⋯ & 0 \\ ⋮ & ... & ⋮ & R_{22} \\ 0 & ⋯ & 0 \end{matrix} ]

= [ \begin{matrix}0 & ⋯ & 0 & p_{11} & p_{11}R_{12} + P_{12}R_{22} \\ 0 & ⋯ & 0 & p_{21} \\ ⋮ & ... & ⋮ & ⋮ & P_{21}R_{12} + P_{22}R_{22} \\ 0 & ⋯ & 0 & p_{m1} \end{matrix} ]

= R′ = [ \begin{matrix}0 & ⋯ & 0 & 1 & R′_{12} \\ 0 & ⋯ & 0 & 0 \\ ⋮ & ... & ⋮ & ⋮ & R′_{22} \\ 0 & ⋯ & 0 & 0 \end{matrix} ]

This implies p_{11} = 1 and p_{21} = p_{31} = ⋯ = p_{m1} = 0. That is P_{21} = 0. Putting this information back into the equation gives

P R = [ \begin{matrix}0 & ⋯ & 0 & 1 & R_{12} + P_{12}R_{22} \\ 0 & ⋯ & 0 & 0 \\ ⋮ & ... & ⋮ & ⋮ & P_{22}R_{22} \\ 0 & ⋯ & 0 & 0 \end{matrix} ] = [ \begin{matrix}0 & ⋯ & 0 & 1 & R′_{12} \\ 0 & ⋯ & 0 & 0 \\ ⋮ & ... & ⋮ & ⋮ & R′_{22} \\ 0 & ⋯ & 0 & 0 \end{matrix} ] = R′.

In particular this implies R′_{22} = P_{22}R_{22} and that

P = [ \begin{matrix}1 & p_{12} & ⋯ & p_{1m} \\ 0 \\ ⋮ & & P_{22} \\ 0 \end{matrix} ] .

But then P invertible implies P_{22} invertible. Also R_{22} and R′_{22} are both in row reduced echelon form and R′_{22} = P_{22}R_{22} along with the induction hypothesis implies R′_{22} = R_{22}.

But this in turn implies that R and R have the same number r of nonzero rows and that the leading ones in each of these rows is in the same column. Let R_{1}, …, R_{r} be the nonzero rows of R and R′_{1}, …, R′_{r} the nonzero rows of R′. Let k_{i} be the position of the leading one in R_{i} (which is the same as the position of the leading one in R′_{i}). Then as every row of R′ is a linear combination of the rows of R for each i there are scalars c_{1}, …, c_{r} so that

R′_{i} = c_{1}R_{1} + c_{2}R_{2} + ⋯ + c_{r}R_{r}.

If we look at the k_{i} position in R′_{i} then it is a 1 (as k_{i} was where the leading 1 in R′_{i} occurs) and the same is true of R_{i}. But all of the other vectors R_{j} have a 0 in this place as R is row reduced. This means that the element in the k_{i} position of c_{1}R_{1} + c_{2}R_{2} + ⋯ + c_{r}R_{r} is just c_{i} and that his must be equal to 1 (the element in the k_{i} of R_{i}). For j ≠ i we look a similar argument shows that the element in the k_{j} position of c_{1}R_{1} + c_{2}R_{2} + ⋯ + c_{r}R_{r} is c_{j} is c_{j} and this must be equal to the element in the k_{j} position of R′_{i} which is 0. In summary: c_{i} = 1 and c_{j} = 0 for j ≠ i. Therefore

R′_{i} = c_{1}R_{1} + c_{2}R_{2} + ⋯ + c_{r}R_{r} = R_{i}.

As this works for any i this shows R′_{i} = R_{i} for i = 1, …, r. Therefore R and R′ have all their rows equal and thus are equal. This completes the proof.

2 Summary of Chapter 1

This chapter deals with solving systems of linear equations over a field. A field is a collection of elements F together with two binary operations addition and multiplication with satisfy the usual rules of high school algebra in the sense that it is possible to add, subtract, multiple and divide as usual. In the context of linear algebra elements of the field will often be called scalars.

The main subject of this chapter is systems of m linear equations in n unknowns with coefficients from the field F. That is a system of the form

A_{11}x_{n} + ⋯ + A_{1n}x_{n} = y_{1}
(1)  ⋮  ⋮ ⋮ ⋮
A_{m1}x_{1} + ⋯ + A_{mn}x_{m} = y_{m}.

The equation is homogeneous iff all the y_{i}’s vanish. That is the system is of the form

A_{11}x_{n} + ⋯ + A_{1n}x_{n} = 0
⋮  ⋮ ⋮ ⋮
A_{m1}x_{1} + ⋯ + A_{mn}x_{m} = 0.

The more general system (1) is the inhomogeneous system. One basic difference between the homogeneous and inhomogeneous system is that the homogeneous system always has at least the solution x_{1} = x_{2} = ⋯ x_{n} = 0 (called the trivial solution) while the inhomogeneous system need not have any solutions in which case it is called inconsistent. A exmaple of an inconsistent system would be x_{1} + x_{2} = 1 and 2x_{2} + 2x_{2} = 1.

Given such a system of equations some other equation is a linear combination of elements in this system if it is formed by multiplying some of the equations in the system by scalars and adding the results together. Two systems of linear equations are equivalent if and only if every equation one system is a linear combination of equations in the other system and vice versa. What makes this notation of equivalence interesting is Theorem 2.1 Equivalent systems of equations have the same set of solutions.

In doing calculations with systems such as taking linear combinations it soon becomes clear that all the actual work of the calculations is done with the coefficients A_{ij} of the system and that the unknowns x_{i} are just along for the ride. This has lead to the notation of a matrix which is just a rectangular array A = [A_{ij}] of elements. To be more precise a m × n matrix has m rows, n columns and the element A_{ij} is in the i th row and j th column of A. Associated to the inhomogeneous system (1) there are four matrices

A := [ \begin{matrix}A_{11} & ⋯ & A_{1n} \\ ⋮ & ... & ⋮ \\ A_{m1} & ⋯ & A_{mn} \end{matrix} ] ,  X := [ x_{1} ; ⋮ ; x_{n} ] ,  Y := [ y_{1} ; ⋮ ; y_{m} ]

The system (1) can then be written as AX = Y and the homogeneous system can be written as AX = 0. Before going to give the relation between matrices and solving systems of linear equations it we note that matrices have a good deal of algebra associated with them. If particular it is possible to multiply a matrix by a scalar so that cA is the matrix obtained form A by multiplying all of its elements by c. The sum A + B of two matrices of the same size is matrix whose (i, j)th entry is the sum of the corresponding elements of A and B and the product of an m × n matrix A with an m × p matrix B is the m × p matrix AB with elements

(AB)_{ij} = ∑_{k=1}^{n} A_{ik}B_{kj}.

This product is associative (i.e. (AB)C = A(BC)) but not commutative (sometimes AB ≠ BA). Note that when we rewrite the system (1) as AX = Y that the product between A and X is matrix multiplication. The n × n identity matrix is the matrix I_{n×n} with entries δ_{ij} = 1 if i = j and = 0 if i ≠ j. Or what is the same thing I_{n×n} is the n × n matrix with 1’s down the main diagonal and all other elements zero. This matrix is special in that for any m × n matrix A we have

I_{m×m}A = AI_{n×n} = A.

For a square matrix A one to define a matrix B to be the inverse of A iff AB = BA = I where I is the identity matrix. (While at first it would seem that one might have to define left inverses, right inverse and two sided inverses we have shown that if A has a one sided inverse then this one sided inverse is the unique two sided inverse to A, thus at least for square matrices we do not have to worry about these distinctions). A square matrix with an inverse will be called a invertible or nonsingular iff it has in inverse and it is called singular iff it has no inverse. The inverse of a nonsingular matrix A will be denoted by A^{−1}. Note if A_{1}, …, A_{k} are all invertible matrices of the same size then the product A_{1}A_{2} ⋯ A_{k} is also invertible and (A_{1}A_{2} ⋯ A_{k})^{−1} = A_{k}^{−1}A_{k−1}^{−1} ⋯ A_{1}^{−1}.

After this digression on matrix algebra we return to systems of equations. The elementary row operations on a m × n matrix A are

1. Multiplying a row of A by a nonzero scalar.

2. Adding a scalar multiple of one row to anther row.

3. Interchanging two rows of A. In terms of the corresponding system of equations these operations are the same as multiplying one of the equations by a nonzero scalar, adding a scalar multiple of one equation to anther and final interchanging two of the equations. Note that any of these operations can be reversed by anther elementary row operation of the same type. We also note that each elementary row operation e can be accomplished by multiplying A on the left by a m × m matrix E. That is a(A) = EA. These matrices are called elementary matrices. Two matrices A and B are row equivalent iff there is a finite sequence of row operations that transform A into B. Or what is the same thing if there is a finite number of elementary matrices E_{1}, …, E_{k} is that B = E_{k} ⋯ E_{1}A. This is very closely related to matrix multiplication because of Theorem 2.2 Two m×n matrices A and B are row equivalent iff there is a nonsingular m×m matrix P so that B = P A.

To study the solutions of the homogeneous system AX = 0 note if A and B are row equivalent then the systems AX = 0 and BX = 0 are equivalent in the sense of Theorem 2.1 and thus have the same solutions. This makes it interesting to try to row reduce the matrix to as simple a form as possible. The first step in this is a matrix is row reduced iff the first nonzero element in any row is a 1 and any column that contains a leading row has all its other elements zero. A matrix is in row reduced echelon form iff if is row reduced, all its nonzero rows appear above all its zero rows and the leading one in any row appears to the right of all the leading ones in the rows above it. This turns out to be our simple form under row equivalence: Theorem 2.3 Every matrix is row equivalent to a unique row reduced echelon matrix.

This allows us to define an important invariant of a matrix, its rank which is the number of nonzero rows in its row reduced echelon form. This invariant is used in understanding when the inhomogeneous system AX = 0 has non-trivial solutions. In particular we have Theorem 2.4 Let R be the row reduced echelon form of the matrix A. Then the general solution to AX = 0 is obtained by letting the variables that do not correspond to leading ones in R have arbitrary values and then solving for the variables that correspond to leading ones in the system RX = 0. Thus if the rank of A is equal to the number of rows (that is the number of equations) then all variables correspond to leading ones and the system has only the trivial solution X = 0. However if the rank is less than the number of rows then there will be nontrivial solutions. In particular this leads to the very important result than any homogeneous system AX = 0 where the number of unknowns is larger than the number of equations will have a non-trivial solution. This tells close to the full story about the homogeneous system AX = 0. To understand the inhomogeneous equation AX = Y form the augmented matrix

A′ := [ \begin{matrix}A_{11} & ⋯ & A_{1n} & y_{1} \\ ⋮ & ... & ⋮ \\ A_{m1} & ⋯ & A_{mn} & y_{m} \end{matrix} ]

Obtained from A by adding Y as an extra column. Let r be the rank of A which is the number of nonzero rows in R. We then row reduce A (not the whole matrix A′) to row reduced echelon form R. The result is

[ \begin{matrix}& R & & Z_{1} \\ 0 & ⋯ & 0 & z_{r+1} \\ ⋮ & ... & ⋮ & ⋮ \\ 0 & ⋯ & 0 & z_{m} \end{matrix} ] where Z_{1} := [ z_{1} ; ⋮ ; z_{r} ]

and each of the z_{i}’s is a linear combination of the variables y_{1}, …, y_{m}. For any given values of y_{1}, …, y_{m} if any of the linear expressions z_{r+1}, …, z_{m} is nonzero, then the system will be inconsistent (these conditions are often called compatibility conditions). If z_{r+1} = z_{r+2} = ⋯ z_{m} = 0 then the system is consistent and the general solution is found as in the homogeneous case by letting the variable not corresponding to leading 1’s in R have arbitrary values and then solving for the variables that do correspond to leading ones by use of the system RX = Z_{1}. Note if the rank of A is equal to the number of rows, there are no compatibility conditions and we have Theorem 2.5 If the rank of A is equal to the number of rows of A (which is the number of equations in the system AX = Y) then the inhomogeneous system AX = Y has a solution for any choice of Y. More generally the number of compatibility conditions is difference between the number of rows of A and the rank of A.

We specialize this to the case of “square” systems. That is where the number of equations is equal to the number of unknowns. Theorem 2.6 For a square n × n matrix the following are equivalent:

1. A is invertible.

2. The homogeneous system AX = 0 has only the trivial solution.

3. The inhomogeneous AX = Y has a solution for every choice of Y.

4. A is row equivalent to the identity matrix I.

5. The rank of A is n.

Finally there are some computational issues that I have not mentioned in the above. These are things such as finding the inverse of a invertible square matrix A (which is done by augmenting A by the identity matrix to get [A, I] and row reducing this to [I, B]. Then B = A^{−1}. When appropriate the best way to deal with this type problem is to use some computer package such as Maple or Matlab.