# 0x011 Linear Algebra

The Hierarchical structure related to vector space is as follows:

With inner product structure, you can induce norm as

$||a|| = \sqrt{\langle a, a \rangle}$

With norm structure, you can induce metric as

$d(a, b) = ||a-b||$

## 1. Vector Spaces

closed under vector addition and scalar multiplication. It is a specicial case of module: module over a field

### 1.1. Vector Space

Definition (addition, scalar multiplication) An addition on a set $$V$$ is a function that assigns $$u+v \in V$$ to each pair of elements $$u,v \in V$$

A scalar multiplication on a set $$V$$ is a function that assigns an element $$\lambda v \in V$$ to each $$\lambda \in F$$ and each $$v \in V$$

Definition (vector space) A vector space is a set $$V$$ along with an addition on $$V$$ and a scalar multiplication on V such that the following properties hold:

• commutativity: $$u+v = v+u$$ for all $$u,v \in V$$
• associativity: $$(u+v)+w = u + (v+w)$$ and $$(ab)v = a(bv)$$ for all $$u,v,w \in V$$ and $$a,b \in F$$
• additive identity: there exists an element $$0 \in V$$ such that $$v+0 = v$$ for all $$v \in V$$
• additive inverse: for every $$v \in V$$, there exists $$w \in V$$ such that $$v+w=0$$
• multiplicative identity: $$1v = v$$ for all $$v \in V$$
• distributive properties: $$a(u+v) = au + av$$ and $$(a+b)v = av + bv$$ for all $$a,b \in F$$ and all $$u,v \in V$$
• Note: $$(V,+)$$ is an abelian group

Definition (vector): elements of a vector space are called vectors or points

### 1.2. Subspaces

Definition (subspace) A Subset $$U$$ of $$V$$ is called a subspace of $$V$$ if $$U$$ is also a vector space (using the same addition and scalar multiplication as on $$V$$)

Criterion (subspace) A subset $$U$$ of $$V$$ is a subspace iff $$U$$ satisfies the following three conditions - additive identity: $0 \in U$ - closed under addition: $$u,w \in \implies u+w \in U$$ - closed under scalar multiplication: $$a \in F, u \in U \implies au \in U$$

Note: the union of subspaces is rarely a subspace, but a collection of subspaces are always subspace.

Definition (sum of subspace) Suppose $$U_1, U_2, ... U_m$$ are subsets of $$V$$, the sum of $$U_1, ... U_m$$, denoted by $$U_1 + ... + U_m$$ is the set of all possible sums of elements of $$U_1, ... , U_m$$. More precisely,

$U_1 + U_2 + ... + U_m = \{ u_1 + u_2 + ... + u_m | u_1 \in U_1, ..., u_m \in U_m \}$

Definition (direct sum) Suppose $$U_1, U_2, ... U_m$$ are subspaces of $$V$$, the sum $$U_1 + U_2 + ... + U_m$$ is called a direct sum if each element of $$U_1 + U_2 + ... + U_m$$ can be written in only one way as a sum of $$u_1 + u_2 + ... + u_m$$, if it is a direct sum, then $$U_1 \oplus U_2 \oplus ... \oplus U_m$$ denotes $$U_1 + U_2 + ... + U_m$$

Criterion (direct sum) $$U + W$$ is a direct sum in either of the following case

there is only one way to write $$u+w = 0$$, where $$u \in U, w \in W$$ $$U \cap W = \{ 0 \}$$ note: direct sum means the linear independence between subspaces

### 1.3. Finite Dimension Vector Space

Definition (linear combination) A linear combination of a list $$v_1, ..., v_m$$ of vectors in $$V$$ is a vector of the form $$a_1 v_1 + ... + a_m v_m$$ where $$a_1, ... , a_m \in F$$

btw, Affine combination has a bias in addition

Definition (span) The set of all linear combinations of $$v_1, ..., v_m$$ in $$V$$ is called the span of $$v_1, ... v_m$$, denoted $$span(v_1, ..., v_m)$$.

Definition (basis) A basis of $$V$$ is a list of vectors in $$V$$ that is linearly independent and spans $$V$$

$v = a_1 v_1 + ... + a_n v_n$

Note that it has to satisfy two properties: 1) linear independence 2) span the whole space.

## 2. Inner Product Space

### 2.1. Inner Products and Norms

inner product is a generalization of the dot product

$x \cdot y = x_1y_1 + ... + x_n y_n$

Definition (inner product) An inner product on $$V$$ is a function that takes each ordered pair $$(u, v)$$ of elements of $$V$$ to a number $$\langle u, v \rangle \in F$$ and has the following properties.

• (positivity) $$\langle v, v \rangle \geq 0$$ for all $$v \in V$$
• (definiteness) $$\langle v, v \rangle = 0$$ iff $$v = 0$$
• (additivity in first slot) $$\langle u+v, w \rangle = \langle u, w \rangle + \langle v, w \rangle$$ for all $$u,v,w \in V$$
• (homogeneity in first slot) $$\langle \lambda u, v \rangle = \lambda \langle u, v \rangle$$ for all $$\lambda \in F$$ and all $$u,v \in V$$
• (conjugate symmetry) $$\langle u, v \rangle = \overline{\langle v, u \rangle}$$ for all $$u,v \in V$$

Euclidean Inner Product

The Euclidean inner product on $$F^n$$ is defined by

$\langle (w_1, ..., w_n), (z_1, ..., z_n) \rangle = w_1\bar{z_1} + ... + w_n\bar{z_n}$

The most general form of an inner product on it is given by

$\langle x, y \rangle = y^*Mx$

where $$M$$ is any Hermitian positive definite matrix.

Function Inner Product

An inner product can be defined on the vector space of continuous real-valued functions on the interval $$[-1, 1]$$ by

$\langle f, y \rangle = \int_{-1}^{1} f(x)g(x) dx$

Definition (inner product space) An inner product space is a vector space $$V$$ along with an inner product on $$V$$

Definition (norm) A norm on a vector space $$V$$ is a function $$|| \cdot ||: V \to [0, \infty)$$ such that

• (positive definite) $$||f|| = 0$$ iff $$f = 0$$
• (homogeneity) $$||\alpha f|| = |\alpha| ||f||$$ for all $$\alpha, f$$
• (triangle inequality) $$||f+g|| \leq ||f|| + ||g||$$

Inner product can induce a norm

Definition (induced norm) For $$v \in V$$, the norm of $$v$$, denoted $$||v||$$ is defined by

$||v|| = \sqrt{\langle v, v \rangle}$

Euclidean norm

### 5.3. Diagonal Matrices

Definition (diagonal matrix) A diagonal matrix is a square matrix that is 0 everywhere except possibly along the diagonal

Definition (eigenspace) Suppose $$T \in \mathcal{L}(V), \lambda \in F$$. The eigenspace of $$T$$ corresponding to $$\lambda$$ denoted $$E(\lambda, T)$$ is defined by

$E(\lambda, T) = null(T - \lambda I)$

Proposition (Sum of eigenspaces is a direct sum) Suppose $$V$$ is finite-dimensional and $$T \in \mathcal{L}(V)$$. Suppose also that $$\lambda_1, ..., \lambda_m$$ are distinct eigenvalues of $$T$$. Then $$E(\lambda_1, T) + ... + E(\lambda_m, T)$$ is a direct sum. Futhermore, $$\dim (E(\lambda_1, T)) + ... + \dim E(\lambda_m, T) \leq \dim V$$

Definition (diagonalizable) An operator $$T \in \mathcal{L}(V)$$ is called diagonalizable if the operator has a diagonal matrix with respect to some basis of $$V$$

Proposition (conditions equivalent to diagonalizability) Suppose $$V$$ is finite-dimensional and $$T \in \mathcal{L}(V)$$. Let $$\lambda_1, ..., \lambda_m$$ detnoe the distinct eigenvalues of $$T$$. Then the following are equivalent:

• $$T$$ is diagonalizable
• $$V$$ has a basis consisting of eigenvectors of $$T$$ there exist 1-dimensional subspaces $$U_1, ..., U_n$$ of $$V$$, each invariant under $$T$$, such that $$V = U_1 \oplus ... \oplus U_n$$
• $$V = E(\lambda_1, T) \oplus ... \oplus E(\lambda_m, T)$$
• $$\dim V = \dim E(\lambda_1, T) + ... + \dim E(\lambda_m, T)$$

Proposition (enough eigenvalues implies diagonalizability) If $$T \in \mathcal{L}(V)$$ has dim $$V$$ distinct eigenvalues, then $$T$$ is diagonalizable.

## 6. Operators on Inner Product Spaces

Definition (adjoint) Suppose $$T \in \mathcal{L}(V,W)$$. The adjoint of $$T$$ is the function $$T^* : W \to V$$ such that

$(\forall v \in V)(\forall w \in W) \langle Tv, w \rangle = \langle v, T^*w \rangle$

Its existence can be guaranteed by Riesz Representation Theorem: when $$w$$ is fixed, $$\langle Tv, w \rangle$$ is a linear functional on $$v$$: $$f(v)$$, so there should exist a vector $$u$$ satisfying $$f(v) = \langle v, u \rangle$$. $$u$$ depends on $$w$$ so we write $$T^*w$$

Consider $$T: \mathbb{R}^3 \to \mathbb{R}^2$$ such that

$T(x_1, x_2, x_3) = (x_2 + 3x_3, 2x_1)$

Then its adjoint $$T^*: \mathbb{R}^2 \to \mathbb{R}^3$$ is

$T^*(y_1, y_2) = (2y_2, y_1, 3y_1)$

Lemma (adjoint is a linear map) If $$T \in \mathcal{L}(V,W)$$, then $$T^* \in \mathcal{L}(W,V)$$

Lemma (The matrix of $$T^*$$) Let $$T \in \mathcal{L}(V,W)$$ Suppose $$e_1, ..., e_n$$ is an orthonormal basis of $$V$$ and $$f_1, ..., f_m$$ is an orthonormal basis of $$W$$. Then $$\mathcal{M}(T^*, (f_1, ..., f_m), (e_1,...,e_n))$$ is the conjugate transpose of $$\mathcal{M}(T, (e_1,..., e_n), (f_1, ..., f_m))$$

### 6.1. Normal Operator

Normal operator is similar to complex number.

Definition (normal operator) An operator $$T$$ is called normal iff $$TT^* = T^*T$$

Lemma (properties of normal operator)

• An operator $$T$$ is normal iff $$||Tv|| = ||T^*v||$$
• If $$T$$ is normal, and $$v$$ is an eigenvector of $$T$$ with eigenvalue of $$\lambda$$, Then $$v$$ is also an eigenvector of $$T^*$$ with eigenvalue $$\overline{\lambda}$$
• If $$T$$ is normal, then eigenvectors of $$T$$ corresponding to distinct eigenvalues are orthogonal

Theorem (complex spectral theorem) $$T \in \mathcal{L}(V)$$ is normal where field is $$C$$ $$\iff$$ $$T$$ can be diagonalized with respect to some orthonormal basis of $$V$$

### 6.2. Hermitian Operator

Hermitian operator is similar to real number

Definition (Hermitian operator, self-adjoint operator) An operator $$T \in \mathcal{L}(V)$$ is called hermitian or self-adjoint if $$T = T^*$$

$\langle Tv, w \rangle = \langle v, Tw \rangle$

Lemma (properties of self-adjoint)

• Every eigenvalue of self-adjoint operator is real
• over C, $((\forall v) \langle Tv, v \rangle = 0) \iff T = 0$
• over C, $$(\forall v) \langle Tv, v \rangle \in R \iff T=T^*$$
• If $$T=T^* \land (\forall v \in V) \langle Tv, v \rangle = 0 \iff T =0$$

Theorem (real spectral theorem) $$T \in \mathcal{L}(V)$$ is self-adjoint where field is $$R$$ $$\iff$$ $$T$$ can be diagonalized with respect to some orthonormal basis of $$V$$

### 6.3. Positive Operator

positive operator is similar to non-negative number

Definition (positive operator) An operator $$T \in \mathcal{L}(V)$$ is called positive if $$T$$ is self-adjoint and $$\langle Tv, v \rangle \geq 0$$ for all $$v \in V$$

### 6.4. Unitary Operator

unitary operator is similar to 1

Definition (isometry, unitary operator) An operator $$S \in \mathcal{L}(V)$$ is called isometry if $$\| Sv \| = \|v\|$$ for all $$v \in V$$

## 7. Operators on Complex Vector Spaces

### 7.1. Generalized Eigenvectors

Proposition (space decomposition) Suppose $$T \in \mathcal{L}(V)$$, and $$n=\dim V$$, then

$V = null T^n \oplus range T^n$

Note that $$nullT, rangeT$$ cannot form direct sum: $$V \neq null T \oplus range T$$

example

Define $$T \in L(C^3)$$ by

$T(z_1, z_2, z_3) = (4z_2, 0, 5z_3)$

The eigenvalues of $$T$$ are 0, 5. The corresponding eigenspaces are $$E(0, T) = \{ (z_1, 0, 0) \}, E(5, T) = (0, 0, z_3)$$. The corresponding generalized eigenspaces are $$G(0, T) = \{ (z_1, z_2, 0) \}, G(5, T) = \{ (0, 0, z_3) \}$$

Definition (generalized eigenvector) Suppose $$T \in \mathcal{L}(V)$$ and $$\lambda$$ is an eigenvalue of $$T$$. A vector $$v \in V$$ is called a generalized eigenvector of $$T$$ corresponding to $$\lambda$$ if $$v \neq 0$$ and for some positive number $$j$$

$(T-\lambda I)^j v = 0$

Definition (generalized eigenspace) Suppose $$T \in \mathcal{L}(V), \lambda \in F$$. The generalized eigenspace of $$T$$ corresponding to $$\lambda$$, denoted $$G(\lambda, T)$$, is defined to be the set of all generalized eigenvectors of $$T$$ corresponding to $$\lambda$$

Proposition (description of generalized eigenspace) Suppose $$T \in \mathcal{L}(V), \lambda \in F$$, Then

$G(\lambda, T) = null (T - \lambda I)^{\dim V}$

Definition (nilpotent) An operator is called nilpotent if some power of it equals to $$0$$.

Note that this is a generalization of zero operator

Lemma (description of nilpotent operator) Suppose $$N \in \mathcal{L}(V)$$ is nilpotent, then $$N^{\dim V} = 0$$

### 7.2. Operator Decomposition

Proposition (operator domain decomposition) Suppose $$V$$ is a complex vector space and $$T \in \mathcal{L}(V)$$. Let $$\lambda_1, ..., \lambda_m$$ be the distinct eigenvalues of $$T$$. Then

$V = \oplus_{i=1}^m G(\lambda_i, T)$

Lemma (basis of generalized eigenvectors) Suppose $$V$$ is a complex vector space and $$T \in \mathcal{L}(V)$$, then there is a basis of $$V$$ consisting of generalized eigenvectors of $$T$$

Definition (multiplicity) The multiplicity of an eigenvalue $$\lambda$$ of $$T$$ is defined to be the dimension of the corresponding generalized eigenspace $$G(\lambda, T)$$ (i.e., $$\dim null(T- \lambda I)^{\dim V}$$)

Proposition (square roots of inertible operators over C) Suppose $$V$$ is a complex vector space and $$T \in \mathcal{L}(V)$$ is invertible, then $$T$$ has a square root

### 7.3. Characteristic and Minimal Polynomials

Definition (characteristic polynomial) Suppose $$V$$ is a complex vector space and $$T \in \mathcal{L}(V)$$. Let $$\lambda_1, ..., \lambda_m$$ denote the distinct eigenvalues of $$T$$, with multiplicities $$d_1, ..., d_m$$. The following polynomial is called the characteristic polynomials of $$T$$

$\prod_{i} (z - \lambda_i)^{d_i}$

Theorem (Cayley-Hamilton) Suppose $$V$$ is a complex vector space and $$T \in \mathcal{L}(V)$$. Let $$q$$ denote the characteristic polynomial of $$T$$. Then $$q(T)=0$$

## 9. Reference

[1] Axler, Sheldon Jay. Linear algebra done right. Vol. 2. New York: Springer, 1997.

[2] Roman, Steven, S. Axler, and F. W. Gehring. Advanced linear algebra. Vol. 3. New York: Springer, 2005.

[3] Boyd, Stephen, and Lieven Vandenberghe. Introduction to applied linear algebra: vectors, matrices, and least squares. Cambridge university press, 2018.