Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Subspaces and Maps to Other Spaces

English · فارسی

Subspaces and Maps to Other Spaces

Status: bridge page between the span/dependence notebooks and the eigen/SVD chapters.

The pages before this one answered questions inside Rn\mathbb{R}^n: what a linear combination is, when vectors are independent, what a span is. This page changes the point of view twice: first a matrix is no longer a table of numbers but a linear map between two spaces, and then a subspace is no longer a set of arrows but the image or the kernel of such a map. Both views are used in every later chapter — projections in PCA, null spaces in least squares, feature maps in the kernel page.

1. Subspaces, in one paragraph each

A subset V⊆RnV\subseteq\mathbb{R}^n is a subspace if it contains 0 and is closed under addition and scalar multiplication. Three consequences we will use constantly:

Data example: the set of all images in a dataset, flattened, spans a subspace of Rd\mathbb{R}^{d} whose dimension is at most the number of images — usually far less, which is the whole premise of low-rank approximation.

2. A matrix is a map

Fix A∈Rm×nA\in\mathbb{R}^{m\times n} and read it column by column: A=[a1,…,an]A = [a_1,\dots,a_n]. Then AxAx is the linear combination of the columns with coefficients x1,…,xnx_1,\dots,x_n. So multiplication by AA is the map

TA:Rn→Rm,TA(x)=Ax,T_A:\mathbb{R}^n\to\mathbb{R}^m,\qquad T_A(x) = Ax,

and it is linear: TA(αx+βy)=αTA(x)+βTA(y)T_A(\alpha x+\beta y) = \alpha T_A(x)+\beta T_A(y). Conversely, every linear map between these spaces is of this form for exactly one matrix: the jj-th column of AA is the image of the jj-th basis vector. This is why the two objects are used interchangeably.

Two subspaces travel with AA:

Col(A)={Ax:x∈Rn}⏟image, in Rm⊆Rm,Null(A)={x:Ax=0}⏟kernel, in Rn⊆Rn.\underbrace{\mathrm{Col}(A) = \{Ax : x\in\mathbb{R}^n\}}_{\text{image, in }\mathbb{R}^m} \subseteq \mathbb{R}^m, \qquad \underbrace{\mathrm{Null}(A) = \{x : Ax = 0\}}_{\text{kernel, in }\mathbb{R}^n} \subseteq \mathbb{R}^n .

Col(A)\mathrm{Col}(A) is the set of targets the map can hit; Null(A)\mathrm{Null}(A) is the set of inputs it collapses to zero. The rank is dim⁡Col(A)\dim\mathrm{Col}(A), the number of independent columns; the nullity is dim⁡Null(A)\dim\mathrm{Null}(A); and the two are tied by

  rank(A)+nullity(A)=n  \boxed{\;\mathrm{rank}(A) + \mathrm{nullity}(A) = n\;}

which is the statement that every independent direction of the input either survives into the image or is lost in the kernel, never both. In linear regression, Null(X)\mathrm{Null}(X) is exactly the ambiguity in the coefficients: if Xw=yXw = y has a solution, it has the affine set w+Null(X)w + \mathrm{Null}(X) of them, and the pseudoinverse picks the shortest.

3. The geometry of the three families

MapMatrix formWhat it doesKernel
scaling along axesdiag(s1,…,sn)\mathrm{diag}(s_1,\dots,s_n)stretches by sis_ithe axes with si=0s_i=0
rotation by θ\theta(cos⁡θ−sin⁡θsin⁡θcos⁡θ)\begin{pmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{pmatrix}turns the plane{0}\{0\}
projection onto a subspaceP=UU⊤P = UU^\top with U⊤U=IU^\top U = Ikeeps the UU-part, discards the restU⊥U^{\perp}
rank-one mapuv⊤uv^\topcollapses everything onto the line of uuv⊥v^{\perp}

The projection row is the one that returns in PCA: PP is idempotent (P2=PP^2=P) and symmetric, and rank(P)=dim⁡Col(U)\mathrm{rank}(P) = \dim\mathrm{Col}(U) — the number of components kept.

4. Composition, inverse, change of basis

5. Data view: a map to another space

Everything above was written for Rn→Rm\mathbb{R}^n\to\mathbb{R}^m, but the same words describe the step that makes modern learning possible: a feature map ϕ:X→F\phi:\mathcal{X}\to\mathcal{F} sends each data point to another space, where a linear model may be enough. Two examples from this course:

Lab pointers

Read these notebooks next to this page, they are the computational half of it: Linear-Dependence-and-Span, Projection, Multiplying-Matrices-and-Vectors, and then Eigendecomposition for the change-of-basis statement of section 4.

Exercises

  1. Take a 2×32\times3 matrix, describe its column space geometrically and give a basis for it.

  2. Give a linear map that is not invertible, describe its kernel, and check that the dimensions satisfy rank plus nullity equals 3.

  3. Explain why a linear map is determined by the images of a basis, and use that to reconstruct a 2×22\times2 matrix from two image vectors.

  4. For A=(123246)A=\begin{pmatrix}1&2&3\\2&4&6\end{pmatrix}, find bases of Col(A)\mathrm{Col}(A) and Null(A)\mathrm{Null}(A) and verify rank plus nullity equals 3. Why does the system Ax=bAx=b have solutions only for some bb, and how do you recognise those bb?

  5. Show that P=UU⊤P=UU^\top with U⊤U=IU^\top U=I satisfies P2=PP^2=P and P⊤=PP^\top=P, and that rank(P)\mathrm{rank}(P) is the number of columns of UU. Interpret both facts geometrically.

  6. Let BB be the matrix whose columns are (1,1)(1,1) and (1,−1)(1,-1). Compute B−1ABB^{-1}AB for a 2×22\times2 symmetric AA of your choice and check that the eigenvalues did not change. What are the eigenvectors of B−1ABB^{-1}AB in terms of those of AA?

  7. Give a feature map ϕ:R2→R3\phi:\mathbb{R}^2\to\mathbb{R}^3 such that the XOR labelling becomes linearly separable, and write the three coordinates explicitly.

Sources