Subspaces and Maps to Other Spaces
Subspaces and Maps to Other Spaces¶
Status: bridge page between the span/dependence notebooks and the eigen/SVD chapters.
The pages before this one answered questions inside : what a linear combination is, when vectors are independent, what a span is. This page changes the point of view twice: first a matrix is no longer a table of numbers but a linear map between two spaces, and then a subspace is no longer a set of arrows but the image or the kernel of such a map. Both views are used in every later chapter — projections in PCA, null spaces in least squares, feature maps in the kernel page.
1. Subspaces, in one paragraph each¶
A subset is a subspace if it contains 0 and is closed under addition and scalar multiplication. Three consequences we will use constantly:
Every subspace is the span of some vectors, and a minimal spanning set is a basis. All bases of the same subspace have the same number of elements, its dimension.
Subspaces of come in dimensions : a point, a line through the origin, a plane through the origin, and so on. The phrase through the origin is not decoration; a line that misses 0 is not a subspace and averaging inside it leaves it.
A subspace can be given in two ways, and the two descriptions are dual: as a span (parametrically, ) or as the solution set of homogeneous equations (implicitly, ). Converting between the two is Gaussian elimination.
Data example: the set of all images in a dataset, flattened, spans a subspace of whose dimension is at most the number of images — usually far less, which is the whole premise of low-rank approximation.
2. A matrix is a map¶
Fix and read it column by column: . Then is the linear combination of the columns with coefficients . So multiplication by is the map
and it is linear: . Conversely, every linear map between these spaces is of this form for exactly one matrix: the -th column of is the image of the -th basis vector. This is why the two objects are used interchangeably.
Two subspaces travel with :
is the set of targets the map can hit; is the set of inputs it collapses to zero. The rank is , the number of independent columns; the nullity is ; and the two are tied by
which is the statement that every independent direction of the input either survives into the image or is lost in the kernel, never both. In linear regression, is exactly the ambiguity in the coefficients: if has a solution, it has the affine set of them, and the pseudoinverse picks the shortest.
3. The geometry of the three families¶
| Map | Matrix form | What it does | Kernel |
|---|---|---|---|
| scaling along axes | stretches by | the axes with | |
| rotation by | turns the plane | ||
| projection onto a subspace | with | keeps the -part, discards the rest | |
| rank-one map | collapses everything onto the line of |
The projection row is the one that returns in PCA: is idempotent () and symmetric, and — the number of components kept.
4. Composition, inverse, change of basis¶
Composition is multiplication. Applying after is the map . Order matters, and the identity in general is a statement about maps, not about arithmetic.
Inverse means invertible map. is invertible iff , which for a square matrix is equivalent to full rank and to . A non-square matrix can be injective or surjective but not both; the pseudoinverse handles the general case.
Change of basis. If has the new basis vectors as columns, then the matrix of the same map in the new basis is . Similar matrices represent the same map and therefore share eigenvalues — which is why the eigendecomposition can be read as “the map is diagonal in the eigenbasis”.
5. Data view: a map to another space¶
Everything above was written for , but the same words describe the step that makes modern learning possible: a feature map sends each data point to another space, where a linear model may be enough. Two examples from this course:
PCA is a map to a low-dimensional subspace, chosen to lose as little variance as possible.
The kernel page replaces by its Gram matrix , so the map is never computed, only its inner products. The kernel is a similarity, and section 2 is why: an inner product is a coordinate-free description of the geometry in .
Lab pointers¶
Read these notebooks next to this page, they are the computational half of it:
Linear-Dependence-and-Span, Projection, Multiplying-Matrices-and-Vectors, and then
Eigendecomposition for the change-of-basis statement of section 4.
Exercises¶
Take a matrix, describe its column space geometrically and give a basis for it.
Give a linear map that is not invertible, describe its kernel, and check that the dimensions satisfy rank plus nullity equals 3.
Explain why a linear map is determined by the images of a basis, and use that to reconstruct a matrix from two image vectors.
For , find bases of and and verify rank plus nullity equals 3. Why does the system have solutions only for some , and how do you recognise those ?
Show that with satisfies and , and that is the number of columns of . Interpret both facts geometrically.
Let be the matrix whose columns are and . Compute for a symmetric of your choice and check that the eigenvalues did not change. What are the eigenvectors of in terms of those of ?
Give a feature map such that the XOR labelling becomes linearly separable, and write the three coordinates explicitly.
Sources¶
linear-algebra-basics.mdand the linear-algebra notebooksGoodfellow et al., Deep Learning, Ch. 2 (as used in these notes)
Strang, Linear Algebra and Its Applications, Ch. 3–4 (column space, null space, rank–nullity)