Dual Space

Definition (Linear Map)

Let $U,V$ be two vector spaces. A map $A:U\to V$ is linear if

$$A({\vec{u}}_1+{\vec{u}}_2)=A({\vec{u}}_1)+A({\vec{u}}_2)\text{and}A(c\vec{u})=cA(\vec{u})$$

Theorem (Set of Linear Maps is a Vector Space)

The set $\left(U\stackrel{\text{linear}}{\to }V\right)$ of all linear maps from $U$ to $V$ is a vector space.

Definition (Covector)

A covector on a vector space $V$ is a scalar-valued linear function

$$\alpha :V\stackrel{\text{linear}}{\to }\mathbb{R}$$

A covector is a linear function that takes in a vector and outputs a scalar. Geometrically, it is best to visualize a covector as a set of equidistant, parallel hyperplanes. In 2D, these are parallel lines. In 3D, they are parallel planes. They are like “slabs”. It is best to visualize them by its level sets.

Definition (Dual Space)

The dual space

$$V^{\ast }=\left\{\alpha :V\stackrel{\text{linear}}{\to }\mathbb{R}\right\}$$

of a vector space $V$ is the vector space of all covectors on $V$. The elements of $V^{\ast }$ are called dual vectors or covectors. The dual pairing is denoted by

$$\alpha (\vec{u})=⟨\alpha |\vec{u}⟩=\alpha [[\vec{u}]]$$

It is the act of “feeding” a vector $\vec{u}$ into a covector $\alpha$ to get a scalar value. Visually, evaluating the dual pairing means counting the number of times the vector’s arrow pierces through the hyperplanes of the covector. If the arrow runs perfectly parallel to the covector’s lines, it pierces $0$, and the pairing is $0$. If the arrow stretches across $3$ level-set lines, the pairing evaluates to $3$.

Theorem (Dimension of Dual Space)

The dimension of the dual space $V^{\ast }$ is equal to the dimension of the original vector space $V$. That is,

$$\dim V^{\ast }=\dim V$$

If $V$ is a finite-dimensional vector space, then $\dim V^{\ast \ast }=\dim V$.

Theorem (Dual Basis)

Let ${\vec{e}}_1,{\vec{e}}_2,\dots ,{\vec{e}}_n$ be a basis for a finite-dimensional vector space $V$. Then there is a unique dual basis ${\beta }^1,{\beta }^2,\dots ,{\beta }^n\in V^{\ast }$ so that

$$⟨{\vec{e}}_i|{\beta }^j⟩={\delta }_i^j=\{\begin{array}{ll}1& \text{if }i=j\\ 0& \text{if }i\ne j\end{array}$$

where ${\delta }_i^j$ is the Kronecker delta. If we write

$$\vec{u}=\sum _{i=1}^n{u}^i{\vec{e}}_i$$

using a basis, then we would write a covector $\alpha$ as

$$\alpha =\sum _{j=1}^n{\alpha }_j{\beta }^j$$

The dual pairing simply becomes

$$⟨\alpha |\vec{u}⟩=\sum _{i=1}^n{\alpha }_i{u}^i$$

Likewise to the visualization of the dual pairing, given some basis vector ${\vec{e}}_1\in V$, the corresponding ${\beta }^1$ dual vector means that the pairing evaluates to $1$, or that it crosses the covector slab exactly once.

Conventional Notation

Vectors are conventionally columnar. Covectors are row vectors. The dual pairing is automatic

$$\left[\begin{array}{ccc}{\alpha }_1& \cdots & {\alpha }_n\end{array}\right]\left[\begin{array}{c}{u}^1\\ ⋮\\ u^n\end{array}\right]=\sum _{i=1}^n{\alpha }_i{u}^i$$

Vectors are labeled with an upper index $u^i$ and covectors are labeled with a lower index ${\alpha }_j$. We can also use the bra-ket notation, where the vector $\vec{u}$ is written as a “ket” $|\vec{u}⟩$ and the covector $\alpha$ is written as a “bra” $⟨\alpha |$. The dual pairing is then written as $⟨\alpha |\vec{u}⟩$. There is also Penrose graphical notation.

Definition (Adjoint Linear Map)

For each linear map $A:U\to V$, there is a linear map called its adjoint $A^{\top }:V^{\ast }\to U^{\ast }$ defined by

$$⟨A^{\top }\beta |\vec{u}⟩=⟨\beta |A\vec{u}⟩$$

for all $\vec{u}\in U$ and $\beta \in V^{\ast }$. The adjoint map is also denoted as $A^{\ast }$.

Definition (Tangent Space)

The tangent space $T_{p}M$ at $p$ is the vector space of perturbations for point $p$.

$$T_{p}M=\left\{(p,v)\right\}$$

This is a family of vector spaces. In type theory, tangent spaces form a dependent type. This is because it depends on the point $p$.

Definition (Tangent Bundle)

The tangent bundle is the space of all perturbation data

$$TM=\underset{p\in M}{⨆}{T}_{p}M$$

In type theory, the tangent bundle is a sum dependent type. It helps because we can say that

$$x\in TM⟺\exists p\in M\text{ such that }{T}_{p}M\ni x$$

Definition (Cotangent Space)

The cotangent space $T_p^{\ast }M$ at $p$ is the dual space of tangent space.

Definition (Cotangent Bundle)

The cotangent bundle is the space of all covectors

$$T^{\ast }M=\underset{p\in M}{⨆}{T}_p^{\ast }M$$

Definition (Vector Field, Covector Field)

A vector field is a function of the following type

$$\begin{array}{rl}X:M& \to TM\\ X(p)& \in T_{p}M\end{array}$$

We also call a vector field a section of the tangent bundle.

$$X\in \Gamma (TM)=\{X:M\to TM\mid X_p\in T_{p}M\}$$

In type theory, this is a product dependent type. This is because the output type $T_{p}M$ depends on the input $p$.

Similarly, a covector field $\alpha \in \Gamma (T^{\ast }M)$ is a function if type

$$\begin{array}{rl}\alpha :M& \to T^{\ast }M\\ \alpha (p)& \in T_p^{\ast }M\end{array}$$

We also call a covector field a 1-form, and denote $\Gamma (T^{\ast }M)={\Omega }^1(M)$.

Types in Differential Calculus

Let

$$f\in \left(M\stackrel{\text{nonlinear}}{\to }\mathbb{R}\right)$$

The differential of a function $f$ at a point $p$ is a linear map that approximates the change in $f$ near $p$. It is a covector field

$$df\in \Gamma (T^{\ast }M)={\Omega }^1(M)$$

where at any specific point $p$, we have $df{|}_p\in \left(T_{p}M\stackrel{\text{linear}}{\to }\mathbb{R}\right)=T_p^{\ast }M$.

Definition (Directional Derivative)

The perturbation at a point is a tangent vector

$$v_p\in T_{p}M$$

The directional derivative of $f$ at $p$ in the direction of $v_p$ is the dual pairing:

$$df|p[[v_p]]\in \mathbb{R}$$

Geometrically, let $\gamma :(-\epsilon ,\epsilon )\to M$ be any curve satisfying $\gamma (0)=p$ and ${\gamma }^{\prime }(0)=v$. Then we can compute the directional derivative as the rate of change of $f$ along the curve:

$$d{f}_p[[v]]=\frac{d}{dt}f(\gamma (t)){|}_{t=0}$$

This function $df$ depends linearly in $v$, but generally nonlinearly in $p$. Each data $(p,v)$ admits a representing curve $\gamma$ that passes through $p$ with velocity $v$ at $t=0$.

What about with with partial derivatives?

Definition (Coordinate System)

A coordinate system is a family of functions

$$x^1,\dots ,x^n\in \left(M\stackrel{\text{nonlinear}}{\to }\mathbb{R}\right)$$

so that $d{x}^1{|}_p,\dots ,d{x}^n{|}_p\in T_p^{\ast }M$ is a basis at evert $p\in M$.

The partial derivatives defined are the coefficients for the covector $df$ under this coordinate-induced covector basis:

$$df=\sum _{i=1}^n\frac{\partial f}{\partial x^i}d{x}^i$$

Definition (Riemannian Manifold)

Let $M$ be a Manifold (domain). A Riemannian manifold is a manifold $M$ together with an inner product structure ${♭}_p:T_{p}M\to T_p^{\ast }M$ at every point $p\in M$.

In general, given a Riemannian manifold, we don’t expect to find a coordinate system so that the coordinate vectors are orthonormal everywhere. The failure to find an everywhere-orthonormal coordinate is the non-Euclidean-ness of the Riemannian manifold.

Definition (Gradient Vector)

Let $(M,♭)$ be a Riemannian manifold. The gradient of a function $f\in C^{\infty }(M)$ at $p\in M$ is defined by

$${\text{grad}}_{p}f:={♯}_p{d}_{p}f\in T_{p}M$$

The gradient does not require coordinates and has little to do with partial derivatives. In index notation, it is

$$(\text{grad}f{)}^i={♯}^{ij}\frac{\partial f}{\partial x^j}$$

The correct way of writing gradient descent is

$$\frac{d}{dt}{x}^i=-{♯}^{ij}\frac{\partial f}{\partial x^j}$$

Chain Rules

The differential of a function $f:M\to \mathbb{R}$ at a point $p$ is a linear map $df{|}_p:T_{p}M\to \mathbb{R}$. This concept generalizes to a map between manifolds $\varphi :M\to N$.

Definition (Pushforward Operator)

The pushforward operator ${\varphi }_{\ast }$ (also called the differential or Jacobian) of a map $\varphi :M\to N$ is a map between tangent bundles:

$$\begin{array}{rl}{\varphi }_{\ast }:TM& \to TN\\ {\varphi }_{\ast }(u_p)& :=d\varphi {|}_p{u}_p\end{array}$$

Geometrically, it “pushes” a velocity vector $u_p$ at $p\in M$ to a velocity vector at $\varphi (p)\in N$. In coordinates, this is matrix-vector multiplication by the Jacobian.

Definition (Pullback Operator for Functions)

The pullback operator ${\varphi }^{\ast }$ for functions takes a function on the target space and “pulls” it back to the source space:

$$\begin{array}{rl}{\varphi }^{\ast }:(N\to \mathbb{R})& \stackrel{\text{linear}}{\to }(M\to \mathbb{R})\\ {\varphi }^{\ast }(g)& :=g\circ \varphi \end{array}$$

If $g$ is a scalar field on $N$, then ${\varphi }^{\ast }g$ is a scalar field on $M$.

Definition (Pullback Operator for Covectors)

The pullback operator ${\varphi }^{\ast }$ for covectors is the adjoint of the pushforward. It pulls covectors (1-forms) on $N$ back to $M$:

$$\begin{array}{rl}{\varphi }^{\ast }:T^{\ast }N& \to T^{\ast }M\\ {\varphi }^{\ast }({\beta }_{\varphi (p)})& =(d\varphi {|}_p{)}^{\top }({\beta }_{\varphi (p)})\end{array}$$

The defining property is the dual pairing: $⟨{\varphi }^{\ast }\beta |u⟩=⟨\beta |{\varphi }_{\ast }u⟩$.

Backpropagation and Machine Learning

In machine learning, we often have a composition of maps (layers) ${\varphi }_1,{\varphi }_2,\dots ,{\varphi }_k$ and a final scalar loss function $L:{\mathbb{R}}^n\to \mathbb{R}$.

Forward Pass (Pushforward)

The forward pass is the computation of the maps themselves: $x\mapsto y_1\mapsto y_2\mapsto \dots \mapsto \text{Loss}$. The sensitivity of the output to a perturbation $\delta x$ is computed by the pushforward:

$$\delta \text{Loss}=L_{\ast }({\varphi }_k{)}_{\ast }\dots ({\varphi }_1{)}_{\ast }\delta x$$

This is the “forward-mode” differentiation.

Backward Pass (Pullback)

Backpropagation is the pullback of the loss differential. We start with the differential of the loss function $dL$, which is a covector (a row vector of partial derivatives). We want to find how the loss changes with respect to the input $x$.

Using the property $(\psi \circ \varphi {)}^{\ast }={\varphi }^{\ast }\circ {\psi }^{\ast }$, we pull the covector $dL$ back through the layers:

$$dx={\varphi }_1^{\ast }{\varphi }_2^{\ast }\dots {\varphi }_k^{\ast }dL$$

Notice the order is reversed. In each step, we apply the adjoint of the local Jacobian (the transpose of the weight matrix in linear layers) to the “incoming” gradient covector.

Why use pullbacks? If the input $x$ is high-dimensional (e.g., an image) and the output is a single scalar (the loss), it is computationally much cheaper to pull back a single covector than to push forward a basis of the entire tangent space. This is why “reverse-mode” autodiff (backprop) is the standard for deep learning.

CNN Example

In a CNN, the trainable parameters consist of the weights within the convolutional kernels (filters) and their associated biases. Let the manifold $M$ represent the space of all possible parameter values. Each point $p\in M$ is a specific setting of the parameters at some training step. The tangent space $T_{p}M$ represents all possible perturbations to the parameters at that point. When an optimizer like SGD updates the parameters, it is essentially moving in the tangent space $T_{p}M$ to a new point $p^{\prime }$ by some update rule.

CNN’s operate on feature maps (multi-dimensional arrays, tensors). Let a single convolutional layer be a map $\varphi$ from an input tensor space to an output tensor space.

$$\varphi :{\mathbb{R}}^{H\times W\times C_{\text{in}}}\to {\mathbb{R}}^{H^{\prime }\times W^{\prime }\times C_{\text{out}}}$$

where $H,W$ is the height and width of the input feature map, and $C_{\text{in}},C_{\text{out}}$ are the number of input and output channels, respectively. The input tensor $x$ is a vector in this domain. The pushforward (or differential) denoted by ${\varphi }_{\ast }$ or $d\varphi$, represents the Jacobian of the convolutional operation. Geometrically, if we perturb the input tensor $x$ by a small amount $\delta x$, the pushforward tells us how the output tensor changes:

$$\delta y={\varphi }_{\ast }\delta x$$

Because convolution is a linear operation, the pushforward is simply the convolution itself.

The network concludes with a scalar loss function

$$L:{\mathbb{R}}^{H^{\prime }\times W^{\prime }\times C_{\text{out}}}\to \mathbb{R}$$

The differential of the loss function $dL$ is a covector, belonging to the cotangent space $T_{\varphi (x)}^{\ast }{\mathbb{R}}^{H^{\prime }\times W^{\prime }\times C_{\text{out}}}$. It is a linear functional waiting to be fed a perturbation vector to tell us how the loss changes. To compute the gradient with respect to the input $x$, we do not invert the matrix. Instead, we use the pullback of the convolutional layer, ${\varphi }^{\ast }$. The pullback is the adjoint of the pushforward

$${\varphi }^{\ast }\beta =(d\varphi {)}^{\top }\beta$$

In the specific context of a CNN, the adjoint of a convolution operation is mathematically equivalent to a transposed convolution (also known as a deconvolution). This is why the backpropagation step in a CNN involves a transposed convolution. During backpropagation, the loss covector $\beta$ is pulled back through the layers by applying a transposed convolution using the same kernel weights. This mathematically mirrors the equation

$$dx={\varphi }^{\ast }dL$$

Eventually, this yields the gradient of the loss with respect to the input $x$, the covector

$$d{L}_p\in T_p^{\ast }M$$

Since we cannot directly add a covector to a vector (manifold), we need a tangent vector. This requires the Riemannian metric ${♯}_p:T_{p}M\to T_p^{\ast }M$ to convert the covector $d{L}_p$ into a tangent vector

$${\text{grad}}_{p}L={♯}_{p}d{L}_p\in T_{p}M$$

This vector ${\text{grad}}_{p}L$ represents the direction of steepest ascent in the tangent space.

The following is unrelated to the class, but this section was further elaborated via Gemini 3.1 pro.

To get back to the original parameter space, we need to apply the “exponential map” ${\exp }_p:T_{p}M\to M$ to move from the tangent space back to the manifold. The idea is to “start at $p$, face in the direction of the gradient, and walk the straightest possible path (geodesic)“. To update our weights, we scale our gradient vector by a learning rate $\eta$ and then apply the exponential map:

$$p_{t+1}={\exp }_{p_t}(-\eta {\text{grad}}_{p_t}L)$$

and so on. In practice, this notation is never used because we always assume the parameter manifold $M$ is a perfectly flat Euclidean space ${\mathbb{R}}^n$. It’s geometry is trivial, so the “straightest possible path” is just a straight line.

Theorem (Derivative and Pullback Commute)

Let $\varphi :M\to N$ be a map between manifolds, and let $f:N\to \mathbb{R}$ be a function on the target manifold. Then the following holds:

$${\varphi }^{\ast }(df)=d({\varphi }^{\ast }f)$$

Theorem (Pushforward Distributes over Composition)

Let $\varphi :M\to N$ and $\psi :N\to P$ be maps between manifolds. Then the pushforward distributes over composition:

$$({\varphi }_1\circ \cdots \circ {\varphi }_k{)}_{\ast }=({\varphi }_1{)}_{\ast }\cdots ({\varphi }_k{)}_{\ast }$$

This is the same as the chain rule for derivatives.

Theorem (Pullback Distributes over Composition and Reverses Order)

Let $\varphi :M\to N$ and $\psi :N\to P$ be maps between manifolds. Then the pullback distributes over composition and reverses order:

$$({\varphi }_1\circ \cdots \circ {\varphi }_k{)}^{\ast }=({\varphi }_k{)}^{\ast }\cdots ({\varphi }_1{)}^{\ast }$$

Because the pullback is the adjoint (transpose) of the pushforward, distributing it across a composition reverses the order of operations. This is why backpropagation goes in the reverse order of the forward pass.