Hessian

If for some reason I take another linear algebra class, I’ll move this file there.

Definition (Hessian Matrix)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$. Recall that

$$\nabla =\left(\begin{array}{c}\frac{\partial }{\partial x_1}\\ \frac{\partial }{\partial x_2}\\ ⋮\\ \frac{\partial }{\partial x_n}\end{array}\right)$$

and that $\nabla f$ gives the Gradient of $f$. The Hessian Matrix denoted as $\mathbf{H}f$ is defined as ${\nabla }^{2}f$ or from a linear algebra perspective, $\nabla {\nabla }^{T}f$. In particular,

$$\mathbf{H}f=\nabla {\nabla }^{T}f=\left(\begin{array}{c}\frac{\partial }{\partial x_1}\\ ⋮\\ \frac{\partial }{\partial x_N}\end{array}\right)\left(\begin{array}{ccc}\frac{\partial f}{\partial x_1}& \dots & \frac{\partial f}{\partial x_N}\end{array}\right)=\left(\begin{array}{ccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \dots & \frac{{\partial }^{2}f}{\partial x_1\partial x_N}\\ ⋮& & \\ \frac{{\partial }^{2}f}{\partial x_N\partial x_1}& \dots & \frac{{\partial }^{2}f}{\partial x_N^2}\end{array}\right)$$

describes the Hessian. In particular, we can describe $\mathbf{H}$ as a matrix in ${\mathbb{R}}^{n\times n}$. The Hessian describes the second order mixed partials of a scalar field.

Example 1

Suppose $f:{\mathbb{R}}^2\to \mathbb{R}$. Then

$$\nabla f={\left(f_x,f_y\right)}^T$$

and

$$\mathbf{H}f=\left[\begin{array}{cc}{f}_{xx}& f_{xy}\\ f_{yx}& f_{yx}\end{array}\right]$$

Furthermore, $f_{xy}$ is equal to

$$\frac{{\partial }^{2}f}{\partial x\partial y}$$

and likewise with the other $3$ values. Note the order.

Hessian Intuitively

We can think of the Hessian operator as a double-derivative in the same way the Gradient or Jacobian is treated as a single derivative. Recall that the first derivative is no longer a scalar but rather a linear map (i.e. derivative of $f(x)=x^2$ is $2x$, a linear map). The second derivative is instead a bilinear map.

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a smooth function. The first-derivative

$$Df:{\mathbb{R}}^n\to L({\mathbb{R}}^n,\mathbb{R})$$

is a smooth function from ${\mathbb{R}}^n$ to the set $L({\mathbb{R}}^n,\mathbb{R})$ of linear maps ${\mathbb{R}}^n\to \mathbb{R}$. From the example above, $2x$ is a map that is given from the derivative of $f$. We can do the same with Hessian

$$D^{2}f:{\mathbb{R}}^n\to L({\mathbb{R}}^n,L({\mathbb{R}}^n,\mathbb{R}))$$

where the codomain is (trivially) isomorphic to the set of bilinear maps

$$\text{Bil}({\mathbb{R}}^n\times {\mathbb{R}}^n,\mathbb{R})$$

But this is precisely what $H$ represents. It is a matrix operator in ${\mathbb{R}}^{n\times n}$. Similar to how double-derivatives model the concavity of a point on some curve/surface, $\mathbf{H}$ models the concavity of a hypersurface at a point.¹

Footnotes

1. See this MSE post. ↩