Normal equations

An important concept is that when energy is minimum, the residual is orthogonal to the fitting functions. The fitting functions are the column vectors $\bold f_1$, $\bold f_2$, and $\bold f_3$.Let us verify only that the dot product $\bold r \cdot \bold f_2$ vanishes; to do this, we'll show that those two vectors are orthogonal. Energy minimum is found by  

$$0 \quad = \quad {\partial\over \partial x_2}\ \bold r \cdot ... ...r\over \partial x_2} \quad = \quad 2\; \bold r \cdot \bold f_2$$ (31)

(To compute the derivative refer to equation (13).) Equation (31) shows that the residual is orthogonal to a fitting function. The fitting functions are the column vectors in the fitting matrix.

The basic least-squares equations are often called the ``normal" equations. The word ``normal" means perpendicular. We can rewrite equation (28) to emphasize the perpendicularity. Bring both terms to the left, and recall the definition of the residual $\bold r$from equation (13):

$$\begin{eqnarray} \bold F' ( \bold F \bold x - {\bf d}) &=& \bold 0 \\ \bold F' \bold r &=& \bold 0\end{eqnarray}$$ (32) (33)

Equation (33) says that the residual vector $\bold r$is perpendicular to each row in the $\bold F'$ matrix. These rows are the fitting functions. Therefore, the residual, after it has been minimized, is perpendicular to all the fitting functions.