Introduction

A recent popular tweet did a "math magic trick", and I want to explain why it works and use that as an excuse to talk about cool math (functional analysis). The tweet in question:

This is a cute magic trick, and like any good trick they nonchalantly gloss over the most important step. Did you spot it? Did you notice your confusion?

Here's the key question: Why did they switch from a differential equation to an integral equation? If you can use when $x = \int$ , why not use it when $x = d / d x$ ?

Well, lets try it, writing $D$ for the derivative:

$\begin{matrix} f^{'} & = & f (1 - D) f & = & 0 f & = & (1 + D + D^{2} + . . .) 0 f & = & 0 + 0 + 0 + . . . f & = & 0 \end{matrix}$

So now you may be disappointed, but relieved: yes, this version fails, but at least it fails-safe, giving you the trivial solution, right?

But no, actually $(1 - D)^{- 1} = 1 + D + D^{2} + . . .$ can fail catastrophically, which we can see if we try a nonhomogeneous equation like $f^{'} = f + e^{x}$ (which you may recall has solution $x e^{x}$ ):

$\begin{matrix} f^{'} & = & f + e^{x} (1 - D) f & = & e^{x} f & = & (1 + D + D^{2} + . . .) e^{x} f & = & e^{x} + e^{x} + e^{x} + . . . f & = & \infty ? \end{matrix}$

However, the integral version still works. To formalize the original approach: we define the function $I$ (for integral) to take in a function $f (x)$ and produce the function $I f$ defined by $I f (x) = \int_{0}^{x} f (t) d t$ . This rigorizes the original trick, elegantly incorporates the initial conditions of the differential equation, and fully generalizes to solving nonhomogeneous versions like $f^{'} = f + e^{x}$ (left as an exercise to the reader, of course).

So why does $(1 - D)^{- 1} = 1 + D + D^{2} + . . .$ fail, but $(1 - I)^{- 1} = 1 + I + I^{2} + . . .$ works robustly? The answer is functional analysis!

Functional Analysis

Savvy readers may already be screaming that the trick $(1 - x)^{- 1} = 1 + x + x^{2} + . . .$ for numbers only holds true for $| x | < 1$ , and this is indeed the key to explaining what happens with $D$ and $I$ ! But how can we define the "absolute value" of "the derivative function" or "the integral function"?

What we're looking for is a norm, a function that generalizes absolute values. A norm is a function $x \mapsto | | x | |$ satisfying these properties:

$| | x | | \geq 0$ for all $x$ (positivity), and $| | x | | = 0$ if and only if $x = 0$ (positive-definite)
$| | x + y | | \leq | | x | | + | | y | |$ for all $x$ and $y$ (triangle inequality)
$| | c x | | = | c | \cdot | | x | |$ for all $x$ and real numbers $c$ , where $| c |$ denotes the usual absolute value (absolute homogeneity)

Here's an important example of a norm: fix some compact subset of $R$ , say $X = [- 10, 10]$ , and for a continuous function $f : X \to R$ define $| | f | |_{\infty} = {max}_{x \in X} | f (x) |$ , which would commonly be called the $L^{\infty}$ -norm of $f$ . (We may use a maximum here due to the Extreme Value Theorem. In general you would use a supremum instead.) Again I shall leave it to the reader to check that this is a norm.

This example takes us halfway to our goal: we can now talk about the "absolute value" of a continuous function that takes in a real number and spits out a real number, but $D$ and $I$ take in functions and spit out functions (what we usually call an operator, so what we need is an operator norm).

Put another way, the $L^{\infty}$ -norm is "the largest output of the function", and this will serve as the inspiration for our operator norm. Doing the minimal changes possible, we might try to define $| | I | | = {max}_{f continuous} | | I f | |_{\infty}$ . There are two problems with this:

First, since $I$ is linear, you can make $| | I f | |_{\infty}$ arbitrarily large by scaling $f$ by 10x, or 100x, etc. We can fix this by restricting the set of valid f for these purposes, just like how for the $L^{\infty}$ example restricted the inputs of $f$ to the compact set $X = [- 10, 10]$ . Unsurprisingly nice choice of set to restrict to is the "unit ball" of functions, the set of functions with $| | f | |_{\infty} \leq 1$ .
Second, we must bid tearful farewell to the innocent childhood of maxima, and enter the liberating adulthood of suprema. This is necessary since $f$ ranges over the infinite-dimensional vector space of continuous functions, so the Heine-Borel theorem no longer guarantees the unit ball is compact, and therefore the extreme value theorem no longer guarantees that we will attain a maximum.

So the proper definition of the norm of $I$ and $D$ are:

$| | I | | = {sup}_{f continuous, | | f | |_{\infty} \leq 1} | | I f | |_{\infty}$

$| | D | | = {sup}_{f continuous, | | f | |_{\infty} \leq 1} | | D f | |_{\infty}$

(and you can define similar norms for any linear operator, including $I^{2}$ , etc.) A good exercise is to show these equivalent definitions of the operator norm for any linear function L:

$\begin{matrix} | | L | | & = & sup f continuous, | | f | |_{\infty} \leq 1 | | L f | |_{\infty} = & sup f continuous, | | f | |_{\infty} = 1 | | L f | |_{\infty} = & inf {b \geq 0 : | | L f | |_{\infty} \leq b | | f | |_{\infty} for all f} \end{matrix}$

So another way of thinking of the operator norm is the maximum stretching factor of the linear operator. The third definition also motivates the terminology of bounded linear operators: each such $b$ is a bound on the operator $L$ , and the least such bound is the norm. Fun exercise: show that a linear operator is bounded if and only if it is continuous (with respect to the correct topologies). Hint: you'll need to work in infinite dimensional spaces here, because any finite-dimensional linear operator must be bounded.

Now let's actually compute these norms! For $I$ , remember that our $L^{\infty}$ -norm is defined over the interval $X = [- 10, 10]$ . First observe that for the constant function $f (x) = 1$ , $I f (x) = x$ , so $| | I f | |_{\infty} = 10$ . Thus $| | I | | \geq 10$ . To show that this is indeed the maximum we use the triangle inequality for integrals:

$\begin{matrix} | | I f | | & = & max x \in X | \int_{0}^{x} f (t) d t | \leq & max x \in X \int_{0}^{x} | f (t) | d t \leq & max x \in X \int_{0}^{x} 1 d t = & max x \in X x = 10 \end{matrix}$

So we have shown $| | I | | = 10$ ! Put a pin in that while we check $| | D | |$ .

For $D$ , we have a problem: for any positive number $k$ , $D (e^{k x}) = k e^{k x}$ . In other words, $D$ can stretch functions by any amount, so it has no norm, or we'd write $| | D | | = \infty$ (and I promise this is a failure of $D$ , not of our definitions). Put another way, $D$ is not bounded as a linear operator, since it can stretch functions by an arbitrary amount.

But now let's return to $I$ . We said that $| | I | | = 10$ (if we're defining it relative to the $L^{\infty}$ -norm on $X = [- 10, 10]$ ), but isn't $(1 - x)^{- 1} = 1 + x + x^{2} + . . .$ only true when $| x | < 1$ ? For real numbers, yes, but for operators, something magical happens: $| | I^{n} | | \neq | | I | |^{n}$ ! (Its like there's a whole algebra of these operators...)

In fact, you can show that $| | I^{n} | |$ assumes its maximum value when applied to the constant function $f (x) = 1$ , and hence have $| | I^{n} | | = \frac{1}{n!} | | I | |^{n}$ . Since $n!$ grows faster than exponential functions, $| | I^{n} | |$ is converging to 0 quickly, so $1 + I + I^{2} + I^{3} + . . .$ is a Cauchy sum, and it is then straightforward to show that the limit is the multiplicative inverse of $1 - I$ . Thus, $(1 - I)^{- 1} = 1 + I + I^{2} + . . .$ is a valid expression that you can apply to any continuous (or bounded) function $f$ on any compact set $X$ . This convergence happens regardless of the choice of the compact set, though it will happen at different rates, analogous to uniform convergence on compact sets.

Summary

Writing $D$ for derivative and $I$ for integral, we showed that $(1 - D)^{- 1} = 1 + D + D^{2} . . .$ can fail, even though $(1 - I)^{- 1} = 1 + I + I^{2} . . .$ is always true.
To explain this, we have to show that $I$ is fundamentally better behaved than $D$ , in a way analogous to $| x | < 1$ .
We built this up in two steps. First, we defined the $L^{\infty}$ -norm for real-valued functions, which lets you say how "large" those functions are. Then, we extended this to function-valued functions (operators), having to make two slight modifications along the way.
With this machinery in place, we could show that $| | I | | < \infty$ , or we can say that $I$ is bounded. The resulting norm depends on the domain of the functions $f$ under consideration, but any compact domain is allowable. Also, since $| | I^{n} | | = \frac{1}{n!} | | I | |^{n}$ , the exact value doesn't matter since the norm of each term goes to 0.
Since $| | I^{n} | | \to 0$ sufficiently quickly, we can say that $1 + I + I^{2} + I^{3} + . . .$ is Cauchy as a sequence of operators. In other words, if you apply the partial sums $1, 1 + I, 1 + I + I^{2}, . . .$ as operators to any function $f$ , the functions $f, f + I f, f + I f + I^{2} f, . . .$ will converge with respect to the $L^{\infty}$ -norm. Writing $L f$ for the function they converge to, it follows that $(1 - I) L f = f$ , so we may write $L = 1 + I + I^{2} + I^{3} + . . . = (1 - I)^{- 1}$ as a statement about linear operators.
In contrast, $D$ is unbounded as an operator, meaning $| | D | | = \infty$ . Thus algebra tricks like $(1 - D)^{- 1} = 1 + D + D^{2} . . .$ will break down if you put in the wrong function $f$ .