ThoughtsOfChains
← Fundamentals

Derivatives

If you take a closer look at how functions work, there's a really important characteristic that became a cornerstone of mathematics and really important for AI.
Take a look at how some functions change at every step

For the function f(x)=xf(x) = x :

visualization of function f(x) = x

but the rate of change between functions can be different

And this function f(x)=x2f(x) = x^2 :

visualization of function f(x) = x

How to discover the rate of change of this functions and why is it important?

The theory behind this is Calculus. It's a huge field of mathematics, so I'm not going to try to explain all of it here. Instead, we only need one idea for now: the derivative.

A good intuition is to think about the velocity of a car.

Imagine a car traveling along a road. We want to know its velocity at a specific moment.
We could measure how long it takes the car to travel from point A to point B: If the car travels 100 meters in 5 seconds, its average velocity over that interval is:

100 m5 s=20 m/s\frac{100\ \text{m}}{5\ \text{s}} = 20\ \text{m/s}

car going through point A to B

But that's only the average velocity. The car could have accelerated, slowed down, or changed its speed somewhere between A and B. So what if we make the distance between A and B smaller?

As point B gets closer and closer to point A, the average rate of change over that tiny interval becomes a better approximation of the car's velocity at that particular moment.

In calculus, this idea is pushed further: what happens as the distance between the two points approaches zero? (this is called a limit)

The resulting value is the "instantaneous rate of change"

The formal math representation of a derivative is:

f′(x)=lim⁡h→0f(x+h)−f(x)hf'(x) = \lim_{h \to 0} \frac{f(x+h)-f(x)}{h}

Finally, derivatives are used to make models learn through a technique called backpropagation. Over the years, a bunch of “shortcuts” have been developed to make calculating derivatives easier. One of them, called the chain rule, is essentially what backpropagation is all about.

This was just a short introduction, but learning this concept will make some algorithms really easier to interpret.

Sources: