There are countless numbers of probability distributions. Some of them are so widely used and beautiful that they deserve a name. Surprisingly, all of those distributions can be derived starting from a single coin toss. Here’s the demonstration.
Consider a coin toss…
Let’s start again with a coin toss. This time, think of a coin that lands heads with probability and probability . This is called a Bernoulli distribution, and we write this as Surprisingly, almost all important distributions we encounter in statistics and machine learning can be derived by combining this single coin toss somehow. Let’s start with a simple example.
This is equivalent to tossing the same coin times. Namely, let , meaning coin tosses with the same probability of landing heads, that are independent of each other. Then, the total number of heads in these coin tosses are:
This was a pretty straightforward example. Now, let’s move on to something a bit more complicated.
If you know what a uniform distribution is, it might seem counterintuitive at first that it can be generated using coin tosses. But we can! Again, let . Then,
Conceptually, we can generate uniform distributions by infinite tosses of a fair coin. To see why, let’s think about the CDF of . Using the above expression of the uniform distribution, we hope to derive . To do so, think of as a binary expansion . Similarly, . To see when , we only need to check the first decimal point in which and defer (e.g. comparison of and can be made by comparing the third decimal point). Conditioning on , the decimal point in which the two numbers differ, we have:
The first equation follows from the fact that . The second equation is using the law of total probability.
Exponential distribution can be defined using a uniform distribution:
where . Because the uniform distribution is generated using coin tosses, we can also think of exponential distribution as generated from them.
If you haven’t seen exponential distributions before, it is a distribution that comes up a lot in Poisson Processes (which is an important probability concept I’ll describe in another post). The most important thing to remember about the exponential distribution is the memoryless property:
To have an intuitive understanding, think of as a waiting time of a bus at a bus stop. following an exponential distribution means that the time you have waited so far tells you nothing about the time you will wait until the bus comes (hence the term “memoryless”). This property is crucial in modeling natural phenomena such as radioactive particle decays in physics. Surprisingly, a continuous random variable is memoryless if and only if it has an exponential distribution (proof).
In contrast to Exponential, a discrete random variable is memoryless if and only if it has a Geometric distribution.
where is exponential.
Gamma distribution is just a sum (or convolution) of i.i.d. Exponentials. Let . Then,
Note that this only defines the Gamma distribution when is a positive integer. More comprehensive definition in case will be left to other sources.
Gamma distribution is important because it is the conjugate prior for Poisson distribution (coming soon). Hence, it is important in Bayesian statistics. The inverse of Gamma distribution is also widely used in Bayesian statistics to model the prior variance of a Normal distribution (again, coming soon). I know this is a lot of information; for now, think of Gamma as some distribution derived from Exponential that is important in Bayesian statistics.
Chi Squared Distribution
Chi Squared distribution is just a special case of Gamma distribution. Namely, Chi Squared distribution with degrees of freedom is
For now, don’t worry too much about “ degrees of freedom” part (it just means that there are parameters we can vary). Chi Squared distribution is often used in hypothesis testing. As noted in the next section, it has a close relationship with the more famous distribution, Normal distribution.
Normal distribution is defined using Chi distribution (which is just the square root of Chi Squared distribution) and a random sign. A random sign is a random variable which takes the value with probability and with probability . Let the former be and the latter be . Then,
is a standard normal, because it has a mean and a variance . Any normal distribution can be obtained from :
Another relationship between normal and Chi distributions is that Chi Squared distribution is sum of i.i.d. standard normals. Since the sum of n iid Gamma(1/2) is Gamma(n/2).
Student-t, Cauchy, Log Normal
These are all distributions that can be derived using Normals and/or Chi-Squared. I will not go into depth with each one, but you will for sure come across each of this as you study more statistics and machine learning.
- Student-t: where Z is a standard normal and . Widely used in hypothesis testing.
- Cauchy: where are i.i.d standard normals. Famous because mean and variance are undefined.
- Log Normal: where X is Normal. It’s just an exponential of a normal random variable, but widely used in modeling.
Beta distribution is widely used in Bayesian statistics because it is the conjugate prior of Benroulli and Binomial distributions.
where , and they are independent. As you can immediately see, Beta distribution has a close connection with the Gamma distribution. Fun fact is that and are independent if and only if are independent Gammas. In such case, is Beta and is also a Gamma. If we think of as waiting time in a bus stop and as waiting time at a station, this tells us that the proportion of time one waited at a bus stop tells nothing about the total time waited.
Last but not least, Possion distribution is defined using Exponential and a Poisson Process. This is tricky, so let me explain it in depth. Think of buses arriving at a bus stop. Let be their arrival times. Let or equivalently, (think about why). Then , the number of bus arrivals until time t follows a Poisson process:
It all started from a coin toss… We’ve come this far. Perhaps in the end, you’ve forgotten about that coin toss in the beginning. But if you carefully track how the distributions are related to each other, you’ll start to see how a coin toss magically opens up the door to the world of probability distributions.