Aydin Jarvis

2022-10-13

Entropy of geometric random variable?
I am wondering how to derive the entropy of a geometric random variable? Or where I can find some proof/derivation? I tried to search online, but seems not much resources is available.
Here is the probability density function of geometric distribution: $\left(1-p{\right)}^{k-1}\phantom{\rule{thinmathspace}{0ex}}p$
Here is the entropy of a geometric distribution: $\frac{-\left(1-p\right){\mathrm{log}}_{2}\left(1-p\right)-p{\mathrm{log}}_{2}p}{p}$
Where p is the probability for the event to occur during each single experiment.

Dobricap

Step 1
Assume $P\left(X=k\right)=\left(1-p{\right)}^{k-1}p$, where $k\in {Z}^{+}$, then the entropy is
Step 2
$\begin{array}{rl}Entropy\left(X\right)& =\sum _{k=1}^{+\mathrm{\infty }}-\left(1-p{\right)}^{k-1}p\cdot {\mathrm{log}}_{2}\left(\left(1-p{\right)}^{k-1}p\right)\\ & =-p\cdot {\mathrm{log}}_{2}\left(p\right)\sum _{k=1}^{+\mathrm{\infty }}\left(1-p{\right)}^{k-1}-p\cdot {\mathrm{log}}_{2}\left(1-p\right)\sum _{k=1}^{+\mathrm{\infty }}\left(k-1\right)\left(1-p{\right)}^{k-1}\\ & =-{\mathrm{log}}_{2}\left(p\right)-\frac{\left(1-p\right){\mathrm{log}}_{2}\left(1-p\right)}{p}\end{array}$

Mattie Monroe

Step 1
It may be faster to derive the above entropy using the memoryless property of the Geometric distribution. Let Y be an auxiliary binary random variable such that $Y=0$ indicates $X>1$. Then $H\left(X\right)={H}_{B}\left(p\right)/p$ follows from the following:
$\begin{array}{rl}H\left(X,Y\right)& =H\left(X|Y\right)+H\left(Y\right)\\ & =H\left(Y|X\right)+H\left(X\right)\end{array}$
Step 2
Note that $\begin{array}{rl}H\left(X|Y\right)& =H\left(X|Y=1\right)P\left(Y=1\right)+H\left(X|Y=0\right)P\left(Y=0\right)=H\left(X|Y=0\right)\left(1-p\right)=\left(1-p\right)H\left(X\right)\\ H\left(Y|X\right)& =0\\ H\left(Y\right)& ={H}_{B}\left(p\right),\end{array}$
where the memoryless property indicates $H\left(X|Y=0\right)=H\left(X\right)$, and ${H}_{B}\left(p\right)$ is the binary entropy function.

Do you have a similar question?