Chapter 1.6 : Information Theory

Christopher M. Bishop, PRML, Chapter 1 Introdcution

1. Information h(x)

Given a random variable and we ask how much information is received when we observe a specific value for this variable.

• The amount of information can be viewed as the “degree of surprise” on learning the value of .
• information : where the negative sign ensures that information is positive or zero.
• the units of :
  • using logarithms to the base of 2: the units of are bits (‘binary digits’).
  • using logarithms to the base of , i.e., natural logarithms: the units of are nats.

2. Entropy H(x): average amount of information

2.1 Entropy H(x)

Firstly we interpret the concept of entropy in terms of the average amount of information needed to specify the state of a random variable.

Now suppose that a sender wishes to transmit the value of a random variable to a receiver. The average amount of information that they transmit in the process is obtained by taking the expectation of (1.92) with respect to the distribution and is given as

• discrete entropy for discrete random variable by
• or differential/continuous entropy for continuous random variable by

• Note that and so we shall take whenever we encounter a value for such that .
• The nonuniform distribution has a smaller entropy than the uniform one.

2.2 Noiseless coding theorem (Shannon, 1948)

The noiseless coding theorem states that the entropy is a lower bound on the number of bits needed to transmit the state of a random variable.

2.3 Alternative view of entropy H(x)

Secondly, let us introduces the concept of entropy in physics in the context of equilibrium thermodynamics and later given a deeper interpretation as a measure of disorder through developments in statistical mechanics.

Consider a set of identical objects that are to be divided amongst a set of bins, such that there are objects in the bin. Consider the number of different ways of allocating the objects to the bins.

• There are ways to choose the first object, ways to choose the second object, and so on, leading to a total of ways to allocate all objects to the bins.
• However, we don’t wish to distinguish between rearrangements of objects within each bin. In the bin there are ways of reordering the objects, and so the total number of ways of allocating the objects to the bins is given by which is called the multiplicity.
• The entropy is then defined as the logarithm of the multiplicity scaled by an appropriate constant
• We now consider the limit , in which the fractions are held fixed, and apply Stirling’s approximation
• which gives
  

  i.e.,

• where we have used , and is the probability of an object being assigned to the bin.
• microstate: In physics terminology, the specific arrangements of objects in the bins is called a microstate,
• macrostate: the overall distribution of occupation numbers, expressed through the ratios , is called macrostate.
• The multiplicity is also known as the weight of the macrostate.
• We can interpret the bins as the states of a discrete random variable , where . The entropy of the random variable X is then

2.4 Comparison between discrete entropy and continuous entropy

H(x)	Discrete Distribution X	Continuous Distribution X
Maximum	Uniform X	Gaussian X
+/-		could be negative

• Maximum entropy H(x) :
  • In the case of discrete distributions, the maximum entropy configuration corresponded to an equal distribution of probabilities across the possible states of the variable.
  • For a continuous variable, the distribution that maximizes the differential entropy is the Gaussian [see Page 54 in PRML].
• Is Negative or Positive?
  • The discrete entropy in (1.93) is always , because . It will equal its minimum value of when one of the and all other .
  • The differential entropy can be negative, because in (1.110) for .
    

    If we evaluate the differential entropy of the Gaussian, we obtain This result also shows that the differential entropy, unlike the discrete entropy, can be negative, because in (1.110) for .

2.5 Conditional entropy H(y|x)

• Conditional entropy:
  Suppose we have a joint distribution from which we draw pairs of values of and . If a value of is already known, then the additional information needed to specify the corresponding value of is given by . Thus the average additional information needed to specify y can be written as
  which is called the conditional entropy of y given x.
• It is easily seen, using the product rule, that the conditional entropy satisfies the relation where is the differential entropy (i.e., continuous entropy) of , and is the differential entropy of the marginal distribution .
• From (1.112) we get to know that
  

  the information needed to describe and is given by the sum of the information needed to describe alone plus the additional information required to specify given .

3. Relative entropy or KL divergence

3.1 Relative entropy or KL divergence

Problem: How to relate the notion of entropy to pattern recognition?
Description: Consider some unknown distribution , and suppose that we have modeled this using an approximating distribution .

• Relative entropy or Kullback-Leibler divergence, or KL divergence between the distributions and : If we use to construct a coding scheme for the purpose of transmitting values of to a receiver, then the average additional amount of information (in nats) required to specify the value of (assuming we choose an efficient coding scheme) as a result of using instead of the true distribution is given by
  and

• It can be rewritten as [see Ref-1]

• Cross Entropy: where is called the cross entropy,

  • Understanding of Cross Entropy: One can show that the cross entropy is the average number of bits (or nats) needed to encode data coming from a source with distribution when we use model to define our codebook.
  • Understanding of (“Regular”) Entropy: Hence the “regular” entropy , is the expected number of bits if we use the true model.
  • Understanding of Relative Entropy: So the KL divergence is the difference between these (shown in 2.111). In other words, the KL divergence is the average number of extra bits (or nats) needed to encode the data, due to the fact that we used approximation distribution to encode the data instead of the true distribution .

• Asymmetric: Note that KL divergence is not a symmetrical quantity, that is to say .

• KL divergence is a way to measure the dissimilarity of two probability distributions, and [see Ref-1].

3.2 Information inequality [see Ref-1]

The “extra number of bits” interpretation should make it clear that , and that the KL is only equal to zero i.f.f. . We now give a proof of this important result.

Proof:

• 1) Convex functions: To do this we first introduce the concept of convex functions. A function is said to be convex if it has the property that every chord lies on or above the function, as shown in Figure 1.31.
  • Convexity then implies
• 2) Jensen’s inequality:
  • Using the technique of proof by induction(数学归纳法), we can show from (1.114) that a convex function satisfies where and , for any set of points . The result (1.115) is known as Jensen’s inequality.
  • If we interpret the as the probability distribution over a discrete variable taking the values , then (1.115) can be written For continuous variables, Jensen’s inequality takes the form
• 3) Apply Jensen’s inequality in the form (1.117) to the KL divergence (1.113) to give where we have used the fact that is a convex function (In fact, is a strictly convex function, so the equality will hold if, and only if, for all ), together with the normalization condition .
• 4) Similarly, Let be the support of , and apply Jensen’s inequality in the form (1.115) to the discrete form KL divergence (2.110) to get [see Ref-1] where the first inequality follows from Jensen’s. Since is a strictly concave (i.e., the inverse of convex) function, we have equality in Equation (2.115) iff for some . We have equality in Equation (2.116) iff , which implies .
• 5) Hence iff for all .

3.3 How to use KL divergence

Note that:

• we can interpret the KL divergence as a measure of the dissimilarity of the two distributions and .
• If we use a distribution that is different from the true one, then we must necessarily have a less efficient coding, and on average the additional information that must be transmitted is (at least) equal to the Kullback-Leibler divergence between the two distributions.

Problem description:

• Suppose that data is being generated from an unknown distribution that we wish to model.
• We can try to approximate this distribution using some parametric distribution , governed by a set of adjustable parameters , for example a multivariate Gaussian.
• One way to determine is to minimize the KL divergence between and with respect to .
• We cannot do this directly because we don’t know . Suppose, however, that we have observed a finite set of training points , for , drawn from . Then the expectation with respect to can be approximated by a finite sum over these points, using (1.35), so that (???don’t know how to derive it)
• the first term is the negative log likelihood function for under the distribution evaluated using the training set.
• Thus we see that minimizing this KL divergence is equivalent to maximizing the likelihood function.

3.4 Mutual information

Now consider the joint distribution between two sets of variables and given by .

Mutual information between the variables and :

• If and are independent, p(x, y) = p(x)p(y).
• If and are not independent, we can gain some idea of whether they are “close” to being independent by considering the KL divergence between the joint distribution and the product of the marginals, given by which is called the mutual information between the variables and .
• Using the sum and product rules of probability, we see that the mutual information is related to the conditional entropy
  through

Understanding of Mutual information:

• Thus we can view the mutual information as the reduction in the uncertainty about by virtue of being told the value of (or vice versa).
• From a Bayesian perspective, we can view as the prior distribution for and as the posterior distribution after we have observed new data . The mutual information therefore represents the reduction in uncertainty about as a consequence of the new observation .

Reference

[1]: Section 2.8.2, Page 57, Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. The MIT Press.