Let X ∈ { 0 , 1 } d be a Boolean vector and Y , Z ∈ { 0 , 1 } are Boolean variables. Assume that there is a joint distribution D over Y , Z and we'd like to find a joint distribution D ′ over X , Y , Z such that:1. The marginal of D ′ on Y, Z equals D .2. X are independent of Z under D ′ , i.e., I ( X ; Z ) = 0.3. I ( X ; Y ) is maximized,where I ( ⋅ ; ⋅ ) denotes the mutual information. For now I don't even know what is a nontrivial upper bound of I ( X ; Y ) given that I ( X ; Z ) = 0? Furthermore, is it possible we can know the optimal distribution D ′ that achieves the upper bound?My conjecture is that the upper bound of I ( X ; Y ) should have something to do with the correlation (coupling?) between Y and Z, so ideally it should contain something related to that.

Question

Let   X  ∈  {  0  ,  1      }    d   be a Boolean vector and   Y  ,  Z  ∈  {  0  ,  1  } are Boolean variables. Assume that there is a joint distribution       D   over   Y  ,  Z and we&#039;d like to find a joint distribution             D        ′   over   X  ,  Y  ,  Z such that:1. The marginal of             D        ′   on   Y,  Z equals       D  .2.   X are independent of   Z under             D        ′  , i.e.,   I  (  X  ;  Z  )  =  0.3.   I  (  X  ;  Y  ) is maximized,where   I  (  ⋅  ;  ⋅  ) denotes the mutual information. For now I don&#039;t even know what is a nontrivial upper bound of   I  (  X  ;  Y  ) given that   I  (  X  ;  Z  )  =  0? Furthermore, is it possible we can know the optimal distribution             D        ′   that achieves the upper bound?My conjecture is that the upper bound of   I  (  X  ;  Y  ) should have something to do with the correlation (coupling?) between   Y and   Z, so ideally it should contain something related to that.

Makhi Lyons · Accepted Answer

We have the following series of inequalities:                    I        (        X        ;        Y        )                            ≤        I        (        X        ;        Y        ,        Z        )                                          =        I        (        X        ;        Z        )        +        I        (        X        ;        Y                  |                Z        )                                          =        I        (        X        ;        Y                  |                Z        )                                          =        I        (        X        ,        Z        ;        Y        )        −        I        (        Y        ;        Z        )                                          ≤        H        (        Y        )        −        I        (        Y        ;        Z        )                                          ≤        1         bit        −        I        (        Y        ;        Z        )        ,            where in the third line we&#039;ve used that   I  (  X  ;  Z  )  =  0.

Let X ∈ <mo fence="false" stretchy="false">{ 0 , 1 <mo fence="f

Answered question

Answer & Explanation

New Questions in High school geometry