## Relation between the two possible KL divergences of two…

Clash Royale CLAN TAG#URR8PPP

Given that I know $$Dleft(Pparallel Qright)<alpha,$$ can I say anything about $Dleft(Qparallel Pright)$ in terms of an upper bound on it?

Also, given this upper bound on $Dleft(Pparallel Qright)$, can I deduce some upper bound on $left|Pleft(Aright)-Qleft(Aright)right|$ for some arbitrary event $A$?

$endgroup$

add a comment |

Given that I know $$Dleft(Pparallel Qright)<alpha,$$ can I say anything about $Dleft(Qparallel Pright)$ in terms of an upper bound on it?

Also, given this upper bound on $Dleft(Pparallel Qright)$, can I deduce some upper bound on $left|Pleft(Aright)-Qleft(Aright)right|$ for some arbitrary event $A$?

$endgroup$

add a comment |

Given that I know $$Dleft(Pparallel Qright)<alpha,$$ can I say anything about $Dleft(Qparallel Pright)$ in terms of an upper bound on it?

Also, given this upper bound on $Dleft(Pparallel Qright)$, can I deduce some upper bound on $left|Pleft(Aright)-Qleft(Aright)right|$ for some arbitrary event $A$?

$endgroup$

add a comment |

add a comment |

##
1 Answer

1

active

oldest

votes

The answer to your first question is no: $D(Q||P)$ may be however large while $D(P||Q)$ is however small. E.g., let $P$ have masses $s$ and $1-s$ at points $0$ and $1$, respectively, and let $Q$ have masses $t$ and $1-t$ at points $0$ and $1$, respectively, where $0<s,t<1$. Then

begin{equation}

D(P||Q)=slnfrac st+(1-s)lnfrac{1-s}{1-t},

end{equation}

begin{equation}

D(Q||P)=tlnfrac ts+(1-t)lnfrac{1-t}{1-s}.

end{equation}

Let now $tdownarrow0$ and $s=te^{-1/t^2}$. Then $D(P||Q)to0$ whereas $D(Q||P)toinfty$.

The answer to your second question is yes: Pinsker’s inequality states that

begin{equation}

sup_A|P(A)-Q(A)|lesqrt{tfrac12,D(P||Q)}.

end{equation}

$endgroup$

add a comment |

## Your Answer

StackExchange.ifUsing(“editor”, function () {

return StackExchange.using(“mathjaxEditing”, function () {

StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {

StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [[“$”, “$”], [“\\(“,”\\)”]]);

});

});

}, “mathjax-editing”);

StackExchange.ready(function() {

var channelOptions = {

tags: “”.split(” “),

id: “504”

};

initTagRenderer(“”.split(” “), “”.split(” “), channelOptions);

StackExchange.using(“externalEditor”, function() {

// Have to fire editor after snippets, if snippets enabled

if (StackExchange.settings.snippets.snippetsEnabled) {

StackExchange.using(“snippets”, function() {

createEditor();

});

}

else {

createEditor();

}

});

function createEditor() {

StackExchange.prepareEditor({

heartbeatType: ‘answer’,

autoActivateHeartbeat: false,

convertImagesToLinks: true,

noModals: true,

showLowRepImageUploadWarning: true,

reputationToPostImages: 10,

bindNavPrevention: true,

postfix: “”,

imageUploader: {

brandingHtml: “Powered by u003ca class=”icon-imgur-white” href=”https://imgur.com/”u003eu003c/au003e”,

contentPolicyHtml: “User contributions licensed under u003ca href=”https://creativecommons.org/licenses/by-sa/3.0/”u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href=”https://stackoverflow.com/legal/content-policy”u003e(content policy)u003c/au003e”,

allowUrls: true

},

noCode: true, onDemand: true,

discardSelector: “.discard-answer”

,immediatelyShowMarkdownHelp:true

});

}

});

### Sign up or log in

StackExchange.ready(function () {

StackExchange.helpers.onClickDraftSave(‘#login-link’);

});

### Post as a guest

Required, but never shown

StackExchange.ready(

function () {

StackExchange.openid.initPostLogin(‘.new-post-login’, ‘https%3a%2f%2fmathoverflow.net%2fquestions%2f320545%2frelation-between-the-two-possible-kl-divergences-of-two-distributions%23new-answer’, ‘question_page’);

}

);

### Post as a guest

Required, but never shown

##
1 Answer

1

active

oldest

votes

##
1 Answer

1

active

oldest

votes

active

oldest

votes

active

oldest

votes

The answer to your first question is no: $D(Q||P)$ may be however large while $D(P||Q)$ is however small. E.g., let $P$ have masses $s$ and $1-s$ at points $0$ and $1$, respectively, and let $Q$ have masses $t$ and $1-t$ at points $0$ and $1$, respectively, where $0<s,t<1$. Then

begin{equation}

D(P||Q)=slnfrac st+(1-s)lnfrac{1-s}{1-t},

end{equation}

begin{equation}

D(Q||P)=tlnfrac ts+(1-t)lnfrac{1-t}{1-s}.

end{equation}

Let now $tdownarrow0$ and $s=te^{-1/t^2}$. Then $D(P||Q)to0$ whereas $D(Q||P)toinfty$.

The answer to your second question is yes: Pinsker’s inequality states that

begin{equation}

sup_A|P(A)-Q(A)|lesqrt{tfrac12,D(P||Q)}.

end{equation}

$endgroup$

add a comment |

The answer to your first question is no: $D(Q||P)$ may be however large while $D(P||Q)$ is however small. E.g., let $P$ have masses $s$ and $1-s$ at points $0$ and $1$, respectively, and let $Q$ have masses $t$ and $1-t$ at points $0$ and $1$, respectively, where $0<s,t<1$. Then

begin{equation}

D(P||Q)=slnfrac st+(1-s)lnfrac{1-s}{1-t},

end{equation}

begin{equation}

D(Q||P)=tlnfrac ts+(1-t)lnfrac{1-t}{1-s}.

end{equation}

Let now $tdownarrow0$ and $s=te^{-1/t^2}$. Then $D(P||Q)to0$ whereas $D(Q||P)toinfty$.

The answer to your second question is yes: Pinsker’s inequality states that

begin{equation}

sup_A|P(A)-Q(A)|lesqrt{tfrac12,D(P||Q)}.

end{equation}

$endgroup$

add a comment |

begin{equation}

D(P||Q)=slnfrac st+(1-s)lnfrac{1-s}{1-t},

end{equation}

begin{equation}

D(Q||P)=tlnfrac ts+(1-t)lnfrac{1-t}{1-s}.

end{equation}

Let now $tdownarrow0$ and $s=te^{-1/t^2}$. Then $D(P||Q)to0$ whereas $D(Q||P)toinfty$.

begin{equation}

sup_A|P(A)-Q(A)|lesqrt{tfrac12,D(P||Q)}.

end{equation}

$endgroup$

begin{equation}

D(P||Q)=slnfrac st+(1-s)lnfrac{1-s}{1-t},

end{equation}

begin{equation}

D(Q||P)=tlnfrac ts+(1-t)lnfrac{1-t}{1-s}.

end{equation}

Let now $tdownarrow0$ and $s=te^{-1/t^2}$. Then $D(P||Q)to0$ whereas $D(Q||P)toinfty$.

begin{equation}

sup_A|P(A)-Q(A)|lesqrt{tfrac12,D(P||Q)}.

end{equation}

add a comment |

add a comment |

Thanks for contributing an answer to MathOverflow!

- Please be sure to
*answer the question*. Provide details and share your research!

But *avoid* …

- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.

Use MathJax to format equations. MathJax reference.

To learn more, see our tips on writing great answers.

### Sign up or log in

StackExchange.ready(function () {

StackExchange.helpers.onClickDraftSave(‘#login-link’);

});

### Post as a guest

Required, but never shown

StackExchange.ready(

function () {

StackExchange.openid.initPostLogin(‘.new-post-login’, ‘https%3a%2f%2fmathoverflow.net%2fquestions%2f320545%2frelation-between-the-two-possible-kl-divergences-of-two-distributions%23new-answer’, ‘question_page’);

}

);

### Post as a guest

Required, but never shown

### Sign up or log in

StackExchange.ready(function () {

StackExchange.helpers.onClickDraftSave(‘#login-link’);

});

### Post as a guest

Required, but never shown

### Sign up or log in

StackExchange.ready(function () {

StackExchange.helpers.onClickDraftSave(‘#login-link’);

});

### Post as a guest

Required, but never shown

### Sign up or log in

StackExchange.ready(function () {

StackExchange.helpers.onClickDraftSave(‘#login-link’);

});

### Post as a guest

Required, but never shown

Required, but never shown

Required, but never shown

Required, but never shown

Required, but never shown

Required, but never shown

Required, but never shown

Required, but never shown

Required, but never shown