Gdtyjr

Russian cases: A few examples, I'm really confused

Know when to turn notes upside-down(eighth notes, sixteen notes, etc.)

Employee lack of ownership

Brexit - No Deal Rejection

Humanity loses the vast majority of its technology, information, and population in the year 2122. How long does it take to rebuild itself?

Could the Saturn V actually have launched astronauts around Venus?

RegionDifference for Cylinder and Cuboid

The use of "touch" and "touch on" in context

What options are left, if Britain cannot decide?

What are the possible solutions of the given equation?

Make a transparent 448*448 image

How to make healing in an exploration game interesting

2D counterpart of std::array in C++17

Why are there 40 737 Max planes in flight when they have been grounded as not airworthy?

Does this AnyDice function accurately calculate the number of ogres you make unconcious with three 4th-level castings of Sleep?

Is it normal that my co-workers at a fitness company criticize my food choices?

How to deal with taxi scam when on vacation?

Rules about breaking the rules. How do I do it well?

Official degrees of earth’s rotation per day

Distribution of Maximum Likelihood Estimator

What is IP squat space

Good allowance savings plan?

Science-fiction short story where space navy wanted hospital ships and settlers had guns mounted everywhere

Ban on all campaign finance?

Distribution of Maximum Likelihood Estimator

Maximum likelihood of function of the mean on a restricted parameter spaceMaximum Likelihood Estimator (MLE)Maximum Likelihood Estimator of the exponential function parameter based on Order StatisticsFind maximum likelihood estimateMaximum Likelihood Estimation in case of some specific uniform distributionsMaximum likelihood estimate for a univariate gaussianFind the Maximum Likelihood Estimator given two pdfsMaximum Likelihood Estimate for a likelihood defined by partsVariance of distribution for maximum likelihood estimatorHow is $P(D;theta) = P(D|theta)$?

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_max$ such that $L(theta)$ is maximized and $theta_max$ is our estimate (once a sample has actually been selected)

Since $theta_max$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$fracf'(x_1;theta)f(x_1;theta)+fracf'(x_2;theta)f(x_2;theta)...+fracf'(x_n;theta)f(x_n;theta)$$

$theta_max$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_max$ such that $L(theta)$ is maximized and $theta_max$ is our estimate (once a sample has actually been selected)

Since $theta_max$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$fracf'(x_1;theta)f(x_1;theta)+fracf'(x_2;theta)f(x_2;theta)...+fracf'(x_n;theta)f(x_n;theta)$$

$theta_max$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_max$ such that $L(theta)$ is maximized and $theta_max$ is our estimate (once a sample has actually been selected)

Since $theta_max$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$fracf'(x_1;theta)f(x_1;theta)+fracf'(x_2;theta)f(x_2;theta)...+fracf'(x_n;theta)f(x_n;theta)$$

$theta_max$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

1 Answer
1

active

oldest

votes

3

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3917

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $fracf'(x)f(x)$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  3 hours ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "65"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f397619%2fdistribution-of-maximum-likelihood-estimator%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3917

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $fracf'(x)f(x)$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  3 hours ago
  

  
  
  
  

add a comment | 

3

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3917

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $fracf'(x)f(x)$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  3 hours ago
  

  
  
  
  

add a comment | 

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3917

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3917

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 3 hours ago

dlnBdlnB

3917

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 3 hours ago

dlnBdlnB

3917