Author Topic: Why statistics is so damn difficult!  (Read 9013 times)

0 Members and 1 Guest are viewing this topic.

Offline SirClickAlot

  • Thread Starter
  • Posts: 285
    • http://www.mrinterface.com
Why statistics is so damn difficult!
« on: Thu, 30 September 2010, 06:52:40 »
A candidate is about to win a prize in a quiz.
He is presented with three curtains; one of them hides the prize, the other two are empty.
The quizmaster asks the candidate to select a curtain and make his choice known.
Before opening the curtain of choice, the quizmaster opens up one of the other curtains, one of which he knows that it is empty. He then asks the candidate if he wants to change to the other curtain.
The dilemma is: does changing from curtain improve his chances of winning the prize or not?



I know some very smart people refusing to believe the solution to this dilemma ;-)
« Last Edit: Thu, 30 September 2010, 07:25:31 by SirClickAlot »
[Succes is when you get what you want
Happiness is when you want what you get]

Optimize your interfaces : http://www.mrinterface.com
Currently using( not necessarily my favorite : I iterate over my collection many times ):
Filco White Majestouch Mx Brown DS White on Black.
Looking for : New and interesting interfaces. Working on :www.deskthority.net

Offline a_fluffy_kitten

  • Posts: 59
Why statistics is so damn difficult!
« Reply #1 on: Thu, 30 September 2010, 07:00:10 »
This is a fun problem to analyze!  Note that this problem is probability and not statistics.

edit - and if I remember right, you solve this by calculating P(A|B) where A=(probability of picking right given two choices) and B=(probability of picking wrong given three choices), right?  Then if P(A|B)>0.5 you switch, if P(A|B)<0.5 you don't switch, and if P(A|B)=0.5 you do whatever the hell you want, up to and including punching the quizmaster in the face.
« Last Edit: Thu, 30 September 2010, 07:16:44 by a_fluffy_kitten »
2x Filco Blue Tenkeyless (and boy are they nice)

Offline Rajagra

  • Posts: 1930
Why statistics is so damn difficult!
« Reply #2 on: Thu, 30 September 2010, 07:15:30 »
By coincidence, I've been looking at this page. I know there's the correct way of working out the probabilities, but the incorrect way doesn't seem to be flawed (though it obviously is.)

I blame Bayes.

Offline Ekaros

  • Posts: 942
Why statistics is so damn difficult!
« Reply #3 on: Thu, 30 September 2010, 07:15:46 »
It's pretty strange thing, that it changes for other one. Still, what if in same case you are asked to choose again? Will both me 1/2 or other one still 2/3?

Probabilitys are easy, complex stuff is complex(integrals and fourier and so on...)
So I should add something useless here yes? Ok, ok...
Filco 105-key NKRO MX Browns Sw/Fi-layout|IBM Model M 1394545 Lexmark 102-key Finnish-layout 1994-03-22|Cherry G80-3000LQCDE-2 with MX CLEAR
[SIGPIC][/SIGPIC]
Dell AT102W(105-key SF) (Black ALPS)|Steelseries Steelkeys 6G(MX Black) ISO-FI-layout|Cherry G84-4400 G84-4700 Cherry MLs

Offline SirClickAlot

  • Thread Starter
  • Posts: 285
    • http://www.mrinterface.com
Why statistics is so damn difficult!
« Reply #4 on: Thu, 30 September 2010, 07:22:15 »
Quote from: a_fluffy_kitten;228343
This is a fun problem to analyze!  Note that this problem is probability and not statistics.


Probability or statistics? Now you got me going ;-)

Definition 01
Probability deals with predicting the likelihood of future events
statistics involves the analysis of the frequency of past events

Definition 02
Probability is primarily a theoretical branch of mathematics, which studies the consequences of mathematical definitions
Statistics is primarily an applied branch of mathematics, which tries to make sense of observations in the real world.

Definition 03
probability theory enables us to find the consequences of a given ideal world
Statistical theory enables us to measure the extent to which our world is ideal.

Hmmm, never thought about the difference..

So given these definitions this indeed seems to be a probability problem although I can solve it without using probability theory ;-)

Regards.
[Succes is when you get what you want
Happiness is when you want what you get]

Optimize your interfaces : http://www.mrinterface.com
Currently using( not necessarily my favorite : I iterate over my collection many times ):
Filco White Majestouch Mx Brown DS White on Black.
Looking for : New and interesting interfaces. Working on :www.deskthority.net

Offline a_fluffy_kitten

  • Posts: 59
Why statistics is so damn difficult!
« Reply #5 on: Thu, 30 September 2010, 07:37:48 »
I think a good real-world explanation of the difference is this:

If you're trying to figure out if something will happen, then it's probability.

If you're trying to figure out what has already happened, then it's statistics.
2x Filco Blue Tenkeyless (and boy are they nice)

Offline SirClickAlot

  • Thread Starter
  • Posts: 285
    • http://www.mrinterface.com
Why statistics is so damn difficult!
« Reply #6 on: Thu, 30 September 2010, 07:39:46 »
Quote from: a_fluffy_kitten;228352
I think a good real-world explanation of the difference is this:

If you're trying to figure out if something will happen, then it's probability.

If you're trying to figure out what has already happened, then it's statistics.


Yeps... That's an easy one I will remember. Thanx!
[Succes is when you get what you want
Happiness is when you want what you get]

Optimize your interfaces : http://www.mrinterface.com
Currently using( not necessarily my favorite : I iterate over my collection many times ):
Filco White Majestouch Mx Brown DS White on Black.
Looking for : New and interesting interfaces. Working on :www.deskthority.net

Offline Ekaros

  • Posts: 942
Why statistics is so damn difficult!
« Reply #7 on: Thu, 30 September 2010, 07:45:13 »
I having a course now about applied statistic, this far one of the simpler courses in maths... Still, it's kinda counter inuative for humans some times...
So I should add something useless here yes? Ok, ok...
Filco 105-key NKRO MX Browns Sw/Fi-layout|IBM Model M 1394545 Lexmark 102-key Finnish-layout 1994-03-22|Cherry G80-3000LQCDE-2 with MX CLEAR
[SIGPIC][/SIGPIC]
Dell AT102W(105-key SF) (Black ALPS)|Steelseries Steelkeys 6G(MX Black) ISO-FI-layout|Cherry G84-4400 G84-4700 Cherry MLs

Offline WhiteRice

  • Posts: 850
Why statistics is so damn difficult!
« Reply #8 on: Thu, 30 September 2010, 08:08:53 »
I always explained it like this. In the beginning you have a higher chance of picking the wrong door. So after one curtain is revealed, the curtain you picked in the beginning has a higher chance of being the wrong one. So switching is the correct choice.

Reminds me of the debates on whether or not a plane on a treadmill will take off.

it will.

Offline a_fluffy_kitten

  • Posts: 59
Why statistics is so damn difficult!
« Reply #9 on: Thu, 30 September 2010, 08:17:47 »
Quote from: WhiteRice;228363
Reminds me of the debates on whether or not a plane on a treadmill will take off.

it will.


Not if the tires explode first!  :D
2x Filco Blue Tenkeyless (and boy are they nice)

Offline erricrice

  • Posts: 326
Why statistics is so damn difficult!
« Reply #10 on: Thu, 30 September 2010, 08:18:04 »
Quote from: WhiteRice;228363
Reminds me of the debates on whether or not a plane on a treadmill will take off.

it will.

Yup, because the wheels on the plane have nothing to do with it's propulsion, the turbines do(so the wheels will just spin twice as fast and the plane will take off)


Quote from: a_fluffy_kitten;228368
Not if the tires explode first!  :D


Lol!
I\'m selling all my Shizz! Please buy it!

White ALPS: Northgate Omnikey 101-NCS(Real-Complicated)****Filco Zero FKBN87Z/EB(Fukka Simplifieds)****Siig MiniTouch(XM Simplifieds)
Black ALPS: Black Dell AT-101W(Real-Complicated)****ABS M1(Modded Black ALPS, Linear)
Buckling Spring: Model M 1391401(1988 & 1993)
Cherry Blues: DAS III Pro
Cherry Blacks: Cherry G80-11900
Cherry Browns: 3X Cherry G80-8113LRCUS-2
Cherry MY: G81-7000HPBUS-2****G81-3000LANUS-0****Modded to 20g
Rubber Dome: HHKB Lite 2 (White & Black)

Logitech G5[/FONT]
Erricrice\'s Song of the Day: Gorillaz - El Maņana
Yup, Blatantly stealing this from you Kishy, hope you don\'t mind, it\'s a great idea.

Offline Co-Op

  • Posts: 38
Why statistics is so damn difficult!
« Reply #11 on: Thu, 30 September 2010, 08:32:25 »
Isn't this the Monty Hall problem?

http://en.wikipedia.org/wiki/Monty_Hall_problem

Offline Mercen_505

  • Posts: 200
Why statistics is so damn difficult!
« Reply #12 on: Thu, 30 September 2010, 11:40:22 »
Quote

I know some very smart people refusing to believe the solution to this dilemma ;-)


Indeed, this is the "Monty Hall" problem. The trouble is that there are a a large number of variants, and they have slightly different results because of that. The aforementioned wikipedia article is particularly enlightening in this respect.

Offline quadibloc

  • Posts: 770
  • Location: Edmonton, Alberta, Canada
  • Layout Fanatic
    • John Savard's Home Page
Why statistics is so damn difficult!
« Reply #13 on: Thu, 30 September 2010, 12:36:24 »
Quote from: SirClickAlot;228341
A candidate is about to win a prize in a quiz.
He is presented with three curtains; one of them hides the prize, the other two are empty.
The quizmaster asks the candidate to select a curtain and make his choice known.
Before opening the curtain of choice, the quizmaster opens up one of the other curtains, one of which he knows that it is empty. He then asks the candidate if he wants to change to the other curtain.
The dilemma is: does changing from curtain improve his chances of winning the prize or not?
The way the problem is stated, there is no answer.

For example, suppose the quizmaster only does this - opening one of the other curtains - if the contestant has chosen the right curtain with the prize. Then, obviously, the contestant shouldn't change. Or if he only does this when the contestant had chosen the wrong curtain - then he should change.

If, on the other hand, the quizmaster always does this - opening one of the other curtains - every time without fail, then one can indeed make a probability calculation without considerations of intent being involved.

Since the quizmaster can, and does, always do this, regardless of which curtain the contestant chooses, this act gives no information about the curtain the contestant chose. The chance of that curtain having the prize was 1/3, and it stays 1/3.

But for the other two curtains - before, the chance for each of them was 1/3. Now, opening one of them up has given information about those two curtains - the 1/3 probability for the one that has been opened has now moved to the one that wasn't.

It was a 1/3 chance of the one you picked, and a 2/3 chance for the two others - and, after the curtain is opened, it still is. But now, if it's the two others, it has to be the one that wasn't opened.

Quote from: Rajagra;228344
By coincidence, I've been looking at this page. I know there's the correct way of working out the probabilities, but the incorrect way doesn't seem to be flawed (though it obviously is.)

I blame Bayes.
Blaming Bayes is right for this one.

With the two envelope paradox, the problem isn't "the probability is 50% that this envelope will have the check for the larger sum of money in it".

Once the envelope is opened, and you see the amount of money you chose, you have new information now. The check in that envelope is for 40,000 pounds.

So, while the a priori probability of the larger check being in the envelope you picked was 50%, with your new information, what you need to decide whether to change envelopes is...

the probability that they filled the envelopes with checks for 40,000 pounds and 20,000 pounds respectively, versus

the probability that they filled the envelopes with checks for 80,000 pounds and 40,000 pounds respectively.

The prize distribution is the distribution that counts now, not the envelope distribution. And that distribution is, given the conditions of the problem, a complete unknown. One data point - a 40,000 pound check - isn't enough to give meaningful insights into that distribution in a Bayesian sense.

Yes, one stands to gain twice as much as one could lose if the amounts of those checks were totally random. But if they were, then a check for 40,000,000,000,000,000,000 pounds would have been just as likely as a 40,000 pound check. So somewhere along the line, the probability of a check being for X pounds has to be less than that of a check for X/2 pounds.

A single check for 40,000 pounds does indicate that the midpoint of the distribution of check sizes is somewhere vaguely around there: that's all that Bayesian statistics will buy you.
« Last Edit: Thu, 30 September 2010, 12:47:01 by quadibloc »

Offline ricercar

  • * Elevated Elder
  • Posts: 1697
  • Location: Silicon Valley
  • mostly abides
Why statistics is so damn difficult!
« Reply #14 on: Thu, 30 September 2010, 15:18:39 »
I pick the blue one.
I trolled Geekhack and all I got was an eponymous SPOS.

Offline Arcias

  • Posts: 1
Why statistics is so damn difficult!
« Reply #15 on: Sat, 02 October 2010, 20:35:51 »
Ah yes, the Monty Hall Problem.

To add another twist, Monty Hall himself said that if the host has complete control, he could do one of two things: 1) Offer a switch. 2) Not offer a switch.

Since the host knows which door has the good prize, if the contestant picks the bad prize right off, he just won't offer the switch (Contestant Loses). If the contestant picks the good prize, he offers the switch. People who know that the statistically correct answer is to switch when offered a switch will switch and get screwed (Contestant Loses). The only way to win is to then pick the correct one right off and stick with it, reducing your chances to 1/3.

Offline Rajagra

  • Posts: 1930
Why statistics is so damn difficult!
« Reply #16 on: Sat, 02 October 2010, 23:28:14 »
Aha. Sounds a lot like the Deal or No Deal TV show.

Offline chimera15

  • Posts: 1441
Why statistics is so damn difficult!
« Reply #17 on: Sun, 03 October 2010, 00:54:06 »
Quote from: Rajagra;229217
Aha. Sounds a lot like the Deal or No Deal TV show.

That's what it's based on. Let's make a deal.  I always get the names confused myself.  Deal or no deal is the one with Howie Mandell.  Monty Hall was the original host in the 60's-80's or so.  It was recently brought back with Wayne Brady as host.

They need to start filming in hd though. lol

Arcias is right, it's a dirty swindler's trick.

Statistically though if the host was on the up and up and didn't know either and offered the switch it I think it would still be a 50/50 shot, if you stayed on the one you picked or not.   Statistics have no memory.  It's like if you rolled a d6, then had to roll a d3.  The d6 doesn't affect the d3, even if you roll a 1 in each instance.  Or flipping a coin.  The original 1/3 choice is negated and was never a choice in the first place since the host was always going to take one away.  It was always a 50/50 decision.




One of the things that makes this show interesting is that Wayne Brady has such a good guy image, and no doubt wants to protect it.  So if he offered you to switch, it might not be a swindle, and he might actually be trying to help you. lol
« Last Edit: Sun, 03 October 2010, 01:26:40 by chimera15 »
Alps boards:
white real complicated: 1x modified siiig minitouch kb1903,  hhkb light2 english steampunk hack, wireless siig minitouch hack
white with rubber damper(cream)+clicky springs: 2x modified siig minitouch kb1903 1x modified siig minitouch kb1948
white fake simplified:   1x white smk-85, 1x Steampunk compact board hack
white real simplified: 1x unitek k-258
low profile: 1x mint m1242 in box
black: ultra mini wrist keyboard hack
blue: Japanese hhk2 lite hack, 1x siig minitouch pcb/doubleshot dc-2014 caps. kb1903, 1x modified kb1948 Siig minitouch
rainbow test boards:  mck-84sx


Offline unicomp

  • Posts: 119
Why statistics is so damn difficult!
« Reply #18 on: Wed, 06 October 2010, 13:25:47 »
Paradoxically probability is a subject in which a paradox is not necessarily a paradox.

Offline quadibloc

  • Posts: 770
  • Location: Edmonton, Alberta, Canada
  • Layout Fanatic
    • John Savard's Home Page
Why statistics is so damn difficult!
« Reply #19 on: Wed, 06 October 2010, 19:52:45 »
Quote from: chimera15;229220
Statistically though if the host was on the up and up and didn't know either and offered the switch it I think it would still be a 50/50 shot, if you stayed on the one you picked or not.   Statistics have no memory.  It's like if you rolled a d6, then had to roll a d3.  The d6 doesn't affect the d3, even if you roll a 1 in each instance.  Or flipping a coin.  The original 1/3 choice is negated and was never a choice in the first place since the host was always going to take one away.  It was always a 50/50 decision.
Well, the idea is that the host does know which door has the big prize, but always offers the switch.

In that case, the probability was 1/3 at the start. You had three doors to pick from. Even though you knew a door was going to be opened, you didn't know which one. So one door had the prize - and there were three doors to pick from - so the chance has to be 1/3.

Since the host always offers the switch, that 1/3 probability is not changed by the fact that he offered a switch. When you picked the door, there were three doors to choose from, and one of them had the good prize.

Now that the host opened one door, though, you know something that you didn't know before about the other two doors. Which is why the chance is now 2/3 that the good prize is behind the unopened door you didn't pick - your original decision to pick one door was made without that information.

In the beginning, there were three cases: let's suppose you picked door #3.

XXP 1/3
XPX 1/3
PXX 1/3

where P stands for the big prize.

With the host opening one door, we now have four possible cases, with the following probabilities:

OXP 1/6
XOP 1/6
OPX 1/3
POX 1/3

where P stands for the door with the prize, X stands for an unopened door without the prize, and O stands for the door without the prize that was opened.

If you picked the right door, the host could have picked either of the two doors you didn't pick, so the probability of that case is split in half.

If you didn't, though, there is only one door for the host to pick to open.
« Last Edit: Wed, 06 October 2010, 19:55:42 by quadibloc »