Probabilistic and statistical thinking

(a preview)

Andrew Pua

2025-10-09

Previously…

  • We calculated summaries and linear regressions using whatever data we have on hand.

    • We explored how these aspects may be related to each other algebraically.
    • We also discussed how to communicate the results.
  • These descriptions could be insightful for what they are but what exactly do we learn from these descriptions?

  • We have to think hard about these summaries and linear regressions before doing any computations.

Empirical vs theoretical quantities

  • What you have seen before are empirical quantities.

    • These empirical quantities are subject to variation.
  • To truly understand what these empirical quantities are actually all about, we cannot avoid abstraction.

    • What I mean by abstraction is that we are engaging in a thought experiment where we think of empirical quantities as having some “order” which could be studied mathematically.
  • We create theoretical (idealized) counterparts of these empirical quantities.

Why?

  • When there is some long-run order, we can apply mathematics with a better idea of what could happen.
  • When there is some long-run order, we can determine what price to pay to accept a little variation.
  • When there is some long-run order, we can use the empirical counterparts of that long-run order and conduct inference on that long-run order.

A typical scientific investigation

A typical scientific investigation

Our focus is Harley

Setup for the experiment

Summary of the data from the experiment

  1. What does the original dataset look like?
  2. Focus on Harley’s result for the cue “Bow look dog”. How do we judge the finding?

Judging findings requires models

  1. Think about what could be explanations for the finding.
  2. Think about the circumstances which led to the finding.
  3. What do the authors mean when they say “the ability of domestic dogs to use human body cues”?

Dogs and coins

  1. Pretend that “the dog chooses the correct cup” is like “getting Heads in a coin toss”.

  2. Pretend we are in the long run.

    • What does it mean when “the dog cannot understand human cues”?
    • What does it mean when “the dog can understand human cues”?
    • As a result, there are two claims to adjudicate.

Dogs and coins

  1. Once again, pretend we are in the long run. How many out of the 1 million times would you expect the dog to choose the correct cup if “the dog cannot understand human cues”?
  2. So, if Harley cannot understand human cues, how many out of the 10 times would you expect him to choose the correct cup?
  3. Think of what we actually observed. How many times did Harley choose the correct cup?

Get a sense of sampling variation

# Number of times to repeat the process of tossing a fair coin 10 times 
nsim <- 2
# Repeat for nsim times "tossing of a fair coin 10 times"
a <- replicate(nsim, rbinom(10, 1, 0.5))
a
      [,1] [,2]
 [1,]    0    0
 [2,]    1    0
 [3,]    1    0
 [4,]    1    0
 [5,]    0    0
 [6,]    1    1
 [7,]    1    0
 [8,]    1    0
 [9,]    1    1
[10,]    0    0
# Calculate the relative frequency or sample proportion for every repetition
props <- colMeans(a)
props
[1] 0.7 0.2

Which claim is supported?

# Number of times to repeat the process of tossing a fair coin 10 times 
nsim <- 10^4
# Repeat for nsim times "tossing of a fair coin 10 times"
a <- replicate(nsim, rbinom(10, 1, 0.5))
# Calculate the relative frequency for every repetition
props <- colMeans(a)
length(props)
[1] 10000
head(props)
[1] 0.3 0.5 0.2 0.4 0.4 0.5
# Summarize in tabular format
table(props)
props
   0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9    1 
   8  101  476 1138 2018 2441 2017 1217  453  120   11 

Which claim is supported?

# Visualize using a histogram
hist(props, freq = FALSE, cex.lab=1.5, cex.axis=1.5, cex.main = 1.5)

How can you decide?

  1. A very popular approach (with dissenting opinions) is to report the following statistic:

If you take the position that Harley cannot understand human gestures, how often can you hypothetically observe something more extreme than what you actually observed in real data?

How can you decide?

  1. In our case, it is given by
mean(props>=0.9)
[1] 0.0131
round(mean(props>=0.9),2)
[1] 0.01
  1. What decision will you make?

Congratulations!

  1. You have been exposed to an argument that is supported by foundations and principles that have some mathematical basis.
  2. This style of argument is called hypothesis testing.
  3. You will learn more as we progress through the course.
  4. There are other statistical ways of thinking depending on the goal. For now, the goal was to adjudicate between two claims about some fixed constant of interest.

Taking stock

  1. What was the simple model of chance that was used? What were the assumptions?
  2. Why were we using the word “pretend”?
  3. What are the steps of the argument? Why could it make sense?
  4. Where was the math?
  5. How do we know that the computer is doing what we think it should do?

When should you use the argument developed?

  1. There is an argument involved in determining whether the data supports “Harley cannot understand human gestures” and “Harley can understand human gestures”.
  2. The argument you have seen is typically used for discovery: meaning figuring out if “something is out of the ordinary”.
  3. It is not used to prove any specific claim. Why?
  4. But it is not surprising that people think that they have proven something. So, there is huge potential for abuse.

The template

  1. A model
  2. A statement about the model which represents a status quo
  3. A statement about the model which represents something out of the ordinary
  4. A statistic which can be computed using data
  5. A way to generate simulated datasets or at least imagine hypothetical datasets under the status quo
  6. A way to decide which statement the data would favor

Your task

Lay out these ingredients for the examples so far:

  1. Is the coin you are using fair?
  2. Does Harley understand human gestures?

River contamination

  1. Several fish deaths were reported just downstream of a large swale through which runoff from a commercial hog facility had escaped.

  2. A pollution control agency decided to closely monitor dissolved oxygen downstream of the swale once every 11 days. The plan is to do the monitoring 10 times.

  3. The agency wanted to see how often the dissolved oxygen was less than the 5.0 mg/L standard. A measurement with a lower level was considered “noncompliant”.

  1. Water quality guidelines suggest that remedial action would be taken if the river was noncompliant more than 10% of the time.

  2. Here is the dataset. An S indicates that the river was not in compliance that day.

Mar 8 Mar 19 Mar 30 Apr 10 Apr 21
S F S F S
May 2 May 13 May 24 Jun 4 Jun 15
F F F F S
  1. What can we conclude based on the evidence?

Organ transplants

  1. People providing an organ for donation sometimes seek the help of a special medical consultant. These consultants assist the patient in all aspects of the surgery, with the goal of reducing the possibility of complications during the medical procedure and recovery.
  2. Patients might choose a consultant based in part on the historical complication rate of the consultant’s clients.
  1. One consultant tried to attract patients by noting the historical complication rate for liver donor surgeries in the US is about 10%, but her clients have had only 3 complications in the 62 liver donor surgeries she has facilitated.

  2. She claims this is strong evidence that her work meaningfully contributes to reducing complications.

  3. Do you think there is evidence to support this claim?

Heart transplant failures

  1. Heart transplants at a London hospital were suspended because of concern that more patients were dying than usual.

  2. Newspapers reported that the 80% mortality rate in the last ten cases was of particular concern because it was over 5 times the national average.

  3. Although there was not an officially reported mortality rate, researchers have determined that 15% was a reasonable benchmark for comparison.

  4. What do you think of the evidence?

How do we extend what you saw here

  1. To what we did in class

    1. (V26) Checking if the net weight of Chips Ahoy “in the long run” is equal to the advertised weight of 38.4 grams
    2. (V27) Checking if the net weight of M&Ms “in the long run” is equal to the advertised weight of 14.5 grams
  2. Harder in these cases because we have to think about how to generate fake data under the condition that the net weight “in the long run” is equal to the advertised weight

Moving on: dealing with uncertainty

  1. This course is not just about data.
  2. Part of what you learn in the course applies to what some call “scenario analysis”.
  3. Every decision we make is a decision made under uncertainty.
  4. Dealing with uncertainty again means imagining potential scenarios.

Should a firm introduce a new product?

A company wants to determine if they should enter a market. But there are other potential competitors. We have the following internal projections:

  1. Fixed costs of entering the market: 26 million
  2. Net present value of revenue minus variable costs in the whole market: 100 million
  1. Profits depend on how many competitors enter the market. The 100 million will be shared equally among those in the market.

  2. Let \(X\) be the number of other entrants in the market. Let \(Y\) be the profit of the company. How would you express uncertainty in this situation?

  1. Let \(X\) be the number of other entrants in the market. This number is uncertain:

    Outcome of X How likely?
    1 0.1
    2 0.25
    3 0.3
    4 0.25
    5 0.1
  2. How about \(Y\)?

entrants <- 1:5
profits <- 100/(1+entrants)-26
a <- sample(entrants, 10^4, prob = c(0.1, 0.25, 0.3, 0.25, 0.1), replace = TRUE)
b <- sample(profits, 10^4, prob = c(0.1, 0.25, 0.3, 0.25, 0.1), replace = TRUE)
table(a)
a
   1    2    3    4    5 
1019 2468 2960 2520 1033 
table(b)
b
-9.33333333333333                -6                -1  7.33333333333334 
              953              2509              2964              2521 
               24 
             1053 
c(mean(a), sd(a), mean(b), sd(b))
[1] 3.008000 1.148768 1.684667 9.434589

Risk profile

probs <- seq(0,1,0.005)
plot(probs, quantile(b, probs))

Multiple uncertainties

  1. Suppose the fixed costs of entry and the total market value are uncertain in the following sense:

    Outcome of D How likely? Outcome of V How likely?
    20 0.2 70 0.3
    26 0.5 100 0.4
    30 0.2 120 0.2
    34 0.1 150 0.1
entrants <- sample(1:5, 10^4, 
                   prob = c(0.1, 0.25, 0.3, 0.25, 0.1), replace = TRUE)
values <- sample(c(70, 100, 120, 150), 10^4, 
                 prob = c(0.3, 0.4, 0.2, 0.1), replace = TRUE)
costs <- sample(c(20, 26, 30, 34), 10^4, 
                prob = c(0.2, 0.5, 0.2, 0.1), replace = TRUE)
# Create a profit function
profit <- function(v, c, e)
{
  return(v/(1+e)-c)
}
# Apply profit function to V, C, E
res <- mapply(profit, values, costs, entrants)
c(mean(res), sd(res))
[1]  1.11750 12.52011

Risk profile under multiple uncertanties

probs <- seq(0,1,0.005)
plot(probs, quantile(res, probs))

Taking stock

  1. What you have seen is a very simplified form of scenario analysis which can be useful for decision making.
  2. You must wonder at this stage where did all these outcome tables come from – and you should.
  3. You have seen how the “long run” is involved in decision-making.
  4. The multiple uncertainties case is extremely special – have you noticed something?