An Interactive webR Approach to Teaching Statistical Inference to Behavioral Science Students

Null Hypothesis Significance Testing via Permutation Tests

Kimberly Henry

Why Permutation Tests?

• Problem: Students struggle with NHST’s abstract logic (p-values, sampling distributions)
• Solution: Permutation tests make inference visual and intuitive

Most students plug and chug formulas without really grasping p‑values or sampling distributions. With permutation tests, they literally reshuffle the data, see the null world emerge, and ask, ‘How unusual is our result?’ It turns abstract inference into a hands‑on experiment.

Exploration via WebR

I developed an interactive WebR tutorial that guides students step-by-step through the logic and implementation of permutation tests for null hypothesis significance testing.

The Study

This activity uses data from Hofman et al., shared on the Open Science Framework. The study investigates how different ways of visualizing uncertainty influence decision‑making.

The Game

Players competed with Blorg in an ice‑boulder game and could rent a special boulder for a potential edge.

Special Boulder

But that boulder might help — or it might not.

The Experimental Conditions

\[ H_{0}: \mu_{\mathrm{CI}} - \mu_{\mathrm{CI\ rescaled}} = 0 \]

To help players decide, they were shown one of four uncertainty visualizations. Each shows the average sliding distance for the standard and special boulders, along with uncertainty around those averages. Today, we’re focusing on the Confidence Interval and Rescaled Confidence Interval conditions. Both show the same data, but the rescaled CI uses a wider y‑axis, which makes the difference between the boulders look smaller. Our null hypothesis is that there’s no difference in willingness to pay between these two visualization conditions.

1: Select & Observe

Pick two conditions & calculate observed difference

To begin the activity, we first pick two groups and compute the observed mean difference in willingness to pay for the special boulder. This anchors students in the real effect before any shuffling happens.

2: Shuffle Labels to Test the Null

Next, we randomly reassign group labels 5000 times. The first panel shows the real data; the rest show shuffled labels — letting students watch the null world form step by step.

3: Build the Null Distribution

We shuffle, calculate the mean difference 5,000 times, and those differences form our hands‑on sampling distribution under the null.

4: Calculate Empirical p-value

What proportion of random shuffles produced effects as large as the real data?

p = 0.048 (4.8% of 5000 permutations)

We simply count the proportion of shuffled differences at least as extreme as what we observed. That fraction is the p‑value.

Our permutation test shows that 4.8% of 5000 random shuffles produced group differences as larger, or larger than, our observed difference (-11.75).

5: Compute the Standard Error

Standard Error (SE) (i.e., SD of null distribution): 5.785 

The standard deviation of the null differences is our standard error — it shows the typical variation we’d expect under the null hypothesis.

6: Construct the Test Statistic

Group Difference/SE = Standardized Test Statistic

We standardize the observed difference using the SE, then plot it on the null curve so students can see how far out it lands.

7: Compare to a Parametric Approach

my_groups |> 
  infer::t_test(wtp_final ~ graph_type, 
         var.equal = TRUE,
         order = c("CI rescaled", "CI"),
         alternative = "two-sided") |> 
  select(estimate, statistic, p_value )

Finally, we run a classic t‑test and compare. This side‑by‑side helps students appreciate when permutation and parametric approaches agree — or why they might differ.

Instructor Benefits

• Zero setup: Runs in-browser (webR)
• Reusable: GitHub repo for easy adoption
• Flexible: Adaptable to other datasets

🔗 Links: https://KimberlyHenry.github.io/UseR2025/

WebR runs in the browser — no installs are necessary. Everything for the activity is on GitHub, ready to use or adapt.