Consider A Hypothesis Test In Which The Significance Level Is

You're staring at a p-value of 0.That's why publishable. 047. But wait — what if you'd set your threshold at 0.In real terms, significant. Your heart does that little jump. Nothing about your data changed. Suddenly it's not significant anymore. In real terms, 01 instead? Only the line you drew in the sand.

That line has a name. It's called the significance level. And it might be the most misunderstood concept in all of statistics.

What Is Significance Level

Significance level — usually denoted by the Greek letter alpha (α) — is the probability of rejecting the null hypothesis when it's actually true. In plain English: it's your tolerance for false alarms.

You're testing whether a new drug works. The null hypothesis says it doesn't. Which means if that p-value falls below your chosen alpha, you reject the null. You run the trial, crunch the numbers, get a p-value. You declare the drug effective.

But here's the thing: even if the drug is completely useless, there's still a chance your data will look impressive just by luck. Alpha is the maximum probability of that happening that you're willing to accept.

The standard default in most fields is 0.In practice, 01. And 05. Day to day, one in twenty. Some use 0.Some fields use 0.Still, 10. That means a 5% chance of a false positive. The choice isn't arbitrary — or at least, it shouldn't be.

The Null Hypothesis Reminder

Before we go further, a quick refresher. No relationship. No difference. Also, no effect. And the null hypothesis (H₀) is usually the "nothing interesting is happening" position. The alternative hypothesis (H₁ or Hₐ) is what you're actually hoping to prove — that there is an effect, difference, or relationship.

Significance level only applies to the null. It's the probability of saying "there's something here" when there actually isn't. Statisticians call this a Type I error. More on that in a moment.

Why It Matters

People treat 0.05 like a law of physics. So naturally, it's not. Also, it's a convention. Think about it: a historical accident, really — Ronald Fisher suggested it in 1925 as a convenient threshold, and generations of researchers just... kept using it.

But the significance level you choose shapes everything that follows.

Choose 0.Worth adding: 01 instead of 0. Think about it: 05, and you make it harder to claim a discovery. Fewer false positives. But also fewer true discoveries — you'll miss real effects that don't quite clear the higher bar. This is the tradeoff. Always the tradeoff.

In medical trials, a false positive means approving a drug that doesn't work. In real terms, patients take it. Side effects happen. Money gets wasted. That said, lives might be lost. So the field often demands 0.Because of that, 01 or even 0. 001.

In exploratory psychology research? Day to day, a false positive means a follow-up study fails to replicate. Embarrassing, sure. But nobody dies. So 0.05 is often fine — or even 0.10 in early-stage work.

The significance level should match the stakes. Plus, that's the principle. In practice, most people just use whatever their field uses and don't think about it.

The Cost of Getting It Wrong

Two ways exist — each with its own place.

Type I error: rejecting a true null. False alarm. Probability = alpha.

Type II error: failing to reject a false null. Missed discovery. Probability = beta.

Statistical power is 1 - beta — the probability of correctly* rejecting a false null. Also, you want high power. But power and alpha move in opposite directions. Lower your alpha, and power drops (unless you increase sample size).

We're talking about why sample size calculations exist. You pick your alpha, you pick your desired power (usually 0.80), you estimate your effect size, and the math tells you how many subjects you need.

Skip this step, and you're guessing. A study with 20 participants per group and alpha = 0.05 might have power of 0.30. That means a 70% chance of missing a real effect. You'd be better off flipping a coin.

How It Works in Practice

Let's walk through a concrete example. You're testing whether a new website design increases sign-up rates.

Current rate: 12%. You want to know if the new design pushes it higher.

Step 1: State Your Hypotheses

H₀: The new design has no effect (sign-up rate = 12%) H₁: The new design increases sign-ups (sign-up rate > 12%)

Basically a one-tailed test. You only care about increases. If the new design decreases* sign-ups, you'll stick with the old one anyway.

Step 2: Choose Your Significance Level

What's at stake? Not catastrophic. Consider this: reversible. If you roll out a worse design, you lose some sign-ups temporarily. But you also don't want to chase phantom improvements.

You choose α = 0.05. Standard. Reasonable.

Step 3: Collect Data and Calculate the Test Statistic

You run an A/B test. Which means 1,000 visitors see the old design. 1,000 see the new.

Old: 122 sign-ups (12.2%) New: 148 sign-ups (14.8%)

You run a two-proportion z-test. The test statistic comes out to z = 2.47.

Step 4: Find the P-Value

For a one-tailed test with z = 2.Now, 47, the p-value is approximately 0. 0068.

Step 5: Compare and Decide

p-value (0.0068) < α (0.05)

Reject the null. The result is statistically significant.

But — and this is crucial — statistical significance doesn't mean practical significance. Because of that, a 2. 6 percentage point increase might matter for your business. Or it might not, if the new design costs $50,000 to implement and only brings in $3,000 more per month.

The significance level only tells you about the evidence against the null*. It doesn't tell you about effect size, practical importance, or whether the result will replicate.

Want to learn more? We recommend how many pounds is 83 kilograms and how effective is it to shadow more senior team members for further reading.

One-Tailed vs Two-Tailed

That example used a one-tailed test. You only cared about increases.

A two-tailed test splits your alpha between both tails. The critical value is more extreme. And 05, each tail gets 0. In real terms, with α = 0. 025. It's harder to reach significance.

Use two-tailed when you care about effects in either* direction. Use one-tailed only when you have a strong, pre-registered reason to only care about one direction — and even then, many reviewers will push back.

Don't switch to one-tailed after seeing the data. That's p-hacking.

Common Mistakes

Treating 0.05 as a Bright Line

A p-value of 0.In real terms, 051 are practically identical. The evidence against the null is nearly the same. But one is "significant" and the other isn't, if you're using 0.So naturally, 049 and a p-value of 0. 05 as a hard threshold.

This is absurd. On top of that, the American Statistical Association has explicitly warned against this. P-values are continuous measures of evidence, not binary switches.

Report the exact p-value. Let readers judge. If you must use a threshold, at least acknowledge the arbitrariness.

Confusing Significance Level with P-Value

Alpha is chosen before* the study. It's a property of your decision rule.

The p-value is calculated from* your data. It's a property of your observed results.

They're not the same thing. Saying "the significance level was 0.03" makes no sense. And it works.

Additional Pitfalls to Watch

P‑Hacking and Data Dredging

When researchers repeatedly slice the data, try different subsets, or keep adding participants until a significant result appears, the nominal error rate no longer reflects the true risk of a false positive. This “p‑hacking” inflates the chance of spurious findings, especially in exploratory analyses where the researcher has no a‑priori hypothesis.

Optional Stopping and Sequential Testing

Stopping data collection as soon as a threshold is crossed—often done in interim analyses of clinical trials—creates a similar inflation of Type I error. If the stopping rule is not built into the statistical plan, the effective α is larger than the nominal value, and confidence intervals become unreliable.

Overlooking Confidence Intervals

A result that just clears the α = 0.05 bar often comes with a confidence interval that barely excludes zero, say (0.001, 0.048). The width of that interval tells you how precise the estimate is. Reporting only the binary “significant / not significant” discards valuable information about the magnitude and uncertainty of the effect.

Ignoring the Context of Prior Evidence

A single study with a p‑value of 0.03 may look impressive, but if it contradicts a solid body of prior literature or if the effect size is minuscule, the finding should be treated with caution. Bayesian thinking encourages updating prior beliefs with new data rather than treating a low p‑value as definitive proof.

Misinterpreting Non‑Significant Results

Failing to reject the null does not prove that the null is true; it merely indicates insufficient evidence to conclude otherwise. Declaring “no effect” after a non‑significant test can be misleading, especially when the study was underpowered or when the confidence interval is wide.

Multiple Comparisons

When many hypotheses are tested simultaneously—common in genomics, imaging, or large‑scale surveys—the chance of at least one false positive rises dramatically. Techniques such as the Bonferroni correction, false discovery rate control, or hierarchical testing are required to keep the overall error rate in check.

Best‑Practice Checklist

Pre‑register the analysis plan (hypotheses, α level, sample size) before looking at the data.
Choose α deliberately, considering the field’s conventions and the cost of Type I versus Type II errors.
Report the exact p‑value (or, better yet, the test statistic and its sampling distribution) rather than merely flagging it against a cut‑off.
Present effect sizes with confidence intervals; this conveys practical significance and precision.
Adjust for multiple testing when conducting many related tests, and document any exploratory analyses separately from confirmatory ones.
Avoid post‑hoc switching of α or test direction; any deviation from the pre‑registered plan should be framed as exploratory.
Interpret results in context, weighing them against prior knowledge, theoretical plausibility, and practical constraints.
Encourage replication and open data sharing to verify whether initially “significant” findings hold up under independent scrutiny.

Conclusion

The significance level is a useful gatekeeper: it forces researchers to articulate, before data collection, how much evidence they require to overturn the status quo. Yet it is only one piece of a larger inferential puzzle. A low p‑value does not guarantee a meaningful effect, nor does it protect against methodological shortcuts that inflate false positives. By treating α as a flexible, pre‑specified criterion rather than a magical bright line, coupling it with effect‑size estimation, and embedding it within a transparent experimental design, researchers can harness hypothesis testing as a reliable tool for scientific progress.

When used responsibly—grounded in pre‑registration, complemented by confidence intervals, and interpreted alongside practical considerations—the significance level helps separate noise from signal, enabling cumulative knowledge to advance with greater confidence.