How to Evaluate EdTech Efficacy Claims and ESSA Evidence Tiers

To evaluate an edtech efficacy claim, check four things in order: how the comparison group was formed, what test measured the outcome, how large the effect was relative to its cost, and whether the study had enough schools to detect a realistic effect. The ESSA evidence tiers give a starting label, but they say nothing about effect size or price. A Tier 1 product with a trivial effect carries the same label as one with a large effect.

Every edtech company claims its product improves learning. Few can show it to a standard that would persuade a skeptical researcher. The gap matters because education effects are small. An analysis of 141 large, rigorous education trials found an average effect of about 0.06 standard deviations, while vendor brochures routinely cite effects five or ten times larger. This guide explains how to tell the difference, using the framework in Chapter 14 of The Learning Equation.

Why is edtech efficacy so hard to prove?

Three features of education make naive comparisons misleading.

Students grow anyway. Typical annual gains are roughly 0.4–0.6 standard deviations in upper elementary school and less later. A product's users improve whether or not it helps, so "students grew 20% this year" proves nothing.

Motivated students use products more. Comparing heavy and light users mostly measures the difference between motivated and less motivated students.

Students are taught in groups. Because students share teachers and schools, rigorous trials must randomize whole classrooms or schools and need many of them.

By the time rigorous evidence arrives, purchasing decisions have usually been made. That is why buyers need a fast way to read what vendors hand them.

Step 1: Identify the ESSA evidence tier

The Every Student Succeeds Act of 2015 defines four levels of evidence.

Department of Education guidance adds that Tier 1 and 2 studies should ideally involve large, multi-site samples, pointing to at least about 350 students across more than one school or district. Tiers 1–3 are required for some school-improvement funds and favored elsewhere, and the relief-era requirement that districts spend at least 20% of ARP ESSER funds on evidence-based interventions made tier claims a sales tool.

Two cautions follow. Tier 3 is relatively cheap to claim: a vendor-commissioned correlational study of usage and outcomes with prior-achievement controls can qualify. And the tiers say nothing about effect size or cost.

Step 2: Check the study design

Rank the evidence by how the comparison group was formed.

Then ask which question the study answers. An intent-to-treat estimate compares everyone assigned to the product, whether or not they used it. It tells a district what happens if it buys. A treatment-on-the-treated estimate rescales for actual use. It tells a designer what the product does when used, but not what a buyer will get. Most districts fall short of vendors' recommended weekly minutes for most students, so the gap between the two can be large. This is why adaptive products such as Curriculum Associates' i-Ready publish recommended dosage, and why buyers should ask what share of licensed students actually met it.

Step 3: Watch for the usage-study trap

The most common vendor claim is that students who used a product for some threshold of minutes a week grew more than students who did not. The research companion to the book simulates 20,000 students in which a product has no true effect but more motivated, higher-achieving students use it more. The naive comparison shows a sizable "effect." Controlling for the pretest shrinks it but leaves a spurious residual, because motivation is unobserved.

Dose-response correlations are consistent with an effect. They are not evidence of one. When a vendor shows a chart of "students meeting recommended usage grew X more," ask for a comparison with similar students who did not have access.

Consumer apps face the same problem. Duolingo has published in-house studies reporting that learners who complete its early Spanish and French units reach reading and listening levels comparable to several semesters of US university study. The studies are company-run, mostly non-randomized and focused on completers, the most motivated users. The book's advice is to read company-commissioned efficacy studies as marketing until independently replicated.

Step 4: Read the effect size properly

An effect size expresses the difference between treatment and control in standard-deviation units, so that studies using different tests can be compared. An effect of 0.20 standard deviations moves the average student from the 50th to about the 58th percentile of the control group.

Do not use Cohen's laboratory rules of thumb (0.2 small, 0.5 medium, 0.8 large). By Matthew Kraft's benchmarks for causal studies on standardized tests, 0.20 or more is large. Translations into "months of learning" are popular but sensitive to grade and test.

Four things inflate effect sizes:

• Developer-made tests aligned to the product, which roughly double effects compared with independent measures.
• Small samples, because publication favors lucky large estimates.
• Short durations.
• Quasi-experimental designs.

A 0.5 effect on a vendor's unit test and a 0.1 effect on a state test may describe the same product.

Step 5: Count schools, not students

A study's minimum detectable effect size is the smallest true effect it has a good chance of detecting. Underpowered studies produce null results that do not mean "no effect," and, when they reach significance by luck, exaggerated estimates.

Clustering makes this worse. Because students share teachers and schools, the information in a clustered sample is reduced by a design effect. With 60 students per school and a typical intraclass correlation of 0.20, 2,400 clustered students carry the information of fewer than 200 independent ones. Many vendor-run studies with 10–20 schools can detect only effects far larger than almost any real intervention produces. Pretest controls help a great deal, but counting schools rather than students is the first check.

Step 6: Weigh effect against cost

A stylized comparison in the book shows why cost belongs in the same judgment as effect.

The software is about 6.3 times as cost-effective, if its small effect is real. But proving a 0.05 effect takes an individually randomized trial of about 12,560 students, and if whole classrooms of 25 are randomized, about 57,800 students in 2,312 classrooms. A cheap intervention with a small effect can be the better buy, but proving it takes a trial most vendors never run.

Read the point estimate, its confidence interval and the cost per student together. A modest effect with a tight interval at a low price can be an excellent buy. A large effect with an interval that spans zero is a coin flip.

Step 7: Check who reviewed it

Independent reviewers add credibility.

• The What Works Clearinghouse, created by the Institute of Education Sciences in 2002, reviews studies against published standards. Its contracts were cancelled in the February 2025 federal cuts, and its future scope remained under review in 2026.
• Evidence for ESSA, from Johns Hopkins, rates programs against the tiers using independent measures and minimum samples.
• Pre-registration registries guard against cherry-picked outcomes.

Privacy, interoperability and design certifications matter in procurement, but they say nothing about learning.

What do landmark studies teach about edtech evidence?

The last row matters for buyers of AI tools. Our explainer on the AI tutoring business model covers what the 2024–26 trials show.

A quick checklist for buyers and investors

• Is the evidence for the product as sold today, or for an earlier version?
• Was the outcome an independent test, such as a state assessment?
• How many schools, and was assignment random?
• Is the estimate intent-to-treat?
• What is the effect per dollar, and what is its confidence interval?
• Who paid for and who ran the study, and was it pre-registered?
• What usage did the study achieve, and can my district match it?

How should edtech companies build evidence?

The book recommends a staged portfolio as revenue and product stability allow: a research-based logic model (Tier 4), rapid-cycle correlational studies with honest controls (Tier 3), a matched comparison using district data (Tier 2), and finally an independent, pre-registered randomized trial (Tier 1). Independent school-randomized trials commonly cost about $1–5 million or more (reported), so the last step usually comes after product-market fit. Publishing null results builds trust with buyers, and district leaders often care more about results in districts like theirs than about a national trial. Evidence is now part of selling to school districts, not a side project.

Which factor of the learning equation does evidence represent?

Evidence is the fourth factor of the learning equation, and it multiplies with the others. A product can have sound learning design, a willing payer and a cheap route to market, and still score zero at a district's renewal if it cannot show results to the standard that district requires. For consumers, the equivalent proof is a first annual renewal. For institutions it is increasingly an ESSA tier, a cost-effectiveness case and data from their own students. See how this fits the wider economics in our edtech business model guide.

The takeaway

Evaluating edtech efficacy means looking past the tier label to the design, the test, the sample and the cost. Randomized trials on independent tests, with enough schools to detect a realistic effect, are the gold standard; usage correlations and vendor-made tests are the most common and least reliable evidence. Modest effects are normal in education, and a small, real effect at a low price can be a better buy than a large, uncertain one. Chapter 14 of The Learning Equation explains how education evidence is produced and judged.