Professional learning · Leaders, Faculty, Coaches · 75 minutes
Two Grade Levels? Vendor Claims vs. Evidence
Leaders trace an AI product's headline result back to the study behind it, read the terms of service line by line, and design the local pilot that would actually answer "Does it work here?"
"Students improved two grade levels!" is the kind of claim that moves budgets. It is also a claim leaders are rarely taught to trace. In this session, teams read a fictional AI tutoring product's marketing, the efficacy summary behind the headline, and excerpts from its terms of service. They sort statements into strong evidence, weak evidence, and privacy flags, then apply the same moves online to a real product their district is considering. The session ends with a design for a small local pilot, because the question that matters is not whether a product worked somewhere, but whether it improves learning for your students.
In this packet
- Handout A: The StepUp AI Tutor Packet (reading)
- Handout B: Evidence or Red Flag? Sort (card sort)
- Handout C: Claim-to-Evidence Ledger (organizer)
- Facilitator key (last page)
TCEA ELE indicators
- L1.3 I am aware of emerging educational technologies and their potential impact on teaching and learning.
- AI3.2 I can evaluate the effectiveness of AI-enhanced learning interventions.
- L3.3 I am aware of various digital assessment tools and their appropriate uses in education.
- AI2.3 I am aware of the importance of transparency and accountability in AI systems.
Full facilitator guide: mglearn.github.io/eles/activities/vendor-claims-vs-evidence.html
Handout A for Two Grade Levels? Vendor Claims vs. Evidence
The StepUp AI Tutor Packet
StepUp AI Tutor and NimbleNorth Learning are fictional, created for training. Highlight claims (yellow), evidence (green), and student data terms (pink).
1From the marketing page: "StepUp AI Tutor: Students using StepUp gained TWO GRADE LEVELS in reading in one semester! Our research-based, bias-free AI adapts to every learner, so no student falls behind. Trusted by more than 1,000 schools. Fully FERPA and COPPA compliant. Your data is never sold."
2From the marketing page, continued: "StepUp's patented engine is built on the science of reading and personalizes every lesson in real time. Teachers love it: 'My students are so engaged, they ask to use StepUp at recess!' — a 5th grade teacher."
3From the Efficacy Summary (a four-page PDF linked at the bottom of the page, titled "Research Brief"): "In spring, 40 students in grades 4–6 at one partner school used StepUp for 30 minutes daily as part of the school's intensive reading intervention program. Students were selected for the intervention because they scored below grade level on the fall benchmark."
4Efficacy Summary, continued: "Students took the StepUp Reading Level Check at the start and end of the semester. The average student's StepUp Reading Level rose from 2.8 to 4.9, a gain of 2.1 grade levels. Of the 52 students who began, 40 completed the program. The study was conducted by NimbleNorth Learning's research team."
5Efficacy Summary, continued: "Teachers reported that students were more engaged. Future research will include additional schools." No comparison group is mentioned. The brief does not say whether it was peer reviewed.
6From the Terms of Service (section 7.2): "User Content, including student responses and writing, may be used by NimbleNorth and its affiliates to operate, improve, and develop our products and services, including training our AI models. Content used for improvement is de-identified."
7Terms of Service, section 7.5: "We do not sell personal information. We may share information with service providers and affiliates, and in connection with a merger, acquisition, or sale of assets."
8Terms of Service, section 11: "We may update these Terms at any time. Changes are effective when posted on this page. Continued use of the Service constitutes acceptance." Section 9: "De-identified data may be retained indefinitely."
Handout B for Two Grade Levels? Vendor Claims vs. Evidence
Evidence or Red Flag? Sort
Sort each statement. On the back of each strip, write the question you would ask the vendor.
Strong evidence
Weak or missing evidence
Privacy or data-use flag
Handout B for Two Grade Levels? Vendor Claims vs. Evidence
Evidence or Red Flag? Sort: cards to sort
Cut apart the strips. Sort each one onto the mat and be ready to explain why.
Sheet 1 of 2
Students gained 2.1 grade levels on the StepUp Reading Level Check.
No comparison group is mentioned.
Forty students at one school.
Twelve of 52 students did not complete the program and were not counted.
The study was conducted by the vendor's own research team.
"Bias-free AI"
"Trusted by more than 1,000 schools."
A teacher quote about engagement.
"Fully FERPA and COPPA compliant."
An independent evaluation with a comparison group of similar students, published with its methods, showing gains on a state or nationally normed assessment.
Student responses and writing may be used to train AI models.
Information may be shared in a merger, acquisition, or sale of assets.
Handout B for Two Grade Levels? Vendor Claims vs. Evidence
Evidence or Red Flag? Sort: cards to sort
Cut apart the strips. Sort each one onto the mat and be ready to explain why.
Sheet 2 of 2
Terms may change when posted; continued use means acceptance.
De-identified data may be retained indefinitely.
Handout C for Two Grade Levels? Vendor Claims vs. Evidence
Claim-to-Evidence Ledger
Use for StepUp first, then for a real product. Trace at least one claim all the way to its source.
| Vendor claim | Evidence that would convince us | What the vendor actually provides | Gap or question for the vendor | Local pilot measure |
|---|---|---|---|---|
| "Two grade levels in one semester" | Independent study, comparison group, independent assessment, similar students | Vendor-run study, 40 students, vendor's own test, no comparison | ||
| "Your data is never sold" | Contract terms covering use, sharing, training, retention, and deletion | |||
Facilitator only for Two Grade Levels? Vendor Claims vs. Evidence
Answer key and notes
Handout B: Evidence or Red Flag? Sort
| Item | Sort | Why |
|---|---|---|
| Students gained 2.1 grade levels on the StepUp Reading Level Check. | Weak or missing evidence | The measure was created by the vendor. Ask: what did students score on an independent assessment? |
| No comparison group is mentioned. | Weak or missing evidence | Students in an intensive intervention might have grown anyway. Ask: how did similar students without StepUp do? |
| Forty students at one school. | Weak or missing evidence | A small sample from one setting may not generalize. Ask: are there results from schools like ours? |
| Twelve of 52 students did not complete the program and were not counted. | Weak or missing evidence | Dropping non-completers can inflate results. Ask: how did all 52 students do? |
| The study was conducted by the vendor's own research team. | Weak or missing evidence | Not automatically wrong, but not independent. Ask: has anyone outside the company evaluated it? |
| "Bias-free AI" | Weak or missing evidence | No AI system can promise this. Ask: how has performance been checked across student groups, including multilingual learners? |
| "Trusted by more than 1,000 schools." | Weak or missing evidence | Popularity is not efficacy. Ask: what results have those schools seen? |
| A teacher quote about engagement. | Weak or missing evidence | An anonymous testimonial about engagement says nothing about learning. |
| "Fully FERPA and COPPA compliant." | Weak or missing evidence | A self-description. Ask: will you sign our district's data privacy agreement? |
| An independent evaluation with a comparison group of similar students, published with its methods, showing gains on a state or nationally normed assessment. | Strong evidence | This is what the packet does not contain. Use it as the standard. |
| Student responses and writing may be used to train AI models. | Privacy or data-use flag | Student work becomes product-development material. Ask: can the district opt out in the contract? |
| Information may be shared in a merger, acquisition, or sale of assets. | Privacy or data-use flag | Student data could transfer to a new owner. Ask: what protections survive a sale? |
| Terms may change when posted; continued use means acceptance. | Privacy or data-use flag | The district might not notice changes. Ask for direct notice and a contract that overrides posted terms. |
| De-identified data may be retained indefinitely. | Privacy or data-use flag | Ask: what is de-identified, can it be re-identified, and when is data deleted after the contract ends? |