Two Grade Levels? Vendor Claims vs. Evidence

Overview

"Students improved two grade levels!" is the kind of claim that moves budgets. It is also a claim leaders are rarely taught to trace. In this session, teams read a fictional AI tutoring product's marketing, the efficacy summary behind the headline, and excerpts from its terms of service. They sort statements into strong evidence, weak evidence, and privacy flags, then apply the same moves online to a real product their district is considering. The session ends with a design for a small local pilot, because the question that matters is not whether a product worked somewhere, but whether it improves learning for your students.

Objectives

• Participants will trace a marketing claim to its source and evaluate the study's design: comparison group, sample, measure, duration, and independence.
• Participants will identify data-use and privacy provisions in terms of service that affect students, including use of student work to train AI models.
• Participants will evaluate a real product's published claims and terms using the same criteria.
• Participants will design a small local pilot with a comparison and a learning measure that would evaluate an AI-enhanced intervention.

Step by step

1. 10–6 min

   Hook

   The headline

   Project only the headline from Handout A: "Students using StepUp AI Tutor gained two grade levels in reading in one semester!" Ask: "If a board member forwarded you this, what would you need to know before you believed it?" List questions on chart paper.

   Facilitator noteKeep the list. By the end, teams will see how many of their own questions the vendor's materials actually answer.

2. 26–21 min

   Explore

   Read the packet with three highlighters

   Each person reads Handout A. Highlight claims in yellow, evidence that supports a claim in green, and anything about student data in pink. Teams then compare: which yellow claims have green evidence behind them? What does the pink tell you about where student work goes?

   Facilitator noteThe efficacy summary is the heart of the packet. Look for teams that notice the 40 students, the lack of a comparison group, the vendor-written test, and the fact that the students were already in an intensive intervention.

3. 321–33 min

   Practice

   Evidence or red flag?

   Teams sort the 14 statements in Handout B into Strong evidence, Weak or missing evidence, and Privacy or data-use flag. For each, they note the question they would ask the vendor. Compare with the facilitator key and discuss any disagreements.

   Facilitator note"FERPA and COPPA compliant" usually gets sorted as strong evidence. It is a claim the vendor makes about itself. Compliance depends on how the tool is used, the district's agreement, and the actual data practices, which your district's privacy review should confirm.

4. 433–53 min

   Apply

   Now a real product

   Each team takes one real product from your list and, using their devices, finds its efficacy claims, any research it cites, its privacy policy, and its terms of service. They fill Handout C: each claim, what evidence would convince us, what the vendor actually provides, the gap or question, and a local measure. They should read laterally too: what do independent sources say about the product and its research?

   Facilitator noteCoach teams to trace at least one claim all the way to its source. Many "research-based" labels lead to a white paper written by the vendor, a study of a different product, or research on a general practice the product claims to use.

5. 553–65 min

   Create

   Design the local pilot

   On the back of Handout C, each team sketches a small, honest pilot for their product: which students and for how long, what the comparison is (similar classes without it, or the same students before and after with a clear baseline), what learning measure you'll use that the vendor didn't create, what you'll look for in student work, and what would make you stop. Include who might be left out of the pilot and how families will be informed.

   Facilitator noteA small pilot with a fair comparison and real student work beats a large rollout with only usage data. Usage data tells you students logged in; it can't tell you whether they learned.

6. 665–75 min

   Debrief

   Back to the headline

   Return to the Hook questions. Ask: "Which of our questions did the vendor answer? Which could only a local pilot answer?" Each team shares its product's biggest gap and one line from its terms of service that surprised them. Close by naming the moves: trace the claim, check the design, read the terms, test it locally.

   Facilitator noteConnect to media literacy: this is lateral reading and claim-tracing applied to procurement. The same moves students use on a viral post work on a vendor's slide deck.

Adaptations

Higher Ed

Swap StepUp for an AI courseware or tutoring platform marketed to colleges with a claim about course pass rates, and involve institutional research staff in designing the pilot.

Curriculum and instruction teams

Extend the Create step into a full pilot proposal with timeline, training plan, and the student work samples teachers will collect.

Short on time (45 min)

Skip the real-product research; use Handouts A and B, then design the pilot for StepUp itself.

Pairs well with

• Generative AI Adoption ChecklistA structured companion for AI procurement decisions.
• PROTECT Rubric v2.0Use to evaluate privacy and safety terms in depth.
• Digital Inquiry Group: Civic Online ReasoningLateral reading lessons; the same moves apply to vendor claims.