The Glass Box Checklist

Overview

AI tools arrive in schools with confident marketing: "safe," "unbiased," "research-based," "personalized." Transparency means being able to see behind those words: what the tool does, what data it uses, where it fails, and who answers for it when it's wrong. In this session teams read a fictional vendor fact sheet, sort a bank of evaluation questions, and build a short checklist that turns vague reassurance into answerable questions. Then they test the checklist on the published documentation of a tool their district already uses or is considering.

Objectives

• Participants will distinguish transparent, checkable vendor claims from vague or marketing-only claims.
• Participants will build a transparency and accountability checklist covering how a tool works, its data practices, its limits, and recourse when it's wrong.
• Participants will apply their checklist to a real tool's published documentation and identify what questions remain unanswered.

Step by step

1. 10–5 min

   Hook

   Glass box or black box?

   Hold up a clear glass jar and an opaque box (or show two pictures). Ask: "If an AI tool made a recommendation about a student, which of these would you want it to be?" Then: "Most AI tools are at least partly black boxes; even their makers can't fully explain every output. Transparency doesn't mean seeing every gear. It means knowing enough to decide whether to trust it, and knowing who answers when it's wrong."

   Facilitator noteThis is an honest technical point: large models can't give a complete step-by-step reason for each output. Vendors can still be transparent about purpose, data, testing, limits, and accountability.

2. 25–17 min

   Explore

   Read the fact sheet

   Everyone reads Handout A, the mock BrightPath Writing Coach fact sheet. Highlight in one color claims you could check (specific, verifiable) and in another claims you couldn't (vague or marketing). Then list, in the margin, the three questions you'd most want answered before 30 students use this tool. Teams compare.

   Facilitator noteLook for teams catching "bias-free," "research-based" with no research named, "may share with trusted partners," and the missing information about what happens when the feedback is wrong.

3. 317–29 min

   Practice

   Sort the questions

   Teams sort the Handout B question strips into four categories: How it works and its limits, Data and privacy, Accountability and recourse, and Evidence of learning. Then each team stars the five questions they consider non-negotiable before classroom use and checks whether Handout A answers them.

   Facilitator noteExpect debate about where some questions belong. That's healthy. The facilitator key gives one defensible placement, not the only one.

4. 429–37 min

   Create

   Build your checklist

   Using Handout C as a starter, teams cross out items they don't need, add their own starred questions, and rewrite any item so it can be answered yes, partly, or no from documentation or a vendor reply. The goal is 8–12 items that someone else on campus could use without the team present.

   Facilitator notePush for plain language. "Does the tool describe known limitations?" is checkable. "Is the tool ethical?" is not.

5. 537–52 min

   Apply

   Test it on a real tool

   Each team picks a real tool from your list and uses its public privacy policy, terms, and AI documentation to complete their checklist. Mark Yes, Partly, or No and note where each answer was found. Anything you can't find goes on the "Still unanswered" chart. Unanswered questions are findings, not failures.

   Facilitator noteRemind teams to judge only what the documents say, not rumors or reputation. If a question needs a vendor reply, write it as an email question the district could send.

6. 652–60 min

   Debrief

   What's still unanswered?

   Review the "Still unanswered" chart. Ask: "Which unanswered question would stop you from using this tool with students? Which is a 'use with caution'?" Then ask leaders: "Who in our district should own sending these questions, and how will teachers see the answers?" Close by agreeing on where the finished checklists will live.

   Facilitator noteTransparency goes both ways: if the district adopts a tool, families and students deserve a plain-language summary of what it does with their data.

Adaptations

District leaders and technology directors

Merge team checklists into one draft for the district's procurement or tool-approval process. Add a column for who reviewed each item and when it should be rechecked.

Librarians

Take the checklist to the library's database and research-tool subscriptions, many of which now include AI summaries or search features, and add a family-facing summary for each.

Classroom teachers

Pare the checklist to five items a teacher can check in 15 minutes before trying a new AI feature, and share it with your PLC.

Pairs well with

• Generative AI Adoption ChecklistA fuller adoption review to compare with the checklist your team built.
• PROTECT Rubric v2.0Deepens the data and privacy portion of your checklist.
• Family App Privacy AuditA family-facing version of the same questions for apps used at home.