Professional learning · Leaders, Coaches, Faculty · 60 minutes
Read the Dashboard Before It Reads You
Leaders take apart eight mock dashboard panels, including an AI-generated "insight," to practice asking what the data shows, what it can't show, and who could be harmed by a hasty conclusion.
Dashboards arrive with confidence: colors, arrows, and increasingly, automatically generated "insights" written in plain English. Leaders are expected to act on them. In this session, teams read eight fictional dashboard panels with the same critical eye they'd teach students to bring to an infographic: checking axes, baselines, sample sizes, what the metric actually measures, and what the data can't tell you. They also examine what personal data each panel exposes and to whom. Teams finish by redrawing one misleading panel honestly and writing a three-sentence summary a staff meeting could trust.
In this packet
- Handout A: Dashboard Panel Cards (cut-apart cards)
- Handout B: Can / Can't Tell Organizer (organizer)
- Handout C: Dashboard Data Use and Privacy Checklist (checklist)
- Facilitator key (last page)
TCEA ELE indicators
- L3.2 I can interpret complex digital data sets and communicate findings effectively using data visualization tools.
- L3.1 I know how to use digital data tools to drive school improvement efforts.
- L3.3 I am aware of various digital assessment tools and their appropriate uses in education.
- AI1.3 I am aware of the current limitations and potential biases in AI systems.
Full facilitator guide: mglearn.github.io/eles/activities/dashboard-critical-read.html
Handout A for Read the Dashboard Before It Reads You
Dashboard Panel Cards
All data is fictional, from Pecan Creek ISD. Each card describes one dashboard panel. Backs are the facilitator key; print separately.
Panel 1 · Attendance
Bar chart titled "Attendance Crisis at Oakview Middle." Five bars for August through December: 94%, 93%, 93%, 92%, 91%. The y-axis runs from 80% to 95%, so the December bar looks about one-third the height of August's. Bars turn red below 93%.
Panel 2 · Reading growth
"Reading Growth by Teacher" ranks eight 3rd grade teachers by average growth points on the fall-to-winter screener. Ms. Tran is last with +2. Class sizes and student names appear on hover.
Panel 3 · Platform success
Line chart titled "Math App Works!" Two lines: minutes on the math app per week (rising) and benchmark scores (rising) from September to February. A caption reads: "More app time = higher scores."
Panel 4 · Discipline
Pie chart: "Referrals by Grade." 6th grade 50%, 7th grade 30%, 8th grade 20%. No counts are shown. The summary line reads: "6th graders are our behavior problem."
Panel 5 · AI insight
A box labeled "AI Insight" reads: "Students in Section 4 of Algebra I are disengaged and at risk of failing. Recommended action: parent contact." A small link says "Based on platform activity."
Panel 6 · Device checkout
Table showing each student's name, home address, device serial number, and "Days since last home Wi-Fi connection." Sorted to show students who haven't connected at home in 30+ days. Visible to all campus staff.
Panel 7 · Survey
Headline number in large green type: "92% of families are satisfied with school communication!" Fine print: "Based on 48 responses to an online survey."
Panel 8 · Assessment tool
"Mastery Heat Map" from a weekly five-question quiz: rows are students, columns are standards, colored red to green. Many standards are marked "mastered" based on a single question.
Handout B for Read the Dashboard Before It Reads You
Can / Can't Tell Organizer
One row per panel. Mark a P in the first column if the panel shows personal information to people who may not need it.
| Panel (P = privacy concern) | Claim someone might make | What it actually shows | What it can't tell us | Next question or data needed |
|---|---|---|---|---|
| 1 · Attendance | ||||
| 2 · Reading growth | ||||
| 3 · Platform success | ||||
| 4 · Discipline | ||||
| 5 · AI insight | ||||
| 6 · Device checkout | ||||
| 7 · Survey | ||||
| 8 · Assessment tool |
Handout C for Read the Dashboard Before It Reads You
Dashboard Data Use and Privacy Checklist
Think of one dashboard you use regularly. Rate it honestly.
| Yes | Partly | No | |
|---|---|---|---|
| I know what each metric on this dashboard actually counts (for example, logins versus completed work). | |||
| Charts show counts or sample sizes, not only percentages. | |||
| Chart axes and baselines are shown and don't exaggerate changes. | |||
| Comparisons to prior years or similar groups are available before we draw conclusions. | |||
| Only staff with a legitimate educational need can see individual student information. | |||
| I know whether any labels or recommendations are generated by AI, and what data they're based on. | |||
| AI-generated labels about students are reviewed by a person before anyone acts on them. | |||
| We pair dashboard numbers with student work or teacher knowledge before making decisions. | |||
| When we share data with staff or families, we say what it can't tell us. | |||
| I know how long this data is kept and who at the vendor can access it. |
Notes
Facilitator only for Read the Dashboard Before It Reads You
Answer key and notes
Handout A: Dashboard Panel Cards
- Panel 1 · Attendance
- Truncated axis exaggerates a 3-point decline. The word "crisis" is an interpretation, not data. Can't tell: which students, which days (flu season? a holiday week?), whether chronic absence changed. Next: compare to prior years' same months.
- Panel 2 · Reading growth
- Can't tell: Ms. Tran has 9 students (a small sample) and a newcomer program. Ranking teachers publicly on one screener invites misuse. Privacy: student names on hover visible to anyone with dashboard access. Next: who needs this view?
- Panel 3 · Platform success
- Correlation presented as cause. Both could rise because it's later in the year and students have had more instruction. Can't tell: whether students with more app time grew more than similar students with less. Next: compare similar groups.
- Panel 4 · Discipline
- No counts: 50% of 10 referrals is different from 50% of 300. Labeling a whole grade is a harmful generalization. Can't tell: whether a few students account for most referrals, or whether referral practices differ by grade team. Next: counts and patterns by referral reason.
- Panel 5 · AI insight
- The model inferred "disengaged" from a proxy, likely login minutes or clicks. Section 4 may do more work on paper. An AI label can bias how adults see students. Can't tell: actual grades, work quality, or context. Next: ask the teacher before anyone calls families.
- Panel 6 · Device checkout
- Useful for identifying students who need connectivity support, but it exposes sensitive household information to far more people than need it. Can't tell: why a device hasn't connected (no internet? shared device? left at school?). Next: restrict access and reach out privately.
- Panel 7 · Survey
- Small, self-selected sample from families who received and opened an online survey. The families least reached by school communication are the least likely to answer it. Next: how many families were invited, and in what languages?
- Panel 8 · Assessment tool
- Appropriate use of assessment tools: one item per standard is too thin to call mastery. Can't tell: whether students can apply the skill. Next: pair the quiz with student work samples before regrouping students.