Ollama Qwen3:8B Test

1. Test Objective

This test evaluates whether the Qwen3:8B model, running locally via Ollama on consumer hardware, can generate accurate, consistent, teacher-facing student performance summaries from Mission HydroSci progress data.

The same prompt was run three times to test both output quality (accuracy, tone, rule compliance) and consistency (whether repeated runs produce stable results). The responses were then evaluated against the actual student data to identify factual errors, rule violations, and patterns of model behavior.

2. Prompt Design

The prompt is composed of four sections, each designed to constrain a specific aspect of the model's output:

Full Prompt

3. Student Data

The test uses grade data for a single student currently in Unit 3. This student has completed all of Unit 1 and Unit 2, with three flagged progress points:

4. Model Responses

The same prompt was run three times. All three responses were nearly identical, differing only in minor formatting (bold markup on "Supply Run" in runs 2 and 3). The content, sentence structure, and claims are the same across all runs.

5. ChatGPT Analysis

The three responses were submitted to ChatGPT for evaluation against the actual student data and prompt rules. The analysis identified high consistency but several accuracy and rule-compliance issues.

Consistency

What the Model Gets Right

What the Model Gets Wrong

• Issue 1: Hallucinated causal chain. "their failure to connect watershed size to flow rate in Investigate the Temple..." — This point was passed with only 1 mistake. There is no evidence of conceptual failure. The model fabricated a narrative bridge between an earlier point and a later difficulty.
  Violates: "do not reinterpret a passed progress point as misunderstanding"
• Issue 2: Using a passed point as a cause. "may have created a conceptual gap..." — The model uses a passed progress point to explain a later difficulty, which is explicitly forbidden.
  Violates: "do not use a passed progress point as the cause of a learning gap"
• Issue 3: Overgeneralization from a single flagged point. "significant gaps in understanding water flow direction and dissolved material transport" — Only one point is flagged in Unit 3. Dissolved materials have not been attempted. No data supports this claim.
  Violates: "do not overgeneralize from a single flagged result"
• Issue 4: Future prediction (too strong). "will remain hindered" — This is deterministic and unsupported by data.
  Violates: "avoid strong predictions about future performance"
• Issue 5: ID leakage. "U2P4 (Investigate Temple)" — Progress point IDs appear in the output despite explicit prohibition.
  Violates: "never include unit or progress point IDs"

Root Cause

The model optimizes for telling a coherent story rather than strict evidence-based reporting. LLMs tend to create narrative bridges ("earlier weakness led to later failure") even when the earlier weakness does not exist in the data. This is a well-known behavior pattern that requires stronger prompt constraints to suppress.

Bottom Line

Suggested Prompt Fixes

ChatGPT recommended three additions to the Final Output Rules to address the remaining issues:

6. Claude Analysis

Independent analysis of the same three responses, evaluated against the student data and prompt rules. This analysis confirms ChatGPT's findings and identifies additional issues.

Agreement with ChatGPT Analysis

ChatGPT's five identified issues are all valid. The hallucinated causal chain (Issue 1), misuse of passed data (Issue 2), overgeneralization (Issue 3), deterministic future prediction (Issue 4), and ID leakage (Issue 5) are all clearly present in the output and clearly violate the prompt rules. The consistency score of 9.5/10 is also accurate: three runs produced effectively identical content.

Additional Issues Identified

• Issue 6: False confidence on ambiguous metrics. The model claims "zero successful sends" based on the metric count=0 in the Supply Run data. However, the meaning of count is not defined in the prompt or curriculum context. The model invents a specific interpretation ("successful sends") for an ambiguous field and presents it as fact. A data-grounded summary should not interpret undefined metrics with this level of specificity.
• Issue 7: "Stalled" overstates the situation. The model describes progress in Unit 3 as "stalled," implying the student has stopped advancing. The data shows only that the student is currently in Unit 3 and has attempted one progress point. This is a snapshot, not evidence of stalling. The word choice injects a judgment that the data does not support.
• Issue 8: Not-started points treated as evidence. "Their incomplete progress in Unit 3 (Pollution Solution I–Balanced Ecosystem) indicates a need to reinforce foundational hydrological concepts before advancing." These are not-started points being used as evidence of a learning need. The prompt explicitly forbids treating not-started content as a weakness, yet the model frames them as reinforcing a deficit narrative.
• Issue 9: Partial recovery ignored. The student was flagged on Which Watershed? I but then passed Which Watershed? II with only 2 mistakes. This is evidence of developing understanding and partial recovery, which the prompt asks the model to recognize. The model never acknowledges this recovery, instead using only the flagged point to build its deficit narrative. This is a significant omission that skews the summary toward a more negative interpretation than the data warrants.
• Issue 10: Additional ID leakage. Beyond "U2P4" noted by ChatGPT, the model also outputs "U2P2 (Foraged Forging)" and "U2P6 (Which Watershed? I)" in the third paragraph. The prompt rule against IDs is violated in multiple locations, not just one.

Structural Observation

The model produces a three-paragraph structure: strengths, concerns, recommendations. This is a reasonable structure, but it creates a narrative arc that pressures the model into escalation: paragraph 1 establishes competence, paragraph 2 introduces problems, paragraph 3 prescribes fixes. This arc incentivizes the model to overstate the severity of concerns in paragraph 2 to justify the recommendations in paragraph 3. A flatter structure (e.g., progress summary, then specific observations without a prescribed fix) might reduce the tendency toward narrative overreach.

Revised Scores

The accuracy score is lower than ChatGPT's assessment because the additional issues (ambiguous metric interpretation, not-started-as-evidence, ignored recovery, multiple ID leaks) compound the original five problems. The output contains correct high-level observations but builds an unsupported deficit narrative that a teacher could act on inappropriately. For a system intended to inform instructional decisions, accuracy at the claim level matters more than getting the overall shape right.

Bottom Line

Recommendations

ChatGPT's suggested prompt fixes are a good starting point. Beyond those, I would recommend:

The most impactful structural change would be to test whether a shorter output (2 paragraphs instead of 3) reduces narrative overreach by removing the pressure to build a three-act arc. The current prompt allows 2-4 paragraphs; testing at the lower bound may produce more disciplined output.