Intelligence Builders
The MHS Grader is the automated assessment engine for the Mission HydroSci project. It continuously polls and processes gameplay event data captured by StrataLog, evaluates student performance against curriculum-aligned criteria, and produces structured progress evaluations that power the teacher-facing MHS Dashboard in StrataHub. The progress evaluations shown on the dashboard, whether a student passed, was flagged for attention, or is actively working on a progress point, originate from the MHS Grader.
The grader evaluates 26 progress points distributed across the five units of the Mission HydroSci curriculum. Each progress point corresponds to a specific learning activity within the game, and each has its own grading rule that defines what constitutes successful performance. Some rules are simple completion checks. Others involve counting mistakes, computing weighted scores, measuring response times, or evaluating whether a student used specific reasoning strategies. This range of rule complexity reflects the diversity of skills assessed across the curriculum, from basic game orientation to constructing persuasive scientific arguments.
The MHS Grader was purpose-built because the project requires an assessment system that can process high-volume gameplay event data in near-real-time, apply domain-specific grading logic that varies by activity type, produce detailed metrics beyond simple pass/fail outcomes, and integrate tightly with the StrataLog event pipeline and the StrataHub dashboard. No existing assessment platform the project evaluated provided this combination of capabilities for educational game analytics.
Mission HydroSci is organized into five units, each focused on a core set of hydrology and scientific argumentation concepts. Within each unit, students complete a series of activities that teach and assess specific skills. The grader maps each assessable activity to a progress point.
Students learn game controls, meet crew members, and are introduced to the structure of scientific arguments. Activities include orientation tasks, crew introductions, and identifying claims within arguments.
Students learn to read topographic maps, understand the relationship between watershed size and water flow, and practice identifying parts of scientific arguments. Activities include matching topographic maps to elevation profiles, navigating using map features, investigating watershed properties, classifying argument components, and constructing a comparison argument about watershed size.
Students learn how water flows through watersheds and how dissolved materials move with water. Activities include identifying flow direction from watershed maps, predicting pollutant spread, constructing arguments with reasoning and supporting evidence, navigating puzzle environments, and predicting nutrient distribution.
Students explore underground water systems, soil properties, and infiltration rates. Activities include water table concept puzzles, soil infiltration experiments across multiple difficulty levels, multi-component synthesis tasks, constructing persuasive scientific arguments, and matching plant water needs to soil properties.
Students investigate evaporation rates, condensation, and the water cycle. Activities include evaporation rate puzzles, water chamber manipulation experiments, countering faulty scientific claims about the water cycle, and fixing a solar still with zero tolerance for errors.
Each of the 26 progress points is identified by a unit and point number (for example, U2P3 refers to Unit 2, Progress Point 3). The grader maintains a rule for each progress point that defines:
For each progress point, the grader assigns one of three statuses:
When a student is flagged, the grader includes a reason code that provides specific information about why the flag was raised. Teachers and researchers can use these codes to understand the nature of a student's difficulty:
Beyond the pass or flag status, each grade includes a set of metrics that capture quantitative details about the student's performance. These metrics vary by progress point but commonly include:
These metrics give teachers and researchers a detailed picture of student performance that goes beyond a simple pass or fail determination.
The MHS Grader runs as a standalone background service, separate from StrataHub and StrataLog. It connects to the same database cluster but operates independently, reading gameplay events from the StrataLog database and writing grades to its own database. This separation ensures that the grader can be updated, restarted, or scaled without affecting the other components of the Strata system.
The grader has no web interface or HTTP server. It is a continuous processing engine that starts, connects to its data sources, and runs until stopped.
The grader operates on a continuous polling cycle:
A key design feature of the grader is its use of attempt windows to isolate the events relevant to each evaluation. When a student triggers evaluation for a progress point, the grader looks back through the event history to find the boundaries of the current attempt:
Only events within this window are considered during evaluation. This design makes the grader replay-safe: if a student replays an activity, only the most recent attempt is evaluated, and prior attempts do not interfere with the current grade.
The cursor-based scanning system ensures that the grader produces consistent results regardless of when it runs or how often it restarts. If the grader is stopped and restarted, it resumes from its last recorded cursor position and processes only events it has not yet seen. This property is essential for a production assessment system where reliability and consistency are critical.
The 26 grading rules span a range of complexity, reflecting the variety of skills assessed across the curriculum.
The simplest rules check only whether a student reached the completion point of an activity. If the trigger event fires, the student passes. These rules are used for orientation and narrative activities where the learning objective is exposure to content rather than demonstration of a specific skill.
Threshold rules count specific types of events within the attempt window and compare the count against a limit. A rule might flag a student if they received more than six instances of negative feedback during a navigation task, or if they made any errors at all in a zero-tolerance activity.
Scoring rules compute a numerical score based on weighted positive and negative events. A correct action might add one point while an incorrect action might subtract a fraction of a point, with a minimum passing score defined for the rule. Some scoring rules use capped penalties, where each category of error can reduce the score by at most a fixed amount, preventing a single type of repeated mistake from overwhelming the overall assessment. Others use bonus systems that award additional credit for using advanced strategies such as including supporting evidence in an argument.
Some rules incorporate response time as a factor in the grade. An activity might award a duration bonus if the student answered within 30 seconds, a partial bonus for answers within 90 seconds, and no bonus for longer response times. Active duration calculations exclude extended idle periods (gaps longer than two minutes by default) to avoid penalizing students for interruptions.
Complex activities that span multiple game elements use multi-component rules that aggregate results across several sub-activities. Each component contributes to the overall score independently, and the component scores are summed and compared against an overall threshold.
The strictest rule type requires perfect performance. If any negative event occurs during the attempt, the student is flagged. This rule type is used sparingly, only for activities where the learning objective demands precise execution and any error represents a meaningful misunderstanding of the concept.
The grades produced by the MHS Grader are stored in a database that the MHS Dashboard reads when rendering the progress grid for a teacher or researcher.
Each student's grades are stored as a single document containing all 26 progress points. For each progress point, the document records the status (passed, flagged, or active), the reason code if flagged, the computed metrics, the timestamps of the start and end events, the duration measurements, and the grading rule version that produced the grade. If a student has multiple attempts at a progress point, each attempt is recorded, providing a complete history of the student's performance trajectory.
When a teacher opens the MHS Dashboard, it queries the grade database for all students in the selected group. The dashboard renders a grid with students as rows and progress points as columns. Each cell displays the color-coded status: green for passed, yellow for flagged, blue for active, and gray for not started. Teachers can examine individual grades to see the reason code and metrics, giving them specific, actionable information about each student's performance.
The grader also records the most recently detected unit for each student, based on the most recent unit-start event encountered. This information appears on the dashboard, allowing teachers to see at a glance which unit each student is working on, even between graded activities.
The complete data flow from student gameplay to teacher-visible grades follows this pipeline:
This pipeline operates continuously and automatically. No manual intervention is required to grade student work or update the dashboard. Teachers see near-real-time progress data as students play.
The grader maintains a persistent cursor (the ID of the last processed event) in its database. This cursor survives service restarts, ensuring that no events are skipped or reprocessed after a restart. The cursor-based approach also means the grader can be stopped for maintenance and resumed without any data loss or inconsistency.
The polling interval, batch size, and database connections are configurable through environment variables. The default polling interval is five seconds, meaning that new grades typically appear on the dashboard within seconds of a student completing an activity. The batch size controls how many events are processed per cycle, allowing the system to handle bursts of activity without falling behind.
The grader's active duration metric excludes idle gaps longer than a configurable threshold (two minutes by default). This ensures that time-on-task measurements accurately reflect engaged work time rather than clock time, which is important for research analysis and for rules that incorporate timing into their scoring.
Each grading rule carries a version identifier that is recorded with every grade it produces. If a rule is updated to reflect a curriculum change or a refinement in grading criteria, the version identifier changes, allowing researchers to distinguish grades produced under different rule versions when analyzing data.
The MHS Grader operates as part of the broader Strata system, with specific relationships to each other component:
StrataLog is the grader's primary data source. Every event the grader processes originates from StrataLog. The grader reads from the StrataLog database but never writes to it, maintaining a clean separation between event collection and event analysis.
StrataHub hosts the MHS Dashboard, which is the primary consumer of the grades the grader produces. StrataHub also manages the student accounts and classroom groups that give structure to the grade data. The grader does not communicate with StrataHub directly; they share data through the grade database.
StrataSave provides game state persistence so students can resume where they left off. While the grader does not interact with StrataSave, the continuity StrataSave provides is valuable for grading: because students resume at their last save point rather than replaying content, their event streams in StrataLog are clean and sequential, making the grader's attempt window logic reliable.
Together, these components form a pipeline in which StrataLog collects the raw data, the MHS Grader transforms it into structured assessments, and the MHS Dashboard presents those assessments to teachers and researchers.
Several categories of existing assessment tools were considered before building the MHS Grader:
Learning management system (LMS) gradebooks such as those in Canvas, Blackboard, or Google Classroom are designed for manually entered grades, quiz scores, and assignment submissions. They cannot process near-real-time gameplay event streams, apply complex scoring rules with attempt windows, or produce the detailed metrics required for research-grade assessment of game-based learning activities.
Game analytics platforms such as GameAnalytics or Unity Analytics provide aggregate metrics about player behavior (session length, retention, feature usage) but are not designed to evaluate individual student performance against curriculum-specific criteria. They do not support custom grading rules or integration with educational dashboards.
Standardized assessment platforms such as those used for state testing are designed for fixed-form assessments with predetermined questions and answer keys. They cannot assess the open-ended, process-oriented activities in Mission HydroSci, where the evaluation depends on the sequence, timing, and pattern of student actions across an extended gameplay session.
Rule engine frameworks such as Drools or business rule management systems provide generic rule evaluation capabilities. However, they are designed for business process automation, not for processing ordered gameplay event streams with attempt windows, cursor-based scanning, and duration calculations. Adapting a generic rule engine to this domain would require extensive customization while introducing unnecessary complexity and dependencies.
The MHS Grader was built because the project needed an assessment engine that combines near-real-time event stream processing, curriculum-specific grading rules with configurable thresholds, attempt window isolation for replay safety, detailed performance metrics beyond pass/fail, cursor-based idempotent processing, and tight integration with the StrataLog event pipeline and the StrataHub dashboard. No existing tool provides this combination.
The MHS Grader provides the following capabilities in support of the Mission HydroSci research project: