Intelligence Builders

AI Research Home

Running Open-Weight LLMs for Student Performance Summaries

Infrastructure evaluation — cloud GPU providers, on-site hardware, and the recommended hybrid architecture

Why We Are Evaluating This

We are currently using LLM APIs to generate summaries of student performance in Mission HydroSci. That works, but API cost can rise quickly because summaries are generated repeatedly across students, activities, and reporting cycles.

The question is not just whether an API can do the work. It is whether we can run our own model stack in a way that is:

• affordable,
• privacy-conscious,
• IRB-friendly,
• stable enough to tune once and use repeatedly,
• and operationally realistic for a university research project.

This is one of the strongest arguments for owning the primary inference path rather than relying entirely on shifting external APIs.

Recommendation

Why this is the best fit

1. Data control is strongest on-site. Student data stays inside infrastructure we manage directly.
2. Output consistency is better. We can pick one model, one quantization strategy, one prompt template, and tune for that environment.
3. Costs become more predictable. Hardware is a capital purchase rather than ongoing per-token spend.
4. Cloud still covers risk. If local hardware is down, overloaded, or temporarily insufficient, we can fail over to Runpod or Lambda.
5. The project keeps the asset. Unlike service spend, purchased hardware remains available after the initial budget period.

The Main Ways These Systems Can Be Used

There are several distinct operating modes. These are often conflated, but they are materially different.

Why Standardizing on One Primary Model Environment Matters

We have already observed this with different hosted models. The same will also be true when switching among API models, cloud-hosted open models, and local/open models on owned hardware.

This means the real product is not just “the model.” It is a summary engine, consisting of:

• the model family,
• quantization level,
• inference runtime,
• prompt template,
• context construction,
• output schema,
• and validation/evaluation rules.

The more we switch model backends, the more tuning and evaluation noise we introduce. That is one of the strongest reasons to propose one primary local/on-site environment and cloud as backup, not as a constantly changing primary.

Cloud GPU Provider Landscape

The full GPU-cloud landscape is broad. Major options in 2026 include hyperscalers such as AWS, Azure, Google Cloud, Oracle Cloud, and IBM Cloud, plus AI-native providers such as Runpod, Lambda, CoreWeave, Crusoe, DigitalOcean/Paperspace, Replicate, Vast.ai, and others.¹

For our purposes, the most relevant providers for serious evaluation are:

• Runpod
• Lambda

Those map well to our needs: open-weight model serving, cost-sensitive inference, relatively straightforward deployment, and realistic use for research workloads.

Runpod Deep Dive

Runpod is an AI-focused GPU platform offering both Pods and Serverless products.²

Main Operating Modes

Typical GPUs Relevant to Us

Pricing

Storage Pricing

Capacity Behavior

Runpod is flexible and cost-effective, but it is not guaranteed reserved enterprise capacity. When a stopped Pod is restarted, it may show “Zero GPU Pods” if the original GPU is no longer available.^{10, 11}

Security and Privacy

• Pods and workers run with containerized isolation in a multi-tenant environment
• Secure Cloud operates in enterprise-grade data centers
• Host policies prohibit hosts from inspecting Pod/worker data
• Can use vetted infrastructure partners meeting SOC 2, ISO 27001, and PCI DSS⁹
• SOC 2 Type II certification announced October 2025⁷

Runpod Summary

Lambda Deep Dive

Lambda is an AI-focused cloud company with roots in deep-learning workstations and servers. Today it provides cloud GPUs, on-demand instances, clusters, private cloud, and an enterprise trust/security program.^{12, 13}

Main Operating Modes

Pricing

Security and Privacy

• Single-tenant, shared-nothing architecture
• SOC 2 Type II certification
• Hardware-level isolation options
• Trust Portal for security posture and compliance documentation¹⁶

Lambda Summary

Runpod vs Lambda

In One Sentence

• Runpod is generally better when we want lower-cost bursty inference and flexibility.
• Lambda is generally better when we want stable AI-cloud instances and a stronger enterprise/security story.

Practical Comparison

Which Is Better for Our Project?

What Is Involved in Actually Running an LLM on Cloud GPUs

This is the mechanics stakeholders often do not understand.

The Pieces

To run an open-weight LLM on cloud GPUs, we typically need:

1. A GPU machine (Pod, instance, or cluster)
2. A model source
3. An inference engine
4. Persistent storage for model caching
5. An HTTP API surface our application can call

Inference Engine

The most important serving engine for our purposes is vLLM, which is designed for high-throughput LLM serving and exposes an OpenAI-compatible API. vLLM supports a large range of open-source models and explicitly supports deployment on Runpod.^{17, 18}

Deployment Pattern

1. Launch pod/instance
2. Mount or attach persistent storage
3. Authenticate to Hugging Face if the model requires a license grant or token
4. Start vllm serve with the chosen model
5. Expose an internal or protected HTTP endpoint
6. Send structured student-summary requests to that endpoint
7. Log outputs and evaluation metadata under our control

Open-Weight Model Sources and What Is Available

The main source for open-weight models today is Hugging Face Hub, which hosts over 2 million models.¹⁹

Common Sources and Registries

Model Families Likely Most Relevant to Us

For student-performance summaries, strong candidates include:

• Llama family (widely supported, common default)
• Qwen family (strong open-model performance)
• Mistral / Ministral family
• Gemma family
• Phi family

Model Size, GPU Requirements, and Quantization

Model Size vs GPU Memory

What This Means for DGX Spark

Quantization Levels

Can Purchased On-Site Hardware Actually Do the Job?

Yes—very likely.

The summarization task we care about is not the same as frontier general reasoning. We are asking the model to:

• digest structured student-performance data,
• identify strengths and struggles,
• and produce clear teacher-facing language.

That is a good fit for smaller and mid-sized instruct models.

Student summaries are especially compatible with smaller instruct models, 4-bit quantization, careful prompt design, and structured input. The biggest quality lever is often not raw model size; it is input organization, prompt design, and consistent deployment.

Approximate Cost Thinking

These are not procurement quotes; they are planning-level approximations.

Cloud GPU Cost Characteristics

Per-summary cost on cloud GPU can be very low when we batch requests, keep output lengths controlled, use an appropriate open model, and avoid premium API pricing. The main variables are model size, actual input length, output token count, and whether the endpoint is warm or cold.

Why Cloud Is Still Useful Even If We Buy Hardware

• Failover
• Overflow workloads
• Trying models before standardizing
• Benchmarking

Why On-Site Becomes Attractive Over Time

For recurring workloads, owned hardware changes the economics from “pay every time the model runs” into “pay once for the machine, then mostly pay power/ops.”

This is especially attractive when the institution can keep using the hardware after the initial project period.

Data Privacy, IRB, and Student Data

The Strongest Privacy Position

Run the model on-site, on hardware we control, inside network and access controls we already manage. That gives the clearest answer to:

• Where did the student data go?
• Who had access to it?
• Was it sent to a public API?
• Was it retained in someone else’s system?

Why Cloud-Hosted Open Models Are Different from API Use

Using a cloud GPU to run our own open-weight model is not the same as using a third-party hosted LLM API. With cloud-hosted open models:

• we choose the model,
• we run the inference server,
• we can minimize or disable application-side logging,
• we can control what is persisted,
• and there is no automatic model training on our prompts.

However, the infrastructure still belongs to a third party, the deployment must be configured correctly, and institutional review may still be required.

What SOC 2 Type II Certification Means — and What It Does Not

Practical Privacy Recommendations

1. Primary summaries run on-site whenever possible.
2. Cloud is backup, not default.
3. If cloud is used, use the provider’s more secure configuration (Runpod Secure Cloud, Lambda single-tenant/private options).
4. Send only the minimum required data to the summarization service.
5. Prefer structured performance signals over raw logs when possible.
6. Log and retain outputs deliberately, not by default.
7. Review IRB / institutional requirements before production use of student records in cloud environments.

Why the Hybrid On-Site + Cloud-Backup Architecture Is Best

Proposed Architecture for the Team

Operational Pattern

1. Student performance data is preprocessed into structured features/signals
2. Summarization requests go to the on-site inference service first
3. If on-site inference is unavailable or backlogged, the request is routed to the backup cloud environment
4. Summaries are stored under our normal project controls

Concrete Recommendation to Present to the Team

Recommendation Details

• Use on-site hardware as the default inference environment.
• Standardize on one primary open-weight model family and optimize prompts for that environment.
• Use Runpod as the most cost-flexible burst/backup option.
• Evaluate Lambda as the more enterprise/security-oriented backup option.
• Avoid making premium hosted APIs the primary system of record for student summaries.
• Treat cloud as backup and overflow, not default.

Why This Is the Best Approach

Because it gives us:

• a repeatable summary engine,
• stronger privacy positioning,
• long-term cost control,
• retained institutional value,
• and resilience when local hardware is insufficient or down.