Benchmarks

Overview

Sparkient's core claim is that compiled decision models retain LLM-quality decisions at sub-100ms latency — fast enough to sit in hot request paths, cheap enough for every transaction.

These benchmarks measure exactly that. For each domain, we:

1. Start with noisy, realistic data (the kind a real customer would bring on day one)
2. Train 6 traditional machine learning baselines on that raw data — representing what a development team without dedicated machine learning experience could build
3. Run Sparkient's compilation pipeline (teacher re-labelling → augmentation → model training → compiled model export)
4. Evaluate both on the same held-out test set with clean ground-truth labels

The question: does the compilation pipeline produce a model that's both faster than an LLM call and more accurate than what a development team without dedicated machine learning experience could build?

Pipeline

Every benchmark follows the same 5-phase pipeline:

1. Generate noisy data — 5,000 synthetic examples per domain with 5–8% label noise, skewed class distributions, and missing archetypes. This simulates what a real customer brings on day one.
2. Run 6 machine learning baselines — six machine learning baselines — three gradient boosting variants, Random Forest, Logistic Regression, and a rule-based classifier. All trained on the raw noisy data with original noisy labels.
3. Teacher labelling — An LLM teacher evaluates a stratified sample, establishing the quality ceiling.
4. Compile — Teacher re-labels the noisy data, the augmenter fills coverage gaps, a fine-tuned text encoder + gradient-boosted classifier is trained on the clean augmented data, and the model is exported to a compiled format.
5. Evaluate — Both baselines and the compiled model are evaluated on the same held-out test set with clean ground-truth labels.

Results

All benchmark domains are shown below. Every result uses the latest compilation pipeline.

What Each Benchmark Tests

• Support Ticket Triage — Every support team needs to route tickets to the right queue instantly. Misrouted tickets mean slower resolution and frustrated customers. This benchmark tests 5-class triage: self-service, standard, urgent, critical, and escalate.
• Content Moderation — Platforms need to classify user content in real time. Too aggressive and you lose users; too lenient and you risk harm. This benchmark tests 4-class moderation: allow, flag, restrict, and remove.
• Gaming Chat — Online games need to enforce chat policies instantly. Delayed moderation means toxic messages reach other players. This benchmark tests 4-class enforcement: allow, mute, restrict, and ban.
• Marketplace Listings — Marketplaces must review every new listing before it goes live. Manual review doesn't scale; automated review must be accurate. This benchmark tests 4-class review: approve, flag, restrict, and reject.

What "Validated" Means

• ✅ Validated — The compiled model's Sparkient F1 beats the best machine learning baseline, compiled latency is under 100ms, and the result was tested using the standard noisy-data pipeline.

Cost Comparison

The LLM Cost/Decision column shows what it would cost to make each decision by calling an LLM API directly (based on typical LLM API pricing at ~500 input tokens + ~200 output tokens per decision, approximately $0.02/call). The Sparkient Cost/Decision column shows the marginal cost of each decision after model compilation — $0.00, because the compiled model runs locally without any per-request LLM calls.

At 50,000 decisions per day, the LLM approach costs approximately $30,000/month. Sparkient's compiled model costs $0.00 per decision — you pay only the fixed monthly plan.

Methodology Detail

Noisy Data

Each benchmark starts with 5,000 synthetic examples generated by an LLM. The data is intentionally degraded to simulate real-world conditions:

• 5–8% label noise — a fraction of examples carry incorrect labels
• Skewed class distributions — some decision options are underrepresented
• Missing archetypes — entire categories of inputs may be absent from the training set

This matters because real customers don't arrive with perfectly curated, balanced datasets. The benchmark measures how well each approach handles the data you'd actually have.

Baselines

Six traditional machine learning models are trained on the raw noisy data with original noisy labels:

• Gradient Boosting A — gradient boosting on tabular + text features
• Gradient Boosting B — gradient boosting alternative
• Gradient Boosting C — gradient boosting with native categorical support
• Random Forest — ensemble of decision trees
• Logistic Regression — linear baseline
• Rules — hand-crafted rule-based classifier

These represent what a development team without dedicated machine learning experience could build: take the data, train a model, deploy it. No label cleaning, no augmentation, no teacher model.

Teacher

An LLM teacher evaluates a stratified sample of the data. The teacher's predictions establish the quality ceiling — the best decision quality achievable for this domain. The teacher is too slow for production use (150ms+), but its predictions guide the compilation pipeline.

Compilation

The compilation pipeline bridges the gap between teacher quality and production latency:

1. Re-label — The teacher re-labels the noisy training data, correcting the 5–8% noise
2. Augment — Gap analysis identifies underrepresented classes and edge cases; the LLM teacher generates additional examples to fill coverage holes
3. Train — A fine-tuned text encoder produces embeddings, which feed a gradient-boosted classifier trained on the clean, augmented dataset
4. Export — The trained model is exported to a compiled model format for sub-100ms inference

Evaluation

Both baselines and the compiled model are evaluated on the same held-out test set. The test set uses clean ground-truth labels — not the noisy training labels — ensuring a fair comparison.

Metrics reported:

• Macro F1 — harmonic mean of precision and recall, averaged across all classes
• Accuracy — percentage of correct predictions
• p95 Latency — 95th percentile inference time for a single decision

Limitations

These benchmarks are transparent about their constraints:

• Synthetic data — All examples are generated, not sourced from real production traffic. Real-world distributions may differ.
• 5,000 examples per domain — Results may vary with larger or smaller datasets.
• Single teacher model — All benchmarks use the same LLM teacher. Different teacher models may produce different results.
• No distribution shift — Train and test data come from the same generator. Real-world deployments face distribution drift over time.