Benchmarks

Overview

Sparkient has four reported controlled synthetic benchmark runs. They are development evidence for the compiled path, not customer-production results.

The reported 33–42ms values are average time per item in one batched inference pass. The runner divides the total batch duration by the number of items. It does not measure individual-request p95, p99, network time, or concurrent traffic. Older Sparkient pages incorrectly labelled these values as p95.

These synthetic benchmarks test that claim under one controlled pipeline. For each domain, we:

1. Start with 5,000 synthetic examples for a bounded decision
2. Train five traditional machine-learning models and one hand-written rules baseline
3. Train and export a Sparkient text-aware compiled model
4. Evaluate the systems against a held-out split from the same synthetic generator

The question is narrow: how did the compiled model perform on those synthetic reference labels, and how long did one batched inference pass take per item? These runs do not establish performance on a new workload.

Pipeline

The repository's benchmark runner follows this broad pipeline:

1. Generate and split data — Create 5,000 examples with one synthetic generator, shuffle deterministically, and reserve 20% for evaluation.
2. Run six baselines — Test Logistic Regression, Random Forest, XGBoost, LightGBM, CatBoost, and hand-written rules. The five trained models use structured fields rather than the free-text fields used by Sparkient.
3. Run a teacher reference — Ask the configured LLM teacher to label a sample. Its results depend on the model, prompt, region, and run.
4. Compile — Train a text encoder and gradient-boosted classifier, then export the result to ONNX.
5. Evaluate — Compare predictions against the held-out synthetic reference labels.

The implementation and configuration have changed since the four figures below were recorded. Exact held-out rows, ordered predictions, fitted models, and run manifests are not currently published, so these values are not independently reproducible from the public repository.

Results

These are the four previously reported controlled synthetic results. They should not be described as a fresh cohort or customer validation.

What Each Benchmark Tests

• Support Ticket Triage — Every support team needs to route tickets to the right queue instantly. Misrouted tickets mean slower resolution and frustrated customers. This benchmark tests 5-class triage: self-service, standard, urgent, critical, and escalate.
• Content Moderation — Platforms need to classify user content in real time. Too aggressive and you lose users; too lenient and you risk harm. This benchmark tests 4-class moderation: allow, flag, restrict, and remove.
• Gaming Chat — Online games need to enforce chat policies instantly. Delayed moderation means toxic messages reach other players. This benchmark tests 4-class enforcement: allow, mute, restrict, and ban.
• Marketplace Listings — Marketplaces must review every new listing before it goes live. Manual review doesn't scale; automated review must be accurate. This benchmark tests 4-class review: approve, flag, restrict, and reject.

How to interpret the table

• Macro F1 and accuracy refer only to the synthetic reference labels in each recorded run.
• The timing is an amortised batch average, not a request-latency distribution.
• The table does not prove parity with an LLM, production accuracy, or a universal improvement over traditional machine learning.
• A new project still needs a representative held-out set and full-path latency measurement.

Cost Interpretation

These runs are not a pricing benchmark. Sparkient operations are credit-metered: a deployed compiled API decision typically uses about one credit, while training, example generation, optional LLM escalation, and hot-model serving use different amounts. Compare current plan credits with the measured tokens, requests, or infrastructure costs of the existing path.

Methodology Detail

Noisy Data

Each benchmark starts with 5,000 examples from a synthetic generator. The current generator introduces class imbalance and synthetic label noise, but that does not make the data equivalent to production traffic:

• Synthetic label noise — a fraction of generated labels are changed
• Skewed class distributions — some decision options are underrepresented
• Single-generator split — training and held-out rows come from the same generator

The current generator applies label noise before the train/test split, so held-out labels may include that synthetic noise. The runs are useful for development comparisons, not as clean ground-truth validation.

Baselines

Five traditional machine-learning models and one rules baseline are tested:

• XGBoost — gradient-boosted trees
• LightGBM — gradient-boosted trees
• CatBoost — gradient boosting with categorical support
• Random Forest — ensemble of decision trees
• Logistic Regression — linear baseline
• Rules — hand-crafted rule-based classifier

The runner removes fields classified as free text before training the five machine-learning baselines. Sparkient's compiled model uses a text encoder. Their F1 scores are therefore not a matched text-model comparison and should not be presented as a universal pipeline uplift.

Teacher

An LLM teacher evaluates a stratified sample. Its predictions provide a live-model reference for those sampled synthetic cases, not a universal quality ceiling. Teacher quality and latency depend on the selected model, prompt, region, load, and run.

Compilation

The current compilation code can use generated labels or teacher labels, depending on its configuration. The four reported figures do not include a public manifest that identifies the exact mode used. Broadly, the pipeline:

1. Selects a training sample
2. Optionally generates additional examples
3. Fine-tunes a text encoder and trains a gradient-boosted classifier
4. Exports the model to ONNX and measures the batched compiled path

Evaluation

The systems are evaluated against a held-out split from the same synthetic generator. Because the current generator applies noise before splitting and the exact historical rows are not published, treat the labels as synthetic references rather than verified ground truth.

Metrics reported:

• Macro F1 — harmonic mean of precision and recall, averaged across all classes
• Accuracy — percentage of correct predictions
• Batch-average time per item — one batched pass divided by its item count

Limitations

These benchmarks are transparent about their constraints:

• Synthetic data — All examples are generated, not sourced from real production traffic. Real-world distributions may differ.
• 5,000 examples per domain — Results may vary with larger or smaller datasets.
• Single teacher model — All benchmarks use the same LLM teacher. Different teacher models may produce different results.
• No distribution shift — Train and test data come from the same generator. Real-world deployments face distribution drift over time.
• No request-latency distribution — The runner does not measure individual warmed requests, network time, concurrency, p95, or p99.
• Unmatched text capability — The traditional models do not receive free-text fields, while the compiled model uses a text encoder.
• No public reproduction bundle — Exact rows, ordered predictions, fitted artifacts, and run manifests for the four reported figures are not published.