APPLE SILICON · LOCAL EVALUATION
A static results page for Local Pro, a local LLM evaluation and reporting toolkit developed on a MacBook Pro.
Static benchmark results — no live inference.
STRICT PROMPT ACCURACY
79.67%431/541 prompts
Fixed P0 · Original IFEval scorer
Public benchmark self-evaluation
01 / THE SCORECARD
Prompt-level accuracy asks whether every instruction in a prompt was followed. Instruction-level accuracy counts individual requirements. These metrics are not averaged together.
| Metric | Correct / total | Accuracy |
|---|---|---|
| Strict prompt · primary | 431/541 | 79.67% |
| Strict instruction | 715/834 | 85.73% |
| Loose prompt | 452/541 | 83.55% |
| Loose instruction | 740/834 | 88.73% |
541 prompts · 834 instruction checks · 78 verified cached + 463 new responses.
All 9 length-limited outputs retained (4 strict failures, 5 passes) · 0 retries.
02 / BUILT LOCALLY
mlx-community/Qwen3.8-27B-4bit
27B, MLX affine 4-bit, group size 64.
Artifact revision3e6447f082e89cc7f0bc6e5441afd38dfce760ff
Python 3.12.13 · MLX 0.32.2 · MLX Metal 0.32.2 · mlx-vlm 0.7.0.
Tokenizer/template hashes and exact settings: frozen configuration.
P0 sends the original user message only: no extra system prompt, few-shot examples or tools.
Original IFEval scorer revisione6890f85757dd84e27ca6df2dd30651dafad28e0.
Developed on MacBook Pro / Apple Silicon. All 463 new loads recorded 48 GiB RAM. The September 9, 2026 historical profile lists Apple M5 Max (18 CPU / 40 GPU cores), macOS 26.6.2; these are not fresh per-run attestations. Environment evidence ↗
Who contributed what: Qwen provides the base model; MLX-community provides the quantized conversion. Aiden built execution/evaluation integration, result tracking, reporting code and reproducibility documentation, with Codex development assistance. No new foundation-model training is claimed; the exact upstream conversion commit remains unverified.
03 / HISTORICAL CONTEXT
Different evaluation setups; not a head-to-head comparison.
English IFEval strict prompt-level accuracy. Historical, nearby and higher published values are shown together. External values were reported by their listed sources, not measured by Local Pro.
| Model / snapshot | Accuracy | Source / evaluation date |
|---|---|---|
| GPT-4 (API snapshot unspecified) | 76.89% | IFEval authors (Google/Yale)2023-11 |
| GPT-4o-mini-2024-07-18 | 80.40% | Qwen TeamEvaluation date not stated |
| GPT-4o 0513 | 84.30% | DeepSeekEvaluation date not stated |
| Claude-3.5-Sonnet-1022 | 86.50% | DeepSeekEvaluation date not stated |
Output budgets, sampling and evaluation dates differ. No statistical significance, general superiority, frontier equivalence or Hugging Face certification is claimed. Detailed source conditions ↗
04 / REPRODUCIBILITY & RIGHTS
Rebuild reports and graphs from stored verdicts, using Python's standard library. This page is generated from a small fixed scorecard; it makes no runtime API calls.
Full raw-response rescoring needs excluded raw outputs and evaluation assets. New inference needs a separate execution core and valid authority. Neither is available here. Fresh-machine generation is untested.
Public, development-exposed benchmark self-evaluation—not hidden testing, third-party certification or proof of general performance superiority.
Original Local Pro code, this static page and associated documentation: MIT License, Copyright (c) 2026 Aiden Oh. Third-party components retain their licenses; see notices. No vendor scorer, model weights, tokenizer, prompts, raw outputs or private ledgers are bundled in this Space.
Source commit: 85ac6a63dcdcf9eb233e6606492cf8f8cd311003
Source hashes · Scorecard data · Space file inventory