Standardized evaluationpublished

Orion Coder 7B on M3 Max · local coding suite

Modelverse local-coding-v1 suite with machine-captured environment.

Orion Coder 7B @alice Jul 18, 2026

Manifest JSON Reproduce this Run

Speed

42.8 tok/s

Time to first token

318 ms

Peak memory

6.1 GB

Coding suite

74.2

Useful

187

12 reproductions

Reproduction command

Use the exact revision and inspect the manifest before execution.

modelverse benchmark --suite local-coding-v1 --model hf:open-lab/orion-coder-7b@d4f7b92 --runtime mlx

Environment and configuration

The comparison context that makes the result interpretable.

Hardware

MachineApple M3 Max · 64 GB

CPUApple M3 Max

AcceleratorApple integrated GPU

VRAM64 GB

System memory64 GB

Operating systemmacOS 16

Architecturearm64

Compute APIMetal

Configuration

RuntimeMLX 0.29.2

Model revisiond4f7b92

QuantizationQ4_K_M

Context length8192

Temperature0

Seed42

Batch size1

Portable Run manifest

Version 1.0 of the Apache-2.0 licensed Modelverse Run schema.

{
  "schemaVersion": "1.0",
  "id": "run_orion_m3max_coding",
  "createdAt": "2026-07-18T11:30:00.000Z",
  "model": {
    "provider": "huggingface",
    "repository": "open-lab/orion-coder-7b",
    "revision": "d4f7b92",
    "quantization": "Q4_K_M",
    "licenseSnapshot": "Apache-2.0"
  },
  "runtime": {
    "name": "MLX",
    "version": "0.29.2",
    "command": "modelverse benchmark --suite local-coding-v1 --model hf:open-lab/orion-coder-7b@d4f7b92 --runtime mlx"
  },
  "hardware": {
    "os": "macOS 16",
    "architecture": "arm64",
    "cpu": "Apple M3 Max",
    "systemMemoryMb": 65536,
    "accelerators": [
      {
        "vendor": "Apple",
        "model": "Apple integrated GPU",
        "count": 1,
        "memoryMb": 65536,
        "computeApi": "metal"
      }
    ]
  },
  "task": {
    "type": "coding",
    "evaluationSuite": "modelverse-local-coding-v1",
    "datasetRevision": "6ef21a9"
  },
  "configuration": {
    "contextLength": 8192,
    "maxGeneratedTokens": 512,
    "temperature": 0,
    "seed": 42,
    "batchSize": 1,
    "quantization": "Q4_K_M"
  },
  "metrics": {
    "timeToFirstTokenMs": 318,
    "tokensPerSecond": 42.8,
    "peakMemoryMb": 6246.4,
    "modelLoadTimeMs": 2390,
    "scores": {
      "Coding suite": 74.2
    }
  },
  "evidence": {
    "machineCaptured": true,
    "redactionsApplied": [],
    "artifactUrls": []
  }
}

Notes and interpretation

Illustrative seed result. All sample metrics in demo mode are fictional and must not be treated as real benchmark claims.

Evaluation suite

modelverse-local-coding-v1

dataset revision 6ef21a9

Other Runs for Orion Coder 7B

reproduced2 months ago

Independent reproduction on RTX 4070 Ti

by @mateo · llama.cpp b5881

Speed

67.3 tok/s

TTFT

201 ms

Peak

5.9 GB

Useful

103

Machine-captured2 months ago

Orion Coder 7B with Ollama inside 16 GB

by @chen · Ollama 0.12.1

Speed

31.4 tok/s

TTFT

480 ms

Peak

7.2 GB

Useful

149

Environment and configuration

The comparison context that makes the result interpretable.

Hardware

MachineApple M3 Max · 64 GB

CPUApple M3 Max

AcceleratorApple integrated GPU

VRAM64 GB

System memory64 GB

Operating systemmacOS 16

Architecturearm64

Compute APIMetal

Configuration

RuntimeMLX 0.29.2

Model revisiond4f7b92

QuantizationQ4_K_M

Context length8192

Temperature0

Seed42

Batch size1

{ "schemaVersion": "1.0", "id": "run_orion_m3max_coding", "createdAt": "2026-07-18T11:30:00.000Z", "model": { "provider": "huggingface", "repository": "open-lab/orion-coder-7b", "revision": "d4f7b92", "quantization": "Q4_K_M", "licenseSnapshot": "Apache-2.0" }, "runtime": { "name": "MLX", "version": "0.29.2", "command": "modelverse benchmark --suite local-coding-v1 --model hf:open-lab/orion-coder-7b@d4f7b92 --runtime mlx" }, "hardware": { "os": "macOS 16", "architecture": "arm64", "cpu": "Apple M3 Max", "systemMemoryMb": 65536, "accelerators": [ { "vendor": "Apple", "model": "Apple integrated GPU", "count": 1, "memoryMb": 65536, "computeApi": "metal" } ] }, "task": { "type": "coding", "evaluationSuite": "modelverse-local-coding-v1", "datasetRevision": "6ef21a9" }, "configuration": { "contextLength": 8192, "maxGeneratedTokens": 512, "temperature": 0, "seed": 42, "batchSize": 1, "quantization": "Q4_K_M" }, "metrics": { "timeToFirstTokenMs": 318, "tokensPerSecond": 42.8, "peakMemoryMb": 6246.4, "modelLoadTimeMs": 2390, "scores": { "Coding suite": 74.2 } }, "evidence": { "machineCaptured": true, "redactionsApplied": [], "artifactUrls": [] } }