Test Matrix 99 results
| Test | deepseek-r1:8b | gemma4:e2b | gemma4:e4b | qwen3:14b | qwen3:8b |
|---|
| data-dbt | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| data-warehouse | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| fin-cashflow | ✓ 5.0 | ✓ | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| fin-model | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| fin-vat | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| health-icd10 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| health-readmit | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| iot-mcu | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✗ | ✓ 5.0 |
| iot-sensor | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| leg-ccma | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| leg-msa | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| leg-popia-breach | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| mkt-brand | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| mkt-gtm | ✓ 5.0 | ✓ | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| mkt-popia | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| orch-brief | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| orch-scope | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| sales-discovery | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| sales-pipeline | ✗ | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
Model Details
deepseek-r1:8b — 18/19 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 106.2s | 2383 |
| data-warehouse | ✓ | 107.5s | 2373 |
| fin-cashflow | ✓ | 107.8s | 2411 |
| fin-model | ✓ | 76.8s | 1869 |
| fin-vat | ✓ | 49.6s | 1324 |
| health-icd10 | ✓ | 106.3s | 2374 |
| health-readmit | ✓ | 107.6s | 2371 |
| iot-mcu | ✓ | 106.7s | 2405 |
| iot-sensor | ✓ | 107.7s | 2406 |
| leg-ccma | ✓ | 106.3s | 2369 |
| leg-msa | ✓ | 107.5s | 2373 |
| leg-popia-breach | ✓ | 106.3s | 2374 |
| mkt-brand | ✓ | 64.9s | 1580 |
| mkt-gtm | ✓ | 109.8s | 2369 |
| mkt-popia | ✓ | 63.0s | 1544 |
| orch-brief | ✓ | 106.1s | 2208 |
| orch-scope | ✓ | 81.9s | 1746 |
| sales-discovery | ✓ | 106.3s | 2400 |
| sales-pipeline | ✗ | 107.6s | 2413 |
gemma4:e2b — 23/23 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 38.7s | 2413 |
| data-warehouse | ✓ | 39.0s | 2400 |
| fin-cashflow | ✓ | 131.2s | 2284 |
| fin-cashflow | ✓ | 112.7s | 2236 |
| fin-cashflow | ✓ | 36.9s | 2330 |
| fin-model | ✓ | 38.2s | 2423 |
| fin-vat | ✓ | 22.8s | 1591 |
| health-icd10 | ✓ | 30.8s | 1977 |
| health-readmit | ✓ | 37.6s | 2315 |
| iot-mcu | ✓ | 38.7s | 2423 |
| iot-sensor | ✓ | 39.1s | 2422 |
| leg-ccma | ✓ | 32.2s | 2051 |
| leg-msa | ✓ | 31.3s | 1984 |
| leg-popia-breach | ✓ | 31.2s | 1963 |
| mkt-brand | ✓ | 25.1s | 1672 |
| mkt-gtm | ✓ | 84.9s | 2393 |
| mkt-gtm | ✓ | 55.4s | 2393 |
| mkt-gtm | ✓ | 42.4s | 2393 |
| mkt-popia | ✓ | 27.7s | 1819 |
| orch-brief | ✓ | 19.3s | 1185 |
| orch-scope | ✓ | 22.3s | 1345 |
| sales-discovery | ✓ | 38.6s | 2422 |
| sales-pipeline | ✓ | 36.0s | 2278 |
gemma4:e4b — 19/19 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 112.8s | 2413 |
| data-warehouse | ✓ | 119.0s | 2400 |
| fin-cashflow | ✓ | 82.7s | 2439 |
| fin-model | ✓ | 135.4s | 2450 |
| fin-vat | ✓ | 75.3s | 2300 |
| health-icd10 | ✓ | 70.3s | 2393 |
| health-readmit | ✓ | 71.0s | 2390 |
| iot-mcu | ✓ | 72.9s | 2423 |
| iot-sensor | ✓ | 71.0s | 2422 |
| leg-ccma | ✓ | 69.5s | 2388 |
| leg-msa | ✓ | 69.2s | 2363 |
| leg-popia-breach | ✓ | 70.0s | 2392 |
| mkt-brand | ✓ | 66.1s | 2021 |
| mkt-gtm | ✓ | 80.3s | 2393 |
| mkt-popia | ✓ | 91.8s | 2388 |
| orch-brief | ✓ | 30.4s | 1051 |
| orch-scope | ✓ | 49.4s | 1606 |
| sales-discovery | ✓ | 69.2s | 2422 |
| sales-pipeline | ✓ | 122.2s | 2437 |
qwen3:14b — 18/19 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 177.3s | 2326 |
| data-warehouse | ✓ | 186.3s | 2386 |
| fin-cashflow | ✓ | 186.8s | 2424 |
| fin-model | ✓ | 184.1s | 2439 |
| fin-vat | ✓ | 161.9s | 2177 |
| health-icd10 | ✓ | 181.2s | 2360 |
| health-readmit | ✓ | 174.4s | 2253 |
| iot-mcu | ✗ | 183.7s | 2418 |
| iot-sensor | ✓ | 186.7s | 2419 |
| leg-ccma | ✓ | 179.1s | 2332 |
| leg-msa | ✓ | 167.6s | 2183 |
| leg-popia-breach | ✓ | 93.9s | 1392 |
| mkt-brand | ✓ | 73.7s | 1146 |
| mkt-gtm | ✓ | 191.4s | 2382 |
| mkt-popia | ✓ | 95.7s | 1395 |
| orch-brief | ✓ | 99.0s | 1274 |
| orch-scope | ✓ | 81.1s | 1076 |
| sales-discovery | ✓ | 167.7s | 2238 |
| sales-pipeline | ✓ | 186.8s | 2426 |
qwen3:8b — 19/19 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 87.0s | 2030 |
| data-warehouse | ✓ | 103.3s | 2302 |
| fin-cashflow | ✓ | 108.3s | 2424 |
| fin-model | ✓ | 106.8s | 2439 |
| fin-vat | ✓ | 106.6s | 2417 |
| health-icd10 | ✓ | 106.2s | 2381 |
| health-readmit | ✓ | 108.0s | 2384 |
| iot-mcu | ✓ | 106.7s | 2418 |
| iot-sensor | ✓ | 108.1s | 2419 |
| leg-ccma | ✓ | 96.4s | 2190 |
| leg-msa | ✓ | 99.3s | 2224 |
| leg-popia-breach | ✓ | 69.0s | 1677 |
| mkt-brand | ✓ | 35.0s | 1003 |
| mkt-gtm | ✓ | 110.8s | 2382 |
| mkt-popia | ✓ | 66.5s | 1610 |
| orch-brief | ✓ | 48.5s | 1109 |
| orch-scope | ✓ | 40.2s | 949 |
| sales-discovery | ✓ | 81.2s | 1934 |
| sales-pipeline | ✓ | 87.8s | 2043 |