Test Matrix 194 results
| Test | deepseek-r1:8b | gemma4:e2b | gemma4:e4b | qwen3:14b | qwen3:8b |
|---|
| data-dbt | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| data-warehouse | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| fin-cashflow | ✓ 5.0 | ✓ | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| fin-model | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| fin-vat | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| health-icd10 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| health-readmit | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| iot-mcu | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✗ | ✓ 5.0 |
| iot-sensor | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| leg-ccma | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| leg-msa | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| leg-popia-breach | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| mkt-brand | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| mkt-gtm | ✓ 5.0 | ✓ | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| mkt-popia | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| orch-brief | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| orch-scope | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| sales-discovery | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
| sales-pipeline | ✗ | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 | ✓ 5.0 |
Model Details
deepseek-r1:8b — 36/38 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 106.2s | 2383 |
| data-dbt | ✓ | 106.5s | 2383 |
| data-warehouse | ✓ | 107.5s | 2373 |
| data-warehouse | ✓ | 108.0s | 2373 |
| fin-cashflow | ✓ | 107.8s | 2411 |
| fin-cashflow | ✓ | 49.4s | -- |
| fin-model | ✓ | 76.8s | 1869 |
| fin-model | ✗ | 106.7s | 2426 |
| fin-vat | ✓ | 49.6s | 1324 |
| fin-vat | ✓ | 53.1s | 1388 |
| health-icd10 | ✓ | 106.3s | 2374 |
| health-icd10 | ✓ | 89.3s | 2048 |
| health-readmit | ✓ | 107.6s | 2371 |
| health-readmit | ✓ | 107.8s | 2371 |
| iot-mcu | ✓ | 106.7s | 2405 |
| iot-mcu | ✓ | 96.0s | 2205 |
| iot-sensor | ✓ | 107.7s | 2406 |
| iot-sensor | ✓ | 108.1s | 2406 |
| leg-ccma | ✓ | 106.3s | 2369 |
| leg-ccma | ✓ | 106.4s | 2369 |
| leg-msa | ✓ | 107.5s | 2373 |
| leg-msa | ✓ | 107.8s | 2373 |
| leg-popia-breach | ✓ | 106.3s | 2374 |
| leg-popia-breach | ✓ | 106.9s | 2374 |
| mkt-brand | ✓ | 64.9s | 1580 |
| mkt-brand | ✓ | 61.4s | 1510 |
| mkt-gtm | ✓ | 109.8s | 2369 |
| mkt-gtm | ✓ | 109.9s | 2369 |
| mkt-popia | ✓ | 63.0s | 1544 |
| mkt-popia | ✓ | 98.3s | 2214 |
| orch-brief | ✓ | 106.1s | 2208 |
| orch-brief | ✓ | 106.5s | 2208 |
| orch-scope | ✓ | 81.9s | 1746 |
| orch-scope | ✓ | 106.2s | 2198 |
| sales-discovery | ✓ | 106.3s | 2400 |
| sales-discovery | ✓ | 104.8s | 2365 |
| sales-pipeline | ✗ | 107.6s | 2413 |
| sales-pipeline | ✓ | 108.1s | 2413 |
gemma4:e2b — 42/42 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 38.7s | 2413 |
| data-dbt | ✓ | 38.6s | 2413 |
| data-warehouse | ✓ | 39.0s | 2400 |
| data-warehouse | ✓ | 39.0s | 2400 |
| fin-cashflow | ✓ | 131.2s | 2284 |
| fin-cashflow | ✓ | 112.7s | 2236 |
| fin-cashflow | ✓ | 36.9s | 2330 |
| fin-cashflow | ✓ | 32.1s | 2080 |
| fin-model | ✓ | 38.2s | 2423 |
| fin-model | ✓ | 38.6s | 2450 |
| fin-vat | ✓ | 22.8s | 1591 |
| fin-vat | ✓ | 24.9s | 1706 |
| health-icd10 | ✓ | 30.8s | 1977 |
| health-icd10 | ✓ | 31.4s | 2013 |
| health-readmit | ✓ | 37.6s | 2315 |
| health-readmit | ✓ | 38.6s | 2366 |
| iot-mcu | ✓ | 38.7s | 2423 |
| iot-mcu | ✓ | 32.4s | 2096 |
| iot-sensor | ✓ | 39.1s | 2422 |
| iot-sensor | ✓ | 39.2s | 2422 |
| leg-ccma | ✓ | 32.2s | 2051 |
| leg-ccma | ✓ | 30.5s | 1961 |
| leg-msa | ✓ | 31.3s | 1984 |
| leg-msa | ✓ | 31.7s | 2012 |
| leg-popia-breach | ✓ | 31.2s | 1963 |
| leg-popia-breach | ✓ | 29.2s | 1898 |
| mkt-brand | ✓ | 25.1s | 1672 |
| mkt-brand | ✓ | 24.3s | 1635 |
| mkt-gtm | ✓ | 84.9s | 2393 |
| mkt-gtm | ✓ | 55.4s | 2393 |
| mkt-gtm | ✓ | 42.4s | 2393 |
| mkt-gtm | ✓ | 49.1s | 2336 |
| mkt-popia | ✓ | 27.7s | 1819 |
| mkt-popia | ✓ | 32.1s | 2052 |
| orch-brief | ✓ | 19.3s | 1185 |
| orch-brief | ✓ | 18.1s | 1128 |
| orch-scope | ✓ | 22.3s | 1345 |
| orch-scope | ✓ | 21.7s | 1317 |
| sales-discovery | ✓ | 38.6s | 2422 |
| sales-discovery | ✓ | 38.5s | 2422 |
| sales-pipeline | ✓ | 36.0s | 2278 |
| sales-pipeline | ✓ | 33.8s | 2161 |
gemma4:e4b — 38/38 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 112.8s | 2413 |
| data-dbt | ✓ | 70.0s | 2413 |
| data-warehouse | ✓ | 119.0s | 2400 |
| data-warehouse | ✓ | 70.5s | 2400 |
| fin-cashflow | ✓ | 82.7s | 2439 |
| fin-cashflow | ✓ | 69.8s | 2439 |
| fin-model | ✓ | 135.4s | 2450 |
| fin-model | ✓ | 69.3s | 2450 |
| fin-vat | ✓ | 75.3s | 2300 |
| fin-vat | ✓ | 64.0s | 2278 |
| health-icd10 | ✓ | 70.3s | 2393 |
| health-icd10 | ✓ | 70.2s | 2393 |
| health-readmit | ✓ | 71.0s | 2390 |
| health-readmit | ✓ | 70.9s | 2390 |
| iot-mcu | ✓ | 72.9s | 2423 |
| iot-mcu | ✓ | 70.1s | 2423 |
| iot-sensor | ✓ | 71.0s | 2422 |
| iot-sensor | ✓ | 70.9s | 2422 |
| leg-ccma | ✓ | 69.5s | 2388 |
| leg-ccma | ✓ | 69.8s | 2388 |
| leg-msa | ✓ | 69.2s | 2363 |
| leg-msa | ✓ | 68.9s | 2348 |
| leg-popia-breach | ✓ | 70.0s | 2392 |
| leg-popia-breach | ✓ | 69.8s | 2392 |
| mkt-brand | ✓ | 66.1s | 2021 |
| mkt-brand | ✓ | 65.2s | 2273 |
| mkt-gtm | ✓ | 80.3s | 2393 |
| mkt-gtm | ✓ | 74.4s | 2393 |
| mkt-popia | ✓ | 91.8s | 2388 |
| mkt-popia | ✓ | 69.0s | 2388 |
| orch-brief | ✓ | 30.4s | 1051 |
| orch-brief | ✓ | 47.9s | 1563 |
| orch-scope | ✓ | 49.4s | 1606 |
| orch-scope | ✓ | 42.4s | 1402 |
| sales-discovery | ✓ | 69.2s | 2422 |
| sales-discovery | ✓ | 69.5s | 2422 |
| sales-pipeline | ✓ | 122.2s | 2437 |
| sales-pipeline | ✓ | 70.2s | 2437 |
qwen3:14b — 36/38 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 177.3s | 2326 |
| data-dbt | ✓ | 161.5s | 2149 |
| data-warehouse | ✓ | 186.3s | 2386 |
| data-warehouse | ✓ | 186.5s | 2386 |
| fin-cashflow | ✓ | 186.8s | 2424 |
| fin-cashflow | ✓ | 175.1s | 2290 |
| fin-model | ✓ | 184.1s | 2439 |
| fin-model | ✓ | 184.4s | 2439 |
| fin-vat | ✓ | 161.9s | 2177 |
| fin-vat | ✓ | 154.3s | 2091 |
| health-icd10 | ✓ | 181.2s | 2360 |
| health-icd10 | ✓ | 151.6s | 2031 |
| health-readmit | ✓ | 174.4s | 2253 |
| health-readmit | ✓ | 186.5s | 2384 |
| iot-mcu | ✗ | 183.7s | 2418 |
| iot-mcu | ✓ | 184.1s | 2418 |
| iot-sensor | ✓ | 186.7s | 2419 |
| iot-sensor | ✗ | 186.8s | 2419 |
| leg-ccma | ✓ | 179.1s | 2332 |
| leg-ccma | ✓ | 183.8s | 2382 |
| leg-msa | ✓ | 167.6s | 2183 |
| leg-msa | ✓ | 186.4s | 2386 |
| leg-popia-breach | ✓ | 93.9s | 1392 |
| leg-popia-breach | ✓ | 161.9s | 2146 |
| mkt-brand | ✓ | 73.7s | 1146 |
| mkt-brand | ✓ | 86.3s | 1286 |
| mkt-gtm | ✓ | 191.4s | 2382 |
| mkt-gtm | ✓ | 190.2s | 2362 |
| mkt-popia | ✓ | 95.7s | 1395 |
| mkt-popia | ✓ | 88.8s | 1316 |
| orch-brief | ✓ | 99.0s | 1274 |
| orch-brief | ✓ | 90.3s | 1174 |
| orch-scope | ✓ | 81.1s | 1076 |
| orch-scope | ✓ | 71.4s | 965 |
| sales-discovery | ✓ | 167.7s | 2238 |
| sales-discovery | ✓ | 181.8s | 2389 |
| sales-pipeline | ✓ | 186.8s | 2426 |
| sales-pipeline | ✓ | 186.8s | 2426 |
qwen3:8b — 36/38 passed
| Test | Pass | Latency | Tokens |
|---|
| data-dbt | ✓ | 87.0s | 2030 |
| data-dbt | ✓ | 71.8s | 1738 |
| data-warehouse | ✓ | 103.3s | 2302 |
| data-warehouse | ✓ | 107.9s | 2386 |
| fin-cashflow | ✓ | 108.3s | 2424 |
| fin-cashflow | ✗ | 108.3s | 2424 |
| fin-model | ✓ | 106.8s | 2439 |
| fin-model | ✗ | 106.6s | 2439 |
| fin-vat | ✓ | 106.6s | 2417 |
| fin-vat | ✓ | 85.3s | 2018 |
| health-icd10 | ✓ | 106.2s | 2381 |
| health-icd10 | ✓ | 100.3s | 2269 |
| health-readmit | ✓ | 108.0s | 2384 |
| health-readmit | ✓ | 106.0s | 2348 |
| iot-mcu | ✓ | 106.7s | 2418 |
| iot-mcu | ✓ | 106.7s | 2418 |
| iot-sensor | ✓ | 108.1s | 2419 |
| iot-sensor | ✓ | 108.1s | 2419 |
| leg-ccma | ✓ | 96.4s | 2190 |
| leg-ccma | ✓ | 90.2s | 2077 |
| leg-msa | ✓ | 99.3s | 2224 |
| leg-msa | ✓ | 101.6s | 2267 |
| leg-popia-breach | ✓ | 69.0s | 1677 |
| leg-popia-breach | ✓ | 66.9s | 1638 |
| mkt-brand | ✓ | 35.0s | 1003 |
| mkt-brand | ✓ | 33.0s | 964 |
| mkt-gtm | ✓ | 110.8s | 2382 |
| mkt-gtm | ✓ | 111.6s | 2382 |
| mkt-popia | ✓ | 66.5s | 1610 |
| mkt-popia | ✓ | 68.1s | 1641 |
| orch-brief | ✓ | 48.5s | 1109 |
| orch-brief | ✓ | 53.7s | 1209 |
| orch-scope | ✓ | 40.2s | 949 |
| orch-scope | ✓ | 42.6s | 996 |
| sales-discovery | ✓ | 81.2s | 1934 |
| sales-discovery | ✓ | 85.2s | 2009 |
| sales-pipeline | ✓ | 87.8s | 2043 |
| sales-pipeline | ✓ | 108.0s | 2426 |