SN103 asks a good question — which combination of off-the-shelf LoRA adapters actually finishes a job — and has built unusually honest machinery to answer it. But recomputing its own published grades from its own published scores shows that 56% of what separates one miner's payout from another is output-token count, and that the ten-place ladder paying 90% of a day's emission to first place spans less than one standard error of its own measurement.
The pitch is narrow and, unusually for a subnet, falsifiable. One pinned base model (Qwen3-8B at a fixed revision). One frozen pool of 30 public LoRA adapters. Miners submit a JSON recipe — which adapters, at what weights, merged by which of seven methods, compressed to which rank — and nothing else. No model weights, no code, no hosted inference. An evaluation engine reconstructs the merge deterministically, serves it, and runs a fixed agent against 1,350 hidden problem instances plus 100 out-of-distribution ones. The best package takes the throne; the ladder pays ten places.
Everything that turns evidence into a number is in the public repository. The engine that runs it is not. That split is the subject's central tension, and most of what follows falls out of it.
This analysis is built from three sources that can each be checked independently: the
Bittensor chain at block 8,976,825, the live console at
capcomp.ai (which embeds the full scored record
for runs 410–421), and the
protocol
repository at commit 2878e59. Every derived number below was recomputed
from the published per-candidate scores; the reproduction is at the end.
The live picture
SN103 is not vapourware. At the sampled block the chain carries 100 registered UIDs, the console shows run 421 at 47% elapsed, and run 419 paid a full ten-place ladder whose weights match the documented shares to five decimal places — 0.899992, 0.049989, 0.029984, 0.009995, 0.004990, then five tail places at 0.000992 each. Miners are real: run 419 took 64 submissions, measured 39, and paid 10.
The two weight-setters are the problem. UID 0 is the subnet owner's own hotkey, holding 1,019,483 α — 89.6% of all stake behind a validator permit. UID 1 is a large multi-subnet delegate registered on 119 subnets, holding 117,994 α. They submitted bit-identical weight vectors: 0.8999958634376526 for UID 83, and so on down the ladder, matching to the last float.
Two independent measurements on different GPUs would not agree to seventeen significant
figures — the repository says so itself, noting that six of the seven merge methods run
an SVD and "an SVD is not bitwise reproducible across devices." Identical vectors mean
both validators derived their weights from the same source: the operator's signed
reports. That is exactly what the subnet's own endpoint mode is designed to
do, and it is legitimate. It also means that at present, one party's engine
determines 100% of SN103's emission, and the "validators measure for themselves"
claim in the README describes a mode nobody is currently running.
The mechanism
Every paid candidate gets one number, the grade. It is documented as three terms: quality 50%, improvement over the base model 40%, cost 10%. Quality is itself a weighted blend, headed by end-to-end completion at 55%. The README concludes that end-to-end completion is "roughly two-thirds of what decides rank."
That is true of the coefficients. It is not true of the outcome, because a weight only matters in proportion to how much the thing it weighs actually varies. On this arena the quality terms score around 0.14 and barely move; the cost and size terms score around 0.95 and move a great deal. Reconstructing run 419's champion grade from its published components — it reproduces to six decimal places, 0.273874 — gives this:
Run 419's champion grade, by term
Grade 0.273874 for UID 83. The 40%-weighted improvement term contributes 7% of it; the 10%-weighted cost term contributes 37%, and is already pinned at its ceiling of 1.0.
| Term | Weight | Value | Adds | Share of the grade |
|---|---|---|---|---|
| Quality | 0.50 | 0.3077 | 0.15386 | |
| Cost (tokens) | 0.10 | 1.0000 | 0.10000 | |
| Improvement | 0.40 | 0.0500 | 0.02002 |
Improvement is (e2e − reference) / (1 − reference). With the base model at
0.11185 and the champion at 0.15630, that is 0.0500 — so the term the subnet describes
as "the network's definition of composition added value" is nearly all headroom the
champion never touched.
Levels are only half of it. What decides rank — and rank is what pays — is spread. Taking all 296 measured candidates the console publishes, multiplying each term's total coefficient in the grade by that term's standard deviation across the field, and normalising, gives the honest weighting:
Designed weight against realised influence
Each term's coefficient in the grade (through every channel it enters) versus its share of the grade's actual spread across 296 measured candidates. Token efficiency is designed to carry 14% and carries 56%.
Coefficients: end-to-end 0.7254 (it enters through quality and again through improvement), token efficiency 0.150, stage balance 0.075, out-of-distribution 0.050, retention 0.025, artifact size 0.025. Spread is population standard deviation over the 296 candidates with published component scores.
Stage balance — the term the README calls "what separates a package that is competent everywhere from one that is excellent at one stage and broken at another" — accounts for 2.8% of what separates one miner's payout from another's.
The lever
Token efficiency is min(1, 3000 ÷ tokens-per-completed-instance). That
denominator is deliberately charged against completions rather than attempts, which
correctly makes cheap failure expensive — the term is really
3000 × completion ÷ mean tokens per instance, so finishing more work raises
it too. The design is careful. The calibration is not.
Inverting the formula on the 239 candidates below the ceiling recovers each package's mean output length: 470 tokens per instance on average, standard deviation 69, range 367 to 781. Crucially, verbosity and completion are essentially uncorrelated (r = −0.162). Length is an independent dial. And it is the dial the grade responds to most:
At the median candidate — 13.9% completion, 451 output tokens per instance — trimming output by 10% with completion held fixed raises the grade by 0.0110. The entire ten-place ladder of run 419 spans 0.00536. So that single edit is worth twice the whole paid field, and is equivalent to gaining 1.5 percentage points of end-to-end completion — an 11% relative improvement on a task where the base model scores 11%.
A second lever needs no measurement at all, and the field has already stripped it bare.
Artifact efficiency is 1 − bytes ÷ 500 MB, and size is a pure function of the
recipe's declared output_rank against the pinned model's shapes — 36 layers,
seven target modules, bfloat16. Rank 8 is 41.6 MB; rank 64 is 333.0 MB. Compressing from
one to the other adds 0.0146 to the grade, nearly three times the entire
spread of run 419's paid ladder, before a single instance is served.
What each output rank costs, before anything is measured
Artifact size is fixed by the rank alone, so the grade contribution is known at recipe time. Rank 128 is schema-valid and can never be paid — it reconstructs past the 500 MB hard gate.
| output_rank | Artifact | artifact_efficiency | Grade vs rank 8 | Field |
|---|
To justify keeping rank-64 capacity a package would have to buy 2.0 percentage points of end-to-end completion with it — a 14% relative gain on a task where the field sits at 14%. Nobody is trying: 232 of the 296 measured candidates ship at exactly 41.7 MB, and so does every one of the top 30. The subnet only ever measures rank-8 compressions of its merges, because the scoring function prices the capacity out before the arena gets to ask whether it helped.
The arena
The default workflow, lora_merger_logic_v1, scores twelve axes drawn from
~3,193 logic puzzles and ~2,944 competitive-programming problems, the latter graded by
executing the submitted program. It is a genuinely well-built harness: nothing is judged
by a model, instances are a pure function of a hidden seed, and code is run in a
network-isolated container behind hard resource limits.
The trouble is that Qwen3-8B at pass@1 greedy cannot do most of these tasks. Across the candidates with published per-axis breakdowns, two axes score exactly zero for every single package, and five more sit between 1% and 3% — one to three correct answers out of roughly a hundred items.
Median per-axis score across the measured field
Twelve declared capability axes. The one the field scores highest on is format compliance, which measures whether the answer is shaped correctly — not whether it is right.
Stage balance is the geometric mean of these twelve, with zeros floored at 0.001. The two permanently dead axes therefore drag the term 2.08× below what the ten live axes alone would give — every candidate equally, so it costs the ranking information without buying discrimination. The per-axis floors that would gate this are set to 0.0 for eleven of the twelve axes, so that hard gate passes unconditionally.
This is the honest core of the subnet's own scepticism, and to its credit the repository says most of it out loud. What the repository does not currently support is the headline claim. The single-adapter reference — the bar that answers "does composing adapters beat just using the best one?" — was removed to save GPU time. The code says so plainly, and names the consequence:
So the only permanent reference left is the untouched base model at 0.11185, and the entry gate is 0.02 above it. In run 419, all 39 measured candidates cleared it. A gate that rejects nobody is not a gate.
The noise floor
Run 419 paid 90% of the day's miner emission to first place and 0.5% to fifth. Here is what separated them:
Run 419's paid ladder
Grade is drawn on a window from 0.2680 to 0.2745 — the full field spans 0.00536, so an unzoomed axis would render all ten bars identical. Payout is drawn from 0 to 90%.
| Rank | UID | Grade | Grade window 0.2680–0.2745 | Payout | Payout 0–90% of the run |
|---|
Adjacent gaps run from 0.002005 (1st→2nd) down to 0.000004 (8th→9th). The standard error of one candidate's own end-to-end measurement over 1,350 instances at p = 0.156 is 0.0099, which propagates to roughly 0.0072 of grade — 1.3× the entire spread of the paid field. The field is compared on a shared instance draw, and pairing shrinks the error on a difference below that; but the ranking that assigns 90% versus 5% is taken on raw grade order with no significance test between neighbours. The paired bootstrap in the code tests each candidate against the base model, not against the candidate above it.
The throne bar has the same problem, one order of magnitude smaller. The dethrone margin was cut from 0.002 to 0.001 to 0.0005 over two days in late August. Every throne decision since run 413 except one has been settled in the fifth decimal place:
How close each throne decision was
Distance between the run's best grade and the bar it had to clear, on a log scale. The shaded band is the measurement's own standard error.
Run 419's leader fell short by 0.000013 and still took 90% of the run, because payment was decoupled from the throne on 28 August ("Pay what cleared the gates, not what beat the throne"). Runs 416–420 crowned nobody; the reigning grade of 0.2733875 has stood since run 415. The constants file is candid about this: "nothing here establishes that the spread is below 0.0005. If a throne starts changing hands on reruns rather than on new submissions, that is the thing to measure."
Trust
The verification story is more thoughtful than most subnets manage, and it is worth being
precise about where it stops. In endpoint mode — the mode both live validators
appear to be running — a validator does not fetch a weight vector and relay it. It fetches
every candidate's signed evaluation report, checks the signature against its own allow-list,
recomputes each grade from the report's own scores, rebuilds the ladder itself, checks the
champion's UID against the chain, and re-scores a sampled run from the engine's published
traces. Anything that fails is burned with the validator's own stake. That is a real design.
Three things it cannot reach:
api.capcomp.ai in a signed request body and are held privately
until the run closes. That fixed a real copying problem, and cost the only independent
record of who submitted what. A dropped submission leaves no trace anyone can point to.
spot_check_run
returns pass when a disclosure is missing or the engine is unreachable; the seed-root
check returns pass when fewer than two runs are disclosed. An engine that publishes
reports but no disclosures is paid with no re-scoring performed at all.
And the engine itself is not public. The README's "Two repositories" table links to
Capcomp-AI/lora-merger-engine; that repository returns 404 and the
organisation has exactly one public repo. The split is defensible — the operator holds the
seed root and wallet material — but as shipped, the run loop, the candidate scheduler, the
anti-copy check and the instance sampler are all unreviewable, and the anti-copy check has
rejected 49 submissions in the console's cached history.
Engineering
The 605-test suite is real and it passes. Installed clean on a CPU-only host, it runs
602 passed, 3 failed — and all three failures are tests that require an actual CUDA device
without being marked gpu, so they are excluded by no filter. The declared CI
command is pytest -m "not gpu and not docker and not chain" on
ubuntu-latest, which has no GPU. CI would be red today.
It has not had the chance to be. GitHub Actions shows six workflow runs on this repository,
all between 13 and 15 August, four of them started by hand. The push trigger
declared in ci.yml has never fired. 97 commits have landed on
main since the last CI run, and they include every consensus-critical
change in the current mechanism: the burn share going to zero, the dethrone margin being
quartered twice, the pay ladder being decoupled from the throne, and the whole of endpoint
mode.
That is a shame, because the code is good. The merge engine is genuinely deterministic and tested for it, the sandbox runs candidate-written Python behind a container with no network and a fresh interpreter per batch, the layering is enforced by a test, and the docstrings are the most candid I have read in this ecosystem — they routinely record what a mechanism gives up and what has not been measured. Several of the findings below are stated more plainly in the repository's own comments than anywhere else.
Findings
Token efficiency carries 56% of the grade's realised spread against a designed 14%, because it varies 10× more than the completion terms do. Verbosity is uncorrelated with completion (r = −0.162), so it is an independently exploitable dial. At the median, a 10% token cut is worth 2× the entire paid ladder.
Fix: normalise each term by its observed dispersion before weighting, or
cap the cost term's contribution to the grade rather than its own value. The
REFERENCE_OUTPUT_TOKENS budget of 3,000 compounds it by landing almost exactly
on the field's operating point — packages spend ~3,300 tokens per completed instance — which
is the steepest possible place to put a threshold. Nineteen percent of candidates are
already pinned at the 1.0 ceiling, where the term stops discriminating entirely.
Run 419's ten paid places span 0.00536 of grade; the standard error induced on the grade by end-to-end sampling alone is ≈0.0072. Ranks 8 and 9 were separated by 0.000004. Ranking is by raw grade order — the paired bootstrap tests candidates against the base model, never against the neighbour whose 18×-larger share they are taking.
Fix: flatten the ladder, or require a paired significance test between adjacent ranks and pool the shares of statistically tied places.
Both weight-setting hotkeys submitted bit-identical vectors, which independent GPU
measurement cannot produce. The owner holds 89.6% of permitted stake. Nobody is currently
running the local mode the README's trust argument rests on — unsurprising,
since it demands 4 × 32 GB GPUs on a subnet with a 4,864 τ market cap.
"Does a merge beat the best single adapter?" cannot be answered while no single-adapter reference is measured. The base model at 0.11185 is the only permanent reference left, and all 39 measured candidates cleared the 0.02 margin over it in run 419. The repository names this as "the first thing to restore."
Artifact efficiency costs 0.0146 of grade to go from rank 8 to rank 64 — 2.7× the whole paid ladder — so keeping any capacity above the minimum requires buying 2.0 points of end-to-end completion with it. 232 of 296 measured candidates, and every one of the top 30, ship at rank 8. The subnet is only ever measuring rank-8 compressions of its merges, which is a strong claim about the answer baked into the scoring of the question.
Fix: the same dispersion normalisation as the token term, or a flat
credit for clearing a size threshold rather than a linear gradient down to zero. Related:
output_rank: 128 passes the recipe schema and reconstructs to 666 MB against
a 500 MB hard gate, so it is an allowed value that can never be paid.
spot_check_run returns (True, …) when a disclosure is missing or
unreachable, and check_draw_was_not_re_rolled returns pass with fewer than two
disclosed runs. Both are documented choices — an outage must not read as fraud — but the
composition means an engine that publishes reports and no disclosures is paid with zero
re-scoring. A staleness bound on disclosures would close it without punishing outages.
Six Actions runs total, none since 15 August, none ever triggered by a push. Three tests
in test_validator_own_preflight.py call the preflight with a CUDA device
without stubbing torch.cuda.is_available, and are not marked gpu.
arrow_maze and web_of_lies score exactly 0.0 for every candidate,
floored at ε in the geometric mean and depressing stage balance 2.08× uniformly. Only
format_compliance carries a non-zero floor (0.5), so
gate_stage_floors skips the other eleven and
BASE_RETENTION_FLOOR = 0.0 makes the retention gate unconditional.
Thirty adapters, of which four carry "license": "unstated" despite a
documented licence review, and several are visibly incidental to the arena —
Qwen3-8B-test, Qwen3-8B-Knowledge_test,
cot-oracle-ablation-stride5-3layers,
terminal-agent-fix-git-overfit-rank256-epoch49. Exactly one
(code-generation-v1) targets the arena's strongest live axis.
The website still describes a 50% burn and a 55/40/5 split; the deployed burn share is 0.0
and the split is 50/40/10. The README states that the qualified score's smaller terms are
"hard gates as well as scored terms"; docs/miner.md correctly states that
"retention is measured and scored, never gated." min_compute.yml still says
"there is no lighter mode" for validators two days after endpoint mode shipped. The
miner guide is the accurate document.
In fairness
Most of the above is calibration, not fraud, and the distinction matters. Several things here are better than the norm:
The single change that would do the most for SN103's credibility is also the cheapest: put the thirty pool adapters through one measurement run and restore the best of them as a permanent reference. It costs one run of card time and converts the subnet's headline claim from an assumption back into a measurement.
Method
Every figure above comes from one of three checkable sources. Nothing was taken on the project's word.
# Chain state — netuid, owner, stake, and the two weight vectors
python -c "import bittensor as bt; m=bt.Subtensor('finney').metagraph(103, lite=False); \
print(m.name, m.owner_hotkey, m.n); print([r.nonzero() for r in m.W])"
# On-chain commitments (the legacy submission route, retired 25 Aug)
python -c "import bittensor as bt; print(bt.Subtensor('finney').get_all_commitments(netuid=103))"
# The scored record: capcomp.ai embeds runs 410-421 as JSON in the page
curl -s https://www.capcomp.ai/ | grep -o '"qualified_score":[0-9.]*' | wc -l
# The engine is up but its read endpoints are not public
curl -s https://api.capcomp.ai/health
# {"status":"up","resubmission_limit":3,"reveal_lag_runs":2}
# The test suite, on a CPU-only host
pip install -e . && pytest -q -m "not gpu and not docker and not chain"
# 602 passed, 3 failed (all three require a real CUDA device)
The grade reconstruction is the load-bearing step: for each of the 296 published candidates,
grade = 0.5·qualified + 0.4·(e2e − 0.11185)/(1 − 0.11185) + 0.1·token_efficiency.
Applied to the run-419 champion this returns 0.273874, matching the console's published grade
to six decimal places, which is what licenses using the same formula for the decomposition
and the sensitivity arithmetic.
That reconstruction, the suite result above and the rest of the checks behind this page are re-run rather than remembered, and they are published as they stand on the testing and results page — which also carries the measurement harness itself: how a candidate is served, on what hardware, at which of the engine's settings, and what every measurement taken so far scored against what the network is currently paying.