The task
Superteam Ukraine opened a brief with one question: how do Solana validators order transactions inside a block. By fee, by arrival, or with no rule at all, and does it differ between clients. The terms were strict: at least 6 validators and 3 client versions, mainnet only, every measurement after the Alpenglow upgrade, a spend cap in code, keys only through the environment, one command to run, raw data with signatures and slots, and a separate “what went wrong” section in the report.
The hard part: between our backend and the leader’s scheduler sit public RPCs, QUIC connections and, with Jito and BAM, one more node. We see when we sent and where the transaction ended up. What happened on the way, we do not.
Why, and what the idea was
Instead of reading someone else’s scheduler code we decided to measure behaviour from the outside. Our probes: a 1-lamport transfer to ourselves with different priority fees. We send them into the 4-slot window of a chosen leader, then read the block and see which came first. If the dearer one lands first although sent later, the leader ranks by fee. If the one sent first lands first regardless of price, the leader works in arrival order. If neither holds, it is noise.
Two decisions set the honesty of the result. First: the rules by which series 4 returns a verdict of “fee”, “no fee rule” or “undetermined” went into ANALYSIS_PLAN.md and were committed before the first probe; the git history shows the order. Second: for series 5 the plan’s hash was written to mainnet in a Memo transaction before any series 5 code existed.
One decision was revised along the way. We first wanted to spot Jito leaders by transfers to tip accounts in their blocks. Such transfers turned up in every block, Firedancer included, because anyone can send a transfer to a tip account. We switched to Jito’s public validator list.
How it went
| When | What happened |
|---|---|
| Series 1 | 30 runs targeted by client version, sequential sends in ascending, descending and random order. Fees up to 60 lamports. The first 9 runs predate the slot-clock fix |
| Series 2 | 67 runs, nine fixed leaders, one per client class. Burst and sequential sends in turn, fees raised a hundredfold, to 6,000 lamports |
| Series 3 | 39 runs, burst only, to the seven frequent leaders, because the block-level test lacked blocks per class |
| Series 4, same night | 477 runs: two probes per run, a coin flip decides who goes first, gap 0, 5, 20, 50 or 150 ms, separate payers or one shared write-locked account. Rules committed before the start; an hour and a half in, the decisive cells were extended by 20 runs each |
| Morning | Full verification: 785 blocks re-fetched with getBlock, all 1,328 probes at their recorded places. Docker checked from a clean clone on the server, README and REPORT written, repository made public |
| A week later | Submission. The same day the series 5 plan was committed, its SHA-256 written to mainnet in a Memo transaction, and series 5 started against four client classes as separate processes |
Tools and method
Python code, no frameworks: survey.py collects the leader schedule, client versions and the Jito and BAM flags; sender.py builds the probes, keeps a local slot clock and sends into the window; budget.py keeps a spend ledger with a file lock, so two parallel series share one 0.05 SOL cap; analyze.py and report.py find the slot, the index in the block and the leader that actually produced it, then count the statistics with an exact sign test and a Holm correction.
Probes went through two public RPCs at once, re-sent every 150 ms. Signing ahead is impossible: a blockhash lives about 150 slots and the needed leader’s window may be minutes away. So the sender sleeps until 10 slots before the window, takes a fresh blockhash, signs, and starts sending 3 slots early.
docker compose up needs no key and spends nothing: it runs the tests, rebuilds the tables from the raw data, re-reads 10 random runs through getBlock and does a mainnet dry run without sending. Keys come only from files named in the environment; the repository holds none.
Result
The answer in three lines. Transactions that reach a leader tens of milliseconds apart land in arrival order, whatever the client: 4.3.2 Jito 17 of 18 blocks, Firedancer 12 of 12, BAM 11 of 13. Transactions that arrive together are sorted by fee at BAM leaders: the dearer one first in 21 of 23 runs with separate accounts and 25 of 29 with a shared one, across 13-14 different leaders, p < 0.001 with the Holm correction. Firedancer and Jito-Agave without BAM lean towards fee but do not pass the threshold, and we call that “undetermined” instead of stretching the result.
Scale, honestly: 613 runs, 1,498 probes, 1,328 landed in blocks, 673 of them in the targeted window, the rest with neighbouring leaders. The analysis attributes every probe to the leader that actually produced the block. Reserved under the cap: 0.0113 SOL of the allowed 0.05.
The report has 14 “what went wrong” entries: a lagging RPC clock, expired signatures, fees too small in series 1, two series that hit the same window nine times, and a “random” order whose seed was fixed at zero.
What we do not know yet: how the brief will rate the work, and what series 5 will show about which fee measure the scheduler sorts by. We will add it when it comes.
Mistakes and what we took from them
