FiredancerResearch · August 2026
One transaction,
two different states
Two programs keeping the same blockchain record should agree. In an isolated test network, they saved different data after processing the same transactions.
01 · The result
One shared record, two different results
Solana is a blockchain: a shared record maintained by many computers. Those computers run validator software to check and execute operations. Agave and Firedancer are two independently developed versions of that software.
An operation, called a transaction, asks programs to read or change data. Transactions are grouped into a block. Starting with the same data and processing the same block should leave both programs with the same record.
In the August 9 test, that rule broke. Both programs reported that the transaction succeeded, but they saved different data. Solana stores this data in records called accounts, which can hold program data as well as balances.
The disagreement carried into the next block. This is what consensus divergence means here: the programs processed the same history but no longer agreed on the resulting record.
Changing one memory-size calculation made Firedancer agree with Agave again. To understand why, the investigation had to follow a much smaller problem: data being written past the space reserved for it.
02 · August 3
The number that didn't add up
Before a program runs, Firedancer prepares the data it will read. One piece of code calculates how much memory to reserve. Another copies the data into that space. The reserved area is called an input frame.
The size calculation forgot one part of the input. The copying code did not: it wrote that extra data too, extending past the end of the reserved area.
On August 3, a targeted test measured 2,304 bytes written beyond the allocation. The transaction was valid and completed successfully. No failed transaction or crash was needed to expose the mismatch.
The omitted input and the first tested version
The investigation began in Firedancer v1.1.2's transaction runtime, the code that executes transactions. It compared the size calculation with the serializer, which arranges account and instruction data into the byte layout a program expects.
The calculation included loaded account data and room for accounts to grow. It omitted the Instructions sysvar, a special account created by the runtime to describe the transaction's instructions. Its data was not charged to the loaded-account total, but the serializer still copied it into the frame.
Adding the omitted allowance removed the overrun. So far, the test covered only part of the transaction runtime. Before moving to a complete validator, there was another question: what was using the memory beyond the frame?
03 · Following the data
The parent still needed that memory
A running program can ask another program to do some work, then continue when that work finishes. Solana calls this a cross-program invocation. Here, the paused call is the parent, and the call doing the nested work is the child.
The parent still needs its input when it resumes. But some of that input had spilled into memory reserved for the next call. Preparing the child's input overwrote it. When the parent continued, it read different data from the data it had started with.
Parent is paused
Child prepares its input
Schematic, not to scale. The parent's data was already beyond its allocation when the child reused the adjacent region.
The changed input did not stay inside temporary memory. The parent used it in later work and saved a different account value. Leaving out the child call preserved the original value; correcting the allocation did too.
What the nested-call tests established
This larger case extended 11,312 bytes past the frame boundary, a separate measurement from the initial 2,304-byte overrun. A debugger confirmed that the overlapping region held parent input that would still be read.
The earliest test showed a byte becoming zero. Follow-up tests with two different source values showed that the replacement followed the supplied data. After the child returned, the parent read the changed byte, used it in a later call, and saved a different account value.
04 · The checks that changed the investigation
Not every difference belonged to the bug
The first comparison with Agave could not isolate the bug. One feature was enabled that should not have been, and one account had a different owner: the program allowed to modify its data. Either mismatch could change the result independently of the memory error.
With the feature settings, account ownership and starting state aligned, the repeated comparison was consistent. Agave preserved the original value. Vulnerable Firedancer changed it. Patched Firedancer agreed with Agave.
Another check removed a setup shortcut and let the program create its input through ordinary execution. The result held. These were still small test environments, called harnesses, that exercised selected parts of the software. The next question was whether complete validators would behave the same way.
An earlier path that did not establish an impact
One route reached temporary memory used while loading programs. The relevant bytes were rebuilt before use, so that route did not demonstrate native code execution. The parent input, which was read after being overwritten, remained the useful observation.
05 · August 6–8
Getting it through real validators
The next test used an isolated group of validators, called a cluster. Agave assembled a block of transactions and sent it over the network. Full Firedancer received that block and executed it independently. Both were complete validator programs, not just selected functions in a test.
On August 6, Agave accepted the transaction through its normal network interface and produced a block. Reprocessing its saved transaction history reproduced its result.
Running the full Firedancer configuration required more memory than the original machine had, so this part of the test moved to a larger AWS instance. Original and corrected versions of Firedancer were built for comparison.
On August 8, Firedancer received the block through its normal network-processing path and saved different data from Agave. The next block carried that disagreement forward. The result repeated across 30 freshly started Firedancer processes.
06 · August 9
The same block, replayed
The final comparison changed the program, not the block. Original and corrected Firedancer builds processed exactly the same recorded network data. Here is what they saved:
The comparison checked both the saved account data and a bank hash: a compact fingerprint calculated from the resulting data and block history. Different fingerprints showed that the two programs no longer agreed on that record.
| Validator | Saved account data | Bank hash |
|---|---|---|
| Agave 4.1.1 | a5 a5 01 00 a5 01 | ECuZSrpu… |
| Firedancer 1.1.3 | a5 c3 01 00 00 00 | E6pwPvS2… |
| Firedancer + fix | a5 a5 01 00 a5 01 | ECuZSrpu… |
Recorded values at slot 1738. Each pair of characters in the account column is one byte. Hashes are abbreviated here; the comparison used their complete values.
What was held constant in the comparison
A slot is a numbered opportunity to produce a block. Slots 1738 and 1739 held the test transaction and its next block. Both Firedancer builds processed the exact same network fragments, called shreds.
At slot 1738, the starting parent hash, proof-of-history hash and signature count matched. The proof-of-history hash records the ordered history. Matching these inputs helped isolate the difference to execution: only the vulnerable run disagreed on the output account, accounts checksum and bank hash.
Each block's fingerprint includes the fingerprint of the block before it. That links the history together. Because the two programs carried different fingerprints into the next block, processing more transactions did not simply erase the disagreement:
Agave and patched Firedancer
Vulnerable Firedancer
A and B represent the different bank hashes at slot 1738. The following block, slot 1739, also ended with different bank hashes.
A second check left out the nested call that overwrote the parent's input. Even the original Firedancer then agreed with Agave, for both blocks. That connected the disagreement to the memory overlap rather than to an unrelated difference between the programs.
Versions and test conditions
- Reference
- Agave 4.1.1, producing the block and reference account state.
- Vulnerable follower
- Full Firedancer v1.1.3, commit
fbf5a1a42679, built with GCC 13.4.0 and run with its 49-tile sandboxed pipeline. - Corrected follower
- The same revision with only the missing Instructions-sysvar allowance added.
- Configuration
- Execution feature settings matched the relevant mainnet settings checked during the investigation. Local paths, network addresses, genesis validation and diagnostic capture were configured for the isolated cluster.
- Repetition
- The August 8 aggregate covers 30 fresh vulnerable full-validator processes. The August 9 comparison adds replay of identical shreds with the original and corrected binaries.
- Network path
- Agave accepted the signed transaction through QUIC, its normal transaction interface. Deployed program code and accounts created through signed transactions were used. Full Firedancer received the leader's shreds through its network and replay pipeline.
- Hardware
- The build machine had about 31 GiB of memory. The full configuration needed 328 GiB and ran 49 pipeline workers, called tiles. The follower test used an AWS instance with 64 vCPUs and 512 GiB of RAM.
07 · Disclosure
The fix shipped the next day
I submitted the report on August 9 with the validator results and reproduction material. The correction shipped the next day, August 10, in v1.1.4.
The fix reserves space for the input that the size calculation had forgotten. The data now fits inside its own allocation, so preparing the nested call no longer overwrites it. In the corrected test build, the same block produced the same saved data and fingerprints as Agave.
+ FD_SYSVAR_INSTRUCTIONS_FOOTPRINTThe team awarded the finding a $25,000 bounty. Thank you to the Firedancer team for fixing the issue so quickly, and for allowing me to disclose the finding and publish this article.