After an “insane amount of data” flooded the proof-of-stake chain, knocking validators out of consensus and grinding still block production, Solana stakeholders rushed to put the network back together Saturday night.
Bots thronged the popular NFT minting tool Candy Machine earlier Saturday with an unprecedented flood of inbound traffic: four million transaction requests and 100 gigabits of data per second – a network record, as per one Solana Foundation source.
This surge shoved validators out of the agreement for causes that are still unknown. At 4:32 p.m. EST, block production became impossible, and the network went dark. Validators restarted the cluster at slot 131973970 by 11:00 p.m. EST, coordinating through Solana’s Discord channels and a Google doc created by one of the validators.
Anatoly Yakovenko, the co-founder, who said he was traveling during much of the squabble, credited the validator community for spearheading mainnet recovery. He was chastised on Twitter on Saturday for allegedly going missing during a network outage.
Unlike the 17-hour outage that occurred in September, Saturday’s hard fork restart did not result in new-and-improved code populating across the validators. They simply resumed where the network left off seven hours before.
To brace for the restart, validators debated whether to incorporate code that would temporarily cease Candy Machine transactions. Some Discord users debated whether such a move constituted censorship. In any case, it would only be successful if two-thirds of validators concurred to cooperate. On Saturday night, it appeared that some did.
Other parts of the ecosystem reacted quickly to fortify their defenses. At 11:36 p.m. EST, Metaplex, a key steward of Solana NFT infrastructure and a close collaborator with Candy Machine, tweeted that it would soon implement a 0.01 SOL “botting penalty” to assist NFT projects in mitigating excessive traffic.
Phantom wallet and decentralized exchange are part of the Solana ecosystem. As RPC node providers slowly came back online, Mango Markets struggled to stay afloat.
The outage played a role in a bloody, albeit brief, drawdown in SOL markets. According to CoinGecko, Solana’s native token fell to a 24-hour low of $83.13 three hours into the outage before rising back toward $89.
According to a member of the foundation, Solana core developers have yet to determine what went wrong Saturday or how the apparent botting attack bypassed existing safeguards to stymie consensus.
