Reasons of network issues on January 28th
Yesterday, January 28, 2022 around 15:00 UTC ErgoDEX users started reporting issues with Yoroi wallet in terms of syncing with the network. In this post we are going to break down the events that led to the incident.
On Ergo ecosystem
Ergo P2P network is a set of nodes, operating under the same protocol in order to maintain consistency of the Ergo blockchain. It consists of Ergo Nodes operating in various modes: full nodes, full pruned nodes, light nodes etc. All these configurations have different capabilities to serve the network’s needs. But none of them are able to provide a convenient API allowing someone to query on-chain data. Therefore there is a special type of Ergo node, the Indexer Node. The Indexer Node is what many of you know as Explorer.
Ergo Blockchain Explorer is essentially an indexer node and most of dApps and wallets in Ergo ecosystem heavily rely on it in order to read on-chain data. Although Ergo Explorer is freely distributed and anyone can run it most of the dApps and wallets, it all still relies on the official Ergo Explorer instance at http://api.ergoplaform.com. The Official Ergo Explorer by the time of the incident consisted of two replicas which the whole Ergo Explorer traffic was split between.
Friday network issues
Yesterday, 28.01.2022 around 01:00 UTC one of the two Ergo Explorer servers had started experiencing issues and this caused a whole instance malfunctioning. The faulty node was plugged off by the load balancer and from that moment all the traffic was routed to a single Ergo Explorer instance that was left. During the day while Ergo team was solving the issue, that instance was handling the traffic pretty ok but as the load increased the response time also increased causing lags in many dApps and wallets including Yoroi.
The catalyst for the dramatic increase in Explorer traffic was the NETA listing — many users began trading the token loading Yoroi and ErgoDEX which in their turn loaded the Explorer even further. As a result the Explorer had to deny many requests in order to stay functional causing Yoroi, ErgoDEX and other dApps to get stuck for many users.
Incident liquidation and further steps
The incident with the faulty Explorer node was resolved at 5am UTC 29.01.2022 and the situation stabilized.
In order to eliminate similar incidents in future Ergo team decided to add one explorer replica.
We want to apologize for any inconvenience this event may have caused.
