February 2020 Work Progress Report: MOAC Delisting, Monitoring System Revealed
March 3, 2020
MOAC delisting, new 2CryptoCalc features, Ethereum ASIC’s news, Updated stratum protocols, 2Miners monitoring system revealed.
MOAC Delisting
We’ve stopped MOAC pools on February 3rd due to the network problems. All the situation was covered in our post MOAC Network is Under Attack. We’ve observed multiple denial-of-service attacks on the network. Blockchain was split many times. Unfortunately, MOAC blockchain developers have done nothing in nearly one month so we have decided to delist MOAC coin from the pool.
2CryptoCalc Mining Calculator Updates
Mining Profitability Calculator 2CryptoCalc is becoming better everyday. This month we’ve added such features:
GPU Memory check. If GPU is not able to mine a specific coin, this coin is not shown in the list;
Specific GPU and algorithm pages;
We are working on GPU ranking page, new design and many more awesome features.
2Miners Pool Monitoring System
Our CTO Erik Van Daal has written amazing post Monitoring for Health. He thoroughly explains many problems encountered by the mining pools and how do we monitor them. Blockchain splits, DDOS attacks, Orphan blocks and more problems are described in this article.
We’ve successfully tested newly-built Innosilicon A10 ASIC machines. We’ve upgraded our pools to support these devices.
Innosilicon A10 is 2 times more effective than previously available ASIC’s and GPU mining rigs. It consumes 750W providing 500MH/s hashrate on Ethash mining algorithm. Check Innosilicon A10 Specs, Profitability and Setup Guide.
Nicehash and ASIC Stratum Protocol Updates
We have updated our pools to support native NiceHash and ASIC-compatible Stratum protocols. This allows our miners to be more efficient both using Nicehash rented hashpower and ASIC’s. Why is this important? Read here NiceHash and ASIC-compatible Stratum Protocols on 2Miners.
Miner since 2017, the 2Miners pool co-founder. Became interested in cryptocurrencies at the dawn of the latest bull run and bought his first graphics cards. After having built and set up a few mining rigs, he realized that existing mining pools didn’t satisfy him – that’s how the idea of creating the 2Miners pool was born.
John published a series of articles about the basics of cryptocurrency mining. He gave valuable tips on how to buy ASICs and GPUs from abroad and then shared their mining performance. On the website, you can find early articles about cryptocurrency mining, blockchain in general and mining pool operation principles. The readers gave positive feedback on John’s practical guides, such as building an Ethereum mining rig and Nvidia 1080ti overclocking.
Hello, Erik Van Daal, CTO of 2Miners here. I thought it could be interesting to talk about how we ensure the smooth operation of our pools for our miners and what infrastructure we have in place.
What Is a Mining Pool
As you may know, 2Miners are a quite intensive operation: we have almost 40 pools, most of which are specialized software on dedicated physical servers located in three locations around the world.
A moment of clarification: each pool is a combination of four essential components:
A cryptocurrency node that hosts the blockchain, submits new blocks found by the pool miners to the network and processes the payouts to the miners.
The specialized custom-written pool software that accepts and verifies miner solutions, distributes new computing work and analyzes miner statistics.
A high-performance database that accumulates all submitted shares, stores the miner-related statistics and every other relevant data.
The Web frontend displays the miner data, graphs, payout history and so on.
As you may guess, any of these four components can face different problems and if any of those fails, it will affect the operation of the pool. How the Mining Pool Works: PPLNS vs. SOLO.
Most Common Mining Pool Problems
The way cryptocurrencies work is by no means error-prone and all kinds of bad situations may occur. Here are the most common ones that we face more or less on a regular basis:
There is a network split causing the cryptocurrency node to mine on a wrong chain causing rejected blocks and the loss of profit for our miners;
Due to some bug in the cryptocurrency node the mining process is affected or interrupted;
There are issues with the network connection between our servers in different data centers around the world (we currently host our servers in Europe, Asia, and America);
There is a DDoS attack running against our servers by some malicious attackers who try to disrupt the operations;
The disk or RAM space on one of our servers is low or there is a hardware component fault so there is a risk of losing some data;
A ghost of Vitalik Buterin is trying to steal some blocks from our ETH pool (okay, just kidding about this one).
We have designated people that are reacting to threats and incidents 24/7/365. But how do those specialists know there’s a problem that needs to be solved?
Enter the Telegram. We use it extensively for our work-related chats, including problem monitoring. Specialized custom-developed bots watch every aspect of day-to-day pools operation and report to us any problems and irregularities. Not every warning means a real problem, so the DevOps team carefully checks each message and reacts to the ones that need attention.
What stuff do we actually monitor? Let me describe in more detail which aspects of the pools are being watched.
Zabbix Monitoring
Zabbix is an open-source powerful monitoring system that can be extended by custom plugins to watch over some custom aspects of each machine we have in operation. Zabbix watches the disk space, CPU and RAM usage, current traffic and network load, the health status of our web frontends and microservices. Also, we have a set of custom plugins that watch over the availability of mining ports, mining luck, and payment issues.
Each incident is tunneled over to a special private Telegram chat and then analyzed and acted upon, if needed.
It is worth noting that we try to keep the CPU load of our servers below 20% to allow for unexpected short spikes in usage and to provide a good mining experience for all of our users. If there is a steady increase in the CPU load that is not going away, it is a good reason to enable backup hardware infrastructure to mitigate the load.
Orphan Blocks Monitoring
Some Ethereum-based cryptocurrencies have a mechanism called “Orphans” for blocks that were mined with the correct solution, but the blockchain network has chosen to pick some other block mined by some other miner or pool because it was mined a few seconds earlier or so; the orphans are usually also rewarded but in smaller amounts. You could find more information on that in our post Orphan, Stale & Uncle Blocks in Bitcoin and Ethereum. In Bitcoin world, term “orphan” is usually used to indicate blocks that have a correct solution, but that were not accepted onto the main blockchain for various reasons (mined later than somebody else, or mined onto an orphaned chain); these are not rewarded.
Either way, an orphan appearing is a reason to do a quick check: are the nodes healthy or they are in the early stages of a split? Is network connectivity stable enough so the solutions are delivered to the rest of the blockchain network as fast as possible?
Blockchain Split Monitoring
So-called “network splits” happen when some network nodes disconnect from each other and start producing a distinct chain of blocks. Once they reconnect, the blockchain chooses one of the chains, making the entire another chain invalid (or orphaned). This is potentially bad for our miners as it means a potential loss of profit. This is why we try to detect these situations as early as possible and try to eliminate them. Chain split could happen accidentally or it could also be a result of a 51% attack.
A special monitoring bot takes various sources of information (our own nodes, some well-known explorers on the Web, some other pools’ data, etc) and estimates whether our cryptocurrency nodes are in the correct chain or not. If not, an alert is emitted so our engineers can verify it. It is worth mentioning that an official coin explorer can often be itself in a split, so we trust pools’ data more (after all, it’s the source of new blocks so they often have more accurate information).
Other Emergency Mining Pool Monitoring
The error conditions above are not nearly the exhaustive list of what can go wrong with a cryptocurrency mining pool. There are lots of times when due to programming bugs the cryptocurrency nodes report an error and stop working, requiring a restart. All of these cases are handled by a separate bot. So whenever something that’s not in the predefined list goes wrong, this bot will pick it up and report to the DevOps team. Then some issues are as easy as “restart the node”, and some require some thoughtful investigation sometimes resulting in a bug report to the cryptocurrency maintainers or even a pull request to fix the bug.
Miner’s Monitor
Last but not least, while all of the things mentioned above are private and used internally, we do have a Telegram bot that’s accessible for any miner: @Pool2MinersBot. It is quite a powerful bot that latches on the same backend as our internal tooling. Not only can it report to you whether one or more of the workers has gone online or offline, but it also notifies when your worker finds a block and provides a useful source of information for the statistics of your mining operations.
I believe a professional and well-thought process of monitoring existing resources is key to a successful mining pool business. While almost anyone with mediocre programming background could assemble a simple pool using the open-source software, making it handle a big load and do so in a performant manner, while watching for all the issues a pool may have, takes a team of highly skilled professionals, just like with anything else. We at 2Miners care about our users as our own income depends on how well we do so.
Miner since 2017, the 2Miners pool co-founder. Became interested in cryptocurrencies at the dawn of the latest bull run and bought his first graphics cards. After having built and set up a few mining rigs, he realized that existing mining pools didn’t satisfy him – that’s how the idea of creating the 2Miners pool was born.
John published a series of articles about the basics of cryptocurrency mining. He gave valuable tips on how to buy ASICs and GPUs from abroad and then shared their mining performance. On the website, you can find early articles about cryptocurrency mining, blockchain in general and mining pool operation principles. The readers gave positive feedback on John’s practical guides, such as building an Ethereum mining rig and Nvidia 1080ti overclocking.
