The risk for decentralized system is not a damage for a single node, but the entire network corruption. A blockchain doesn’t have any centralized server or repository, then it’s a key factor for maintaining network to keep the state consistent in each decentralized nodes. Hence the consensus mechanism is essential to make a blockchain reliable in the decentralized manner.
With this consensus, many people can use a variety of blockchain safely and it made a blockchain more popular.

In this post, we’ll learn the blockchain consensus for both public and enterprise blockchain (also known as, private blockchain, permissioned blockchain) by picking up each examples of those, Bitcoin and IBFT (Geth, Quorum).
These are not all consensus algorithms, however, you might understand each respective background, ideas, and standpoint.

Bitcoin Consensus – PoW

First, we see how the cryptocurrency (Bitcoin) is securely communicated and protected with decentralized manner. Bitcoin relies on the following Proof-of-Work (PoW) consensus.

In the Bitcoin blockchain, each block was explored by a person called a miner. A miner picks up multiple transactions, and seals these transactions as a block, and finally this block is added in a chain of blocks.
Each block has a block header (about which I describe details later) and a list of transactions.

The consensus is realized using the block header as follows.

Each block header includes the transaction hash (called Merkle root (tree)) which is generated using a list of transactions. If you add one character (even a space, a period, and so on) to some transaction in a list, this hash will be totally different. To make things more simple, this hash is representing a list of transactions.
Each block header also includes the hash of a previous block. This hash is also totally different, when the content of previous block is changed.

Each miner is having a key pair, i.e, a pair of a private key and a public key. (Only this miner can get his own private key, and his public key can be retrieved and used for every persons.) Using this private key, a miner signs this block header (including transaction’s hash and previous block’s hash) and create a new signature, e.g, a new hash of a block. This generated hash is also used in the next block as “previous block’s hash”. A chain of block is realized by this consecutive connections of each blocks. (See the below illustrated.)

Each block hash (signature) can be verified using a public key. (The public key can be recovered by the signature.) Then, if some transaction is tampered by a malicious user, all subsequent blocks should be changed using a new private key.

In order to keep an eligible state of a chain, Bitcoin mechanism also introduces a new parameter called nonce in block header.
In Bitcoin, the block hash should start with some predefined number of consecutive zeros. For instance, if 19 zeros are required in the hash, the block hash should be like this. (See Bitcoin Explorer for the real hash examples.) :

000000000000000000011206f10af1db...

In order to get this hashed value (signature), a miner should change and find a nonce (which can take an arbitrary string) in the block header. As you can imagine, this needs much of computations and is not a easy task for miners.
In Bitcoin, in order to keep the average time (approximately 10 minutes) for mining, the parameter called difficulty is also used in the block header.

Consequently, a miner should sign the header value including all these parameters (Merkel root, nonce, difficulty, and so on) to meet the requirement, in which the hash must be started with zeros. Once a miner luckily find a hash, the block is added into a blockchain in all nodes ! (Congrats !)

I illustrate this chain of flow in the following picture.

Note : In this post, we focus on the creation process of each “block”, however, each “transaction” (such as, “A gives 1000 BTC”, “B gains 1000 BTC”, …) is also signed by each participants and broadcasted.
See the original paper – “Bitcoin: A Peer-to-Peer Electronic Cash System” (Satoshi Nakamoto) for transaction chains.

By this mechanism, if a malicious user tries to tamper a chain of blocks, he will also need some average time for mining each blocks in the chain. Then, during his mining, a new block will be found and added in a chain by another miner. As a result, tampering is not realistic in the actual blockchain network. (Furthermore, a chain is distributed (copied) in every nodes in Bitcoin network. It’s also not realistic for 51 % of attacks.)
On the contrary, everybody can quickly verify chain using public keys and identify whether it’s correct or not.
This is the reason why this consensus is called Proof-of-“Work” (PoW).

I think you have a last question : “How the miner is motivated for doing such a hard work ? Is it voluntary ?”
The mining is incentivized by a reward of BTC. If a miner finds a hash (succeeded for exploration), he gets some bitcoins (BTC) for a reward. To gain this reward, a miner consumes a lot of computing resources (energy) for mining task.

Note : The bitcoins (BTC) is increasing by every mining. In Bitcoin platform, this mined reward halves every 210,000 blocks, and the supply is limited to 21 million bitcoins.

In this mechanism, multiple miners could find nonce and add each blocks synchronously. This is called a fork (accidental fork), and in Bitcoin, it’s resolved as more deeper (longer) chain wins against alternatives in a chain.

PoW is tolerant, but it also has several disadvantages, such as needing high energy or taking a long time for sealing a block. It will not be a problem because a public blockchain, such as cryptocurrency and so on, assumes the social trust and has a ton of anonymous users.
However, it won’t suit for a private blockchain, which is often used in the enterprise application.

IBFT (Istanbul Byzantine Fault Tolerance) – PoA

For a private blockchain, it assumes the enterprise use and PoW consensus doesn’t work, because it’s costly. Furthermore, there’s no way to incentivize to seal blocks for miners. (PoW will only work behind the enough computing capacity.)
For one of examples for a private blockchain consensus, here I show you IBFT (Istanbul Byzantine Fault Tolerance), which is used in Quorum protocol on Azure Blockchain Service.

Note : In Quorum implementation, you can choose any of 3 consensus protocols, Clique, IBFT, and RAFT. Quorum on Azure Blockchain Service uses IBFT, and currently you cannot change this protocol in Azure Blockchain Service.

IBFT is the modification of Practical Byzantine Fault Tolerance (PBFT). This type of consensus is not appropriate for a public blockchain. (Later I’ll show you the reason why.)
First, I’ll show you Byzantine fault tolerance (BFT) which solves famous Byzantine generals’ problem in short.

Byzantine fault tolerance was introduced in the work of Robert Shostak, Marshall Pease, and Leslie Lamport. (See this paper.) In this post, I use the example and diagram same as in this paper.

Now we assume that a number of generals reside in different locations and one of these is a commander.
Among these generals, they can send pairwise message one another. However, a general might be a traitor (for example, captured by an enemy), and he might disturb our command intentionally. For instance, a commander sends a message “RETREAT” to all generals, but some general (a traitor) might falsely send “ATTACK” to other generals.
Now we (our team) want to get one consensus in this situation.

As you can see below, among 3 or less generals, we cannot determine a consensus for this problem.

Whereas, if there exist 4 generals in the network, honest generals can decide a correct command by a majority of consensus.

Even when a commander is a traitor and lies to Lieutenant 1 in order to trick him, they all will gain the same value with the majority consensus. (See the following illustrated.)

Marshall Pease has generalized this idea by any positive integer F (> 0) in his work. If there exist F faulty nodes (i.e, traitors, also known as “Byzantine nodes”), at least N = 3F + 1 nodes are required for a consensus by the majority of 2F + 1.
Hence the number of nodes (generals) should be >= 4 to get a consensus in the network.
Shortly, both PBFT and IBFT include this idea to get a common consensus.

In order to prevent a false proposal and invalid blocks, IBFT (and PBFT) also introduces a 3-phase commitment.
If some proposer has submitted a new block request, each nodes determine whether they should agree this proposal. If each of them have agreed, each of them will broadcast messages. Then if each of them have received 2F + 1 agreement, each of them will enter into the ready state (PREPARED state).

Here I describe the state transitions in a round. (See here for details.) :

• “NEW ROUND” state : In this state, only a proposer broadcasts a new block request (“PRE-PREPARE” message) to all nodes.
• “PRE-PREPARED” state : If each of nodes receives above message and the request is valid, each of nodes enters into this state. Then each of nodes broadcasts “PREPARE” message to all nodes.
• “PREPARED” state : If each of them receives valid 2F + 1 “PREPARE” messages, each of nodes enters into this state. Then each of nodes broadcasts “COMMIT” message to all nodes.
• “COMMITTED” state : If each of nodes receives valid 2F + 1 “COMMIT” messages, each of nodes enters into this state. Then each of nodes inserts a proposed block into their own chain.
• “FINAL COMMITTED” : If the insertion succeeds, each of nodes enters into this state.

In each round, one of them (nodes) becomes a proposer by round-robin fashion.
Each round is changed by a timer and it’s also initiated by 2F + 1 majority. (i.e, Each nodes send “ROUND CHANGE” message in the same interval, and enter into “NEW ROUND” state by 2F + 1 majority.)
For this reason, there’s no fork in IBFT consensus unlike Bitcoin or other blockchain consensus. (Only one proposer in each round)

Note : This interval and the fashion (round-robin or sticky) are configurable in IBFT, but it’s not configurable in Azure Blockchain Service now.

One of disadvantages of BFT is so many message interactions. When the number of nodes increases, the required number of messages will be rapidly increased. For this reason, this mechanism is not applicable for a public blockchain.

In order to maintain network and avoid this overheads, only pre-approved nodes (a subset of nodes), named validator, can attend this process in IBFT. A consensus mechanism by permissioned validators (including the mechanism for maintaining validators) is called Proof-of-Authority (PoA). (For this reason, a private blockchain is sometimes called “permissioned blockchain”.)

Note : In public PoA Network, the validators (authorities) are also incentivized by rewards, then the network is maintained by eligible validators (authorities).
Here we discuss PoA in a private blockchain, not in a public blockchain.

As I mentioned above, the number of validators is also recommended to be 4 or much more in PBFT. As you can see, a large number of validators will result into more tolerant for the faults or security breaches, but much more costly. There’s a trade-off between security and computing cost (performance).

IBFT maintains a list of validators by a voting mechanism, which is inspired by Clique, one of PoA (Proof-of-Authority) implementations. In this voting mechanism, the network can add/drop a validator node by agreement of majority floor(N/2) + 1 amongst N validator nodes. Once some validator node is not eligible for a consensus, soon this node is dropped from a list of validators. (As I described in my previous post, you cannot use this voting mechanism in Azure Blockchain Service. Instead, you should invite new members in Azure Blockchain Service.)

Note : Here I don’t describe details about Clique, but Clique uses existing ethash header (beneficiary, nonce, and extraData) in an Ethereum block in order to keep a list of validators and a voting mechanism. As a result, Clique is implemented with a minimum code changes of Ethereum implementation. IBFT protocol also comes from this idea.
See here for the specification of Clique. (Clique doesn’t take BFT consensus, instead it takes its own “eventual” consensus approach.)

In my previous post, I have also explained about a private transaction (including a private smart contract), which can only be read and used between some specific users (participants) in Quorum transaction. When a user creates a private transaction, the transaction is encrypted using participant’s public key. Then each participants can decrypt using their own private key. Other users outside this participants’ party cannot decrypt, because they don’t have a key for decryption.
As I described above in Bitcoin consensus (see “Merkle root”), block validation includes a step to confirm hash of transactions. However, a private transaction is not included in the hash for this validation. (Strictly speaking, the database between public transactions and private transactions are separated.) As a result, the block validation is skipped for a private transaction in Quorum.

Here I showed a brief outline of blockchain consensus for both public blockchain and private blockchain, using Bitcoin and Quorum (or Geth) IBFT.
Next I focus on the decentralized identity technology, which is built on a decentralized ledger (blockchain) and a decentralized storage (IPFS) in the background.

Reference :

Istanbul Byzantine Fault Tolerance :
https://github.com/ethereum/EIPs/issues/650