[Paper Review] Evaluating Permissioned Blockchain Using Stochastic Modeling and Chaos Engineering

Shiv Sondhi, Sherif Saad+3 more|arXiv (Cornell University)|Jan 14, 2023

Blockchain Technology Applications and Security4 citations

TL;DR

This paper evaluates permissioned blockchain platforms using stochastic modeling and chaos engineering to assess consensus protocol performance under normal and faulty network conditions. It finds that while stochastic models offer efficient relative performance insights, actual platform architecture significantly impacts real-world results, showing that identical consensus protocols can perform very differently across platforms like Hyperledger Fabric and Tendermint.

ABSTRACT

Blockchain and distributed ledger technologies rely on distributed consensus algorithms. In recent years many consensus algorithms and protocols have been proposed; most of them are for permissioned blockchain networks. However, the performance of these algorithms is not well understood. This paper introduces an approach to evaluating consensus algorithms and blockchain platforms in a hostile network environment with the presence of byzantine and other network failures. The approach starts by using stochastic modeling to model the behaviors of consensus algorithms under different typical and faulty operational scenarios. Next, we implemented a blockchain application using different consensus protocols and tested their performance using chaos engineering techniques. To demonstrate our generic evaluation approach, we analyze the performance of four permissioned blockchain platforms and their consensus protocols. Our results showed that stochastic modeling is an inexpensive and efficient technique for analyzing consensus protocols. But they do not represent the actual performance of the consensus protocols in a production environment. Moreover, an experiment with chaos engineering indicates that if two different blockchain platforms use the same blockchain algorithm or protocol, we should not assume they will have similar performance. Therefore, It is also essential to consider the role of platform architecture and how the protocols are engineered in a given platform.

Motivation & Objective

• To evaluate the performance of permissioned blockchain platforms under normal and faulty network conditions.
• To compare the effectiveness of stochastic modeling versus real-world blockchain deployments in assessing consensus protocol behavior.
• To investigate how platform-specific architectural choices influence consensus protocol performance.
• To assess the reliability of stochastic models in predicting real-world blockchain system behavior under Byzantine and network failures.
• To demonstrate that identical consensus protocols can yield vastly different performance outcomes across different blockchain platforms.

Proposed method

• Developed stochastic models using probabilistic automata to simulate consensus behavior under various network faults, including delay, packet loss, and Byzantine failures.
• Implemented a lightweight blockchain application across four permissioned platforms—Hyperledger Fabric, Tendermint, Clique, and Raft—using their native consensus protocols.
• Applied chaos engineering by injecting controlled faults such as network delays and message corruption to evaluate system resilience and performance under stress.
• Measured key performance metrics: throughput, latency, and system load capacity under both normal and fault-introduced conditions.
• Used a standardized benchmarking approach to compare relative performance across platforms and protocols while isolating the impact of platform-specific design choices.
• Correlated results from stochastic models with real deployment outcomes to assess model accuracy and platform influence.

Experimental results

Research questions

• RQ1 How do stochastic models compare to real blockchain deployments in predicting consensus protocol performance under normal and faulty conditions?
• RQ2 To what extent do platform-specific architectural decisions affect the performance of identical consensus protocols across different blockchain platforms?
• RQ3 How do network faults such as delay and message corruption impact the throughput and latency of different consensus protocols?
• RQ4 Can stochastic modeling reliably rank consensus protocols by performance in a way that aligns with real-world deployment results?
• RQ5 What role does platform implementation play in determining the actual performance of a blockchain system, beyond the consensus algorithm itself?

Key findings

• Stochastic models provide a cost-effective and efficient method for analyzing consensus protocol behavior, showing relative performance trends, but do not accurately predict absolute performance in production environments.
• The performance of the same consensus protocol varies significantly across platforms: for example, PBFT performed worse in deployment than predicted by models, while Tendermint outperformed expectations.
• Tendermint and Clique demonstrated better throughput and fault tolerance than PBFT and Raft under load and fault conditions, with Clique maintaining stable performance even under corrupted messages.
• Platform architecture has a critical impact on performance: for instance, Tendermint's design caused higher latency than predicted, while Fabric's implementation required optimization to improve performance.
• The results show that identical consensus protocols (e.g., PBFT) can yield vastly different real-world performance depending on the underlying blockchain platform, invalidating assumptions of protocol-agnostic performance.
• Chaos engineering revealed that real-world system behavior under failure is not fully captured by models, emphasizing the need for platform-aware evaluation.