Paxos (lamport_tcs98; lamport_sigact01) and its variants (lamport_msr05; camargos_podc07; yanhua_osdi08; sutra_srds11; moraru_sosp13; ongaro_atc14) provide reliable solutions to the problem of distributed consensus (fischer_jacm85). Thanks to their excellent fault-tolerance properties and proven consistency guarantees, these algorithms often underpin the replicated state machines (schneider_cs90) at the heart of many industrial cloud and distributed systems, e.g., Chubby (burrows_osdi06), CockroachDB (taft_sigmod20), and PaxosStore (zheng_vldb17).
Traditionally, the Paxos family of algorithms uses majority quorums, guaranteeing that any two sets containing the majority of nodes intersect, ensuring that previously decided values are not lost. Flexible Paxos (howard_opodis16) relaxes the requirement for intersecting quorums in Paxos, proving that quorum intersection is only required between phases, permitting disjoint quorums to be used within each phase. This result enabled subsequent algorithms to improve performance by adjusting quorums depending on the phase of the algorithm (ailijiang_tpds20; nawab_icmd18; muhammed_nsdi19; enes_eurosys20; eischer_popoc20; logdevice).
Paxos is usually implemented using Multi-Paxos (lamport_tcs98; lamport_sigact01), an optimisation that elects one node to be a leader. This single leader can then achieve distributed consensus in just one phase, but unfortunately also becomes a performance bottleneck.
Seeking to improve performance, a new family of leaderless consensus algorithms emerged, starting with Fast Paxos (lamport_msr05), which forms the basis for many subsequent algorithms including Generalized Paxos (lamport_msr04) and Egalitarian Paxos (moraru_sosp13). Paxos is based on the idea of rounds in which at most one value can be proposed. Fast Paxos introduced the notion of fast rounds where multiple values can be safely proposed in the same round. However, such fast rounds require stronger quorum intersection than classical rounds. Specifically, Paxos only requires that any two quorums intersect whereas Fast Paxos also requires that any quorum intersects with any two fast round quorums. Fast Paxos’ quorum intersection requirements can be satisfy by requiring fast round quorums to contain at least three quarters of nodes. Due to this additional quorum intersection requirement, Fast Paxos and its variants cannot directly benefit from Flexible Paxos.
In this paper we show that the approach of Flexible Paxos can be safely applied to consensus algorithms that rely on stronger quorum intersection by extending Flexible Paxos to Fast Paxos. The resulting algorithm, which we refer to as Fast Flexible Paxos, relaxes the quorum intersection requirements of Fast Paxos. Specifically, Fast Flexible Paxos proves that the only additional quorum intersection requirement is between phase-1 quorums and any pair of fast round phase-2 quorums.
Relaxed quorum intersection in Fast Flexible Paxos permits new performance tradeoffs by manipulation of the quorum systems. For example, reducing the size of fast round quorums reduces the contention in the algorithm. This may further improve overall performance on top of an improvement attained by a smaller quorum alone. For instance, we illustrate that the Fast Flexible Paxos with smaller fast quorum achieves up to 10% better latency than Fast Paxos in low conflict scenarios.
We begin by recapping how distributed consensus is currently solved by Paxos, Flexible Paxos and Fast Paxos. The relation between these algorithms is shown in Figure 1.
The Paxos algorithm distinguishes between two roles a node can take: a proposer and an acceptor. A proposer initiates a decision by executing Paxos using a round. Rounds are integers allocated to proposers, and each proposer must propose only one value in each round. A proposal is a pair of a round and a value. The protocol runs runs in two phases per round, each requiring a quorum of acceptors to proceed.
In phase-1, the proposer learns if a value was decided in any previous round by asking the acceptors to send the last proposal they voted for. Acceptors promise not to vote in any smaller round.
In phase-2, the proposer asks acceptors to vote for a value . If during phase-1 the proposer learned that a value might become decided, then it must use that value for . Provided that an acceptor has not promised otherwise, it updates its last proposal voted for and acknowledges that to the proposer. Once the proposer has completed phase-2, it will learn that is decided.
Paxos requires intersection between any two quorums. If denotes the set of quorums then this intersection requirement can be expressed as:
Consider a simple quorum system based solely on the cardinality of quorums. If denotes the cardinality of quorums in and denotes the number of acceptors, then we can express Paxos’ quorum intersection requirement (Eq. 1) as:
Paxos is often used to decide a sequence of values, where the instance of Paxos decides the value. Multi-Paxos improves the protocol’s performance by pre-executing the phase-1 of every instance by the same stable proposer, known as a leader. The leader can then decide each value in just two communication steps, compared to the four communication steps needed by Paxos.
2.2. Flexible Paxos
Flexible Paxos differentiates between the quorums for phase-1, and phase-2, . This approach allows the weakening of Paxos’ quorum intersection requirement (Eq. 1) to the following:
That is, quorum intersection is not required within each of the two phases. As previously, if denotes the cardinality of quorums in , then we can express Flexible Paxos’ quorum intersection requirement (Eq. 3) as:
In Multi-Paxos, phase-1 is executed rarely compared to phase-2 so applications can decrease the phase-2 quorum to improve performance at the cost of decreased fault-tolerance.
2.3. Fast Paxos
In Multi-Paxos, the leader is a bottleneck and other proposers must first send values to the leader, adding a communication step. Fast Paxos addresses these issues by allowing all proposers to propose values directly to the acceptors. Fast Paxos can thus decide a value in one phase, an optimal solution to distributed consensus (lamport_tr03).
Fast Paxos achieves this by introducing fast rounds, where it is safe for multiple values to be proposed in the same round. In a fast round, if the leader is free to propose any value in phase-2 then instead of proposing a specific value it proposes a special any value to the acceptors. Proposers can then send proposals directly to the acceptors and each acceptor will vote for the first proposal it receives as if it had been sent by the leader.
Fast Paxos needs stronger quorum intersection for fast rounds to ensure safety and progress in the case of conflicts. If denotes fast round quorums and denotes classic round quorums then Fast Paxos requires:111The original paper asserts that (a) any two quorums intersect and (b) any two fast quorums and any classic or fast quorum intersect. Since (b) already covers the intersection between two fast quorums as well as a fast and classic quorum, we have reduced (a) to any two classic quorums intersect.
In other words, Fast Paxos requires that any pair of classic round quorums intersect (Eq. 5), that any pair of fast round quorums intersect with any classic round quorum (Eq. 6) and that any three fast round quorums intersect (Eq. 7).
Fast Paxos suggests using or and to satisfy these requirements.222The former is sometimes written as and where is the number of faults which can be tolerated (junqueira_hotdep07). The larger quorums required by Fast Paxos have been shown to significantly decrease performance compared to Paxos (junqueira_hotdep07).
3. Fast Flexible Paxos
Following the approach of Flexible Paxos, we will differentiate between the quorums used for each phase of Fast Paxos.
Recall that quorum intersection is required between the two phases to ensure that a proposer learns in phase-1 any value which may be decided in phase-2. In phase-2, the proposer needs to pick a value from the highest round it learned during phase-1. In Fast Paxos, there may be multiple such values as acceptors may vote for different values during the phase-2 of fast rounds, requiring a proposer to determine which single value (if any) could be decided in previous rounds. Fast Flexible Paxos achieves this by ensuring that each phase-1 quorum intersects with any pair of fast round phase-2 quorums.
The quorum intersection requirements are the same regardless of whether phase-1 is for a fast or classic round. Therefore denotes the phase-1 quorums (fast or classic) whereas and denote the phase-2 quorums for classic and fast rounds respectively. The weakened intersection requirements for Fast Paxos are as follows:
In other words, we find that quorum intersection is only required between phase-1 quorums and phase-2 classic round quorums (Eq. 11) and between phase-1 quorums and any pair of phase-2 fast round quorums (Eq. 12). Note that quorum intersection is not required between phase-1 quorums, between phase-2 classic round quorums, or between phase-2 classic round quorums and phase-2 fast round quorums.
Fast Flexible Paxos must ensure that at most one value is decided. We can show this by proving the following two properties:
Property 1 ().
At most one value is decided per round.
If the round is classic, then at most one proposer can propose (and therefore decide) a value. If the round is fast, then at most one value will be decided as any two fast phase-2 quorums will intersect (Eq. 12). ∎
Property 2 ().
A proposer will only propose a value in a given round if no smaller round decided a different value.
Assume that a value is decided in round . Consider the next round () where a value is proposed. In phase-1, the proposer of round will ask the acceptors to promise not to vote in any smaller rounds and to reply with the last proposal they voted for.
Due to quorum intersection between the phase-1 and phase-2 (Eq. 11 & Eq. 12), at least one acceptor will reply to the proposer in round with value and round . This is because the acceptor must have voted for value in round before participating in round as it promises not to vote in any smaller rounds. The acceptor also cannot have voted in any round since as round is the first round after where a value is proposed. The proposer in round will propose the value as the proposer will not receive any proposals from rounds greater than . If the round is fast then the proposer in round may receive multiple values from round , however, the proposer will choose value since is decided.
Consider the next round () where a value is proposed. The value proposed in round must be value as only the value has been proposed since value was decided in round . By induction, we can see that for all rounds larger than , if a value is proposed then that value will be . ∎
Fast Flexible Paxos must also ensure liveness to solve distributed consensus, and in particular it must satisfy the following property:
Property 3 ().
Upon completion of phase-1, a proposer can determine at least one value which is safe to propose in phase-2.
Consider a proposer which has just completed phase-1 of round . A value is safe to propose in round only if the proposer knows that no smaller round has decided a different value (Property 2). If a proposer receives multiple proposals in phase-1 then it proposes the value with the greatest round (). However, if the round is a fast round the proposer may receive multiple values and so must determine which of the values (if any) could be decided in round . Note that a value could be decided in round only if there exists a phase-2 fast round quorum of acceptors which may have voted for the value in round .
For every pair of phase-2 fast round quorums, at least one acceptor which will reply to a proposer in phase-1 of round must also vote in both quorums if both quorums decide a value in round (Eq. 12). The acceptor will only vote for one value in round and thus will reply to the proposer with only one value. The proposer thus learns that the other value cannot have been decided in round by any quorum containing that acceptor.
Once the proposer has heard from a phase-1 quorum of acceptors, the proposer can safety eliminate either all or all but one of the values received with round . ∎
We adapted the Fast Paxos specification (lamport_msr05) to model check a formal specification of Fast Flexible Paxos using TLA+ (lamport_tla). Both our specification and model checking configurations are available online (ffpaxos_tla).
The weakened intersection requirements show that phase-1 of a fast round can use the same quorum as phase-1 of a classic round. Since the requirement of fast round quorums is stricter than classic round quorums then fast round quorums must be at least as large as classic round quorums.
For example, Fast Paxos suggests using and , but our relaxed intersection requirements demonstrate that a simple majority of acceptors is sufficient for phase-1 of fast rounds. Similarly, Fast Paxos also suggests using and again we observe that is conservative and only one third of acceptors are needed for phase-2 of classic rounds ().
More generally, by weakening the intersection requirements of Fast Paxos, we provide more flexibility to choose quorum systems and tradeoffs. In a stable system, phase-1 is rarely executed compared to phase-2 so we can decrease the size of our phase-2 quorums, fast and classic, provided we increase the size our phase-1 quorums. For example, a system of 11 acceptors could use phase-2 quorums of 7 acceptors for fast rounds and 3 acceptors for classic rounds, if it uses quorums of 9 acceptors for phase-1.
Note that the liveness of such a system does depend upon both phase-1 and phase-2 quorums. For example, we could minimize fast round phase-2 by using a simple majority for , but this would require all acceptors to start a new round.
6. Preliminary Evaluation
The main contribution of this paper is the observation that the quorum intersection requirements of Fast Paxos can safety be relaxed. We have also implemented Fast Flexible Paxos to illustrate the potential performance improvements this result enables, even with a simple quorum system based solely on quorum cardinality.
We evaluated Fast Flexible Paxos with the aforementioned quorum configuration (, , and ) using Paxi (ailijiang_icmd19) on AWS EC2 m5a.large VMs. We focused on two key aspects of Fast Flexible Paxos: latency and conflict reduction due to a smaller phase-2 fast round quorums. We compared our protocol against a Fast Paxos ( and ) baseline.
In Figure 1(a) we illustrate the performance of two protocols under a workload of 1400 requests/second with no conflicts. Smaller fast round quorums allowed Fast Flexible Paxos to reduce the average and median latency by 5—8% compared to Fast Paxos.
We also evaluated under conflicts by generating a workload with several clients racing to propose different commands for the same consensus instance. For this workload we generated a steady stream of operations with only small intervals between them. We also pre-assigned each operation to an instance to control the potential for conflicts. In about 10% of the cases, we assigned the same instances to two consecutive operations, creating a race condition between them. In this setup there are two possible outcomes for such races: one of the operations reaches the fast round quorum, causing the second to abort; or none of the operations reach the fast round quorum, causing entry into the conflict resolution phase. We then measured the conflict avoidance ratio to study the impact of the Fast Flexible Paxos on conflict handling.
We found that Fast Flexible Paxos entered the conflict recovery almost one-third as frequently as Fast Paxos due to the smaller fast quorum. However, the overall frequency of recovery phases increased substantially for both protocols as the throughput rises and the interval between the commands shrinks, as Figure 1(c) shows. Considering the overall performance in the conflict workload, Figure 1(b) shows that our Fast Flexible Paxos continues to maintain a roughly 5% latency advantage over Fast Paxos even under high load compared to our non-conflict experiment.
We believe Fast Flexible Paxos will enable further performance improvements if quorum systems are used that are not based solely on quorum cardinality (gifford_sosp79; garcia_jacm85; peleg_podc95; naor_siam98; junqueira_hotdep05). This has already proven to be the case for Flexible Paxos (ailijiang_tpds20; nawab_icmd18; muhammed_nsdi19; enes_eurosys20; eischer_popoc20; logdevice). In particular, Fast Flexible Paxos can benefit from the existing literature on Byzantine and Refined quorum systems (malkhi_dc98; guerraoui_podc07) as these quorum systems provide stronger quorum intersection.
Fast Paxos allows any proposer to decide a value in two communication steps in the absence of collisions. This is the optimal number of communication steps for distributed consensus. However, to achieve this it needs a stronger quorum intersection than Paxos and thus has not benefited from recent work on relaxing quorum intersection requirements.
Fast Flexible Paxos weakens Fast Paxos’ quorum intersection requirements by differentiating between the quorums used in each phase of the algorithm. We find that quorum intersection is only required between any phase-1 quorum and both (a) any phase-2 classic round quorum, and (b) any pair of phase-2 fast round quorums. This shows that the quorum systems used by Fast Paxos are conservative and that alternative quorum systems could be safely used.
More generally, we have proven that the approach of Flexible Paxos generalises to consensus algorithms beyond Paxos. We hope more consensus algorithms, particularly those which extend Fast Paxos such as Generalized Paxos (lamport_msr04), Egalitarian Paxos (moraru_sosp13), MDCC (kraska_eurosys13), Alvin (turcu_opodis14) and Caesar (arun_dsn17), adopt this approach to relax their quorum intersection requirements, giving applications greater flexibility to determine their performance and fault-tolerance tradeoffs.