Replication in Distributed systems
DRY code vs Redundancy:
One of the most famous idioms of programming is “DRY” — Don’t repeat yourself. The idea is pretty straightforward, don’t write same lines of code again and again, rather abstract it and use it multiple times.
But over the years of my career, I have learnt and realized that it makes more sense to repeat yourself at the risk of over-engineering. And when it comes to distributed system it makes much more sense to have duplicate data which can actually benefit to whole system.
Why we need replication:
There are many benefits of replication in distributed system:
High availability:
If one of the node of a large scale distributed system fails, the whole system should not go down. With replication we have replica nodes which are in same state as of the dead node. Hence the traffic of dead node can be sent to any of the other replica and system will continue to work smoothly.
Reduced latency:
Replication helps in reducing the network latency of data queries by having replica nodes in multiple geographic regions. If a person from Asia is querying something in North America, network latency adds to the query processing latency which can be avoided by having a replica of same data in Asia (read CDN)
Read scalability:
Since we have multiple replicas having same data, we can read from any of them. Hence the read traffic can be distributed between multiple nodes and response time is much faster.
Network interruptions:
System works even under network faults.
Maintaining state:
Let’s say that we copied a node and created a replica one. But later on, we change something in the original node — perhaps we change its state, some value that it contains, or some internal behavior. What would happen to the redundant node? Well…nothing!
If we created just a simple redundant node, there is nothing in place within our system that will ensure that the redundant node will 1) know that something changed and 2) will update itself or be updated by someone else to have the correct state.
Replication attempts to solve this problem, because it ensures that all replicas of a node will be identical to one another, and will match the original node that they were replicated from. There are some ways in which we can sync up the original node and replica nodes.
Synchronous way:
In this approach, data is sent to original node and replicas and when its written on every replica and original leader, the write is termed as successful.
Asynchronous way:
In this approach, write is considered successful as soon as its written on leader. After that, data is sent to replica nodes. This leads to eventual consistency in system.
Semi-Synchronous way:
Synchronous approach can have a lot of latency as writing to each node will take a lot of time. While asynchronous approach is risky if leader fails just after write and replica nodes are yet to update. Hence there is semi-synchronous approach, wherein write is considered successful when its written on original node and some of the replica nodes.
Replica strategy:
Leader based replication topology:
One of the Node is designated as the leader (master or primary) while other replicas are known as followers (read replicas, slaves, secondaries or hot standbys). When client sends a write request, it gets written on Master node. Follower nodes copies data from Leader node and maintains the reliability. When client makes a read request, they can read it from either master/ follower nodes based on the configuration of system. Leader based replication are the most famous and used replication topology as they are easy to understand and implement.
Multileader replication topology:
Leader based replication approach has one drawback — there is only one leader and all writes must go through it. If clients can’t connect to it, write stops which hampers the availability of the system. We can solve this by having multiple leaders in the system, all having their follower nodes. In Master-Master application, client sends write query to one of the leader which sends stream of data to other nodes and children nodes to maintain reliability. Here system is available even when one whole data center goes down, as leader from other data-centers are available to do write operations.
Leaderless replication topology:
In Leader based approaches writes wont work if leader/leaders are unavailable until the failovers are recovered. In leaderless application there are no leaders, so client sends write request to all nodes and if some of these node confirm writes, the write operation is considered successful.
The exact numbers of node on which write/read should be implemented before considering it successful is based on the concept of Quorum.
For example, if there are 10 replicas and if at least 6 of them accepted the write, and during read operation say at least 6 of them responded. This guarantees that out of 6 that we read at least 1 of them was write successful and have latest information.
For an optimal configuration: writes + reads > number of replicas.
Outages:
Leader Outage:
When leader goes down, a new leader is elected among the follower nodes and write requests are sent to new leader. When old leader is up, it becomes follower of new leader node.
Follower Outage:
After restart, follower node connects to leader and requests all the data changes occurred during the downtime and maintains the consistency.
Replication is small and one of the most important part of large scale distributed systems which help maintain the reliability and availability of the system.
If you liked this post, please clap, share, comment and recommend.
If you want to see more back-end related stuff don’t forget to follow me on Medium
References:
Base Ds publication on Medium.com
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems Book by Martin Kleppmann