Methodologies of Large Scale Distributed Systems

Large scale Distributed systems are typically characterized by huge amount of data, lot of concurrent user, scalability requirements and throughput requirements such as latency etc. The major challenges in Large Scale Distributed Systems is that the platform had become significantly big and now its not able to cope up with the each of these requirements which are there in the systems. Also at this large scale it is difficult to have the development and testing practice as well. This technology is used by several companies like GIT, Hadoop etc. Large Scale System Architecture :

1. Architecture has to play a vital role in terms of significantly understanding the domain. It is very important to understand domains for the stake holder and product owners. Also they had to understand the kind of integrations with the platform which are going to be done in future.
2. Another important Aspect is about the security and compliance requirements of the platform and these are also the decisions which must be done right from the beginning of the projects so the development processes in the future will not get affected.

Distributive Systems :

• Now Let us first talk about the Distributive Systems. Distributed Systems contains multiple nodes that are physically separate but linked together using the network. Each of these nodes contains a small part of the distributed operating system software. For better understanding please refer to the article of Distributed Systems
• Large Distributed systems are very complex which means that in terms of fault tolerance (how much resilient your system).It means that did you have considered all possible cases when your system can crash and can recover from that.
• For the distributive System to work well we use the microservice architecture .You can read about the microservice architecture form the article. But the challenge is to have a microservice based architecture you should draw boundaries of microservices otherwise you might overengineer the system and you might break it into unnatural boundaries and the system does not work cohesively together in a natural condition. The other side of microservice architecture that we consists about is Cap Theorem.

The boundaries in the microservices must be clear.

CAP Theorem :

• Cap theorem states that you can have all the three aspects of Consistency, Availability and partitioning. You can have only two things out of those three. Now you should be very clear as per your domain requirements that which two you want to choose among these three aspects.

Choose any two out of these three aspects

Message Queue : Message Queuesare great like some microservices are publishing some messages and some microservices are consuming the messages and doing the flow but the challenge that you must think here before going to microservice architecture is that is the order of messages. If you do not care about the order of messages then its great you can store messages without the order of messages. One more important thing that comes into the flow is the Event Sourcing. Event Sourcing : Event sourcing is the great pattern where you can have immutable systems. If we can have models where we can consider everything to be a stream of events over the time and we are just processing the events one after the other and we are also keeping track of these events then you can take advantage of immutable architecture. Immutable means we can always playback the messages that we have stored to arrive at the latest state. It means at the time of deployments and migrations it is very easy for you to go back and forth and it also accounts of data corruption which generally happens when there is exception is handled. Note – Event Sourcing and Message Queues will go hand in hand and they help to make system resilient on the large scale. Deployment Methodology :

• You cannot have a single team which is doing all things in one place you must have to consider splitting up you team into small cross functional team. You must have small teams who are constantly developing there parts and developing their microservice and interacting with other microservice which are developed by others.
• Also one thing to mention here that these things are driven by organizations like Uber, Netflix etc. These Organizations have great teams with amazing skill set with them. So the thing is that you should always play by your team strength and not by what ideal team would be.

Single Team doing things in one place

Small teams constantly developing there parts/microservice

Security and TDD (Test Driven Development) : The development in the team has to secure the coding practices and developing system where data in motion and data at rest are encrypted according to the compliance and regulatory framework. TDD (Test Driven Development) is about developing code and test case simultaneously so that you can test each abstraction of your particular code with right testcases which you have developed. By this you are getting feedback while you are developing that all is going as you planned rather than waiting till the development is done.