What are some best practices for building applications at scale?
Great answer! Adding for his perspectives as well
The foundational element of scalability is resiliency and one also has to remember that every component of an application can, and will fail, so architecting for resiliency improves scalability by default. To increase resiliency, first start at the (cloud) infrastructure layer and use industry best-practices such as Multi-AZ (Availability Zone) and Multi-Region with Elastic Load Balancers in front of every component/(micro)service.
To enhance that, look at how to use new and emerging architectural patterns like microservices to assist with both performance as well as being able to recover from any outages/disruptions. Test this using Chaos Engineering and the OSS tools that assist with this. Once again, assume all services will fail at some point and how can you quickly recover from those disruptions.
Finally, use Observability tools/solutions to understand at a deep level of how your application/service behaves under different loads and scenarios.
There is no Done state to all of this, it will require a continuous effort and make sure you articulate that important point to Sr. Management at the onset.
By 'building applications at scale' I mean -
1. You have a large application to be built, with a large team as fast as possible.
2. You have a lot of applications to be built with a large team, and you are churning out applications at scale.
By 'building applications that scale' I mean the application is built for horizontal scaling. In other words, a 'cloud native application'.
May take on 'building applications at scale'.
1. Divide into micro applications. Such that each micro application should not need a team of more than 20 odd developers. Build teams which can work concurrently.
2. Obviously, micro applications got to interact, so interfaces must be defined clearly (becomes critically important as concurrent teams are working). Adopt API first approach. Define interfaces and interactions between the applications using API. You may want to use api blueprint (https://apiblueprint.org/) to define the interfaces and use the likes of apiary (https://apiary.io/) to host API mocks. This reduces friction between teams and requirements of communication are well documented. Inter-application seams become more seamless.
3. Needless to say, this kind of development essentially indicates a microservice-based architecture. Each application is developed as a microservice with a definite set of requirements. Development of such microservices can happen fast and concurrently. Advantage with microservices is also that deployment time reduces drastically.
4. Microservices for back-end is well-understood design pattern, however how about front-end. If it is one big application, it's difficult to divide UI development like microservices and still achieve a coherent user interface. It's a challenge. However, do check out design pattern of 'Micro Frontends'.
5. I am not a great fan of reusable components. If the requirement of the component in itself is reusable, then the component automatically becomes reusable. For instance, a component which establishes a connection to cache service. This is naturally reusable component and does not change service to service. Generally, when we try to build a re-usable component, we spend a lot of time in generalizing it so that it fits in various situations. Analysis becomes paralysis. Eventually, the component becomes heavy and buggy.
6. Heroku team had come out with 12 Factor application concept. Kevin Hoffman adds few more to it in 'Beyond the Twelve-Factor App' for building applications that scale on the cloud. Excellent suggestions. I believe with 12 Factor++ concept we can build applications that scale and also build applications at scale (by adopting API first, microservices, micro-frontend, and automation).
Thanks for your inputs, Someshwar. What I meant was applications working at scale (huge number of transactions per second).
Building applications at scale also requires careful thinking about how you can create reusable components. You end up building a lot of components if you don’t make reusable ones, and all of them require testing. Every time you create a component, you have to test it for the functionality, as well as for the non-functional aspects, like the scale security. You also have to put a lot of focus on the architecture of what you are building, because the traditional way of thinking may not work. When we are designing a solution, we often try to fit the end result into the technology we have, rather than defining our end result first and then fitting the technology to that. It has to be reverse-engineered. Otherwise, you will have to change the technology when your requirements change.
It is also important that you leverage the tools that are most relevant to that problem, rather than going with what you already know. If the architect has more experience with databases, for example, the solution ends up being inclined towards databases. That is a hindrance, so we have to remove our bias toward the tools we know when building applications for scale. We need to put the end result into perspective, establish what is required, architecturally build that, and then we can choose what technology will solve that particular problem, whether we have used it before or not.