Real-World Containerization - Monolithic Applications, Microservices, Distributed Computing, Containerization, Point-of-Sale Software

Distributed computing and containerization is sweeping the IT landscape. Yet there remains large pockets of applications that are stuck in the world of single tenancy, monolithic architectures. One area where we are seeing significant challenges with the adoption of distributed computing and containerized workloads is in the area of retail point-of-sale systems.

A seminal paper came out in 2010. It described a new way of architecting distributed solutions. As the paper gained traction, containerization also started to become popular. DataBricks and Mesosphere launched as startups, with DataBricks releasing Spark and Mesosphere releasing Mesos, then DC/OS.

Figure 1: The paper that launched an industry

The 3 dominant players in this space are: (1) Mesosphere DC/OS; (2) Google Kubernetes; (3) Docker Data Center. Those are cluster managers/orchestrators, which are beyond the scope of this article.

The big outcome out of the Hindman paper is Mesos. His whitepaper focuses on the decoupling of the scheduler, giving you the flexibility to control how your containers are scheduled.

Figure 2: Mesos: Abstracting the scheduler

There are some good reasons why adoption for new distributed architecture and containerized is a challenge in the area of point-of-sale systems. Retailers can't afford downtime - ever. Even in my lifetime I've been told maybe twice that I cannot order something because the system is down.

Figure 3: You absolutely, positively must be able to take orders

Retailers always need to be able to take money from customers. That's why most point-of-sale systems run on-premises. Point-of-sale systems that depend on cloud connectivity are vulnerable due to networking woes. After all, many retail locations exist in the hinterlands, areas that could be considered rural and somewhat disconnected from the Internet.

Clearly, the world is becoming increasingly connected so the time is now to start thinking about moving to a cloud-based point-of-sale system.

Figure 4: Eventually, the cloud will be good enough to run most POS systems

This blog post addresses some of the challenges as well as some of the opportunities to move in the direction of distributed, containerized workloads. I work closely with some of the biggest retail brands in the world around this challenge, and here are some of my thoughts.

We can begin our discussion by viewing the diagram below, which is a rather simplistic version of what most retailers actually face. Many retailers have stores in the tens of thousands across the entire planet. What I want you to notice is that typically point-of-sale systems are on-premise, single tenant, monolithic applications.

Figure 5: Mostly thick client and on-premises

This typically means there is some large binary file that contained the application typically organized as a three or four tier architecture. In addition, there are various dependencies present on the VM in order for the monolithic app to function. In other words, the monolithic application references DLLs or external software packages that must be installed on the VM itself so that the application can run on that VM.

Figure 6: Challenges with Monolithic Architectures

• Large monolithic code bases can make it difficult to onboard new developers, due to the fact that they become increasingly complex and messy over time.
• The modularity of design breaks down as new features are added and changed.
• Despite the fact that IDE's are built to support this design, the code base can grow quite large and can slow the IDE down.
• Web-based, monolithic applications tend to suffer from GUI bloat, where the business logic and visual components grow over time and slow down the user experience, particularly when loading the browser components.
• Scaling is difficult. Typically, monolithic applications share a single data store. Even if you scale the application, each copy of application instance will access the same single store of data. The database quickly becomes the bottleneck as I/O traffic increases. In additions, caches become stale quickly and their value is compromised. You also have to scale the entire app, not individual tiers. If your business logic tier requires more compute resources, you have to scale the entire monolithic application.
• Monolithic applications lock you into an architecture. You cannot easily decide to re-write one of the tiers using a different technology stack, or even incrementally adopt a new technology stack.
• Coupling becomes crippling. As systems grow, coupling limits how much change can take place and you end up coding up even more interdependence. The ability to separate a logical module from another module becomes very difficult.

A great deal of friction can result in the CI/CD pipeline.

• When you update one component, you end up redeploying entire application.
  • Sometimes this can be mitigated by using a plugin architecture, but this is not typical.
• Testing is difficult
  • Slow build and application start-up time compromises timely integration, unit, and functional tests
  • This results in a slow feedback loop with large test suites
  • Oftentimes, it can become so slow enough that developers think twice about running the full test suite locally before checking in
    • This compromises the critically important feedback cycle

Frequent deployments provide well documented value:

https://en.wikipedia.org/wiki/Release_early,_release_often

In many cases it can take over a year to roll out new upgrades of the software. In a retail world where agility is paramount, this inability to increase deployment velocity can be crippling. The deployment issue gets progressively more difficult where you have thousands of retail outlets scattered throughout the world. These rollouts can be tricky as you can't afford any store downtime. In addition, many retail outlets have customized software, not to mention localization-specific deployments. Finally, the hardware platforms running these point-of-sale systems may be in various forms of upgrade, meaning that the hardware footprint may differ dramatically among store locations.

But being able to update point-of-sale systems frequently can provide other benefits:

• Redesign for better user interfaces based on customer and employee feedback
• Higher return on investment for software investments due to faster time to market
• Continuous testing and improvement results in defect reduction
• Self managed development teams are more efficient and generally happier
• Customers are happy because products just work

Specifically with respect to point-of-sale systems there is the ability to implement loyalty programs more quickly, respond to changes in the competitive landscape.

This need to synchronize the application along with its dependencies becomes another pain point. One problem is that the virtual machines are optimized to run that single application. If additional apps need to run on VMs, those dependencies must be installed alongside existing dependencies. This leads to versioning challenges with dependencies.

It also introduces a level of coupling meaning that the application must be tested with those specific DLLs. If an engineer would update the DLLs and not the application, the application may fail to operate properly.

Now your monolithic apps are locked into running on a specific, pre-configured VM. If that VM doesn't have the correct dependencies installed, the monolithic app would fail. It would be better to have generic VMs that can run any workload, any monolithic application. This means that monolithic application would come pre-configured with all its dependencies.

Figure 7: Dependencies on external DLLs

There is no question that connectivity and internet bandwidth is continuing to improve. There will be a point at which the public cloud will play a dominant role in point of sale systems, as explained previously.

By now, most of know the benefits of the public cloud. Here is a quick outline:

• Scaling: You can scale to meet any demand without having to having to worry about capital expenditures for on premise hardware
  • This is particularly interesting to franchisees, who typically frown upon upgrading or improving the hardware unless there is a clear value proposition
• Disaster recovery: Public cloud providers generally triply replicate persistent data.
• Support for mobile/kiosks: Providing functionality to mobile users is very difficult using on premises architectures.
  • There may be scenarios where businesses want to provide the ability to make purchases or transact business using the mobile device.

Figure 8: Leveraging the public cloud

Absolutely.

Given all this context, can you containerize so that you could have a greater deployment velocity and reduce external dependencies? It isn't simple, but the answer is "yes."

Let's make some big assumptions so that we can at least talk about re-architecting. Let's assume that you have redundant and excellent network connectivity to the cloud (this is becoming increasingly the case throughout the world). Let's also assume that you have control over the infrastructure running on premises. This would enable you to run your monolithic app in a container in multiple locations:

1. On-Premises
2. In a public cloud

Some important concepts to understand from the figure below:

• The container includes the monolithic app and all of its dependencies
  • This enables you to have a generic VM with nothing installed - one less headache
  • This enables you also run multiple containers in a single VM, notwithstanding privacy concerns where 2 tenants run on a single VM

Figure 9: Running in a container

Containerized applications benefit both public cloud deployments as well as on premises deployments. The key factor here is that a point-of-sale application, along with all its dependencies, are packaged in a standardized unit. Because the application is packaged with all its dependencies, it can be rolled out to virtually any VM, even bare-metal computers, in a predictable fashion. A generic virtual machine can be the target for the application to run on. The side benefit here is that this generic VM can run practically any application, not applications whose dependencies have been pre-installed. The container brings everything it needs with it.

There are many valid strategies for migrating to a microservices architecture from a monolithic architecture.

To get some background on monolithic architectures, this is a good resource:

https://en.wikipedia.org/wiki/Microservices

The good news is that the migration can take place in stages. You can begin by moving an existing monolithic application from a VM into a container. That is just a first step on a journey, not an end goal. You do not need to wait until an app is completely rewritten to starting gaining the benefits of Docker immediately.

Figure 10: The move to Microservices

Over time you can move pieces of the monolithic application as a series of stateless microservices. You are essentially starting with a “lift and shift," with a general and gradual move to a de-coupled services oriented architecture. The development teams can start breaking down the tiers piece by piece. They can move some functions out of the monolith, and begin deploying them as loosely coupled services in Docker containers.

Over time the entire application is deconstructed, and deployed as a series of portable and scalable services inside Docker containers.

The advantage is that you can take the containerized services and components and run them at massive scale in a data center. With Docker containers, this becomes a pretty straightforward effort. Because Docker containers are inherently portable, and can run in a VM or in the cloud unmodified. In addition, the containers are portable from VM to VM to bare metal without a lot of heavy lifting to facilitate the transition.

The biggest challenge of all, of course, is asking a company to rearchitect its line of business application away from a monolithic architecture. Moreover, asking the company to also make their application multi-tenant is another huge and expensive ask. In a nutshell, what you are really asking the company to do is decompose their application into a series of services that support multiple tenants.

You can see below that each store location below is really a separate tenant. There for you need to run a separate monolithic application in a container for each tenant. Imagine a company that has 6,000 stores. That would be 6000 containers. You might argue that you could run multiple containers in a single VM. BUt that might not be feasible given the paragraph below.

Although containers are not as completely isolated from one another as virtual machines, they are more secure than just running applications by themselves in their own VM. Your application is really more secure when it's running inside a Docker container," said Nathan McCauley, director of security at Docker.

Notice in the diagram below that the microservice architecture below is multi-tenant. This means that each store location could be considered a separate tenant in the overall architecture. Each store location could operate completely isolated from other stores - if the requirements dictate that as such. You could even architect the microservice to be "owner aware," meaning if a franchisee owned multiple store location, the definition of tenant could mean an "owner," not a "store location."

The key takeaway is that each microservice (like Register, Inventory Management, Web UI) is a multi-tenant service completely de-coupled from the other microservices. Here are some characteristics:

• Each microservice has its own DB and caching layer
• Each microservice has its own dev team and own schedule for CI/CD
• Each microservice is a self-contained unit of OS, runtime, framework, third-party libraries and code
• Each microservice can be deployed independently and scaled appropriately based on its requirement

Figure 11

Martin Fowler is considered one of the founding fathers of microservice architectures. He cites some pros and cons.

https://martinfowler.com/articles/microservice-trade-offs.html

• Strong Module Boundaries
  • Microservices reinforce modular structure, which is particularly important for larger teams.
• Independent Deployment
  • Simple services are easier to deploy, and since they are autonomous, are less likely to cause system failures when they go wrong.
• Technology Diversity
  • With microservices you can mix multiple languages, development frameworks and data-storage technologies.

• Distribution
  • Distributed systems are harder to program, since remote calls are slow and are always at risk of failure.
• Eventual Consistency
  • Maintaining strong consistency is extremely difficult for a distributed system, which means everyone has to manage eventual consistency.
• Operational Complexity
  • You need a mature operations team to manage lots of services, which are being redeployed regularly.

You can read more here:

• Netflix is perhaps the largest user of microservices. Lots of great detail here.
  • https://techblog.netflix.com/2015/02/a-microscope-on-microservices.html