Authored by Charles Young from Solidsoft Reply, in conjunction with Russell Young from Microsoft.
This article is part of a series about customers who’ve worked closely with Microsoft on Service Fabric over the last year. We look at why they chose Service Fabric, and we take a closer look at the design of their application.
This customer profile is also available as a technical customer story on Customers.Microsoft.com:
- Solidsoft Reply platform powers e-verification of pharmaceuticals
- Technical Customer Stories - An Azure.com gallery of all our technical customer stories.
In this installment, we profile Solidsoft Reply, the European Medicines Verification System (EMVS) running on Azure, and how they designed the architecture using Service Fabric, Azure Cosmos DB, and Azure Event Hubs.
Based in the United Kingdom, Solidsoft Reply has more than twenty years of experience in building integrated business solutions. It is part of the Reply Group, a cooperative network of highly specialized companies that support their customers in defining and developing business models to optimize and integrate processes, applications, and devices using modern technology and communication paradigms. Solidsoft Reply is a Microsoft Gold Partner specializing in integrated cloud-based and on-premises solutions on the Microsoft platform.
Detecting falsified medicines in the European supply chain
EMVS represents the pharmaceutical industry’s response to the European Union’s Falsified Medicines Directive (FMD). The FMD harmonizes legislation across Europe to protect 520 million European citizens from falsified and counterfeit medicines. Paid for by drug manufacturers, FMD calls for a European-wide system to detect, identify, and eliminate falsified medicines. The European Commission worked with pharmaceutical industry stakeholders to agree to an approach. In early 2016, additional legislation was enacted to mandate the use of the EMVS.
Solidsoft Reply won the contract to build and operate the EMVS. Starting in February 2019, every pack of prescription medicine entering the European pharmaceutical supply chain must, by law, bear a unique identifier (a two-dimensional Data Matrix barcode; see Figure 1). The EMVS must be used to verify every pack identifier at the point where the pack is dispensed. Every drug manufacturer, parallel distributor, wholesaler, hospital, and pharmacy operating across 32 nation states must be connected to the EMVS.
Figure 1. A two-dimensional Data Matrix barcode.
The EMVS consists of a central European Hub that handles product, batch, and pack data uploaded by drug manufacturers and parallel distributors (the businesses that support the distribution of pharmaceutical products across national borders). As Figure 1 shows, the European Hub distributes this data to national systems for each market in which the product is authorized. Marketing authorization is granted at a national level, and each country has its own national system.
Wholesalers, third-party logistics providers (3PLs), hospital pharmacies, and community pharmacies connect to the national system in the market in which they are located. They use the EMVS to verify the unique identifier of each pack of prescription medicine and to record any event in which the pack identifier is decommissioned. In some cases, pack state changes are communicated across multiple markets via the European Hub. When a potential falsified medicine is detected, the wholesaler or pharmacist is informed, and an alert is generated and returned to the manufacturer and stakeholder organizations for further investigation.
Figure 2. The EMVS components.
The EMVS must reliably and consistently support the legal requirements placed on all European pharmacists and wholesalers to verify every pack of prescription medicine. It must minimize the cost-per-pack of verification and scale and perform effectively across the continent. To accomplish this, Solidsoft Reply built the European Hub as a cloud-based solution on the Azure platform. Most markets have opted to purchase a blueprint of the national system, a type of reference architecture that meets a set of requirements specified by European industry stakeholders. Solidsoft Reply provides a cost-effective, cloud-hosted National Blueprint System (NBS) built on the same technology and architecture as the European Hub.
The design, development, deployment, piloting, and rollout of a system of this size represents a considerable challenge, especially given the fixed timescales imposed under European law. The system must scale effectively and must be resilient and highly available. It must provide excellent performance to all its users. The NBS, in particular, must be built incrementally using an agile, DevOps-oriented approach to ensure timely delivery of sufficient functionality to support national pilot programs.
Building on Service Fabric
Solidsoft Reply chose Service Fabric as the most appropriate platform on which to build a microservice-based solution that meets these demands. Service Fabric provides the intelligent cluster-based service orchestration, routing, and state management that the EMVS requires. Service Fabric offers high levels of resilience, availability, and scalability, so much of the risk associated with meeting demanding SLAs is eliminated. The Solidsoft development team took advantage of the first-class support in Service Fabric for stateful service models to support the asynchronous, persisted interchanges that the EMVS needs and enable scalable, resilient workflows.
To maintain performance at scale, the EMVS depends on careful partitioning of different workloads and data. As Figure 2 shows, the solution uses a layered architecture with separate clusters for different types of stateful and stateless workload. Each cluster can be sized and configured appropriately and scaled independently and dynamically.
Figure 3. EMVS node cluster layered architecture.
The hot paths are kept as lightweight and uncluttered as possible, handing off background work asynchronously to separate microservice domains. In addition, message mediation, routing, and workflows are handled as separate asynchronous processes. The solution uses stateful queue services to partition asynchronous messages and workflows. These queues are persisted and automatically replicated across multiple datacenters. The system implements checkpointing across each type of message flow to support robust recovery from failure with minimal risk of data loss.
Service Fabric Reliable Services programming model
In the EMVS microservices-based architecture, each service has a single purpose and is independently deployable and configurable. Service Fabric manages the placement of each microservice instance within a cluster and the routing of network traffic to and from that instance.
EMVS was built using the Service Fabric Reliable Services programming model and uses both stateless and stateful microservices. As Figure 3 shows, stateless services in the web gateways ingest the product, batch, and pack data uploaded by drug manufacturers and parallel distributors. This nonvolatile master and reference data is persisted in Azure Cosmos DB, a low latency NoSQL document store that stores data formatted in industry standard JSON notation. Stateful microservices manage their own data and are used for the queues that handle the volatile service state associated with processing and distributing the pack data. Stateful services rely on Service Fabric to replicate their data transactionally and manage fine-grained recovery and failover of individual service instances.
Figure 4. EMVS microservices overview.
EMVS also uses the Service Fabric Reliable Actors programming model to support concurrency. In this model, an actor is an isolated, independent unit of compute and state with single-threaded execution. EMVS workflow actors perform lightweight business processes such as bulk verification of pack data.
Founded on microservices
The main microservices in EMVS are:
- Gateway services, including the web front ends, the API used for public access to the European Hub, and a private API used for administration and configuration of the system. Stateless gateway services upload master data store, verify packs, and synchronize their state against the master store.
- Latency services include the queues that handle nonvolatile master and reference data, including product, batch, and pack-related information. The solution must, by law, meet demanding low-latency requirements to ensure that verification of all packs of prescription medicine does not result in a significant loss of performance for existing systems and applications. Low latency services work with the data partitioning capabilities of Cosmos DB to ensure consistent low-latency pack verification at scale.
- Throughput services include the queues that handle volatile service state during the processing and distribution of pack data. Pack state triggers the notification systems used for reporting and escalation.
- Processing services use workflow actors to represent product pack data and master data. Actors also represent product recalls, reporting services, and escalation requests.
- Event services are stateless microservices that monitor the activities of the transaction queues and provide history for the auditing and other logging systems.
- Exception services handle system-level issues such as retries and deferred requests, then notify the system administrators and reporting services.
The EMVS must scale appropriately to handle different European-wide and national workloads. The scalability strategy is founded on microservices, allowing the system to adapt quickly to changing conditions and demands. Microservices also support the very high availability necessary for a system of this size. Any significant outage could adversely affect a large part of the pharmaceutical industry across Europe or within individual national markets, resulting in very large backlogs of deferred work that must be processed as part of the recovery cycle.
Service Fabric automatically replicates service instances and their data and handles recovery when a service instance fails. This self-healing feature significantly reduces the risks associated with service unavailability and data loss as well as eliminating many of the scenarios that might otherwise require a full system failover between datacenters.
The requirements for the EMVS have been specified by industry stakeholders, and are expected to change and evolve over time. In addition, individual markets may want to customize their national system. Microservices support rapid evolution and change to the system with minimal disruption to running services.
Figure 5: EMVS microservice architecture.
Managing master data with Cosmos DB
The EMVS uses Cosmos DB to store data on an array of SSD disks, ensuring that the service is scalable, robust, and performant.
The impact of data loss in the EMVS is potentially severe. Although it is possible to recreate data by asking manufacturers to upload it again to the European Hub, this approach is expensive and disruptive. Even worse, if pack data is lost together with audit and transaction logs, the correct pack states cannot be recreated, and citizens could be put at risk.
Cosmos DB supports high availability by providing geo-replication through an eventual consistency model. Writes to the primary store are replicated via intermediate queues to remote replicas. The EMVS provides a configurable approach to balancing the overhead of geo-replication against other processing, allowing intelligent trade-offs between risk and performance.
Other Azure services
As a multi-layered solution, the EMVS works with many Azure technologies to meet the requirements for data storage, auditing and reporting, secure authentication of users, and data sovereignty.
Figure 6. EMVS relies on several Azure services to provide security, authentication, auditing, and storage.
Storing long-term data
The EMVS must retain audit log data for long periods (up to a decade) for regulatory, supervisory, and reporting purposes. With 10 billion packs of prescription medicine entering the European supply chain each year, the volume of stored data will grow very quickly. The EMVS uses Azure Blob storage and Table storage to provide low cost storage capacity for large quantities of data. The data is geo-replicated to minimize the risk of data loss and allows read access for inspection and reporting purposes. Sensitive data is encrypted by the storage service.
The future roadmap for the EMVS includes the option of storing data using Azure Data Lake. As well as additional cost advantages, this change will also better support analytics. Data ownership rules are strictly applied within the EMVS to prevent any organization from obtaining a commercial advantage from analysis of the data. Over time, more sophisticated data analytics will be introduced to detect suspicious behavior within the supply chain and to support the requirements of national and European medicine agencies.
Managing audit logging through Azure Event Hubs
The EMVS must retain extensive and comprehensive audit trails of the activities of the system at both European and national levels. The audit trails may be used during forensic investigation of potential counterfeit medicines. The solution uses Azure Event Hubs as a high-performance queuing mechanism to support a very high rate of message ingestion. Event Hubs decouple the services generating event and audit log data from the services that consume and process those events.
Security and authorization
Although the EMVS does not store sensitive personal data, it must implement high standards of data protection and privacy in accordance with strict data ownership rules. The solution is a shared resource accessed by thousands of organizations and companies operating across the pharmaceutical sector. It must guard against unauthorized access and data upload, as well as prevent any misuse or abuse of the data it holds.
The security subsystem includes the Public Key Infrastructure, user authentication via Azure Active Directory (Azure AD), and Azure Key Vault. A set of virtual machines in the European Hub manage the process for signing client certificates. These virtual machines operate in a Certificate Authority role and are used to sign client certificates, allowing the Hub to verify the authenticity of those certificates.
Azure AD is configured with user accounts and provides password authentication. Additionally, Azure AD is used to allow parts of the Hub to authenticate with each other. Key Vault, with its standards-compliant Hardware Security Module (HSM), is used to store cryptographic keys and application secrets such as passwords and access keys. Access is granted through Azure AD integration.
Manufacturers and parallel distributors upload product and pack data to the system. They connect to the European Hub using certificate-based authentication, multiple layers of encryption, and non-repudiation approaches. Access is only granted to organizations that have passed a stringent verification and accreditation process.
The security requirements for pharmacists and wholesalers are less stringent. The NBS implements an OAuth 2.0 Identity Provider using a third-party service hosted in Azure alongside each national system. This system is used to authenticate client systems and authorize their use of the verification API using a straightforward and appropriate approach.
Managing data traffic
The EMVS uses Azure Traffic Manager to control the distribution of user traffic to specified endpoints, which includes Azure cloud services, websites, and other endpoints. Traffic Manager works by applying an intelligent policy engine to Domain Name System (DNS) queries for the domain names of the Internet resources. The Azure-hosted cloud services provided by the EMVS can run in different datacenters across Europe.
Benefits of Service Fabric
Before Service Fabric, the first version of the European Hub was built and piloted using earlier Azure PaaS features. Despite the success of the pilot in a single market, it was clear that the original architecture did not provide sufficient scalability for European roll-out. The second generation of the European Hub, built using Service Fabric and Cosmos DB, addresses these concerns.
The Solidsoft Reply development team worked closely with the Microsoft Service Fabric product team for a year before the release of Service Fabric to ensure that the architecture would meet its scalability, performance, and availability requirements. In particular, the design of Reliable Queues in Service Fabric evolved to address a number of concerns raised during EMVS development.
Service Fabric and Azure services also provided the following benefits in the ongoing development of the NBS:
- Accelerated development: The investment made in designing and implementing Service Fabric patterns in the European Hub has directly benefitted the implementation of the NBS. It uses the same patterns and approaches to handle interchange with the European Hub and to manage other asynchronous processing at the national level. The ability to repurpose architectural and design patterns has significantly accelerated the development of the NBS in the face of demanding timescales.
- Agile development: Service Fabric, with its support for microservice architecture, complements the agile methodology used to deliver NBS functionality. The Solidsoft Reply development team has been able to support national pilot programs and roll-out before the fixed end-date in February 2019.
- Automated testing: Comprehensive and rigorous testing of complex distributed systems is always challenging. Solidsoft Reply built its own automated test framework to support the needs of the EMVS program. The framework provides a comprehensive approach to black-box and grey-box testing of the system as a whole, as well as the individual European and national-level components of the EMVS. The framework provides readily available test metrics via Microsoft Power BI.
- Compliance: The pharmaceutical industry is highly regulated. The EMVS complies with GAMP 5 GxP best practice standards and constitutes a verified system with detailed traceability of functionality back to the formal requirements and functional specification, and an emphasis on robust configuration management of individual releases and environments. Solidsoft Reply operates the European Hub and National Systems on behalf of its customers and implements comprehensive operational management processes used Azure-hosted tooling.
Service Fabric, together with other Azure services, provided Solidsoft Reply with a modern platform to address the requirements of the European Union and the European pharmaceutical industry in building, deploying, and managing the EMVS. Of the many bidders for this important European Union service, only Solidsoft Reply proposed using a solution with a fully public, cloud-based back-office infrastructure on Azure.
Service Fabric gave Solidsoft the edge with its powerful, approach to building highly scalable, available microservice-based solutions that can evolve over time in an agile, cost-effective manner. The use of this platform represents significant cost savings for the pharmaceutical industry, helping them to meet their legal responsibilities and to provide world-class protection to citizens across the continent.
For Solidsoft Reply, this technological platform has enabled the company to move beyond its strong history as an enterprise-level integration specialist to a cloud-centric provider of large distributed systems and services.