How to Integrate Your Billing System with the Usage Metering System

This post is a part of the nine-part “ What’s New in Windows Server & System Center 2012 R2 ” series that is featured on Brad Anderson’s In the Cloud blog. Today’s blog post covers Service Provider experiences in enabling the billing and chargeback of Tenant Resource Utilization and how it applies to Brad’s larger topic of “Transform the Datacenter.” To read that post and see the other technologies discussed, read today’s post: What’s New in 2012 R2: Service Provider & Tenant IaaS Experience .

As described in that blog post, enabling billing scenarios for the service providers is a key investment area for this release. Service providers cannot successfully monetize their services in the absence of a system that tracks and reports on tenant resource utilization. These services are offered on a subscription basis and therefore it is critical that the resource utilization is reported at the subscription granularity to assist in billing scenarios. 

Overview of the Usage Metering System

The usage system is located alongside the Service Management API in the Windows Azure Pack (WAP) stack, enabling it to access tenant utilization data for all the services provided in the WAP stack and provide an REST API which is leveraged to integrate with the billing system of the provider, as illustrated in the figure below.

Service providers have invested a lot in their own billing system and it was critical that the 2012 R2 release be able to integrate with the existing systems in place. Therefore, we targeted our investments to ensure that 2012 R2 integrates easily with various billing providers and ITFM (IT Financial Management) products that are in the market.

It is important to note that there is no billing system being shipped in 2012 R2 release. Service providers have to create the billing integration module (also referred as “billing adaptor”) to provide data to the billing system they are using.

Now, lets go a little deeper to look at the building blocks of the usage metering system and how its architected.

The Usage Metering System has four main components. Three of these components, the Data Generator, The Data Collector and the Usage Database are internal to the system and the fourth component the Usage API is an external facing API that the billing adapter will interface with to extract the tenant resource utilization data.

Data Generator

The Data Generator tier represents the services (resource providers) registered as part of the system. They collect information specific to a subscription and expose it to the Usage Collector. The Usage Collector expects information to be made available following a specific data contract. This contract is the same across all the providers. All providers in the system adhere to this contract to provide information. IaaS metrics in Windows Azure Pack are provided by VM Clouds resource provider.

Data Collector

The Data Collector is an internal component that periodically collects usage information from all the registered Data Generators and stores it in the Usage Database.

Usage Database

The Usage Database is a transient store, which stores all the data from the various Data Generators for a period of 30-40 days. The expectation is that during this time, the billing system would have extracted the data from this database for billing purposes.

Usage API

This is a RESTful API and is the only way to extract the data from the Usage Database. Since Service Providers typically have a billing system which allows them to generate monthly bills to their subscribers. Customers can easily create an integration to their billing system by extracting data from the Usage Database through the Usage API. The component that customers develop to integrate with their billing system is called a “Billing Adapter”, which serves as a bridge between the Usage Metering system and the customer billing system.

In the figure below, in the red circles, you can see the VM Clouds resource provider, alongside other resource providers such as Service Bus, generating the IaaS resource utilization data, which is collected and stored in the Usage Database and made available through the Usage API.

The Usage API can be leveraged to create the billing adaptor and interface with the billing system within the provider data center. In the figure below, you can see that the role of the “billing adaptor” serving to integrate the Usage Metering System and the billing provider within the provider datacenter.

The “Service Reporting” component and the analytics it provides is discussed in the blog post titled “Creating Usage Analytics Reports using Excel and Performance Point” while this blog post details on how to create a “Billing Adaptor”.

Interacting with the Usage System

This section explains the ways an external system can interact with the Usage Metering System. Two different types of information are available through the Usage API:

1. Tenant resource utilization for all subscriptions
2. Plan, add-on, subscription, and account information

The information is presented via two channels:

1. Usage API that queries all the historical data
2. Real time CRUD events via the Event Notification System.

The billing adaptor uses both these channels to be able to effectively create a billing reports while being able to respond in real time as plans, subscriptions and accounts get created and managed in the environment.

Usage API (Exposed on the Usage Endpoint)

Usage Data

The Usage endpoint exposes an API to return tenant resource utilization data pertaining to every subscription across services. The caller (“Billing Adapter”) needs to provide the “startid”. This parameter informs the Usage Metering System to return usage data, starting from that ID. The Billing Adapter advances the “startid” based on the number of records returned for the subsequent call.

Plans\Addon\Subscription Data

The Usage endpoint also exposes APIs to return data on existing plans, addons, subscriptions, etc

Notes:

STARTID is the record id of the first record you want to fetch in a particular cycle.

BATCHSIZE is the maximum number of records you want to fetch.

USAGE-RESTAPI-ENDPOINT can be found at https://<Admin-API-Machine-Name>:30022

Configuration

The administrator needs to ensure that the Usage Metering Service is configured correctly to authenticate the Billing Adaptor. That can be done by ensuring that the service is capable of accepting the correct credentials that will be used to authenticate. The steps below describe how to ensure that the credentials are set properly. Note: During the installation process, the password used is a random sequence and hence this step is necessary to establish connectivity.

On the WAP deployment launch the Management Service PowerShell Module on the Admin API server.

Then, run the commands below:

· Set-MgmtSvcSetting -Namespace UsageService -Name Username -Value '<EnterUserName>'

· Set-MgmtSvcSetting -Namespace UsageService -Name Password -Value '<EnterPassword>' –Encode

Once the username and password are set to known values, these values can be used by the Billing Adaptor to authenticate.

Consuming the Usage REST API

The following steps are required to consume the Usage REST API:

• Define an httpClient
  • Define the base address (https://<Admin-API-Machine-Name>:30022 )
  • Define the request header
    • Set the media type (application/json or application/xml)
    • Authorization Type (basic)
    • Username and Password
• Construct a URI to query the Usage Metering Service
• StartID is the record id of the first record you want to fetch and BatchSize is the maximum number of records you want to fetch.
• Execute the API call and read Usage Data
• Data Contracts can be used to de-serialize the response returned (as in Sample below)

Usage Data Model

The Usage Data Model is shown in the figure below and can be used to associate the data returned by the Usage API.

Event Notification System

The Service Management API keeps track of events within the Usage Metering System and sends notifications to any registered subscriber (e.g. a Billing Adaptor). Examples of the events are plan, addon, subscription creation\updates and account creation.

Notifications are sent as a Post call to an endpoint registered with the Usage Metering System. The Management Service PowerShell Module should be used to define the required notification end point. Note that the notificationEndPoint must end with a trailing slash.

Description:

Subscribing for plan, add-on and account changes.

SubscriberType:

• BillingService
• MandatoryService
• OptionalService

Example:

Set-MgmtSvcNotificationSubscriber -Name Billing –SubscriberType BillingService -Enabled $false -Endpoint https://localhost/ -AuthenticationMode Basic

The Billing Adaptor can be set up to handle the event in a blocking or a non-blocking manner. The SubscriberType BillingService & MandatoryService are both blocking. The only nonblocking option is OptionalService. If the Billing Adaptor is set up to be blocking, a plan creation event in the service management API should trigger a corresponding plan to be created in the billing system. If this operation is not successful, the plan creation at the service management API will fail. This enables consistency between the platform and the billing system.

Notification Data Contracts

Notifications sent to the billing adapter adhere to type - NotificationEvent<T> type. T could be replaced by the below objects.

• Plan
• PlanAddOn
• AdminSubscription
• ResourceProviderReference
• PlanAddOnReference
• PlanAddOnReference

When you download the WAP (Windows Azure Pack) the data contracts can be found under:

· \SampleBillingAdapter\DataContracts\*

Following are the two important properties of NotificationEvent

1. NotificationEvent Method could have following values:

1. Post to create a new account/subscription/addon/plan

2. Delete to delete an account/subscription/addon/plan

3. Post an update to a plan

2. NotificationEvent Entity sends an event when any of the above objects are created\updated\deleted.

Pricing APIs

The Pricing API is designed for billing system in the Service Provider data center to specify prices for Plans and Add-ons to flow into the 2012 R2 system. The billing adaptor can choose provide prices for each Plan, or Plan add-on in real-time. As part of implementing the notification subscriber, we have specifications for the below APIs that the billing service can implement to enable pricing data to flow back into the system. The implementation of these APIs is optional. If the below APIs are enabled the price values for the plans and add-ons will be visible in the WAP Tenant site at the time of addition of the Plan\Add-On.

Notes:

• This API is expected to return a string with pricing information. The 2012 R2 system will display this information alongside plans for the subscriber, but these are textual and not typed.

Detailed Description of the Sample Adapter Project Files

This section explains the content of the sample billing adapter (SampleBillingAdapter.sln). At a high level the billing adapter consists of the below parts:

1. SampleBillingAdapter .cs provides an example of the different calls to the Usage REST API

2. The set of Data Contracts that can be used to deserialize the API responses

SampleBillingAdapter.cs

This is the entry point for the application. The file contains the below:

1. Instantiation of a UserServiceHttpClient with the required configuration data.

2. This UsageServiceHttpClient is then used to query the usage service. There are seven types of calls that can be made for the Billing data. This data is deserialized into instances of the data contracts that are included in the DataContracts directory

3. The data is then printed to the console.

Example:

using Microsoft.WindowsAzurePack.Usage.DataContracts;
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;

static void Main(string[] args)
{
// create a WAP usage service http client (this data can be read from a config file)
string mediaType = "application/json"; // application/json or application/xml
string authenticationType = "basic";
string Username = "UsageClient";
string Password = "specify the correct pwd";
string Machine = "specify the machine where the usage service is running";
string Port = "30022";
string BaseAddress = String.Format("https://{0}:{1}/", Machine, Port);
var usageService = new WAPUsageServiceHttpClient(Username, Password, authenticationType, BaseAddress, mediaType);

    // gather usage and billing data asynchronously using the Usage API
var usage = usageService.GetDataAsync<UsageRecordList>("usage", 0, 50);
var plans = usageService.GetDataAsync<UsageEventList<Plan>>("billing/plans", 0, 50);
var subscriptions = usageService.GetDataAsync<UsageEventList<Subscription>>("billing/subscriptions", 0, 50);
var addOns = usageService.GetDataAsync<UsageEventList<AddOn>>("billing/addons", 0, 50);
var planAddOns = usageService.GetDataAsync<UsageEventList<AddOnReference>>("billing/planAddons", 0, 50);
var subscriptionAddOns = usageService.GetDataAsync<UsageEventList<AddOnReference>>("billing/subscriptionAddons", 0, 50);
var planServices = usageService.GetDataAsync<UsageEventList<ResourceProviderReference>>("billing/planServices", 0, 50);

    #region Print the usage and billing data to the console ...
Console.WriteLine("Printing Usage Data - Press Enter to Proceed...");
Console.ReadLine();
usageService.PrintUsageData(usage.Result);
usageService.PrintPlanData(plans.Result);
usageService.PrintSubscriptionData(subscriptions.Result);
usageService.PrintAddOnsData(addOns.Result);
usageService.PrintPlanAddOnsData(planAddOns.Result);
usageService.PrintSubscriptionAddOnsData(subscriptionAddOns.Result);
usageService.PrintPlanServicesData(planServices.Result);
#endregion
}

Data Contracts

The DataContracts directory contains all the required Data Contracts to interact effectively with the Usage API.

VM Data Gathered from the Usage API

As you can see, this is a powerful API that allows bi-directional data flow. The usage data from the 2012 R2 stack to the billing adaptor and the pricing data (business logic decides the prices) and that data flows from the billing system into the 2012 R2 stack.

In subsequent blogs, we will provide more details as we hear more from our customers.

To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.