As a team, from Visual Studio App Center engineers to product managers, designers, and QA, our goal is simple: build an amazing product for developers. We decided early on that to help our team deliver on our goal, we’d create a set of values that would guide our decisions and keep us on the right path as we add new features, prioritize enhancements to services, or build integrations. We wanted these values to drive not just the product, but our team culture, and after many discussions, we established four core values: customer obsession, simplicity, no silos, and growth mindset.
As an Engineering Manager, I’m responsible for my teams’ execution, and I contribute technically to the product. Yet as a team, we are all empowered to improve our product and design the world we want to work with. This laser-focus on a central goal and ground rules sounds great on paper, but what does it actually mean? It means we constantly assess our tools, infrastructure, team structure, and development processes to make sure they’re simple, open, and freeing us to focus on delivering the best possible developer experience.
Goals and principles are different for every organization, but they make it easier to make hard decisions, prioritize one developer’s project over another’s, and stop doing things that just aren’t working. You may already have goals and values in place; if not, it’s never too late to get started, because it never ends.
In this post, I’ll share how and why we implemented one of our most recent changes: moving to Mono-repo and One Infra, including why we unified our development and infrastructure and what’s next.
App Center uses a micro-services architecture to give us the flexibility to break things down into small units that perform a single business function and can be developed and deployed independently. When we started, we chose to use multiple repositories.
When we were shipping a first version of the product, this made sense for two reasons:
- Clear Ownership: A small team owns the codebase end-to-end. Each App Center service team—Build, Test, Distribute, Crashes, and Analytics—was free to independently develop and deploy updates.
- Scale: Smaller code bases are easier to manage and cause fewer merge conflicts. Large codebases can be inefficient, taking a long time to clone, pull, and push.
We have several teams spread across multiple time zones, and, with the above approach, it was challenging to share and standardize code across the multiple repositories. A team member working on one service couldn’t easily clone a repository and know how to get started. Since developers didn’t always have a full understanding of the “big picture” across App Center, our code reviews weren’t as effective as they could be, and shared code didn’t feel “shared, so we were doubling effort in some scenarios.
Multiple repositories across multiple teams meant each team created their own infrastructure. We’d started with a common direction, using DC/OS and multiple data sources like SQL Server, Casandra, Table, and Blob storage. But, as teams pushed the product forward, how we set up infrastructure drifted in multiple directions. We were duplicating how we managed and accessed our infrastructure, and each team had to do secret management and rotation.
As we compared this to our “no silos” value, it went against building one team with knowledge across the product. We decided to move to Mono-repo and trunk based development and use a common infrastructure for our build and deployment pipeline. Value and goal check: this eliminates silos, helps us work better as one team, and frees us to constantly deliver customer value.
Going from Multi to Mono(-repo)
To move our code from multiple repositories in different source providers into one, each team migrated their development or master branch (including its history) into a new Visual Studio Team Services project repo:
- We wrote a script to move the current repo into a “Mono-repo” branch.
- To simplify the transition and to plan for a folder structure in the future, repositories went into a sub-directory under the root path.
- Once code was working and the build definitions were migrated, we created a Pull Request and merged to our new Mono-repo master.
We used kebob format (tools-swagger-codegen) for our folders, making descriptions as robust as possible (within a reasonable length) and removing team-specific names for code. For example, we called our user service Vanaheim which was confusing to new team members and teams working on other services. When we moved to Mono-repo, Vanaheim became accounts-management-service a clear, descriptive, and easily understandable identifier.
After a few weeks, we’d fully transitioned, but it’s a constant work in progress; we want it to be a never-ending, iterative process. As we continue development over time, we’ll continuously make our code better, simplifying processes and working together to build better product.
Creating One Infra
Shifting to Mono-repo gave us an opportunity to rework and consolidate our infrastructure too. Each team used Docker to deploy code into different orchestrators, read secrets their own way, and had their own approach to logging and telemetry. In addition to consolidating repositories, we’d unify and share code in way that didn’t require cloning and duplication.
We use Visual Studio Team Services to consolidate builds, packages, and deployments. Promoting a single artifact reduces our total number of build definitions to one per service or component we create. To get our build definitions working inside of our new Mono-repo, we followed the below steps:
- Rename build definition to match the repository folder name.
- Set a path filter to the Trigger, so Continuous Integration is only triggered when changes are made under the components folder.
- Add a ROOT_FOLDER variable that specifies your directory name.
- Add a Shell Exec step to the Build Tasks to move your service into the root, so all existing tasks execute as normal (refer to CI above for Shell Exec code).
- Add any other required steps to build the code.
We use three environments: integration, staging and production, and two Visual Studio Team Services control our releases. Our integration definition automatically deploys changes merged into master to the corresponding integration environment.
The majority of our existing integration definition didn’t require any alterations, other than adding a new step to create the git tags. The production definition deploys the integration artifacts to staging and creates an automatic pending approval for a production deployment. After qualifying our staging environment is ready, we generate an approval for production. In the future, as we grow and add more features, we’ll fully automate this, too, allowing releases to go straight to production.
Given our growth mindset approach to development, we’re continuing to iterate and refine our processes. Our next step is to consolidate subscriptions and services, including moving our Docker deployments to Azure Service Fabric as our orchestrator to consolidate our logging and telemetry infrastructure, as well as standardizing how we access and rotate secrets.
This automation and consolidation brings us back to our core values and goal, since every simplified process means each team member spends more time delivering value to developers and less time managing infrastructure.
The move to Mono-repo and a common infrastructure is ongoing, but we’re already seeing some positive results. Team members are more productive, code is less complex, and there’s more knowledge sharing across teams. Teams don’t just work and own one feature area, they work on the most important set of features across the product, so we have all hands on deck for high-priority items.
We’re committed to delivering customer value and making App Center the best CI/CD experience for all developers, so you can ship better apps faster and focus on delivering value to your users.
We’d love to hear from you. Sign in and click the “chat” icon from your App Center dashboard to tell us what you’d like to see next, your team values, or how you’ve streamlined your development pipeline. We’re all ears!