CQRS Journey guidance project released

  • Article

Today we, the patterns & practices team, are happy to announce the release of the CQRS Journey guidance project.

Customer Value

The project is focused on building highly scalable, highly available and maintainable applications with the Command & Query Responsibility Segregation and Event Sourcing patterns. Right from the start we positioned this project as a learning journey. Our vision was to provide an extended experience report that described building, deploying (to Windows Azure), and maintaining a sample real-world complex enterprise system as a reference implementation (RI) to showcase various CQRS and ES concepts & techniques. A famous explorer Jacques Cousteau once said that “the best way to observe a fish is to become a fish." The best way for us to provide this guidance was to immerse ourselves in the project where we could experience the challenges first hand, overcome them, and then share the lessons we learned in doing so.

This journey is now complete. We’ve shipped three pseudo-production releases and attained the goal of no-downtime migration. We’ve also worked hard in the last milestone to empirically prove the scalability of our system. The map below depicts our main milestones. To learn more, we invite you to read the guide and play with the sample code we are shipping.

CQRS Journey map - click to enlarge!

Acknowledgements

The dev team didn’t embark on this journey by themselves and didn’t work in isolation. We actively sought input from industry experts and from a wider group of advisors to ensure that the guidance is detailed, practical, and informed by real-world experience.

We’d like to thank many people who have contributed to the journey in many different ways:

  • Greg Young for his pragmatism, patience with us and continuous mentoring;
  • Udi Dahan for challenging us and offering alternative views on many concepts;
  • Clemens Vasters for pushing back on terminology and providing a very valuable perspective from the distributed database field;
  • Kelly Sommers for believing in us and bringing sanity to the community as well as for deep technical insights;
  • Adam Dymitruk for jumpstarting us on git and extending the RI;
  • Glenn Block for encouraging us to go all the way with the OSS initiative and for introducing us to many community members;
  • Our GM Lori Brownell and our director Björn Rettig for providing sponsorship of the initiative;
  • Scott Guthrie for supporting the project and helping amplify the message;
  • Josh Elster for exploring and designing the MIL (Messaging Intermediate Language) and pushing us to make it easier to follow the workflow of messages in code;
  • Cesar De la Torre Llorente for helping us spike on the alternatives and bringing up terminological incongruities between various schools and thought leaders;
  • Rinat Abdullin for very active participation at the beginning of the project and contributing a case study;
  • Bruno Terkaly and Ricardo Villalobos for exploring the disconnected client scenario that would integrate with the RI (look forward to your out-of-band release!)
  • Einar Otto Stangvik for spiking on the Schedule Builder bounded context implementation in Node.js (look forward to that too);
  • Mark Seemann for sending the very first pull request focusing on code quality;
  • Christopher Bennage for helping us overcome GitHub limitations by creating the pundit review system and the export-to-Excel script to manage iteration backlog more effectively;
  • Bob Brumfield, Eugenio Pace, Carlos Farre, Hanz Zhang, and Rohit Sharma for many insights especially on the perf and hardening challenges;
  • Chris Tavares for putting out the first CQRS experiment at p&p and suggesting valuable scenarios;
  • Tim Sharkinian for his perspectives on CQRS and for getting us on the SpecFlow train;
  • Kelly Leahy for sharing his experience and making us aware of potential pitfalls;
  • Dylan Smith for early conversations and support of this project in pre-flight times;
  • Evan Cooke, Tim Walton, Alex Dubinkov, Scott Brown, and Gabriel N. Schenker for sharing their experiences and contributing mini-case studies;
  • RoAnn Corbisier, Nelly Delgado and Nancy Michell for being patient with our many revisions and handling edit and production;
  • Alexander Ustinov and Anton Rusecki for clean and intuitive styles (both UI and book illustrations) and for the fast turnaround.

We gratefully acknowledge our reviewers who commented on the code and the early drafts of the guide. Our 70+ member advisory board was exemplary in many ways, especially in terms of the activism both during the meetings and via the discussion list. Thank you for many great ideas, questions, critiques, and suggestions.

We thank all community members who have followed our journey and provided feedback.

What’s in the box?

─  Reference Implementation – a working sample to illustrate many of the concepts applied. The RI is designed to take advantage of the cloud computing—we’ve deployed it on Windows Azure.

Go play with the live system!         

─ The Guide “Exploring CQRS and Event Sourcing” —to complement the RI by describing how it works, what decisions were made during its development, and what trade-offs were considered.

Exploring CQRS and Event Sourcing book

The guide includes three sections:

I. Journey—the chapters in this section follow the chronology of the project to develop the RI; each chapter describes relevant features of the domain model, infrastructure elements, architecture, and UI that the team was concerned with during that phase of the project. as Also covered are how and why particular CQRS patterns and concepts apply to the design and development of particular bounded contexts, describe the implementation, and highlight any implications for testing.

II. Reference—provides a collection of reference material collated from many sources. It is not the definitive collection, but should contain enough material to help you to understand the core patterns, concepts, and language of CQRS.

III. Tales from the trenches—includes mini-cases studies of other teams that describe their experiences with implementing the CQRS and ES patterns in the real world.

How to get it?

Today, the final state of the RI is available as a self-extractable source zip from the Download center. If you are interested in code evolution, please check out the git history.

The guide is published on MSDN. It is also be available in print via Amazon.

How to get started?

There are many different ways that you can benefit from the project.

One way is by reading the detailed notes from the journey or perhaps by jumping to the most important Lessons Learnt chapter.

Another way would be to check out and play with the code—either the final version or any of the previous pseudo-production releases. A series of posts discussing the planning and the releases is helpful.

Make sure to check out the release notes and the detailed installation instructions.

We’ve recorded one video talking about the journey while we were still at an early stage. We plan to record additional videos with code walkthroughs which would be intended to help you familiarize yourselves with the code more quickly. Check out the CQRS Journey landing page on MSDN for updates.

The code also contains many inline comments about additional improvements that could be made to the RI. You are welcome to attempt those and if willing, submit them as pull requests to the repo so others can benefit from your implementation.

More importantly, we hope the guidance will entice you to consider CQRS and ES as viable patterns for your domains, to embark on your own journey of exploration, and to apply them to your systems. If you fork our RI, feel free to blog and share your experiences and link back so others can learn from your implementation.

We really hope the project will have a life on its own within the community. There are many business stories that have not been implemented. There are also technical challenges still to be tackled. Check out the backlog and pick up what you think will help you learn the most. Remember, we, humans, learn best by doing, not just by reading. Some of the learning may come with initial pain. Please don’t be discouraged. CQRS and ES are not cookie-cutter solutions, they require serious analysis, experimentation, and reflection.

What other people are saying about it?

I think patterns & practices has delivered the community a valuable service in the presentation of this guidance. The view point the guidance is written from is both an uncommon and valuable one. It has also really been in good overall exercise for the community in terms of setting the bar for what is being discussed and refining of the vocabularies that people speak in. Combine this with the amount of previously difficult to find Azure guidance and the guidance becomes quite valuable to someone getting into the ideas. ~ Greg Young, CQRS Inventor

This is another excellent guide from the patterns & practices team—real software engineering with no comforting illusions taken or offered. This guide provides a detailed journal of the practitioners implementing a real production system using the CQRS and Event Sourcing patterns, and also highlights the tradeoffs and teaches the principles that underlie them. The topics presented are relevant and useful, especially if you are building highly scalable Windows Azure applications. You’ll be both challenged and inspired! ~ Scott Guthrie, Corporate Vice-President, Azure App Platform, Microsoft

The p&p team’s dedication and hard work go hand-in-hand with the very high level of competency present on the team. Their attention to detail, insistence on clarity, and open collaboration with the community all led to the creation of material representing enormous value to consumers of the guidance. I definitely plan on referencing this material and code in future engagements because I think my clients will derive many benefits from it–a win-win for everyone! ~ Josh Elster, Principal, Liquid Electron

patterns & practices assembled many of the active and key people in the CQRS community to join them on the their journey with CQRS and along the way discovered confusing terminology and concepts that created opportunities for leaders in the community to bring clarity to a broad audience. The material produced is influenced from the results of building a real world application and expresses the experiences from advisors and the patterns & practices team during the development process. By request from the community to allow outside contributions, everything has been open sourced on GitHub. Anyone interested is encouraged to take a look at the guide or implementation. The patterns & practices team has been very welcoming to anyone who wants to collaborate on covering additional areas, alternative implementations or further extending what is currently in place. ~ Kelly Sommers, Developer

Having participated and co-authored various guides from patterns & practices, the "CQRS Journey" follows the same walkthrough, scenario-based style, but adding even more fresh empirical content. It's a true testament of a skilled development team without previous CQRS experience, going through the journey of implementing a complex system and documenting their adventures and lessons learnt in this diary. If I had to recommend to someone where to start with CQRS, I would definitely point them to this guide. ~ Matias Woloski, CTO, Auth10 LLC

For more, please see “What other readers are saying about the guide” section.

How to provide feedback?

If you’d like to comment on the guide, please enter them online or send your feedback to cqrsjourney at microsoft dot com. You can report code issues here.

Start your own journey

The CQRS pattern and event sourcing are not merely simplistic solutions to the problems associated with large-scale, distributed systems. By providing you with both a working application and written guidance, we expect you'll be well prepared to embark on your own CQRS journey. Bon voyage!