Some A/B Testing for OneNote layout

Back in May, I wrote a little about A/B testing for OneNote and used napkin math as an example. We are about do some more of this and it will involve our entire team so I wanted to mention that here.

We are considering a small change to our layout code – the code that control where and how outlines, images and all other page content appears on the page. The change is small but we want to ensure it makes our layout even better. The complication is that even though our change is small, it may consist of several changes made to different areas. Since they all interact with each other – adding an image can cause an outline to scroll offscreen for one example – we need to understand how each change works on its own in addition to how they work when combined.

So what we will do is take our first proposed fix into our dogfood builds and use them for a while. Once we verify the behavior of the change, we can take the second proposed fix, again in isolation. Then we can combine the two once all validation is complete.

The advantage this gives us is that if we see some odd behavior at any given point, we can see if the first change was the likely culprit, or the second change (if that is what we are currently using for our day to day work), or if it is the combination of the two changes together. That makes tracing and debugging much easier for everyone involved. It is always easier to isolate a problem with fewer changes to a system than with many changes to that same system.

I hope this makes sense. It came up this week internally when someone was asking when we would see the combined proposed changes, and I used that opportunity to add some core testing concepts into the conversation. The more testing we can make part of our daily work by making little process changes like this the better we will all be.

Questions, comments, concerns and criticisms always welcome,


Comments (4)

  1. Andy Bryant says:

    Talking of layout testing – have you ever looked into the outlining bug between email and OneNote?

    This has been bugging me for years; and I've yet to find a solution.  Currently on Office 2013 in Win8.1.

    i.e. when I have a numbered or bulleted list in email; and copy/paste it to OneNote – the outlining level of the numbers and bullets all drop by one; so you manually have to go back and re-indent.


  2. John says:

    Let me follow up on that.  Are you using HTML or RTF mail format (or both)?

    And does the same thing happen when you copy/paste from Word?

  3. Andy Bryant says:

    I've just tried both RTF and HTML; and in both cases pasting an outline from email to OneNote results in the odd looking embed.  Pasting the same thing from email into word works fine.

    Word into OneNote fails in just the same way as Email to OneNote.


  4. John says:

    Can you send me an email from the upper right link?  I want to get a reply from you that has the bullets and numbers and repro this bug.