OneNote 12 - Working as a team with shared notebooks

First, let me point out that Owen Braun is blogging now. He's the lead program manager on my team responsible for OneNote. He's got a great post up about how we now support multiple notebooks. And he's promising a whole feature on hyperlinks...

Recently I've had some mail from various OneNote users telling me that they are using OneNote for team work - shared sections and folders of notes which every member of the team can read and contribute to, and of course search. I've also been reading blogs like this one. I like this post because sbreck describes exactly the type of scenario we are targeting OneNote 12 at. Of course, he is trying to do it with OneNote 2003, so he has a few issues. It works, but it runs into limits. OneNote 12 has had significant work done to make it a breeze to work in small to medium groups effectively and quickly. Allow me to drone on….

How did we get here?
A lot of people view OneNote as a single-user application designed for personal notes. This is a pretty narrow view of an application we have much grander ambitions for. Of course it is not as narrow as the "it only works on TabletPC" myth (grr), but it's still pretty narrow.

As we started planning OneNote 12 in the summer of 2004, for kicks we dusted off our original vision for it. Right there in black and white in the original vision doc from early 2001 there is a sentence I wrote at the urging of my dev manager to specify what we were NOT going to do in the first release as a guide to the team. It states, "Scribbler [OneNote] will be a single-user application for personal use. Although we intend to make it a multi-user application in the future, our first release will focus on the single user scenario to make sure we get it right".

Our dev team was actually very excited to design into OneNote from the very beginning the ability to treat every page and every little bit of content on a page as a node on a graph that can be modified by any number of user agents. They did not want to make any assumptions in the architecture or the code that required a single user to be doing the modifications to this graph in a particular sequence. So we had to put in a little explicit "Down Boy!" to make sure we stayed focused for the first release.

When we had a chance to do SP1 with a load of features, we enabled real-time multi-user editing of pages. We had already found OneNote a compelling way to enhance meetings - it would be even better if we could all use it together - including people attending by phone. We hooked ourselves up to a convenient peer-to-peer sharing technology that was in every version of Windows we supported: DirectPlay. This is used for multi-player games on LANs, but it also works for note taking apps that work using the same principle of passing changes to the other members in the "game'. I wrote more about this here.

The upshot was that we had a non-trivial multi-user editing surface which could be used in real-time. Multi-user editing of a document is one of those hard problems in computer science. It's been done for simple documents such as text files or pixel-drawing whiteboards, but to do it for something tricky and complex like a OneNote page was novel. I am not aware of any non-research application out there that does this to the degree we do (where any user can edit at once in an unstructured way). There were a few limitations to this approach - namely it was really focused on synchronous multi-user editing (i.e. you are all working together in real-time), so it was not a good candidate for collaboration that lasted more than a few hours.

Another thing we added in SP1 was the ability to open a folder on a file share. We had some customers early on (even before we shipped the final code for the original release) who immediately saw that OneNote was a great place to keep the notes for a team (of, say, lawyers and paralegals) for one or more projects. They had a problem today where only one person could read and access notes and documents at one time with their existing tools, and they needed to move faster. They were quite motivated to make it so each user could see and add to or annotate a shared set of information, such as the case file for a client. It's sort of interesting to have a PDF file in a shared location such as a file share or document management system. More interesting is if you can see the other lawyers' opinions of the various clauses in that contract or brief - written in ink or typed beside the pages, or highlighted passages.

OK, still with me? Here's where OneNote 12 comes in.
OneNote 12 allows something we are tentatively calling "shared notebooks". Other working titles include "Nirvana and you", or "Happiness on a stick". [:)]

With a shared notebook:

  • You can work by yourself or with others - it is asynchronous
  • You can come and go as you please - there is no "session" to reconnect to, and no duplicated pages when you reconnect
  • Everyone can edit at once
  • Everyone or anyone can be offline (no net connection) and still edit
  • Offline edits are synced and auto-merged with the shared notebook when the user gets back on-line (even several weeks' worth of changes)
  • Changes are automatically propagated to a server and thence to any of the clients involved in the shared notebook
  • The contents can be searched instantly since they are cached locally and indexed.
  • Team members can be added at any time and their machines will sync up to the latest bits.
  • Everything written is tagged with who wrote/modified it and when it was created/modified
  • You can query the notebook to show you what was written since you last looked at it - the actual text gets highlighted.
  • Creating a shared notebook is as simple as creating a new notebook and picking a file share or SharePoint (or any WebDAV-capable server) for it - we'll pick a decent default as well if you don't care. If you make a notebook shared, we'll offer to send an email invitation to the team members you specify and that's all there is to it.

Shared notebooks may sound fancy but really they are just the most obvious thing. I can work on page as I would with my private notes. So can you. You can see what I wrote, and vice versa. So can Hans, Reza, Mary and Yuriko. We can continue this way for months if necessary. It's the most natural way to work together. It's like sitting around a big desk with your team and having a space in the middle where there are post-its, documents, notes etc. and you can see what others are doing or have done while you stepped out. There's no barrier to grabbing a doc and commenting on it while some one else writes the conclusion. Or add comments to a post-it note already on the desk while someone else is reading it.

Best of all you can go on a trip and take the whole desk with you for reference, make changes, then when you come back through the magic of software your modifications are just merged in with the work that has been going on. (Ok, you can see that the physical metaphor sort of runs out of gas because this is something you really can’t do without software.) In software there's no bumping into each other and holding a paper at an awkward angle while someone else tries to see it. Its also trivial to search and locate anything in the massive archive you can build up - very unlike paper.

Today when a team works together they have a file share or a SharePoint site where they put files. You can browse to that place and see what files are there. You can't see what's in the files without opening them. You can't see what others think about each file. You can’t access these files offline unless you take them offline manually and lock others from editing them to avoid nasty conflicts when you go to put your changes back. All those barriers go away with OneNote 12.

When I try to describe this in words I usually get a few questions:

  1. Won’t this be chaos? No, it won't, because we've done it and it isn't. Just because you don’t have barriers doesn't mean that people go insane and start modifying everything everyone else is working on. What happens is that everyone is able to READ what other people are contributing on different pages, and maybe also comment on it.
  2. Isn't this just a wiki? Yes and No. A wiki is great because of the same sort of thing - everyone can read what others have added and modify it if it is wrong or incomplete. But a wiki still forces you to get a write lock on an article or page unless you want to deal with the risk of it getting modified while you are making your changes. And then merging the changes is a manual process if that happens. And wikis do not support offline - that's a killer benefit and in fact the really hard part. Not to mention that the OneNote writing surface is much richer than a wiki: pictures, annotations, drawings, ink, embedded documents, etc. Wikis are also hard to get a handle on since they can't be "organized". It turns out that having the notebook structure available is very important to show people what is available and how to use the space.
  3. What if I don’t have a server? Any file share including one you make on your own PC will work. Unlike a wiki, the other members do not have to be able to reach your PC in order to work on the shared notes. As long as you are both on the network at the same time periodically the changes will be synced and merged.
  4. Does this mean I can edit a Word document or PowerPoint with other people at the same time? No, we didn't enable this capability in other applications - they weren't designed for it from the ground up like OneNote was. You can of course put a Word doc or PPT file in the shared notebook and it will be replicated to your teammate's machines like any other content. They can edit the file and the changed document will be synced back to all the machines. There's no protection against conflicts though, so be sure no one else is going to modify that document.
  5. Is this based on Groove/LiveMeeting/Office Communicator/SharePoint/Windows offline files? No. All of those technologies more or less work at the file level. From the end-user perspective they move a file around (not SharePoint, which doesn't move files except when you edit/read), just as Shared Notebooks move Word docs (see previous question). Groove does some bandwidth optimization to move only diffs of docs, but this is below the user level. If you have modified a doc in two places you are on your own. None of these technologies will solve a conflict for you that is inside a document. These technologies are not focused on letting several people edit the same content whenever they want to without regard to what others are doing. (note: A partial exception is Groove, which has a few tools that let you make lists with each other, edit discusisons, scrawl on a simple whiteboard, etc and changes to these are merged automatically. But there is no Groove tool yet for collaborating this way on a document surface - basically because it is really hard to do even for one app, let alone generically for arbitraily complex document types. Maybe in the future now that Groove is a part of Microsoft we'll have some cool synergies...) BTW, of course you can host a Shared notebook on a file server or a SharePoint site - OneNote handles the offline caching, syncing, and merging.
  6. Isn't this just going to cause conflicts all over the place? Um, no. The reason is that we have some clever merge smarts and we also get to take advantage of the scenarios OneNote is used for (vs. say Word or PPT). Usually when you sync you have only modified an area no one else has modified, so the merge is straightforward. If you add a page to a section in a shared notebook, it can be easily merged into that section even if others have added or modified other pages. Likewise if you add a picture to a page that others have edited, that can be slipped onto the page without bothering anyone. If you added item 3 to a list of two items and someone else added item 3 as well before they saw your change, we just make one of the items #4 and continue on. If you delete a word and someone else modifies it, we just say that modifying wins over deleting (partly because deleting is easy to do again, but recalling the modification is harder). The only times you actually see a conflict is when there are two content modifications to the same paragraph - maybe I changed "great" to "swell" and you changed it to "awesome". In that case we will pick one winner (the first one to update the master) and then show a ghosted page that indicates where the conflict edit is. In practice this basically doesn’t happen if you are not offline, and even then it is very rare to work on the exact same text in the same paragraph as someone else - you tend to self-police this. And that is more of a Word scenario where you are "polishing" content and wordsmithing it. It's more likely with OneNote that you are just pasting in content from the web or other documents, adding a picture, building you own list, etc.
  7. Do I have to invite others to the shared notebook - or can I just "collaborate with myself"? Bingo! In fact this is the answer to working on two or more machines with OneNote. You can just set your own personal notebook up to be shared and replicated across as many machines as you own. And that's all there is to it.
  8. Ok, how does this really work? Every copy of OneNote12 keeps a local cache of all the notebooks it is aware of. Periodically OneNote looks to see if the server share where the shared notebook is located is available, and if so, is the notebook there (the master) updated since it last checked? If there is an update, OneNote downloads the part that has changed and applies it to its local copy. Likewise if you make a change, we now autosave to the cache (way faster than autosaving directly to the server!). Periodically we try to push up the changes to the server, again just moving the part that has changed if possible. Until we successfully update the master, we keep around a "baseline" which is a version of the master as it appeared when we last synced to it. That way we know what changes to push up, and more importantly we know what to do when the server copy has changed as well. Once a sync completes successfully we update our "baseline" to reflect the new "truth". Obviously if you are offline we can't sync, but you can keep working merrily along and we'll sync everything later.

Ok, that's fine, but what use is this?
There are lots of uses as I've mentioned above. An example for us was when we were trying to get our plans firmed up for the last milestone of development. Usually that process entails putting specification documents on a server and having devs read those docs and make private comments in a non-shared place. We usually try to have at least two separate devs estimate the work to develop the proposed capability to cross check and make sure nothing is missed. The downsides of this process are that no one can see if the devs have started reading the specs or making comments. No one can see the comments. Devs have to ask questions via email or physically locating the PM, and no one else can see those questions.

For our last milestone we used OneNote 12 and created a shared notebook, using several sections but one especially held a page for every proposed feature. We put a table at the top of each page, and embedded the spec document in question. The table also had a place to list the devs, testers, comments, unanswered questions (marked with note flags), etc. Everyone on the team could see the notebook at any time, even on the bus ride home. You could see which devs had started making comments on which spec. Two devs could comment on the same spec at the same time. We could query the whole notebook to see how many unanswered questions there were and what they were.

There were some neat side effects too. For example, previously we used to put these specs out and dev would say they would estimate them. Now we could actually see that they didn’t start estimating them until the following week (with an integrated development team it is gold to know what is really happening, not just what is supposed to be happening). Some PMs took advantage of that knowledge to put up an updated spec that had more detail - something we had been asked not to do in the past as devs had often started estimating the spec unbeknownst to us.

The most amazing thing was that we were done with the whole exercise and had higher faith in its being well executed in a matter of a week - usually it takes a month. This underlines one of the selling points of this approach - it makes your organization more agile. I was talking to a law firm the other day which is interested in OneNote. I asked them why a firm that charges by the hour is interested in time-saving productivity tools? The answer: law firms that engage in litigation are more interested in winning the case than in hourly fees. Anything that allows them to put a case together and move faster than the competition increases their chance of winning. The CIO told me that he learned something from a General when he served with the military: winning is all about not waiting your turn.