Document Format Legislation - Something To Be Avoided

Recently, a number of elected officials in U.S. state governments have been taking a closer look at document formats. Minnesota, Texas, California, and Oregon are the most active voices today while Massachusetts was the first to start examining the issue. Given the genuine intent that underlies the issues being discussed, and the effectiveness of lobbying efforts by various large commercial interests, I would expect to see more of the same from other governments.

It is important to note that Open XML meets the requirements of openness in these bills. While I am sorry to see the legislative process used as a mechanism for product competition, I am more concerned about the implications of the bills and the precedent they set.

Real Issues Being Contemplated:

The elected officials who are putting forward the three current draft bills are seeking to address three very real issues. All three are laudable goals, and ones we share with the lawmakers.

· Archival: As society has progressed from the exclusive use of physical paper (and related goods) for the exchange of information to the digital world, the issue of long-term archival of electronic data has become a real concern. It is incumbent upon policy makers to retain the long-view and think carefully about how future generations will be able to learn from our labors.

· Interoperability: Governments have an interest in improving the ability of their IT infrastructure to efficiently exchange data in a government-to-government or government-to-citizen context. The data exchanged includes documents, email, images, financial transactions, medical records, and more. Interoperability is the connection of people, data, and diverse systems; be that at the technology (infrastructure, app, data…) or the organizational (org. structure, business process, or legal) levels.

· Communications: All constituents must be able to communicate with government, and the various levels of government (local, state, federal, international) need to communicate effectively with each other as well.

Governments should be in control of their data. They should have the maximum choice of technologies available to them for solutions. Vendors will respond to market pressures, and bring answers to market in various ways. Innovation will continue to take place with the result of increased choice of solutions, competitive pricing pressures, increased functionality, etc. Governments should also have the ability to take advantage of these factors as well.

Factors To Be Considered:

Governments should be in control of their data. They should have the maximum choice of technologies available to them for solutions. Vendors will respond to market pressures, and bring answers to market in various ways. Innovation will continue to take place with the result of increased choice of solutions, competitive pricing pressures, increased functionality, etc. Governments should also have the ability to take advantage of these factors as well.

· Most governments have existing laws in place governing the archival of electronic data. Some gets kept for the long-term while different types may be disposed of based upon set timelines. Archivists at the nationals, state, and agency levels pay close attention to hardware, software, and services that have become available to answer archival concerns. There are solutions out there across a full range of data types.

o Document standards reach back to ASCII Text, through the advent of HTML, and on into XML-based documents. This progression shows how technology continues to move forward with one design feature of many being archival.

o Established commercial players like Microsoft, Adobe, Corel, Novell, IBM, and Sun are actively thinking about archival and interop in their product offerings. Just look at the different flavors of PDF that adobe has produced to understand the market effect on producers of software.

o New comers, like Google, to the document creation market have hybrid approaches as they seek to get a foothold in the market

o There are entire niche markets that have come into existence to accomplish better document management, and archival. There is healthy competition among those players and they are brining ongoing innovation to that space.

· Interoperability is more than just standards. Interop is built into products (documentation, interfaces, protocols, data formats, development kits, etc.), collaboration with other firms to build bridges between systems, providing access to core technologies (through commercial licensing, community licensing, and open specifications), AND through standards.

o This is an IBM argument I hear all the time, interop=open standards. Not only is that statement incorrect, it doesn’t even reflect what they do with their own business. Simply consider the software industry of 1980 with that of today, and you will see that interoperability is the cover charge for bringing a software product to market rather than the completely integrated stacks sold by DEC, IBM, NCR, et al back then.

o Governments should want to utilize the full spectrum of interoperability offerings and not be limited to just one, standards.

· Translation is available today, enabling communications between disparate systems. This is the basis for building a bridge between disparate systems. Software is fundamentally different than anything in the physical world. It is subject to infinite manipulation, and XML has made that much easier.

o Microsoft, Sun, Novell, Google, IBM and others are working with translators to build the capacity for document formats to be exchanged between competing products that provide valuable and needed choice in the marketplace.

o Translation is used every day to enable newer software to reach back and work with software produced 10, 20, and even 30 years ago. IBM is a direct beneficiary of these technologies and what they have enable for keeping their mainframe monopoly alive.

o There are billions of documents out there in formats from all kinds of sources – web pages, simple text docs, complex binary documents (not just from MS) – all of these rely on translation to be brought forward and used effectively today. Mandating a single document format today means that there really only should be one-way translation from old formats into the new, mandated format. Not only is this needlessly limiting, it carries real concerns about cost, performance, and a host of other factors.

The Challenge of Complexity:

In conversations with folks involved with these pieces of legislation, it has been clear that the sheer depth of this issue presents a challenge. There are more points to be made than I’ll put in this blog entry, but here are a few additional thoughts.

· By putting forward a preference for formats as described in the MN bill, lawmakers are being told that it will solve their archival problems. Not true.

o As I pointed out above, most governments already have archival rules in place. The challenges around archive reach well beyond the formats of documents. For any number of reasons, agencies will have chosen specific types of software applications to meet business needs (such as processing fishing licenses), and the output of that data will not necessarily be done in an open format.

§ The language in MN, for example, specifically says “documents including text, spreadsheets, and presentations.” Any lawyer will point out that including becomes a very important word here, and thus all government systems (financial, image libraries, mapping data, etc.) would be within scope of the legislation. This is probably not what was intended, thus showing why legislating in this space is so risky.

§ The precedent set by any of these bills says that in the name of archival, all electronic data should be stored in open, royalty free formats, that anyone can build in the future.

· Does this mean all financial data currently stored by governments in SAP systems? How about data stored in Oracle databases? How about IBM data formats for mainframe, middleware, and database solutions?

· The big guys might be able to do this with no problem, but the harder road is for the thousands of small software providers who sell their solutions to governments who will be forced to either re-develop their products or forcibly open up their product in order to retain their government customers.

· There is no standard use of the word “standard.” There is a definite difference between a procurement “standard” established by the office of the CIO and an industrial “standard” created in a body such as OASIS or Ecma.

o The MN bill came about because of the desire of a state employee to use a different office suite than MS Office. To be clear, I have no problem with someone wanting to use a different product. The challenge for the employee was the fact that his IT organization had a procurement “standard” in place (to simplify purchasing, gain volume purchase discounts, streamline deployment, establish uniform training practices, control long-term management costs, etc.). All large IT shops utilize this type of “standard.” Moreover, vendors work very hard to have their technologies adopted as enterprise standards. This is where product competition is often most pronounced, and where IT professionals are able to drive the best value for their organizations.

o Industrial standards are also part of government procurement, but in varying degrees depending on the agency and the systems they would like to put in place. While governments often have procurement guidance based on standards, it is the implementation of the standards that matter most – in other words, the software that is produced and sold in the market that may include a given standards as a subset of a larger collection of technologies.

· The pace of change matters. Legislation that lays down specific technology guidance will end up out of synch with technology – every time. Laws are made slowly, they change slowly, and they are replaced even more slowly. Within that context, the changes in technology are exponentially faster. Thus, if the state CIO is locked into a decision of what technology to use by law – that makes it very hard for him to select the best technology to do the job from his perspective.

o In the time that these laws are being considered, ODF will go through multiple revisions. PFD will go through multiple revisions. The same may be true for Open XML or any other standard.

o IF the government locks its systems to a legislated format, then it potentially does the same for its citizens. Citizens will inevitably move on to new technologies more rapidly than the government’s mandated solution. This reintroduces communications problems the government is seeking to avoid.

o Another example of specifying technology in the legislation leading to a disconnect is when “open XML-based” formats are the only allowed solution. This precludes the state from using PDF.

· Legislation like the bills proposed in MN, TX, CA, and OR create precedents that would be challenging on many fronts.

o The “royalty-free” text goes to the heart of more than 100 years of industrial standards law and policies. There have been clear examples of royalty-bearing standards being superior to royalty-free specs, and that end up cheaper for the end customers due to market adoption bringing economies of scale to bear. Or, having a better technology in the specification enables a better range of solutions to be implemented and benefit consumers more because of the quality of the technology.

§ This is not and issue for Open XML as it has been released under an Open Specification Promise, and is covered by royalty-free terms in Ecma as well.

§ The risk is in governments not recognizing the value of innovation in any standard and that rights-holders may wish to make submissions to a standard and yet retain commercial value in that submission.

o The language in these bills is essentially saying there has been a failing of procurement practices to be neutral and value-based. I submit that this is not the case. In fact, States have in place strong procurement rules, and most have explicit archival rules in place as well.

§ Governments have long-standing procurement practices in place designed to ensure an open proposal and selection process for contracts/goods.

§ There are many, many factors such as price, suitability for use, minority owned-businesses, accessibility for disabled citizens, etc. that are mandatory factors in a purchase decision. If a law is put in place that reduces choice, and removes these other balancing factors – that concerns me.

Overall, this is an enormously complex discussion. Organizations with commercial interests to sell solutions dependent upon ODF are working hard to create problems that need to be solved…by them. I am not saying that all is perfect and there is no room for improvement. In fact, I would say governments face legitimate challenges on all three issues listed at the top of this posting – but if we look at what is already taking place in the marketplace it becomes clear that using legislation to address these concerns is not only unecessary, but dangerous.