Why all the secrecy?


Alex Brown, a long time member of SC34 and the convener of the upcoming BRM for DIS 29500 provided some additional insight into the ISO/IEC rules behind keeping the national body comments confidential. National Bodies submit their comments as part of the fast track process with the understanding that they will be kept confidential, and because of that Ecma TC45 has to be careful about how public it can make the replies: http://adjb.net/index.php?entry=entry071209-141310

Ecma Secrecy


There seems to have been something of a kerfuffle about the secrecy in which Ecma has supposedly shrouded the standards process. However, the instruction to keep the current to-and-fro between Ecma and NBs confidential came directly from ISO/IEC itself at the October JTC 1 meeting in Brisbane, and is not Ecma’s initiative. It is not Ecma’s responses themselves which are sensitive, but the National Body comments to which they are attached. These are, by ISO/IEC rules, confidential and should not be republished in public. Now, as a matter of fact these comments were published in public for several weeks anyway, but this was an aberration (the current SC 34 web site is not password protected; before the current controversies privacy through obscurity was enough to keep documents confidential). Ecma simply have to follow the rules. And they should have applied to ballot comments on ODF too.

-Brian

OpenXMLCommunity.org Quote of the Day:

Accenture – Portugal

“The adoption of Open XML as the standard will greatly facilitate the integration of information sources (internal and/or external), thus streamlining processes, increasing productivity, and creating new business opportunities.”

– Manuel Mira Godinho – Partner

Comments (111)

  1. Rob Weir says:

    Brian, since the Ecma proposed resolutions, as you and I know very well, are distinct from the NB’s  comments, then why doesn’t Ecma make these public?  Experts in both SC34 (Alex) and Ecma (Jan van den Beld) have now confirmed that the Ecma responses themselves do not require secrecy.  So can we expect them to be made public soon?

    If not, what are you afraid of?

    Of course, as a private organization, Ecma has every legal right to keep their work secret, as Ecma has kept secret almost all aspects of the development of OOXML, from mailing list archives, to meeting agendas, minutes, public comments, etc.  But it is inconsistent to have such a public communications effort, involving Ecma press releases and Microsoft bloggers, promoting the existence of these proposed resolutions, but then have them not be public.

    -Rob

  2. jones206@hotmail.com says:

    Rob, you act like you have no insight into Ecma, but you guys are one of the strongest members in that organization. It’s my undersatnding that you have access to all of the documents from TC45, and you guys even had an IBM employee attend one of our TC45 meetings in Kyoto last week where we discussed issues like this; conformance; and other details.

    The Ecma responses on their own are kind of useless if you can’t see the question that is being responded to. The system is set up so that the people who raised the issues have access to the responses, as do you. We didn’t set up the system, and I don’t know why you keep trying to imply that we did.

    You know what Rob, how about if you just take the questions and responses and post them on your own site? You are a member of the US national body and you have access to all of the materials. If it’s no big deal, then post them for everyone.

    Clearly you have a vested interest here as we’ve seen from your blogs, so I’m sure it’s not an issue of you just not having the time. If you don’t feel comfortable doing that yourself (personally), then you probably shouldn’t give Ecma a hard time either for walking a fine line between following the rules and wanting to be transparent. While you claim that TC45 was closed, we actually provided public releases of the specs throughout the development process, and even had a mechanism set up where the public could voice their opinions. So you had two options to provide feedback (attend the TC45 meetings since you are Ecma members; or use the public feedback tool). Instead you decided to keep all the feedback for the fast track review, which is fine, we can still address your concerns. Are you unhappy with the way we’re responded to your issues so far?

    -Brian

  3. Rob Weir says:

    @Brian,

    More than my website, I think the comments would be better hosted by a neutral site like http://www.dis29500.org/

    That site already has the NB comments from when they were publicly available. (You can’t put the genie back in the bottle there.)  But if Ecma could give their permission for that site to host a copy of their proposed resolution PDF’s, then you would have my thanks (and that of others).

    I wouldn’t want to give the Ecma password out, because the Ecma server is already slow as it is, and I wouldn’t want to put more load on it.  Best to keep that for NB-access only.

    As you said, IBM is an Ecma member.  So I am familiar with Ecma rules, and one of the rules is that the degree of openness of an Ecma TC is determined by that TC.  Ecma neither mandate nor forbids making things like meeting minutes, mailings list archives, etc., available.  That was TC45’s decision alone.

  4. Wu MingShi says:

    Brian,

    While I do agree that Rob’s original posting is misleading as it, IMHO, accused ECMA of keeping things secret when he, being on a ISO committee, should had known that there is a confidentiality clause on NBs’ comment, I really believe that if ECMA could just publish the working draft the way Weir suggested in his first comment on the post, DIS29500 will end up being a better standard, to the charing of Rob but to the advantage of DIS29500.

    Yes, you will get a lot of flame, but stripping away the useless internet chatter, there are a lot of things that is worth discussing, if only to present Microsoft’s point of view on controversial topics.

    With working draft, I mean simply the full text of the standard as it stand at this point in time. There is no controversy on this, no condfidentiality problem, allows problems to be identified and show good will on the part of ECMA.

    Knowning the interest with DIS229500, believe me, if you publish the working draft, someone somewhere will create a "diff" version of both. You probably have a lot of work done for you for free. The actual comments from NBs need not be there and is uninteresting details in this respect.

    For better or worse, the working draft will be given the "SCO" treatment. Believe me, I am sure SCO’s lawyer learn a lot from groklaw.net. So can both side in this debate if we can just get the working draft.

  5. Drunken Sailor says:

    Brian said:

    "You know what Rob, how about if you just take the questions and responses and post them on your own site? You are a member of the US national body and you have access to all of the materials. If it’s no big deal, then post them for everyone."

    To which Rob Weir responded:

    "I wouldn’t want to give the Ecma password out, because the Ecma server is already slow as it is, and I wouldn’t want to put more load on it.  Best to keep that for NB-access only."

    A fine piece of sophistry.

    Nobody is asking for the ECMA "password".   That seems retarded.   So why not go ahead, download the documents from the NB servers or the ECMA or whatever servers (the ones that IBM has access to, and you confirmed) and post them yourself.   Am sure IBM can afford the bandwidth, and if not, just say so here and am sure we can get you the funds to pay for 9.99/month over at Dreamhost or GoDaddy to host the content (hurry up, Dreamhost has a 5.99 sale!)

  6. Reggie says:

    Watching Rob Weir reply to these posts makes me think that ODF has a serious problem if he is sitting on their TC… it makes me also wonder why US V1 hasn’t gotten rid of him either.

    Rob (Grima Wormtounge) Weir is a deceitful liar. look at him squirm his way out of his mistruths by commenting here. Rob, your mother really did love you. There’s no need to take it out on everyone here.

  7. Rob Weir says:

    I must follow the official fiction that Ecma TC45 owns these documents and not Microsoft alone.  I must pretend that there is a democratic committee there, and this isn’t just whatever Microsoft wants.  This it is insufficient for a single Microsoft employee to suggest that I put them up on my web site.  

    This is true, isn’t it?

    Since Ecma legally owns these documents and their copyright, I ask again for Ecma to allow these PDF files to be posted on other websites: mine, dis92500.org, wherever.

    What are you afraid of?

  8. Drunken Sailor says:

    Of course, would the TC45 or Microsoft publish the documents, we would see a post on Rob’s site in big bold letters stating:

    "OMG OMG MICROSOFT VIOLATES ISO RULES BY PUBLISHING RESPONSES THAT WERE SUPPOSED TO BE PRIVATE KTHKSBAI"

    Followed by the regular smear with lack of details and half-truths that we are used to.

    Rob, if you want to get those documents published, why dont you take this with ECMA?   Your company is a member as much as Microsoft is.

    I await your non-response.

  9. Rob Weir says:

    Brian is on Ecma TC45. I am not.  I think I’m raising this request to an appropriate person.  Similarly, if you had a request of the ODF TC, to make particular information available, it would be appropriate to ask me to bring this request forward.

    Note that, as I explained before, Ecma TC45 determines how open the process is, not the overall Ecma membership.  

    Remember, the consistent interpretation, from me, from Ecma and from SC34, is that these proposed disposition are not covered by ISO rules.  They are Ecma’s to do with as they will.  My request remains that they make these public.

    What are you afraid of?

  10. Sour Grapes says:

    @Rob

    You said "I wouldn’t want to give the Ecma password out, because the Ecma server is already slow as it is, and I wouldn’t want to put more load on it."

    What a lame response. I expect better from one whose sole purpose in life seems to be to bash Microsoft in the standards world.

    What a joke!

  11. Wu MingShi says:

    Dear Reggie

    "Watching Rob Weir reply to these posts makes me think that ODF has a serious problem if he is sitting on their TC… it makes me also wonder why US V1 hasn’t gotten rid of him either."

    I do not know him personally. If he do use his "web style" in the committee stage, I will be "all hands to battle stations" if I were proposing something through any committee that he does not like.

    However, it is not unknown for people to have multiple-personality disorder, ie, one style on the web, another for committees.

  12. hAl says:

    Rob, I asked you, a member of the OASIS TC for Opendocument, several times already why OASIS has not submitted Opendocument spec 1.1 to ISO yet.

  13. Robertlilly says:

    The fact is Drunken and Reggie, these documents are not published which makes for a "closed" process. The suggestion not to host them on ECMA due to slow servers seems completely reasonable. Find some place to publish them and open up the process. This is a fundemental element of a stardard being Open, which is, it is created in an open enviroment.

    The 2 Chairmans of the ECMA TC45 are both Microsoft employees, they are calling the shots more than likely and can get this done ASAP if they want..

  14. Rob Weir says:

    @Sour Grapes, I recently downloaded all 1766 current proposed dispositions from the Ecma website.  It took over 10 hours to do so, even using automation.  This single server is clearly not up to the task.  That is my basis for suggesting that these documents be mirrored elsewhere.  Consider also latency delays — A website in Geneva is not going to give adequate response time in Asia or the US West Coast.  Anyone who has attempted to distribute information on a global scale knows the importance of having distributed mirror sites.

    @hAl, Yes, I remember you asking about ODF 1.1.  I thought I responded.  But let me give it another try.  We are committed to performing maintenance on ODF 1.0 according to ISO rules, and this includes responding to error reports from NB’s and producing technical corrigenda within an ISO timetable. We also have a responsibility under OASIS rules to respond to comments and defect reports submitted via the OASIS public comment list.

    We are in the process of producing a corrigenda document for ODF 1.0.   My prediction — not a guarantee but my best guess — is that this will be ready to submit to ISO in Q1 2008.  It will contain corrections suggested from multiple sources.  It will not be identical to   ODF 1.1, but may contain fixes from ODF 1.1.

    If you monitor the ODF mailing list archives, especially agenda and meeting minutes, you will be able to track the progress of this effort.

  15. Bruno says:

    Ugghh.

    More sophistry from Rob Weir.  But it’s not even *good* sophistry, because it’s so transparent.  It just never stops with this guy, the lies and deceptions just keep spewing forth from his mouth and keyboard.

  16. While you guys are all chatting about how sophistic Rob’s points are or are not, I’d like to ask Brian if he feels that closing the TC45 work is a good thing when elaborating an open standard (remember, drafting the answers to the NB comments is part of the development of an open standard).

    This is a very important issue, imho.

    Charles.

  17. skc says:

    Charles, it depends on whether or not that’s Brians (or Microsofts) call. Are you asking the question to the right person(s)?

  18. @skc: Since Brian is on the TC 45, I guess he’s the right person who can tell us why the TC 45 works and pages are being locked.

    Charles.

  19. Luc Bollen says:

    "I’d like to ask Brian if he feels that closing the TC45 work is a good thing when elaborating an open standard"

    It seems to me that the obvious conclusion is that OOXML is a closed standard, not an open standard.

  20. hAl says:

    It seems that Luc now declares it an obvious conclusion that all ISO and W3C standards are closed standards and not an open standard because the TC committee work in those organisation is also not published openly.

    Even not all OASIS TC’s seem to do that.

  21. Luc Bollen says:

    @hAl : Indeed, but I see nothing wrong with being a "closed standard".  It is just a choice made about the way of working.  The issue I see is to work in a closed way, but to pretend to work in an open way, as Microsoft is doing.

  22. hAl says:

    @Luc

    That is probably because just about everybody in ICT considers those ISO and/or W3C standards as open just like OOXML is open. If seems like opponents of OOXML are trying to redefine open standard to fit only ODF.

  23. hAl, Luc,

    What it boils down to is confusion about what constitutes an "open standard". Most of the anti-OOXML-lobby seem to mix the standard itself with the development process of the standard – hence dismissing just about everything from being called an "open standard" than ODF.

    To be clear: OOXML /is/ an open standard just as ODF is. ODF was developed in a more "open manner" than OOXML, but this does not make OOXML closed in any way.

    /Jesper

  24. Luc Bollen says:

    It seems like nowadays "open standard" is used by many people simply as a synonym of "standard"…  

    There is obviously a lack of common agreement about the definition of "open standard".  But the the processes in place at ECMA and at OASIS have clearly different degrees of openness.

  25. Luc,

    Yes – this is clear to me as well. Due to intense lobbying the definition of an open standard actually made it into the Danish parliamential decision "B103" where it now defines an open standard as

    1. well documented with complete documentation publically available

    2. freely implementable without political, economical or legal limitations

    3. standardized and maintained in an open forum  (a so-called standardization organization) and with an open process

    It is a major victory for the anti-OOXML-lobby that this definition made it into the decision in Denmark … I just find the 3rd bullet a mistake and it has nothing to do with a standard being "open" or not.

    And something to ponder: Does open maillist and meeting minutes automatically provide openness and transparency … or do they just provide the perception of openness?

    /Jesper

  26. @hAl:

    since when does the OASIS close its TCs? Being a member of the OASIS I find this allegation quite interesting…

  27. Ian Easson says:

    There seems to be a massive confusion about the use of the use of "open" in "open standards".  All (non-defacto) standards are open, by definition, since they publically provide a detailed description of how something works.  Attempts by anti-OOXML people to say that OOXML is not "open" are nonsensical, since it is an accepted ECMA standard.

  28. hAl (and others),

    I am currently looking into MathML and SVG and their application in ODF, so due to your comment I tried to find relevant information about the WG-activities of the two W3C-standards (I have looked into MathML WG at this moment).

    "Open standard" is usually thought of as providing transparency and openness in the process itself and Rob has often touted ODF as providing all of the above in terms of mail lists, meeting minutes etc (I don’t agree with Rob in this case, but that’s a different story). So I tried to find the relevant artifacts with respect to MathML and SVG, but my efforts have been fruit-less … so far. I have not been able to find meeting minutes nor mail lists for development of MathML – all I have found is the generalt statement/rule at http://www.w3.org/Consortium/Process-20010208/groups.html#three-month-rule where it says that a WG-status report should be provided every three months; but it does not give any requirements for these status reports. Indeed the Charter for MathML states at http://www.w3.org/Math/Documents/Charter2006.html (section: Communication) that the development of MathML is hidden for the public and is archived for member-only access.

    Do you know where I can find this information – or if it is not available at all? Am I missing something here? We will begin implementing support for ODF in our products within a relatively short timeframe, but with all the uncertainty the anti-OOXML-lobby has spread about problems with using standards, that are "not open", we are reluctant to start implementing ODF-support since it is dependant of standards, that are not open.

    … no, seriously, I am just kidding … we don’t see it as a problem … but I am sure you can see my point.

    :o)

    /Jesper

  29. Luc Bollen says:

    @Ian: There seems to be a massive confusion created by OOXML supporters, claiming that all standards are equally open.  (By the way, if all standards are open by definition, why do Microsoft and ECMA insist that much to call OOXML an "open standard".  Just calling it "a standard" would be enough…)

    @Jesper: I do not agree with you that the 3rd bullet is a mistake.  It is surely important that standards can be "openly used" (bullets 1 and 2), but it is equally important for some standards, impacting a very large public, that the content of the standard is "openly defined" (bullet 3).

    There was (rightly) a lot of criticism of Sun for their lack of openness in the definition of Java.  The same apply to Microsoft for the lack of openness in the definition of OOXML.

    Microsoft is of course allowed to define alone the format of the files used in their products (they did so for 15 years).  But they should now choose whether they want to retain control of the format (which is fine for  a standard published at Ecma level), or they want to make it an "openly defined", "openly used" standard, as they are currently claiming.

  30. Luc Bollen says:

    @Jesper: if you enter "mathml w3c mailing list" in Google, the 3rd entry shows : "Public discussion of MathML and issues of support through the W3C for mathematics on the Web takes place on the public mailing list of the Math Working…"

    With a couple of additional clicks, you end up at http://lists.w3.org/Archives/Public/www-math/, containing the mail archives starting from April 1997.  This took me less than 30" to find.

    Hope this help.

  31. marc says:

    >It is a major victory for the anti-OOXML-lobby

    >that this definition made it into the decision

    >in Denmark …

    yes, and it is a major defeat to vendors and partners who collect $$$ abusing monopoly positions

    instead of just work to support open standards like ODF  in its products, they prefer to spread FUD and simplificate this debate as a

    "anti-ooxml" versus "pro-ODF" thing

    the worst thing is that some of this people participate in ISO SC… like Stocholm

    really worring

  32. Luc,

    Well – I simply beg to differ. I agree with you that it is nice if an open standard is also developed through an open process, but it is certainly not an requirement for it to be usefull. Good examples of this are PDF which is (now) an open standard and is today an even bigger monopoly than MS Office-files. Yet – it was not developed in the "open". An open process is clearly in the "nice-to-have"-category and does not have anything to do with an open standard being usefull or not.

    :o)

  33. hAl says:

    @marc

    It is rather scary that you feel that there is a need to ‘defeat’ vendors an partners trough use of standardization processes in ISO.

    It should be about the format and not about the party and/or supporting submitting the formats.

    You show that you think that is more important than the standardization proces itself which is not what ISO is about. ISO is not about regulation competetion but allowing competition and not on limiting people by standards but allowing them choice through standards.

  34. Luc,

    Thanks for your reply.

    I found this list as well but it is not the one I am talking about. I am talking about the mail list that the members of the WG use to support the development of MathML, and it is not public – as far as I can see. The section I referred to says:

    "The Math Working Group communicates via the archived member-only member-math@w3.org mailing list. During development, all working documents are visible to the W3C membership."

    The public mail list is fine and it provides a way for you and me a channel for feedback – but the development of OOXML is being bashed because there is no insight into why decisions were made and how they were made. The content of the public mail list of MathML does not answer these questions.

    (at least it does not seem like it to me)

    :o)

    /Jesper

  35. marc,

    "the worst thing is that some of this people participate in ISO SC… like Stocholm

    really worring"

    I wonder if you have the slightest idea of what goes on in the meetings in ISO and the various NBs? Between you and me … most of the work is really, really, really boring.

    :o)

  36. Luc Bollen says:

    @Jesper:

    "Good examples of this are PDF which is (now) an open standard and is today an even bigger monopoly than MS Office-files."

    PDF is surely (and has always been) an "openly used" standard, but I would not call PDF an "openly defined" standard, even today.

    And I strongly oppose the concept that PDF is a monopoly : don’t confuse a successful standard and a monopoly company.  No company has a monopoly build on PDF, and this is showed by the dozen of independent implementation of the PDF standard.

    "An open process […] does not have anything to do with an open standard being usefull or not."

    I fully agree with you (this happens :-), and I never implied the opposite.  A perfect example of this is PDF : very useful (and successful), despite a closed definition process.

    "there is no insight into why decisions were made and how they were made"

    The W3C charter you pointed to says "In support of public accountability, the Working Group will periodically make public a summary of all technical decisions made since the last public summary, and the rationales for these decisions."  This is probably the public information you are looking for.

  37. Luc,

    "don’t confuse a successful standard and a monopoly company.  No company has a monopoly build on PDF, and this is showed by the dozen of independent implementation of the PDF standard."

    Yes – and where we now have a situation where Microsoft has a dominant position in the market of Office applications (they are the only ones who can genuinely create "real" binary Office files) with OOXML we will be able to move to a market with "dozens" of independant implementations – of which several has already been implemented. By denying OOXML in ISO, we all loose since we will be putting OOXML back into the hands of Microsoft and ECMA, effectively saying: Sorry, we don’t want it – you can shove it where the Sun don’t shine for all we care.

    Luc, OOXML is a unique opportunity to create a market of equal Office implementations without loosing compatibility with existing binary documents. I know that some would like to portray it as Microsoft being the only ones able of implementing OOXML … but of those I cannot help to think: "Seriously, have you even looked at the spec – other than the crap posted at Grocdok"?

    About MathML: You are trying to "pull a ‘Rob’ on me" by beating around the bush :o). The periodical summary of decisions of MathML WG is already in effect with ECMA, where they explain changes and the reasoning for it. I am sure you know that’s not what is being asked for be the anti-OOXML-crowd. What MathML WG is doing is merely, once every three Months they come down from the mountain with a list of changes. They explain them and the reason for making them … and then they go back on the mountain. What is being requested by ECMA is insight into the decision-process and not just the decisions + some well-fitting reason …

    Either way I read about MathML WG it seems clear to me that the decision process is closed and the development process is closed … exactly as it is with ECMA.

    Btw, this is not a "bait and catch"-tactic of mine. I genuinely would like to be proven wrong that MathML (and SVG) are closed standards.

    :o)

    /Jesper

  38. Luc Bollen says:

    Jesper, we are making progress !  You explain that MathML and OOXML have similar development processes (this should perhaps be qualified…), and you conclude that MathML and SVG seem to be closed standards.  So I assume we both agree now that OOXML is also a closed standard.

    "OOXML is a unique opportunity to create a market of equal Office implementations"  Do you seriously believe that Microsoft is fighting so hard just to be sure to loose control of their file format ? That’s the whole point…  

    "Seriously, have you even looked at the spec"

    Yes, I have, and I noticed they are really of a poor technical quality.  MS assembled in a hurry the existing documentation (they are even requiring the use of a SharePoint Server in the normative text! see Part 3, section 4.5.1)

  39. Luc,

    "So I assume we both agree now that OOXML is also a closed standard."

    No – my conclusion is, that if you say that OOXML is a closed standard, then MathML and SVG must also be closed standards per deduction. I would then question the openness of ODF, since it heavily depends of both MathML and SVG.

    I have no problems with MathML nor SVG, though, – even if they clearly do not fit into "bullet 3" of the open standard definition. I care about my posibility of implementing the spec and not so much about how it was developed.

    "Do you seriously believe that Microsoft is fighting so hard just to be sure to loose control of their file format ? That’s the whole point…  "

    There are more aspects at stake here. I don’t think you should underestimate the importance of OOXML getting ISO-approval. The anti-OOXML-crowd has effectively created a cardinal point that any standard should have the ISO-stamp – regardless of quality. If OOXML doesn’t get ISO-approval, OOXML will always be on its heels and hence the entire OOXML-eco-system. I’m not sure what the English term for this, ut it could be something along "Who gets first dips". So yes – handing OOXML to ISO means that Microsoft looses control over it but at the same time it preserves their position at the playing field.

    "Yes, I have, and I noticed they are really of a poor technical quality. "

    Ok – but if it’s that bad – shouldn’t we let the market decide which standard to use? As hAl said, ISO is not about limiting competition it’s about enabling competition.

    About the SharePoint-thingy: I am sure I have seen a paragraph somewhere in the spec about which parts are normative and which are not, but I cannot find it. Are you sure, that the section you refer to is not informative? Chapter 4 in itself is marked as informative.

    :o)

    /Jesper

  40. marc says:

    >@Jesper:

    >"Good examples of this are PDF which is (now) an open

    >standard and is today an even bigger monopoly than MS

    >Office-files."

    monopoly!! i would like all monopolies were like PDF…

    i have lot of  options in my linux box ( linux mint Daryna ) to see PDFs, all free:

    evince

    kpdf

    gv

    xpdf

    ocular

    adobe reader

    etc….

    all with  near 100% pixel-level equal rendering

  41. marc,

    Well – the only to make sure you never get there with various independant Office implementations on Linux is to bury your head in the sand and say no to ooxml. Having OOXML in ISO gives you a choice that you would otherwise not have.

    /Jesper

  42. A says:

    marc,

    You’re comparing apples and oranges.  It’s a lot easier to get 100% pixel perfection in PDF seeing as each character has it’s position on the page stored.  It’s more a vector format than a document format.

    True document formats such as Word are flow based.  Thus, if you change the font on your machine (i.e. change nothing within the file itself) the format of your document may change.  Nothing, and I mean nothing, is absolutely positioned…well ok, except those things that are anchored to page, but only those on the first page.

    I’m not saying that the fact that the same document could look different on two different machines is a good thing, but have you ever tried to edit a PDF by hand?  They each serve a different purpose.

    Word doesn’t store pages.  Word doesn’t store character positions.  Even object positions depend on the the position of the character to which it is anchored.  This is why it is so very hard to implement right.  Get one thing wrong at the beginning of the document and the rest just fall apart.

    A better example is this – Can you find me two independent applications that support ODF 100% accurately? And if you can, can you find 6 (the number listed for PDF viewers)?  And I don’t mean 6 viewers, I mean 6 near 100% perfect viewers.  I suspect you’ll have difficulty with the first challenge let alone the second.

    Besides PDF has been an standard for a while, OOXML hasn’t even been around for a year yet.  Give people time to implement it!  Brian has posted several times about which applications support OOXML.  Granted, I doubt any are 100%, but for the same reason you can’t find multiple viewers with 100% support for ODF too.

  43. marc says:

    mmmm… i don’t understand your way to argument…

    i mentioned PDF to counter-argument this assertion:

    "jesper said:

    Good examples of this are PDF which is (now) an open

    standard and is today an even bigger monopoly than MS Office-files."

    and you throw me a lot of arguments to convince me that ODF can’t be 100% reproducible in +2 implementations

    what have to do with it!!!  :-)

  44. I don’t want to abuse Brian’s space with too much discussion of MathML, but since it’s been mentioned a few times in the comments here:

    It is true that we have a member-only list (w3-math@w3.org) but the official comment list for all drafts is public www-math@w3.org. So the comments and all resolutions of them are supposed to be in public.

    Personally I’m happy to have a completely open list, and having taken the XML entity declarations out of MathML into a separate spec, the editor’s drafts for that are in public, although that does lead to some logistical problems in clearly marking the drafts as being editor’s drafts not w3c publications, and then making sure that those warnings are removed during the final w3c publication process. We published three relevant documents last week (I think that the comment system will link to my blog which has details).

    Basically the W3C rule is that while a document is on track to become a recommendation the Working group is supposed to publish a public draft of it at least every three months. They are not required to publish meeting minutes and internal mail archives, except to w3c members.

    I don’t want to argue whether that qualifies as open or closed, or is good or bad, certainly not in this forum, but just stating what (my understanding of) the position is, as one of the editors of mathml.

    David

  45. Dave S. says:

    @A – "You’re comparing apples and oranges.  It’s a lot easier to get 100% pixel perfection in PDF seeing as each character has it’s position on the page stored.  It’s more a vector format than a document format."

    Portable Document Format is a document format. It has vector, text, and bitmap capacity as well as bookmarks, page ordering, and many other key features.

    PDF does not require a position for every character. That is up to the creator of the PDF file.

  46. Dave S. says:

    "OOXML is a unique opportunity to create a market of equal Office implementations without loosing compatibility with existing binary documents."

    There is no compatibility between MSO-XML and the MS Office binary formats. MSO-XML requires new applications to process it.

    There is also little way to create equal Office implementations in much the same way there is little way to start a new car company in the US that makes cars just like the ones that are out there. It’s not impossible, but you won’t have many investors.

  47. Dave S,

    Backwards-compatibility goes the other way (from the binary files to OOXML).

    I’m really surprised that even after all of this time, all of these discussions and all of these blog-entries, there are still someone out there, that hasn’t understood this crucial point.

    PS: I tried to write the same to you on Doug’s blog at http://blogs.msdn.com/dmahugh/archive/2007/12/13/open-xml-links-for-12-13-2007.aspx, but my comment has not been released yet. Granted, my words there were not so "polite" as here, so it might have been banned.

    :o)

    /Jesper

  48. David Carlisle,

    Thanks for your input on the process of your W3C-WG. I’m not sure whether anyone will be able to shout "victory!", but it is always good to get information directly from "the horse’s mouth".

    :o)

    /Jesper

  49. A says:

    marc,

    I only brought it up because you said PDF was a "good" monopoly but implied OOXML a "bad" one.  In return I just wanted to point out that it is easier (though by no means easy, PDF is a huge format) to support PDF 100% than Office, since it seemed to be a criteria of yours that the support should be 100%.

    It’s also been around longer, so no surprise that more people support it, and more people support it better.

    Thus you cannot yet compare the "goodness" of the new OOXML monopoly to that of PDF’s.  Give it a few years and then we’ll see.

    As a side note, you did know that everyone can implement PDF *except* Microsoft?  I don’t know if they’ve backed down since but they threatened to sue when they first heard Office 2007 would support PDF.  Though I understand why Acrobat would do that, and will not comment on whether they should or not, it kind of negates its "openess", no?

    DaveS,

    You are probably right, but we’ve never encountered it.  But when such is the case, does it reformat the document completely whenever the font changes (and thus possibly the width of the characters and thus line breaking and eventually page breaking)?  We compared it to document and CAD formats and ended up lumping it under CAD since it was more similar to that. CAD formats like Catia also support text, bitmaps, pages, etc… but you’d be hard pressed to convince me it is a document format!

    And I didn’t say PDF wasn’t a document format, I just said it behaved more like a vector format than Word or WordPerfect.  :)

    And we are getting totally off topic here!

  50. marc says:

    "marc,

    I only brought it up because you said PDF was a "good" monopoly but implied OOXML a "bad" one. "

    you ( and jesper ) are redefining the meaning of

    the word "monopoly" applying it to a product or

    format and not to a market actor

    you may say "Adobe is a monopoly", or "Microsoft is a monopoly" , but not "PDF" is a monopoly

  51. Dave S. says:

    @Jesper

    Applications can be backwards compatible, not formats. If an application is designed to read only the MSO binary formats, it will not be able to read the MSO-XML formats.

    Think for a moment on what compatibility means.

    If you are lost at this point, then "Compatible: capable of being used with or connected to other devices or components without modification"

    wordnet.princeton.edu/perl/webwn?s=compatible

    MSO-XML is -not- compatible with MSO binary.

    To suggest the new format can represent structures that are found in the old format moves the discussion to whether that has been independently established. If there is not a one-one correspondence between the old features and the new, then the new format represents a subset of the old features.

  52. Dave S. says:

    @A – PDF doesn’t have text re-flow, at least as far as I know, but it is also not just pasted letter by letter like a ransom note. Get a copy of Adobe Illustrator and see for yourself.

    Vector formats have only vectors in them. It’s like pregnant, either yes or no. Turtle is a vector format. Maybe some from CalComp – I never got their manuals. I don’t recall any others.

    I’m sticking with these definitions for document:

    d. Computer Science A piece of work created with an application, as by a word processor.

    e. Computer Science A computer file that is not an executable file and contains data for use by applications.

    http://www.thefreedictionary.com/document

    Although I can argue the second definition is poor. When a computer document is opened the reader of the document is acting as an interpreter, which makes the document an input to an interpreter. The differences between a document interpreter and the usual interpreters is the document interpreter is a visual code generator and there is low skill level required in the hand coding.

    If you want to create a "Hello World" program in Word, just type "Hello World" and anyone who executes that file on a Word interpreter will see it say "Hello World."

    Other conventional programming languages are similar. I believe the National Instruments software interface is light on the hand coding and heavy on the drag-and-drop.

  53. Dave S,

    "Applications can be backwards compatible, not formats."

    Well, I do not agree.

    "MSO-XML is -not- compatible with MSO binary."

    There is no need to put an emphasis on XML/binary since it is not relevant here. OOXML is backwards compatible with the binary files since each and every setting in the binary files can be persisted in OOXML.

    "To suggest the new format can represent structures that are found in the old format moves the discussion to whether that has been independently established."

    Eh – has anyone said that is was independently established? OOXML was created on the basis of the existing binary format of the Microsoft Office Suite. I thought that was common knowledge and I am pretty sure it is even stated in the charter for TC45. I don’t really see the problem in this since it shares point of origin with several standards like PDF and even ODF.

    "If there is not a one-one correspondence between the old features and the new, then the new format represents a subset of the old features."

    Don’t you mean _super_set_ ?

    /Jesper

    :o)

  54. S says:

    Jesper said "since each and every setting in the binary files can be persisted in OOXML."

    Prove it. Where is the document with the mapping tables?

  55. Ian Easson says:

    S:

    Why the intense distrust on the point of preserving the binary data in OOXML (in another form, of course, i.e., in an XML structure)?

    Are you not aware that this is the *design intent* of OOXML, and that enormous efforts have been expended by Microsoft and ECMA to ensure that this is precisely the case?  

    Are you also not aware that the anti-OOXML lobby has made a big fuss over this preservation of information, saying that OOXML should not have preserved some of the legacy information, because to their minds it made the documents less portable?

  56. S says:

    Nice diversion Ian. When you fall short, try to put some spin. Right?

    Are you by any chance a Microsoft employee?

    Where is the document with the one-to-one mapping tables between the binary formats and the new formats ?

  57. Ian Easson says:

    S:

    My comments were not intended as a diversion or as a spin.  You just seem to be quite uninformed.  If you were informed, you would know that the information you seek is contained in the document known as the OOXML spec.  You can download it from ECMA.  Read it, and then come back to this blog with a more informed approach.

    And no, I am not a Micrsoft employee, and I have no connection whatsoever with them and never have.  I comment in these discussions occasionally when I see that someone (e.g., like you) is arguing from an uninformed, purely emotional perspective.  So far, this lack of information and the emotional zealotry has all been from the anti-OOXML side, but if in the future there are any supporters of OOXML who have their facts wrong or who are acting purely emotionally, I will speak against that as well.

  58. S says:

    Ok, Ian, let’s assume I am uninformed. Now please can you tell me where in the documentation I can find the mapping tables between the binary formats  and the new formats?

    I thought that, because the specs were about XML, that binary formats were barely mentioned at all and, as a result, the chance to see the tables with the mappings is less likely. Perhaps I have overlooked something obvious.

    Now something as big and complex as binary formats could take several thousands pages just to map to the equivalent in the new formats. It’s hard to miss it, in principle.

    PS : I am playing the devil advocate. Now is your turn to show me the stuff.

  59. Ian Easson says:

    S:

    As I said above, the *information* you seek is the the OOXML spec.  I didn’t say that it was in the form of mapping tables.

    Your main point, surely, is that you don’t believe, for whatever reason, that the OOXML format can represent the data that is contained in the binary files.  You can disabuse yourself of that belief by reading the spec.

  60. S says:

    Spin, spin, spin.

    The point : you can’t tell me where the "mapping tables" are in the specs, probably because they are just not there. So I did not overlook something obvious, they are not there, period.

    Perhaps you want to stop making less grandiose statements about backwards compatibility then.

    You have nothing to show.

  61. A says:

    marc,

    :)  Actually, I only called the formats monopolies because you wrote:

    "i would like all monopolies were like PDF"

    Of course I know that only the company that creates the format can be the monopoly.  

    DaveS,

    "piece of work created with an application, as by a word processor"

    Isn’t that pretty much any file then?  Thus even what you would call a vector format would still be a document, no?  A raster image would also be a document.  I’m not trying to argue against you here, I’m actually intrigued by the definitions.

    "computer file that is not an executable file and contains data for use by applications"

    I also agree this one is a little shaky.  Postscript is an interpreted language as well as a file format.

    My POV was a little different.  I saw a document format as something that was at its core text based requiring text formatting on the fly by the interpreter, most things being relatively positioned.  It also flows between pages…or to be more accuate, there are no pages, one merely cuts the flow to create them.  Maybe a better term would be a word processing format.  A vector format being more drawing based and most everything being absolutely positioned and fixed to the page on which it was created (no overflow).  A spreadsheet format being grid based, and mostly numerical, again with no flow between pages.  A raster format being more pixel oriented, generally images, though some images formats are vector based of course.

    This is how I came to the result of PDF being more like a vector file than a "document" file.

    And as for Adobe Illustrator, we’ve found that they may contain hooks for the application to implement some functionality not stored in the file itself.  So though they may be stored in PDF, we can’t show them correctly with normal PDF support without those additional application-specific elements being implemented.  I don’t know if text would fall under that or not.  But I really know very little about Illustrator.

  62. Ian Easson says:

    S:

    Grow up.  Just because the information you seek is not in the format that you demand it must be in, doesn’t in any way falsify the truth of the fact that OOXML is designed for backward compatibility.

    Since you seem to be alone in the world in this point of view, perhaps you can tell us why this is important to you?  I.e, what are you seeking to prove?

  63. S says:

    @Ian,

    It’s the second time in a few comments that you resort to insults. That’s what liars do.

    As for the rest, I’m a technical person. If a vendor going to ISO tells us format X is backwards compatible to format Y, I want to see proof of it. If the vendor can’t offer any evidence of the claim, it’s deception.

    Frankly I love your logic, you said "doesn’t in any way falsify the truth of the fact that OOXML is designed for backward compatibility.". In other words, it’s true because you say it’s true.

    If you show me some evidence, I may be more inclined to believe it. That would be a start.

  64. Ian Easson says:

    S:

    This is the last response from me on this subject.  You say you want proof.  You have been pointed at the documentation.  It’s up to you to read it.

  65. S says:

    Well, if you were just trolling, why did you participate in this thread at all then?

    The first comment I posted was to Jesper who posted a claim about backwards compatibility. I asked proof of that. We haven’t heard back from him  yet, so perhaps he has something for us. But you chose to join the party and made an even bolder statement : you said that not only this stuff is backwards compatible (meaning that you have proof of that too), but the only reason I can’t see it myself is because I’m uninformed : it’s all in the specs.

    But when I ask where are the mapping tables between the binary formats and the new formats, you seem to be unable to tell me where exactly are those mapping tables.

    And then you tell that those mapping tables are not in the specs, at all.

    I think you’ve painted yourself into a corner.

    Now that you resort to insults to exit the issue you created, is rather interesting.

    I’m still waiting for Jesper’s answer, and hope you will stop polluting this thread with bogus comments. Thanks.

  66. Dave S. says:

    @jesper

    If set A has elements that are not in set B then set B is a subset of set A with respect to the elements in set A.

    If MSO-XML does not match all the elements in MSO binary, then it is a subset of MSO binary. So, no, I did not mean ‘super-set.’

    You seem certain that each and every setting from MSO binary can be represented in MSO-XML. You should publish your comparison. I doubt either ECMA or ISO have seen that documentation.

    A good part of describing the settings is to describe what an application does with them. Elements that are added to a new specification requiring the understanding of behaviors that are part of old software do not count as description.

    Were your view of compatibilty correct VHS, BETA, DVD, Blu-Ray, 35mm movie film, and MPEG movies are all compatible.

    Are they?

    It’s a rhetorical question – the only answer is no.

    Someone can make a device that is compatitible with all of them, but they are not compatible.

  67. Ian Easson says:

    Dave S said "If set A has elements that are not in set B then set B is a subset of set A with respect to the elements in set A."

    No, this is not so (basic set theory).  You don’t know what "subset" means. Google on "Venn diagram", and you will see what I mean.  

  68. Dave S. says:

    @A – I would hold that most all formats are document formats – that all those mentioned contain various combinations of document elements. It’s pretty hard to find any of them that are particularly pure. Many bit-map/raster formats also contain meta-data, such as creator, time, date, et al.

    Not necessarily authoritative, but Open Document Format includes word processing, spreadsheet, and presentation elements. These in turn are each likely to hold character, text (strings of characters), bitmap, vector, meta-data, and whatever other elements they cram in there.

    I too am intrigued. There was certainly a good reason for you and/or your compatriots to try to classify document formats. Did you have downstream needs?

    The DoD has been concerned with file formats, because in the past vendor support for formats tended to wane when the company moved to other interests.

    Both media formats and file format can disappear. Do you remember the 20Mb 3.5 inch floppies? Sad it did not catch on. Just too late and CDs swept over them.

    What I missed in my Illustrator explanation was that Illustrator can’t make** relationships that aren’t already in the .pdf, in the sense that were the characters individually placed and appeared to form a word, Illusrator would not know they formed a text string. If the characters are a text string, then Illustrator can edit and reposition it as a string.

    Thanks for your sanity. It’s rare.

    ** Not that they can’t be made – OCR is designed to do just that. I just don’t see that Illustrator can do so.

  69. Dave S. says:

    @ Ian –

    Not so! Basic typing! Do you know I missed the word ‘not’ between ‘is’ and ‘a’? Do you know you noticed it made no sense without the ‘not’? :) Thanks for the find. It’s almost a not-not joke.

    As this was in the context of MSO binary (set A) and MSO-XML (set B) if MSO binary has elements that are -not- in MSO-XML then MSO-binary is -not- a subset of MSO-XML.

    OK- the nots are good.

    For jesper’s edification: en.wikipedia.org/wiki/Subset

  70. Dave S. ,

    "You seem certain that each and every setting from MSO binary can be represented in MSO-XML."

    Yes – but I have no proof for you in terms of a mapping between the binary MS Office file format and OOXML. However, I do not see that it is a problem. In fact, I have not seen a single person utter the same critique as you do here – that alle data in the binary files can be loyally persisted in OOXML. Seriously – not a single person. This is not the same as saying that you are wrong – you actually might be right and you could in fact be the one shouting at the Emperor (Microsoft): "But, he has no clothes on at all". I wish you the best of luck with this. Seriously, I do. This just happens to be one of those times, where I am happy to side with the other "six nines" that are quite content with the claim from ECMA that content in the binary files can all be persisted in OOXML. It’s not so much because I simply eat everything Brian tells me – it’s because my experience with working with the Microsoft Office suites and office files through (endless) upgrade cycles gives me no reason to believe that it is not so. It could of course be that your experiences are different from mine.

    About set theory: We are not talking about the same thing. You see (as far as I can tell) .DOC and .DOCX as to sets with some but not all elements in common. I see .DOC as an inclusion of .DOCX (feature-wise). The illustration at the top right of the Wikipea-article is what I mean with A being .DOC and B being .DOCX.

    And finally: That I cannot provide you with 100% watertight proof in terms of a mapping table between .DOC and .DOCX should really make you happy. It means that all you have to do is find a single feature of .DOC that is not part of .DOCX … and you will have proven me (or ECMA, more importantly) wrong. All you have to do is aquire the specification of the binary file format from Microsoft and do a comparison.

    It should be relatively easy.

    :o)

    /Jesper

  71. Rob Weir says:

    I think we’re all missing the fundamental word play in Jesper’s original statement, "OOXML is backwards compatible with the binary files since each and every setting in the binary files can be persisted in OOXML."

    Take items of the legacy formats like macros, scripts, DRM, etc.  None of these are described by OOXML.  Certainly they can be represented in an OOXML document, via the extension mechanisms provided by OOXML.  But this is not the same as saying that the OOXML specification defines or describes these features.  Ditto for many other items in OOXML which are just binary blobs.

    Now, if OOXML can avail itself of proprietary extensions in order to solve its backwards compatibility needs, then on what principle would you deny the use of the same mechanism to ODF?  So, extension are allowed, then I can say with  100% certainty that ODF, using only the extension mechanisms defined by ODF 1.0, can also persist each and every setting of Microsoft’s legacy binary formats.

    In any case, those who point out the need for a mapping from the binary formats to OOXML have the right idea.  Without that mapping this is all just voodoo with proprietary extensions.  It isn’t interoperability.

  72. S says:

    @Rob Weir : yes, that was what I meant to say, and you have drawn the conclusion. Indeed, there is no reason that ODF+extensions can’t do what OOXML+extensions supposedly do.

    @Jesper : I fail to understand your logic. You said "every single setting in binary files is persisted in OOXML", and when asked to prove it, you say that you say so because you like Microsoft and therefore have no reason to think this is not true. Duh? It is YOU who made that statement.

    Are you by any chance a Microsoft employee?

    Why is it that the backers of OOXML keep making statements that they are unable to prove? We are talking about an ISO proposal, an open standard, therefore anything we are talking about here should be easily verifiable. If we can’t verify the charter of the ISO proposal, all the wrong conclusions follow.

  73. Rob,

    One of the really, really cool things about the blog-sphere is the speed with which it moves. Combine that with not being able to control how the things you say are used – you have a genuine roller coster ride. Seing how my comment has been interpreted I kindda wish I hadn’t been so categorical in my comment about backwards compatibility. Even though I have used this asyncronous way of communicating through quite some time (first usenet, then blogs), every once in a while something you say comes back and bites you in the ass. What comforts me, though, is that I’ll bet you and I will have this in common when the comment you made on my blog to Gary Edwards is more widely known:

    "Working code rules. Paranoid delusional ramblings are a dime a dozen."

    I’ll keep this in mind next time "someone" talks about Microsoft being the only one able of implementing OOXML.

    http://idippedut.dk/post/2007/12/Software-politics-Hyprocrisy-101.aspx#id_2d9a5709-e886-443c-ba8a-dd9fe0a04e6e

    :o)

    Now, you are perfectly right in a couple of the things you said. The first thing is extensibility. Both of OOXML and ODF are extensible so in theory everything could be persisted in either format.

    The second thing is interoperability. I think you and I share the same view of the importance of interoperability – it’s really, really, really important. But do you think interoperability is best achieved by:

    a) Using a standardized document format whose intention it is to be backwards compatible with the exiting amount of MS Office files

    b) Extending a standardized document format whose intention it has been not to be backwards compatible with these files?

    Please also note that I am not in this work to extinguish ODF. I am a "pro-choice" kindda guy that see availability of choice as being to the benefit of our customers (and us, naturally). We have customers that ask us to extend our software with them to use ODF and others to use OOXML. Each of them has chosen the format that they feel fits best in their organization. We actually believe that choice provides the best value and not a "one format fits all".

    :o)

    /Jesper

  74. Dave S. ,

    "when asked to prove it, you say that you say so because you like Microsoft"

    Did I? Reading my comment again I think the words "so much" were tapped in by mistake. Sorry about that.

    "Are you by any chance a Microsoft employee?"

    I think I’ll leave that up to you to figure out – it’s really a no-brainer.

    :o)

  75. hAl says:

    "Indeed, there is no reason that ODF+extensions can’t do what OOXML+extensions supposedly do."

    That is incorrect.

    For instance the packaging mechanism has no extension mechanism. So the OPC packaging in OOXML cannot be done in ODF without altering the specification.

    Also you cannot extend something like MathML because it is a w3c spec so to use the OOXML possibilities which interegate imml and office tags you would have to transplant the entire omml math specs and have ODF with two math languages which would be farcical.

  76. S says:

    @Jesper

    You made a bold statement and don’t back it with anything substantial. Let me repeat it. YOU said "every single setting in binary files is persisted in OOXML".

    Alright, show me the proof. Show me the document with the mapping tables between the binary formats and the new formats.

    "I think I’ll leave that up to you to figure out – it’s really a no-brainer."

    You smell like a Microsoft employee. Beware not to pull a Karl Rove of what you advocate. We have enough alternate realities already.

  77. Dave S. ,

    "Let me repeat it. YOU said "every single setting in binary files is persisted in OOXML"."

    Yes – have I stated otherwise? Am I saying that YOU said it? Have I said that I have a mapping table that I won’t show you?. Indeed I told you:

    "Yes – but I have no proof for you in terms of a mapping between the binary MS Office file format and OOXML."

    What is unclear to you?

    "You smell like a Microsoft employee."

    Ok – I’m not sure if I should take it as a compliment or not. Maybe we should leave this with the jury.

    "Beware not to pull a Karl Rove of what you advocate. We have enough alternate realities already."

    Being a non-American I fail to see your analogy, but since I am increasingly getting the feeling that you are more talking _to_ me than talking _with_ me, maybe it doesn’t matter so much.

    :o)

  78. S says:

    Jesper said "What is unclear to you?"

    You lied and don’t want to be held accountable for those lies.

    Your name will be remembered as a Microsoft shill.

  79. Dave S. ,

    "You lied and don’t want to be held accountable for those lies."

    Accountable – in which way? Do you want to send me to jail (and will I collect $25 if I pass ‘Start’) ?

    Let me repeat what I said:

    OOXML is backwards compatible with the binary files since each and every setting in the binary files can be persisted in OOXML.

    This is really my view of it and this is how I see backwards compatibility. I do not think "backwards compatibility" can be defined any other way (at least when it comes to document formats).

    By now it seems pretty clear that you do not agree with me – well, so be it. I think the world is big enough to hold both of us, don’t you?

    "Your name will be remembered as a Microsoft shill."

    Well, I will see if I can live with that. At least I write as "me" and am thereby accountable by everyone for everything I say. I actually feel good about that.

    Also – about being on a "wall of shame", this is nothing, baby. If you find some of the crap I wrote on Usenet when I was younger – then you can talk about my potential embarassment in relation to current and/or future employers.

    :o)

  80. Ian Easson says:

    By now, it should be clear that "Dave S", or "S", whatever they call themselves, is exhibiting the classic trollish signs, namely, that in response to rational questioning of or disagreement with their point of view, they:

    – accuse the other person(s) of being a liar

    – accuse the other person(s) of being a shill

    – pronounce that the other person(s)will meet retribution later in life for what they have said

    – pronounce that they must be an employee of such-and-such a company

    – pronounce that other person to be a troll

    – consistently misrepresent what other people have said

    I suggest therefore to the participants in this blog that no further responses be given to Dave S / S.

  81. Rob Weir says:

    @Jesper,

    I can’t code to intentions.  I can’t review intentions.  I can’t verify intentions.  I can’t standardize intentions.  Saying that OOXML was designed with the intention of being backwards compatible with legacy binary formats means nothing.  I could say ODF was designed with the intention to stop world hunger and reverse global warming, but unless I can point to the specifics in the documented format that achieve that intention, then you would be right to laugh at me.

    In an open standard, the text of the standard should be sufficiently detailed so that any practitioner in that field can achieve the goals set out in the standard.  Some may struggle or fail for lack of money, resources or talent, but no one should be prevented for lack of information.  That is the key — the information is in the standard.  

    So Microsoft can claim all they want, using circuitous and waffling language, that OOXML was designed with the intent of being backwards compatible with all legacy documents, but the fact remains that the standard does not give enough information to do this.  If allowed the same loose interpretation, the same claims of legacy compatibility can be claimed by ODF as well.

    I find it useful to look at this from the patent law perspective, where a patent, in the US at least, must have sufficient detail that it enables a person having ordinary skill in the art to practice the invention without undue experimentation.  A patent claim that is not backed by that level of details can be rejected for lack of enablement.  I think it is a useful model for reasoning about whether something is sufficiently documented.  We can apply it to standards by analogy.

    The concept of a "person having ordinary skill in the art" is a useful abstraction.  A standard doesn’t need to explain the basics, the fundamentals.  You can assume a basic facility with the field of expertise.  But the standard should be implementable, and achieve the stated claims, without undue experimentation.

    So the question I have then, is whether OOXML — the text of the standard — enables backwards compatibility with Microsoft legacy documents, so that a person having ordinary skill in the art can achieve that without undue experimentation?

    I think you’ll agree with me that OOXML does  not do this.

    @hAl, you say that it would be farcical if ODF had two math markup languages?  But isn’t this what OOXML does by having two vector markup languages, VML for legacy compatibility as well as the new DrawingML?  Would you call that ‘farcical’ as well?  

  82. S says:

    @Ian,

    Jesper did not express a point of view, he made a statement so bold that not even Brian Jones said such thing on this blog. Brian Jones never said that "every single setting in binary files is persisted in OOXML".

    Jesper thus has to back his statement with something factual, a document that he can point to.

    Which he has failed to do so far.

    Because he’s a liar.

    You? You are a troll supporting a liar being nice about a deceptive corporation (Microsoft).

  83. Luc Bollen says:

    @S :

    "Jesper thus has to back his statement with something factual, a document that he can point to. Which he has failed to do so far."

    I think you made a good point, but your insistence to get a proof is now undermining your own argument.

    Jesper said :

    – "I kindda wish I hadn’t been so categorical in my comment about backwards compatibility"

    – "I have no proof for you in terms of a mapping between the binary MS Office file format and OOXML"

    I think this clearly summarise Jesper’s position.  You do not agree with it (nor do I), but personal attack and rude language will not help the discussion.

  84. Dave S != S says:

    @ Jesper,Ian,

    It would be better if you responded to the correct party. "S" <> "Dave S".

    I refrain from abusive methods, preferring instead to examine the chain of evidence to see where it leads.

    An informitive example –

    In the trial of Microsoft, one of the Microsoft reps – a vice-president of something, perhaps – clearly stated on the witness stand that he had been with the team who applied a program that removed Internet Explorer in total, which should not prevent the OS from operating otherwise normally. This witness provided video recordings of the process, which showed the procedure failed and the computer did not work, bolstering the MS contention that IE was critical to the OS.

    The prosecuting attorney reviewed the tapes and the next day asked why an icon (or some similar small detail) had changed on the desktop during the process, but no icons should have.

    The witnees response – an I don’t know. Further questioning showed he had -not- been there during the entire process and knew it and had perjured himself, perhaps by accident – the night of the software install had been a long one.

    Just follow the evidence.

  85. S says:

    @Luc

    It’s a central question, hence why I insist on getting this sorted out. If this question gets answered positively, with the mentioned document, many other questions are solved (some of IP hidden in binary formats, undocumented binary bits, migration scenarios, interoperability…)

    In fact, this is the question that should be answered by THE document, ECMA 376. But since we know Microsoft has paid their way to ECMA (the only organisation able to fast-track whatever piece of crap to ISO), this is where we are today.

  86. A says:

    Dave S,

    We separated the formats by behaviour, as it is difficult to write a single app that will render many format types, as the the underlying concepts of those formats are so completely different.  After all, one could build a spreadsheet sheet as a complex table in a word processor, but it doesn’t really make sense.  So to show a file, we need to know what "format group" that document falls under to know how best to handle it.

    —-

    As to that other discussion, it will pretty much be solved by one person finding one single property that didn’t persist between the formats.  Though I was willing to believe that not all elements survived, I didn’t see anyone proving that they didn’t, just a bunch of people asking the opposite to be proven.  Just because there was no easy mapping table didn’t mean that everything wasn’t there.  It just meant it was hard to tell.

    The purpose of the Spec was to define the format, not to compare with another format.  I would assume that’s why there was no mapping table, no?  Otherwise why not include mappings to ODF, WordPerfect and others?  I’m sure if there was such a table the spec would not get through the scrutiny of the readers of this blog because that table does not belong in the spec.  A no-win situation.

    I think Rob, with his mention of macros not being documented in OOXML satisfies the requirement of proof.  It isn’t in the spec, so the spec doesn’t cover everything that is in OOXML, let alone what is in the binary. So the answer is no, the pure ECMA spec does not support everything the binary did.

    However, the format might still support everything the binary did given those undocumented extensions.  Anyone have an example of something that really didn’t make it?  I’m actually interested to know :)  There were some things you could no longer add through the UI, but you could still convert from an older file.  I’ve yet to find anything that’s just completely gone (but that in no way implies that I believe there are no such features!).

  87. Reggie says:

    "Indeed, there is no reason that ODF+extensions can’t do what OOXML+extensions supposedly do."

    Yeah, and Rob and "S" would be happy to have Microsoft add dozens of "extensions" to ODF, right?  Get real.

    But here’s the main point:

    Rob, you continue to act like ODF is some app-neutral format, designed from scratch as an uber-format intended to serve everyone’s needs. When the reality is that ODF is essentially OO.o XML 2.0, based on OO.o XML 1.0 (OO.o’s website even admits this), and as such, is built around OO.o’s features and OO.o’s code base.  

    If ODF were really a from-the-ground-up pure app-neutral format, then sure, maybe everyone should use it.  But that’s not the case.  ODF is essentially is OO.o’s format, period.  So demanding that Microsoft use ODF rather than OOXML is demanding that the suite with 90% share must use the format of a suite with 2% rather than simply offer their own format to the public.  It’s almost comical.  It’s like Microsoft coming up with a new audio format for Zune, standardizing it, then demanding that iPod support it, and bitching when they don’t.

    And it makes the claims of by Rob et al of Microsoft being "invited" to work on ODF all the more disingenuous.  "Dear Microsoft, We invite you to participate in the development of a format based on OO.o’s current XML format which we will then lobby governments to use exclusively and ostracize anyone that uses a different format."  What a laugh.  Rob, the key phrase there is "based on OO.o’s current XML format".  Take that out, and you have a case for complaining about Microsoft not working on ODF and not wanting to use it; but ODF wasn’t a new from-scratch app-neutral format, and you know it.  And it’s illogical for a suite with 90% share to force its needs into a format based on the suite with 2% share’s previous format rather than standardize their own format.

    Neither ODF nor OOXML are app-neutral formats of purity, neither one has any moral high ground on that score.  Rob and S like to pretend otherwise, but it’s just more lies from Rob.

  88. S says:

    A said "However, the format might still support everything the binary did given those undocumented extensions.  Anyone have an example of something that really didn’t make it?"

    Interesting take. You are asking to let those proposing a standard to be let alone, and that everyone else prove what the standard is meant to achieve. Doesn’t it work the other way around?

    Don’t you think a proposed standard should be clear enough to make everything it tries to do verifiable. And not just verifiable : if it’s really good, it should be easily verifiable. Given that the typical standard is 40 pages long, what does a 6,000 page proposal leave us with?

    As for the standard itself, it’s a migration format. In ECMA 376, in the Scope page 2, it says : "The goal of this clause is to define conformance, and to provide interoperability guidelines in a way that fosters broad and innovative use of the Office Open XML file format, while maximizing interoperability and preserving investment in existing files and applications". Have you noticed "existing files" ?

    Since the mapping (migrates from, migrates to) isn’t provided, the scope is wrong. All the wrong conclusions follow : nothing related to the legacy can be verified. Which also means interoperability is not provided, one of the ISO rules is violated.

    As for other questions that the mapping tables between the binary formats and the new formats would answer, I guess if you were implementing it, it would be a no-brainer why they are so much important. If this stuff is specified, it’s something you don’t have to find or re-invent yourself. Again, that’s what a standard is for.

  89. Dave S. says:

    @A – I gotcha. I feel your pain as it seems another format pops up most every day. So what to do with them? Did you work for Rosetta or Myriad? They built multifunction apps using a modular approach. One app with a hundred little extension programs, each for its own format.

    I’ve had a quarter century with computer graphics. I admire Jim Blynn’s work and when I see a teapot in some odd place in a computer rendered movie I get the joke.

    Most of what I’ve come across is that only the obvious chunks get dealt with, not the entire capability. HPGL/2 offers all sorts of features to the dedicated programmer, but most HPGL interpreters I’ve seen only handle the basics, so it looks like it is vector only. Half the HPGL/2 book is about raster graphics.

    The interest I have about MSO-XML, which MS labels OOXML (named way too much like it is related to their competitors product.) is the repeated assertions that the goal for it was compatibility with the binary formats.

    I’m thinking if a company publicly states that something is their goal and they go for international approval of it as a standard there should be covered, in complete detail, how it does that so that others can move their MSO binary documents to the new MSO-XML format and verify they did it correctly.

    I wouldn’t hold ODF to the same measure, only because their charter is different.

    I will mention that I use MSO-XML because MSO-XML is the best name for the standard, given the origin and the charter. "Open Office" is any office that is open, not much of an identifier unless one means the only product that is so named in the category, Open Office – but that is a competing product.

    Thanks for the sound reasoning.

  90. Rob,

    Come on, you and I know very well that intentions are the very foundation for standards. OOXML was supposedly built with one of the intentions being ensuring backwards compatibility with respect to the existing binary files. You may naturally have doubts that this goal will be fullfilled. One of the intentions of ODF was supposedly to create a document format that provided interoperability between applications and platforms. I have doubts that this goal will be fullfilled for ODF. Who knows – maybe they will both fail on these two merits.

    In a nutshell I think you are setting the bar higher for OOXML than for ODF. You said that

    "In an open standard, the text of the standard should be sufficiently detailed so that any practitioner in that field can achieve the goals set out in the standard."

    Now, let’s compare it to the charter of ODF:

    "OASIS Open Document Format for Office Applications (OpenDocument) TC.

    Statement of Purpose

    The purpose of this TC is to create an open, XML-based file format specification for office applications.

    The resulting file format must meet the following requirements:

      1. it must be suitable for office documents containing text, spreadsheets, charts, and graphical documents,"

    I do my daily work as a consultant for various departments and offices in the Danish public sector and I have to tell you – they are practically run on the basis of spreadsheets. The financial data might  end up in some system like SAP, but the day-to-day processes are done using spreadsheets. Then comes along a new document format that claims that it can handle this information – but the very key in these spreadsheets – formulas – is not a part of the document format. How can you seriously tell me that ODF "is suitable for office documents containing […] spreadsheets"?

    You may be correct that "I can’t code to intentions.  I can’t review intentions.  I can’t verify intentions.  I can’t standardize intentions." – but are you not asking exactly this from us when we try to implement ODF? Isn’t this exactly what you were asking ISO when ODF was put up for ISO approval (we intend to put formulas in later)? All of your arguments are valid when talking about market penetration of a given technology (i.e. let the market decide) but your arguments are not relevant for this ISO-process. ISO has already showed us, that it didn’t care about these things when ODF was put through the process – why should it be different with OOXML?

    :o)

    PS: This is my last day at work before Christmas, so happy holidays to you all and a happy New Year … if I don’t have time to get back in the thread until then.

    PPS: Dave S – I appologize for the misunderstanding of you and S’s "nicks".

  91. hAl says:

    @rob

    <blockquote]I can’t standardize intentions</blockquote>

    It seems you already have with ODF and the spreadsheet formula’s

  92. A says:

    I now have to agree with S on some level, the scope does state that it is meant to be interoperable with the binary, and thus perhaps a mapping table of some sort was warranted, but still perhaps not necessary.  Just because something is difficult to prove doesn’t make it less true.  

    Are there any other standards out there (not necessarily file format related) that have such a charter, i.e. to move from one standard to another, that includes a mapping as to how to do it?  Be interesting to compare.

    BTW, how big is the ODF standard, I assume its more than 40 pages too.  I have seen Brian give proof that parts of the ODF format are not fully documented, so in truth the Spec should be bigger than it already is.  (I know *nothing* about ODF, btw, though I understand they are in the process of filling in the missing bits).  

    Personally I’d rather have 6000 pages with lots of pictures and explanations, than say 600 that basically leave it up to you to figure out the details.  I wouldn’t even mind more pages, much as it’s a daunting thought.  I already don’t have time to read all the useful bits they’ve included (though there are a lot of useful bits they’ve left out, I’m not saying their documentation is perfect!  Far from it.)  But those bits I do read I find very useful.

    Now I’ve implemented a fair chunk of the format thus far and didn’t have any trouble figuring out the mapping myself.  The names were pretty clear in general and I pretty much just looked inside the files themselves and refered to the documentation afterwards to ensure I covered the element’s attributes.  This being such an improvement after staring at random bits and bytes that I generally have no qualms about the improper XML they use (e.g. bitmasks).  It is so much easier than the binary I probably could have done it without the Specs at all.  Keep in mind though, my goal was never to implement each and every feature!  Macros are nowhere in my plans, so I’m sure others have hit obstacles that I have not.

    For most people, they won’t care if there is one missing mapping (depending on what that missing piece is).  And most mappings can be figured out just reading the file internals themselves, not even refering to the specs.  This whole thread is really just nitpicking.

    And yeah, I did think the name OOXML was a little too close for comfort to the competition’s name. I felt that just added to the confusion and was actually a bad move on MS’s part.  Though some marketing team probably thought that the muddied waters would be a good thing.

  93. Rob Weir says:

    @Jesper, the difference is no one is claiming that ODF 1.0 defines spreadsheet functions, whereas you, and others, are claiming that OOXML is represents every feature of the the legacy binary formats.  Do you see the essential difference here?  

    There seems to be a tendency on this blog to make arguments with the following outline:

    1) Microsoft claims X, where X is that Ecma will hand over control of OOXML to ISO, or that OOXML represents every feature of the legacy binary formats, or other such nonsense.

    2) Microsoft’s claim X is refuted

    3) Microsoft fanboy says, well look at ODF, they don’t do Y, where Y is similar, but not identical to X.

    4) But no one every claimed that ODF did Y.

    There must be a fancy Latin term for this type of logical fallacy.  But let’s just call it "the weasel" for now.  You’re avoiding the main issue.  Lying about maintenance,lying about Ecma secrecy, lying about the features of OOXML — you cannot escape from these allegations by pointing elsewhere.

  94. S says:

    @A

    "BTW, how big is the ODF standard, I assume its more than 40 pages too."

    ODF is a proper standard, it only references other standards it is based on, and for this reason the number of pages is irrelevant. I am sure that if you add all specs that ODF references, it will be more than 6,000 pages. As for ODF itself, anytime vector drawings are needed, the specs simply references the SVG specs, which separately defines all you need to know about vector drawings. ECMA 376 uses a spaghetti of crappy vector markup, combined using their secret sauce.

    So if you wanted to compare ECMA 376 and the ODF specs, you have to be astounded by the sheer mediocrity of ECMA 376, which is basically an alphabetical dump of XML elements, which also has the gal to redefine anything it can (such as dates). ODF, however, does not do such thing. It is a high quality spec, building on other standards. It never tries to redefine its own world.

    I’m perfectly fine with Microsoft redefining their own world, as long as they provide the mapping tables.

    In fact, everything would be fine for me if Microsoft was not going to ISO with it. It’s just a proprietary file format, and I can live with that. The ISO "thing" though is a violation of standards organizations.

  95. Reggie says:

    <i>"@Jesper, the difference is no one is claiming that ODF 1.0 defines spreadsheet functions"</i>

    Yet another lie from Rob Weir.  Rob, you and your company are lobbying governments to mandate the use of ODF 1.0, and in so doing, telling those governments that ODF 1.0 is sufficient for all of their needs.  Since their needs include spreadsheet formulas, your company implies to these governments that ODF 1.0 supports spreadsheet formulas.  And a lie by implication is still a lie.  Or do you really expect us to believe that when IBM lobbies a government to mandate ODF 1.0 usage that IBM explicitly says, "Oh, it doesn’t work for spreadsheets"?  I bet you just leave out that part when telling governments about the greatness of ODF 1.0, right?  

  96. Rob,

    "@Jesper, the difference is no one is claiming that ODF 1.0 defines spreadsheet functions, whereas you, and others, are claiming that OOXML is represents every feature of the the legacy binary formats.  Do you see the essential difference here?"

    For real? Is this the best answer you could come up with for my question?

    Let me ask you in another way:

    Do you think it provides for better or for worse interoperability that formulas are not a part of the ODF-specification?

    (Please don’t reply by saying, that ODF was never claimed to provide interoperability for document formats)

    On completely different note – do you have some insight to what is happening with OpenFormula? From looking at their webpage it seems that the work has almost completely stopped with only 23 emails on their maillist in the second half of 2007.

    :o)

    /Jesper

  97. One of Rob Weir’s statements does worry me greatly, as a commercial software development organization.

    We are both IBM Partners and Microsoft Partners.

    "We are in the process of producing a corrigenda document for ODF 1.0"

    ODF 1.0 was finalised well over 2 1/2 years ago in April 2005, actually being approved by the OASIS TC in December 2004.

    I sincerely hope that this process is more towards the end than the beginning, or I will have retired before we have a fixed ODF 1.0.

    We can’t really do too much with ODF as it is anyway, without spreadsheet formula support, you might as well put out a XLS to support OpenOffice users.

    Gareth

  98. marc says:

    Jesper Lund Stocholm said:

    >Yes – but I have no proof for you in terms

    > of a mapping between the binary MS Office

    >file format and OOXML. However, I do not

    >see that it is a problem. In fact, I have

    >not seen a single person utter the same critique as

    >you do here

    well, actually an NB expert group  ( IST41/-/1 from BSI from UK ) asked Microsoft ( aka ECMA )

    in one of its +500 observations provided to ISO as an input to this crazy process, to:

    "Provide such a mapping or remove this claim." in one

    http://www.xmlopen.org/ooxml-wiki/index.php?title=Part_1_-_Fundamentals&diff=343&oldid=261

  99. hAl says:

    @S

    "it only references other standards it is based on"

    Actually that is not nescesarily correct. For instance the drawings parts rely on some SVG compatible elements but the ODF specification refines them with their own namespace on them.

  100. hAl,

    So, I suppose we will see some sort of proof for the bold statement of S that ODF actually only references the standards it utilizes?

    "ODF is a proper standard, it only references other standards it is based on"

    Output form could be some sort of … ahem … table with the references of existing standards on one side and confirmation for each of them on the other side that the refererence is actually only a reference and not redefining the referenced standard.

    ;o)

    /Jesper

  101. Btw,

    I just browsed through the OASIS-ODF spec and the first time SVG is mentioned it says:

    prefix: svg

    Description: For elements and attributes that are compatible to elements or attributes defined in [SVG].

    Namespace: urn:oasis:names:tc:opendocument:xmlns:

    svg-compatible:1.0

    Does anyone know what determines if an element is compatible with SVG or not? I do not see any explanation of this in ODF and neither in SVG (also the parts dealing with extension of SVG).

    It seems that the extension model of SVG (section 23) is basically the same as for ODF (just mark any foreign element and attribute with it’s own namespace), but will this not make any element or attribute "compatible to SVG"?

    /Jesper

  102. S says:

    @Jesper,

    On details, you may be right. But you quibbled to sidestep the big picture. Here is what I mean :

    – ODF is kind of pyramid-shape standard. It uses other standards. In full ISO/W3C standards tradition.

    – ECMA 376 however, is a nuclear-shape standard. It redefines everything just to keep the legacy (and therefore a number of things that should be fixed by now, thanks to better engineering, and existing ISO standards) afloat. Why? There is a difference between preserving old files and preserving them with all internal bugs. In essence, Microsoft is shoving their own mistakes right down the throat of ECMA/ISO. It’s ironic that, given how much different that is from the ODF standard, Microsoft has the gal to say the standard meets a different need (which they’ve really said many times), when all they mean is : we don’t wanna fix our bugs, because that would force us to use non-Microsoft standards, and that is unacceptable.

    The analogy with Internet Explorer is obvious. As Internet Explorer became the dominant web browser (it managed to force OEMs to stop pre-installing Netscape, and so on), Microsoft did not bother fix the bugs in their HTML, Javascript and CSS implementations. Heck, they were the dominant web browser, therefore their implementation became THE standard. So Microsoft kept the bugs in Internet Explorer for 5 years, ruining the lives of web developers out there.

    And then Firefox came out of nowhere, got a lot of traction, which caused Microsoft to ship Internet Explorer 7, cloning a number of Firefox’s features, and fixing some of HTML+CSS issues.

    Which company is being reactionary here? Who is  creating proper standards that benefit all?

  103. First of all Merry Christmas to everyone who reads this blog and happy upcoming New 2008 from me and

  104. S,

    I do not necessarily disagree with you with respect to your picture of ODF as a "pyramid", but what do you do, when existing standards does not actually fit your requirements?

    You basically have three options:

    1. Use the standard anyway and then have limited "features"

    2. Extend/enlarge the standard with your own extensions so it fits your requirements

    3. Create your own stuff.

    (of course you could also do the foot-work to have the existing standard improved)

    ODF has chosen the first two points. Microsoft has chosen point 3. with respect to OMML and DrawingML.

    The tricky thing is – it’s not a black/white world. ODF uses MathML for formulas in text documents, but the caveat is that you cannot make parts of a formula with specific formatting and you cannot (at least I couldn’t figure out how) insert "stuff" into a formula – should you want to. Microsof has created OMML to fit into the OOXML model and it does not at all look like MathML – but then again, they’re not saying it is. Microsoft has created XSL to extract MathML from OMML so the interface to the outside world is pretty sharp. The downside is naturally that we have to learn something new instead of reusing our MathML-skills.

    You can make the same argument with DrawingML. I cannot say why Microsoft chose not to use SVG, but maybe it didn’t fit the requirements they had. So they made something new which is not SVG at all, so there is no confusion.

    To me the bottom line is, that if you find yourself in a position where an existing standard does not meet your requirements, I’d much prefer you to create something new than extend/enlarge the standard but not tell anyone you did and at the same time claim, that you are only referencing existing standards.

    /Jesper

  105. jones206@hotmail.com says:

    I don’t really think that ODF even used SVG. They used concepts from SVG, and reused some element names, but they also have a bunch of stuff in their own namespace.

    If you took an existing SVG drawing, from what I can tell you would not be able to place that into an ODF file and expect it to work.

    The same is the case for the "use" of XSL-FO.

    -Brian

  106. Owner Blog says:

    First of all Merry Christmas to everyone who reads this blog and happy upcoming New 2008 from me and

  107. Doug Mahugh says:

    It’s been quite a year for those who have been blogging about the Open XML file formats. Here’s a look

  108. Embrace and extend – SVG in ODF revisited

  109. Embrace and extend – SVG in ODF revisited