CDF and Grand Convergence

This a lengthy response to "ODF calls time on da Vinci coding .... Still seeking OASIS approval", By Lucy Sherriff of the Register.

Well, for starters, we are no longer waiting for anything from OASIS. That ended in April of 2007. We've moved on to the W3C's Compound Document Format in hopes of solving the same market requirements that we were unable to use ODF for.

It does look like The Register got lost in the acronym alphabet soup. Nevertheless, da Vinci still lives and breathes. We're no longer trying to hammer the OpenDocument format into a task it was not designed for though. A task outside the ODF Charter, and the scope of ODF's purpose. This news should cheer up our friends at OASIS. That we are refactoring da Vinci to support the W3C's Compound Document Formats ("CDF") instead of trying to eXtend ODF to perform a challenge it was never intended for has certainly made our work enjoyable again.

So, what should people understand about this change? ODF continues to do exactly what it was intended to do. MS-OOXML continues to do exactly what it was intended to do. And it looks to us as if CDF can bridge the gap between while opening the world wide webs beyond.

There are some bumps of misunderstanding that must be addressed.

For instance, people continue to insist that if only Microsoft would implement ODF natively in MSOffice, we could all hop on down the yellow brick road, hand in hand, singing kumbaya to beat the band.

Sadly, life doesn't work that way. Wish it did.

Sure, Microsoft could implement ODF - but only with the addition of application specific eXtensions to the current ODF specification.

The MSOffice specific eXtensions might be based on the sprawl of feature sets and business development efforts that distinguish MSOffice from other desktop office suites such as OpenOffice. This approach might result in a thousand pages of application specific element/attribute pairs that are stuck into the ODF spec, but would still not implementable by other ODF ready applications.

Or hey, maybe either MSOffice or OpenOffice would be willing to throw out their current layout - implementation model and adopt that of the other? With 550 million desktops expecting some sort of continuity and backwards compatibility, this is a tall order for Microsoft. Even taller for the 550 million desktops we would be asking to live through the transition.

On the flip side of the coin, Sun has already made it clear at the OASIS ODF TC that they are not going to compromise (or degrade) the new and innovative features and implementation model of OpenOffice just to be compatible with the existing 550 million MSOffice desktops. Their solution is to point out that it's far easier and more sensible to have the 550 million MSOffice desktops download OpenOffice and replace MSOffice. Instant interoperability, with innovative features; as long as everyone is running the same version of OOo.

OK. So we're stuck between a rock and hard place, leaving the file format as the only place to turn to. And even there, concessions and accommodations must be made. Everyone has to give up something if the markets are to get what they really want - a single universal file format useful to all kinds of applications while being bound to none.

Let's dig deeper.

Beneath the feature set - business process development - accessibility add-on layers of MSOffice, what we're really dealing with is basic document structures. The bottom line is that MSOffice and OpenOffice differ at the feature set, feature set implementation model, and business process development models. These feature set – layout engine differentials are expressed through the file format implementations of basic document structures such as lists, tables, fields, sections and page dynamics. ODf is perfectly designed for OpenOffice features and layout engine. And since MS-OOXML is a reflection of the in-memory-binary representation of MSOffice documents, it's also “near” perfectly designed for MSOffice features and layout methods.

A file format designed specifically for one applications features and layout methods is not going to work for another application without some kind of serious flexibility at the document structure layer.

The challenge is to find simplest method of bridging these fundamental differentials with a single file format. For us, the answer was in a simple set of five generic elements for lists, fields, tables, sections and page dynamics. We called these the ODf iX “interoperability enhancements” - or iX for short. (Note, MS-OOXML uses generics to maintain backwards compatibility wherever deprecated legacy issues threaten to shred interop).

These basic document structures account for nearly all “MS-Binary <> MS-OOXML <> ODF” conversion fidelity problems. Although there are many ways to go about eXtending and enhancing ODf with this generic model, our favorite approach was to use the new ODf Metadata RDF feature as the implementation means.

Between July of 2006 and April of 2007, there were five comprehensive proposals submitted to OASIS ODf members for discussion and consideration. Some parts of iX made it to formal proposal vote. Most didn't. We gave up on ODf iX in April of 2007 with the defeat of the “List Enhancement Proposal”, and the gutting of W3C RDF.

Note that ODf iX would only enable us to convert (with a high level of “round trip” fidelity), MS binaries and xml to ODf iX. Application interoperability would still depend on ODf ready applications also implementing iX. This is the only way these apps would be enabled to properly read and write ODf iX documents within workgroup oriented business processes that include (and are most likely driven by) MSOffice desktops.

We believe that integrating into those MSOffice bound business processes is the primary barrier blocking the use of OpenOffice and Linux desktops in enterprises, smb's and governments.

In short, we can't crack the MSOffice workgroup barrier without ODf iX. End of story.

To understand why we turned to the W3C's CDF “Compound Document Format”, and continue to fight, one has to grasp how seriously important this MSOffice bound business process issue is to the future of collaborative computing. If we can't neutralize and re purpose MSOffice, the future will belong to MS-OOXML and the MS Stack. Note the MS Stack noticeably replaces W3C Open Web technologies with Microsoft's own embraced “enhancements”. Starting with MS-OOXML/Smart Tags as a replacement for HTML-XHTML-RDF Metadata.

HTML and the Open Web are the targets here. ODf is being used as a diversion from the real end game – the taking of the Internet.

Before issuing their RFi, “Request for information concerning the feasibility of a ODf plug-in for MSOffice”, the Commonwealth of Massachusetts had conducted a year long Pilot Study on implementing ODf. The RFi was nothing short of than a cry for help. Massachusetts had already mandated ODf, but the Pilot Study fully demonstrated how difficult and costly it would be to implement ODf. Hence the RFi.

Keep in mind that all the big ODf vendors participated in the Pilot, and were unable to implement ODf short of the costly replacement of MSOffice. There were two aspects to this “cost” factor. The first was the disruption cost caused by critical day to day business processes bound to MSOffice. The second cost was that of re engineering business processes for OpenOffice based replacements. The RFi was a clear signal to all that Massachusetts was unwilling to pay those costs.

So after a year long Pilot Study, Massachusetts had come to the conclusion that the only way they could realistically implement ODf was through a clone of the MS-OOXML Compatibility Pack plug-in for existing MSOffice workgroup desktops. Non MSOffice workgroups and HTML based processes could easily move to OpenOffice ODf. (It turns out there were only 24 of those desktops in the Massachusetts government).

The big ODf vendors insisted that such a plug-in clone was impossible. Massachusetts countered that there was no other way forward. Enter da Vinci, first demonstrated to Massachusetts on June 19th, 2006 at an event hosted by IBM. On July 4th and 5th, da Vinci was publicly demonstrated at a EU-IDABC conference in Brussels, Belgium.

Here's a hard fact. Although the ODf 1.0 version of da Vinci was optimized for compatibility with OpenOffice ODf 1.0, it did not meet the Massachusetts “requirements” as determined by the year long Pilot Study. CIO Louis Gutierrez and his Massachusetts ITD group knew exactly what they needed, and were able to explain to us why the requirements were so demanding. We took those requirements and cross referenced them with other interested governments in California, Belgium, Denmark and the EU-IDABC to find just how universal this MSOffice bound workgroup – business process problem was. The Massachusetts position was fully confirmed, and this lead us to revise our “market requirements” for the second version of da Vinci - which had to include the essential ODf iX.

Briefly stated, the Massachusetts market requirements are:

• Compatibility with existing documents - file formats :: including the volumes of MS binary documents.
• Interoperability with existing applications :: including the over 500 million MSOffice bound workgroups.
• Grand Convergence of desktop, server, device, and web systems as fluid and highly interoperable routers of documents, data, and media.

The first two requirements, “compatibility – interoperability” involve the conversion of existing documents, applications and processes to XML. The third “grand convergence” requirement is all about a W3C Web ready XML.

This migration – conversion to XML is currently a primary driver in enterprise, smb and government decisions. The only question is, “Which XML?”

Take a look at the July statement from Sun's Jon Bosak, made when casting Sun's ANSI – ISO vote in favor of ISO approval of MS-OOXML:

“We wish to make it completely clear that we support DIS 29500 becoming an ISO Standard and are in complete agreement with its stated purposes of enabling interoperability among different implementations and providing interoperable access to the legacy of Microsoft Office documents.”

Sounds like our government driven “market requirements” doesn't it? There is no way to interpret this statement other than to say that Sun agrees with Microsoft that ODf does not meet the same implementation requirements as MS-OOXML.

In October of 2007, at the GOSCON Conference in Portland, Oregon, Sun once again sided with Microsoft on the issue of multiple file formats. IBM's Arnaud Le Hors was passionately arguing and making the case of a single file format standard useful for many purposes. Microsoft's Jason Matusow of course argued in favor of multiple file formats, pointing out that ODf and MS-OOXML fulfilled different purposes. The surprise was that Sun jumped in with both feet in support of Microsoft's argument, opposing IBM's single file format theory.

If Sun agrees that only MS-OOXML is able to meet these compatibility – interoperability with legacy Microsoft document and application market requirements, and ODf cannot, what's the point of arguing this any further? Sun controls both the OASIS ODf TC process and the OpenOffice reference implementation.

ODf was not designed to fulfill our three market requirements. And efforts to adapt ODf to those needs have failed. That leaves us with only one option; apologize for trying to make something of ODf that it was not intended to be, and move on.

We apologize and move on.

The fact remains though that the market requirements are real, and they are not going away. The migration to XML is going to happen with, or without us. If we can't convert existing MS documents, applications and processes to ODf, then the market has no other choice but to transition to MS-OOXML. It's that simple.

And at this point, ISO approval of MS-OOXML matters little. With upwards of 85% or more marketshare for MS-OOXML ready MSOffice desktops (using the easy to download and install Compatibility Pack plug-in), the stage is set for a very non disruptive and low cost transition. At the other end of the portable document pipeline sits the Exchange/SharePoint developers Hub, with upwards of 65% marketshare and climbing.

The E/S Hub automagically converts all MS binary attachments to MS-OOXML, with Compatibility Pack enabled MSOffice desktops able to automagically convert them back into application useful binaries. This routine has been in place for two years with most E/S Hub users blissfully unaware of what is going on behind the scenes. Is there anything ISO can do that will cause governments to rip out and replace this burgeoning pipeline?

I mean, that's what it really comes down to. Microsoft is expert at commercializing interoperability to meet their monopolistic needs, and then garnering so much marketshare that becomes too too painful and punishing to consumers for governments to rectify the damage.

The MSOffice <> MS-OOXML <> E/S developers Hub juggernaut is happening right before our eyes. We believe that the only way to stop this Juggernaut is to neutralize and re purpose MSOffice, enabling E/S alternatives like Zimbra, Alfresco, Plone and a host of other open source web applications to have a shot. The name of the game for many Office 2.0 initiatives is to perfect a non disruptive integration into existing businesses processes such that “collaboration” is a value added feature and not a process splitting thorn. For most workgroups, this means cracking into MSOffice with a web ready integration “re purposing”.

Ripping out and replacing MSOffice is a disruptive cost beyond the threshold of what markets will bear. The events in Massachusetts, California and the EU-IDABC prove this. So why not take a page out of the SAMBA / WiNE / Zimbra (Outlook) interoperability with Microsoft technologies book, and make a fight for the world of opportunities that will be within our reach if we act now to break the monopolists iron grip?

The simple truth is that ODf was not designed to be compatible – interoperable with existing Microsoft documents, applications and processes. Nor was it designed for grand convergence. And as we found out in our five years participation at the OASIS ODf TC, there is an across the boards resistance to eXtending ODf to be compatible with Microsoft documents, applications and processes.

CDF on the other hand was designed exactly for grand convergence. No problemo there. The only question for us was the challenge of using CDF to solve our first two “compatibility – interoperability” with existing Microsoft documents, applications and processes requirements. We are very confident we can do it, but it's involved to say the least.

The primary usefulness of converting existing MS documents, applications and processes to CDF is a web ready, W3C approved grand convergence possibility not driven and directed by Microsoft or the MS-OOXML specific MS Stack. This is not a desktop application to desktop application exchange process that falls within the design range of ODf. It really is about neutralizing and re purposing the MSOffice installed base of desktops (all 550 million) to become full participants in an Open Stack desktop, server, device, and web grand convergence model.

We intend on distributing CDF da Vinci as a free download, no strings attached. Hopefully this will help crack the monopolists iron grip, enabling the many Zimbra's, Alfresco's, Web 2.0, Office 2.0, Enterprise 2.0, SaaS and SOA efforts to integrate into MSOffice bound business processes with the same fidelity and round trip advantages applications in the MS Stack enjoy. We might not be able to undo the MS desktop monopoly, but maybe we can slow down the Microsoft's insidious leveraging that monopoly into servers, devices, and the web.

If anyone thinks they can do this with ODf, we urge them to get out of the blogs and out of the standards committees, and get busy doing it. There is no time to waste. Pragmatism rules the migration to XML. Meaning this is a fight that will be won or lost in the real world trenches of converting existing documents, applications and processes to XML. And doing so without disruption or re engineering cost.

Hope this helps,

~ge~ Hey buddy, could you spare me a garage?