An ODF Quest Grounded by Hard Reality

One of the hard realities is that this particular blog post has been reviewed by a major publication for reprint. Stay tuned.

~ge~
~marbux~

Connecting the dots

Is the Failure of ODF in Massachusetts related to the question, “Is Open Source Dying?”

eWeek has posted an interesting series of articles, “Is Open Source Dying from 1,000 Cuts? Among these is a good “connect the dots” story from Michael Hickins, “Is Open Source Dying?

Michael makes the interesting comment: “State and local governments have latched onto the idea that having their documents hostage to a single vendor, no matter how well-intentioned, might not be such a good idea. Dell recently jumped on the Linux bandwagon and is offering Ubuntu on its PCs. And Microsoft's attempt to have its partly-proprietary OOXML (Office Open XML) format rubber-stamped by a friendly standards body hasn't gone as smoothly as expected. ”

“But behind the scenes, things are not quite as rosy. The Commonwealth of Massachusetts ..... broke important ground by mandating that state agencies switch to open-source platforms. There's just one problem: They can't seem to manage the transition. Sources close to the situation tell me that former state CIO Peter Quinn's resignation happened at least in part because of delaying tactics by vendors who publicly support open source but do their best to scuttle it behind the scenes”.

I wonder if Michael knows just how close he's coming to discovering a dark ODf suspicion that seems to grow by the day. Are big ODf vendors supporting ODf? Or are they trading away ODf Interoperability and implementation for sweet sweet deals with Microsoft?

Could the same thing be happening throughout Open Sourcedom?

I'm hardly alone in my thinking that once again, Sir Richard Stallman has been proven right. The big vendors are trouble. If not today, then likely at some point in the future when the interests of open source communities come into conflict with corporate interests.

Whether we're talking LiNUX, OpenDocument, or patent indemnification - interoperability deals with Microsoft, big vendors are willing to trade the efforts of open source communities as if they were just strategic corporate assets. It's not that big vendors mean opens source communities any harm. It's simply that they treat collaborative efforts in the same way they regard all corporate asset investments. Something that can be used for corporate advantage when the opportunity arises.

Everyone one of the big ODF vendors, with the exception of Google and Oracle, now has or is involved in an interoperability deal with Microsoft.

Sun cut their deal in 2004, and ODF interoperability has never been the same. Novel cut their deal in 2006, and now LiNUX vendors everywhere are being offered plump deals that require them to support and ship the Novell OfficeOpenXML Translator plugin for OpenOffice. The MS-Novell deal promises to put IBM hardware back into competition with Sun, and could be an explosive boon to the PowerPC 6 line of processors. The ability to run Microsoft applications is that important. Even if it is through a VM over SuSE LinNUX.

So far the tech media has focused on the patent indemnification – son of SCO aspects of the Microsoft interoperability deals. The patent threats however involve both LiNUX and, LiNUX desktop darling OpenOffice – which is also the OpenDocument champion. Time to connect the dots indeed.

The 2004 MS-Sun deal indemnified StarOffice, while leaving OpenOffice, the open source version of that same code base, hanging out to dry. Fast forward to the 2006 MS-Novell-IBM deal and we find that the Novell Office version of OpenOffice has been indemnified! Including both Windows and LiNUX versions. Very cool. That ends any thoughts Sun might have had about monetizing their $175 Million investment in the OpenOffice/StarOffice code base by invoking MS patent claims against OpenOffice as a pretext for migrating the entire OOo install – ODf user base to StarOffice.

The interesting thing to note is that with the continuing wave of Microsoft interop deals, we have all these big ODF - LiNUX vendors now supporting the full conversion of OpenOffice to a OfficeOpenXML application.

There's a list of links documenting Novell and Sun support for OfficeOpenXML here.

Meanwhile, many of these same big ODF vendors declined to support the ODF Plugin for MSOffice that the future of ODF in Massachusetts was riding on! After conducting a year long ODf Pilot Study, Massachusetts concluded that ODf plugins for MSOffice were the only way they could make the transition to ODf.

No doubt the plugin approach Massachusetts proposed was in serious conflict with the big vendor business plans behind the many desktop alternatives that bombed in the Pilot Study. So where Massachusetts was trying to figure out how to overcome the MSOffice bound business process barrier to successfully implement ODf, the big ODf vendors were more worried about their heavily invested corporate assets and plans. The thought of leaving MSOffice on the desktop, even if the documents and business processes were in fully compliant ODf, was too much for the big ODf vendor business plans to bear.

With Microsoft bringing on the political pressure, and eventually succeeding in cutting off his budget, Massachusetts CIO Louis Gutierrez had no choice but to turn to the big ODF vendors for support. Thinking that there was no way these vendors would ever let ODF fail in Massachusetts, Louis believed he could overcome the budget problem. He was wrong about the big ODf vendors. They left him hanging.

Shortly after Louis Gutierrez was notified by his selected group of big ODF Vendors that they would not support his community effort to develop an internal ODF plugin for MSOffice, Louis resigned. Of course, the ODf plugin was a comparatively small part of projects impacted by the lack of an approved IT budget. But the impact was big. They had serious HomeLand Security projects hanging in the balance because of Microsoft's determination to stop ODf.

The failure of ODF in Massachusetts has had global ramifications. In California the CIO's asked the question as to whether or not it's even possible to implement ODF? They all know about Massachusetts and the Pilot Study. In Europe, the IDABC has set out to create their own non big vendor - non big vendor standards consortia file format; a highly interoperable fork of ODF called ODEF; "OpenDocument Exchange Format". No big vendors, no ISO, no OASIS, no Ecma, no big vendor standards consortia of any kind were invited to the ODEF table.

So here we have the strange situation where the same big ODF vendors who declined to support an internal ODF plugin that would be able to convert 500 million MSOffice bound desktops to ODf, and save ODf in Massachusetts, are supporting OfficeOpenXML solutions. These same big vendors are making sweet sweet deals with Microsoft, agreeing to provide MS OfficeOpenXML ready versions of OpenOffice.

On the one hand they are against an ODf plugin for MSOffice. And on the other hand they are rushing to provide the OOXML plugin for their shipments of OpenOffice.

Doesn't make sense until you realize two things. The first is that the Microsoft anti trust settlement allowed Microsoft to commercialize the very same interoperability critically important to the goal of creating competitive markets for information technologies. It's hardly surprising that sooner of later Microsoft would be weaving that interoperability into the push and pull deal making fabric of these marketplaces exactly to stifle any threat of competition. First up; ODf and LiNUX. Once the first big vendor fell, the rest were doomed.

2004. Not a good year for ODf and Open Source.

~ge~

Politics is only part of the problem

ComputerWorld: Microsoft legislatively TKO’s open document formats. At least stateside. by ZDNet's David Berlind -- ComputerWorld has a five-page report this morning detailing how Microsoft has managed to score a technical knockout of open document formats (not necessarily the OpenDocument Format) in five out of six states. The story sheds a bit of light on how closely (and in some cases quickly) vendors are working with legislators to sway public [...]

Sorry David, politics is only a small part of the problem.

The question we should be asking is why State CIO's and IT divisions are not backing the legislative proposals? It's not the lobbying that is killing ODF. It's the lack of support from IT departments responsible for the challenge of implementing ODF solutions if the legislation is ever approved. The silence of the CIO's is deafening.

One has to wonder why? I've had more than a few conversations with CIO's and their IT warriors, and there is no lack of enthusiasm for ODF. They would implement ODF in a heartbeat if they could. No legislative mandate necessary.

Of course, it seems not a week goes by without a major ODF vendor announcing some sweet interoperability - patent protection deal with Microsoft. But where's the beef?

Even at the ISO-OASIS ODF Technical Committee level, where the essence of interoperability with MS Office lays within the broader structural enhancement requirements of "compatibility with existing file formats and application interoperability", these issues have been pushed aside as being "out of bounds", "out of scope", "outside the charter", and "that's a problem for converters and translators-let them solve it".

The marketplace of CIO's and IT warriors in California, Massachusetts, Belgium, Denmark, NATO, and throughout the EU IDABC are consistent in their demands. Provide them the interoperability tools they need to migrate existing documents and business processes to ODF, and it will be done. No legislative mandates needed.

There are three quotes i've seen batted about that pretty much say it all:

• ...... "Interoperability isn't just a feature. It's the basic requirement for getting your XML file format and applications considered"

• ...... "The challenge is that of migrating our existing documents and business processes to XML. The question is which XML? OpenDocument or OpenXML?"

• ....... "Under those conditions, is it even possible to implement OpenDocument?"

The challenge for OpenDocument isn't at the legislative level. Nor is it at the International Standards level. No, the challenge for ODF is at the implementation level where there is a serious lack of MS Office compatibility tools and solutions needed to make that difficult transition to ODF.

So we are left with the real world question of whether or not the "demand side" of the information technology equation can get from where they're at today, 500 million desktops bound to MS Office workgroup-workflow processes, to where they would like to be tomorrow? Which is ODF.

Sadly, ODF doesn't get to start with the world as a clean slate. Otherwise this decision would be simple and done. With no amount of political lobbying able to stop real world uptake. ODF wouldn't even need legislative proposals.

But that's not the case. Hardly. With upwards of 500 million workgroup desktops bound to MSOffice bound business processes, we are a long way from the 1995 office suite "feature set" wars of yesteryear. All the new and innovative features sets in the world aren't going to help ODF office suite applications crack into those MS Office bound business processes. What's needed instead is an all out - no compromise focus on compatibility, interop, and convergence issues.

The documents and business processes that make these critical day to day workgroup-workflow tasks possible are going to transition to XML. The question is, "Which one? ODF or MS OOXML?"

Where the rubber meets the road, the challenge for ODF is that of matching doc for doc, proc for proc, the non disruptive cost Microsoft offers with their OfficeOpenXML plugin.

The real world doesn't have the luxury of evaluating OpenDocument and OfficeOpenXmL based on the expert level of proper XML, open standards governance, IPR encumbrances, or reuse of existing XML standards. No, instead they have a bigger problem of getting existing documents and business processes into XML. And from there they can move into SOA, SaaS, and the Web 3.0.

So CIO's are forced by the everyday reality of MSOffice bound business processes to demand from ODF solutions three primary characteristics:

• .... Compatibility with existing file formats (MS Binaries/HTML/XHTML/RTF)

• .... Interoperability(application level - including existing apps like MSOffice!)

• .... Convergence (the portable XML document as the end user interface into information systems that span desktop-server-device-web)..... If your tied to the desktop, you're dead. But if your an XML file format tied instead to something like the Vista Stack, well, you've got a shot as long as the other guy remains a no show.

• .... Harmonization (the worst of all compromises; the successful implementation of the above three characteristics, but on terms dictated by MOOXML.

These are serious questions the ODF community has to come to terms with if the demand side of the equation is to have some sort of choice other than OfficeOpenXML.

That's not to say that the demand side is sitting still. No way. Check out the EU IDABC "Advanced eGovernment" Conference held February 28th-March 1st, 2007 in Berlin. They've got their own "optimized for interoperability" proposal known as ODEF - The Open Document Exchange Format.

You've got to read this stuff to believe it. No big vendors need apply. No ISO either! I take that as a swipe at both the big vendor standards consortia, OASIS and ECMA.

The EU IDABC has even gone so far as to identify the "interoperability break points" that can be found throughout ODF and OpenOfficeXML: the "optional" methods of implementation.

There is a famous quote from the infamous Doc Searls that goes like this, "Open source is where the demand side has taken over their own supply".

Could it be that we are now witnessing the same thing with Open Standards? One has to wonder.

Meanwhile though, The OpenDocument Foundation has made the decision to go with the marketplace - to stick with the demand side. If the EU gives us the ODEF spec, we'll provide them with an ODEF version of our da Vinci plugin for MSOffice. As applications (and application converters :)) move to ODEF, i have no doubt OpenDocument will jump to make whatever compatibility-interop-convergence changes needed. And that would be a good thing.

A very good thing,
~ge~

Three Stages of XML Migration and the Challenge to OpenDocument

1. Conversion Fidelity - the billions of binaries problem
2. Round Trip Fidelity - the MSOffice bound business processes, line of business integrated apps, and assistive technology type add-ons
3. Application Interop - the cross platform, inter application, cross information domain problem

First it was the EU's controversial Valoris Report that centered on a fictional open standard XML universal file format known as “OpenDocument”. This was fictional open standard XML universal file format known as “OpenDocument”.

This was followed by the Massachusetts's pilot study and mandate for the real thing, the ISO/OASIS XML OpenDocument file format. The States of Texas and Minnesota weighed in with similar legislative proposals once again describing a open standard XML universal file format in measures that only the ISO/OASIS OpenDocument can measure up to.

And now comes the crusher. The State of California, touting what would be the world's eighth largest economy if it were a nation, steps forward into the breach with a legislative proposal describing in such detailed parameters what can only be the OpenDocument universal file format that one can almost see the butterfly tattoo on the specifications buttocks.

No doubt the great migration to XML is upon us, with XML universal file formats leading the way. Deciding to go with OpenDocument is the easy part.

Getting there is something else. A quote from Peter Quinn, the legendary CIO of Massachusetts who threw that great Commonwealth into the breach, says it all:

"Open document formats: I get it! But how do I get there? Discuss."

Sam Hiser has provided us with another fine commentary,“Open Standards Mandatory in Denmark”that's loaded with insight. He discusses the recent announcement by Denmark of their intentions to mandate support for two conflicting, contradictory, and irreconcilable XML file formats; the ISO/OASIS OpenDocument and the Microsoft Ecma Office Open XML proposal. For the life of me i don't see how mandating both ODF and OOXML will be helpful to any government. It's true that both are “XML” by design, but beyond that the promise and expectations of XML are broken by the proprietary application, platform and system specific dependencies that make up MS Ecma OOXML. In fact, the quality of XSL transformation filters between the two XML file formats is so bad it might as well be zero.

Although Microsoft assured Massachusetts that such a transformation would be trivial, nearly two years later they have nothing to show for their braggadocio.

Which means Denmark will be hoist on the petard of having their documents in two file formats that are not interchangeable. And except for the MSOffice ODF Plugins from Sun and the Foundation, application interoperability might as well be zero.

A Moving Target – MSOffice 2007 Plugin Architecture:

It's important to keep in mind that both daVinci and ACME 376, the Foundation's Plugins, were working fine in every beta release of MSOffice 2007, but broke in the final public version. I suspect Sun had the very same problem we did. It seems that Microsoft altered the system default for “zip” file format packages such that whenever a “zip” package like ODf is double clicked, the OOXML conversion engine automatically starts to convert the package to MSOffice 2007 in-memory-binary-representation. Of course, the OOXML conversion engine knows nothing about ODf. So the problem we now have is reverse engineering what looks to be a low level system “default” setting that loads the OOXML conversion engine instead of the daVinci conversion engine. Microsoft has a long history of dirty tricks and an ever moving APi, effectively protecting their monopoly from would be poachers and thieves, er “competitive application and service providers working the Windows – Vista platform”. And here we go again. This first act of aggressively blocking the ODf Plugins should signal governments loud and clear that moving into the Vista Stack of desktop, servers, devices and services is going to be the equivalent of exclusively mandating OOXML. We've known this to be the case in the past, the primary example being the Exchange/SharePoint Hub and developer platform which is optimized for MSOffice 2003 MSXML. So much so that the E/S Hub has to be considered an ODf killer. Yet even those governments exclusively mandating ODf are boldly going forward with E/S Hub purchases, totally unaware of the consequences. As i've said many times, Massachusetts is just one E/S Hub Court Docket System away from revoking their ODf requirement and standardizing on OOXML. (Watch carefully now, the hand is quicker than the eye; ViSTO 2005, which was released with MSOffice 2007, dropped support for MSXML entirely in favor of the MS version of OOXML. (i mention this because there is clear evidence that MOOXML, legacy MOOXML, and now MOOXL Binary InfoSet for Excel all include eXtensions and dependencies that differ from the Ecma 376 version submitted to ISO/IEC). Remember our Peter Quinn quote, "Open document formats: I get it! But how do I get there? Discuss.”? The simple truth is that the ODf Community is not providing the means to get to ODf. There is no bridge from the legacy installations of MSOffice and the billions of binary documents, to ODf ready applications and services.

Get to XML:

What most ODF-OOXML warriors forget is that the real issue for workgroup and workflow oriented consumers is getting their binary documents into XML. Which XML, ODf or OOXML, is a secondary consideration. The one thing everyone understands is that the way to connect important information domains, systems and architectures is through the skillful use of open XML and open Internet technologies. Important information domains and architectures includes:

• Desktop Productivity Environments (the Office Suites)

• Enterprise publication, content management and archive systems

• SOA – Service Oriented Architectures

• SaaS – Software as a Service systems

• The Internet – Web 2.0 and beyond

You can't efficiently exchange information across these domains if it's trapped in application – platform specific and unstructured binary formats. Conversion fidelity requirements will break a binary bound exchange process every time. Even Microsoft realizes the need to move to XML, although they will of course strive to control that XML through continued use of application and platform specific dependencies and, artificially contrived implementation constraints.

XML Migration: The Three Stages

The problem as we saw in Massachusetts, Munich, and Bristol is that there are three stages of XML file format migration. The three stages for migrating to XML file formats are inextricably linked and must be tackled in exactly this order:

1. Conversion Fidelity - the billions of binaries problem
2. Round Trip Fidelity - the MSOffice bound business processes, line of business integrated apps, and assistive technology type add-ons
3. Application Interop - the cross platform, inter application, cross information domain problem

Sadly, the ODf Community is near entirely focused on stage three, leaving the critically important first and second stages up to customers.

Interestingly, OOXML nails all three stages perfectly, with the proviso being that all interop applications be Microsoft application – platform specific. This is what they mean when they say, “interoperability by design”; applications designed to speak fluent OOXML, making exclusive internal use of .NET 3.0 system dependencies.

Note well that since the starting point of XML migration is that of MSOffice bound binaries and business processes, OOXML is today the only XML file format proposal that can perfectly answer all three stages of XML migration requirements. The OpenDocument Foundation does have three products in the works that are designed to also perfectly answer these same XML requirements, but the Foundations approach has met with considerable resistance, argument, doubt and continued outrage from the ODf Community at large. Note also that these three stages are only important to workgroup and workflowsituations – situations where MSOffice bound business processes are critical day to day to day operations. People who are involved in simple document exchange can easily move to ODf through an OpenOffice download and some simple conversion artifact fixes as documents are converted as needed. Workgroups and workflows require a very high level of round trip fidelity. Which is why mixed application environments present workgroups with an near insurmountable problem; even where the mix is limited to that of different MSOffice versions! Of course no one wants to see the life span of MSOffice eXtended indefinitely,which is exactly the conclusion most come too regarding the daVinci Plugin. daVinci makes it too too easy to stay with MSOffice since there is zero disruption to MSOffice bound business processes, line of business integrated apps, and the functioning of assistive technology add-ons. The ODf community doesn't realize three things.

• First, it's an absolute real world requirement that the three stages of XML migration be met perfectly by any file format contestant.

• Second, Microsoft has handed the ODf Community the opportunity of a lifetime regarding the most difficult stage for ODf; stage two, the migration of MSOffice bound business processes.

• Third, consumers migrating to XML have no stomach, tolerance or funds for the highly disruptive and costly rip out and replace” approach most in the ODf community favor.

The Massachusetts RFi:

Massachusetts ITD deserves credit for figuring this out. The three stages are the reason behind their ODf Plugin for MSOffice RFi. Which is about as desperate a cry for help from the ODf Community as i've ever seen. There is no other way to describe the events in Massachusetts except to say that the ODf Community refused to listen, and worse, made no attempt whatsoever to understand the exact nature of problem consumers wanting to migrate to ODf face. Hence the unprecedented RFi (Request for Information). So what's going on here? It's the reality of being caught in the clutches of a monopolist for over ten years.

In 1995 Microsoft won the Office Suite application wars, with the next two years mopping up as they paved the way for the real crusher, MSOffice 97. The office suite wars were marked by the vicious marketing of comparative feature sets and competitive price inducements. The feature wars ended in 1995, were set by 1997, a time period during which a new movement had begun – the use of the ubiquitous MSOffice as a development platform for business processes, line of business integrated applications, and a proliferation of add-ons like those for assistive technologies.

In short, by 1997 it was no longer about “features”. It was all about MSOffice bound business processes. And the binding occurs at two levels; the MSOffice application layer, and the binary file format layer. This creates a binding trifecta of MSOffice, MS Binary Documents, MSOffice bound business processes.

The real battle today is over how to migrate or participate in the MSOffice bound business processes developed since 1995. This is the core of the problem every organization trying to migrate to ODf must confront. Massachusetts, to their everlasting credit, came up with a rather magical solution; the idea of using ODf Plugins for MSOffice to solve the problems of stages one and two. The stage three ODf problem of application interoperability is a work in progress, and i would direct everyone's attention to the ODf 1.2 progress ofthe three OASIS ODf sub committees; Accessibility, Open Formula, and Metadata RDF/XML. A great leap in application interop is brewing there, and expectations for ODf 1.2 run high with good reason.

The Problem With Rip Out and Replace:

Many people mistakenly believe that migrating to ODf is as easy as downloading for free OpenOffice 2.0. Which is true if your collaborative work is that of simple document exchange. For workgroups and workflows currently based on MSOffice bound business processes and binaries though, the disruptive and re engineering cost of moving those processes to ODf ready OpenOffice is impossibly high. While it's true that Novell is working on an automated means of converting bound business process VBA scripts to an OpenOffice footing, it's not there yet. But that's just a fraction of the problem. These business processes include compound documents for templates, forms, formula bound spreadsheets, and reports - often involving systems dependent data bindings. Rather than trying to unwind these MSOffice bound business processes and re write them to somehow become application independent and ODf ready, we believe there is another approach demanding consideration. Instead of migrating to an ODf ready desktop MSOffice alternative, we believe the ODf Community should instead be focused on the Microsoft migration of these same business processes to the OOXML enhanced Exchange/SharePoint/Groove Hub.

The Great Opportunity: The Vista Information Processing Chain

Briefly the greatopportunity is this: Microsoft has been very busy migrating those MSOffice bound business processes to the Exchange/SharePoint Hub, which will soon be joined with Groove Collaboration servers. XML Hubs are interesting in that they are an indispensable core to any SOA effort. The Hubs are a very effective point where access, aggregation and repurposing of XML services and information streams connecting to such things as backend data, transaction, inventory and billing processing or content –archival management systems. As a universal transformation layer able to connect to many disparate and purpose specific backend legacy systems, XML is unparalleled.

But what if these same XML Hubs are integrated as a way station connecting desktops and devices to Internet server systems as well as the legacy backends? And there my friends is the magic of an Exchange/SharePoint/Groove Hub. On the desktop side E/S Hubs integrate directly with the MSOffice productivity environment using OOXML (versions Ecma 376, MOOXML, legacy MOOXML, and MOOXML Binary InfoSet) as the both the XML container and transport. The E/S Hub provides the integration point where eMail, documents, workflows, project participants (people), and information resources (web services, XHRequest streams, media streams, and message streams) can be aggregated, sorted, re purposed, published, managed and scheduled. At the E/S Hub it's easy to bind data from MS SQL Server systems, MS CRM, MS ERP and MS Live to the portable OOXML document transport.

The most important point about XML Hubs is that migrating business processes to them always results in extraordinary productivity gains. Yes, the E/S Hub represents a new lock-in point for MSOffice bound workgroup customers. The immediate productivity gains however will far outweigh the long term cost of having your business lock-in to MS only applications and services for the next fifteen years. So where's the opportunity you ask?

By intercepting this migration at the head point, MSOffice, converting the documents to ODf, and provisioning ODf ready Hubs the ODf Plugins could potentially walk off with the entire monopoly base of over 500 million MSOffice bound desktops. The ODf Plugin route is infinitely cheaper in that there is no need for desktop upgrades. And, the cost of long term MS lock-in is completely negated.

The ODf Plugin alone however isn't enough - which is exactly why the Foundation began development of the lightweight, efficient and highly portable ODf InfoSet Engine and APi. We need ODf ready Hubs to complete the ODf Information Processing Chain alternative. ODf Hubs that can compete with E/S Hub Juggernaut.

C'mon Alfresco, Lotus, and Zimbra - the whole world is waiting.

One more thing about these ODf Hubs and ODf ready applications that span desktop, servers and device platforms. The Internet has ushered in a new world of universal connectivity, exchange and collaborative computing. Some ODf ready applications will be designed to participate in specific information processing chains, so they will be built to act as efficient routers of information.

Other applications will profess to be ODf ready, but continue to act in the legacy traditions of information end points - ignoring the ODf document needs as defined by other information processing chains; dropping objects and data binding mechanisms they don't understand or lack the feature sets needed to make use of. Or, they might add value by asserting application or platform specific dependencies that would otherwise corrupt the documents use in other information processing chains.

Application Interoperability

The problems of application interoperability are far more difficult for ODf than for OOXML because ODf was designed for a very different objective.

OOXML was designed exactly for MSOffice bound binary and business process compatibility, (stage I & II of the XML migration requirement) including application and platform specific dependencies.

ODf was designed to be an application and platform independent universal file format, dependent on open XML and other Internet technologies available to any application or service.

For ODf, proprietary or platform specific dependencies are an interop killer. For OOXML, proprietary and platform specific dependencies are the monopolists life blood of "interoperability by design”.

The world is going to move to XML, and from there to RDF/XML. That's a given. OOXML has the incredible advantage of meeting the three “migration to XML” requirements. But the migration comes at the cost of long term business process lock-in, with interoperability demands limiting applications and services to the Vista Stack of:

MSOffice <> OOXML <> IE <>ViSTO .NET <> E/S/G Hub <> MS Active Directory <> MS SQL Server <> MS CRM <> MS ERP <> MS Live etc. etc. etc.

The core of this chain is the MSOffice <> OOXML <> Exchange/SharePoint Hub

ODf can similarly meet the the three “migration to XML” requirements, and do so just as well as OOXML – in spite of MS claims otherwise. Although they anger and upset the rip out and replace bent ODf community crowd, the ODf plugins for MSOffice from Sun and the Foundation are the only way ODf can crack and perfect these critically important “migration to XML” requirements. Danes please take note!

And of course, there is today no ODf Stack or ODf information Processing Chain comparable to the one Microsoft has unleashed. Some great stuff cooking with Lotus Notes, but they are obviously lacking a strategy for stage I & II of the “migration to XML” requirements. Alfresco is hard charging, but they have that same problem. Zimbra and Google Hubs are still mired in the application as end points approach where they see conversion of documents as a one way process where continuing and persistent loss of fidelity and the dropping of feature related objects or bindings is the expected and unavoidable collateral damage cost, so get over it.

Meanwhile, the ISO/IEC consideration of Ecma 376 is a diversion from where the real action is; the optimized for MSOffice OOXML Exchange/SharePoint juggernaut.

It's high noon for XML. Do you know where your ODf information processing chain is?

~ge~

Notes: The OpenDocument Foundation is working on three products that we believe to be essential to the development of an ODf Information Processing Chain that can connect desktops, servers, devices and Internet systems transitioning information based on the highly interactive and collaborative portable XML document/data model/streaming media model.

These products are still in development. What follows are the design goals and objectives:

• daVinci ODf 1.2 Plugin for MSOffice

• InfoSet Engine & APi

• Interop Wizard for OpenOffice

• ACME 376 XML-RTF Plugin for MSOffice

The daVinci ODf 1.2 Plugin:

To the best of our knowledge, daVinci continues to be the only ODf plugin for MSOffice designed to work “internally”. What happens is that daVinci triggers the internal conversion process Microsoft uses to do MSOffice conversions.

DaVinci triggers the internal process, intercepting an undocumented internal super structure we call “MS Universal RTF”, for lack of a better name. This undocumented super structure accounts for the extraordinary near perfect conversion fidelity achieved by daVinci and demonstrated by the ACME 376 plugin.

We do not believe similar conversion fidelity results can be achieved by either the MCN XSL Translator plugin approach, or the traditional “external” binary file reverse engineering approach used by OpenOffice and the Sun ODf Plugin. Which is perhaps why Microsoft themselves use this same internal process to convert legacy MS binary documents to MS OOXML.

The daVinci conversion process follows this “internal” sequence (and it'sreverse):

imbr <> MS Universal RTF <> daVinci InfoSet <> ODf 1.2

imbr :: Microsoft in-memory-binary-representation which becomes a MS Binary (dump) on save (the reverse on load).

MS Universal RTF: This is a very special structure that MSOffice uses to do all conversions from imbr to MSXML, OOXML, and RTF (the RTF the rest of the world uses).

DaVinci InfoSet: A super InfoSet structure daVinci imposes on the MS Universal RTF structure.

ODf 1.2 :: daVinci maps from InfoSet to ODf 1.0 (first daVinci version) and ODf 1.2 to produce the ODf file. The mapping mechanism could be redirected to Chinese UOF or even a subset of OOXML with some work.

The ODf 1.2 metadata RDF/XML model provides daVinci with the descriptive flexibility needed to maintain a high level of interoperability. The first version of daVinci provided to Massachusetts in June of 2006 targeted ODf 1.0 and high interoperability with OpenOffice. This came at the expense of perfect conversion fidelity and much needed round trip fidelity with the legacy of billions of MS binary document. In essence, the 85% conversion fidelity one expects from OpenOffice was exactly the same conversion fidelity delivered by daVinci ODf 1.0.

ODf 1.2 enables daVinci to hit near perfect conversion fidelity without compromising on the high level of application interoperability users expect.

The ACME 376 plugin is available for public download and testing - (the ODf 1.2 metadata RDF/XML proposal will not be completed until April of 2007). Although ACME 376 perfects a XML encoding of RTF, it demonstrates the extraordinary near perfect conversion fidelity provided by the daVinci engine.

The daVinci Plugin design objectives includes these features:

• daVinci plugin for MSWord, Excel and PowerPoint :: MSOffice versions 1998, 2000, XP Office, XP 2003 and 2007 (note- unexpected problems with final release of MSOffice 2007)

• Accessibility Interface installed with daVinci (tagging graphic & media objects in MSOffice with ODf 1.1 accessibility eXtensions)

• PDF/ODF w/Digital Signatures (XML-XForms sig model) :: a combination PDF/ODF file that can be read in Acrobat, but edited in any ODf ready application given proper digital rights :: excellent for data binding, transport and extraction

• XForms Interface for MSOffice installed with daVinci

• Enhanced document library storage search, re use, and re purposing through the advanced ODf 1.2 metadata RDF/XML model

• Enhanced cross platform/cross information domain application

interoperability through the ODf 1.2 metadata RDF/XML model

InfoSet Engine & APi:

Based on our knowledge and experience of working with the MSOffice internal conversion process, we began development on a ODf InfoSet Engine with developers APi. The objective is to provide developers, IDE's, Server-Device-Desktop Application providers and cross platform run time engines with a complete ODf conversion and layout engine that is light weight, portable, and easy to embed. Work on the “layout” engine portion has not yet begun.

The Interop Wizard for OpenOffice:

This is a plugin for OpenOffice that enables workgroup user to set either the individual document or default document settings to provide perfect interop with MSOffice desktops.

The issue here is that mixed environment workgroups (ODf Plugin MSOffice & OpenOffice desktops) will face a near insurmountable problem of round trip fidelity corruption. This has nothing to do with file formats or conversion fidelity processing, and everything to do with an issue known as layout engine impedance mismatch.

OpenOffice has a very sophisticated and complex layout engine that is optimized for the advanced presentation and use of “styles”. MSOffice on the other hand has a comparatively simple layout engine design and does not attempt to do common presentation tasks such as “table within a table”.

The layout engine mismatch limits and defines certain feature sets that make it structurally impossible to map documents between the two application suites. The “structural” problems are well known, and fall into these five categories: lists, fields, sections, page breaks, and tables.

The Interop Wizard shuts off the advanced feature sets of OpenOffice, limiting functionality to mimic near exactly the structural features of MSOffice. When the wizard is on, the advanced features are not available unless the end user overrides the warning.

God Save the Queen!

The British Standards Institute, which represents the UK with the International Standards Organization, has issued a " contradiction" to Microsoft's specification.

And it is just one of many national bodies that had until today to contradict the application, which was being fast-tracked following its endorsement by the European Computer Manufacturers' Association (ECMA). Reports are that Malaysia will follow suit with their own encyclopedia of contradictions.

"Microsoft standards bid faces failure"

"Microsoft's bid to get its Open XML formats recognised as an international standard faces a delay for at least three months and could fail altogether, it emerged today."

Yankees in the Court of King Arthur, with a Microsoft Agenda

ANSI/INCiTS has completed their review of Ecma 376, and is ready to cast their ISO/IEC Contradiction Review Phase Fast Track Ballot in favor of Ecma 376 being rammed through ISO. As Sam Hiser points out in his PlexNex blog, not only are the findings of contradictions, inconsistencies, and proprietary dependencies pouring into the public view, there's not much an American can do about it. ANSI/INCiTS has determined that no contradictions exist."

Hi Sam,

As a fellow American, prepare yourself for humiliation and shame. ANSI/INCiTS has decided that they will not object to MS Ecma 376 on the ISO/IEC Fast Track Ballot. This in spite of the massive compilation of contradictions and inconsistencies compiled by GrokDoc, whose members raced through the weekend just to have the document ready for the ANSI/INCiTS meetings.

Rather than confront the clear evidence of contradictions and inconsistencies, the brave hearts at ANSI/INCiTS choose to narrow the definition of what a contradiction is. And narrow it they did.

They decided that one standard contradicts another standard only if the proposed standard causes the existing standard not to work.

Sounds good doesn't it? Au contraire mon ami. This is the kind of self serving maneuvering only a bureaucrat could love.

Using the analogy of the Chinese WAPI WiFi networking standard that was defeated last year because the protocol caused radio interference with existing 801.11 networks, our standards champions came up with their mechanical interference measure of “contradiction”.

Because both files can physically exist on the same disk without interfering with each other, our champions determined that OOXML did not contradict ODF.

Maybe they thought this would go unnoticed, but as one disheartened friend of open standards pointed out, “this argument can be used for every XML format, every programming language, every operating system, in fact every software standard, since software is ultimately data, and data can be segregated on disks. So they essentially chose a definition so narrow that it nullified the concept of "contradiction" for most of what JTC1 has authority over”.

So narrow a definition of “contradiction” that you can drive a fleet of monopolist trucks through the hole they've carved out of the ISO/IEC standards process.

So here we are. The champions appointed by our National Institute of Standards and Technology to represent our interest in the International Standards process have carved out a dangerous and possibly enduring loophole in the ISO/IEC fast track process. A loophole designed to serve the interest of a single proprietary monopolist seeking to control mankind's digital future.

Sam, where do we hide?

To our friends abroad fighting desperately to preserve the integrity of international open standards, with our digital freedom and the future of open Internet on the line, we ask that they dig in their heels and fight this to the end. Those Yankees you see striding into the Court of King Arthur are not from Connecticut, and the Camelot of ISO had better beware. Stalking standards to the tune of that ugliest of all Americans, the corporation from Redmond, ANSI/INCiTS has sold our souls that the world might be fodder for Vista.

Dig in your heels friends,

~ge~

Brace Yourselves! ACME 376 XML Is Here!

The ACME 376 Proposal for International Standardization
The following letter is to Rob Weir, who somehow managed to get his own personal XML language approved as a standard. We too would like our personal XML language and file format approved. But we don't have Rob's connections. We also don't pack the massive checkbook he carries! Still, our XML is just as worthy of international consideration. We also are providing the standards bodies members with an easy to download ACME 376 Compatibility Kit for versions of MSOffice 97-2007 so that they can test and use our personal XML language. We need help Rob!



Dear Rob,
We were very inspired by your "ECMA Weirish" effort. We too have ideas, thoughts and business needs so unique that we need our own XML language. So we invented ACME 376. We even have a ACME 376 Plugin for MSOffice so that coyote's everywhere can immediately improve their road runner catching productivity using our own XML language, "ACME 376". Now coyote's, plumbers and amateur bomb-makers everywhere can finally join the XML revolution, converting their billions of legacy binary documents to perfect, 100% high-fidelity ACME 376!

It's a great day for coyote's the world over. Woe to the hapless roadrunners who dissed us in the past.

So Rob, how did you get ECMA to rubber stamp "ECMA Weirish"? We need the same no-questions-asked rubber stamping for ACME 376. And from there we need to push it through ISO/IEC, but in a way so that the road runners of the world are caught unawares of our nefarious plot for their destruction.

Could you please review the following proposal and advise,

~ge~

Wanted: A Standards Body Willing To Rubber Stamp
Inspired by Rob Weir’s blog who intended to standardize “Weirish” as an international standard, we thought that if Rob has his own standard then we need one too.

So, we invented the “ACME 376” file format. It meets all the requirements of the ECMA TC 45, namely:

• * it is XML! Actually “.acme” files are ZIP files which contain pure XML.
• * it is compatible with Microsoft Office (2000, XP and 2003).
• * it can convert those billions of binary documents with perfect fidelity
• * it is great!

Make your Microsoft Word desktop productivity environment “ACME 376” ready today. You can download the installer here. This is not a joke! If you have problems, leave a comment here.

Standards bodies of the world! ACME 376 needs your consideration and approval! We need your vote for ISO/IEC approval now. We made sure not to use any other existing standard so a review for contradiction is not necessary.

Make ACME 376 an ISO standard today!

Download the installer for ACME 376 Compatibility Kit for Microsoft Office.

Running On OpenDocument Inside of Microsoft Office

Perfect Conversion Fidelity & The daVinci ODF Plugin for Microsoft Office

• The ODF Plugins for Microsoft Office Appear
• Three Conversion Approaches to Consider
• The daVinci Plugin "Internal" Conversion Process
• Blanketed with Unspecified Binary Objects - The dark spots
• The Skinny on daVinci inside
• Conversion Fidelity & Interoperability
• The flexibility of ODF 1.0 - The Interop of ODF 1.2

By now it's clear that Microsoft's Ecma-approved Open Office XML file format specification is filled with contradictions to existing ISO/IEC standards products. Beyond the traditions of international standards consideration, there is a second perhaps even more important concern: How do we reasonably migrate from a world where Microsoft Office bound business processes drive critically important economic, governmental and organizational concerns? How do we migrate these processes to OpenDocument ("ODF"), which is an international standard designed for interoperability? And is there any possibility of converting with acceptable fidelity the billions of binary documents trapped in Microsoft's proprietary file formats? The world wants to move to ODF XML. But the question is, "Can This Be Done?". And further, "Can this be done without costly disruption to our day to day business processes?" Microsoft has long claimed that only their proprietary Office Open XML could convert those billions of binaries to XML without loss of fidelity (data loss, or "lossiness"). The claimed that ODF was inadequate and unable to handle the rich feature set of Microsoft Office. This is a strange claim in that the "X" in XML stands for eXtensible. Since XML formats are eXtensible, of course ODF can handle anything Microsoft Office or those billions of binary documents have to throw at it. The real truth about this issue is that no one has ever been able to crack the secret code of those Microsoft binaries that hold so much of the world's documents. And Microsoft is not about to disclose the specifications for those formats. Years of reverse engineering efforts by non-Microsoft develoeprs have brought us within range, but the binaries are an ever-moving target for interoperability. With each new version of Microsoft Office applications, the binary formats change arbitrarily. For example, there is no single DOC format; there are a host of DOC formats that all use the same file extension, DOC. Do not kid yourself. Microsoft is not about to be the one that converts all those binary documents to the open standard ODF. Instead, they set out to convert those billions of binaries to their own proprietary XML. So, the next question is of course, "Is there any possibility of converting those binaries to Microsoft's EOOXML and from there a transformation to ODF?" After all, easy and perfect transformation is the promise of XML. Microsoft's Steve Ballmer answers that question for us when he claims that conversions between EOOXML and ODF can be done, but that Microsoft's plugin will never provide full fidelity conversions between EOOXML and ODF; in other words, only a core set of features will be converted and the conversions of documents implementing other features will be lossy, resulting in data loss in largely unpredictable situations, depending on the differences in particular documents. So Microsoft's solution will not allow the automated conversion of those billions of binary documents to ODF. Not without data loss. We'll have transformation processes between EOOXML and ODF, it just won't be worth doing unless you are willing to manually compare documents when rendered both on Microsoft's applications and on an application that fully supports ODF, just to ensure that no crucial data was lost in the conversion. Essentially Microsoft is claiming that only they can convert the billions of binary documents to XML with the fidelity needed, perfect fidelity (no lossiness), and only to Microsoft's flavor of XML, EOOXML. And Microsoft wants the ISO to award it a monopoly in converting its legacy formats to XML by making its own personal XML its own personal international standard. Microsoft is also funding an open source project to perfect conversions (more properly, "transformations" in XML lingo) between EOOXML and ODF. Maybe just to prove they are good guys who can be trusted with our information, and that they are not out to "replace" ODF, but rather to perfect the conversion of those legacy billions of binaries to XML. One objective of the Microsoft-Novell Translator Project is to provide an easy to install EOOXML <> ODF plugin for both Microsoft Office and OpenOffice. Betas for both of these plugins are expected to be released this month. Because this work is somewhat in the open, we are well aware of the intransigence of continuing conversion fidelity problems in both versions. (The Novell work on the OpenOffice.org Writer Translator plugin is complete, but not open to the public, or contributed back to OpenOffice.org -- yet). Note that the Translator Project is based on a XSL Transformation process. So it's expected to be both application and platform independent. The only question is, "Can they achieve the quality of fidelity needed to be of any use?" Steve Ballmer says no. And he's funding the project. There's also the consideration that a scorpion can't help but continue to act like a scorpion. This famous quote from Bill Gates continues to haunt the technology industry to this day. He might as well been referencing the Microsoft binary file formats:

"I doubt they [Digital Research] will be able to clone Windows. It is very difficult to do technically, we have made it a moving target and we have some visual copyright and patent protection. I believe people underestimate the impact DR-DOS has had on us in terms of pricing." (May 18, 1989 - Bill Gates)

The ODF Plugins Appear::

When Massachusetts announced their Request for Information concerning the possibility of an ODF Plugin for Microsoft Office, there were many responses. Each with a different approach to the problem. Microsoft responded to the RFi with the promise of their XSLT based Translator Plugin. Sun provided two different plugin designs; the first based on a OpenOffice Server side conversion, the second based on a C# routine connecting Microsoft Office functions to a locally installed OpenOffice conversion. In both cases it was the OpenOffice conversion engine we know and love that was doing the work of MS binary formats to ODF and back. Importantly, neither the Microsoft nor the Sun plugins allow ODF to be set as the default file save format in Microsoft Office, what is known as "native support" for a file format, a situation sure to produce lots of accidental non-ODF files. Imagine what it would be like to have to train yourself never to hit the "Save file" option or keystroke shortcut in Microsoft Office. Instead, you must open a special menu option to save a file as ODF. Do you think you just might accidentally use the normal file save commands every now and then? An unpredictable mixture of file formats in a network can have unpredictable consequences, particularly when automated processes are involved. A third ODF Plugin for Microsoft Office was proposed and submitted by the OpenDocument Foundation. This plugin conversion process was based on internally triggering a Microsoft Office native conversion process; one the Foundation believes is the same or similar to that which is used when the EOOXML Compatibility Pack is installed, and the Microsoft Office in-memory-binary representation of the document ("IMBR") is converted to EOOXML. Unlike the other plugins, the Foundation plugin adds full native support for the ODF file formats to Microsoft Office. (The present version adds that support to Microsoft Word; later versions will add it to Excel and Powerpoint.) You can use the normal file save dialogs and commands. Moreover, you will not need to study/rewrite all of your existing scripts to ensure that files are saved to the right format. Just set the default file save format to ODF. There were other submissions, but these are three we know the most about. They each represent a different approach to the problem of converting those billions of binary documents to XML. They each offer a different quality of conversion fidelity. Only one allows ODF to be set as the default file save format in Microsoft Office. The real question is whether or not any of the three provide file conversion fidelity acceptable enough so that there would be little or no disruption to existing Microsoft Office bound business processes, line of business dependencies and the functioning of assistive technology add-ons. In short, without near-perfect conversion fidelity, there is no measure of "interoperability" worth talking about. That is the reality of Microsoft's EOOXML blackmail attempt. For years they have withheld from competitors and customers the secret binary file format details needed for perfect conversions, reserving that advantage for themselves. Only Microsoft holds the key to unlock your information from Microsoft applications it remains bound to. So because they alone hold the key to so much of our binary-bound information, they insist that the world must adopt their proprietary, self serving, application and platform bound, monopoly leveraging XML as an international standard. We are being blackmailed by the problems of converting those legacy billions of binary documents to XML. I have a counter offer that ISO/IEC might consider; Give us the keys to those legacy binaries and the documentation for the new MSXML InfoSet binaries that first appeared in Microsoft Office EXcel 2007, and we'll give you international standardization for EOOXML. A fair trade i think, because it will break the monopolist's grip, level the competitive playing field, and restore competition wherever desktop, server and device systems need to interconnect and exchange information.

Three Conversion Approaches to Consider::

What we have then are three conversion methods, all enabled through an easy to install plugin model, and each with a different level of conversion fidelity:

• The MS Translator XSLT method (EOOXML <> ODF) :: note that this initially was an application and platform-independent approach. In the strict sense, this is a "transformation", not a "conversion"

• The Sun external OpenOffice.org conversion engine method (MS Binary Files <> ODF) :: note that the OOo conversion engine is based on years of reverse engineering needed to understand the secret structure of those mysterious and enigmatic billions of binary documents. A secret only Microsoft can unlock with perfect fidelity.

• The Foundation's daVinci "internal" conversion process (MS in-memory-binary representation <> ODF) :: note that this process harnesses the internal conversion methods of Microsoft Office applications in much the same way as the EOOXML Compatibility Pack.

A brief description of the daVinci internal process:

So how does daVinci do this magic of triggering an internal process and letting the native resources of Microsoft Office perfect the conversion? Well, the first problem was getting inside Microsoft Office applications and working "natively". The know how for this was provided by Microsoft themselves when they mounted their first MSXML add-on to Microsoft Office 2003. And what a great job they did! Once inside Microsoft Office and working natively, the entire view of how to best convert MS binaries to ODF changes radically. Rather than trying to crack the intransigent and enigmatic binaries externally, on the inside you simply let the Microsoft Office applications do it for you. We don't know for sure, but there is every indication that daVinci works very similar to how the EOOXML Compatibility Pack works. There is no doubt that without the public information Microsoft has provided concerning the early versions of MSXML, we would not have had the series of breakthrough discoveries that make daVinci possible. So the key to daVinci is in letting Microsoft Office apps handle the billions of binary documents, especially their conversion to IMBR (the Microsoft Office apps in-memory-binary representation). Internally, when a conversion process of any sort is triggered, Microsoft Office apps follow pretty much the same routine. There is a point where this internal conversion process can be intercepted and routed (mapped) to a different, non Microsoft, file format structure. Imagine if an internal conversion process from IMBR <> EOOXML is triggered, and you intercept the process the moment before mapping to EOOXML begins. And you reroute by mapping to ODF. That's daVinci. DaVinci triggers, intercepts, and maps to ODF. It could just as easily be configured to map to Chinese UOF. Or to EOOXML (well, forget the easy part regarding EOOXML; the haphazard and sprawling structure of EOOXML makes this mapping difficult - but as the MS Compatibility Pack proves, it is possible :). daVinci could even be configured to map to Romanian XML or Oracle XML. The conversion quality of the daVinci process really depends on the flexibility of the XML schemas it is mapping to. Let me say that again, "The conversion quality of the daVinci process really depends on the flexibility of the XML schemas it is mapping to". Since EOOXML was made expressly and specifically for mapping Microsoft Office IMBR to, you better get perfect fidelity. What about ODF? Yes, you can get the same perfect fidelity. The flexibility is there, and has been there since the February 2003 addition of the <foreign element> tags, section 1.5 of the ODF v1.0 standard (casually referred to as the <microsoft tags> because of what they can do). So yes, if you can break the secret of the proprietary IMBR, understand their hidden structure and function, you absolutely can get perfect-fidelity conversions to ODF and EOOXML. This is an incredible achievement for the OASIS ODF Technical Committee ("TC"). ODF was designed to be a universal file format, totally application and platform-independent, and it has the built in flexibility to easily handle anything the enigmatic billions of binaries might throw at. Tapping into the Microsoft Office IMBR just makes it easier for daVinci to see what's actually happening inside those unspecified binary blocks that blanket the billions of binary documents were trying to convert to XML. As Rob Weir has remarked: "If Microsoft supported ODF 1.0 in Office today, using the foreign attribute support already specified in ODF 1.0, they could achieve backwards compatibility with their legacy documents. There is nothing that prevents them from adding a "DoItLikeWord95" attribute to an ODF document."

Blanketed with Unspecified Binary Objects - The dark spots ::

The real problems of converting those billions of binary documents or working as a near native file format within Microsoft Office has nothing to do with either EOOXML or ODF. And everything to do with the secret, enigmatic binary file formats. Microsoft is busily spinning the world to convince us otherwise, but it only takes one demonstration of daVinci to set things straight. We shouldn't give in to blackmail. Especially blackmail designed to leverage the Microsoft desktop monopoly deep into our future of converged and highly interoperable multi platform systems. Control of the file formats, and keeping them bound to proprietary applications and platforms, is control of everyone's information and information processes. So Microsoft Office will do a great conversion of those billions of binaries to IMBR for us. And when triggered, IMBR will set things up for daVinci to intercept an internal conversion structure and map to ODF. Because there are billions of binary documents out there, with years of file level application feature tweaking and enhancements by independent LOB - business process developers and assistive technology add-ons to deal with, there's no telling what kind of unspecified binary objects daVinci will encounter and have to map to ODF. daVinci needs the mapping flexibility in the XML target structure to place these unspecfied anomalies otherwise called "dark objects". The thing is that these binary object anomalies are unspecified on both ends of the conversion equation. They are unspecified with regard to the historical annals of reverse engineering, which itself is based on the cryptic, enigmatic, and often misleading documentation Microsoft has provided for RTF and the MS binaries. And, they are unspecified by the XML structures at the other end of the conversion equation. Like ODF.

The Skinny on daVinci inside::

When a user loads a binary document (or creates a binary document in Microsoft Office applications), the apps themselves convert the binary documents to IMBR (the Microsoft Office apps in-memory-binary representation). The user works the document in IMBR mode. This means all application features, business process adaptations, assistive technology add-ons, whatever, are available and cooking without disruption or change. When an internal MS conversion process of IMBR is triggered, daVinci intercepts the results, and maps to ODF. The ODF version is saved to file. An internal conversion process is triggered whenever functions like save, save as, open, or open most recent is called for.

Conversion Fidelity & Interoperability ::

We fully believe that ODF version 1.0 provides daVinci with the flexibility we need to hit the same quality of fidelity of conversion of those billions of binary documents that EOOXML promises. Which is to say that ODF 1.0 has long offered Microsoft the same opportunity to convert everything to ODF and back. There is no technical reason for Microsoft not to have implemented ODF. And there is no technical reason for them to now ask that ISO/IEC consider a second universal file format specification as an International standard. But what's beyond the issue of conversion fidelity? Inter application and cross platform interoperability; the ability to transport and exchange documents across many different kinds of information domains without loss of fidelity or structural compromise. Interop is a tall order. Especially after years of living with application bound file formats that only the application and platform vendors can transport and exchange effectively. In their EOOXML pitch, Microsoft promises something called, "Interoperability by design". Translated this means that all Microsoft applications will be designed to work perfectly with EOOXML. Most likely we will also see MS applications able to handle the binary extensions of EOOXML that showed up Excel 2007. This includes desktop, server and device systems written to .NET 3.0 and the Vista platform. To make certain this happens, Microsoft has provided us with a new version VSTO 2005 where they drop support for MSXML and introduce support for EOOXML. They make it easy. The thing is that if you're a non Microsoft application, most likely you won't be able to fully implement EOOXML. Definitely you won't have access to the binary InfoSet extensions of EOOXML. Least ways not without a price, and never if you're a competitor like Oracle, IBM or Sun. ODF Interoperability is open and freely available to anyone wanting to implement ODF. Participation in the OASIS ODF TC specification process is open and affordable. There are no application or platform specific dependencies, or licensing restrictions, or patent encumbrances - legal risks holding anyone back. Universal file format interoperability is a given with ODF. Application interoperability is another matter. Especially existing applications that might have layout engines developed long before ODF became available. With new ODF applications this won't be a problem since they can develop directly to the specification. This is one of the reasons so much work is going into ODF 1.2, to accommodate the differences of traditional layout engines as they implement ODF. We don't have the power or authority of a Microsoft to rewrite every application to work perfectly with ODF. Nor do we have a similar command of the marketplace to force a user base of over 550 million desktops to upgrade to a Vista platform of Microsoft Office 2007 - VSTO -IE 7.0 -Exchange/SharePoint/Groove - MS SQL Server, MS Active Directory Server, etc. So instead, we have ODF 1.2 waiting in the wings. That's where our solution to universal application interop lies.

About the flexibility of ODF 1.0 - The Interop of ODF 1.2 ::

It is true that ODF has had, since February of 2003, an extremely flexible set of tags were added to the specification. They are called the <foreign element> and <alien attribute> tags, and were designed exactly to handle the billions of unspecified and conversion defiant binary object anomalies known to comprise years of Microsoft proprietary binary file format use. When you're mapping from an IMBR conversion process to ODF, you have to have something to put the unknowns in. You have to map to an existing tag. Since the very nature of these dark objects is that they are "unspecified" and previously unknown, they are also outside the range of ODF. Using these tags, daVinci can get perfect fidelity between the billions of binary documents and ODF. And get it right every time, with every Microsoft Office version from 1997 to 2007. But this "perfect fidelity" comes at a high cost of interoperability with other ODF ready applications. Simply put, the other ODF applications have no idea what to do with the daVinci <foreign elements>! An Microsoft Office with daVinci knows. But for all other ODF ready applications these dark objects are still a mystery. In many ways the <foreign element> tags are the equivalent of the what EOOXML does with the same volumes of unspecified legacy tags. They're there, but no one excepting Microsoft Office Compatibility Pack enabled installations know what to do with them. The same is true with daVinci ODF. Only daVinci knows what to do with these dark objects. In other words, from day one ODF has had the exact same means wrapping in proper XML an unspecified binary object or processing instruction as that which EOOXML is now parading about as something absolutely necessary (and unique to EOOXML) for converting those billions of binary documents to XML. With ODF 1.2, daVinci gains the flexibility to map whatever dark objects found in ways that will dramatically improve interop with other ODF 1.2 ready applications. (see the <interop eXtensions> proposal submitted by Florian Reuter to the ODF Metadata SC). Using the new metadata model, daVinci can then proceed to fully describe everything known and intuited about the dark object. Keep in mind that daVinci has an inside view. What daVinci sees is the IMBR context and conversion structure that is missing from a binary file format as well as EOOXML's cryptic tags. This descriptive model will provide every other ODF 1.2 ready application a much better chance to handle and render the dark objects. With ODF 1.0, we were limited in how we describe for interop purposes these unspecified creatures. With ODF 1.2, daVinci can field these objects on the fly, and give other ODF 1.2 ready applications a fighting chance to properly render them. Sticking a binary object into an an XML wrapper is just kicking the can forward. It's passing the problem onto someone else. Yes, it solves the momentary problem of an XML file format plugin running inside Microsoft Office (EOOXML and ODF). No problemo for those users. But it punts the problem of roundtrip interoperability with other ODF ready applications. They are left hanging. With the generic <interop eXtension> approach, and the metadata descriptive model, my guess is that ODF 1.2 ready applications will handle upwards of 98% of these problems instead of having to ignore the entire binary block. Over time of course, we will come to understand, specify properly, and map directly these binary objects. Years of reverse engineering has brought us to upwards of 85% conversion fidelity. Now we need to nail that remaining and highly elusive 15%. Uncompromising demands from Massachusetts and the EU have forced Microsoft to come out in the open with their proprietary XML. They are fighting tooth and nail to keep their application bound binary secrets secret. And with good reason. If we crack that last 15%, and do it in a way that provides users with a totally non disruptive migration to ODF path, the monopoly will have been cracked open. Sometimes i wonder if the ISO/IEC JTC-1 members realize that they have it in their power to do what no government has thus far been able to do - stop the Microsoft monopoly from illegally leveraging their control into other markets, and restore open competition to technology marketplaces. The daVinci ODF 1.2 and ODF 1.0 plugin demonstrations will be made available to ISO/IEC members as positive and irrefutable proof that "it can be done". Hopefully we can get a video demonstration to walk them through daVinci so they can see for themselves. Hope this helps, ~ge~

Notes:

Microsoft joined the original OASIS Open Office XML effort in November 2002 (now OpenDocument or "ODF"). But they refused to participate or comment, instead quietly observing the work of the ODF Technical Committee for the next four years. Meanwhile, they began work on a proprietary XML file format designed specifically and solely to meet the "XML" needs of their Microsoft Office applications and emerging Vista platform of desktop, server and device systems. In 2004, Microsoft presented their proprietary XML effort to the European Union in response to a famous study known as the "Valoris Report". The report recommended that EU governments and organizations mandate information technology purchase requirements based on a far reaching but uncompromising infrastructure of SOA, Open Standards, and Open XML mandates. Valoris also recommended the development of a universal XML file format that was application and platform independent, able to service the portable document needs of an SOA infrastructure stretching over desktops, servers, devices and across the Internet. The universal file they envisioned was tagged "OpenDocument". When Massachusetts followed the EU with a clear mandate for Open Standards and open XML file formats that were recognized by Open Standards bodies, Microsoft formed the MS Ecma 45 workgroup with the objective of developing an open standard XML file format perfectly compatible with the existing Microsoft Office XML file formats. Meaning the end result was in the hands of the Ecma 45 workgroup before they even began work. All that needed to be done was some massive documentation of what in essence is a Microsoft Office binary dump into XML. Or, if you prefer, an XML encoding of Microsoft Office proprietary binary file formats.

OpenDocument as the perfect Microsoft Office file format

How to:

Add native file support for OpenDocument to Microsoft Office

What is the da Vinci plugin for Microsoft Word? Where did it come from? How does it work? And can it really convert files generated by Microsoft Word versions 97-2007 into fully compliant, OpenDocument format-ready applications? Can it do so so without disruption to bound business processes, dependent line of business applications, and assistive technology add-ons?

Let's begin from a simple truth, discussed in more detail below. The OpenDocument format ("ODF") is able to handle anything Microsoft Office can throw at it, and handle it at least as well as Microsoft's new EOOXML file formats. That includes the bound business processes, assistive technology add-ons, line of business dependencies and advanced feature sets unique to various versions of Microsoft Office. The converse however can not be said of Ecma Office Open XML -- the new Microsoft EOOXML file formats -- as evidenced by Microsoft's own ODF Translator Plug-in Project difficulties.

In April of 2006, the Commonwealth of Massachusetts Information Technology Division broke ranks with IT tradition and put out an open Request for Information concerning the feasibility of an ODF Plugin for Microsoft Office. It is not that Massachusetts ITD had rejected the many current software offerings of ODF vendors and open source community efforts. But they lacked a critical tool to perfect a migration to ODF -- from years of Microsoft Office business processes and binary document dependencies to ODF-ready solutions -- without impossibly costly disruptions to their day-to-day business activities.

Massachusetts ITD correctly reasoned that ODF plugins for the Microsoft Office major apps would solve this problem, enabling a mixed environment of interoperable desktop solutions, giving their business processes much needed time to be migrated for the right reasons at a manageable pace. The right reasons are productivity gains and efficiency enhancements, as opposed to merely reactive efforts to squelch the costly insanity of file format madness. The manageable pace is variable, depending on human and other resource availability, rather than on a disruptive rip-and-replace of most software across entire networks to achieve interoperability.

Imagine what your own office would be like if you came to work one day and discovered that all of the software staff normally used had been replaced on every machine by different software. Could your office afford to shut down long enough for everyone to learn all of the new software and procedures? Is that kind of trauma something you would like to go through? People need to be able to exchange documents. That means software interoperability is key, and it simply is not workable for people to exchange files in formats that are incompatible with software used by the recipients.

Now imagine how much worse the problem would be if your office is dependent on fully-automated business processes where processing is entirely dependent and bound to specific applications, and those applications spanning the process pipeline have a secret language only they can speak. Additional workstations must have the right software if they are to participate in the process. Business process change becomes an issue for the application vendors since they alone can tweak the language specific applications. That's expensive. So much so that many a business lets much-needed process changes lapse until they simply are no longer competitive or profitable, and the cost of rewriting the applications to re engineer the business process must be paid; at any cost. Still, a driving and determining factor is that any new adjustments must be compatible with the information that drove prior iterations of that business process. Anything less than full fidelity in converting legacy documents to new formats greatly limits the changes and productivity enhancements that can be gained as well as ensuring that your applications and service continue to come from the same vendors -- for years and years to come. -- No matter what the cost.

Full fidelity is a term used by the file conversion congnoscenti. It refers to the quality of conversions from one file format to another. Full fidelity means the conversion from one format to another cannot result in data loss. If data loss is possible, for example depending on what software features are invoked in a given document, then we refer to the conversion as lossy.

For example, there is the always problematically lossy and always one way-only conversion of a Microsoft Word 97 binary file format (.doc) to a Microsoft Word 2003 binary file format (.doc). There is also the common file format conversion of a Microsoft Word binary .doc to Rich Text Format ("RTF") and back. For our purposes, we are concerned with the fidelity of file conversions between OpenDocument and binary files bound to Microsoft Word versions from Word 97 to Word 2007. Microsoft is able to perfect a full fidelity conversion of those same binary documents to and from EOOXML, using the Microsoft XML compatibility pack plugin. We argue that the same full fidelity of conversion is possible between Microsoft Word and OpenDocument, using the daVinci plugin for Microsoft Word. In fact, the process of how this is done is very similar to that used by the MS XML compatibility pack.

Automated business processes are key to increased productivity and profit, but they are still relatively few and far between, a situation not expected to last long. Automated business process systems are regarded as one of the next Big Things in the software industry. Legacy business process systems often require much end user interaction such as manual management of information, verification of data, tracking of data as it transits the process, etc. Many are dependent on particular legacy software applications' proprietary binary file formats.

Basically, an automated system provides exacting interfaces so that the system handles the data, workflow and intelligent routing of documents, and users concentrate on the decision points and architecture of the process itself. The Workflow Management Coalition defines workflow, in general, as "the automation of a business process, in whole or parts, where documents, information or tasks are passed from one participant to another to be processed, according to a set of procedural rules," where participant can mean either a human being or another automated process.

To get from the partial automation of legacy systems to the full automation of future systems, information transfer must be accelerated systematically and human participants must be provided with interfaces that enable rapid reorganizing of people, resources, documents, eMail, messaging, wiki collaborative documents, direct access to data bindings within those documents, and the ability to push change into advanced workflows managed by "the system". There is simply no practical way to do this without having a universal file format based on the XML portable document with system-bound data model. The universal XML file formats exist.

The systems-level XML hubs are coming on strong as evidenced by the Exchange/SharePoint/Groove Hub, Workplace/Lotus Notes/Domino, Scalix, and Zimbra. This is where data and various information streams merge with the portable document model, with information streams collapsing into a single easy to schedule and manage aggregate; the merge occurs in the context of end user project and workflow management interfaces.

Full fidelity conversions of the information that drives our current business processes to universal portable XML document models is the only way we can make that productivity leap to fully automated business processes. Every time there is a loss of fidelity in data conversion we basically lose a node in the process. We have to go manual and fix a problem that should have been handled by the system. That impedes any potential productivity gains.

The major problem with anything less than full fidelity conversion is that human intervention to fix the artifacts is required. This is both disruptive and costly. Imagine a situation where an important binary document is worked in MSOffice and NOT converted to ODF at the headpoint by daVinci. That binary document then flows through the system and is converted to ODF and further processed at an OpenOffice workstation. There is some loss of presentation, but no big deal for the OpenOffice wiz. From there it goes to a Workplace enabled workstation where it is processed and responded to. No problemo, WorkPlace can read and write ODF perfectly. Including an ODF-XForms document. From there it goes back to an MSOffice workstation for review and data extraction. The individual sends it back to the WorkPlace desktop and asks for a Word 2000 .doc format. If it is an ODF-XForms document, there is no way to convert it back to Word 2000 binary .doc. If it is a compound document of any complexity, there will be loss in the conversion. No matter what happens, this process simply isn't going to work. In fact, no one in their right mind would ever set up or allow for such a poorly mixed environment in the first place.

Enter daVinci. The conversion of binary documents occurs at the headpoint of the process, Microsoft Word, and it is full fidelity. No need to waste expensive human attention fixing conversion artifacts. Once the process is in ODF, these documents can be passed through a mixed environment without loss of that original fidelity.

Let's take for example a more extensive and document centric business process. One that handles a trillion dollars per year in transaction-after-transaction volumes; the real estate transaction industry. For each and every transaction many professional participants must come together, exchange hundreds of documents, work under a strict and unforgiving schedule, and do this with a perfection and data exactness that, if it fails, could entirely wipe out the life time savings and asset accumulations of their clients. Today the entire real estate transaction is driven by paper documents, just as it was 50 years ago. The only difference is that today much of data and document content is provided by back end data processing systems. Almost none of which are able to talk to each other.

The documents involved must be continually reviewed and re written to make certain the data and information is current. The cost and availability of a mortgage changes by the hour. A bank might pre-approve the buyer and the property to be purchased. Even lock in the rates. But there is always the possibility that secondary mortgage market evaluations, purchases, and constantly changing loan ratios can take a lender out of the market on the very day they are scheduled to deliver a loan product. It's all an incredibly time-sensitive process.

The average real estate transaction brings together many professionals: buyer and seller representative real estate agents, their brokers and transaction coordinators, the buyers and sellers, their attorneys, a lender, a title company, inspection companies, insurance companies, appraisers, contractors filing contingency removal estimates, and key information systems providers such as the Multiple Listing Service, county records, probate courts, and courts of law recording such things as liens and encumbrances. All in all, every real estate transaction has at least twelve professionals, each having to deliver specific documents and reports, and each having a back end data processing system different and unable to speak with any other back end system. It is a mess. It is impossible to automate this process without a universal digital file format that can be produced in and translated back into the many proprietary back end systems involved. And do so with absolutely perfect fidelity of document conversion and data extraction.

So here we have a paper driven process that costs consumers upwards of 12% of the actual transaction. And it's a trillion dollar a year industry once all the before and after sale costs are factored in.

A common situation in the real estate industry is that of the professionals having to exchange documents electronically. Faxing is great for paper, but there is a problem of version control as the volume of documents approaches the final closing. Most of the industry relies on eMail attachments as the primary exchange method. What a nightmare. To exchange documents, participants first have to synchronize the versions of Microsoft Office used to produce these usually compound and data centric transaction instruments. They call it version madness in that there is no way to exchange Microsoft Office documents without first having everyone involved using exactly the same version. Meaning, the quality of fidelity conversion between different versions of MSOffice is a killer. Data loss occurs unpredictably. It is unacceptable.

How do you force everyone who must participate in a real estate transaction to all use the exact same versions of software? Well, the same way governments routinely force tax payers to use the same versions of Microsoft Office. You either get the right versions or you don't get to participate.

There are other problems with eMail attachments of proprietary binary documents. Let's say you are exchanging a sales report spreadsheet with inventory, pricing, availability, special discounts or price cuts, sales, customer name and ID, etc. Every time that spreadsheet changes hands, the data has to be verified and updated. The more rapid and wide the exchange, the more intense the data management - data extraction problem becomes. It is a mess. A dangerous and risky mess.

Those spreadsheets could have important formulas and data bindings. Using the XForms or Jabber P2P data binding models, an ODF spreadsheet document can manage itself. Every time the sales people connect to the Internet, the inventory, discounts, sales totals, etc fields can be updated by "the system" with exacting precision. The end user no longer has to worry about data verification and integrity. The system takes care of it, no matter how fast and furious the changes occur. It's automated within the document. Bullet proof. The same can be done with a Microsoft EOOXML document.

However, try to convert one of these documents, and risk dropping formulas, data bindings, rules based routing, or workflow scripting, and the whole process will come to a screeching halt. As it should. This is but another way some proprietary vendors lock in their customers. If you can't do a full fidelity conversion, and be confident of the results, the only alternative is to turn to a single vendor and year after year pay the price.

Enter the OpenDocument Foundation. Since early 2003, a loose gaggle of ODF developers had been seeking methods to improve the OpenOffice.org ("OOo") conversion of Microsoft Office binary formats to what later evolved into ODF. By December of 2003, the OOo community actually launched an official project to develop an ODF plugin for Microsoft Office. The difficulties were enormous, and at the time there was no perceived “business case” for the effort that might have justified allocation of sufficient resources from the Sun Microsystems StarOffice group, the primary developers for OpenOffice.org.

However, as Microsoft began its own long march to XML formats, openly discussing important MS binary format to XML conversion issues, these “difficulties” began to be understood. In 2005 there were a series of breakthroughs based partly on released information from Microsoft. The breakthroughs also sprang from our own intuition of what was really behind some of the oddities in the Microsoft approach. Here's a quick list of reference materials Microsoft provided that really helped:

• Office 2003 XML Reference Schemas

• Overview of WordprocessingML [Word 2003 XML Reference]

• Transforming Word Documents into the XSL-FO Format

• the WordProcessingML Transform Inference Tool

When Massachusetts put out their request for information, as it happened we were only an installation program away from our first ODF 1.0 compatible plugin for Microsoft Word, the most heavily-used application in Microsoft Office. In June Massachusetts sent us our first set of 150 primary importance test documents. We sent them their first plugin install, which was dialed for perfect interoperability with the ODF 1.0-compliant reference implementation, OOo 2.0. In August, we sent Massachusetts the first da Vinci version of the plugin, which was dialed for perfect, 100% round trip conversion fidelity with the legacy of Microsoft Word binaries.

A lot happened in Massachusetts between June and August of 2006. The short form is that as Massachusetts came to understand how our plugin worked, we came to understand the true nature of the challenges they faced. We saw up close and personal the challenges of the Microsoft Office bound business processes and the need for perfect fidelity “round-tripping” of documents between different formats within automated business processes. The Microsoft assistive technologies were never an issue for us because the plugin works natively within Word. Microsoft Word handles the assistive technologies, and all the plugin does is the conversions from Word's in-memory-binary-representation to ODF <> and back.

The situation with the MSOffice bound business processes was such that everyone agreed we had to have a much higher fidelity converting those binaries than the extraordinary 85 per cent fidelity credited to the OpenOffice.org file conversion engine. We had to have perfect fidelity because there was no reasonable expectation of ever successfully migrating those business processes to a Microsoft Office alternative like ODF-ready OpenOffice.org, StarOffice, WorkPlace or Novell Office. Such a re-engineering of the business processes would be costly and beyond disruptive.

If however we could achieve full fidelity conversions of legacy Microsoft binary documents to ODF, and were able to guarantee the roundtrip process of these newly christened ODF documents in a mixed desktop environment, one comprised of ODF-enabled Microsoft Office, OpenOffice.org, Novell Office, WorkPlace, and KOffice, the existing MSOffice bound business processes could continue being used even as new ODF ready workstations were added to the workflow. Massachusetts could migrate to non-Microsoft Office software in manageable phases, restoring competition -- and sanity -- to the Commonwealth's software procurement program.

If on the other hand, there is no full fidelity conversion to ODF of legacy documents available at the head point of the migration -- Microsoft Office -- then the business process will break under the weight of users having to stop everything to fix and repair the artifacts of lossy file conversions. What Massachusetts discovered is that users will immediately revert to a Microsoft-only process wherever the business process system breaks down due to conversion fidelity problems. It is a productivity killer and a show stopper for migration to ODF-supporting software.

Most people believe that full fidelity file conversion is impossible, largely influenced by the poor fidelity generally experienced in converting documents from one vendor's binary file formats to another vendor's binary file formats. The major reason some people transfer such beliefs to the daVinci plugin is that they do not realize the plugin works natively inside Microsoft Word. The Word program itself performs the conversions using its internal processes for native support of a file format. The da Vinci plugin triggers the native Microsoft Word conversion process, intercepts the output at precisely the right moment, and maps it to ODF. When opening an ODF document, da Vinci simply reverses the process (although there is nothing simple about what da Vinci does or how da Vinci does this).

Think of it this way. The Microsoft Compatibility Pack is a EOOXML plugin for older versions of Microsoft Office. If these versions of Microsoft Office can use a plugin to perfect a conversion process between legacy binary formats and EOOXML, the same process can be used for ODF. In fact, it's actually easier to perfect a similar conversion process to and from ODF instead of EOOXML, particularly for Microsoft. Easier in that the ODF specification is lean and clean by comparison and very carefully structured. Even easier if you have the blueprints for those binary file formats.

My educated guess is that it would take all of two weeks for Microsoft to write a Microsoft Word ODF plugin. Indeed, Microsoft developers reportedly told Massachusetts that it would be "trivial" for them to implement ODF in Microsoft Office. If they can do it for something as complex and convoluted as EOOXML, ODF will be a snap. The trick is not in XML. It's in having the secret legacy binary format blueprints.

There's a reason we call our plugin “da Vinci”. Yeah, you guessed it. There is in fact a secret code needed to unravel the remaining mysteries of the legacy binaries; the elusive 15% just beyond the reach of the OpenOffice.org conversion engine.

The da Vinci plugin is conformant with version 1.2 of the ODF specification still working it's way through the OASIS ODF sub-committees and is dependent on features of the draft specification that may conceivably change. Massachusetts was well aware of this fact, and knew we were treading on dangerous ground because ODF 1.2 was not yet final. They gave us the go-ahead anyway. Such is the importance of round trip high fidelity conversions with the legacy binaries.

Why ODF 1.2? Microsoft is notorious for frequently changing its file formats. There is even a Microsoft document jarred loose in the antitrust litigation where they brag about maintaining a "moving target" for other vendors' efforts to achieve interoperability with Microsoft software. For example, there is no single "DOC" file format. There are versions after versions of just the word processing format, all different. And their specifications are still secret. Imagine the volumes of unspecified binary objects, application version-specific or add-on specific processing instructions, and aging system dependencies buried in those billions of legacy binary files. There is no telling when or where they will pop up and da Vinci will have to somehow handle what are otherwise one unmappable black hole after another.

The easy solution is to simply wrap (enclose) these dark objects in foreign element tags. While this method is perfectly legal ODF because of ODF's strong interoperability features, see e.g., ODF v. 1.1 section 1.5, it comes at the cost of interoperability with other ODF-ready applications. Sure, inside da Vinci-enabled Microsoft Word the conversions and handling are perfect. Absolutely perfect. Send one of these dark object loaded documents to OpenOffice though and the application has no idea what to do with it.

ODF 1.2 provides da Vinci with two important advantages. The first is that ODF 1.2 ready applications “must” preserve tags whether they understand them or not. This is important to round-tripping. ODF 1.2 applications become routers of documents, instead of document end points.

The second aspect of ODF 1.2 is that of metadata XML/RDF based flexibility. Instead of wrapping these dark objects in a non descriptive tag, ODF 1.2 will enable da Vinci to wrap with a loosely descriptive generic element tag, but then nail into the more flexible attribute model everything da Vinci can describe about the blob. This gives ODF 1.l2 ready application like OpenOffice or KOffice a better than even shot at reading and rendering dark objects that would otherwise have been dropped. Over time, the elusive 15% of lossage during their conversions of Microsoft binary formats will disappear.

This is a far better way of handling dark objects than what EOOXML tries to do. Under that specification, the implementing application simply wraps the object and passes it along. There is no effort to describe these objects in XML-speak except for the tag name. The specifics needed to render these objects are left out. Which means, only those applications with access to the secret binary blueprint can properly render these dark spots.

As a proof of concept, da Vinci provides a most valuable service. First of all it proves the ODF can handle anything EOOXML can handle. ODF can handle everything and anything Microsoft Word can throw at it. Second, da Vinci proves that Microsoft developers could at least as easily have written an ODF plugin for Microsoft Word as they did an EOOXML plugin. For Microsoft, knowing the secret binary blueprint as they do, this simply can not be seen as a technical challenge. It was purely a business decision. Third, da Vinci demonstrates that it is possible to perfect a migration to ODF at minimal cost of disruption to existing business processes and application bound routines. The migration to ODF is possible. The starting point of this migration is exactly where Massachusetts ITD thought it would be – with ODF plugins forMicrosoft Office.

We have not yet developed plugins for the other major applications in Microsoft Office; Excel and Powerpoint. However, we have investigated the involved issues enough to have confidence that those tasks are also achievable. The proof of concept and working prototype of Excel is completed. Only a proof of concept for PowerPoint exists.

How to:

Interact with Microsoft business processes using OpenDocument

One of the interoperability challenges the OpenDocument format faces is that of the developers of existing non-Microsoft applications trying to implement a file format that follows after the fact of the design and development of the specific layout engines. The tradition of productivity applications is that the application was written with the file format an afterthought based on the in-memory binary representation needs of the application. Every application had a perfect binary dump of these in-memory-representations, and from there conversions to other file formats began. Early and open cooperation between existing application providers in the specification development process is essential for later interoperability.

One hope for ODF is that a new generation of ODF-ready applications will appear. And these applications will be geared from the git-go to produce perfect, highly interoperable, and round trip ready ODF.

To that end the da Vinci developers began work on the InfoSet Engine and API. This is designed to be a lightweight conversion (and someday layout) engine useful to mobile devices, server side applications, rapid development desktop productivity applications, and the easy ODF enabling of IDE's. The principal is the same as da Vinci except that the InfoSet Engine must do the MS binary conversions without the native assistance of the Microsoft Office applications. That is a step for which da Vinci relies on Microsoft Word to do natively.

So it's safe to say that the more we watch da Vinci do its work, the more we learn about how Microsoft Office perfects its own conversions. This knowledge is then passed to the InfoSet conversion algorithms. Going native gives one a whole new perspective on an enormous problem that spans decades.

The InfoSet API is intended to provide developers with easy-to-call interfaces that do all the conversions for their applications and services. We have not yet begun work on the InfoSet layout engine, but so far this is one of the more promising ODF projects we've worked on, maybe more important and certainly more lasting than da Vinci. We believe that da Vinci will have its day, cracking open Microsoft Office. But that day will only last until the Microsoft Office bound business processes and their related binaries are migrated to ODF-ready information processing chains. After that, (perhaps a three-year process) the day belongs to InfoSet and apps fully supporting ODF such as OpenOffice.org.

InfoSet is important because the world is moving towards Internet-enabled information processing chains that mesh desktops, servers and device-based information systems. We need ODF-ready chains, and the only way to get there is to make it as easy as possible for application developers to provide interop-perfect ODF.

To understand this urgency, one need only to look at the Microsoft Vista Stack, an information processing chain where every application speaks perfect EOOXML. The head point of the Vista chain is of course Microsoft Office. But the core of this chain is the Exchange/SharePoint/Groove Hub. The Hub is where information is aggregated, sorted, managed, scheduled, re purposed and bound into intelligent collaborative workflows. The driving force is that of the portable XML document, with data being bound to interactive document objects, extracted and worked as the business process flow demands.

These chains are extremely productive. So much so that i believe every Microsoft Office business process will migrate as soon as possible to either the Vista – EOOXML chain, or, to an ODF ready alternative.

Some interesting thoughts here. The da Vinci plugin and those on the development road map for the other major Microsoft Office apps will be able to convert the Microsoft Office head point to an ODF pump. This opens the way for ODF ready Hubs. Zimbra comes to mind. An InfoSet-enabled version of Zimbra (just an example :). Once the Microsoft Office bound business process moves to the enhanced productivity of an ODF ready Hub, the desktop workgroup space opens up for any ODF enabled alternative to Microsoft Office. That means, for example, any Linux desktop running OpenOffice.org or KOffice can fully participate in these Hub-hosted workflows.

Many will no doubt disagree. But I personally believe the above information processing chain scenario is the secret to Linux desktops running wild and finally replacing the 485 million Microsoft Office bound desktops that dominate business.

Related information

The da Vinci plugin and InfoSet Engine – API have been presented at technology conferences and to interested governments serious about migration to ODF. This includes:

• EU IDABC Experts Group Contacts: Barbara Held, Peter Strickx, Elmar Geese

• Commonwealth of Massachusetts ITD Contacts: Louis Gutierrez, Timothy Vaverchak, Claudia Bowman

• State of California ITD Contacts: Bill Welty, CIO of the Air Resources Board

The Massachusetts Plugin Feature Priority List:

• Microsoft Word perfect conversion fidelity plugin (ODF 1.2)
• Plugin Accessiblity Enhanced Feature Set (ODF 1.1 compatible)
• MSExcel perfect conversion fidelity plugin (ODF 1.2 with formulas)
• MSPowerPoint plugin
• Roundtrip Interoperability with ODF 1.2 version of OpenOffice.org
• XForms Interface for da Vinci enabled Microsoft Word
• XForms Interface for MSExcel and MSPowerPoint
• PDF - ODF Digital Signature Interface for Microsoft Office

OASIS Open Office XML Application Vendors:

The interoperability of any universal file format very much depends on the open cooperation of existing application vendors. New applications written to the universal file format specification will of course have near perfect interoperability, but there are serious application layout and feature impedance difficulties for existing applications trying to implement a universal file format, no matter how open or consensus driven the specification development process happens to be. ODF was most fortunate to have a number desktop and enterprise application developers participati ng in the process. Although Microsoft was part of the original group of founding members, and continues to maintain that membership in the OASIS ODF XML TC, they have never actively participated. Their "observer" status did however give Microsoft direct access to all discussions, meetings, conference minutes, eMail exchanges, listserves, documents and proposals. Clearly if Microsoft had participated in the ODF process, the stubbornly defiant interoperability problems we see today between ODF and Microsoft's own proprietary XML could have been resolved.

Resources:

• Wired Magazine article, "MS Fights to Own Your Office Docs"
• Rob Weir, "The Vast Blue-Wing Conspiracy", "A Foolish Inconsistency", "The Chernobyl Design Pattern", Calling Captain Kirk, Guillaume Portes Redux, How to hire Guillaume Portes
• Open Stack, "Game Time for OpenDocument"
• Groklaw: GrokDoc, EOOXML at JTC-1 and EOOXML Objections
• Groklaw: Searching for Openness in Microsoft's OOXML and Finding Contradictions