The documents chosen for the corpus reflect a wide a variety of sources, in order to avoid skewing our results to a particular kind of data. It is possible that the real uses of XML data will be different from the somewhat artificial uses listed above. Thus, any conclusions drawn based on our corpus must be tentative. Nevertheless, because XML is not yet a mature, everyday technology, this would likely be true of any alternative proposed corpus.
We compressed each document using "gzip"[10], "bzip2"[19], "ppmd+"[5,21] (with a context bound of 5), and "ppm*"[4,21]. The compression results, reported in bits per original XML document character, are shown in Table 1(a). The first column, ``XML'', lists the sizes of the documents in bytes, and the other columns show the bits per symbol rate of the compressors. For each file, the best compression in a table is in bold, and the best compression overall (including Tables 1 and 3) is in bold italic. At the bottom of each table is listed the average bit rate, weighted average (total compressed bits divided by total bytes), and compression time on the corpus.
Markup tends to be more redundant and compressible than text. _Textual_ documents (mostly text, little markup) tend to have high bit rates of around 1.5-2.0. This is below the 2.1-2.2 bpc usually seen with English text, and reflects an averaging of very redundant markup and less redundant text. _Structured_ documents (mostly markup, little text) compress up to several times smaller than text documents. Documents mixing text and structure tend to fall in between these extremes. This distinction between textual and structural data is a recurring theme. Some compressors compress text well, others compress structure well, but few compress both well.
There is no clear winner. The "bzip2" compressor is best overall; it does 20-30% better than "gzip" and about as well as "ppm*". However, each compressor performs poorly on at least one document. For structured documents, there is a large gap between "gzip" and "ppmd+" and the other compressors (as much as a factor of 4), but otherwise there are no obvious _a priori_ reasons to believe a particular compressor will be effective on an unknown document.
Because all XML documents are generated by a fairly restrictive context-free grammar, perhaps grammar-based compression techniques such as Nevill-Manning and Witten's SEQUITUR [18] or Kieffer and Yang's grammar-based codes [13] would do better. We have experimented with a version of YK compression and found that it compressed about as well as "bzip2" and "ppm*" on our corpus. Nevertheless other forms of grammar-based compression, perhaps tailored to XML, might offer improvements.
Plain-text XML compression is easy, fairly fast, and relies on existing, robust compressors. On the other hand, good compression is either very slow (using "ppm*"), or fairly slow and off-line (using "bzip2"). Off-line compression is undesirable for XML because it forces a long wait before document parsing and processing can begin. Moreover, there is no reason to believe that text compressors can take full advantage of the redundancies present in structured XML data, especially in light of the high variation in bit rates among the compressors.