# PD4ML PDF Optimizer API - Programmer's Manual _Available starting from PD4ML v4.1.1_ ## 1. Architecture overview [`com.pd4ml.pdf.optimizer`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/package-summary.html) cleans a PDF down to exactly what's actually reachable, and collapses its cross-reference/trailer history into one fresh cross-reference section - built entirely on top of [`com.pd4ml.pdf.cos`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/package-summary.html), the same foundation [`com.pd4ml.pdf.sign`](https://pd4ml.com/javadoc/com/pd4ml/pdf/sign/package-summary.html) and [`com.pd4ml.pdf.merge`](https://pd4ml.com/javadoc/com/pd4ml/pdf/merge/package-summary.html) are built on. **Dependency direction is one-way**: `com.pd4ml.pdf.optimizer` depends on `com.pd4ml.pdf.cos` (and, for optional output encryption, [`com.pd4ml.pdf.cos.security`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/security/package-summary.html)); `com.pd4ml.pdf.cos` never depends on it. ### 1.1 What accumulates in an incrementally-updated PDF Two things pile up in a PDF that has been edited via **incremental update** (ISO 32000-1 §7.5.6) - e.g. by `com.pd4ml.pdf.sign`, [`CosCli`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/cli/CosCli.html)'s `set`/`delete`, or any other tool that patches a file this way rather than rewriting it from scratch: * **Unreferenced objects.** An incremental update only _adds_ new object bodies and xref entries; when it drops a reference (e.g. replacing a page dictionary that used to point at some content), the object that's no longer pointed to anywhere still has a live "in use" xref entry and its bytes are still sitting in the file. [`COSParser`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/parser/COSParser.html) happily parses it into the document's object table right along with everything still actually reachable. * **Trailer/xref history.** Each incremental update appends its own xref section and trailer, chained to the previous one via `/Prev`. `COSParser` already resolves that chain down to the current value of each object number (older, superseded bodies are simply never parsed in the first place - see §1.2), but the chain of old xref sections/trailers itself is still physically part of the file until something rewrites it away. ### 1.2 What `optimize(...)` does about it ``` [ base file ] [ update 1 ] [ update 2, /Prev -> update 1's xref ] \___________object table (COSParser: latest value per object number)___________/ | v walk reachable from /Root and /Info only | v COSObjectImporter (renumbers, dedupes shared resources) | v ContentDeduplicator (merges byte-identical duplicate objects, if enabled) | v COSDocumentWriter (one fresh cross-reference section + trailer) ``` `COSParser` already collapses the chain of revisions down to the current value of each object number - a superseded object body from an earlier revision is never even parsed into the resulting [`COSDocument`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/COSDocument.html). What it does _not_ do is drop an object that a later revision simply stopped pointing to; that object is still "in use" per the xref table and still gets parsed. [`optimize(...)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(byte[])) handles that half: it deep-clones only what's actually reachable by walking from the trailer's `/Root` (and `/Info`, which is otherwise unreferenced by design - it's a trailer-only entry) into a fresh `COSDocument` with compact object numbering, and writes that out with a single, from-scratch cross-reference section and trailer. Anything not reachable is simply never visited, and every old revision's xref/trailer is gone because the output isn't an incremental update at all - it's a full rewrite. The reachability walk and renumbering are done by [`com.pd4ml.pdf.cos.util.COSObjectImporter`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/util/COSObjectImporter.html), a general-purpose deep-clone primitive with no knowledge of optimizing specifically - it's the same class `com.pd4ml.pdf.merge` uses to combine several sources' selected pages into one document (see the [**PD4ML PDF Merge Manual**](../pd4ml-pdf-merge/pd4ml-merge-manual), §1.2, for its full API and identity-preservation guarantees). Here it's seeded from a single document's own `/Root`/`/Info` rather than from several sources' pages, which is the entire difference between "merge" and "optimize" at this layer. By default, the result of that pass is then also deduplicated by content (§3) before being written out. **Output layout.** _From v4.2.0._ [`COSDocumentWriter`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/writer/COSDocumentWriter.html) writes a PDF 1.5 or later document compactly: its objects other than streams packed into Flate-compressed object streams (up to 100 objects each), its cross-reference data a compressed cross-reference stream (ISO 32000-1 7.5.7, 7.5.8) - as PD4ML writes its own PDFs, so optimizing a PD4ML-made file keeps it about its size. The classic layout -- every object on its own, a cross-reference table -- is kept where it is required: for a PDF 1.4 document (PDF/A-1 among them, which allows neither) and for encrypted output (§5.2: a string in an object stream is not encrypted on its own). Linearized output (§6) follows the same rule. The object streams and cross-reference streams of the input file are never copied: they describe the old file's layout, not the document. Before v4.2.0 every output used the classic layout, which made a file whose input packed its objects several times larger. A side effect worth knowing `/Encrypt` lives only in the trailer - nothing under `/Root` ever points to it. Optimizing an encrypted input document (successfully decrypted via the given password) therefore drops the old `/Encrypt` dictionary along with everything else unreachable, and the output is a plain, unencrypted PDF unless `Options.encryptOutput` is used to re-protect it (§4). ## 2. `com.pd4ml.pdf.optimizer` - the public API ### 2.1 Entry point: `PdfOptimizer` ```java PdfOptimizeResult result = PdfOptimizer.optimize(pdfBytes); System.out.println("Removed " + result.getRemovedObjectCount() + " unreferenced object(s)"); Files.write(Paths.get("cleaned.pdf"), result.getOptimizedPdf()); ``` [`PdfOptimizer`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html) has no instance state - every method is `static`. | Method | Notes | | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `optimize(byte[])` / [`optimize(byte[], Options)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(byte[],com.pd4ml.pdf.optimizer.PdfOptimizer.Options)) | returns a [`PdfOptimizeResult`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizeResult.html) | | [`optimize(File)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(java.io.File)) / [`optimize(File, Options)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(java.io.File,com.pd4ml.pdf.optimizer.PdfOptimizer.Options)) | reads the file, then as above | | [`optimize(InputStream)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(java.io.InputStream)) / [`optimize(InputStream, Options)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(java.io.InputStream,com.pd4ml.pdf.optimizer.PdfOptimizer.Options)) | reads the stream fully (not closed), then as above | | [`optimize(byte[], Options, OutputStream)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(byte[],com.pd4ml.pdf.optimizer.PdfOptimizer.Options,java.io.OutputStream)) | convenience: optimizes and writes straight to a destination, still returns the `PdfOptimizeResult` | | [`optimize(File, Options, File)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.html#optimize(java.io.File,com.pd4ml.pdf.optimizer.PdfOptimizer.Options,java.io.File)) | same, file-to-file | ### 2.2 `Options` ```java PdfOptimizer.Options options = new PdfOptimizer.Options() .password("inputPassword") // if the input is encrypted .encryptOutput("openPassword", "ownerPassword"); // opt-in; see §5.2 ``` | Method | Default | Notes | | ----------------------------------------------- | ------------------------- | ------------------------------------------------------------------------ | | [`password(String)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.Options.html#password(java.lang.String)) | `""` | tried as both the user and owner password if the input is encrypted | | [`encryptOutput(userPassword, ownerPassword)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.Options.html#encryptOutput(java.lang.String,java.lang.String)) | _(output stays plain)_ | AES-256 output encryption, the modern default | | [`encryptOutput(PdfEncryptor.Options)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.Options.html#encryptOutput(com.pd4ml.pdf.cos.security.PdfEncryptor.Options)) | _(output stays plain)_ | full control - algorithm choice, permission bits (§5.2) | | [`deduplicateContent(boolean)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.Options.html#deduplicateContent(boolean)) | `true` | merge byte-identical duplicate objects into one; see §3 | | [`linearize(boolean)`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizer.Options.html#linearize(boolean)) | `false` | write the output linearized ("Fast Web View"); see §6. _From v4.2.0._ Not together with `encryptOutput` | ### 2.3 `PdfOptimizeResult` ```java byte[] getOptimizedPdf(); int getInputByteCount(); int getOutputByteCount(); int getInputObjectCount(); // total indirect objects the input's object table had -- includes unreachable ones int getOutputObjectCount(); // always exactly what's reachable from /Root and /Info, post-dedup // (not the objects a linearized layout adds, see §6) int getRemovedObjectCount(); // getInputObjectCount() - getOutputObjectCount() int getDuplicateObjectCount(); // subset of removed: merged as duplicate content specifically (see §3) ``` [`getInputObjectCount()`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizeResult.html#getInputObjectCount())/[`getOutputObjectCount()`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizeResult.html#getOutputObjectCount()) let a caller confirm something was actually cleaned up (or confirm nothing needed to be - [`getRemovedObjectCount()`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizeResult.html#getRemovedObjectCount()) is legitimately `0` for an already-clean document) without re-parsing either the input or the output themselves. [`getDuplicateObjectCount()`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizeResult.html#getDuplicateObjectCount()) narrows that down further: it's `0` whenever `deduplicateContent` was turned off, and also legitimately `0` when it was left on but nothing turned out to be duplicated. ## 3. Duplicate-content dedup By default, `optimize(...)` also merges byte-for-byte duplicate objects still reachable after the pass above - typically the same font, image, or color space embedded twice under two different object numbers (e.g. left over from two separate incremental updates each adding their own copy). Every reference to a duplicate is rewritten to point at one surviving ("canonical") copy, and a second reachability pass drops the now-orphaned duplicate and renumbers everything compactly again - the diagram in §1.2 shows where this fits (`com.pd4ml.pdf.optimizer.internal.ContentDeduplicator`). ### 3.1 What counts as a duplicate Two indirect objects are duplicates when their fully-resolved content is identical: the same dictionary entries (order-independent - PDF dictionaries are unordered by definition), the same array elements (order-sensitive - arrays are ordered), the same raw (still-encoded) stream bytes, the same primitive values. An indirect reference nested inside either object counts as identical content only once the object it points at has itself already been found to be canonical - comparison walks the object graph bottom-up, so a parent referencing two now-identical children is correctly recognized as a duplicate of another parent that references the single canonical child. ### 3.2 What's deliberately excluded Two kinds of object are never merged into each other, even when byte-identical: `/Type /Page` dictionaries, `/Type /Annot` dictionaries, and interactive form field dictionaries (anything carrying an `/FT` entry, terminal or not). Two blank/default form fields, for instance, can easily be byte-identical today yet get filled in independently later by a viewer that writes into a specific field by object identity - merging them would make filling in one silently affect the other. Everything else content-bearing - fonts, images, ExtGState/color-space/function dictionaries, content streams, and so on - has no such identity significance and is always safe to merge when byte-identical. ### 3.3 Turning it off ```java PdfOptimizer.Options options = new PdfOptimizer.Options().deduplicateContent(false); ``` Disable this to skip the extra content-hashing pass on a large document where every bit of processing time matters and duplicate content isn't expected - the reachability cleanup (§1-§2) still runs either way. ## 4. Error handling Every checked failure from `com.pd4ml.pdf.optimizer`'s public API surfaces as one type: [`PdfOptimizeException`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/PdfOptimizeException.html) (`getCause()` carries the real underlying exception, if any). | Message (paraphrased) | Cause | | ------------------------------------------------------------------------------------ | ---------------------------------------------------------- | | "Failed to parse input PDF" | wraps a [`COSParseException`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/parser/COSParseException.html) - malformed/non-PDF input | | "Input PDF is encrypted and the given password (or no password) did not open it" | wrong/missing password for an encrypted input | | "Input PDF has no /Root in its trailer; nothing to optimize" | malformed input with no usable Catalog reference | | "Failed to encrypt optimized output" | wraps a [`COSSecurityException`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/security/COSSecurityException.html) from [`PdfEncryptor`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/security/PdfEncryptor.html) | | "Linearized output cannot be encrypted" | `linearize(true)` together with `encryptOutput(...)` (§6) | | "Failed to write optimized PDF" | wraps a [`COSException`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/COSException.html) from [`COSDocumentWriter`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/writer/COSDocumentWriter.html) | | "Failed to read ``" / "Failed to write ``" | wraps an `IOException` from a `File`-based overload | ## 5. Encryption ### 5.1 Input `COSParser` decrypts transparently while parsing, so an encrypted input document is handled exactly like a plain one throughout the rest of the pipeline. `optimize(...)` rejects an input only when [`COSDocument.isEncrypted()`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/COSDocument.html#isEncrypted()) is true **and** `getEncryptionError()` is non-null, i.e. the given (or empty) password genuinely failed to authenticate as either the user or the owner password. ### 5.2 Output The optimized output is **not** encrypted by default, even if the input was (see the callout in §1.2 - the old `/Encrypt` dictionary is simply unreachable garbage from the reachability walk's point of view, and gets dropped like anything else). Opt in explicitly: ```java PdfOptimizer.Options options = new PdfOptimizer.Options() .encryptOutput("openPassword", "ownerPassword"); // AES-256 (default algorithm) PdfOptimizer.Options fullControl = new PdfOptimizer.Options() .encryptOutput(new PdfEncryptor.Options() .userPassword("openPassword") .ownerPassword("ownerPassword") .algorithm(PdfEncryptor.Algorithm.AES_128) // RC4_128 | AES_128 | AES_256 .permissions(-4)); // raw /P bits, default "everything allowed" ``` `PdfEncryptor` (`com.pd4ml.pdf.cos.security`) is shared, general-purpose infrastructure - the same class `com.pd4ml.pdf.merge`'s `PdfMerger.encryptOutput` uses. See the [**PD4ML PDF Merge Manual**](../pd4ml-pdf-merge/pd4ml-merge-manual), §5.2, for how it works internally (it's the write-side counterpart of the read-only [`StandardSecurityHandler`](https://pd4ml.com/javadoc/com/pd4ml/pdf/cos/security/StandardSecurityHandler.html)). ## 6. Linearized output (Fast Web View) _From PD4ML v4.2.0._ ```java PdfOptimizer.Options options = new PdfOptimizer.Options().linearize(true); PdfOptimizer.optimize(new File("report.pdf"), options, new File("report-web.pdf")); ``` A linearized PDF ("Fast Web View", ISO 32000-1 Annex F) is laid out for reading over a network: a viewer that loads it progressively -- a browser plugin, a web viewer requesting byte ranges -- shows the first page as soon as it has arrived, and then jumps to any other page without waiting for the whole file. Acrobat lists such a file as "Fast Web View: Yes". The optimizer writes the cleaned-up document (§1-§3) in that layout: - the first page's objects first, together with a linearization dictionary and hint tables telling a viewer where every page's objects are; - the other pages' objects, then the objects shared between pages, after it. A PDF 1.5 or later document is written compactly here as well: object streams and cross-reference streams, as in the plain output (§1). The catalog and the page objects stay out of object streams, as ISO 32000-1 7.5.7 requires for a linearized file, and an object stream holds only objects of one part of the file, so a viewer loading a page does not fetch the objects of other pages. A PDF 1.4 document keeps cross-reference tables and every object on its own. The layout and hint tables follow ISO 32000-1 Annex F as implemented by [qpdf](https://qpdf.readthedocs.io/en/stable/linearization.html), so `qpdf --check-linearization` accepts the result. Page attributes a page inherits from the page tree (`/Resources`, `/MediaBox`, `/CropBox`, `/Rotate`) are copied onto each page, as linearization requires; the document looks the same. A linearized file is a little larger than the plain optimized one: the hint tables and the linearization dictionary are extra. They are not counted in `getOutputObjectCount()` (§2.3), which stays the number of the document's own objects. **Not together with encryption.** Linearized output cannot be encrypted: `optimize(...)` throws a `PdfOptimizeException` ("Linearized output cannot be encrypted") when both `linearize(true)` and `encryptOutput(...)` are set. An encrypted input is fine -- it is decrypted with `password(...)` (§5.1) and written linearized and plain. **Linearize last.** Any later change to the file -- an incremental update, such as adding a signature or form values -- leaves it no longer linearized for viewers. Linearize as the last step. ## 7. Not optimized (out of scope) **Stream recompression** - an uncompressed or suboptimally-compressed content/image stream is copied through as-is; `optimize(...)` never re-encodes stream data. This is a size optimization orthogonal to both reachability cleanup and content dedup (§3) - a different kind of work, not attempted here. ## 8. Threading `PdfOptimizer` holds no state at all (every method is `static`); call it freely from any thread. Each `optimize(...)` call is fully independent - there's no shared mutable object comparable to `PdfMerger` that would need one instance per operation. `com.pd4ml.pdf.optimizer.internal.ContentDeduplicator` (§3) is likewise stateless between calls - it builds fresh hash maps for every `optimize(...)` invocation. ## 9. See also * [PD4ML Programmer's Manual](https://pd4ml.com/docs/pd4ml-manual.md) - the core HTML/CSS-to-PDF engine `com.pd4ml.pdf.optimizer` and the rest of the post-processing toolkit sit alongside. * [pd4optimizecli Reference](pd4optimizecli-reference) - the companion command-line documentation for [`com.pd4ml.pdf.optimizer.cli.Pd4OptimizeCli`](https://pd4ml.com/javadoc/com/pd4ml/pdf/optimizer/cli/Pd4OptimizeCli.html) * [PD4ML PDF Merge Manual](../pd4ml-pdf-merge/pd4ml-merge-manual) - `com.pd4ml.pdf.merge`, which shares `COSObjectImporter` and `PdfEncryptor` with this package * [PD4ML COS API Manual](../pd4ml-pdf-cos/pd4ml-cos-api-manual) - standalone `com.pd4ml.pdf.cos` programmer's manual (object model, parsing, writing, filters, encryption reading) * [pd4coscli Reference](../pd4ml-pdf-cos/pd4coscli-reference) - command-line reference for `com.pd4ml.pdf.cos.cli.CosCli` * [PD4ML XFDF Manual](../pd4ml-pdf-xfdf/pd4ml-xfdf-manual) - [`com.pd4ml.pdf.xfdf`](https://pd4ml.com/javadoc/com/pd4ml/pdf/xfdf/package-summary.html), which offers its own `--optimize` post-import cleanup delegating to this same package