Fun with Microsoft Word-made PDFs

A potential client came to me the other day looking for a solution to replace something called "adlib."  I am not sure what that is but what it does is munge PDF files to handle certain aspects of digital signatures.

Basically the company uses Windows-based technology and Word to create PDFs and they needed some help with that.  Particularly in the area of tracking objects through that process.

So the idea is that Word makes it hard to do certain things, like control placement of objects in PDFs it generates.  So, for example, if you have a large Word document and you want, say, a signature in a certain place in the resulting PDF (created by Word) you can't do it.

So people instead come up with novel solutions using these other products, like "adlib."

So this potential customer asked how could you put something into Word and later come up with the exact location of that "thing" in the Word-generated PDF (Word sort of flows things along and you cannot easily know exact things about how this works).  Further, they asked, what could you tell us about the location (page, x, y, width, height) of the object in the final PDF.

So I came up with a feature to the pdfExpress Manufacturing product line to allow someone to place an image in Word, generate a PDF, and locate the image along with its exact position, page, height and width.

Basically the idea is this: create a "placeholder" image (though its easy to support non-image objects like lines and curves and circles as well).  Set certain attributes in that image.  Place it into Word using one of the myriad of Word API's from something like .NET.

Now Word does bad things to images which appear to be beyond he control of mere mortals, e.g., re-sizing them, re-rasterizing them, etc.  But, with a specially crafted image, you can make it work very well.

Once Word spits out the PDF you pass it though pdfExpress Manufacturing.  pdfExpress removes all these specially marked images (or other objects) and emits corresponding page, position, height and width data to tell you were the objects that were removed were.

Now you can post process the PDF to add annotations and other Word/PDF junk that apparently many government agencies and big corporations like to see in PDFs, e.g., digital signature junk, etc.  Of course, pdfExpress and friends make this easy as well.

pdfExpress can do this on huge PDFs, say 100K pages, as well as smaller PDFs.

PDF Structure

I have spent 15 years working with PDF structure.  I am impressed with how little various "hackers" and "security" people really know about its internal format - how little the tools really do.

Most tools simply dump out superficial dumps of streams and other content structures.

The real vulnerability of PDF is the CosObj structure and how its processed.  Most constructs are poorly defined in the "standard" and therefore subject to "issues".  Acrobat has been a source of consistent issues for me for the last decade and a half or so...

PDF at the Forefront of iOS/iPhone Hacks...

I have neglected this blog for many, many months.   But today I have some startling news PDF.

Virtually every version of iPhone/iOs/iPad/ from Apple appears to have a serious PDF security issue.

I found this due to the site jailbreakme - not that I am interested in jailbreaking per se - but it just showed up like chloroform chlorophyll on Casey Anthony's web browser.

The issue has to do with JavaScript (surprise, surprise).  The problem can also surface in things like Safari and other iOS applications (presumably) that use browser technology.  You just go to this site, click on some links and viola - your iDevice is jailbroken.  (No messary hacking or mucking about with reds0wn and so forth.)

I have tracked down several sites (this, this, and this) with gory technical details but basically it works like this:

1) Inside a PDF you embedded some malicious JavaScript.  There are a number of ways to do this.  As an expert I am duty bound not to spill the beans.

2) When the PDF is loaded the JavaScript either calls some specific JS functions known to corrupt memory or use something called HeapSpray to fill memory with things you would like to execute.

3) Once the memory in the JavsScript memory in the PDF interpreter is corrupted and local OS ARM code is inserted by JavaScript the code is ARM code executed.  The code then uses an exploit to access "root" space which is the highest-privileged operating mode for iOS.

Typically this exploit (as of July 9th, 2011) is the iOS IOSurface exploit - which creates bogus parameters for the IOSurface API call.  Since IOSurfaces work in kernel, aka root, space, the hack is basically complete.

4) The IOSurface, now corrupted, executes code for the hack as root.  It checks for a previously installed hack (via the existence of bash).  If none is found it then downloads something called 'wad.bin' which contains more executable code as well as a Cydia installer for jailbreaking.

Interestingly enough these PDF JavaScript problems have been around for years.

My guess is that there are a huge variety of PDF hacks one could create by carefully manipulating a PDFs structure to cause any number of equivalent to JavaScript hacks.

I have believed for many years that Adobe no longer cares about PDF or print.  And, because of this, their policies for upgrading the CS suites, and the fact that PDF and PS are outsourced to a foreign land where no one cares about print - we have holes that go unpatched for a long, long time.

Now certainly there are more consequences to jailbreaking than I am implying here - irreversible things, at least for iPad 2's - so don't just rush out to try this.

However, I am confused by some of this.  Certainly it would seem that the right thing to do on an iPhone would be to install some App that runs other apps in kernel mode - rather than mucking up the security of the iOS in general by jailbreaking.  That is, create an App that is root-owned, that runs in kernel mode, but is otherwise a normal app.  Then fire up that app when you wanted to do things.

That way you're not breaking security....

At any rate... Some day Adobe will fix all this.  There have been problems like this before (see the links) and there will surely be more in the future.

At any rate its nice to see that PDF, Adobe's legacy, is getting a whole new lease on life in the iOS world.

Where's the PDF Development?

Though Adobe has passed PDF off into the public domain there are still active pockets of PDF development going on around the world.

Google's CHROME seems to have some active PDF development activity though the focus seems to be on making Chrome use non-Adobe plug-ins for PDF rendering.

There is also "PDF Quick View" when you do a Google search that turns up a PDF (probably in gmail and other Google products as well - but I don't use those...).  Its covered in this blog.  This seems to work well over all - its generally a lot nicer than fooling around with the Adobe plug-in to see something.

The sad part of all this, though, is that Google is too stupid to understand the full picture of PDF, the one that includes print.  (See this from Lone Wolf.) They only care about PDFs that fit into their model.  A little digging will demonstrate that they don't fully support PDF rendering yet.  I suppose they are working on it and one day will.  But as we have seen they don't get color or many other things that many of  us in the printing world know, love, and, most importantly NEED from PDF files.

Google is the king of ad placement - though I don't know if Adobe's exit from "PDF Ads" is good or bad in that regard.

More PDF activity is going on relative to SilverLight, PDF editors, and other tools.

There is also the ubiquitous http://www.planetpdf.com/ - though the forums and things there have not had much for years.

My fear is that without any sort of leadership PDF will become fully bastardized over the next couple of years.  By that I mean that no one will step in to fill the leadership vacuum left by Adobe's exit from that position.

Each company with its own ax to grind will pick up the parts they care about and ignore the rest.  Companies like Google will tear PDF apart faster than the rest because of their ubiquity.

PDF - Technology to Live Without?

I found an interesting post here.  Basically from an enterprise perspective PDF is a no-no - at least on the web - and it comes in at #3 on the list.  I have seen this in the real world - many corporate types cannot receive PDFs in emails because they are blocked by the corporate fire wall.  My belief is that IT types don't like PDF for a couple of reasons. 

First, though it is relatively secure there are some clumsy problems like Javascript that make it seem like a risk.
 
There are various hack-schemes associated with PDF, javascript hacking chief among them from what I can see.  This basically involves some mechanism to run or get you to run a nefarious javascript that has either been embedded in the PDF or is somehow linked to it via web browsing.

Adobe offers fixes for the elements that involve using the PDF to display a dialog that tricks the user into running a malicious app from the PDF as documented here.   This is all linked together via the Zeus BotNet.

Second, the machinery of PDF is opaque to IT types.  This is kind of an interesting point.  I tracked down a Black Hat document on PDF threats (itself a PDF!)  Eric Filiol, the author, is the Head Scientist Officer of the Virology and Cryptology Laboratory at the French Army Signals Academy.

Basically this document outlines some of the attacks I describe above as well as covers some PDF basics.

What is of interest to me is that its relatively shallow in the nature of what it covers.  PDFs are relatively complex files and there are quite a few malicious holes in them.  But this analysis stops short of doing much more than a superficial inspection.

They do cover the various Forms actions you can associate with elements of a PDF and they also cover some about registry settings and what they can allow or not allow in terms of security.

I suspect the reasons for this are that to process the guts of a PDF you need some relatively sophisticated technology.  The paper describes the PDFStructAzer which is a tool they wrote to monkey with PDF files for hacking purposes.

I sent this guy an email offering to discuss PDF with him - but so far I have not received a response.

Third, and probably most importantly, is that the Adobe Acrobat and Flash worlds are relatively closed.  What I mean by this is that on the IT side of the world there is a lot of activity and interaction between the developers and the corporate folks.  Back and forth on the Microsoft side over formats, developer kits, and so on.  IT folks don't like closed because it makes their jobs harder to do.

Silverlight, for example, is kind of a Flash/PDF replacement for web use.  This went through a long beta period with lots of user input from developers.

Try that with an Adobe product.

From the AFP perspective there is much to learn here.  AFP is much less complex security-wise than PDF so I doubt you will have nearly the issues coming from that side of things.

Patent war update...

Follow it here.

Off topic at Lone Wolf and some interesting history...

Post at Lone Wolf. 

From a historical perspective readers might find this interesting: Argon.

New post...

At Lone Wolf today.

How do you know your PDF is correct? (cont.)

I guess over time I have come to adopt the PDF correctness model that says "if the PDF is generally correct, i.e., works in a variety of applications across versions, operating systems, and processors and it works correctly for a given application then we say the PDF is in fact correct."

There are a few PDF validating tools out there you can buy.  Some large printing houses have them to validate their internal workflow or external PDF inputs (like ads in a newspaper).  But these are of little use if the PDF that comes out doesn't work or if you don't have the money to buy one.

We have used this type of model for more than a decade and it has always worked well.

Periodically we come across new vendors or vendors updating their PDF output and we run into interesting problems.  Recently a customer came to me quite panicked and said "some of my output is missing".

This is the second cardinal sin of workflow - having an application make some output that is supposed to be on the page go away.  (The first cardinal sin is having the wrong output on the page.)

So I took a look at the PDF.  All PDFs have an "Info" section where you have a string containing the authoring application's name, a time stamp, etc.  Well, lo and behold, way down deep in the resources for the page in question there was a color space definition.  (A color space definition says things like "CS1" means to use calibrated RGB for all colors marked by the CS1 tag.  Well here was the name "CS1" and its definition, instead of legal PDF color space, was a link to the Info section of the PDF document.

Our application, in this case pdfExpress XM, iterates through the color spaces because it might want to change them.  As it does so it checks to see if each color space is one it cares about.  Part of this checking is to check whether or not it conforms the PDF standard for color spaces.  When if found this particular color space definition it generated an error.  Unfortunately several levels up in the code we made the assumption that the return value was always correct and we placed the page contents based on this.  When this error occurred we branched around the code that injected content into the page and so that part of the page was blank.

So in this particular case its easy to say "the PDF is wrong" - which it is.  But our handling of the bad PDF was also a problem and the customer was unhappy. 


The real question here is as a PDF application developer how do you anticipate arbitrarily wrong PDF input.

The portion of the PDF in question is basically a dictionary (as we discussed a few days ago) where the entry in the dictionary for the value is incorrect.   This is many levels down in a structure which is otherwise correct.  While its fairly easy to check for what should be there handling the cases where something that's not supposed to be there is not.  For example, most places in PDF including dictionary entries, can be indirect.  This means that instead of an actual value being present there is a pointer to some object.  Normally you have to locate this object and then examine it as if it were the entry.

How do you know your PDF is correct? (cont.)

While there are tools available to purchase is this regard their use does not ensure that a PDF will work in practice.   Each version of PDF has specific features, e.g., transparency, that it enables, has specific features that are replaced with alternate constructs, and so on.

So the first thing you have to figure out is which version are you trying to create.  In my experience you always want to check that the PDF is compatible with the oldest version that supports all the features you need.  You can check for correctness relative to new versions as well but this limits the usefulness of the PDF.  Of course, "A" list companies always want to force your PDFs to be the latest, most complex version - but that's not always the best for you or your customer or your application.

Once you have decided on a PDF version the simplest way to "validate" a PDF is to use a variety applications to process it and see if the results are correct.  For RIPs and viewers this basically means processing the PDF and checking the output and logs.   We tend to use a spectrum of newer and older tools, RIPs and applications for this.  The reasoning is that if it opens and works in older tools as well as newer tools the PDF is much more likely to be right.

Our tools have been in operation and continuous customer use for almost a decade at this point so we only contend with new "features" for the most part.

Backward compatibility is also important and in general we tend not to add features just because we can.  Why?  Mostly because the customers who use our products have their own set of tools which they have been using a long time and don't want to have to re-verify that any changes we have made to not negatively impact their workflow.

When you think about all of this together you start to see that there really isn't such a thing as "correct PDF" because that depends on the application and usage.  I can continually update my PDF output modules but I may break customer workflows by doing so.

How do you know your PDF is correct?

This question comes up from time to time.  As far as I know there is not programmer or application which can tell you if your PDF is in fact "correct" PDF.

Before we get into that let's look into what "correct" means in this context.  PDF is both a language and a structured file format.  Unlike PostScript, which is also a language, PDF cannot be extended.  PostScript, based on the Forth programming languages, allows you to define new language constructs and make use of them in your documents.

There have been all sorts of problems with PostScript in this regard - at least from the perspective of someone trying to make use of the PostScript content of documents.  It was easy to create convoluted, buggy ways to do things and one could never be sure if the constructs created really worked right.  A big issue with this was "page independence".  Programmers used to create PostScript files that, like programs, had to be executed sequentially, i.e., page by page, in order for the PostScript programming to work right.

PDF was meant to solve this - which it did by creating a language that was not could not be extended.  However, it also opened another can of worms. 

The structure of a PDF file is organized loosely around that of a hierarchical database.  At the top of the tree you have the "Root".  Below the root you have indexes of pages.  Page have elements like Resources.  Resources have things like fonts and images.  Most of the important parts of the file are organized around "dictionaries".

A dictionary is a structure with keys and values:

  << /EntryA 1

       /EntryB 2

  >>

In this case there are two keys (EntryA and EntryB) and two values (1 and 2).  The keys are used to retrieve the values - so I can lookup "EntryA" and get the value 1.

Using simple dictionaries doesn't present a problem.  But PDF uses dictionaries to hold other dictionaries, arrays, and other complex entities.  In addition PDF allows two dictionaries to share the same value.  So, for example, if a font is used on every page I don't have to duplicate the definition of the font; I merely store a link to a common definition in the dictionary.

So what does all this have to do with defining "correctness?"  While there are certainly many documents that define PDF a lot that has gone on over the years has left legacy issues open, i.e., things were done a certain way early on and never changed to reflect changes to PDF.  Another big issue is that things like dictionaries in PDF have defined entries, i.e., in a Resource dictionary you have a /Font entry and that's where fonts for the page are found.  But PDF in general doesn't say anything about putting other things into these structures, e.g., I can add an Elephants entry in the Resource directory and it will be ignored by most applications because they are not looking for it and don't care about it.

(Note: This is not an issue in file formats like AFP which are not based on a structured dictionary type of model.)

So what does it mean for a PDF to be correct?  The answer is I am not sure.  We always use a fairly complex test to determine "correctness".

Data Driven Color Debugging... (part 3)

The final part is at the AFP Outsider.

Tomorrow we will return to this blog for a few days.

Debugging Color...

Part two is at the AFP Outsider.

Debugging Color...

I've had some interesting experiences in this regard - particularly with those unskilled in color.

First off - what do I mean by "debugging color"?  In large complex systems color failures do not occur on a screen where a message box pops up and says "sorry your job failed".  In these system things are put together in such a way as to work 24x7.  They process standard inputs and standard outputs - all tested and carefully organized.  So a "failure" in this context occurs only when someone actually notices the problem.

What do I mean by this?

Large systems are typically built to be, at least to a certain extent, fault tolerant in the sense that at various steps along the way checks are performed to ensure what is supposed to be going on is actually going on.  Jobs that have various incorrect elements "pop out" in fail folders and humans collect them, diagnose them, and resubmit them for processing (presumably after fixing the problems).

Color problems do not manifest themselves in quite the same way as, say, a missing database entry for a mail merge job.  In this later case a logical integrity check says - "hmmm - there is no entry for this" and the job stops processing.

If the "color is wrong" what happens?  First off - there are no systems along the way to check.  Maybe that logo is now red and not blue.  The software cannot not know this because there is nothing to check it against.  Secondly, something - a human or a machine - has to look at the result of processing the entire job - usually by looking at what's in an output bin - before the problem is identified.  So, after maybe two to four hours of processing through five different servers and ten different processes we discover something is wrong.

So the night operator, whose job it is is to run the jobs nightly, notices that something doesn't look like its supposed to.  In the production world people are trained to notice things that don't look the same as everything else.  In this case the job doesn't look like it did yesterday (or, if you're lucky there's a sample to compare it too).  The logo is a different color or shade.

So at this point the operator probably does not know what is wrong - if anything.  The system that provides working job samples for comparison might be broken, i.e, the operator did not get updated production sample, or the output might be wrong, or the device producing it might be working incorrectly.

Generally an operator will be able to identify problems related to things under their control, e.g., are other jobs producing wrong colors.  So generally the problem does not escalate unless these "standard" sort of issues get ruled out.

So now the problem has "escalated" to the supervisor.  At this point generally things become more interesting.  Someone has to decide if the output is actually "wrong" or if the criteria to judge it is incorrect.  As remarkable as it seems often no one is able to make a precise diagnosis.

The reasons are very interesting.  Typically in a large shop the world of operations, job preparation, QA, graphic arts, and customer service all fall into different silos.  Each silo has its own managers, contractors, vendors, processes, and so forth.

Now the job has to "work backwards" through the system.  At each step those involved (from whatever silo is responsible) have to look at what inputs they received and compare them to the incorrect output.  In the world of color things get more interesting than in the case of, say, data value, i.e., a bad mailing address.

Each silo has its own idea of how color is handled and measured.  Some don't really know anything about it other than how it "looks".  Others tend, like CSRs, tend to think about it in terms of "approved color" or "not approved color".  Programmers tend to think about the numbers in the code that produced the values.

So as the job works backwards each silo applies its own criteria to the output and decides if its responsible for what it sees.  Generally this occurs in an ad hoc meeting on the shop floor where fingers are pointed.

(This will be continued tomorrow on the AFP Outsider blog as it applies to both PDF and AFP workflows.)

PDF Size and Performance...

This is a topic which comes up frequently (and no, this is not Viagra spam).

People say "This PDF is too large and it will take to long to RIP."   Basically most people make a direct link between the size of the PDF and the performance they are going to get RIPing the file.

In general this is wrong for several reasons.  First of all, until very recently PDF was always turned into PostScript before rasterizing on a RIP.  Now PostScript is a programming language which means that the RIP must process the language statements in order to create the raster.  All this takes time - especially when you have a lot of nested PDF forms.  So any PDF file would be effectively processed twice - once to parse the PDF to PostScript, again to process the PostScript.

There isn't a one-to-one correspondence between PDF operators and PostScript operators, particularly in terms of complexity, so seemingly simple and short PDF might not be simple PostScript as far as the rasterizer is concerned.

PDFs can be very large due to embedded images.  The most profound effect I have seen on performance (and I mostly work on non-plating devices) is extra resolution, i.e., too many bits for the job.  Shear volume is the first problem - a 2,400 dpi CMYK image takes a long time for the RIP to consume because there are a lot of bytes.  If you only need 600 dpi then don't put a 2,400 dpi image into the file.  RIPs process images quickly but can be overwhelmed by sheer volume.

So even though there are lots of images the file may still RIP quickly.

Font usage is a weakness in many RIPs - particularly using a lot of fonts.   There are many cases of PDFs having a completely new set of fonts on each page in, say, a thousand page document.  RIPs don't deal well with this and for me its been a problem over the years.  This issue compounds another common font issue - bitmap fonts.  Many programs, particularly those that convert to PDF, tend to convert source bitmap fonts to PDF bitmap fonts.  My experience is that the higher the percentage of bitmap fonts in a file the slower, in general, it will RIP as page count goes up.

Applications go to great lengths to obfuscate fonts so that font licenses can't have their intellectual property stolen.  Unfortunately you may be paying the price for this with crappy RIP performance - so you're paying twice - once for the font if you need it and again to RIP hundreds or thousands of copies to prevent you from stealing it.

The last problem area is transparency.  There are two types - masks and true PDF transparency - and both create performance issues.  (Most of what I will say here is more or less invisible to a use because "A" list applications try very hard to "hide" the fact that transparency is being used.)  Basically any situation in a PDF file where you can "see through" a transparent part to some underlying PDF element is a problem.  For transparency masks, which can occur in any PDF, the problem increases as the size of mask increases.  For true PDF transparency (controlled by the graphic state) any use is a problem for performance.

The issue is simple - a printer cannot render "transparent" ink. So, if a blue line crosses over a black line and we want to see some of the black line through the blue one the RIP has to calculate what colors the effect of the transparency would have and print a gray-blue color to present that effect.  The calculation requires the RIP to rasterize the transparent areas, create colors for them, then merge that rasterization result with the result of rasterizing everything else.

The bottom line is that transparent areas are rasterized twice - which slows things down.

So very large PDF files, as long as they avoid these and other pitfalls, will RIP very fast.  At the same time, very small files using many of these bad constructs, will RIP slowly.

Imposition Aside...

I got involved in another PDF project the other day...  This involves imposition.  However, there were a few catches.

First off, the imposition itself was fairly simply - single sheet front and back.  Unfortunately the imposition has to occur over a stream of front/back sheets, i.e., the underlying sheets change.  Secondly pages to impose are chosen not by sequence but by bookmark - and bookmarks can also have an indicator which says to leave that position blank.

So basically you have a stream of documents to impose:

  DOCID_01 DOCID_03 DOCID_99 DOCID_12

And so on.  The blanks can appear like this:


  DOCID_01 blank DOCID_03 DOCID_99 DOCID_12 blank

where "blank" is a special document ID meaning skip document pages here (pages per document is constant over the run).

Other than that we're basically dealing with cell positions for each page of the document.

I used our pdfExpressXM software as a platform for this.  It supports the multi-stream imposition (though I had to fix bug relating to relocating resources).  Internally the imposition model looks like this:

    item2={source2=+1,clip={0,0,792,612},ctm=    { 0, -1, 1, 0, 41.04,     1431 }}
    item4={source2=+3,clip={0,0,792,612},ctm=    { 0, -1, 1, 0, 693.36,    1431 }}
    item6={source2=+5,clip={0,0,792,612},ctm=    { 0, -1, 1, 0, 40.04,     1143 }}
    item8={source2=+7,clip={0,0,792,612},ctm=    { 0, -1, 1, 0, 693.364,   1143 }}
    item10={source2=+9,clip={0,0,792,612},ctm=   { 0, -1, 1, 0, 40.04,      855 }}
    item12={source2=+11,clip={0,0,792,612},ctm=  { 0, -1, 1, 0, 693.364,    855 }}
    item14={source2=+13,clip={0,0,792,612},ctm=  { 0, -1, 1, 0, 40.04,      567 }}
    item16={source2=+15,clip={0,0,792,612},ctm=  { 0, -1, 1, 0, 693.364,    567 }}
    item18={source2=+17,clip={0,0,792,612},ctm=  { 0, -1, 1, 0, 40.04,      279 }}
    item20={source2=+19,clip={0,0,792,612},ctm=  { 0, -1, 1, 0, 693.364,    279 }}

    item1={source2=+0,clip={0,0,792,612},ctm=    { 0, 1, -1, 0, 1305.36,   1157.4}}
    item3={source2=+2,clip={0,0,792,612},ctm=    { 0, 1, -1, 0, 653.04,  1157.4}}
    item5={source2=+4,clip={0,0,792,612},ctm=    { 0, 1, -1, 0, 1305.36,    869.4}}
    item7={source2=+6,clip={0,0,792,612},ctm=    { 0, 1, -1, 0, 653.04,   869.4}}
    item9={source2=+8,clip={0,0,792,612},ctm=    { 0, 1, -1, 0, 1305.36,    581.4}}
    item11={source2=+10,clip={0,0,792,612},ctm=  { 0, 1, -1, 0, 653.04,   581.4}}
    item13={source2=+12,clip={0,0,792,612},ctm=  { 0, 1, -1, 0, 1305.36,    293.4}}
    item15={source2=+14,clip={0,0,792,612},ctm=  { 0, 1, -1, 0, 653.04,   293.4}}
    item17={source2=+16,clip={0,0,792,612},ctm=  { 0, 1, -1, 0, 1305.36,      5.4}}
    item19={source2=+18,clip={0,0,792,612},ctm=  { 0, 1, -1, 0, 653.04,     5.4}}

    item21={source1=+0,clip={0,0,1440,1345.68},ctm={1,0,0,1,0,0}}
    item22={source1=+1,clip={0,0,1440,1345.68},ctm={1,0,0,1,0,0}}

There are two "sources" defined source1 and source2.  source1 is the stream of backgrounds and source2 is the stream of items to impose.  Each itemn entry defines a location where an imposed item is placed.  A clip and CTM (transformation matrix) is also supplied. 

There is also a way to specify the cycle for each input stream - cycle being the number of pages to step each time (items are offset, e.g.,  "source1=+2", from the current page in the cycle).

The interesting part of this is that the stream of imposed pages is "virtual", i.e., defined by the document id stream I described above.

The internal architecture decodes the documents ID's into a stream of pages.  The XM architecture is defined such that page number that are out of range may occur in the input page stream.  When such an occurrence is found a blank is produced.  So the document ID stream gets converted to something like:

  5 6 99999999 99999999 11 12 1 2

Where 99999999 basically causes a blank page to be produced.

Looking ahead...

So as I complete the posting over in Lone Wolf on "Industrial Color Management" I am starting to think about what to cover here.

Basically a less capable version of our color management system has been in real-world production for half a year or so.

What I am interested in now is finding others who have needs in this area - specifically PDF color transformations. 

At the end of the day color is just another data driven variable.

But I want to be clear - we are not parameterizing the creation of PDF with color - like a data driven Illustrator or something - we are parameterizing the alteration of existing PDF (or TIFF or JPG or AFP).

Anyway I hope to complete the Lone Wolf discussion within a week or two.

New Color Management Technology to be Released...

I think this will be of great interest to those following this thread...

follow it here...

The core technology will be discussed on Lone Wolf as it is not PDF specific.

Processing Color...

Given all this complex machinery for manipulating a PDF - what do we do in the second pass that interesting?

For one thing we can add PDF resources, like new color spaces, that were not previously part of the PDF file. So, for example, if I wanted to change /DeviceRGB to a color space with an ICC profile I would have to add the new color space and profile into the Resource dictionary.

Another thing that happens is that resources are renamed to be compatible as described in the previous post. Because we rename them uniquely we can easily associate subsequent processing (described below) with the correct resource.

A big part of the second pass is something called "scan". Scan proceeds to process each and every PDF operator on a PDF page. It starts at the first object and proceeds to the end. At a minimum it must rename references to resources to use the new portable names. We call this "replace". In general we have machinery that can arbitrarily match some PDF operator and parameters and replace them other other operators or parameters (or remove them entirely).

On top of this machinery we added color processing. Color processing involves replacing one color with another (more on this in future posts). So as we process the PDF operators we may encounter a simple color operator, e.g., "0 0 0 rg". Again, this is basically (at least on the simplest level) a "scan" and "replace" operation; but in the case of color an elaborate color engine has been added to handle defining matches and replacements. The color machinery is able to look up the notion "0 0 0 and rg" and come up with a replacement. A simple replacement might be "0 0 0 1 k".

In the case of resources there may be images involved with also have associated color spaces. Again the same sort of processing occurs - but this occurs at the time of resource processing because the color handling for images is usually part of the image. Images typically involve RGB to CMYK translations for most of our customers.

So our solution for this customer is based on a product we call pdfExpress XM. This product is like our basic pdfExpress product but its designed to handle these large files. We created it because the customer called up a few years ago with a problem related to an inkjet printer they had. Seems the printer had performance problems RIPing files but the problems went away when the PDFs went through out application.

Only problem was that the files were larger page-count wise then we typically can handle and our application ran out of memory. Since this was a "rush" I had to come up what a scheme to make very large files work.

The idea I came up with is a PDF compiler that prepares the PDF files for a second pass in such a way as to ensure that very little memory is needed. Our original software was designed with the basic strategy of consuming the entire PDF file before it starts to process anything. While this was a reasonable strategy in 2001 its no longer valid.

Basically when we inspect the entire PDF file we have to compute the set of resources used by each page so that if the page is used in an output file we know what resources to include. We also have to process the contents of the file to see what resources are used by the page. So, for example, virtually every PDF file has a Font resource call "F1". So if I wanted to place two PDF pages from two different files (or even from the same file because there is no guarantee that the font "F1" on page one is the same font called "F1" on the second page) on the same page as part of an imposition there are only two ways to do it. The first is to use a PDF form object - which is kind of like a PDF subroutine. It "hides" the resource issue by creating a new scope for the F1 on each page.

While doing the form thing in PDF is fairly easy it doesn't require you to have full control over the page content in the sense that you can know every resource used, i.e., you can embed the page in a form without having to deal with the content. Since they path I can from was PDF VDP I had created (and patented US 6,547,831) involved "editing" PDF content to insert variable data I declined to follow the form route.

So I inspect all the named resources like "F1" and give them a guaranteed unique name relative to what I am doing with them. Long story short all this uses a lot of memory - especially editing PDF content streams - so I created a means to "compile" most of this processing in one pass and have a second pass that uses the predigested resource data to do the real work.

The "compiled" PDF is still a PDF. We just add metadata to the Cos object tree for our second pass. If you're not our second pass no problem - it just looks like a PDF. If we find our metadata we use it instead - saving tons of memory and allowing us to handle really large files.