Post at Lone Wolf.
From a historical perspective readers might find this interesting: Argon.
Monday, August 30, 2010
Thursday, August 26, 2010
Thursday, August 19, 2010
How do you know your PDF is correct? (cont.)
I guess over time I have come to adopt the PDF correctness model that says "if the PDF is generally correct, i.e., works in a variety of applications across versions, operating systems, and processors and it works correctly for a given application then we say the PDF is in fact correct."
There are a few PDF validating tools out there you can buy. Some large printing houses have them to validate their internal workflow or external PDF inputs (like ads in a newspaper). But these are of little use if the PDF that comes out doesn't work or if you don't have the money to buy one.
We have used this type of model for more than a decade and it has always worked well.
Periodically we come across new vendors or vendors updating their PDF output and we run into interesting problems. Recently a customer came to me quite panicked and said "some of my output is missing".
This is the second cardinal sin of workflow - having an application make some output that is supposed to be on the page go away. (The first cardinal sin is having the wrong output on the page.)
So I took a look at the PDF. All PDFs have an "Info" section where you have a string containing the authoring application's name, a time stamp, etc. Well, lo and behold, way down deep in the resources for the page in question there was a color space definition. (A color space definition says things like "CS1" means to use calibrated RGB for all colors marked by the CS1 tag. Well here was the name "CS1" and its definition, instead of legal PDF color space, was a link to the Info section of the PDF document.
Our application, in this case pdfExpress XM, iterates through the color spaces because it might want to change them. As it does so it checks to see if each color space is one it cares about. Part of this checking is to check whether or not it conforms the PDF standard for color spaces. When if found this particular color space definition it generated an error. Unfortunately several levels up in the code we made the assumption that the return value was always correct and we placed the page contents based on this. When this error occurred we branched around the code that injected content into the page and so that part of the page was blank.
So in this particular case its easy to say "the PDF is wrong" - which it is. But our handling of the bad PDF was also a problem and the customer was unhappy.
The real question here is as a PDF application developer how do you anticipate arbitrarily wrong PDF input.
The portion of the PDF in question is basically a dictionary (as we discussed a few days ago) where the entry in the dictionary for the value is incorrect. This is many levels down in a structure which is otherwise correct. While its fairly easy to check for what should be there handling the cases where something that's not supposed to be there is not. For example, most places in PDF including dictionary entries, can be indirect. This means that instead of an actual value being present there is a pointer to some object. Normally you have to locate this object and then examine it as if it were the entry.
There are a few PDF validating tools out there you can buy. Some large printing houses have them to validate their internal workflow or external PDF inputs (like ads in a newspaper). But these are of little use if the PDF that comes out doesn't work or if you don't have the money to buy one.
We have used this type of model for more than a decade and it has always worked well.
Periodically we come across new vendors or vendors updating their PDF output and we run into interesting problems. Recently a customer came to me quite panicked and said "some of my output is missing".
This is the second cardinal sin of workflow - having an application make some output that is supposed to be on the page go away. (The first cardinal sin is having the wrong output on the page.)
So I took a look at the PDF. All PDFs have an "Info" section where you have a string containing the authoring application's name, a time stamp, etc. Well, lo and behold, way down deep in the resources for the page in question there was a color space definition. (A color space definition says things like "CS1" means to use calibrated RGB for all colors marked by the CS1 tag. Well here was the name "CS1" and its definition, instead of legal PDF color space, was a link to the Info section of the PDF document.
Our application, in this case pdfExpress XM, iterates through the color spaces because it might want to change them. As it does so it checks to see if each color space is one it cares about. Part of this checking is to check whether or not it conforms the PDF standard for color spaces. When if found this particular color space definition it generated an error. Unfortunately several levels up in the code we made the assumption that the return value was always correct and we placed the page contents based on this. When this error occurred we branched around the code that injected content into the page and so that part of the page was blank.
So in this particular case its easy to say "the PDF is wrong" - which it is. But our handling of the bad PDF was also a problem and the customer was unhappy.
The real question here is as a PDF application developer how do you anticipate arbitrarily wrong PDF input.
The portion of the PDF in question is basically a dictionary (as we discussed a few days ago) where the entry in the dictionary for the value is incorrect. This is many levels down in a structure which is otherwise correct. While its fairly easy to check for what should be there handling the cases where something that's not supposed to be there is not. For example, most places in PDF including dictionary entries, can be indirect. This means that instead of an actual value being present there is a pointer to some object. Normally you have to locate this object and then examine it as if it were the entry.
Wednesday, August 18, 2010
How do you know your PDF is correct? (cont.)
While there are tools available to purchase is this regard their use does not ensure that a PDF will work in practice. Each version of PDF has specific features, e.g., transparency, that it enables, has specific features that are replaced with alternate constructs, and so on.
So the first thing you have to figure out is which version are you trying to create. In my experience you always want to check that the PDF is compatible with the oldest version that supports all the features you need. You can check for correctness relative to new versions as well but this limits the usefulness of the PDF. Of course, "A" list companies always want to force your PDFs to be the latest, most complex version - but that's not always the best for you or your customer or your application.
Once you have decided on a PDF version the simplest way to "validate" a PDF is to use a variety applications to process it and see if the results are correct. For RIPs and viewers this basically means processing the PDF and checking the output and logs. We tend to use a spectrum of newer and older tools, RIPs and applications for this. The reasoning is that if it opens and works in older tools as well as newer tools the PDF is much more likely to be right.
Our tools have been in operation and continuous customer use for almost a decade at this point so we only contend with new "features" for the most part.
Backward compatibility is also important and in general we tend not to add features just because we can. Why? Mostly because the customers who use our products have their own set of tools which they have been using a long time and don't want to have to re-verify that any changes we have made to not negatively impact their workflow.
When you think about all of this together you start to see that there really isn't such a thing as "correct PDF" because that depends on the application and usage. I can continually update my PDF output modules but I may break customer workflows by doing so.
So the first thing you have to figure out is which version are you trying to create. In my experience you always want to check that the PDF is compatible with the oldest version that supports all the features you need. You can check for correctness relative to new versions as well but this limits the usefulness of the PDF. Of course, "A" list companies always want to force your PDFs to be the latest, most complex version - but that's not always the best for you or your customer or your application.
Once you have decided on a PDF version the simplest way to "validate" a PDF is to use a variety applications to process it and see if the results are correct. For RIPs and viewers this basically means processing the PDF and checking the output and logs. We tend to use a spectrum of newer and older tools, RIPs and applications for this. The reasoning is that if it opens and works in older tools as well as newer tools the PDF is much more likely to be right.
Our tools have been in operation and continuous customer use for almost a decade at this point so we only contend with new "features" for the most part.
Backward compatibility is also important and in general we tend not to add features just because we can. Why? Mostly because the customers who use our products have their own set of tools which they have been using a long time and don't want to have to re-verify that any changes we have made to not negatively impact their workflow.
When you think about all of this together you start to see that there really isn't such a thing as "correct PDF" because that depends on the application and usage. I can continually update my PDF output modules but I may break customer workflows by doing so.
Tuesday, August 17, 2010
How do you know your PDF is correct?
This question comes up from time to time. As far as I know there is not programmer or application which can tell you if your PDF is in fact "correct" PDF.
Before we get into that let's look into what "correct" means in this context. PDF is both a language and a structured file format. Unlike PostScript, which is also a language, PDF cannot be extended. PostScript, based on the Forth programming languages, allows you to define new language constructs and make use of them in your documents.
There have been all sorts of problems with PostScript in this regard - at least from the perspective of someone trying to make use of the PostScript content of documents. It was easy to create convoluted, buggy ways to do things and one could never be sure if the constructs created really worked right. A big issue with this was "page independence". Programmers used to create PostScript files that, like programs, had to be executed sequentially, i.e., page by page, in order for the PostScript programming to work right.
PDF was meant to solve this - which it did by creating a language that was not could not be extended. However, it also opened another can of worms.
The structure of a PDF file is organized loosely around that of a hierarchical database. At the top of the tree you have the "Root". Below the root you have indexes of pages. Page have elements like Resources. Resources have things like fonts and images. Most of the important parts of the file are organized around "dictionaries".
A dictionary is a structure with keys and values:
In this case there are two keys (EntryA and EntryB) and two values (1 and 2). The keys are used to retrieve the values - so I can lookup "EntryA" and get the value 1.
Using simple dictionaries doesn't present a problem. But PDF uses dictionaries to hold other dictionaries, arrays, and other complex entities. In addition PDF allows two dictionaries to share the same value. So, for example, if a font is used on every page I don't have to duplicate the definition of the font; I merely store a link to a common definition in the dictionary.
So what does all this have to do with defining "correctness?" While there are certainly many documents that define PDF a lot that has gone on over the years has left legacy issues open, i.e., things were done a certain way early on and never changed to reflect changes to PDF. Another big issue is that things like dictionaries in PDF have defined entries, i.e., in a Resource dictionary you have a /Font entry and that's where fonts for the page are found. But PDF in general doesn't say anything about putting other things into these structures, e.g., I can add an Elephants entry in the Resource directory and it will be ignored by most applications because they are not looking for it and don't care about it.
(Note: This is not an issue in file formats like AFP which are not based on a structured dictionary type of model.)
So what does it mean for a PDF to be correct? The answer is I am not sure. We always use a fairly complex test to determine "correctness".
Before we get into that let's look into what "correct" means in this context. PDF is both a language and a structured file format. Unlike PostScript, which is also a language, PDF cannot be extended. PostScript, based on the Forth programming languages, allows you to define new language constructs and make use of them in your documents.
There have been all sorts of problems with PostScript in this regard - at least from the perspective of someone trying to make use of the PostScript content of documents. It was easy to create convoluted, buggy ways to do things and one could never be sure if the constructs created really worked right. A big issue with this was "page independence". Programmers used to create PostScript files that, like programs, had to be executed sequentially, i.e., page by page, in order for the PostScript programming to work right.
PDF was meant to solve this - which it did by creating a language that was not could not be extended. However, it also opened another can of worms.
The structure of a PDF file is organized loosely around that of a hierarchical database. At the top of the tree you have the "Root". Below the root you have indexes of pages. Page have elements like Resources. Resources have things like fonts and images. Most of the important parts of the file are organized around "dictionaries".
A dictionary is a structure with keys and values:
<< /EntryA 1
/EntryB 2
>>
In this case there are two keys (EntryA and EntryB) and two values (1 and 2). The keys are used to retrieve the values - so I can lookup "EntryA" and get the value 1.
Using simple dictionaries doesn't present a problem. But PDF uses dictionaries to hold other dictionaries, arrays, and other complex entities. In addition PDF allows two dictionaries to share the same value. So, for example, if a font is used on every page I don't have to duplicate the definition of the font; I merely store a link to a common definition in the dictionary.
So what does all this have to do with defining "correctness?" While there are certainly many documents that define PDF a lot that has gone on over the years has left legacy issues open, i.e., things were done a certain way early on and never changed to reflect changes to PDF. Another big issue is that things like dictionaries in PDF have defined entries, i.e., in a Resource dictionary you have a /Font entry and that's where fonts for the page are found. But PDF in general doesn't say anything about putting other things into these structures, e.g., I can add an Elephants entry in the Resource directory and it will be ignored by most applications because they are not looking for it and don't care about it.
(Note: This is not an issue in file formats like AFP which are not based on a structured dictionary type of model.)
So what does it mean for a PDF to be correct? The answer is I am not sure. We always use a fairly complex test to determine "correctness".
Monday, August 16, 2010
Friday, August 13, 2010
Subscribe to:
Posts (Atom)