Colour Palettes Not Displaying Colours

[kalm]

Richard,

I have created one or two colour palettes, and tried to use them in our CAD system, which uses standard ACO palettes. None of the custom palettes that I have created show the colours in the CAD palette. When selected, they always insert a BLACK colour no matter what is selected. The CAD (SOFTPLAN) technical support have no idea what is causing this. Do you have any idea. You know the Crayola colours work since you've seen the palette. This is how the CAD system shows it ...

If I can help you to trace this issue in any way, let me know.

Keith

[Richard Moss]

Hmm, that is bizarre.

I tested a few of the palettes generated using the Palette Editor in Photoshop CS2 (the latest version that I have access too  ), and they loaded fine, so at first glance it looks like I'm interpreting the specification correctly.

I've generated two version of the Crayola sample palette, one using the version 1 ACO format, and the other version 2. Would you mind trying to load these palettes and see if either works?

Thanks;
Richard Moss

[kalm]

Sorry Richard, you're still out of luck. I had hoped that one would show and the other not.

Here's how both V1 and V2 look. Also screenshot showing Resene Total Colour System Master - to be fair I'm not sure whether the System Master was installed by me or the CAD system, so I went to Resene's site and downloaded their Resene Metallics file and installed that. This also displays fine in the CAD system.

Now just for your information, I also use the created ACO files in XARA PHOTO AND GRAPHIC DESIGNER 10, and they display and work fine in there. It's only the CAD system that shows only black.

Keith

[Richard Moss]

Interesting. I'm not entirely convinced the Palette Editor is at fault if the palettes are working fine in multiple applications, but I'm happy to dig further.

I don't suppose you have a direct link for the ACO file you reference from Resene? I'd like to to a base palette that you say works fine, then generate my own version so I can binary compare the two and check for errors. I had a poke around their site but the only thing I could find was Adobe Color Books, which I don't yet support. (That's the next thing on my list after I wrap up the Adobe Stack Exchange format support I was recently working on).

Edit: Never mind, I asked Google and the answer was http://www.resene.co.nz/comn/services/Photoshop_colour_files.htm. I'll try to get some time tomorrow to set up a compare and test this.

Edit2: A direct binary compare shows a difference, so there is indeed something up. Again though Photoshop still loads the file very happily so it's quite odd. Time to do some visual compares.

[Richard Moss]

The differences in the bytes are most likely rounding errors. Palette Editor is written in C#, using the .NET framework. In this framework, a colour is presented by 4 values of between 0 and 255 representing the alpha, red, green and blue channels.

In an ACO file, the RGB channels are presented by values between 0 and 65535. So the number I read from the file is then divided by 255 to get my 0-255 range. As it is using integer division, there is some loss of precision.

For example, one of the values in the Resene data is 35723

I divide that by 256 to get 139

However, if force it to divide by floating point division rather than integer, I get 139.54296875.

If I multiply 139 by 256 I get 35584, which doesn't match the original value. If I multiply 139.54296875 by 256 then I get 35723 which of course does.

So, there is some slight shifting of colours which is a definite bug. I actually came across this a couple of weeks ago when working with Adobe's Swatch Exchange Format, which saves floating point numbers by default. I haven't built ASE support into production code yet, but in order to resolve the problem I was going to store custom structures with the original colour data in it, so I could be damn sure there was no loss of precision (this has been a long term goal as currently it's impossible to store extra information such as colour spaces, even swatch names have to be hacked in).

Looks like I need to do that work sooner than I planned, which is fair enough.

The problem however, is the colour data is a match (aside from the slight shifting). You can see in the screenshot the tool I wrote to help me diagnose the issue, by comparing files down the byte level with highlighting showing the different sections (blue is version, orange is colour count, red is colour space and salmon is the RGB data (plus an extra block of data not used), yellow is the length of the swatch name, and green is the swatch name data).

So while the colours are shifted slightly depending on how the file is de-serialized, the files are perfectly valid. There is no "junk" data (this appears as light grey in the tool).

Due the rounding issue, the second byte in the colour data is always zero. I would almost suspect there is a bug in the CAD software itself that is reading the file wrong by not processing the first byte of the pair is the second is zero. If you own the software and/or have a support contract, perhaps you could pass this information on, it would be interesting to know for definite.

If this is the case, then at the very least when I fix my rounding error that will work around the issue, but I would really like to know *why* this is breaking. Perhaps it's actually an invalid combination and Photoshop itself is working around *my* error. Working with integers and floats at an actual byte level is something I'm still feeling my way around as I just haven't needed to even think about it or endian-ness prior to working with Adobe's file formats and I should have noticed the rounding issue long ago  . Still, you learn from your mistakes!

Edit: Corrected examples above, so busy thinking of bytes I used 255, when the code actually uses 256.

[kalm]

Interesting. I'm not entirely convinced the Palette Editor is at fault if the palettes are working fine in multiple applications ...

Although the converse could be true ... the CAD software views multiple ACO palettes correctly, except mine. So the fault may lie in the palette editor. Just suggesting!

[Richard Moss]

Although the converse could be true ... the CAD software views multiple ACO palettes correctly, except mine. So the fault may lie in the palette editor. Just suggesting!

Well yes, that big wall of text right below the text you quoted kind of outlined that

[kalm]

To be honest Richard, I really didn't understand much of that big wall of text ... I'll take your word for it!

I understood that you had a rounding error, that may have caused a colour shift, but I imagined that would just cause it to show with a colour shift. After that, I got seriously lost.

[Richard Moss]

Yes, your summing up of the rounding error is spot on. But it shouldn't affect the CAD program.

To try and simplify it, each RGB color in an ACO file is represented by 3 pairs of values and each of those 6 values is between 0 and 255. My code is writing the second value of each pair as zero, due the rounding issue. However, this is probably a red herring and the real issue (as far as the CAD software is concerned), that value should never be zero. There's a fairly clear pattern in the original source data of pairs of values that are the same - so once I've done some more testing I may have a solution, although I still need to find out why or I'll have learned nothing.

[Richard Moss]

I have a new set of colour format libraries for dealing with colour values of differing ranges, thus avoiding the rounding error and which seems to fully resolve the problem. However, it needs more work before the Palette Editor can use it, and also opens a new can of worms for me to think about in regards to PE's UI, which is geared towards 0-255 ranges for colours, meaning it would probably be difficult to represent the full gamut of colours that ACO supports with its 0-65535 range. I'm still a little wary of the new code as the problem just vanished (unexpectedly) so I will be writing quite a few more tests.

So, it won't be in the upcoming update which has its own set of bug fixes, but it should be in the one after that.

[kalm]

If you want me to test a palette (or even a single colour) in SOFTPLAN, just send one over and I'll test it for you. If I can assist in any way, just let me know.

I know you're not doing this just for me, but it feel like you are. I really appreciate everything you are doing.

Thank you.

Keith

[Richard Moss]

Better to know about these bugs and get them fixed, then it's a more useful program for everyone.

[kalm]

Richard,

A new wrinkle. I just downloaded an ACO palette creation tool called Just Color Picker from http://annystudio.com/software/colorpicker/, and created a colour palette from it. This also shows FULL black in my CAD software. Now that would imply that either BOTH your and their palette software has issues - UNLIKELY but possible, or the CAD system is not reading something correctly - MORE LIKELY. However, this still doesn't explain why it seems to read professional palettes correctly. Will forward this information to the CAD company for their information, and see what they have to say.

Comments?

[Richard Moss]

It's a bug with our Palette Editor software. I know what the bug is, I have a fix for it, I just don't know why it's a problem - I'd rather know before I start pushing random changes out, even if they seem correct. Unfortunately, it's a technical issue.

Each RGBA channel in an ACO color ranges from 0-65535. This requires two bytes to store. My palette editor currently uses the more standard 0-255 range, which requires a single byte to store.

The bug is caused by the fact that converting 0-255 to 0-65535 is done by simply multiplying by 256. When I break that value down into the two bytes to save, one of them is always zero. That is one key factor. The other key factor is that the ACO format requires the values to be stored in big-endian format. Windows itself uses little-endian, so I need to swap bytes around.

Now for the bit I don't understand.

If I save that pair of bytes, one non-zero and one zero, programs such as Artweaver (and your CAD software one would assume) seem to treat both bytes as zero. Which boils down to - black. Programs like Photoshop seem to be slightly more intelligent and don't discard the non-zero byte.

The latest version of the Palette Editor internally stores colour data correctly for each format, but the UI itself still works in the 0-255 range. So as soon as you modify a colour, the bug pops up again. I have solved that by simply duplicating the byte when one is non-zero and one is zero.

So if that fixes the problem, then why haven't I release an update to the Palette Editor? Simply because I don't know why these programs are treating both bytes as zero if one is. Is it something to do with how BE architecture works? Am I misunderstanding something fundamental when converting integers into their component bytes? I've done a fair bit of searching over the last few months and I don't have an answer. And so the update is just sitting on a CI server waiting to go.

You can see the bug in that colourful screenshot I posted in an early reply - if you look at the pairs of columns in the right hand set, the right hand column is always zero. If you look at the exact same data in the left hand set, you can see several identical pairs, mirroring how I correct the converted data.

I have tested certain images that use BE for storing their data, and I observed the same behaviour in those, which lead me to believe my solution really is the correct one. And yet I still don't know for sure. It's probably time I posted on stackoverflow and actually asked!

I surmise that this other software you're using is doing exactly the same thing as me, ie storing the channels in a single byte then multiplying them up without pairing them.

[Richard Moss]

I recently started having our CI server automatically upload the last successful build each day to cyotek.com - there is a nightly build of the palette editor available which incorporates a fix for this issue (amongst others).