Very strange…there appears to be something fundamentally wrong or different with the X3100, as best I can tell from my extremely limited experience with such things. Which probably means I’m completely wrong.
D4rk helped me get set up compiling my own fragment programs (thank you!). The approach I took is what I always do, which is get the working code on one end, the non-working code on the other, and then move towards the middle until you find out what breaks.
I took out all the matrix multiplication, and just started with some basic transformations to see what worked. While rgb.r = 1.164 * yuv.r; produced the expected pixel color rgb.r = 1.164 * (yuv.r – 16.0/256.0) + 1.596 * (yuv.b – 128.0/256.0); doesn’t (at least according to my Numbers spreadsheet). So tomorrow I’ll do some more back and forth to see if I can figure out why the second formula isn’t working. (Note that it DOES work on an NVidia card). Doing debugging where the only way to output debugging information is through 0-1 values in pixel color components has given me new respect for people who work in that field.