OK, my head is still spinning from all the CoreAudio reading and VLC/Portaudio source browsing I’ve been doing. I’ve made a few changes that seem to help out the passthrough situation for people with the combination analog/digital ports and AC3 audio streams. Check it out here.
Note: DTS is definitely broken, I need to look at this more.