You should definitely read through it, but here’s the gist for movies that contain AC3 audio files:
Install a copy of the free Perian QuickTime plug-in on your Mac. String an optical cable between your Mac and a compatible receiver. Launch Audio Midi Setup (found in the Utilities folder) and in the audio output area select Digital Out from the Source pop-up menu and 48000.0Hz and 2ch–16-bit in the Format area.
Launch the Terminal app and enter 

defaults write com.cod3r.a52codec attemptPassthrough 1
 Press Return.

If iTunes is running, quit and relaunch it.
Locate the movie you want to view, and play it with iTunes’ Volume slider set to its highest level. Also be sure that the equalizer is off. Place your cursor over the movie so that the on-screen controls appear, click on the small bubble icon, and choose the surround sound track. With a little luck, iTunes should now play the movie with surround-sound audio.

source: