For PCM encoded (uncompressed) audio data, there are a few different cabling standards you'll see mentioned:
  • optical / TOSlink
  • coax / S/PDIF
These actually send the same information (stereo, uncompressed) through different kinds of cables -- optical, coaxial, or XLR/balanced. Assuming the cables are of good quality and not too long, there will be no quality difference between the different options, but you need to have the same input on the receiving side as the output on the sending side.
Now, through this same digital cables, can be sent something which is NOT linear, PCM, "uncompressed" audio. Most common is the Dolby Digital 5.1 format, which compresses the bits of the sound to use the same bandwidth as uncompressed audio, but fits six channels in the same bandwidth (Front, Center, Right, RearLeft, RearRight and LFE). Another format that can be sent through the same digital cables is DTS.
So, in the simplest of worlds, the digital formats that come in through these cables is either PCM (which is uncompressed, stereo), from a source such as a CD player, or DD (compressed, 5.1) from a DVD.
In either case, the receiver needs to decode the signal. In the PCM case, the receiver forwards the bits from the wire, to a stereo DAC (digital-to-analog converter), which puts out an actual audio signal on your pre-outs and/or your speaker outs. In the DD case, the receiver first has to do some computation on the compressed bits to de-compress them, and then send the six separate channels to a six-channel DAC for conversion to surround outs. There is typically some other processing going on before converting to analog, such as volume control, tone controls, etc.
Now, HDMI comes in and confuses things a bit. HDMI can support more than two uncompressed channels in the cable. Thus, a HD-DVD or BR player can decode a 7.1 signal, into separate PCM streams, and send across the HDMI cable. The receiver then forwards those bits to 8 separate DACs. However, HDMI can also forward a variety of compressed digital formats, such as DD-Ex, DTS-ES, etc. Again, the receiver must do more processing on those compressed signals to turn them into decoded (but still digital signals), and then send the decoded signals to the DACs.
Different receivers have different amounts of decoding capability. A basic home theater receiver may do only PCM (uncompressed) and DD (compressed dolby digital). A fancy HDMI 1.3 receiver may do something close to 20 different formats. In general, the media that's played makes sure to include at least one of the two base formats (DD 5.1 or PCM stereo) so that everyone can at least hear something, even if they don't have the fanciest receiver.
If your playback device has settings for what to output, set it to output a compressed format that your receiver knows how to decode, if you want multi-channel audio over TOSlink or S/PDIF, else you'll only get stereo PCM. If you're using HDMI, the format is usually smart enough to figure out what the best option is to send data between the player and the receiver, so that you get the highest quality with the most channels that your system can deliver.