The TOC points to sensible L/R pair alignment points, rather than random bit positions (it might even be to a CIRC frame boundary). A CD isn't simply a stream of bits; there is a complex CIRC frame structure as well. After all, the whole purpose of the recorded bitstream is to deliver error-corrected 16-bit sample pairs to a DAC.
When ripping, most software starts with simple, 16-bit PCM WAV files, as they're the most appropriate format to store the extracted track data.