after digging around all day, I am willing to bet the the problem is with the output transistors, as the headphone output is fine, and it is a resistive load from the outputs.
Also, the output signal doesn't clip without speakers installed so its almost like the outputs are impedance...