Ok. What is the last picture? (both ~14/15mVrms?)
Yes, tube strength is important. The high voltages look great, so I think the 5Y3 is working fine. When it comes to a push-pull set of outputs, we generally look at the plate voltages, and they are fairly balanced (325 and 329) but given the only series resistance is the OPT primary that may not tell the clear story. The combined bias currents of 78mA for the two tubes is pretty healthy, so it does not seem there is a problem with either one of those, but swapping the two output tubes and seeing no change will help you gain confidence there. If you want to gauge how well the tubes are actually matched, you can put a 10 ohm or lower resistor in series between each cathode and the shared 270 ohm cathode resistor to measure how much actual cathode current is being contributed from each. The 6AN8A also seems to be performing about as expected, so to a first order I don't suspect a tube issue, but a biasing issue in either the pentode, triode, or both. It will be interesting if you swap in the NOS 6AN8 and see if there is a difference here.
Pin 3 (triode cathode = 1.01V) is about what is expected (~1.1).
Pin 9 (pentode cathode = .8V) is a bit low, but works OK for now.
AV = 2.02/.0139 = 145.3 X (pretty high!)
AV(dB) = 20*LOG(145.3) = 43dB
That is a HUGE amount of gain- very sensitive. Once you have open loop gain measured in dB, you can then make a decision to trade off how much sensitivity you want (closed loop gain) versus amount of negative feedback. The final closed loop gain in dB is the open loop gain minus the amount of negative feedback. Say you wanted an amplifier that puts out maximum power with 1V RMS input. A 6V6 push-pull is going to be about a 10W maximum output into rated load. Maybe a bit more here because of your higher voltages, but let's say 10W. I assume you are using an 8 ohm load. 10W into 8 ohms is ~8.9V RMS. 8.9V with 1V input is a gain of 8.9X, but there is some squashing that happens as you get close to clipping, so it needs to be a bit higher, so let's say it's 10X. That's 20dB of closed loop gain. With 43dB of open loop gain to start with, you could apply theoretically 23dB of global negative feedback. That's really, way too high. The amplifier will likely be unstable if you try to use that much. Generally, 20dB of negative feedback is considered a healthy, maximum amount if you have really good transformers. But usually, with cheaper transformers you can't keep the amp stable with more than about 11 to 18dB of feedback. Rather than try to see if we can use that much feedback, I think it's wise to drop the open loop gain back a bit, to maybe 38dB (80X) or less.
Since you seem pretty comfortable with the scope, it would be interesting to use 10X setting on the probes, and see what the grid drive signals look like at the input (plate) side of each coupling cap as you approach clipping. That will tell us how symmetrical they are and which one is crapping out at higher amplitude. Given the previously identified super low triode plate voltage I bet that is the problem- too little swing room. Given the super high gain, we should be able to drop the two 470K plate resistors down some (I'd try 330K, 220K, etc) and see how the voltages, max power waveform, and open loop gain changes. If the lower grid signal is pretty small compared to the upper grid signal, we can up the gain on the lower inverter triode as well- you can actually just jumper between the pin 3 cathode of the 6AN8 to the positive side of the 100uF bypass cap that is right there. That is basically what was done on the Magnavox 9300, and should work great.