We've discussed this in scattered bits (use search terms VFET* and V-FET*), but let me see if I can summarize, and correct me if I veer:
Strike 1: The amps were very expensive for their power output. Remember, this was the era of the receiver horsepower race. The devices themselves were expensive to make. The reject rate at the semiconductor fab plants must've been rather high, and the VFET line wasn't suitable for making anything else, so VFETs would have to go over big or they wouldn't make it at all.
Strike 2: With the associated gear of the era, especially the speakers, I doubt that anyone could hear enough of a difference to make the expense worthwhile.
Strikes 3 and 4: The VFETs themselves weren't fragile, but they required a special bias supply that had to be absolutely reliable or else the VFET effectively became a short circuit across the main power supply. Sony chose some diodes for this bias supply that for reasons known only to themselves went cocoapuffs after a year or three, and there was no backup. So many of us had our wonderamps go foom-- but that generally happened after the VFETs had already exited the market. Yamaha's amps didn't seem to have this trouble, but they were "statement" amps, and that sort of thing always seems to have a limited life. Halos tarnish and must ever be replaced by something newer, faster and shinier.
It must be said that VFETs have undeniable advantages, and the amps that survive do have a unique sound, at least in my limited experience. EW can tell us if there are any devices/circuits out there now that have some of that sound, but the audio-only market isn't the monster it was back in 1975, and it's not likely that anyone will mass-market an amp that uses unique devices if the mass market isn't so massive.