While the following doesn't conclusively prove changing the motor would work, I think the answer is yes! And I achieved the desired result via a different method, so I consider the problem solved.
Instead of changing the motor, I designed a 1:4 step-up planetary gearbox to put on top of the extruder motor, yielding a 10:1 net reduction instead of the original 40:1. This works, even without much attention to part strength or backlash in the gearbox (thanks to the subsequent 40:1 reduction), yielding the desired sub-50ms retractions with PLA. This seems to establish that the rotational rate or acceleration of the flex shaft is not a limiting factor.
One thing I noticed after adding my gearbox is that, even after scaling up the extruder speed and acceleration to 4x the values I was using without the gearbox, the stepper motor did not make the sharp "chirp"/"squeal" sound on retractions that I got before. This seemed odd, because it should be the same speed/acceleration for the output shaft and slightly higher load than before. So I think there may be something other than just exceeding the stepper's capabilities going on with the original configuration.
The manufacturer's recommended (provided STL files) coupling of the motor to the flex shaft is this long rigid coupler and guide cap:
That looks like a suspiciously long lever on the motor output shaft. And while the cap retains the flex cable sheath fairly rigidly, it can't be perfectly aligned with the motor shaft, and there's room for the flex shaft to move slightly side to side inside the sheath. So, my guess is that spinning the flex shaft at high speeds with it directly coupled to the motor like this put some serious lateral forces on the motor output shaft, interfering with the motor's performance and likely leading to long-term damage if not corrected.
The gearbox I added avoids extending the motor shaft at all (the planetary gear carrier slides fully over the D-shaft) and the small amount of backlash in the gears themselves allows the flex shaft positioning to be imprecise with (apparently) no serious ill effects. And if it does cause unwanted wear, well these are printed parts that are easily replaceable.
As for the 40:1 reduction in the extruder itself, after working with all of this to get it printing well, I don't think it's necessarily a wrong design. The flex shaft can handle the speeds needed just fine, and the 40:1 reduction keeps torque from the extruder gear off of it while extruding. The Zesty Nimble (which some call a clone of this design) used 30:1 instead of 40:1, and has dropped to 20:1 in their latest iteration, probably for reasons related to my troubles with the 40:1. But I think the gearbox at the motor side is actually a rather nice solution that preserves all of the potentially good properties of the high reduction and none of the bad ones. It does make the setup more complicated to manufacture and install though.