Damping factor is only influenced by speaker impedance - nothing to do with sensitivity. Sensitivity is simply how much power a speaker requires to produce a certain level of loudness.
Sensitivity is is a tradeoff based on design and like most things is based on physics and therefore there’s no way around it.
Some general rules of thumb:
A lower sensitivity speaker driver is generally caused by two things: a stiffer suspension and a heavy cone material. Stiffer suspension is required for smaller box volumes. If you look back 40 years, speakers were a lot more sensitive. They also had cabinets the size of refrigerators. Cone weight is all about another set of compromises. A heavier cone will reduce the resonant frequency. This is good for reproducing low frequencies. A cone can also be made heavier by the addition of dampening materials. These are used for control of ringing. There are a bunch more reasons for cone weight but I don’t want to write a book lol.
Another factor that makes a difference is the crossover design. The more parts in a crossover, the lower the efficiency (every part has its own internal resistance). More parts are generally used when “sculpting” how a speaker sounds. Basically the more “built-in” equalization you want (for better frequency response), the more parts you’re going to use. This equalization is often required due to shortcomings of speaker drivers. Because there is no optimal transducer (physics).
Long story short, it’s all compromises. Don’t get too hung up on sensitivity. Just pay attention to it based on the amplification you want to use (lower power = you need more sensitive speakers). Or conversely, buy the speakers you want and then choose amplification based on them.