It really comes down to a personal thing. But I wouldn’t consider a switch because of aesthetics. I would have to have the switch close by for easy access and that won’t look good. Sure its practical, but it won’t look nice. Just imagining the clutter I would have to organize is daunting.
Instead I went with a Topping mx5 which is a good dac/amp combo for both speakers and headphones. It’s got a lot of other features but the fact that I have it all in one device really keeps the wires and clutter down to a minimum.