High sample rates

I currently own a Cambridge dacmagic 100 and I just started using digital instead of USB as the input to get access to the higher sample rates. I noticed that when I play audio in 24bit/96khz or higher my audio becomes lower and I have to raise my headphone amp volume to compensate. Is this normal when playing music, at higher sample rates?

No, not if it’s the same signal.
Some hires masters are better and are recorded quieter, you probably also want to check you don’t have loudness compensation turned on or some sort of digital volume in the chain.

I had the dac too, or at least it is not in the cupboard as a replacement.
I connected it to the Playstation 4 via optical and had to turn it up a bit but not much on the amplifier.