Measurements, measurements, measurements… They are always the forefront of discussion and are used as a guide about a headphone’s sound.
I am not very educated about the process of taking these measurements. I know there are multiple companies making ear simulators that aim to replicate our ears and measure/record the sound that would be going through our ears. For example, ASR is the most famous place for its objective data obsession, but how are these measurements carried out? I am specifically interested in the amplifiers and DACs used to measure headphones.
Correct me if I am wrong, but this is my simple understanding of it: the ear simulation rig has a proprietary DAC, and this DAC is connected to an audio analyzer. Also, how is it ensured that the headphones are properly driven?
My concern is that the measurements aren’t taken on consumer audio products. And if that’s the case, the whole validity of these measurements is up for question.
Why?
You need your test equipment to be more accurate than the products it tests otherwise it would be limiting the results.
The AP has analogue and DAC cards (you can also buy other input cards such as HDMI E-ARC, etc). But it’s important to remember this device is used to analyze test signals not play music.
If you are interested - ASR has articles and videos documenting the setup equipment and measurement decisions as well as explaining what those measurements detail.
IIRC Archimago also showed the insides and gave a detailed explanation of the APx555 that ASR uses.
This sure is true for amplifiers and DACs, but for headphones, I propose it’s very different. I spent some time reconsidering the validity of headphone FR graphs, as well as other headphone measurements, and this is how I feel about them now.
Headphone measurements are fundamentally limited by their reliance on professional-grade lab equipment.
One of the core problems of headphone measurements is how they have been presented to the consumers. There is so much educational content out there teaching how to read frequency response graphs, yet they don’t mention how it is that the measurement are taken, nor what they represent. Imo, the biggest offense headphone measurement people and database websites commit is presenting these measurements as how the headphone sounds—instead of how the headphone sounds through that measurement rig (which is conveniently rarely disclosed).
i.e. consumers can’t accurately extrapolate these results to their own use cases (further explained in the point below)
The headphone measurements taken on such setups only allow for objective comparisons between headphones when isolated from gear variance. Let’s say there is a database of headphone measurements (assuming all of them were taken with the same setup and same method), you can objectively compare how different headphones sound on that specific measurement rig. In other words, their validity is limited to that exact measurement setup. No consumer equipment matches the performance of the professional-grade equipment used to make the measurements.
My conclusion is the following: The validity of headphone measurements relies on the consumers’ blind assumption that gear variations won’t deviate the headphone’s performance far from the measured one. After all, they are nowhere near as objective as they are proclaimed to be. They have the potential to serve as a very limited reference point, but dishonest practices are preventing them from being that.
Given that each gear piece in the audio chain has its own set of physical properties (which aren’t measured), we can ask a scientific question whether or not these properties affect the sound, and if so, how they affect the sound (to make it more relevant to what I am asking, we can say the sound of headphones).
There are no feasible or convenient solutions to the limitations of headphone measurements, and this also extends to the answer to the question I posed at the end.
These are my thoughts. I would love to hear some counter-arguments and have a healthy debate. Kenyon, I thank you in advance if you take the time to do just that.
ASR & Archimago have articles about precisely this so may be worth checking out - I’ll try and dig out links if your having trouble finding the stuff - let me know
Hey chap - haven’t forgotten you just super tied up this week. When I get some time will respond to your points
I don’t frequent ASR, so I’m having a hard time navigating through it. I know Amir has multiple posts up about measurements for DACs and amplifiers, but not for headphones, at least not in written form (post) AFAIK. He includes a video in each one of his headphone reviews, and to be completely honest, I have sat through the whole video. I did notice that he mentioned the gear he uses, but didn’t hear him emphasizing that the headphones will be driven by this gear—it might be self-explanatory and rather logical to anyone who is familiar with measurements, but I think it would be a little much to expect the average Joe to connect the two together and conclude that the graph they are seeing represents the headphone’s sound through that measurement rig. The guides on how to read graphs are constructed in a way that anyone can understand them, and I would expect them to include an explanation of what the measurements explain (with the same easy-to-understand nature). I really feel like that should be a disclaimer. We have disclaimers for reviews, so it would be nice to also have them for measurements.
I myself am guilty here of jumping to conclusions and making assumptions. I haven’t watched Amir’s video from beginning to end, and I also haven’t gone through Archimago’s article(s). It would be great to hear your response to my points, and I just want to say this: I am not expecting or asking you to waste your time to do the reading for me. Please! I respect your precious time, and it wouldn’t be correct to ask you to do my homework =)
Thank you very much! I don’t know whether that rig is the same one used today, but I haven’t seen Amir attach this to his headphone reviews. If we don’t know whether all measurements were taken with the exact same setup and method, their validity and accuracy take a big hit.
The complaint thread sounds like something that would be interesting to read. But nearly 70 pages is a little much for me.
The third link you shared, I sorta read in full. I wouldn’t say skimmed over, but was definitely reading fast, so I might’ve missed some of the points. I don’t think I found answers to most of the things I am questioning, but I did find this thread
I think more of my concerns were addressed here.
You seem like a knowledgeable person and someone who is willing to have a civil debate. So I still look forward to hearing from you. You strike me as someone who has absorbed a lot of information and concerns surrounding this topic. Take your time, no rush. And thank you for sending these links to me!
