From mathematician John von Neumann:

The fundamental reform that will have to take place is…to stop publishing figures with the pretense that they are free from error. There are no such figures, no matter what the layman may think and no matter what the producers of…statistics may assert.

Q: I’m looking for a reviewer that reviews audio components like an engineer would. While I’m not an engineer, I’m fascinated by technical details and measurements. Is there anyone out there that reviews audio components who not only describes what they hear but breaks down the components i.e. parts of the crossovers, gauge of the internal wiring, makers of the capacitors, etc? I’m also interested in more scientific methods of reviewing as well (blinding the reviewer to the make and model of components prior to the review).

A: This, and related ideas, seems to be a relatively common interest, so this seems like a good a time to explain our methodology for reviewing equipment. Those of us who joined this community in the early days of The Absolute Sound (1970s and 1980s) can easily forget that the methods we worked out long ago are not obvious to newer readers and viewers.

We’re going to explain why we mostly don’t do what is described in your question. That is largely because we think many readers, if we could give them what they’ve requested, wouldn’t find it useful (it might be entertaining, but that’s different). We do what we do because it works, and much of what sounds ideal, we think doesn’t work in the real world.

This is perhaps controversial and involves detailed logic, so we invite readers to consider our methodology and logic and decide for themselves whether it suits their purposes. We aren’t making a moral argument, so readers could certainly want something else. If you do, at least you know we aren’t aiming to provide it. And that we have professional, logical reasons for not doing so.

1. What are you trying to accomplish with your reviews?

a. Musical Enjoyment. It is worth reminding ourselves and readers that a main part of the goal here is musical enjoyment. We love music well performed, and assume our audience does too, so we are always searching for enhancements to that experience. We often say “your sensitivities may differ from ours, so we offer insights into what equipment and software and set-up procedures do, but you have to conclude how you value those.”

b. A Quest. One of the great things about the audio-for-music hobby is that it can be conducted as a quest. To support this, we set forth a goal of “believable musical performance”. Such a goal is one that, arguably, cannot be fully achieved, which makes audio-for-music an almost ideal quest. Audio-for-music is also ideal as a quest because it is logically and scientifically complex, which makes it potentially rewarding over decades.

c. Appreciation of Progress. Our experience doing this over more than 50 years is that great progress has been made toward believable musical performance at home. We, and those who join us, enjoy following the development of science and engineering and art in service of the core goal. The people involved in this progress are often interesting and their thinking is interesting and their effort is admirable. We examine all kinds of products and procedures and technologies and histories and personalities as part of this. We are hopeful that the sometimes high price of innovations is simply the beginning of a technological wave that cascades down or up the price ladder.

2. Why do you do subjective reviews?

We don’t. Or for the most part we try not to make that the core of our reviewing. We aim to do observational, objective reviews. Now, there is some confusion about terminology in which “quantification” is “objective” whereas human “observation” is “subjective”. But this is wrong. That notion incorrectly glosses over a critical distinction. “Subjective” in the dictionary means human reactions that primarily involve feelings. But humans are also capable of observing objectively. Losing sight of that second source of objectivity, the use of careful observations, comes at a price, as scientists generally know, but regular conversation seems to forget. It is wise not to let sloppy conceptual categories direct your thinking.

A simple example may help make some sense of this important distinction. If your car is parked next to your house and we ask “which is taller?” you will observe that your house is taller than your car. It isn’t that you feel your house is taller, it is that you are fully capable of objectively observing the height differences. Humans can do this with facial recognition, color identification, bird calls, food smells, voice association and many, many other perceptual tasks.

Professional audio reviewers train to make these observations about audio equipment and report them. Professional reviewers compare notes and very frequently find listening notes that match almost perfectly. But we observe that most new listeners, as well as more experienced consumers, can make these observations too. The training of reviewers tends to involve knowing the signals that may trigger different observable behaviors and being able to use a standardized lexicon to communicate these findings.

The scientific method generally is said to start with observation. Scientists do observations of, say, planetary motion or the proverbial apple falling on Newton’s head, and then form hypotheses, which are tested in various ways. In physics, where scientists are seeking universal laws that can be used for prediction, often the goal is to find a mathematical formula (like F=MA) that applies in many circumstances. This is then verified (or not) with measurements that necessarily must be quantified. The physics model of mathematical formulas and the resulting quantification of things like spacecraft flight is a wonderful development where it can be done (low complexity phenomena). It also seems to lead to much confusion about what people are trying to do outside of physics. Attempting this is scientism, not science.

An extreme example is what we are trying to do when reviewing audio equipment. We are trying to observe the macro behavior of speakers and amplifiers and DACs. You can then form hypotheses about the sound and gather data about the performance of such devices that seem to match you needs. When you get a good match, you buy the product and you’re done. We are not trying to develop a mathematical description of the McIntosh 2800 or the KEF Blade. We and you do not have a mathematical hypothesis that needs quantified data to verify its predictive power.

Engineers also are practiced observers. Many measurements currently in use came to be invented or deployed after observing a phenomenon and then figuring out what measurements might shed light on the phenomenon. Note also, that the existence of a measurement that sheds light on a phenomenon is not the same as saying the measurement completely characterizes the phenomenon. So, measurement is often partial, although helpful to engineers. And so, observational listening is a frequent and often the final step in design work.

3. Why not just measure equipment?

The biggest issue is that the measurement suite (the list of required measurements) needed to characterize equipment is large and complex. In simple terms, one needs to do a lot of measurements, and, very problematically, those measurements are hard to add together into a mental profile of what a piece of gear sounds like if you are a consumer (our audience). What you gain in apparent precision, you lose in meaning (for consumers). Consider for example: frequency response, polar response, power response, time domain response vs frequency, interference, diffraction, harmonic distortion with frequency, intermodulation distortion with frequency, impulse distortion, dynamic compression, sensitivity, impedance with frequency, changes in each variable with level and temperature, etc.

Those are just examples from the speaker world. Amplifiers and DACs are just as difficult. To be clear, by “difficult” we mean difficult to comprehend and difficult to add up into a musically meaningful sonic picture. Most consumers don’t know what half or more of the measurements are measuring or what sonic phenomena they point to. And even if they did know (and surely some do), the individual measurements and their many data points are measuring very specific, narrow technical phenomena that are difficult even for experts to connect with musical results. Note that this “measuring narrow phenomena” element is handy for the engineer who has to solve discovered problems by tracing them to the source. But it doesn’t work for consumers.

That doubt about any consumer’s ability to integrate a full audio measurement suite into a useful understanding of how equipment will perform on music may seem insulting. It may seem as if we doubt your ability to do the integration. This is not the case. What we doubt is that anyone can do it (and most audio engineers would agree, which is why they insist on listening as the final frontier of knowledge). What is behind this is something known in the philosophy of science as “the knowledge problem”. There are many predictive tasks where the knowledge base needed to do passable prediction is simply not available. An example of an unpredictable task that falls under the knowledge problem is a simple coin flip. It is not the case that coin flips are random. Most scientists would agree that coin flips are completely deterministic: they follow the rules of Newtonian physics. The problem is that we don’t know all the Newtonian initial conditions and inputs, so we call the phenomenon “random” when we mean “practically unpredictable”. See this article for more on the knowledge problem, a constant issue in many aspects of life.

A common attempted solution for consumers, then, is to reduce the measurement suite used to a few popular items (e.g. on-axis frequency response or THD). This simply gives a vastly incomplete picture (that is still difficult to connect to the sonic experience except by poor rules of thumb). History shows that this approach often leads to disappointing results for consumers.

Note that in addition to the difficulty of integrating a vast data set to predict sonic results, measurement also has the problem of reference standards. If we have a measurement result from a piece of equipment, what is the goal or reference to which we should compare our result to establish its goodness or badness? As a simple example, is flat on-axis frequency response the goal? Under what measurement conditions, for example what mic distance? If not, what is the reference goal? And how should power response data look? Is a -1db monotonic rolloff desirable? Why? From what frequency? If, instead, the power response rolls off at 1.2 db/octave, how does this affect what we want on axis? We could go on for these two measurements and then continue in this vein for another 30 or so metrics. Measurement results are extremely complex and standards are few and debatable, thus audio engineers spend years mastering measurement, creating some of their own proprietary standards along the way. It, again, seems impractical for consumers to come even close to the required knowledge to render this meaningful. And even if consumers were willing to do the significant coursework needed, there remains the reality that measurements need to be connected to sonic results, generally by…objective observation.

Note also, perhaps ironically, that the reference standard for measurements is often established by…observation (a.k.a. listening to music and then characterizing preferences quantitatively). The observations used to set references are often old and/or developed under debatable conditions (“average listeners” and “typical equipment” and “current recordings”). This isn’t to say that all references are meaningless, but that they require deep understanding for interpretation.

We also must mention the six major issues with audio believability that TAS has identified. These generally aren’t addressed in the standard model of measurement (science is often confined to phenomena that can be modeled with manageable mathematics). Two of these major issues of audio are largely outside the consumer audio equipment realm (the problem of visual images and the problem of recording standards). But the other four (the problem of spatial imaging, the problem of bass in real rooms, the problem of dynamics and the problem of digital distortions) are in the audio equipment wheelhouse but lack quantitative measurement standards. The scope of quantified audio measurement is simply narrower than the scope of observational capability. (It is also the case that measurements can detect phenomena we may not be able to observe; -150 db s/n is measurably different from -145 s/n, but is that observable? – and note that if it isn’t observable we can ask if it is meaningful whereas the inverse is not true).

If we contrast this with objective observations, communicated via words or diagrams, we gain several advantages. We gain quite a bit of simplicity because the characterization of musical audio involves 8-10 key concepts (frequency balance, octave-to-octave output variations, micro and macro dynamics, soundstage, sound space, harmonic and a-musical distortions). While these may benefit from some study, they are relatively intuitive as concepts. They are relatively intuitive because humans have extensive hearing practice (due to a life in which hearing is a survival skill) and humans have extensive vocabularies used day to day for describing experiences. Humans come into audio with some expertise and they can easily learn to expand or sharpen their vocabularies. This is not so simple with reading impulse response measurements or phase diagrams.

Adding to the simplicity in the objective observational approach, we note that sound quality observation concepts tend to be mostly common across all types of equipment. In contrast, amps and DACs and speakers tend to have different measurement parameters, leading to perhaps 100 or more measured parameters characterizing a full system.

With objective observation, we gain the advantage that our observations are conveyed with musically meaningful terms much like those people naturally use to describe that they hear. This, we believe, greatly aids understanding for a greater number of consumers. Consumers can also verify our observations by listening themselves, something they cannot generally do with comprehensive measurements.

And, with objective observation, we can test for all musically relevant phenomena, not just those we happen to be able to measure at the current state of the measurement arts. This allows us to address real but not fully characterized phenomena, for example our “major problems of audio believability”. Since these are extremely important to consumer satisfaction, a methodology that includes them has significant advantages over one that doesn’t.

4. But I like numbers and they seem more “solid”, why do you hate them?

We like numbers too. Some of our reviewers are engineers and some are scientists. And we use measurements frequently for setup purposes and other narrow applications. We’re simply pointing out that the subjective feeling that numbers and measurements are more reliable comes at the price of limited information delivery. Or, as we said above, with measurement “precision comes at the expense of almost all meaning” in audio for consumers. And meaning is what we must have or we have nothing.

Well, that’s not quite true. Some of the interest in measurements and technical description may come from the entertainment value of these things. While we are aiming at providing useful information for consumers trying to improve their audio experience, we intuit that some people want reviews and articles and videos largely as entertainment. We’re working on understanding whether we could occasionally serve that need and how we could do it without interrupting the primary mission. Properly done measurements have the difficulties we’ve described and doing them properly is very expensive. But if such information were for entertainment purposes alone, it might be simplified and become economically feasible if potentially misleading.

We add that the deck isn’t completely stacked in favor of objective observation. The advantages of using objective observation skip over an important issue with the application of words to describe sound quality: words are harder to stack rank when comparing two products. If two speakers both have “tight mid-bass”, it is a little hard to know if speaker A has tighter or less tight bass than speaker B. So, we’ve gained meaning but lost precision. It is frustrating, and we think The Absolute Sound needs to do some work advancing this art. However, you can go to hear the products under consideration and evaluate them for yourself (using TAS reviews to get to a short list).

It helps with regard to comfort with observation to understand that your powers of observation are excellent, an understanding which we think is enhanced once you drop the idea that quantified measurements would be better for evaluating sound quality by definition. Once you start to view measurements as borderline meaningless (for you, though valuable to engineers) and start to view objective observation as an audio superpower that you have, and can further develop, this process becomes more attractive. We find that many consumers assume that they can’t hear differences in sound quality and thus assume that quantitative measures will solve the knowledge problem. We think both assumptions are incorrect. You can hear differences and measurements won’t add meaningfully to your insight.

5. You talk about objective observation as a superpower, but how is your opinion objective?

To use observation as a meaningful measurement technique, you must have a reference standard. This is the case with quantified measurements too, just as it is the case with objective observation. Comparing what we (or you) hear to a reference gets us out of the realm of opinion (subjective feelings). As we said above, it isn’t your opinion that your house is taller than your car. It is an observable fact. It is an observable fact whether a guitar sounds like a guitar, and if it doesn’t, to what degree and in what way.

In music audio, we use the sound of real music and real musical instruments (the absolute sound) as the reference standard. An audio system that can reproduce a guitar or a singer or a jazz band or a symphony so that it sounds believably real, will tend to be more satisfying for most listeners most of the time. This latter point is our experience from over 50 years of listening to live music and audio reproduction of music across hundreds of reviewers.

To be clear, “it sounds good” or “I like it” are pretty much meaningless points to you. Unless you know what my reference is. The existence of a reference shifts observations from subjective to objective, at least if care is exercised. “It sounds good” is too vague to be a useful observation.

Note, again, that with quantified measurements we have to go through two steps to develop a reference: first, coming up with a quantified standard to compare each measurement to and, second, compare the proposed standard to observed musical results to calibrate it to something meaningful. This critical second observational step is often overlooked as a point of error by consumers when thinking of the quantified measurement approach. And sometimes the basis for establishing these references isn’t as clearly meaningful to music as you might think. Similarly, you might imagine that establishing these references may involve either the limitations of the state of science at the time of standard setting, or opinions of the humans setting standards, or practical engineering limitations. In contrast with this problem, music as a reference is pretty close to automatically relevant to music listening. At least if your goal is a believable portrayal of the real thing.

You may not agree with the objective observational approach, but we invite you to try it. And if you don’t agree, you can still use our methodology by understanding what distortions you prefer and comparing equipment sound quality to your reference distortion profile. And you can read our reviews and look for products that fit with your profile (though if your profile is quite far from believably real, we may pass over equipment that you would prefer as we search for gear to review).

6. Why do you have to talk about sound with so many gobbledygook words?

If you assume that there is no methodology used in generating our descriptions of audio phenomena, then perhaps the words look like gobbledygook. But if you consider that this is an objective and carefully executed methodology, then words are useful, relatively simple and probably more powerful than the alternatives.

In addition, if you can learn the hardware parts of a stereo system, you can learn the descriptors of sound quality. The number of basic terms isn’t that vast. And the observational terminology is pales in comparison with the number of concepts needed for proper measurement.

We’ve tried to make the terms relatively simple, but it may help to do a little study of the terms we use for reviews. With this in mind, we have prepared a Glossary of Sound Quality terminology that is within our Audiopedia.

Now, as is common in English, there are plenty of synonyms. Since these are fairly standard words, once you have the basics down, it isn’t hard to understand the synonyms. And, as we pointed out above, the number of basic concepts is much smaller and easier to grasp than with the vast array of measurements.

That said, learning all of this is easier when you look up a few terms and then listen to some music and then repeat.  It can help to have a musical instrument to experiment with as well (piano or guitar for example). And attending concerts can be informative too. You will hear the phenomena we describe and then be able to attach the words to them to give the words richer meaning.

It also helps to spend some time listening to different equipment, noting what you hear and comparing this with review descriptions. Words describing phenomena that you’ve never experienced are tough to understand and warm up to.

7. Why Do You Use Old People As Reviewers When They Can’t Hear Most Of The Musical Spectrum?

There are frequent questions about how The Absolute Sound (and other publications) can use older listeners to review equipment (our reviewers typically range in age from mid-20s to mid-70s). The logic, for those open to logic, is easy to understand if not intuitive (for those in a hurry skip to items 1 and 6 in the list below, but to address common misconceptions about the need for “perfect hearing” be sure to read the whole list):

1. Everyone Has Hearing “Loss”. So-called age-related hearing loss is the proposed issue. Some roll off in high frequency hearing typically is noticeable in humans at the age of 20. The amount of high-frequency sensitivity loss generally increases with age, but varies person by person. Some 40 year olds have more roll-off than some 60 year olds.

2. Loss is Really Roll-Off. The effect of age-related hearing “loss” is actually a reduction in hearing sensitivity at higher frequencies. It is a roll off, not a brick wall filter.

3. Main Impact At Very High Frequencies. Typically, the largest effect is at frequencies above 8-10 kHz. Because the effect is a rolloff of sensitivities, not a brick wall filter.

4. Tones Are Not Everything. It is commonly said that hearing extends from 20 Hz to 20 kHz. This range is based on the use of sine wave tones. But recent (post-2000) studies have revealed that the ear/brain is also a discriminator of time, and that timing discrimination extends up above 100 kHz. Assuming similar age-related sensitivity loss, one could expect older listeners to hear time-based signals above 40 kHz or higher.

5. Less Than 0.1% of Music Affected. Musical energy is not linear with frequency, it is logarithmic. Because of this, as measured and reported in AES papers, less than 0.1% of musical energy is at frequencies above 8 kHz. Note that the highest fundamental note of the piano is 4180 Hz, violin is 3520 Hz, piccolo is also 4180 Hz although technique can raise that a bit. 2nd harmonics will thus generally be below 8 kHz. For further reference, middle C on a piano is 262 Hz.

6. Relative Measurement. A core methodology of The Absolute Sound is to compare the sound of stereo equipment to the sound of real music in real spaces. When listeners are trained in this approach, they are comparing two inputs, each subject to any rolloff in individual frequency sensitivity. The results, therefore, should be relevant to listeners of different hearing frequency sensitivities. Example: If listener A, age 40, listens to a violin, and the top note (A7, 3520 Hz) is played at 90 db, the listener might hear it at 70 db. If the same violin sound is recorded perfectly and played on a perfect speaker, this listener will hear A7 at 70 db. He or she will be able to say that the speaker sounds “like a real violin”. Now, if listener B, age 60, listens to this violin, the A7 may be heard at 50 db. And the perfect recording/speaker will reproduce it at…50 db. Listener B will say “this speaker sounds just like a real violin”. Just as listener A hears it.

7. Ear is Not a Microphone. Thus, a logical mistake is to assume that reviewers are using the ear like a calibrated microphone where the voltage from the mic must be the same for all frequency inputs (an absolute reference). But that isn’t what we are doing. We are comparing device sound to a known reference sound so that observations can apply across listeners. Listeners with different sensitivities will hear the same relative response between the equipment and real instruments.

8. Study Required. To be good at doing this, reviewers must study the sound of real instruments in real space. Basically, this means attending many concerts and doing this regularly. Our reviewers do this and pay attention to learning the sounds of real instruments. Some also play instruments and some keep several instruments on hand to check and update references. This is critical and either not a function of age or a skill that improves with age.

9. Audio Knowledge Required. In addition, reviewers must be able to attend to the many details of audio distortion. This requires study of and familiarity with the sound of resonances, harmonic distortion, intermodulation distortion, filter error, pre-ringing, soundstage stability, timing coherence, octave-to-octave frequency errors, balance errors, roll offs and more. Again, it takes time to learn to observe all of these phenomena. We have frequently experienced our younger reviewers (ages 29-35) spend hours trying to identify some quality that reviewers with decades of experience can identify in less than a minute with no prior input.

10. Ear and Brain Involved. It helps to understand that hearing is not simply a mechanical phenomenon, it is a combination of ear and brain workings of enormous complexity. The role of the brain is so significant that it indicates a role for learning in the process of hearing. This learning process plays to older listeners simply because it takes time to learn. Research indicates some compensation for hearing roll-off by the brain with age.

11. Test Music Crucial. Reviewers also need to know the music that triggers various distortion phenomena in equipment. Reviewers don’t just play some music, they play test music that is revealing of errors, a library of which is built up over years. Of course, younger reviewers have these libraries too, but time and experience are helpful.

12. Everyone Can Have A Golden Ear. Note that these last few points suggest that many reviewers have hearing capabilities that exceed those of typical consumers. That is true in a sense, but this is not a genetic difference, it is the product of study and work and this work can be done by anyone. So, we like to say, everyone has hearing as a superpower, but just like athletes can develop capabilities like running and jumping and throwing, every listener can develop a “Golden Ear”.

8. Why do you insist that I play specific music that you call “the absolute sound” but which seems to be confined to acoustic music performed live? The worthiness of stereo equipment does not relate to the “absolute” nature of that being reproduced. The equipment is good if it can produce the sound for what it is, as it was intended by the creator to be.

We don’t insist on or even recommend that. What we call “the absolute sound” simply defines test signals that we have found useful in characterizing audio equipment distortions. If you’ve followed the methodology we use as outline above, we need references to assess equipment performance. The references are known musical sounds (because we can’t use unknown sounds) as test inputs. And music is used largely because it invokes ~the full ear/brain system and triggers the resulting audible phenomena that are central to “produce the sound” and of necessity to perceive the sound.

Now, very important: there is no presumption that such specific signals must be used by the consumer in his or her listening. The absolute sound is not the music you play, it is the source of the test signals we use that have known attributes that allow the signal to reveal distortions added by equipment.

Now we add the observation that very often the distortions revealed by a broadly challenging set of test signals (music) will apply to music that isn’t part of the test suite. This makes the approach potentially useful. And there is an observation that lower distortion of many types helps to get closer to a satisfying result (i.e. produce the sound for what the creators intended it to be — as far as we can know).  This makes the approach meaningful, for some.

We allow that there could be listeners for whom lower distortion is irrelevant or for whom lower distortion of certain types is not important. They can cherry pick our comments or ignore them. And we allow that listeners to music that is not well recorded may want to ignore this model or may want to cherry pick it if they think it possible to define compensating distortions (e.g. for the 1950s and 1960s standards for bass roll off). The focus of our work is on observing the distortions added by devices. You need to interpret that in light of your sensitivities and musical preferences.

You can of course “just” buy what sounds good to you. Everyone should buy what sounds good to them, we’d say, but “just” doing that is the issue we try to help with.  Many of us tried doing “just” buying what sounds good without any framework for how to proceed or additional input on what works well, and we found it to be too much of a random walk (too many unsatisfying purchases and too many trade ins). So we built The Absolute Sound.

9. If you claim to use ‘the absolute sound’ as your reference for evaluating equipment, why do you have ‘reference equipment’ as well?

Reference equipment is used by reviewers because all equipment has distortions. The more that reviewers can understand and limit the characteristic distortions of the other equipment used in a review, the more the review can focus on the performance of the equipment under test. Conversely, the more the associated equipment is unknown, the more the reviewer doesn’t know what is causing a given distortion from the system being listened to (since only systems can be listened to). Ideally reference equipment has wide bandwidth and very low distortion. This is why our reviewers of (plausibly) the lowest distortion equipment tend to have reference equipment that is of a very high standard.

10. I want proof that equipment does what I want it to do. I need to have validation of your observations by measurements and listening panels. The listening panels and measurements must be certified. I need detailed descriptions of what measurements mean in terms of sound as I would perceive it. The measurements and observations must be backed by specific engineering and scientific logic. Academic or published references must be cited for all theory.

This is not what we are trying to do. In fact, we don’t believe it is possible in the real world. A typical example of the difficulty is the one mentioned above: we know of no algorithm that can integrate a suite of measurements into a set of listening phenomena like tonal balance or sound staging or edginess or blur. In fact, we know of no algorithm that can integrate a suite of measurements into a figure of merit or figures of merit, except by oversimplification.

So we are not aiming to prove things to readers and viewers. We aim to give you a good, useful sense (but not a perfect or complete or exact or unconditional sense) of the character of the sound of equipment under typical conditions. If your usage is unusual or your conditions are unusual, our observations may not apply. And since your perceptual sensitivities may differ, we ask you to learn what those sensitivities are and apply them in judging whether our observed qualities would work well for you. And, for equipment that seems interesting, we strongly suggest that you validate its performance by listening.

Now, it may help to point out that we are aiming for a decent representation of what the engineers might have intended, not aiming for literal accuracy to the input signal (we can’t know when we have literal accuracy). We use the term ‘believability’ to differentiate from ‘accuracy’. The believability idea is also takes into account that we are looking for relatively large/meaningful distortions and accuracy seems to imply any tiny deviations are of interest. Large distortion really means significant distortion, and significant really means “upsetting the sense of a believable lifelike performance”. Measurements, which almost always reveal many deviations from “accuracy” can distract the listener into an obsession with small things that we don’t know how to address and away from solving the big issues. What seems like a drawback (“I want measurements to reveal what I can’t hear”) is to a large degree an advantage in a world where there are so many easily audible issues.

11. Why don’t you do double blind testing?

There are several reasons. Double blind testing to some degree presumes that the comparison of interest is between two pieces of equipment. But, as we have seen, our comparison is with the sound of real instruments and voices in real spaces. For a designer interested in comparing two prototype designs, double blind listening might be useful. In our case, writing for consumers evaluating equipment, there are so many possible comparisons that it is impractical to do them at all, much less double blind. And double-blind listening tends to drag the conversation into relative benchmarking (“speaker A has more bass than speaker B” – okay, but we still have to compare with the absolute sound, so what did we accomplish?).

Double blind testing is difficult to execute well. In particular, level matching is a requirement and is quite time consuming, which means costly. Speakers need to be, presumably, in the same position and this is quite hard without complex mechanical systems. And are we level matching at 1 kHz or level matching the integrated power response or?

Then there is the question of whether such testing should be done with A/B switching. A/B switching has arguable deficiencies that render it possibly misleading. These include memory effects, stress states that do not mirror conventional listening and errors introduced by the switching equipment.

In principle the equipment must not be visible, and, again, that is impractical to arrange in the home environment where listening evaluations occur.

There are certainly potential errors made with objective observational listening too. But adding significant time and cost to the process to generate questionable gains simply doesn’t make sense. This is a common thread we’ve touched on, but to say it bluntly: if you understand what we’ve said above, we could indeed do measurements and double-blind testing, but we’ve argued that this wouldn’t add much value, might be misleading and it would distract consumers from developing their powers of observation. It is unclear how the very large costs of doing this well would be funded in a free media world, as well.

12. Why don’t you explain circuit designs and loudspeaker structures in detail?

 As described above, we think the most effective and efficient way to understand audio equipment is with listening evaluations. Measurements add an order of magnitude to the complexity of this process, and make the process accessible to far fewer listeners. Explaining circuits and structures and materials goes another step further back in the reasoning chain, so that one must attempt to reason along an even more complex path:

Circuits/Structures/Materials>Measurements>Listening Results

As you can imagine, this is much harder than a measurements>results chain, and it leaves even more listeners on the sidelines. We want to invite more people into the audio world.

We should add that there is really no practical way to have a team of reviewers (or one reviewer) who knows as much as practicing engineers about all the circuit topologies and design details and structures and design tools and materials for all the component categories. And, then, somewhat obviously, if we imagined using practicing engineers (as if they’d have time) to do the work, we find that practicing engineers disagree.

We do, however, often mention what manufacturers claim is the differentiating element or elements of the products we review. We do this not because we think we or our viewers can reason from design features to listening results. We do it because we want to explain our motivation in listening carefully to certain products. We find that sometimes a product really sounds different and better than others in its class and to help bring readers up to speed on such discoveries it seems helpful to explain what got us interested in the first place. Sometimes this is “just” a demonstration (high initial sound quality got us to wondering is sound quality would hold up under more extensive evaluation). Sometimes this is a technical feature. Sometimes it is a measurement.

A big point for understanding our reviews, which bears repeating, is that these reasons for investigating a product are not attempts to prove by reasoning (along the chain above) how a product performs in listening evaluations. People with restricted listening experience or people with extensive design experience will often be confused about our purposes here because they are accustomed to attempting to work on the above reasoning chain. That reasoning is fun and perhaps helpful for them, but we are not here primarily to support it. We are not capable of supporting it fully. We don’t think any third party is.

Not only do our reviewers not know everything about every technology, but the designers and manufacturers often do not provide more than basic information about their designs. Sometimes the designs are patented, sometimes they are felt to be proprietary, sometimes the explanation is extremely complex and the manufacturer doubts the ability of consumers to understand so they don’t provide the information. Again, when we have such information and it seems motivating and connected to sound quality, we try to pass it on, especially if we can add value from interviews or discussions with the designers. When such information seems to be mainly “reading the internet to viewers” we skip it or do brief summaries.

We should add a final word about correlation and causation. If we attempt to describe what motivated our interest (and might motivate your interest) in a product, and then we find that the product performs well in certain ways, this is not the same as saying the feature that motivated our interest is the cause of the listening result. The world just doesn’t work that way. There are far too many simplistic reasoning chains whereby a certain type of capacitor or power supply or driver configuration leads to good or bad results. If only things were so simple.

Summary Thoughts

In the end, it seems that some of these understandable questions come from a basic lack of trust in humans and human faculties. If that is the case, we hope we’ve made it clear that it is possible to design a methodology using human observation that is sensible and practical and more useful than quantitative measurement for consumers. We probably haven’t fully addressed the trust issue, but we hope we’ve made it clearer that the seemingly obvious solutions are not as obvious as they appear because they are not the solutions they appear to be.

From Lao Tzu in ‘The Tao Te Ching’:

…The further one goes, the less one knows…