The Problems with Electronic Testing of Audio Equipment
A guest post by Michael Levy owner/designer of Alta Audio (full disclosure, Alta Audio is a longtime advertiser with Audiophilia).
A controversy over the validity of testing and measurement has been slowly brewing in high end audio.
In our industry there are two methods by which we evaluate products.
One looks at tests done by programs that send signals through the product to be evaluated by recording the output response.
The second method is to simply have a highly experienced professional reviewer use the product in his reference system and evaluate the results.
Both methods have validity logically, but a discrepancy has become apparent.
A significant percentage of the reviews end up with electronic tests finding deficiencies in a product that has been very highly reviewed by the same magazine and others in listening tests.
You might think that the reviewers had simply missed the flaw, but according to those who were doing the electronic tests, the problems indicated should have been easily detected by the reviewer. Instead, upon listening to musical passages that were expected to highlight the problem indicated, the reviewer never found it, and often raved in praise of the sound.
Very odd, don’t you think?
I am not surprised this is happening.
Michael Levy, Alta Audio.
The entire premise upon which testing is based is wrong in my mind. It is based on a two-dimensional signal graph-oriented concept, and we already have many examples of why that is just not enough. We are wearing blinders, or in our case, ear plugs.
If we want to evaluate an audio product, we must first understand what we want to achieve, and how that product is expected to help us achieve our goal.
The goal of an audio system is to deceive a listener’s brain into thinking it is somewhere else listening to something that is not there.
Therefore, anything that improves the sense of reality is better, and anything that makes the sound less natural is to be rejected. Electronic tests are only useful when they guide us in that goal.
Instead, the opposite is often true. For example, using negative feedback has long been known for the improvement it makes lowering distortion and increasing the linearity of the frequency response of amplifiers and speakers. Unfortunately, listening tests have found that negative feedback creates a sterile sound. It is simply unnatural and most good designs try to minimize or eliminate its use.
Speaker design is where this controversy becomes most obvious. It is also where the testing regiment is most out of line with the intended use of the product.
The reference for loudspeaker testing is an anechoic room. The method used is to play various test signals such as sine waves or square waves and then graph the response through a microphone that has been calibrated.
There are so many things that are wrong with this method that it is hard to imagine that it is the reference. Square waves and sine waves have little in common with music. Also, my experience has been that changes in design are often undetectable by electronic tests, but obvious to the listener.
A better understanding of the relationship between testing and listening is needed.
First, we must understand that we exist at the bottom of a sea of air. The way the air reacts to a speaker (acoustical impedance) varies greatly with the environment. An anechoic room is so unnatural that in a recent New York Times article about one, they mentioned that being in one was torturous. We are using an anechoic room because it is supposed to be a neutral environment. It is not. As most audiophiles know, the room is half the sound of a system. You must include it in your tests, not eliminate it.
Also, a stereo system is taking two two-dimensional signals and using them to recreate three-dimensional sound. That has little in common with reassembling a square wave at one meter into a microphone.
And as for the microphone, it bears no similarity to the ear-brain listening system. One look at the inner ear will tell you that. Why is the cochlea not directly connected to the tympanic membrane instead of being separated by three bones, the hammer, the anvil, and the stirrup? Also, we have hundreds of cilia in the cochlea which each respond to the sounds inputted through the bones into the cochlea with neural output, unlike the singular output of a microphone.
The cochlea looks like a snail shell, and you would think that the cilia would vary in length and run from the wide opening to the narrowing end. That would be the best orientation for a frequency response-oriented system. Looking at an electron microscope image of one shows something surprisingly different. The cilia do not vary in length and are arranged in rows that are perpendicular to the direction of the sound.
All in all, this indicates a system that has been designed by natural selection to capture the whole of an input impulse and define it in such a way that we are able to determine the size,weight, distance, and direction of the source. It served us well in the wild to either help us find prey or prevent us from becoming prey. It also helped us find our friends in the jungle.
The way the brain uses the signals it gets from the cochlea may vary from person to person and is so complex that the only way to determine how good a job we have done deceiving the listener is to ask.
Therefore, measurement should only be used to direct us, with listening as the final judge. If a measurement indicates a flaw that cannot be heard by the listener, it is to be ignored. If measurement finds no flaw, but the listener does, we must continue to work to find out why, and try to find better methods for electronic testing that are more in line with listening.
Michael Levy,
Alta Audio
