How We Test

Like all of the web sites in the Reviewed.com family, our approach to testing media players  is to use scientific testing methods. While many other testers rely on 'golden ears' (those with good hearing who subjectively judge audio quality), we use proven scientific test methods developed in collaboration with industry experts. This means that we can not only quote specific numbers for tests (instead of saying that the music sounded bassy, we can quote a specific frequency response), but that we can run exactly the same tests and get the same results, even if two tests are run at different times. The human ear changes over time, but our testing system does not.

Audio performance is the most important thing we test on media players, so we put a lot of time and effort into testing the quality of the sound that the media players themselves and the headphones they come with produce. We evaluate the electroacoustic performance of media players (meaning the electrical and acoustic performance of the device) objectively using a test system from Listen, Inc which is compromised of a SoundCheck^TM electroacoustic test system. This enables us to do precise, scientific testing of media players, rather than the subjective testing that is often used. These components of our testing system are described in more detail below.

SoundCheck
SoundCheck, from Listen, Inc. is an electroacoustic measurement and analysis package widely used for testing audio devices, both on the production line and in R&D applications. It is a PC and sound card based system which communicates with both analog and digital equipment using standard, non-proprietary interfaces.

What We Test
Our audio testing of media players is broken into 6 parts: frequency response, distortion, cross talk, noise,
output power and the headphones. All of the tests are done with the player on its maximum volume level. The first 4 tests are carried out with the output of the media player connected directly to our testing system ; the headphone tests are carried out using a head and torso simulator (HATS). Each of the tests is detailed below.

In order to test the full function of the players, our test sounds are WAV files. This enables us to make sure that the issues we might see in testing are caused by the player, not by MP3 or other compression issues.

Frequency Response
Frequency response is how well a media player reproduces sounds at different frequencies. To test this, our test system uses a stimulus file that contains tones of between 20 and 20,000Hz, which covers the entire frequency range of the human ear. The SoundCheck system analyzes the sound to determine how much of the sound was output by the media player. This produces a graph like the one below, which shows the frequency response of the media player from low frequencies (at the left) to high frequencies (at the right). The two lines are for the left (in green) and right (in red) channels of the headphone output of the media player.
 

The ideal media player would have a flat response, meaning that it reproduced all of the frequencies exactly as they were present in the original stimulus sound. So, our score is based on how close the player comes to the ideal flat response in the frequency range of 200 to 15000Khz; if there are any peaks, troughs or other issues, the player loses points.

Distortion
Distortion is a measure of how accurately the media player reproduces the waveforms that make up the music you listen to; if it clips, compress or otherwise mangles the waveform, your music won't sound the way it should. Using the SoundCheck system, we measure something called the Total Noncoherent Distortion (TND) of the player, using a multitone stimulus.  This approach to distortion testing provides a better simulation of how a player will distort music, because it simulates the multiple frequencies and harmonics that are present in music bette than a conventional distortion test. It is important to note here that this TND analysis is not compatible with the Total Harmonic Distortion (THD) that we measure for our headphone testing.  TND is a measure that examines how the output signal differs from the original source stimulus, including any noise that the players internal electronics add. The pioneers of this approach to distortion testing (Steve Temme and Pascal Brunet of Listen, inc) have produced a technical paper that describes the mathematics and practicalities of how the test works here (PDF link).
 

Cross Talk
The music you listen to on media players is composed of two channels: the left and the right. To accurately represent the music, the two should remain separate, but they don't always do so; on some players, sounds leak from one channel to the other.  That's called cross talk, and it's what we test for here. The result of our test is a graph like the one below that shows the mount of cross talk in decibels; the further the number is below zero, the less sound has crossed over from one side to the other. Similarly, our score is based on this number; the further below zero it is, the better the score.

Signal to Noise
All devices that play back audio introduce some level of noise; the question is how much. We determine how much noise the players introduce by measuring the signal level across the audio spectrum when the device is playing back a file of digital silence, producing a graph like the one below.

We then use these figures to calculate the Signal to Noise (SNR) ratio; the higher this ratio, the bigger the difference between the signal (ie; the music being played back) and the noise (ie; the noise introduced by the amplifier built into the player). The scoring for this test is based upon the SNR; the higher the SNR, the better the perormance of the player, and the higher the score.


Output Power
The raw power that a device can output is an important part of its performance; the higher the output power, the better it will do at driving a high impedence set of headphones. Although the headphones that come with most media players have a very low impedence (and are thus very easy to drive), a player that can output more power will generally do better when driving a better quality set of headphones that have a higher impedence (headphones usually range from around 8 ohms to 35ohms). We measure the output power by outputting a 1kHz tone at maximum volume, then measuring the voltage level across a fixed resistive load, then calculating the output power in milliwatts. We typically expect to see a result in the range of 20 to 50mW, and our scoring is based on the principle of more being better.

Included Headphones
We test the performance of the headphones included with a media player using a subset of the testing done by our sister site headphoneinfo.com; this testing examines the frequency response, distortion, tracking and maximum usable volume of the headphones. For more details on how we test, see this how we test page on headphoneinfo.com. Our score for the headphones is based on an amalgam of the 4 test results to produce an overall score. We test the headphones included with a media player and the media player itself separately because we often find that a decent media player is undermined by the manufacturer saving money by including a cheap pair of headphones. This way, we can ascertain if this is the case, and recommend that the user buy another pair of headphones if the media player would benefit from it.