Conducting shootouts-part 1 Philosophy

I’m going to do a couple of posts here about conducting gear shootouts. In this first installment I’ll discuss the purpose and philosophy of shootouts that I personally subscribe to, and in the second I’ll go over doing mic shootouts specifically.

So generally, why do a shootout?

I do these to clearly define the strengths and weaknesses of one piece of gear vs another in order to make buying and session decisions. I personally took a lot of inspiration from the zacuto great dslr shootouts conducted last year, and you’ll see the principles that they applied in those tests echoed in my own.

With that in mind, what are the main things that need to be dealt with when conducting a shootout?

Here’s my list:

• The gear needs to be tested fairly against one another.

• The gear needs to be tested in situations that approximate common and specific real world conditions.

• The gear needs to be stress tested.

So fundamentally here’s how I go about covering those three main points:

Testing fairly:

Testing apples to apples isn’t always possible in the purest sense of the word because manufacturers will do different things with the gear in order to differentiate themselves, but the objective should be to get as close to that reality as you can.

Generally testing fairly means:

• Setting up all devices in a manner that is as close to equal as possible
• this means setting all filters, sample rates, bit depths, and other things to the same settings and otherwise bypassing anything proprietary to a single device if possible.
• calibrating all devices to the same baseline
• I’ll go over calibration in more detail later, but generally you want to lock down a baseline testing level for any signal that’s going to pass through the devices being tested in advance of the tests. Pink noise and db spl meters are your friends here.
• removing or reducing all outside influences on the test
• Outside influences are the things that are required to be in the test in order for the test to happen, but that aren’t being tested themselves. This can be things like the sounds that are being recorded in order to test, the mics, cables and recording devices and preamps and rooms etc that are part of a recording signal chain that are both part of the recrording chain and not part of the equipment being tested.

Testing situations that approximate specific real world situations

Tests are not useful unless they can give information that will illustrate how gear works in the situations you’re most likely to use them. This means that testing an akg D112 as a distant mic on a woodwind section (while potentially interesting) is not particularly useful.

A whole lot of recording gear is tuned up to record the human voice, so generally voice recording is a good baseline for a wide variety of gear. Shootouts can either be wide and shallow or narrow and deep, but not often both wide and deep. Find the appropriate balance there and choose your subject matter accordingly.

As an example, in my handheld recorder shootout I focused on a few handheld recorders that I had available to me. But beyond that I consciously tested their performances using the internal mics only, and I tested them in the context of sound effects recording. I could have tested the preamps separately with external mics, and I could have tested them in other contexts like interviews or musical performance recording, but I wanted my tests to be focused on how these devices perform as impromptu sfx capture devices specifically so I tailored my tests to that specific end.

Think about your scope before doing the tests, then work that scope thoroughly.

Stress Testing

With a scope defined, the next step is to really test the boundaries of what the gear does within that scope. That means if you’re testing a recording device you’ll do things like peg its limiters to see how they react, roll quiet room tone to see what happens down on the bottom end, and otherwise put the gear in a variety of tough recording situations. Microphones can be stress tested similarly, but can also have things like off-axis response, transient response, and proximity effect really run through the ringer.

Stress testing gear is important because a fair amount of what differentiates one piece of gear from another is how it handles extreme situations. The better a piece of gear is designed, the more gracefully it will deal with what you can throw at it. Its often a combination of price and extreme end performance that determines one’s buying and usage decisions.

A note on calibration:

Calibration is a little more complex than just setting everything at the same place. It requires thought into what you’re about to record.

Calibration is usually done by sending tone or pink noise through the devices and setting inputs to a level that is consistent across all devices. In the case of my HH shootout I sent pink noise through at 70db spl and adjusted the pre on each to -12dbfps. This meant that at about 92db I was going to clip, which was a pretty hot setting. That setting was done on purpose because my intent was to both test the limiters on the devices and to test the room tone.

The level at which you calibrate for the test has a big influence on how those tests come out, so choose your calibration (and headroom) level carefully.

Presentation:

One of the things that I really liked about the Zacuto presentation that I tried to incorporate into my own shootout was the presentation.

Specifically, I loved the quick edits from one camera to the next of the exact same footage. I really felt that gave me the best opportunity to make a blink assessment of what I was seeing. In my tests I modeled my edits the same way, and really put the devices right next to each other in the presentation. If I had put each test in its own soundcloud player, the reader (listener) would not really get to make a split-second evaluation, and prejudice with regards to brands and price points would have an opportunity to creep in.

I also did my best to document and describe every step of the test so that the reader could listen to them with a clear sense of what they’re actually hearing. This included descriptions and photos of the setups, tables describing features and functions of the devices, and wraps ups with my own personal impressions and thoughts.

So that’s the philosophy and general theory behind how I run my shootouts. In the future I’ll do a mic shootout and get into some details about how to do mic shootouts specifically.