1. We already conduct focus groups. What's the advantage to conducting dial groups? What will it tell us we don't know already?
Here's an analogy: How much more does a doctor know with an MRI than with a stethoscope?
A focus group alone won't indicate with much accuracy at what moments your audience became confused or bored with your presentation. A focus group alone won't indicate whether that boredom or confusion was age- or gender-specific. A focus group alone won't indicate quantifiably which were the strongest and weakest parts of the presentation. And most importantly, a focus group alone won't indicate whether there's a correlation between certain sub-groups' understanding of (or interest level in) the presentation and their ultimate purchase of your product. But in conjunction with our dial testing methodology, which diagnoses exactly where the "weak spots" are, a focus group such as the type we conduct is enormously effective in helping find out why these problems exist and how they might be fixed.
2. How do you separate the presenter’s delivery from the content?
During a moment-to-moment dial test, our focus is on content, and content only. We instruct audiences—to the best of their abilities—to focus solely on the content and not on the presenter’s demeanor, dress, friendliness, or any other personal characteristic.
That said, we recognize that presenters obviously do matter. If we are testing for moment-to-moment claimed understanding, for example, audiences might turn their dials down if the speaker mumbles, or if the content is confusing. To minimize the times when delivery might interfere with an evaluation of content, we only dial test proficient speakers. If presentation skills are lacking, we will refer the presenter to a presentation coach. We strongly recommend 3D Communications.
We also find that once an audience views a proficient speaker for a few minutes and becomes accustomed to him or her, they tend to discount the delivery and focus on the content, as they were instructed.
3. Won't the audience forget to turn the dials while the presentation is going on?
Before a presentation starts, audiences are told to turn the dials as often as they are comfortable doing so. For some people that translates into continuous motion, for others it means a periodic turn once every few minutes or so. It all depends on the person.
As is common practice in this field, we remind audience members every 3-5 minutes to turn the dials by saying the words "keep reacting" from the back of the room. In a live presentation, these reminders don't occur randomly, but rather at natural break points in the presentation, worked out in advance with the speaker. Usually these break points occur when the speaker is changing slides and/or topics, so it does not interfere with the flow of the presentation. The audience is told in advance that it will be periodically reminded to turn the dials. In a video presentation, we edit the reminders into the footage. We have a full editing facility in our office, as well as a teleprompter.
4. Doesn't the act of asking audience members to turn the dials affect their moment-to-moment responses?
No doubt it's possible people will pay more (or less) attention to a presentation if they are being continually dial tested on its contents. Until we can monitor continuous audience feedback by inserting a computer chip into the minds of audience members, this type of imprecision will occur!
What is most important is to compare other methods of data gathering to this one. And compared with traditional post-presentation focus groups conducted without dial testing, our method that employs the dials—while admittedly imperfect—is far more detailed and superior.
5. How is dial technology mostly used today?
The dial technology that we use at Presentation Testing has been widely employed for 20+ years for testing audience reaction to TV and radio ads, audience agreement/disagreement with political speeches, student feedback to college lectures, and mock-jury testing of the believability of prospective witnesses in trials.
Presentation Testing has broken new ground by using this dial technology to test the effectiveness of business presentations made to consumers and to iteratively create and refine advocacy presentations made to voters.
6. How does one interpret the data on the moment-to-moment graph, below?
The image shown here is a single-moment freeze-frame from a speech where the audience was dial-tested. The four jagged lines (in red, white, blue and green) super-imposed over the speaker correspond to the second-by-second reactions that various sub-groups of viewers had to the speaker as he was talking. This particular image displays the most recent 30 seconds' worth of moment-by-moment audience reaction. During the test, each member of the audience held a dial in his/her hands, similar to the one shown here. Audience members watched the presentation while simultaneously turning the dial between zero and 100, registering their positive or negative reactions to the speaker.
A single-moment freeze-frame from a speech where the audience was dial-tested.
Women ages 18-34, represented by the red line, registered the most positive reaction to the speaker. Women ages 35 and older, represented by the green line, showed the most negative reaction. (The entire group's response is represented by the white line.)
In the bottom left-hand corner, the data indicates for each sub-group three things: the number of people in the sub-group, the average (mean) reaction to the speaker for the entire sub-group at that particular second, and the sub-group's cumulative average response to the speaker from the beginning of the presentation to that particular second.
For example, there are seven women ages 18-34 who were tested in this sample. For the second that is displayed in this illustration, their average score was 79 (out of 100). For the entire presentation, this sub-group of women gave the speaker a score of 67.8.
7. What if someone in the audience doesn't want to participate in the dial test?
As part of the pre-presentation instructions to the audience, Presentation Testing encourages members of the audience to indicate whether anyone is uncomfortable or unwilling to use the dials. When this occurs, the particular audience member is relocated to a seat in the audience where no dial is located. Sometimes, when there are extra attendees at a meeting, we recruit one to fill the vacancy. This person will have the same demographic characteristics as the person who relinquished his or her dial.
8. What happens if the technology malfunctions during the test?
The dials are very reliable, and we use the latest model produced by the manufacturer. The dials emit a radio signal that is picked up on a specific frequency. If, in the odd circumstance there were interference with this frequency, it likely would be noticed during the set-up and equipment check phase of the test, hours before the audience arrives. In this situation, the dials would be connected to the computer using standard telephone wires.
The technology used for the test has an extremely low failure rate (well below 1%). Prior to each use, every dial is tested to ensure that the computer is receiving an accurate signal.
9. What are the logistics of setting up a dial test? What do you need a client to provide in advance?
We would need your company to arrange a typical sales/HR/marketing/employee communication meeting anywhere in the United States, and give us two-to-four weeks of advance notice to plan the testing logistics around it. With this lead time, we can schedule a technician, ensure the availability of the testing equipment, and work out the mechanics and content of the test.
Specifically, we will need to see an advance copy of the slides that will be used by the presenter. (If you need us to sign a non-disclosure agreement, that is no problem.) This way Presentation Testing will have an understanding of the meeting and can jointly determine where to remind the audience to "keep reacting" (see question 3, above). We would work closely with you to devise the multiple-choice questions and zero-to-10 scale questions we would ask the audience. These questions will reflect the contents of the presentation.
10. How is the price of Presentation Testing's service determined?
Our dial-testing price is determined by a number of factors:
a) Size of audience(s)
b) Number of times the presentation will be delivered and tested
c) Number of different presentations tested
d) Extensiveness of final report required
e) Availability of audiences (i.e. Does the audience need to be recruited and paid, or is it already going to be attending an event?)
Please call us to discuss the scope of your project and we can give you a quote. If you are accustomed to conducting traditional focus groups, our pricing is moderately higher considering the technological requirements of dial testing and the effort required to interpret results and make actionable recommendations.