1. We already conduct focus groups. What's the advantage to conducting dial groups? What will it tell us we don't know already?
Here’s an analogy: How much more does a doctor know with an MRI than with an X-ray? A focus group alone won't indicate with much accuracy at what precise moments your audience became confused or bored with a presentation, or disagreed with a public-policy video. A focus group alone won't indicate whether that boredom or confusion was age- or gender-specific. A focus group alone won't indicate quantifiably which were the strongest and weakest parts of the presentation. But our dial testing methodology diagnoses exactly where the "weak spots" are, so we can eliminate them.
2. How do you separate the presenter’s delivery from the content?
During a moment-to-moment dial test, our focus is on content, and content only. We instruct audiences—to the best of their abilities—to focus solely on the content and not on the presenter’s demeanor, dress, friendliness, or any other personal characteristic.
That said, we recognize that presenters obviously do matter. If we are testing for moment-to-moment claimed understanding, for example, audiences might turn their dials down if the speaker mumbles, or if the content is confusing. To minimize the times when delivery might interfere with an evaluation of content, we only dial test proficient speakers. If presentation skills are lacking, we will refer the presenter to a presentation coach. We also find that once an audience views a proficient speaker for a few minutes and becomes accustomed to him or her, they tend to discount the delivery and focus on the content, as they were instructed.
3. Won't the audience forget to turn the dials while the presentation is going on?
Before a presentation starts, we do a practice run with respondents to help them get comfortable with the dials, and to make sure they’re using them correctly. Prior to the presentation we remind them to turn the dials as often as they are comfortable doing so. For some people that translates into continuous motion, for others it means a periodic turn once every few minutes or so. It all depends on the person. If a respondent cannot get the dial testing process right during the warm-up, we discreetly pull them out of the session and send them home (with pay).
4. Doesn't the act of asking audience members to turn the dials affect their moment-to-moment responses?
No doubt it's possible that people will pay more (or less) attention to a presentation if they are being continually dial tested on its contents. Until we can monitor continuous audience feedback by inserting a computer chip into the minds of audience members, this type of imprecision will occur!
What is most important is to compare other methods of data gathering to this one. And compared with traditional focus groups conducted without dial testing, our method is far more detailed and superior.
5. How is dial technology mostly used today?
The dial technology that we use at Presentation Testing has been widely employed for 30+ years for testing audience reaction to TV and radio ads, audience agreement/disagreement with political speeches, student feedback to college lectures, and mock-jury testing of the believability of prospective witnesses in trials.
Presentation Testing has broken new ground by using this dial technology to test the effectiveness of business presentations made to consumers and to iteratively create and refine advocacy presentations made to voters.
6. How does one interpret the data on the moment-to-moment graph, below?
The image shown here is a single-moment freeze-frame from a speech where the audience was dial-tested. The four jagged lines (in red, white, blue and green) super-imposed over the speaker correspond to the second-by-second reactions that various sub-groups of viewers had to the speaker as he was talking.
This particular image displays the most-recent 30 seconds' worth of moment-by-moment audience reaction. During the test, each member of the audience held a dial in his/her hands, similar to the one shown here. Audience members watched the presentation while simultaneously turning the dial between zero and 100, registering their positive or negative reactions to the speaker.
Men 50+, represented by the red line, registered the most positive reaction to the speaker. Women ages 50+, represented by the blue line, showed the most negative reaction.
In the bottom left-hand corner, the data indicates for each sub-group three things: the number of people in the sub-group, the average (mean) reaction to the speaker for the entire sub-group at that particular second, and the sub-group's cumulative average response to the speaker from the beginning of the presentation to that particular second.
For example, there are 12 men ages 18-49 who were tested in this sample. For the second that is displayed in this illustration, their average score was 59 (out of 100). For the entire presentation up to this point, this sub-group of men gave the speaker an average score of 54.0.
7. What happens if the technology malfunctions during the test?
The dials are very reliable, but like all electronics, they have their moments! The dials emit a radio signal that is picked up on a specific frequency. If, in the odd circumstance there were interference with this frequency, it likely would be noticed during the set-up and equipment check phase of the test, hours before the audience arrives. In this situation, the dials would be connected to the computer using standard telephone wires. The technology used for the test has an extremely low failure rate (well below 1%). Prior to each use, every dial is tested to ensure that the computer is receiving an accurate signal. We travel with spare dials in case a dial is inadvertently damaged by a respondent.
8. What are the logistics of setting up a dial test? What do you need a client to provide in advance?
We would need your company to arrange a typical sales/HR/marketing/employee communication meeting anywhere in the United States, and give us two-to-four weeks of advance notice to plan the testing logistics around it. With this lead time, we can schedule a technician, ensure the availability of the testing equipment, and work out the mechanics and content of the test.
Specifically, we will need to see an advance copy of the slides that will be used by the presenter. (If you need us to sign a non-disclosure agreement, that is no problem.) This way Presentation Testing will have an understanding of the meeting and can jointly determine where to remind the audience to "keep reacting" (see question 3, above). We would work closely with you to devise the multiple-choice questions and zero-to-10 scale questions we would ask the audience. These questions will reflect the contents of the presentation.
9. How is the price of Presentation Testing's service determined?
Our dial-testing price is determined by a number of factors:
a) Size of audience(s)
b) Number of times the presentation/video will be tested
c) Number of different presentations/videos tested
d) Depth of final report required
e) Availability of audiences (i.e. Does the audience need to be recruited and paid, or is it already going to be attending an event?)
f) Whether the dial-testing work is preceded by in-depth interviews, and if so, how many
Please call us to discuss the scope of your project and we can give you a quote. If you are accustomed to conducting traditional focus groups, our pricing is moderately higher considering the technological requirements of dial testing and the effort required to interpret results and make actionable recommendations.