Plan Sponsor Magazine
August 2004
by Janet Aschkenasy

Taking the Pulse of Plan Participants

A few years ago, Rich Thau teamed with OppenheimerFunds on a survey of GenXers' attitudes toward money and investing.

He had OppenheimerFunds ask: "Have you ever attended an informational or enrollment meeting concerning your current employer's retirement plan?" Nearly half (46%) said yes. Of that group, a follow-up question was asked: "Which of the following best describes your memory of this meeting?" The choices were "interesting," "neutral," or "boring." A full 61% said "boring" (17%), or "neutral" (44%). Only 39% said "interesting."

If only there were a foolproof way to make 401(k) presentations more rewarding for plan participants and potential enrollees—a way to ensure that your next retirement education session is not a snoozer. Thau, the former president of Washington think tank Third Millennium, says he has just the thing, and it is small enough to hold in one hand.

Thau's work as a political advocate and Social Security reformer has given way to a new career as head of a New York company called Presentation Testing, whose hot new item comes in the shape of a hand-held dialing device able to take the pulse of plan participants moment by moment as they take in your latest enrollment information, retirement plan changes, or asset allocation guidelines.

Whereas the technology is not new, Thau has hit on a brilliant new application for the testing methodology in question, which has a 20-year history analyzing audience response to TV ads, jury trial arguments, and political speeches like the State of the Union. Thau wondered, "Why couldn't we use that same technology to test 401(k) participant education meetings and pinpoint the problem spots?"

Soon, he had T. Rowe Price convinced and, since then, has added other retirement powerhouses like Prudential to his client list.

In early 2002, T. Rowe's Retirement Plan Services unit tried dial-testing with two groups: a hotel chain's eligible non-participants and an insurance company's 401(k) participants. Within each client group, 30 to 40 employees were treated to the identical presentation and were tested for either their understanding or interest in the material.

"Sometimes interest would dip because we went into too much detail or talked too much about a topic, so we really tightened up our scripts [in those areas]," says Carol E Waddell, director of product development and technology at T. Rowe Price, which dial-tested the hotel and insurance company groups over several sessions.

Among other things, T. Rowe learned that employees were not always able to accurately link the terms cash, bonds, and stocks with the appropriate asset class description (stability, income, growth). Some also had difficulty with the most basic calculations (such as calculating the value of 1% of pay), and charts and graphs often required clarification or discussion.

How It Works

In addition to learning what makes an entire group of participants tick, Thau's clients are able to look at how different segments of the group take in a presentation. Aided by software from Portland, Oregon-based MS Interactive, the inventor of the dials, "We can divide the total group into subgroups based upon any breakout the client wants, such as gender, age, investment savvy, or educational level," says Thau. Before the retirement presentation begins, employees twist the dial on the Perception Analyzer to answer a series of multiple choice questions—for example: "What is your gender? Turn the dial to ‘A' if male and ‘B' if female."

"Then, we ask the audience a series of questions of fact and attitude they answer using the dials." The questions of fact—such as, "What is a 401(k)?"—offer a series of multiple-choice answers, while attitude testing might include a question like "On a scale of 1 to 10, how confident are you in your ability to invest?"—with audience members twisting the dial up by increments to give their responses.

Finally, the dial is recalibrated and the actual presentation begins, with audience members being asked to assess just one thing, moment by moment—their interest in what is being said, their understanding of it, or its believability.

"We take a reading from every dial every second and aggregate and average that information. Once the presentation is over, we ask many of the questions of fact and attitude again," says Thau, who follows it all up with a post-presentation focus group that throws the videotape of what the audience just saw back on the screen with the moment-to-moment data superimposed over it.

The price for Thau's presentation testing method is $9,000 for the first audience tested, plus travel and the cost of recruiting that audience. To test the same presentation more than once, Thau charges $6,000 per audience, plus travel and recruitment. The fee generally covers all pre- and post-presentation testing and focus groups. Waddell calls it "affordable research."

Whereas Thau says he has not worked directly with plan sponsors as yet, he says that employers might find it valuable to pre-test key HR presentations or videos explaining new benefits offered. "We could pre-test it with a sample audience of employees and identify the weak spots so they could be removed and replaced by more effective messages," says Thau, adding that basically any major announcement or explanation to employees from the HR department could be pre-tested with a few turns of the dial.