Webinar Replay: Effective Surveys for the Member Journey →

We recently held a webinar, Effective Surveys for the Member Journey, with Jessica Knapp, PhD. As usual, some of the best moments came from your questions, so I'm sharing a few of them here with fuller answers.

‍

Your questions, answered

Q: Should a survey be automated, or sent as a one-off?

A: Both have a place. Jessica's view is that anything tied to a moment in the member journey should be automated: a new member signing up, a first gift, a purchase. You don't want a staff member having to remember to send it, and you want it to arrive while the experience is fresh.

One-off surveys fit questions that aren't tied to a single moment, like an annual satisfaction check or a survey after a big event. Those work best when someone owns them from start to finish, including what happens to the results.

Q: Should we start with interviews or a survey?

A: This comes up with clients all the time, and there isn't a perfect answer. As Jessica put it, it's depth versus breadth. Surveys reach more people, faster and for less money. Interviews give you a much deeper understanding of fewer people.

One piece of advice I was given years ago, and I'm not a researcher, is to start with a handful of interviews. If you spot a pattern, send a survey to the larger group to see whether it holds. It works the other way too: if a survey turns up a theme you don't understand, a few conversations can fill it in. Interviews also help you reach people who would never fill out a survey.

Q: Which survey tool should we use?

A: Jessica isn't picky. She likes SurveyMonkey and Microsoft Forms, and both export easily to a spreadsheet. Qualtrics is the gold standard for many researchers because you can do more advanced analysis inside it.

Her real advice: use whatever your organization already knows. You won't have to retrain anyone, and you'll make fewer mistakes with a tool your team is comfortable with.

Q: Which channel works best: email, text or an app?

A: Jessica prefers email. People see it quickly, and it stays in their inbox, so when they go back through their email they're reminded of the survey they meant to answer. Plenty of people want to take a survey and simply forget.

Text messages get seen almost immediately, but they're also easy to ignore or mistake for spam. An app can work well if people actually use yours, though few organizations have that kind of engagement. I read recently that push notifications aren't dead, but getting people to opt in is hard. An in-app survey may be the better bet.

Q: We get hundreds of open-ended answers. What do we do with them?

A: Start by reading them. Jessica's first step is always to read through the responses and see what themes start to emerge.

From there, you have two options. You can read with a question in mind, for example, are people mentioning our growth? Or you can let the themes come to you without any preconceptions. If you just want good quotes for marketing, a careful read is enough. If you need data for a board or a grant report, go a step further: tag each response by theme and count how often each one comes up. That turns open-ended answers into numbers you can report.

Q: How do we collect stories we can use in marketing and grant reports?

A: Ask for permission up front. Jessica suggests building it into the start or end of the survey, so people agree to let you use their answers in marketing and grant materials.

Then ask open-ended questions about their experience: how was your experience with this program, or what else would you like to tell us? It also helps to explain the trade. Tell people what their story will do, like helping you serve more members or making the case for funding. People are usually glad to help a nonprofit, especially when they know why you're asking.

Q: Our annual survey keeps hearing from the same people. How do we reach everyone else?

A: An attendee from a food co-op described this exactly. Their annual member survey kept drawing the same longtime members, many of whom had belonged for decades, so it rarely told them anything new. It also served the board's questions more than the questions staff needed answered.

A few ideas came out of the conversation. Separate the board's survey from an operational one, since they're really different audiences with different needs. Keep surveys short, or frame questions so they feel different from a traditional survey. And step outside surveys altogether. Watching what people do in a store, holding a focus group, or having someone with a clipboard ask a few questions can reach people who will never fill out a form.

The people raising their hands in the front row will always answer your survey. Most of us aren't in the front row. That's useful to know: protect your most engaged group, and plan other ways to hear from everyone else.

A final thought and author note

A final thought.

A good survey doesn't need to be complicated. Know who you want to hear from and what you need to know. Then show people you listened. That's what brings them back next time.

The author, Tim Yeadon, is the founder of Clyde Golden, the member retention agency for nonprofits.

‍