When your board asks how the education program did last year, what number do you reach for first? For many of my clients, it’s registrations, completions, or hours of content consumed, because those are the numbers the learning management system (LMS) easily provides.

Those numbers aren’t useless; they tell you whether the marketing worked and whether people stayed until the end. What they can’t tell you is whether anyone did anything differently once they got back to work, and that’s the question that should matter most.

I don’t think most associations skip outcome measurement because they don’t care. Evaluation takes time and money, and education teams are already stretched thin. Here are five ways associations, especially small ones, can measure the impact of their learning programs.

A quick refresher before we start—Kirkpatrick’s four levels are:

• Reaction (did they like it?)
• Learning (did they learn it?)
• Behavior (are they using it?)
• Results (did it make a difference?)

A lot of association evaluations stop at level 1; many certification programs measure level 2 as well. The approaches below roughly follow levels 3 and 4.

1. Ask about behavior instead of satisfaction.

To get at level 3, send a short survey 60 to 90 days after the course, which is long enough for people to have applied what they learned and recent enough that they still remember it. In place of “How satisfied were you with this course?” you’d ask something like “Since the course, how often have you used the resource [name it, e.g., risk checklist] in client engagements?”

Two things make these surveys more useful:

• Write the questions from the learning objectives. If an objective won’t turn into a behavior question, that tells you something about the objective.
• Ask supervisors too, when you can. Their view is a helpful check on what learners report about themselves.

2. Have learners name their metric before the course starts.

This one moves into level 4. Before the course, ask each learner to name one work metric they hope to move. It should be something they or their employer already track, such as error rates, audit findings, or client retention. Then follow up and ask them to report the change.

Self-reported data is imperfect, but a metric the learner committed to in advance is more credible than a vague “was this useful?” rating.

Then add the isolation question from Phillips ROI Methodology: “What percentage of this improvement do you attribute to the training?” Other things changed at that learner’s workplace too (new software, a new manager). This question keeps you from claiming credit for all of it. A modest number you can defend will serve you better in a board report than a large one you can’t.

3. Track credential and career outcomes.

Associations have unusually good access to this kind of data. Your association management system (AMS) and LMS let you look at outcomes a commercial training provider would have to guess at:

• Exam pass rates for course-takers compared with nontakers
• Time to certification
• Recertification rates
• Promotions or role changes reported in member profile updates

One caveat: People who sign up for a prep course may differ from those who don’t. I’d present a pass-rate gap as a strong signal and stop short of calling it proof. 

4. Look at member value and retention signals.

This one measures value to the association as well as the learner, which matters to the board.

Compare members who completed meaningful learning against similar members who didn’t and look at:

• Renewal rates
• Engagement depth
• Upgrades, such as premium tiers, conference registration, or volunteer roles

If you put your most active learners up against the whole membership, the education program will look better than it may actually be. Instead, match on member type, tenure, and career stage where your data allows, and decide up front what counts as “meaningful” learning (a certificate program as opposed to one free webinar).

5. Collect stories with evidence.

The first four approaches give you numbers. Robert Brinkerhoff’s Success Case Method helps you understand the reasons behind them.

Interview a few of your most and least successful learners (the behavior survey from the first approach will help you find both). From the first group, you’re looking for concrete, documented results, something like “we cut onboarding time from six weeks to four using the framework from module 3.” From the second, you’re trying to find out what got in the way.

It’s inexpensive, and it pays off in two ways:

• You get compelling stories for marketing and board reports, with evidence behind them.
• You see where support after training needs to improve, such as job aids or a community of practice.

Where to start

You don’t have to do all five at once. Measurement is much easier when a course was designed to move a specific number, so pick one course and try this:

1. Choose the one outcome the course is meant to move (fewer audit findings, say) and write it into the learning objectives.
2. Ask learners at registration which work metric they hope to improve.
3. Send a short behavior survey at 60 to 90 days, with the isolation question included.

Registrations still belong in your board report, but alongside evidence of what the training changed.

Which of these are you already doing, and what has gotten in the way of the rest? I’d like to hear what’s working for you.