In this episode, we dive into the critical topic of measuring the success of a conference, be it online or offline. We discuss defining success, identifying goals, and considering all stakeholders involved, such as managers, sponsors, speakers, and most importantly, the audience. We also explore the importance of crafting well-designed surveys and conducting follow-up interviews to gather both qualitative and quantitative data that can challenge assumptions, evaluate the effectiveness of content, and inform future planning. Ultimately, setting the right goals, measuring qualitative and quantitative data, and being open to feedback are crucial for improving future events. Join us as we dissect this fascinating topic, and don't forget to share our conversation with others.

Timestamps 

0:00:27 - Show start

0:03:42 - Considering Stakeholders in Conference Goals

0:09:03 - Measuring Success 

0:11:58 - Measuring Virtual Event Sponsorship ROI 

0:14:31 - Measuring Engagement 

0:23:28 - The Value of Post Event Surveys 

Highlights

"I think there are a couple of things that you have to define when you're looking at how to measure your success. First of all, what are your goals? Why are you even having an event? What is the motivation behind doing this? If you don't have a clear understanding of that, I think that that's gonna make it hard to determine if you met those goals. Because if you can't clearly tell me this is why we're doing it. How am I able to say then, it was successful because x y z?"

"When thinking about your goals, you also have to think about who is involved in the conference. Who are the stakeholders of the conference? [...] if you're a company or or an organization that is supporting the production of this conference, you're gonna have stakeholders in your managers, in the executives, in the people who are leading that organization. Even if you're self organizing, where you're the primary stakeholder, you're also going to have stakeholders in the form of sponsors or partners in the event. Your speakers are stakeholders too. Right? They're contributing content to your event. And so they have a stake in how that is produced, how that is received, and how the conference succeeds or does not because that will benefit or work against their needs.”

"We wanna know are people actually interacting? Are they engaging? What percentage of people actually showed up and are watching? That kind of stuff is helpful because we wanna know how many bodies were in the room, so to speak. But getting down into the granular; Oh, well, did they click over to the schedule page? Well, Who cares? [...] if you're talking about success, broadly speaking, and how we're measuring that, really fine grain metrics aren't going to be that helpful. It can be a little deceiving."

" I think that helps in your evaluation process. If you know clearly why you are asking each question on a survey, then when those results come in, you're gonna know exactly what those results are telling you and how that's going to affect your future planning. And whether it's survey responses or whether it's attendee interviews, that data, whether quantitative qualitative, is going to help you challenge the assumptions you've made around your attendees goals."

Transcript

This transcript was automatically generated by Podium.page using AI and has been edited for clarity.

0:00:10

N: Hi. Everyone. Welcome to the Kickass Conferences Talk Show. I'm Nessa Jimenez, the operations manager for Kickass Conferences. 

I: And I'm Isaac Watson, Executive Producer. 

N: And this is a talk show where we sit down and talk about everything at conferences. Thanks for joining us this week.

0:00:27

N: Our topic is measuring the success of a conference, either online or off. So like always, I always like to start with definitions before we start talking about what the answer is just to make sure that we're all on the same track. Right? So why don't we start first with defining success? Because I feel like for an event for a conference, it's different than for a business itself.  Is that good? 

I: Yeah. Absolutely. I think it's always good to define these things upfront. Because if you don't know what they are, how are you gonna measure them? Right now I think there are a couple of things that you have to define when you're looking at how to measure your success. First of all, what are your goals? Why are you even having an event? What is the motivation behind doing this? If you don't have a clear understanding of that?I think that's gonna make it hard to determine if you met those goals. Right? 

N: Yeah. Because if you can't clearly tell someone this is why we're doing it. How am I able to say then, it was successful because x y z. Right? 

I: Yes. That's absolutely true.

0:01:41

I: Let's talk about some of the actual goals. So this could be a marketing thing. Right? Maybe you have a product or a service you're trying to sell and hosting a conference around it. Is a good way to sell that, I would question that motive, but that is a motive that we encounter. 

N: Yeah. We've talked about that before, and I think we'll have a topic on that alone. Yeah. Another day. 

I: But marketing is definitely one thing. Community building. And that's I think one of the one of the most, I guess, relevant and valuable things you could do with the conference is bringing people together and strengthening the relationships that you have with them and that they have with each other. And then I think that there is also an element of- pardon the air quotes, but ‘thought leadership’. Right? So being able to demonstrate your expertise in a particular industry, being able to gather your network of individuals who have expertise on a particular topic and convene people around consuming that content is a way to look at it.

0:02:56

N: I think those are probably the main goals that we see as we look at people who organize conferences. Some of it is also just education. Right? 

I: I was just gonna say that.

N: Like, related to the last point, education. So if you're there for a topic. Right? 

I: And if your goal is to teach, that's a valid goal because that's something you can measure. Right? Like, if I wanted to teach people to cook, did I actually achieve that. And that's through the content and all of that. So definitely And that's easy to measure because it's a very tangible thing. 

N: Yeah. Exactly. Learn x y or z. Yeah. Yeah. So I agree. Those are, like, the main goals for conferences. There are of course little sub goals that we can get into, but we won't do that right now.

0:03:42

I: But I'd also say, like, when thinking about your goals, you also have to think about who is involved in the conference? Who are the stakeholders of the conference? And there are different stakeholders that can come from different places. So if you're a company or or an organization, that is supporting the production of this conference. You're gonna have stakeholders in your managers, in the executives, in the people who are leading that organization. Even if you're self organizing where you're the primary stakeholder, you're also going to have stakeholders in the form of sponsors or partners in the event. Your speakers are stakeholders too. Right? They're contributing content to your event. And so they have a stake in how that is produced, how that is received, and how the conference succeeds or does not because that will benefit or work against their needs. Because you're adding to the list of goals. Because maybe your primary goal is education. But since you brought a sponsor in, their primary goal is getting emails. Right? Adding emails to their list. So, obviously, their goal is not the same as yours, but it doesn't necessarily take away from each other. Like, you can achieve both of those things, but you have to be aware that they're there. 

N: Yeah. Exactly.

0:05:12

I: Now I would argue that the most important stakeholder that often isn't considered a stakeholder is actually the audience. And the people who are attending your events. 

N: Yes. Yes. 

I: So when we're talking about defining success, I think that we really need to think about what the audience needs and what the audience's goals are so that you can then define the audience's experience and their journey through your event to be able to evaluate whether that was successful. What does the audience need from this conference whether it is education, whether it is networking, meeting, other people. Maybe it's just fun. Right? Like, they just wanna enjoy the weekend of doing whatever at the conference. That absolutely needs to be on the list of goals.

0:06:00

I: And I find that sometimes, unfortunately, they get dropped off the list, and that's not great. Right? We need to think about what the attendees need from this event. What are their goals? Why are they coming? Because, again, my favorite question: Why do they care, essentially? They care about their goals, whatever that is.

0:06:24

I: So if you're not adding that to your own goals list, I feel like that's a big zero. I hate like, that's a big failure, right, from the beginning if you're already, like, dropping the attendees off your list. Yeah. I think if you ignore the fact that the audience is trusting you as an organizer with their time, with their attention, if they're surrendering themselves to what you are planning, and you're not considering what they need or want. 

N: Right. Like, you're right. Big fat zero. That's like failure already. That's not cool. Yeah. 

0:07:11

I: So I think we have a clear list of when we define success, what should be on the list? What are your goals, the stakeholders, the attendees? Everybody. So you bring this all together in a brainstorming session. And you should have a list at the end of the day of, okay, these are the goals of the event and these are the goals for which people. But and I think especially now that we're seeing more virtual conferences, we get into the metrics of actually measuring success, of actually proving that I succeed in this event with my goals, I think we're starting to get muddled because when we're at an in person event, the metrics that that we're using are still gonna be used, right, once we get back in in person. They're slightly different from virtual conferences. The metrics And I think a mistake that a lot of people are making with the virtual events is, just because you can measure certain things online that you couldn't do in person doesn't mean you should.

0:08:22

N: Right? Like, just because you can know if somebody clicked on this thing for two seconds. Does that really tell you anything about your goals? it really depends. Let's get into talking about that, the metrics. We have our goals. So how do we prove that we succeeded or not? 

I: So I think when it comes to metrics, it's kind of similar to the way that people tend to measure success on social media through analytics in the way that they tend to measure, you know, website traffic or things like that. So from a broad perspective, that kind of data can be helpful. We wanna know are people actually interacting? Are they engaging? What percentage of people actually showed up and are watching? That kind of stuff is helpful because we wanna know how many bodies were in the room, so to speak.

0:09:22

I: But getting down into the granular. Oh, well, did they click over to the schedule page? Well, Who cares? Right? Maybe if an attendee comes and says, ‘Oh, I had no idea what was going on’, then you can go back and say, ‘oh, well, it looks like you missed this schedule page’. So maybe that's a communication error. Right? So that could be helpful.

0:09:50

I: But, if you're talking about success broadly speaking and how we're measuring that, really fine grain metrics aren't going to be that helpful. It can be a little deceiving. You could say, ‘oh, well, we can measure ROI so much better’, except that you can't really because they don't tell you what you actually need to know. Is a number of clicks on a button going to tell you how you achieved your goal? No. It's just data overload. 

N: Yes. And from the app's perspective, they always wanna sell it to you. Like, yes. We're giving you all these data points and that's great. We're giving you control. But at the end of the day, it's data overload.

0:10:42

I: Because if you don't know what to do with the data, if you don't have a clear reason for why I'm collecting this and a clear way to interpret it– Because data is just for data, it doesn't mean anything. It has to be interpreted. Like, what does it mean? This piece of data means x y z. K? But we get to the point of data overload just because I can collect it. Now I have it. But what does it mean? Right? And this is granular. And you're right,  I'm not saying, like, there are just data points that don't matter, but it doesn't matter to everyone and in every situation.

0:11:23

I: If I'm trying to measure how many people visited the sponsor areas, for example, which is a big one, which sponsors tend to like to have that information. That's fine. But when we start getting into, ‘They spent two seconds looking at the set of this thing, of this page, and this’ that doesn't necessarily mean anything. If you don't know how to interpret that. Right? 

N: Exactly. 

I: So on the sponsor note, and maybe this is a tangent. But I think one of the things that drives me nuts is that virtual sponsors are getting more discerning about what they will and what sponsor and what value they have in sponsoring a virtual event. And in some respects, that's it's always good to be discerning and to be very intentional.

0:12:20

I: But this has turned an organizer's focus onto demonstrating ROI through analytics like this. How many clicks through your sponsor booth? How many, you know, people engage. What I would challenge out of that is how is the sponsor tracking ROI in an in person event? Yes. And how is that different? Is a  sponsor actually clicking a little attendee counter for every person that comes up to their booth in person? No, they're measuring it in the relationships that they're building. They're measuring it in the conversations that they're having, which cannot be tracked with data points. 

N: And we had a conversation with a sponsor rep not that long ago where they were asking us for, like, these types of data points and things like that. But as we kept the conversation going, they actually said that the biggest value for them was, like, the conversations and building relationships and making friendships with the attendees. And I was thinking, okay, then that doesn't make sense. Then why are you trying to solely measure–  for the report, I guess? For the supervisors?  like, these data points, and then you're telling me, ‘Oh, no. But the biggest benefit we get from events is, these other things that you can't measure in numbers.’ Right? 

I: Yes. And I think that that's the problem is that because the analytics exist, it therefore gets prioritized even though it's not helpful. 

N: Yes. Yes. Yes. 

I: So that's I think that's a clear distinction to make.

0:14:09

I: Shifting slightly. I wanna talk about the types of data that are good to collect. Like, what are some beneficial things? I mean, we mentioned sponsor visitors to booths.I could see how that could be helpful.  even if it's not actually useful. I can see how it'd be helpful from a sponsor's perspective.

0:14:31

I: From an organizer's perspective, some of the things that I'm interested in are the relationships. we're talking again about virtual events where people are coming together distributed from their homes or from their laptops. And they are participating in this event. So we're really trying to measure engagement. Well, what does that engagement look like? And it's about networking. It's about the personal relationships that they're building. It's about how they're engaging in the chat.

0:15:06

N: Those are the types of things that aren't as easily measured through traffic analytics as they are by those platforms who say, oh, yes. This person scheduled, you know, thirty one on one meetings with other attendees. Oh, that's actually used. That shows somebody who is interested in engaging with other attendees that has nothing to do with the conference content, that has nothing to do with  sponsors content in a booth. That's about person to person stuff. Okay. So that kind of data is helpful. But you're right. It's the really granular stuff that it's like, well, it's there, but just because it's there doesn't mean we should prioritize it. 

0:15:52

N: And coming back to– it comes down to qualitative versus quantitative. Right? And there's value in both, but not necessarily in everything. You have to really be clear on what is the goal you're trying to measure and what will tell you what you wanna know. Yes. So I think we can move on then into how? Like, what kind of things do we typically do that we can maybe suggest to other people to analyze these goals? 

I: Yeah. I think as much as we've been trained to dislike them.

0:16:42

I: Surveys are key. Those surveys can help gather both qualitative and quantitative data. And I think that it's important to focus on quality quantitative data. Right? Like, we want the quantitative information that actually tells us something, not and this goes back to the analytics, that not data for data's sake. So crafting a survey to both gather concrete data points as well as anecdotal information. The qualitative end of things is the stories and the experiences and giving people an opportunity to share that information with the organizers to help inform their discussions.

0:17:35

N: I just attended a virtual conference last week, and they sent out an event survey after. a post event survey. And it included both of that, you know, rating of one to five, things like what was your favorite type of content because they had various formats as well as open ended questions like, what was the thing you learned the most? Or if you could change one thing, what would it be? Things like that. 

I: Yeah. Let's keep things open ended And then the other thing that we like to do beyond surveys is actually go deeper because people are generally unless you're talking about, like, super negative reviews where people are just, you know, going on and on at length as if they were posting.

0:18:27

Generally speaking, open-ended survey responses are gonna be less in-depth than conversations. So what we do with our clients at the end of each event is we identify, you know, three to five attendees that kind of stand out to us across a demographic spectrum. And schedule follow-up one on one interviews. Or in our case, it's two on one because we're both involved with that. We're actually having a conversation with them.

0:19:04

I: Just open ended, like, tell us about your experience. Have you been to this event before? What were your goals going into it, if any? Did you feel like you met those? And we can have people open up and tell us more about what they learned and what they experienced through the event in a way that a survey will not expose that data.

N: And it's not designed to.

I: Right. Yeah. But those interviews, those attendee interviews that we have, inform the reporting of the experience that we do because they are deeper quantitative data points that really help us measure the success. Now, obviously, five attendees are not going to be representative of five hundred. But it does help to kind of learn more in-depth about the individual experiences and the impact you're having on individual people, which can have a lot of meaning as you're measuring your success.

0:20:15

N: Yeah, and in conversations, people will share things that they would never share in the survey. because a good survey, It's quick. You fill it in, you move on. It's not meant for a story. And in these conversations, people will tell you stories. Deep stories about themselves, about their experience, about things that happen after the event. You're not gonna get from a survey. And I think it helps us write the story of the conference from a different perspective. Right? Like, it's an eye in the room for perspective that as an organizer, you're never gonna get because you're running around taking care of the behind the scenes things. Right? 

0:21:15

N: surveys are no one's favorite. But a big issue with that is when people ask questions on surveys, that they don't need to be asking. Why are you collecting all this data when you're not really gonna do anything with it? And so the survey becomes like this thirty question thing that nobody wants to fill out. 

I: Every single question in that survey should have a specific end goal in mind as far as what you were trying to assess from it. 

N: Ranking speakers from most favorite to the least favorite. Is that actually helpful? Why? Are you giving that to speakers? Do you really want to know this? Yeah. Like, if you had a valid reason, great. Fantastic. But I find that with surveys, they're just asking because they saw another conference ask that same question. And so they just kinda think that that's what you're supposed to do, and then they never do anything with that information.

0:22:25

I: Would we also say we need to start questioning why we ask for gender and race and all of this stuff? In these types of surveys, if that's not gonna tell us anything either. Like, I understand an event where their goal is. Okay. I Want to have more women this year. Right? And a way to measure that, like, hey, did more women attend? We can ask that question. But if that's not something that you have a specific goal around? Why are you asking this question? 

N: Yeah. I mean, if your event is specifically targeted at people who identify as women, then why do you need to collect that? It doesn't matter because it's kind of obvious. But again, that's one of those questions that people ask because that's kind of what we've always asked, and we don't even think about it. We just write it down.

0:23:24

I: The questions that you're asking the audience, like, why do you wanna know those things? Yeah. And I think that that helps in your evaluation process. If you know clearly why you are asking each question on a survey, then when those results come in, you're gonna know exactly what those results are telling you and how that's going to affect your future planning. And those whether it's survey responses or whether it's attendee interviews, that data, whether quantitative, qualitative, is going to help you challenge the assumptions you've made around your attendees goals. It's gonna help you evaluate the effectiveness of the content or the programming that you put together, it's gonna help you evaluate whether or not you met your stakeholders goals. So I think that that is really important.

0:24:13

I: And I've seen some people recently challenge the notion of doing a post event survey in the first place, I think they're still really valuable. I think that's information that you're not gonna be able to gather inherently through the production of an event. And I think that that information is really useful if you do it correctly. And not only is that informing, did we succeed with this event? but it should also inform what we 're gonna do the next time. What are we doing next year? So, yeah, they totally have value.

0:24:46

N: I can't imagine not doing them because that feels to me like throwing something out there and then having no idea how people felt about it. Right? Like, like no. I can't. I need to know. I need to know if people were enjoying themselves. Alright. So Summing it all up then.

0:25:08

N: What can we say as a parting thought for people? Like, how would we measure success? 

I: You have to know your goals going into it and the goals across your stakeholders. You have to be able to measure qualitative and quantitative data and then evaluate it. be honest, like, don't take it personally when you get negative feedback. You have to see it for what it is and be able to discern, ‘is that valid? Is this specific to this individual's experience?’ How can that inform a better experience the next time around? Same with the positive feedback.

0:25:47

I: We don't wanna live in an echo chamber of, you know, cheerleading ourselves. But we wanna be able to make good decisions and set better goals or set the right goals going into future events based on the feedback that we receive. So I think those are pre and post. You've gotta have the right goals and the right definitions of success upfront, and then you have to have the right data points to be able to measure against those goals. 

N: Absolutely. Alright. Thank you so much, Isaac, and thank you everyone for listening to this week's episode. If you found our conversation helpful in any way, I invite you to share it with other people. And you can find us online at kickassconf.com. Thanks for joining us. See you later. Bye.