This is an Eval Central archive copy, find the original at evalacademy.com.

Transforming Evaluation

Part 2

The holidays are over (boo!) and we have all accumulated a few extra pounds, well-intentioned resolutions and a few new gadgets we’re testing out. Maybe some of you are testing out your new virtual reality (VR) headsets? If not, keep reading because this blog post explores augmented and virtual reality and how evaluators can use this technology. If you don’t know what VR is then keep reading – you may find some ideas for transforming your evaluation practice.

What are they and how do they differ?

Augmented Reality (AR) is just that – elements added to the real world to enhance the user’s experience. There are lots of companies embracing this technology:

Not sure if a piece of furniture is going to fit in your space? Ikea has you covered with its Ikea Place app.
Not sure if that wall colour will look right? Benjamin Moore’s Color Capture app lets you visualize the colour on your wall.

The list goes on and on. The point being that AR uses the existing environment and adds information to make a new artificial environment.

VR, on the other hand, does not center on reality but instead generates and transports users to a computer generated, different world. It is a wholly immersive experience where you can leave the world you’re in and enter the world of your making. Not surprisingly companies are beginning to see the power of VR and how it can be used to not just show people their brand and what they are all about, but experience it. Check out this blog for some great examples of how companies are embracing VR technology:

How and why should evaluators embrace AR and VR?

As evaluators we want people to understand what we have uncovered through our evaluations and feel compelled to act on our findings and recommendations. To do this we need to move beyond simply providing information. Most do not truly learn through passively listening or reading – we learn by doing. AR and VR change the way people see and feel the world and thus can process information in a more experiential way. With this in mind, there are numerous applications for how to apply this technology in our evaluations:

1

Project Management

Sure we can bring people together via teleconference or video conference but what if we could actually bring our evaluation team together in a virtual meeting space? There are lots of evaluators conducting multi-country evaluations, VR provides an opportunity to bring people from various countries together in virtual meeting spaces. Feeling uninspired by your meeting room? Why not meet on the top of Machu Picchu? That is sure to stir your team’s creative juices!

2

Data collection

Similarly, what about organizing virtual focus groups? We all know the amount of time and money that goes into organizing those, only to have a handful of people show up. What if, instead, people joined virtually? It also makes the whole ethics piece around anonymity a non-starter since the person meeting in the space could be an avatar and could remain anonymous. Organizing a virtual focus group also allows the facilitator to project slides, videos or other concepts on the screen for participants to easily see and comment on.

We work with a lot with clients who are trying to understand and capture the stories of the people they work with and how they are making a difference. In the past we have been hired to interview people to try and understand their stories. There are numerous issues with trying to elicit information from people (especially if they are a vulnerable group):

Who are you? (building rapport) – We all know it is important to build rapport with our interviewees. Very rarely do people spill their guts and get into the nitty gritty of their experiences with someone they just met and don’t trust. But in the real world there often isn’t time for the relationship building touted in our university textbooks – sometimes we need information from people now!

The word(s) are escaping me (verbal communication) – Expressing abstract concepts, memories or complicated ideas is difficult and especially difficult for kids, people with cognitive impairment, ESL or low literacy. When we rely on verbal language to describe experiences, we often times collect superficial information or underdeveloped ideas, leading to less robust data.

I can’t remember (recall bias) – Speaking of poor data, recall bias is a real thing and can lead to incorrect or incomplete data.

What if you set your participants up with a 360-degree camera and asked them to document their experience? We already do this using methods like photovoice, but imagine how much more powerful a 3D experience is? What better way to understand the actual or lived experience than to have your participants capture it in real time, so others can be surrounded in their world? It eliminates the need for people to recall and express their stories to strangers and as a result provides more valid data.

Not only do we end up with better data, but we also end up with data that people understand and are compelled to use. Research suggests we retain more information and can better apply what we have learned after participating in VR exercises. Interestingly, emotional understanding and empathy are also found to improve with the usage of AR and VR. Stanford researchers found that people who underwent a virtual reality experience, called “Becoming Homeless,” were more empathetic toward the homeless and more compelled to sign a petition in support of affordable housing than other study participants.

3

Reporting and Use

This blog series is all about how we transform how clients utilize information. If we want people to act on our evaluation findings then let’s compel them by immersing them in the data.

“Data parties” are used to create a more participatory environment for uncovering and discussing findings. This is great if you can get everyone in one spot. If that’s not possible, an alternative might be to use VR to create a space to get people together to discover findings. You could even make a game of it. Imagine a virtual Amazing Race where participants could visit the Great Barrier Reef, the Grand Bazaar in Istanbul, and the pubs in Ireland to discover, discuss, and answer questions about findings or come up with recommendations. VR could also be used to show people alternate realities based on your recommendations; you could walk people through different options or recommendations and what those would look like.

Screen Shot 2020-01-06 at 10.26.01 AM.png

“Whoa, whoa, whoa” you say, “that sounds complicated” or “that’ll be way too expensive” or “that won’t work because <insert naysayer response>.” Just like any technology it starts off complicated until it is simplified. It starts off expensive and then isn’t. The technology is already here. There are 360-degree cameras now that can attach to your phone for less than $300, free software to upload your videos and create your own 3D experiences, and super inexpensive viewers that your audience can use to watch/participate in your experience. Check out Google’s AR and VR site for various products.

AR and VR have the power to transform evaluations from a process that provides information to one that creates knowledge. Just like anything if you don’t embrace it you will be left behind. Change happens; embrace it, leverage it. Will your evaluation practice be Blockbuster? Or will it be Netflix?

Check out Part 1 in this series: Visual Storytelling Through Videos

Email Address

We respect your privacy.

Thank you!