string theory

This is an Eval Central archive copy, find the original at camman-evaluation.com.

Here’s something I’ve been thinking about.

Imagine a big knotted clump of yarn, the kind of thing that happens when you leave a crafting project at the bottom of your bag for too long. (Alternatively, what happens to your headphones if you leave them in your pocket for five seconds.) It’s a solid mass of strands going every which way at the centre and a bunch of loops and looser tangles hanging off it. If you’re lucky you might know where the two ends are (assuming there’s only two!), but you probably have no idea what else is going on in there and how it’s gotten this tangled and useless. The fastest way to get rid of it would be to cut it (ah, Occam’s Razor), but that wouldn’t help you finish the crafting project, the scarf or hat or sweater that this thing is supposed to be.

Every knot is its own beast, so there’s no single specific sequence of steps to follow to untangle this one. But there are some basic things you’ll probably do, either from trial-and-error learning or past experience if you’ve done this before. To start, you’ll look over the whole thing and try to see if there’s a likely place to begin, a bit that seems amenable to a little tugging. You might give the whole thing a quick shake to see what that loosens up. You’ll want to track down those end strands if you can and keep an eye on them. As you pull a little bit at one spot and then another, you’ll keep looking back at the whole thing to see what’s shifting, what you might pull on next. There might be some false starts before you finally get some traction. With a bit of luck, things will begin to pull apart and the knots-within-knots will start to surface. There may be parts that have been pulled so tight it seems like you’ll never pry them apart again. You’ll be confronted with moments of guessing whether something is a slipknot, where one sharp yank will pull it through, or not, and the same yank will fix the knot in place for good. You might find yourself suddenly at an impasse after a whole lot of good progress with a something that just won’t give. You may end up with several separate equally confounding tangles, smaller but no more straightforward than the one you began with. At various points in the whole frustrating process you’ll wonder why you’re even bothering, why you don’t switch to watercolours or needle-felting. Or you might hit a flow where you press on and the act of disentangling feels as much a part of your hobby as the deliberate ‘entangling’ you do with hook or needle. (It’s definitely a good way to get familiar with the material of your craft.) Hopefully you hit a point where the tangles start to fall away on their own, when you find that little loop that got stuck inside another loop and then another, kicking the whole mess off, and then all of a sudden you’ll have a lap full of soft strands, ready to be rolled up into a ball so that crafting can continue, like the knot never existed. (Or maybe you transcend the paradigm, give up on the sweater, and turn the knot into a cat toy or cut it up and use it for stuffing.)

What earthly relevance is all of this?

One of the things I’ve been working on in my evaluation practice is my own internalized framing of what my work is—what it’s about, what it’s for, why I’m doing it—and shifting that from a focus on specific programs, “is it good or not? Was it a success or failure?”, and into something that is more contextual and strategic—“what is happening here? How does this relate to the larger change effort?”. Some of this is from moving deeper into developmental evaluation, driven by the needs of clients working in spaces of deep complexity and uncertainty, who are adapting and inventing practices and approaches as they go. But as I go further into this space, I also find that traditionally-focused summative and formative evaluations benefit from being contextualized within this wider lens. I can’t think of anyone I’ve spoken to recently in the social sector who didn’t have a sense that big shifts are needed and that individual programs and services are meant to be part of larger transformations. This is where I find the developmental perspective to be liberating and necessary, even when not doing strictly developmental evaluation work, because it moves out of the either/or paradigm of ‘success/failure’ that informs the summative and formative models and offers a different frame in which to think about and judge what we’re doing.

Here’s another way I think about it:

This is my understanding of an ‘ideal’ evaluation scenario, with evaluation involved at several points in a program development and refinement process. In the planning and design phase, evaluation helps surface questions about models and theories and assumptions, lay the groundwork for data collection, and gather early impressions and context. Then throughout the implementation, there are formative assessments, asking, “How can this be better?”, checking in on implementation quality and fidelity and beginning to look at outcomes, first the early ones and then later ones, seeing how the program theory is panning out. Finally, at the summative phase, going in-depth into those outcomes, what it takes to make the program work and whether it can and should be scaled and replicated, asking, “Does it work and is it worth it?” The underlying assumptions of this process are that things that ‘work’ can be reliably expected to keep ‘working’ in their own context and even in other contexts, and that what ‘working’ means will also be relatively stable. If something is good, then it is good, by whatever measure of ‘good’ we have agreed on. (And how “good” and “working” are defined and by whom and to what ends are separate questions that are often under-explored.)

Here’s another picture, one that captures more of what I have experienced and witnessed working in spaces of change, both in the formal social sector and in activism, and what another ‘ideal’ evaluation process might look like:

Looks a lot more like a tangled ball of yarn, right? The evaluation line is running parallel to the whole development process (which isn’t neatly distinguished into “design” and “implementation” either, since the whole thing is a constant exercise of doing and re-working and adapting). Learning and doing are happening together, and the nature and focus of the learning is shifting and adapting as much as the doing is. There’s an overall guiding direction (from one side to the other), but the moment-to-moment directions change, including sometimes going back over itself. It could just as easily be a journey-map of someone lost in a mall, or of the way we handle our yarn knot.

To me the disentangling of a yarn knot is metaphor of how big change happens, scaled down to something I can hold in my hands and my head. I zoom in to individual stands and knots-within-knots, then zoom back out again to assess the whole, then zoom in again. My immediate outcomes, their indicators, and my interventions are shifting and relative. I tug on a strand and notice the way it leads to movement on the other side of the knot, so I move to that part next and see what headway I can make there, looking back to the part I was working on before as well as keeping an eye out for what else might be happening elsewhere in the knot (and to ‘scale up’ the metaphor, imagine a knot the size of a Buick and a group of people working on different parts of it simultaneously, affecting each other’s work and hopefully also communicating effectively). This is the systems-view understanding that everything is connected and influencing each other, often in ways we can’t immediately decipher, with the need for timely data and insights to inform and support the work as it’s happening as an integral part of what the work is. There are formative and summative queries embedded through out this developmental process (“is this working? Could it be better?”), but they are contextual rather than absolute, and don’t require complete, definitive answers each time because the context will keep shifting anyway as we continue to interact with the system.

In these spaces, it’s not about “what’s good?” or even the more pragmatic “what’s good enough?” It’s about “what can we do right now that might suit our present situation as well as work in service of (albeit not always immediately directly toward) our ultimate purpose?” It’s strategic and deeply, deeply contextual, to time, place, purpose, resources, people, opportunities, etc. In these spaces ‘successes’ aren’t always successes and ‘failures’ aren’t always failures. Success and failure are relative, dynamic, and sometimes just irrelevant.

What does this look like in practice?

I find it challenging to bring up specific examples from my practice that exemplify the difference I’m talking about because so far I’ve found it quite rare to work on projects that fully embody this understanding. Even in developmental projects, the formative/summative way of thinking is deeply entrenched and often reinforced by funding models that serve short-term, linear, and compartmentalized thinking and ways of working (and I can’t single clients and projects out until their findings and process are at a stage for public sharing). But fortunately activism abounds in examples of working in strategic adaptation supported by deep evaluative practice (despite not being formally recognized as such).

Here’s one: sharing our pronouns. You might have experienced people sharing what pronouns they use, maybe in their email signature, on a button on their lapel, or as part of introductions at a workshop or event. You might have a practice of sharing your own pronouns or asking people for theirs. As someone who uses ‘non-standard’ pronouns (they/them), I end up advertising mine a lot and have written a whole blogpost on the topic which I share often, partly as an end-run around having the same conversation about them on repeat. Pronoun sharing is a fairly recent phenomenon, taking off within the last few years, though it’s far from universal. It’s also garnered a fair share of critique, both on form and function, and some of these have produced modifications. One early criticism was the language of “what are your preferred pronouns?”, which was argued to suggest that some people’s pronouns are merely ‘preferences’, and we’ve generally moved away from that language to just, “what are your pronouns?”. We’ve also pushed for pronoun-sharing to become a normalized activity for everyone so that we aren’t singling out the people with ‘non-standard’ or ‘non-obvious’ pronouns.

If we took a formative lens to this, we might be thinking, “yes, we’re tweaking the intervention to improve it, make it the best version of itself that it can be”. But some of the criticisms aren’t easily resolved. By making pronoun-sharing a routine activity, we’ve also fostered scenarios in which people have to choose between being ‘out’, with all the risk that entails and the infringement on self-determination, being misgendered, or, if they chose not to disclose at all, being perceived of as less ‘progressive’ than their cis and binary-gender peers who can put “she/her” and “he/him” in their email signatures without concern. I actually dislike having to constantly advertise my pronouns. I find it distracting and a constant reminder of the awkwardness of not fitting in. When I do it, it’s because I want to find and connect with other nonbinary people and because I believe it, to some extent, normalizes the existence and presence of nonbinary people (“to some extent” because it also runs the risk of turning my uniqueness into a ‘standard’ if people see me into a reference point for nonbinary-ness instead of as a single datum in massively diverse constellation of people). But to me it’s not a solution, it’s a tactic—part of a larger, emerging strategy.

The risk is that we settle into complacency, “Okay, we’ve got pronoun-sharing now, so that solves that problem and there’s nothing else we need to do to structurally shift our harmful, rigid, binarist conceptualizations of gender that are also bound up in white supremacy, colonialism, and misogyny” (which is the direction here, the purpose, the knot being untangled). Pronoun-sharing is a useful intervention to the extent that it gets people thinking differently, creates opportunities for generative conversation, expands imaginative horizons, eases the pain of erasure and being made invisible, and demonstrates respect for a wider expression of what being human looks like. It also represents and reflects progress made so far to trouble and disrupt entrenched understandings of gender in this particular social, political, temporal context. It is also imperfect and imperfectible. Pronoun-sharing only makes sense in this given context because of where we are coming from and where we are trying to go and it comes with trade-offs and limitations. It is not a solution we should expect to scale or replicate ad infinitum. In fact, if we did we would only become committed to upholding or even creating the conditions in which pronoun-sharing makes sense—i.e., the binarism, colonialism, racism, misogyny, etc. (Which is what happens when we assume that ‘progressive’ gender concepts that are specific to white, Western, Anglophone contexts apply without modification to all other linguistic and cultural contexts.) ‘Success’ in the immediate would mean failure in service of our larger goals. And the critiques of pronoun-sharing are actually part of what makes it valuable, because these are conversations that help us move beyond the present moment into the next opportunity. When we can see and talk about what we don’t want, we can also have new inspirations about what we do. So ‘failure’ carries the seeds of success (or change).

And what kind of change? I often see incremental and transformative change presented as mutually-exclusive and oppositional approaches—slow and steady gains versus the total upheaval, the progressive versus the radical—but in practice I have only ever seen real transformative change emerge from a collection of strategically-leveraged incremental changes, climbing a rock face handhold by handhold. To go back to the yarn analogy (I think almost entirely in metaphors and puns, you’re welcome), the need to work with the yarn in whatever state it is in at any given moment is to look for what smaller shifts can give way to the release of the whole knot. The last few disentanglings may seem like the most transformative, but it was the whole process that mattered. We can also get caught up in those incremental changes in ways that work against our guiding purpose—going back and forth over the same bit of freely-moving strand because it’s easier to work with even though it’s not actually helping unravel anything else, or finding that the process of disentangling one part tightens up another part and makes the whole thing worse. (Honestly, there so much in this metaphor for me that I’m not even going to try to get at all of it in one blog post. Hopefully some of you reading this are inspired to take it and run with it.)

(Also, for another example of what it looks like to pursue change adaptively over time, with an emerging and shifting strategy that unfolds in unpredictable ways and through collective and collaborative efforts, watch this video of an event that took place in Vancouver recently, Decolonizing the City: The Future of Indigenous Planning in Vancouver, where a panel of Indigenous planners describe and discuss their work from over the past decade, what’s shifted and how and what is yet to come. Also basically any activist movement from the US Civil Rights movement to the current Hong Kong protests is a case study of adaptive, emergent strategy. What’s harder to find is explicit examples of evaluators being useful in these scenarios, though evaluative thinking abounds.)

Why does this matter?

I know we’re not always able to (or feel capable of) working in spaces that are acknowledged to be about systems change or complexity (though that doesn’t mean that we aren’t still in those spaces), but I think it’s ultimately a more hopeful, helpful frame to recognize that our formative and summative evaluations are often actually us working on just one brief segment of that bigger tangly process that happened before we got there and will keep going after we leave, and we can make our work more useful by accounting for that. In fact I think many of us already try to, if we are attending at all to the contextual factors informing the spaces we’re working in and people we’re working with, crafting questions and processes and recommendations that speak to strategic needs (having sussed these out through thoughtful discussion and exploration with various stakeholders, since they’re not necessarily the ones outlined in the 5-year strategic plan that’s been gathering dust for the last three years) and helping people take “the next elegant step” in their work, as a friend of mine says. Hopefully we’re illuminating the strategic nature of the work at the same time, and using evaluation as an opportunity for people to connect dots between micro- and macro-level shifts and see their work in context, which can be both humbling and inspiring. Keeping the whole knot in mind even while working on one part of it at a time reminds us to think in terms of context and to hold onto our definitions of “success” and “failure”, “working” and “not working” lightly.

It can also help us not fall in love with our interventions (and our designs, tactics, and even our strategies) to the detriment of our purposes. Some of the inspiration for this post came from a recent experience I had attending a presentation and watching someone totally shut down in the face of a valid critique of the intervention she was working on. She was presenting the intervention as a model practice, eager for it to be scaled and replicated, and was committed to seeing it as a ‘solution’ where any shortcomings were actually features, not bugs, so when the critiques came up, she had nowhere to go with them. (She literally ended the presentation and shooed us from the room!) But when I heard more about the reasons behind certain ‘design’ features of the intervention and the constraints they were operating within from her co-presenter, it all sounded reasonable to me—I could see what they were trying to do and why it was happening the way it was at the moment and what it might lead to in the future. I wondered if the ‘model practice’ couldn’t have been viewed instead as a useful workaround for a specific time and place, from which valuable learnings could be shared with people working on similar challenges in their own contexts. With so much invested in this particular intervention being the way it is right now though, I wonder how adaptable it will be to future opportunities, and I’m trying to hold this lesson in mind for the next time I’m tempted to fall in love with a particular change effort.

The structures we work within (looking at you, current funding models! And you, characteristics of white supremacy culture!) pressure us to come up with best practices and replicable models and scaleable results and all sorts of things that fit snugly within the success/failure, “either it works or it doesn’t work” paradigm (hm, sounds like a mix of either/or thinking, perfectionism, sense of urgency, quantity over quality, and progress is bigger and more… seriously, characteristics of white supremacy culture). We don’t always have the power to resist these pressures fully when they are structurally embedded—that’s a knot we are still untangling—but we can resist internalizing the assumptions that keep us in this narrow frame. Otherwise we end up fighting off despair with false hope while we struggle through, or redefining our goals into things so achievable as to be irrelevant and we never have to worry. Working with a systems view doesn’t have to be overwhelming, abstract, or impractical. We work with what’s in front of us, we think about what we’re ultimately trying to do, we get as real as we can about the relationship between the two, we change it up when we need to, and (outside of imperfect-but-useful tangled yarn analogies) we work with others to do it.

Gratitude

There were a lot of different sources of inspiration for this particular post and I want to name some of them. I wrote the first draft of this post right before attending the 2019 American Evaluation Association conference and was delighted to find that my conference experience ended up reflecting back, reinforcing, and deepening so many of the ideas in it, including (among MANY) the plenary with Glenda Eoyang of the HSD Institute, Khalil Bitar of EvalYouth, and Dr. Dominica McBride of Become Center, on Prioritizing What Matters In Evaluation, and a session with Dr. Donna Mertens, Dr. Katrina Bledsoe, Dr. Dominica McBride (yup, twice) and Jen LoPiccolo, and Dr. Gail Dana-Sacco, called, An Intersectional and Transformative Future for Evaluation. These sessions (and others!! but for this post these stand out particularly) dropped the mic on using evaluation to collectively disrupt, strategize, advocate, and transform in service of love and justice, which is about as big picture as I can imagine. I’ve also had my attention tuned to the need to integrate evaluation and strategy as a result of work being done by Zena Sharman and Julia Langton of Michael Smith Health Research Foundation, Jara Dean-Coffey and Jeanne Bell in their recent Non-Profit Quarterly webinar, Developing Practical Systems: A Masterclass on Linking Strategy and Evaluation, and adrienne maree brown’s incomparable Emergent Strategy. Many, many conversations with friends also shaped this post (you know who are—thank you for letting me dry-run it on you), and of course the reason I actually finished it was the encouragement from friends, old and new, at a recent Equitable Evaluation workshop, for whom the yarn metaphor will live on. 😉 Thanks, friends.

h/t to Zena Sharman for the title of this post. Alternate titles included: a good yarn, honing the ‘craft’, from wicked problems to ‘knotty’ ones, and, simply, tangled

Coda

If you’re hungry for more metaphorical forays into what it looks and feels like to do systems change in complexity, read the chapter “A Mother’s Work”, in Robin Wall Kimmerer’s Braiding Sweetgrass. And then read the rest of Braiding Sweetgrass. It’s all there.