This is an Eval Central archive copy, find the original at depictdatastudio.com.
I recently had the opportunity to be a guest on James Pann’s YouTube show to talk about reports. James is an Associate Professor, program evaluator, and psychologist who specializes in evaluating health, human service, and educational programs.
Watch Our Conversation
Here’s a recap of our conversation:
My Introduction to Dusty Shelf Reports
I recently had my 7th year anniversary of working for myself. I started out as an evaluator and I always loved the data viz side of my job. Some people love doing logic models or lit reviews, me I love poring through spreadsheets trying to figure out the pattern.
Early on in my career, I was throwing all my effort into a report, staying late, working on the weekends, doing whatever it took to do my best work. My boss sat me down and kindly said, “Ann, you’re a great team member and I’m so glad to have you. I’ve noticed you’re working really late on this evaluation report. It matters, sure, but I don’t want you to sacrifice your personal life for a dusty shelf report.”
I’d never heard that term before and I instantly felt crushed, but also thinking, “there’s got to a better way- this can’t be the standard”.
Reasons Evaluators Should Be Learning About Data Visualization and Reporting
If you’re working in academia or are a student in grad school students, then you absolutely need to write academic reports. But if you go outside of academia, there are many other workplace settings that have their own standards. The bar is being raised so much (in a good way) in thinking beyond the report. Can we also have interactive dashboards? Infographics? Amazing things are being done.
The tricky part is that none of us were trained for how you go from really technical reports in academic settings to non-technical reports, non-technical audiences and really quick turnaround times.
Resources Someone Just Starting Out in Evaluation (and Others) Can Use to Get Proficient in Data Visualization
For a student just getting started or if you’re new to the evaluation field, you can take one of my courses (hint, hint, nudge, nudge).
Soar Beyond the Dusty Shelf Report is our free mini-course where you’ll gain practical techniques that you can apply immediately to improve your visuals.
For reporting the best place to start would be to do a self-assessment. Ask yourself: what is the impact of your reports (if any).
There are four red flags to really look for with reports (or slideshows or dashboards, etc.). Look out for these:
- No Response
- No response at all after you send it out.
- Promise to Follow-Up Later
- You get a response that says, “Thanks, I’ll let you know if I have any questions.”
- You get a response that says, “Thanks, we can tell that a really technical team worked on this report. It’s very detailed and thorough.”
- Won’t Read It
- They ask for another format entirely like a slideshow, one-pager, etc.
Creating a Dataviz Wall of Fame
One thing I did for myself early one that was so helpful was to create a data viz wall of fame. I used to look for examples of great graphs, dashboards, infographics, etc. and would print them out and plaster them on the wall above my desk.
Nowadays, you could have a Pinterest board, folder on your computer or favorite them on Twitter. Start by actively collecting examples that you love but then push yourself to the next level by thinking, “What’s the rubric for this? Why is it good- is it the writing, the type, etc.”. Once you can identify why it’s great, you can start to replicate it.
There’s a lot of great podcasts out there too. The ones that come to mind are:
- Data Viz Today with Alli Torban
- Data Stories with Enrico Bertini and Moritz Stefaner
- Data + Love with Zack Bowders
- PolicyViz with Jon Schwabish
Techniques for Engaging Project Stakeholders in the Reporting Process
I co-authored an article on using Data Placemats, which is a technique you can use to engage project stakeholders in the reporting process.
I want you to step away from the idea of waiting until the end of the project to give the completed report/slideshow and hope that it gets used. Instead there’s a three-step process you should follow:
- Develop data placemats with preliminary findings- traditional format.
- Make a data placemat. Use Word or PowerPoint and lay out one graph per finding.
- Meet with stakeholders and have them explain their findings in their own words; you’re a facilitator, not a lecturer.
- Hold a meeting with the stakeholders where you pose discussion questions and then sit back and listen. Walk them through how to walk through the data and have some of the ‘lightbulb’ moments themselves.
- Write the final report- storytelling graphs- using their interpretations.
- Take all that data you collected from them and their insights and put that into the report.
Easy Fixes for Reports
Here are the most common mistakes–and easy fixes–that I see.
Going Beyond the Report
You only have a report. A report can’t meet every audiences’ need. You need one product per audience.
Structure Your Report
Not starting with the “So What?” The most important information is buried in the report. Put the “So What’ on the first page and get your audience in the game right away. Don’t risk losing your audience.
Not following brand guidelines. Use your fonts and colors, especially think about using your recipients brand guidelines. It’ll take you about 10 minutes but shows that you’re willing to go the extra mile and it looks more professional.
Design Each Page
Not aiming for 1+ visual per page. I define visuals very broadly. You might use the logos of organizations, or include a timeline to show the numerous steps taken.
Forgetting to lower the reading level. The average American reading level is 6th through 8th grade. Plug your text into a readability level checker and make sure your reading level matches your audience. Aim to be about two levels below your audience’s educational level. For example, if most of your audience members have completed Master’s degrees, then aim for two levels below that (high school).
How to Obtain Feedback on Your Work
When you blog and YouTube like I do where your work is front facing, people will tell you! I remember posting my first YouTube video and the first comment was from an anonymous person who wrote in all caps, saying they couldn’t hear anything over the clicking of my keyboard and said it was such terrible quality. I didn’t even know people used a microphone! I watched YouTube videos on mic and lighting comparisons to get to my setup now.
I did a webinar a week or two ago and my phone number is on my website (I forward my business phone number to my cell phone). This guy called and said, “I paid for your webinar you did yesterday and I have one problem with it”. I instantly went into flight or fight mode wondering what he was going to say. He went on to say how most presenters just read off a script but you didn’t, it was one of the best he’d seen. People will just call and will tell you very openly how they feel about your work so you have to have a poker face.
I’m also in a few masterminds with some other business owners where we act as each other’s sounding boards and brain trusts. I’ll throw out ideas to them and they’ll flat out tell me that it’s not a great idea. It’s really helpful to have a peer group that you trust.
Collect Feedback After Trainings and Courses
When I do trainings, I collect feedback forms or do an internet survey. In my larger projects, I build it in to have a debrief. They turn into problem solving for when hiccups may happen in the future. In shorter projects, I look for repeat clients or referrals. If I never hear from them again, that would worry me.
At the end of my courses, student receive a course survey. I also have behind the scene data for course students that let me know how much of a course they’ve completed, how often they’re logging in, if they’re watching all of a video or not and more. I can see which videos people are re-watching or even which sections they’re re-watching which tells me where they’re getting stuck.
Inspiration for Developing Blog Posts
I usually write blog posts on questions that workshop or course participants are asking. If everybody is struggling on how do I make sure my colors are accessible, that’s a great blog post. I also do a lot of before/after makeovers which help people see all the little steps that can make a big difference. I try to show the behind the scenes thought process so that people can then replicate it in their work.
More recently I’ve been doing interviews to try and get to know the people in data viz better and then I turn that interview into a blog post.
Finding a topic that you personally care about is key. I don’t think blogging is supposed to feel like work. You could monetize your blog but it would take a lot to make a full time living from a blog, but it’s mostly a hobby and supposed to be fun. I love the idea of thinking about what’s interesting to you and what do you want to explore more.
I have gifted all of my siblings-in-laws, Your Money or Your Life by Vicki Robin. She published it back in the 90s for the first time and just did a re-do a couple of years ago. It’s a financial book kind of, but it’s more about thinking about your life energy. You’re only alive for so long and you have a limited number of days, how can you make the best use of your time? It’s about trying to avoid just consuming things that give you temporary happiness, but thinking about how do you create for long-term, life satisfaction?
My husband I listened to the audio version during a long car drive and kept pausing it to talk about the insights we were having. She says to not throw away your life in a job that’s only decent, you need to aim for a great job and set up. Or maybe you should stay home and be a parent. This is what inspired us to have my husband quit his job ad have our family join me on the road while I traveled teaching. This book clearly changed our life trajectory.
Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones by James Clear is so actionable for everybody in every industry. I think everyone can learn something from it.
James: Steve Kotler’s stuff on flow like the Rise of Superman: Decoding the Science of Ultimate Human Performance. Also, Bernardo Castro has a philosophy book on idealism vs materialism. It’s not some academic nonsense, but it’s very practical. I think he’s one of the most important writers right now.
How to Structure Your Day/Week to Enhance Productivity
That’s a tricky one because I think there’s perception that Ann Emery must wake up at 5 am, run 10 miles, then make a green smoothie and work, work, work until midnight. She never sleeps and must be miserable, clearly. In reality, I get a lot done but I also rest and have fun. I don’t see those as opposing dichotomous topics at all.
I think you can have this wonderfully fulfilling personal life and achieve a lot professionally at the same time. I’m into minimalism in terms of thinking about what really matters in your work and personal life (which ideally are the same thing!). But really thinking about what matters and let go of everything else.
My Husband is a Stay-At-Home Dad
My husband let go of his very good salaried job with a pension and top-secret security clearance. We let that go so we no longer have to do things like both waking up with alarm clocks, rushing the kids off to daycare, both working and then figuring out who makes dinner, etc. We now don’t set alarm clocks and get enough rest each day.
We Bought a House with a Quiet Office
Now if I have to set an alarm, I think to myself, “My life is out of control! Why did I schedule something before 10 am?” We purposefully stopped fulltime travel due to COVID and so when house hunting in Florida, we were looking for a very specific home. We wanted a one level house with a bonus space I could use as an office. It’s my little retreat- I go to my office, get all my work done and then I go downstairs and go ‘home’.
Be Smart About Scheduling
I used Calendly for scheduling meetings and I’m only available from Mondays at noon until Thursday. I don’t book anything on Monday morning, that’s a terrible way to start the week! And I don’t boo anything on Fridays, because if my brain is tired, I’m not going to be any good on a Zoom call. Fridays are usually a day for myself to write a blog post or record videos for online courses. It feels really good to end the week like that, doing something creative for yourself.
Creator vs. Manager/Delegator
Identifying whether you’re a creator or more of a manager/delegator and making sure you’re not trying to go back and forth. Instead of trying to spend one hour writing a blog post (creating), then a 30-minute meeting about a project and then spend one more hour writing that blog post. I try to really separate it and have a half day of uninterrupted to do the actual, creative work. Then I try to book all my meetings, back to back at another time.
Morning, Afternoon or Late-Night Person
I’ve listened to whole podcast episodes about being careful to know if you’re a morning, afternoon or late-night person. I used to be a late-night person and would do my best work at 9 pm-midnight. But now I’m a morning person so making sure that my mornings are my uninterrupted time. I rarely look at emails and instead am building courses and writing. In the afternoon, I take calls. You really need 100% of your brain energy for the creative work.
James: I think it’s great to get good sleep, that’s probably the first and most important thing. Getting that honed in has been huge.
And then being okay with taking breaks during the course of the day and actually building it in. Making sure that you’re going through a cycle and not just powering through.
Subscribe to James Pann’s YouTube show.
Comment below: Which reporting tips were new to you? Which tips are you going to put into practice? Have reporting tips of your own to share?