Category: uncategorized

line-height: 1; Read More

The charitable sector does not need any more reasons NOT to measure impact, yet a Stanford Social Innovation Review article lists 10. 
I’ve just had another person send me a link to the Stanford Social Innovation Review (SSIR) article by Mary Kay Gugerty & Dean Karlan titled  https://ssir.org/articles/entry/ten_reasons_not_to_measure_impact_and_what_to_do_instead Ten Reasons Not to Measure Impact – and What to Do Instead. I thoroughly enjoyed the article, but as the comments below it state, the title is misleading. Had it, instead, been called Ten Reasons Many Organizations Should Not do Strict Randomized Controlled Studies, I would have no issue with anything in the article at all. But as it is, I hate the title.
Full-blown impact studies using controls can take months and many, many thousands of dollars. I believe that it would be wasteful for most charities in Canada to undertake such studies. As the article points out, money spent on research that doesn’t help is money wasted.
The key difference between their title and the title of this article is the definition of “measuring impact”. If we were to use their wording, “collect good monitoring data that informs progress”, as the definition of measuring impact, then my title would have been more appropriate for their article. Each of their ten reasons has to do with what I believe are mostly wasteful, overblown studies that are not required in the vast majority of cases.
It is often said that Good is the enemy of Great. In the world of charity program evaluation and reporting, I would argue that Great is the enemy of Good. Charity leaders believe that to get great data, it would cost too much time and money that they don’t have, that they would rather spend on helping their clients.
However, the crux of it, to me, is that good data is exactly what will help clients the most. Those charities that we have found to have the most impact on their clients collect good data. It may not be randomized controlled data (almost never is, nor should it be) but it is good enough that they can understand what is happening because of their programs and what happens when they change things in an attempt to improve their programs. That’s the main reason to collect data – to continually improve program outcomes and to make sure that you are using donor dollars to create as much impact as possible.
It could be argued that we are simplifying things too much, that you cannot adequately understand the impact of a charity without some sort of strict control group evaluation. But I do not believe that, and I worry that this is one of the key reasons that charities are NOT collecting the right data. It is too intimidating to get Great data.
Good data would let you see how much you spend to help someone. And Good data would then allow you to compare that to what happens to that person. Do they improve their health? How much? Do they become employed? How much more money are they making? Are they housed in a better situation than they would have been in? Did you provide them food? How much? Answers to relatively simple questions will allow the charity to understand how much difference they are making with the dollars they spend and if changes in programs are improving the outcomes. And, as Kate Ruff and Sara Olsen argue in another SSIR article,  https://ssir.org/articles/entry/next_frontier_in_social_impact_measurement The Next Frontier in Social Impact Measurement Isn’t Measurement at All, this data will also allow analysts to determine how much change the programs are creating.
We have analyzed over 200 charities, looking to determine if they are creating a lot of value with the money given to them, or if they are creating a little value. We are not splitting hairs, wondering if, for a $100 donation, the charity has produced $210 or $230 of value. That is immaterial, and it is not cost-effective to be able to understand that difference. But if you can compare two charity programs, A & B, where program A is creating between $150 and $250 worth of value per $100, and program B is creating between $300 and $400 worth of value, it’s an easy decision.
Most charities do not currently have the data to do this. Some are focused on hair-splitting, which can paralyze a charity, and most are just counting bodies. They could calculate that it cost them $1,345 to help each of their clients, but they have no idea about the associated value created by helping this client. Is it close to $1,300?  Could it be more – $2,000, perhaps?  Or are you really making change and it’s more like $5,000? With a bit of relatively inexpensive data on clients, most charities could provide data that would allow for such necessary calculations.
I applaud Gugerty & Karlan’s article, but I frown on their title. At least for those people who have, likely blindly, sent it on to me, it perpetuates the notion that there are so many reasons not to do impact evaluation that we may as well just forget it. We don’t need any more reasons not to measure impact.

David Common asked today why I rant on about financial transparency and audited statements. It is food for thought. My colleagues tease me about this obsession. Who takes the time to read audited financial statements? And does it really matter? With gaps between headlines and facts, audited financial statements matter.
Audited financial statements are the fine print. Financial statements show a charity’s “ingredients”. Like the current fuss over “ginger” “ale”, while the label says one thing, the list of ingredients shows how much ginger is in the can.
Here’s a good example. The few donors who care about their giving may have read Canadian Red Cross’ 2-year donor report on Fort McMurray released in May 2018.
More will likely be reassured by  https://www.cbc.ca/news/canada/edmonton/red-cross-still-helping-people-in-need-2-years-after-fort-mcmurray-wildfire-1.4643954 news headlines like CBC’s: “89.5 per cent of the $325 million Canadians and governments donated has been spent, says Red Cross.”

Canadian Red Cross’s audited financial statement report that $254 million was spent – 78% of total funds received. The difference is in the fine print. Canadian Red Cross reported it had spent and committed $291 million. Charity Intelligence highlighted the https://www.charityintelligence.ca/images/Not-so-fast-_-Ci-1-year-donor-report-on-Fort-McMurray.pdf  twist of “spent and committed” in May 2017.
$291 million vs. $254 million. Is 12% material? Does $37 million matter? That’s a decision a donor needs to personally assess. It matters to me. It likely matters to the people and charities in Fort McMurray. This was why Canadians gave in the first place. And $37 million is a lot of money, especially in the charity sector.
Maybe it also matters to Canadian Red Cross and that is why it headlined with the bigger number. Donors want fast disaster responses. Those affected by disasters also want fast support. Speed matters in disaster responses.
“Another $50 million went to community groups such as shelters and food banks”, CBC reports. Wrong. In June 2016 Canadian Red Cross promised to share $50 million with Fort McMurray’s local charities and organizations. The audited statements show that in Year 1 Canadian Red Cross shared $8.2 million and, in Year 2, $12.2 million. So far, $20.4 million in support for community groups – only 40% of the promised $50 million.
As to shelters and food banks getting support? Wood Buffalo Food Bank has received three significant grants of more than $1.4 million. In contrast, Fort McMurray’s shelters have received chump change.  http://www.redcross.ca/how-we-help/current-emergency-responses/alberta-fires/alberta-wildfires-community-organization-partnership-program/current-partnerships Canadian Red Cross names the local groups receiving money, not the dollar amounts as United Ways report. 
This is sad. Many of Fort Mac’s local charities are still hurting and struggling after the fire. These https://www.charityintelligence.ca/news-and-views/ci-articles?id=236  local frontline charities needed funding over the last two years, not three years from now. 
CBC goes on to report “Our team is still very busy in Fort McMurray,” McManus said. “We are meeting one on one with families and individuals in the community. And what we are seeing is that people’s needs are emerging over time.
Total spending on helping individuals and families was $12.2 million over the entire year ending March 2018. From the donor report released earlier, Charity Intelligence reported $30 million was spent and committed to help individuals and families. Again, it turns out that most of this $30 million is commitments for future periods rather than actual cash spent in the last year.
Only spending $12.2 million over Year 2 is unsettling. At the start of Year 2, Dana Woodworth, team leader of Wood Buffalo Recovery Committee said, “We are in for the toughest period of the recovery so far.” Year 2 should have been a year of larger spending.
Canadian Red Cross is seeing and monitoring needs. It has a hard decision. Does it spend now in Year 3 to meet needs, or hold back to meet future emerging needs? Evidence-based research funded by the Gates Foundation shows that meeting needs within the first two years is more effective in getting people back on their feet after a disaster. Left neglected, needs compound into other issues like mental health, domestic abuse, and foreclosure. Does a dollar spent in the first two years have greater impact than a dollar spent in Year 5? Most likely.
Canadian Red Cross has $71.4 million in funds remaining. It has signed contracts and commitments to spend $37.9 million over Year 3 through Year 5. It holds $33.5 million in “uncommitted” funds.
Canadian Red Cross’ audited financial statements show it plans to spend $53.7 million in Year 3. For comparison, it budgeted to spend $55.0 million in Year 2. It missed this estimate by a wide mark, only actually spending $31.3 million. Let’s hope Year 3 gets back on track. 
Journalists are not at fault. Journalists can only go by what the charity reports. The audited financial statements are not posted until months after the disaster anniversary, and without fanfare. It takes time and expertise to crunch the numbers. Charity Intelligence’s analysts focus on financials and charity results. We troll websites. It’s all we do.
If donor reports and the audited financial statements both showed the same numbers, one wouldn’t need to read audited financial statements. Yet when donor reports lack full disclosure or only highlight stories and snippets, the audited financial statements are essential to see the full picture.
Until we get to that glorious age of “reconciliation” with full and frank disclosure, I’ll keep campaigning for financial transparency and audited financial statements. Sorry, David.
 
Sources:
David Thurton, “ https://www.cbc.ca/news/canada/edmonton/red-cross-still-helping-people-in-need-2-years-after-fort-mcmurray-wildfire-1.4643954 Red Cross still helping people in need 2 years after Fort McMurray wildfire: 89.5% of the $325 million Canadians and government donated has been spent, says Red Cross”, CBC Edmonton, May 2, 2018 
Canadian Red Cross  http://www.redcross.ca/crc/documents/About%20us/About-the-Canadian-Red-Cross/CRC-statement-alberta-fires-fund-2018-03-31-EN.pdf Alberta Fires Fund Audited Financial Statements, March 2018 
Canadian Red Cross, 2016 Alberta Fires  http://www.redcross.ca/crc/documents/Where-We-%20Work/Canada/Alberta/AlbertaFiresInfo/ABFires_2Years_en.pdf Two Year Donor Update
Food bank again, not shelters – https://www.charityintelligence.ca/news-and-views/ci-articles?id=251  List of Fort McMurray local organizations supported in Year 2
Dana Woodworth’s comments reported by Cullen Bird,  http://www.fortmcmurraytoday.com/2017/04/05/spring-rebuild-will-be-tough-recovery-committee-told “Spring rebuild will be tough, Recovery Committee told”, Fort McMurray Today, April 5, 2017 
Canadian Red Cross  http://www.redcross.ca/crc/documents/About%20us/About-the-Canadian-Red-Cross/CRC-audited-statement-signed-2018-03-31-EN.pdf 2018 audited financial statements, Financial Note 8: Deferred contributions for Alberta fires fund. In accounting, “current” is distinct from “long-term”. Current refers to the next 12 months from the balance sheet date. CRC’s balance sheet date is March 31.
 
More background from Charity Intelligence
https://www.charityintelligence.ca/images/Not-so-fast-_-Ci-1-year-donor-report-on-Fort-McMurray.pdf Not so fast: Fort McMurray recovery (May 2017)
https://www.charityintelligence.ca/research/charity-profiles?id=251 2 year update on Fort McMurray recovery (May 2018)
https://www.charityintelligence.ca/news-and-views/ci-articles?id=206 Canadian Red Cross promises $40 million to Fort Mac local charities (June 2016)
 

 
If you find Charity Intelligence’s research useful in your giving, please consider donating to support our work. Being entirely funded by donors like you maintains our independence and objectivity to help Canadians be informed in their giving. Canadians donate over $17 billion each year. This giving could achieve tremendous results. We hope Charity Intelligence’s research helps Canadians give better.
Legal disclaimer:
The information in this report was prepared by Charity Intelligence Canada and its independent analysts from publicly-available information. Charity Intelligence and its analysts have made endeavours to ensure that the data in this report is accurate and complete but accepts no liability.
The views and opinions expressed are to inform donors in matters of public interest. Views and opinions are not intended to malign any religion, ethnic group, organization, individual or anyone or anything. Any dispute arising from your use of this website or viewing the material hereon shall be governed by the laws of the Province of Ontario, without regard to any conflict of law provisions.