Policy in Practice: Addressing Poverty with Behavioral Economics

Although the global rate of extreme poverty is at a historic low, the pace of poverty reduction is slowing and the World Bank estimates that more than 700 million people still live on less than $1.90 a day. The 2019 Robert B. Menschel Economics Symposium discusses the ways behavioral economics can inform development policy to create effective solutions to poverty at the international, national, and local levels.

BESCHLOSS: Good afternoon. Welcome to the second session of the CFR Robert Menschel Economics Symposium. We heard in the earlier session a lot of very interesting things about both theory and some policy, and what we’re going to talk about in the second session is looking at policy but specifically policy in practice and how it’s implemented, and taking advantage of the recent research on behavioral economics and how it impacts poverty-reduction programs.

My name is Afsaneh Beschloss, and I’m founder and chief executive officer of Rock Creek.

I’m very, very excited to introduce our great panel. Of course, they do not need an introduction.

But starting with Varun Gauri, who is at the World Bank. As you know, he is a senior economist at the bank and co-head of the Mind, Behavior, and Development Unit there—and the author of the great report that I’m sure he’ll be talking about.

Liz Hardy is the senior lead of behavioral insights, and she’s at the Impact and Innovation Unit for the Privy Council Office at the government of Canada. And she’s going to be sharing her thoughts about her experience in terms of policy and practice in Canada with their population.

Matthew Klein is the executive director of the New York City Mayor’s Office for Economic Opportunity, and also has been both designing policy and implementing it.

And very, very excited to have all of you here today. I was eager to hear from each of you, if you don’t mind, very, very briefly if possible, what you’re doing today. And maybe we start with you, Matthew, what you’re doing today that is relevant to the topic.

KLEIN: Sure. So very briefly, the Mayor’s Office for Economic Opportunity helps the City of New York use tools of evidence and innovation to address issues of poverty in the city. So that takes a number of different forms. We have an innovation fund where we start and test new models. We use evaluation on broader-scale policies. We integrate data from across agencies to try and help the city more holistically serve low-income New Yorkers.

About two years ago we, in partnership with the nonprofit firm ideas42, launched a behavioral design team, a team that—of behavioral scientists who are essentially embedded within the city, and who we then assign to different projects across the city, with a particular focus on issues of poverty and equity but also some other operational questions. So that team helps us design a variety of interventions or often small nudges to help existing programs work better. Happy to talk about those as the panel goes on.

HARDY: Great. And hi. I’m Liz Hardy. I work at the Impact and Innovation Unit, where I manage the behavioral science team there. We run randomized controlled trials, small- to medium-size experiments testing to see what works using behavioral insights and behavioral science principles.

We work in a variety of domains—public policy domains across Canada. That includes diversity and gender issues, financial inclusion, and also environmental issues. And we do, as well, have a focus on poverty reduction.

GAURI: So I co-lead the Mind, Behavior, and Development Unit—eMBeD, we call ourselves—at the World Bank. We’re the behavioral science team at the World Bank. We work in sixty-plus countries across all of the sectors. Our mandates are to improve development effectiveness by integrating behavioral science into development projects, World Bank projects as well as government programs and other partner programs, to institutionalize behavioral science in governments. We spend a lot of time building capacity and working with partners. And we also provide public goods like research and training. The team grew out of the World Development Report 2015: Mind, Society, and Behavior, in which we tried to summarize what behavioral science and behavioral economics means, and the implications of those fields for development altogether.

BESCHLOSS: The field we’re discussing, the use of behavioral economics in—using it to reduce poverty, is a relatively new area, and some would say risky. I’m just curious, when did you all decide to get into this area? And maybe, Matthew, we’ll start with you because you were a lawyer, I think, to start with.

KLEIN: Well, I went to law school. (Laughter.)

BESCHLOSS: OK.

KLEIN: So one of—we were attracted to behavioral science for a number of different reasons. I mean, one is that a sort of a core element of it or a core promise is that small changes can have a meaningfully significant impact at a very low cost. So large scale, low cost, very attractive to government. And we’d seen sort of proof of concept of this, at least within the U.S., out of the White House. Under Obama there was a—and that, of course, built on work that was happening in the U.K. And these folks have a better sense of the international context.

But we wanted to do that at the city level because everything is already designed. There’s already—you know, a notice is already designed, an intervention, a program, and government is about trying to get people to do the things that we as a society want them to do. And since everything is already designed, if we can change it or tweak it using the insights that come from behavioral science and get a better result at a low cost, a very attractive proposition.

BESCHLOSS: Great. Liz?

HARDY: Sure. Prior to joining the federal government I set up the behavioral insights team in the Ontario government at the provincial level, where we ran a series of randomized controlled trials testing to see what works in a variety of areas. The focus at the time was mostly on cost efficiency. Like, as you were saying, we can make small—somewhat small tweaks to the way in which we deliver services and programs and see really remarkable returns.

And I think for me the interest is really around, one, problem solving. So how can we use innovative approaches to tackle some of our, you know, big societal problems? But also now at the federal level looking at how multiple levels of government can work together to tackle some of these big challenges that we’re facing, where we have a policy or program that may be owned at the federal level but where the touchpoints and the access to clients perhaps is at the provincial level, and then maybe even at the city level.

So I think we’re getting into a really exciting time with behavioral science. We’re starting to see these kinds of groups and units working together.

I also think what’s really quite interesting to me right now is this notion of, yes, we’re working on problems—our existing programs and service that aren’t working the way that we hope that they will work, and we want to improve them. But what about new policies and programs? How can we incorporate behavioral science in the way people make decisions in the real world to design that policy or program more effectively from the beginning? Instead of fixing, OK, this is already broken, let’s start thinking about how we incorporate these insights earlier on in the policy development process.

GAURI: So I was recently thinking back over my career and realized that for many years I’ve been doing behavioral economics without knowing it, kind of like that Molière character who’s told you’re speaking prose. Oh, thank you very much. I didn’t know I speak prose.

My dissertation was on education reform in Chile, which has and I think still has one of the largest school voucher-like programs, closest to Milton Friedman’s proposal of any country in the world. And one of the things that I found back then is that a lot of parents weren’t taking advantage of the choices available to them. Looking back now we would say that sort of was like a bandwidth/attention scarcity problem, although the vocabulary didn’t exist at the time.

When I came to the World Bank, one of my first assignments was to work on the HIV/AIDS program in Brazil, which is one of the world’s most successful programs on HIV/AIDS. But there were issues of stigma and there is—and there is the problem that people were getting medicines but not using them. And now in behavioral science, behavioral economics we spend a lot of time talking about the take-up of problem—you know, the take-up and adherence—drug adherence. And so, I mean, these problems were surfacing, but we didn’t have the tools and the vocabulary to really address them. And now we really do.

And I think—I mean, sometimes people ask, so what’s new about all this? And I think what’s really new is that not only do we have the language, but we have a lot of research that gives us actually estimates of the impact, you know? So you can put a magnitude on the coefficients. So if we send reminder letters to people to pay their taxes, that’s probably going to give you a 5 to 15 percent bump, right, in tax payment. We sort of more or less know those numbers, and that’s what we didn’t have when I began my career. If we addressed scarcity, we didn’t know what that would really do or what impact, you know—how much could we move—improve drug adherence. So I feel like I’ve been working on it for a long time, but now I’ve come back to it in a new way.

BESCHLOSS: I’m curious, Liz, as you were talking about program design and then implementation and how that sort of proceeds, and then what you learned from the implementation and how do you put that back into design. Can you talk a little bit more about that? I’d actually be curious to hear from everyone on that. And then what you learned in that sort of loop, and how can you use the inputs.

HARDY: Sure. I think that’s an important question and one that our keynote kind of addressed, too, is the importance of implementation.

So, as I mentioned, we do research where we test to see what works. So we’ll run a randomized controlled trial. I’ll take an example. In the Ontario government we looked at organ donor registration, so how can we encourage people to register to be organ donors. We know that in the province of Ontario a large majority of Ontarians, when asked, about 85 percent say yes, definitely, I really want to register. But yet, registration rates are around 24 to 25 percent. So it sort of begs the question, like, why is that happening? Why is there this gap between our intention and then actually following through?

So we did some research around that area, tested a number of interventions. Some may seem very simple, like a simple form design. We also tested what we call sort of under the bucket of timing. So timing has a big impact on whether people follow through with the decision. So if you ask them in the right moment or the wrong moment, you may actually change the decision that they’re—that’s made. Regardless, we ran the trial, and the most successful intervention increased organ donor registration rates by 144 percent. It was beyond what we could ever even imagine.

But the implementation, in my mind at the time I thought that would just sort of roll out. We’ve proved it. It’s a 144 percent increase. Let’s just roll this out. But the implementation became challenging.

So I think one thing I learned earlier on this field is that there is certainly—there is the importance of the research and the testing, but then we have to make sure as practitioners we’re following through with that implementation to make sure, one, it happens, but also that it happens in the way that the research sort of tells us it happens, that it doesn’t get changed along the way. And I think part of that is capacity, and I think it was well said in the previous session around, you know, we’re sort of all kind of taxed, and government and public servants have a lot on their plate, so we need to make sure as behavioral scientists we’re there to help them through that final phase.

The other thing I’ll say on this is sometimes things don’t work. And in situations where we do test things and we find a null result or sometimes things actually do worse, which I could—(laughs)—talk a bit about if that’s of interest, then we need to not be afraid and not hesitate to go back to the drawing board. It doesn’t mean that, you know, we shouldn’t continue to test; it’s just means that we need to think maybe what we’re testing just wasn’t the right thing, or we need to maybe do a bit more qualitative research, a bit more what we call behavioral mapping, figure out sort of what are the steps sort of to the process and how can we maybe rethink the way they were designed.

KLEIN: I’d echo all of that. I mean, I think you were making a very interesting point in terms of how can behavioral science shape the creation of a policy.

I think we’re still at the early stages of that, at least in New York City in the work that we’ve done. Lots have been small tweaks to existing practices: redesigning the letter so that people—you know, increasing uptake. An example of that, SNAP is the program that provides nutritional assistance to low-income New Yorkers, and folks have to recertify periodically. And there’s often a churn where people just don’t follow through on the recertification process, and so they come off and then they need to reapply, and it’s cumbersome for them and it’s—it takes resources from the city to process that. So just implementing various nudges—emails, texts, letters, redesigned letters—to increase—

BESCHLOSS: Simplification, as you’ve written.

KLEIN: Yeah. And so there’s lots of examples of that.

That by itself is useful. And some of the ancillary benefits to government is we’re now rigorously assessing what happens—like, what is the percentage uptake, as you were saying. It introduces a culture of, like, more constant improvement, iteration, things that aren’t just set and forget and go on. And acceptance of failure. So, you know, often in the public-sector context there’s a real incentive not to fail because you get the bad headlines, whereas when you can do lots of these little experiments and you expect that some percentage of them are going to have a null effect there’s more of an embrace of trying things. And then, of course, you know, innovation improvements can happen. So there are all those benefits, and I see that.

But then there are the big-scale things. So if we take seriously these notions of, like, the hassle factors and the bottlenecks, and it’s a tradeoff between do you want the policy to go forward in the way that it’s intended versus potentially embracing questions of the way it’s not intended.

So take fraud, for example. We do a lot in government in safety-net processes to ensure that there is no fraud. But as a result of all those steps we probably are discouraging uptake of certain benefits, and that’s a tradeoff.

So I think as we are looking at these larger-scale questions of should there be a universal basic income—one experiment that we’re doing out of our office is supporting a child allowance for low-income mothers in New York City, which is unconditional cash—those have low friction, but embrace some of these other kind of questions around, well, will people, you know, use them for sin purposes and those kind of things. So there’s where, you know, I think both, you know, politically, but in terms of the social goals that we have we have to confront how seriously we take the lessons of behavioral economics.

BESCHLOSS: Varun?

GAURI: Yeah. So we take this, you know, test, learn, adapt model very seriously, at least we try to whenever we can in our work. For example, we did some work in Peru using growth mindset. Some of you may know this. This is this idea that in schools kids sometimes don’t work hard because they think they’re not smart. Growth mindset’s the idea that, you know, brains adapt. You know, you don’t say I can’t do this; you say I can’t do this yet.

So we did a two-hour lesson—you know, with the government did a two-hour lesson with two hundred and fifty thousand kids in Peru. This two-hour lesson was randomly applied. Among kids who got the lesson, they took a standardized test a couple months later and they tested about .1 standard deviations higher compared to people who didn’t do this two-hour lesson. So that’s about three or four months of learning; .1 standard deviation is a lot on one of these standardized tests. It was, you know, twenty cents per child, a very powerful intervention.

But, you know, then we thought, well, how do you deliver this—so when we—this was mailed out in Peru. The compliance rates are about 60 or 70 percent because it got lost in the mail or schools didn’t apply it or whatever. So how can you—you know, if you know it works, then how can you adapt it so that it reaches more people?

So we in South Africa are working with the government of the Western Cape to put it on YouTube. So that’s going to be the delivery channel. In Indonesia we’re working to do it—use comic books with another—with another two hundred and fifty thousand kids. So you have something that works, but then you sort of, like, adapt it to the context to try and make it more effective.

There are a variety of political challenges in any one place because I think, you know, public officials often want—they’re committed to a program, this is my program. (Laughs.) It’s hard to sort of—to shift gears. But if you keep getting the examples out there you hope that, you know, they’ll be taken up at some point down the road.

BESCHLOSS: Looking at sort of some of the ways that these programs affect the financial sector, could you give us some examples of what’s worked and what’s not worked in your experience?

GAURI: In the financial sector?

BESCHLOSS: Financial sector.

GAURI: Yeah. So we’ve done some work in Tanzania trying to get people to save. You know, there’s—mobile money is everywhere these days, or almost—you know, it’s increasing. And that’s good, and it can be dangerous if poor people are then subject to loan opportunities at very high interest rates. So you sort of want to encourage people to save, but then discourage the sort of—you know, the Phishing for Fools, right? That’s sort of—there’s a book out there on this topic, you know, where people go looking for people who are willing to sort of—you know, to sign up for some of these things that aren’t in their own interest.

But in Tanzania we sort of sent out messages to encourage the savings side on phones and found that these little reminder messages, certain ones of them—the social norms message increased savings by about 11 percent on relative to not getting a message. Some of the other ones, actually one about agency that said you can redo your life, that one actually—it lowered savings compared to the control. So we are not exactly sure why.

I mean, I think one of the challenges in the field is that—you know, the main takeaway from behavioral economics is that decision-making is very contextual. You know, you just can’t sort of say I raised the price, I’m done, like in the old days in economics. You’ve got to look at the local context. But the context is complicated. It varies and changes over time, even in the same place.

BESCHLOSS: Thank you. There are a lot of ethical dimensions to what we’re talking about, and I was wondering, Liz, if you have any good examples of ethical issues that you have seen as you’ve designed programs in your experience.

HARDY: Sure. And I think the first thing I’ll say about that is, obviously, ethical considerations are at the forefront of everything that we do. Whenever we’re doing randomized controlled trials, we make sure that we’re taking all the necessary sort of due diligence in planning them out.

I would say that in the government we roll out programs and services all the time without testing them. And I think in this case we’re testing things mostly with the broad population. So we very—in very few cases—I actually can’t even think of any particular case where we’ve isolated a vulnerable population, so most of the time we’re dealing with a large population.

One thing I will say that came up was there was a trial that we were doing around the Canada Learning Bond, which I’ll probably end up talking a bit more about because it relates to the topic today. But essentially, the Canada Learning Bond is an amount of money—free money, essentially—that is given to low-income families in Canada to help them save for postsecondary education. So the policy objective here is we want parents to save for postsecondary education, known research showing that, obviously, postsecondary education is a huge deterrent—determinant of success later in life, so let’s encourage these families to save. So the government decides, OK, we’ll give families this incentive. We’ll give them this amount of money and they’ll start saving. All they have to do is open up this really simple account, this RESP it’s called in Canada.

But it turns out 1.8 million kids who are eligible in Canada are still not receiving the bond. Now, why would that be? It seems very obvious; there’s this free money on the table. Well, that’s because the account is complicated to set up. You have to get a social insurance number, which is something that we have in Canada where you have to go in person to locations across the country, stand in line, and get one of these numbers for yourself and your child. You then have to choose a financial institution, a credit union, or a bank. You then have to go make an appointment, show up in person, fill out paperwork. You have to sign. And then you have to then apply for this bond, and then you get it. So, anyway, obviously, some friction points there.

But with respect to this question here, there’s a case where we were looking at or a team in government was looking at changing the correspondence that we send to people to tell them that they’re eligible. And in one of the designs there was one group that was not going to get any letter at all. So, like, if you think about ten thousand kids that were going to receive this letter, we were going to have a group or they were going to have a group that doesn’t receive the letter. And I think—I can see the importance of testing that way to see whether the letter has an impact, but I think the important thing here is that the children get the letter eventually so that everybody gets access to the same information and the same content and they have the same opportunity to open the account. So what we did in that case was simply phase the letter out after the fact.

But I think certainly it’s something that we do have to take quite seriously as practitioners. I don’t know if you guys have anything to add to that.

KLEIN: Well, I think often what comes up in the behavioral science context around ethics is are you tricking people. You know, are you—is the messaging such that you’re using psychology to—in nefarious ways? And I think the response to that is the aim for the interventions that I think are most often done are ones where you’re trying to help people do the things they already want to do, not do the things that they don’t want to do. And so often the—you know, the very basic kind of psychological things, whether it’s like framing things as you’ll lose this if you don’t take advantage of it—because people, you know, feel worse about losing something than they feel good about getting something—are just trying to get people to make the decisions that they—and to follow through the decisions that they already want to make.

BESCHLOSS: And I think you may want to also sort of talk about paternalism because it’s very related to what you just said.

GAURI: Yeah. This is, obviously, one of the criticisms of behavioral economics. The thought is that, you know, if you opt people into a retirement savings account, for example, rather than saying we’re going to wait for you to sign up, you’re sort of deciding for that person that they ought to be saving for the future. And people are uncomfortable with that.

I guess my view and our view, as you know—and in our team is that they’re really like any other policy. You need to disclose what you’re doing and you need to sort of, like, deliberate upon it, right? In fact, it’s less coercive than, say, taxation, conscription, other things the government does, frankly, you know?

And the good news is that as far as we know, when you disclose to people they’re being nudged, the nudges still work. There was an interesting study on advanced directives that was published a couple years ago that sort of told people, you know, we’re going to automatically sign you up for an advanced directive in which you only want comfort care when you’re—if you’re in a terminal condition, and other people were not told this, and the people who were nudged—who were told this—the nudge still worked for them. So interesting, you know. So that’s good, I suppose, for those of us in the field. (Laughs.)

For us, because we do a fair amount of work on gender relations and social norms around gender, that’s where some of the ethical issues sometimes come up. You know, we have some work in Jordan trying to understand why more women aren’t in the workforce—only 14 percent are in the workforce—even though women are more educated than men. It has to do with social norms, of course. We try and measure social norms. And in our survey that we sort of, like, asked, we had to—we were very wary of, like, you know, what does your husband think, and what’s really appropriate to ask a wife about what her husband thinks and ask the husband about the wife thinks, sort of behave so that we don’t complicate, you know, relations in the family. So even asking a question. This is not so much a behavioral economics question, but just a general survey research ethics question that comes up quite a bit.

BESCHLOSS: Thank you. Before we go to the audience, I have one last question. In terms of sort of using tools—using cellphones, using the internet, big data—is that things that you’re integrating? And given that 50 percent of the population doesn’t have access, and obviously that’s the poorest part of the population, how does that factor into the biggest tools that you all use?

KLEIN: We certainly do incorporate it. You all probably find this, too; text messages are opened at a much higher rate than email messages are, probably higher than a, you know, bland envelope that comes into your house. A lot of—higher and higher percentage of folks—low-income folks, New Yorkers—do have devices, and often the challenge is around the data plans not the—not the device. So increasingly that opens up many new opportunities to engage with residents in different ways.

But on the data side, you know—you know, like everyone, we’re enthralled by the potential of big data and AI and machine learning. I think in our context there—we’re very, very early in the road. There’s a lot of work that has to be done to make sure not only the data is clean enough to be utilized and analyzed, but also that we can navigate important privacy and legal constraints.

But within that, you know, one can imagine going forward thinking about looking at the—you know, not only like which messages between the, you know, three that we just sent out work best, but which messages work best for whom and starting to tailor different kind of messages to folks who have different kind of profiles—which, of course, is Marketing 101. But in government, when it comes to thinking about nudging people around different services, or different courses they might want to consider as a community college student, or what kind of decisions they want to make as they are looking at job opportunities, all those kinds of things are down the road.

BESCHLOSS: Very good. Thank you.

We’re going to turn to the members now and get your questions. I think there are going to be microphones, so if you could say your name and affiliation. And just a reminder that we are on the record for this session. I think over there.

Q: Hi. Maria Fidler from Linden Global Strategies.

All of you spoke a lot about the different mechanisms which you employ in your respective fields to nudge people to act in a certain way. Have any of you ever come across or used gamification as a mechanism? And if so, in what instances, and has it worked?

BESCHLOSS: Anyone who’d like to go first?

KLEIN: We have not done any gamification. I’m aware that there’s—particularly in, like, encouraging savings, there’s tools and approaches that do gamification. You could—one example—I don’t know if you’d count it particularly gamification—but is like a lottery, where everyone saves and then one person whose save gets, like, a winnings, and that encourages lots of people to save because they might be the one who gets the winnings.

HARDY: So I myself have not done a trial with gamification. But I do know in Ontario they were working with youth in the hopes of encouraging them to be safe at work. So they were finding that young people would be hurting—were getting hurt at work, so whether that’s they work in a bakery, they were getting burned, or they’re doing some part-time work in trades and they were getting hurt. So they designed this game, basically, where youth would kind of go into these buildings and they would—a little guy would, like, do the job and then get very hurt, like dramatically injured. And I think it was supposed to be kind of funny, and then they would learn this little thing about how they were supposed to remain safe.

And I think in the end they found that there was an increased awareness of the risks, but I don’t know if they ever really were able to measure whether there was a reduction in the injury. And if they’re not setting it up in sort of a trial basis, it’s difficult to know what the actual impact was. But I think definitely they did see a greater awareness because, of course, it was kind of extreme, so the kids would talk about it and say, oh my gosh, the guy fell off the ladder and his head popped off, and it was, like, really funny. (Laughter.) So that, certainly, I think they found was successful. But in my work not so much.

Don’t know about you, Varun.

GAURI: Nothing’s coming to mind. I don’t know if this is—this is the style of gamification, but we are sort of exploring virtual reality kinds of things. Putting yourself in a position of a poor person is pretty hard for many of us, and it comes out in our own data. We’ve sort of studied World Bank staff and try to get them to predict how a poor person would behave in context, and we’re not as right as often as we think we are. And so to put yourself in that position we sort of have a—you know, just last week we sort of, like, you know, put a—you know, sort of several dozen staff in this position. Sort of we’re going to see if it makes a difference. We’re going to try and measure.

I know it’s also out there in terms of sexual harassment, you know, sort of putting yourself in the position of a woman who’s being harassed. And so that’s another potential use of that particular technology.

BESCHLOSS: OK. I was just going to add to that. I think there’s a lot also in sort of private ed tech that is starting to get built. So obviously, you know, this is sort of the younger population that gets covered. I know that PBS has also done a lot in trying to introduce games to help especially, again, the very young to deal with some of the things you asked about.

I think Mrs. Bruce. Over there. Oh, sorry. OK.

Q: Hi. Kim Davis from Charles Bank Capital.

I have a—maybe it was triggered by Matthew—a political question. How do you balance the policy prescription from behavioral economics with the public polling, which probably comes out pretty differently? For example, on the fraud and uptake issue, public polling is very sympathetic to stopping fraud and not particularly sympathetic to the uptake issue, even if the data points you in a different direction. And I’m sure this has probably happened in both the World Bank and Canada. I’m kind of curious how those conversations actually get adjudicated between the behavioral economists and the political—the political actors.

KLEIN: Well, look, on the fraud stuff, particularly around, you know, low-income benefits, we haven’t moved much. We’re strict on the fraud. We haven’t taken away any of the requirements for qualifications or anything like that.

There are—but, you know, you’d be surprised on the kind of default settings that exist. So one of the things we have done is tried to make the hassle factor less in sort of just the logistical ways. So in SNAP, for instance, the nutritional program, the default is that, as part of the application process, you have to have an interview. And the interview is that you have to be available at the four-hour bloc that the government gives you and we’re going to call you, so be home during that four-hour bloc. And as you can imagine, a lot of times you don’t reach it and now the process is delayed.

New York City applied for and received a waiver to be able to give people the opportunity to do on-demand interviews. So they call us whenever they want and we have a call center to react. That took a waiver granted to us by the state. So there are things to do to reduce the hassle factor along those lines that don’t compromise fraud prevention.

But I was trying to, I think, make a larger point that they’re probably in—if you—the more we want to move towards systems that presume eligibility or are universal and don’t care about meeting certain criteria, then it’s going to run into the political kind of concerns.

I would just—just one other example is, you know, voting fraud. There’s a big debate on, oh, is there—you know, on the one side voting fraud big problem, very little evidence that there is voting fraud, lots of evidence that people who are entitled to vote don’t vote because of intentional policies around licenses or whatever it may be—when you can vote, it’s only on Tuesday, et cetera. So there is a tradeoff where, you know, the fraud argument is made, but yet we have the millions of people who are not voting. So how do you weigh those two is the kind of thing that can come up over and over again, I think.

BESCHLOSS: There’s a question over there.

Q: Judith Bruce, the Population Council.

The populations I work with are excluded in some ways that I really think need exploration, including we’ve got examples in Tanzania in which the communities—about 60 percent of the girls will be seriously off-track and out of the range of any entitlements, if they ever were in the range of any entitlements or communication approaches, by age fifteen. Seventy-five percent of them will be single mothers for substantial periods of time; I mean, they’re children. And when I hear about the design of interventions, they often begin with an externally-defined problem or even a device, rather than with a segment.

So I’m concerned not just about gender, but about age of intervention. And I would say with girls you want to begin about age ten. And so I—any examples you have with working with very excluded populations early, and creating both access on-ramps and also figuring out what you want to measure since their lives are rolling out. Any examples of what you—of programs that you’re pursuing?

BESCHLOSS: Varun?

GAURI: So we do a lot of work with, you know, very poor populations, you know—you know, in, say, rural—(inaudible)—India on sanitation, getting people to sort of use latrines; nutrition programs in Bangladesh.

So one of the vehicles is sometimes women’s self-help groups, though those tend to skew a little bit older than, I think, you know, the groups you’re talking about. But that’s one platform. You know it’s almost a technology, really, the way to reach people. You know, a lot of NGOs have established them and you can use them to get them out.

We have been doing some work on the uptake of long-acting reversible contraceptives in Africa. This raises some of the ethical questions about, you know, providing something to a young girl. We sort of ended up focusing on married. You know, we couldn’t quite politically get to, you know, the—you know, the unmarried population, which I think you’d want to do. But sort of—you know, the ethical question of sort of, like, making it available, keeping in mind the health challenges, while also creating kind of a choice architecture where if it’s right for you you will go down and take that choice. And so there’s, like, an algorithm that, you know, we helped set up—sort of, you know—sort of a decision tree, you know, that was automated to help girls make those choices.

But I completely agree. It’s a—it’s an enormous challenge.

BESCHLOSS: Any other comments?

KLEIN: None of our behavioral science-related stuff have been targeting children, like, directly. A fair number target families with young children. So, for example, we’re trying to boost the number of New Yorkers’ children who take the gifted and talented test who come from low-income communities, and so have had an intervention around nudging that. The child allowance that I mentioned.

One other thing I just want to say, because I’m a big fan of the child allowance, is that in relation to the scarcity question that, you know, the professor was so articulate on, the benefit of unrestricted cash to reduce, you know, the cognitive scarcity has multigenerational benefits, potentially. We’re testing this in a very rigorous way. But the notion is that if I’m a parent and I’m stressed for—financially, not only do I not—am I not able to sort of provide material goods or experiences and those kinds of things for my child, but I’m probably parenting in a different way because I’m stressed, and anyone who’s stressed parents in a different way. And that can have developmental effects in terms of the stress levels of children, particularly infants and zero to three brain development and cortisol levels. So by virtue of, you know, poverty, we’re affecting child development.

And so the child allowance test that we’re a part of is going to be measuring not only sort of what is the impact on the parents and their use of funds and all of that, but what are the cortisol levels of the children, and what are the other social, emotional, and developmental effects on young people. So it’s—and that’s a randomized controlled trial.

HARDY: One thing I’ll just add to that is mentioning, again, the Canada Learning Bond, we’ve been focused for a number of years on trying to encourage parents to act. So let’s tell the parents that they need to open the account. Let’s send them a notice. Let’s bundle services together. And it sort of occurred to us at one point, what about the kids? We’ve got these kids that are—they’re eligible for this amount of money up to the point in which they go to postsecondary. So what if we reach out to them and say, hey, you’ve got this money, so it’s available to you. This is what you need to do to get it; you can get your grandparent to sign, and give them kind of options.

So right now we’re looking at designing two things. One is sending the correspondence directly to them, which legal is looking at. (Laughter.) And then the other thing is around designing a workshop. So similar to a program that was designed in Ontario called Life After High School, where basically they brought kids in and had them kind of work through the application for university and college, looking at designing a workshop where we bring the kids in and talk to them about the bond, and how do they get the bond, and how much money it would be. And also, I think really importantly, what does that money mean? So it means that your college tuition—it will be college tuition for you for the next three years. And when you put it in terms like that, it sort of starts to resonate that now, at thirteen, we’re putting it in their minds that this is possible for you; you can—you can—you can get this. You can get to college. You can get to university or go to trade school.

So I think that’s a really interesting sort of, I think, angle on the—on the issue.

Q: Elmira Bayrasli, Foreign Policy Interrupted.

On the RCTs, it’s great when all of the results come back and they’re all pretty similar. What do you do in situations when the results are completely all over the place? Do you—do you move ahead? And if you do move ahead, how do you actually know that the choice that you’ve selected is working?

HARDY: I’m trying—it’s a very good question. I’m trying to think of an example where—it’s not so much of where the results were all over the place; it’s more that something really surprising ended up working. Like, we did this trial around encouraging women to enter the armed forces, Canadian armed forces. One wants to encourage more women to apply, so we did this social media trial where we tested a bunch of different messages to see if we could encourage them based on some false assumptions of the armed forces.

So we did some qualitative research where we found that women were concerned, as an example, that they would not pass basic training; that basic training would be too challenging for them. So we went back and looked at the data and found that, in fact, that’s not true; that nine out of ten women are successful at basic training, and in many years they’re actually, in fact, more successful than men. So we designed some ads around that assumption, as an example.

And then one ad we put into the test was supposed to be our control, like was supposed to be kind of the one that didn’t work and then we would measure everything against it, and it outperformed all the others—(laughter)—by like a mile. And so we looked at the data—I said to Harris (sp), who was doing the data analysis, OK, Harris, you’ve got to run this like three more times; there’s no way that’s right. And yeah, it turned out that this very kind of benign ad that just was the badge of the armed forces performed really well, so in this case we’re running it again. So we’re going to run it again, and we’re going to unpack kind of this—the badge and what was it about it? Maybe it was that it was so different from all the other ads that it caused somebody to stop and actually engage with it. Or it could have, in fact, been that it did actually resonate with the audience, that they felt some sense of—whether it was patriotism or some sense of connection with that and it did actually work. So we’re—as I said, the answer there is retesting.

KLEIN: Mmm hmm.

BESCHLOSS: That’s really cool.

GAURI: Well, I mean, it’s a tough—it’s a challenge, right? I mean, I think—on some level, I think you’ve got to at a certain point fall back on your theory. You know, the ICT is identifying certain mechanisms, implementation challenges, this or that, but meanwhile you’ve got a theory that was guiding some of the work and that’s going to be—well, sort of, like, guide some of your choices going forward.

So, I mean—so what comes up for us in the context of some—a lot of these tax letters, you know, sort of little—nudging people to pay taxes, and the original work in the U.K. found that the social norms—you know, if you tell people, nine out of ten people in the U.K. pay their taxes, you get a five to ten percent increase in the U.K. of people paying their taxes.

We did it in Guatemala—one of the lowest ratios of tax revenues to GDP in the world, it worked there, and the social norms letter was the most effective, just like the U.K. Poland, when we did it, it worked but the social norms was not as effective as the hard tone letter, you know? Sort of, like, we’re watching you now, and if you do it again—

BESCHLOSS: (Laughs.)

GAURI: —you know, so we can wave our hand and speculate about why in polish society that could be different, but we don’t quite know. We just published some results in Kosovo. It didn’t quite work, you know, also, you know, that much. You know, Latvia sort of mixed.

So it’s a little bit all over the place, but you know, I—we still believe that—from the theory that in some context social norms are going to work, right? Social comparisons are going to work. And we know that it’s not only from the tax letters, but things like all of these little letters that you all get about your utility—your electricity consumption, you know? Like I’m below similar houses; I’m above similar houses, you know. And a little smiley face if you’re below and a frowny face if you’re above. Like that’s been done with millions and millions of consumers, and you get a two to five percent drop in utility—from those letters and that’s—so those social comparisons work.

Why not in Poland? You know, we need to sort of, like, tease that out, you know? But we’re still—our theory is still guiding us in terms of—we’re not abandoning it altogether going forward, but we need to do a more careful investigation in particular context about how it’s being interpreted or whether it’s a delivery problem or something else.

Q: Hi, I’m Cillian Nolan, and I work at JPAL with Abhijit from the first session.

And I had a question about your take-up of the research that you’re doing and implementation. Liz, you mentioned on—I think it was the organ donation study—sort of thinking that the study would implement—the findings would sort of implement themselves, which, I think, is a very familiar kind of reflection. But I’m wondering what you’ve learned—perhaps particularly between Liz and Matt but maybe Varun as well—about promoting uptake of the research you’re actually finding and sort of moving towards implementation. Do you find that once the research is published sort of different parts of the government you’re working in just take it and go, or is that part of the battle? And I might—(off mic).

HARDY: (Laughs.) This is being videotaped, right?

No, I’m not finding that, unfortunately. I think it’s getting better.

So a couple things are happening. In the federal government, there is very keen, very strong interest in experimentation, which is wonderful. I—a few years ago, the clerk, which is the head of our public service, released this directive to say that everybody should be experimenting in their work. That had, actually, a huge impact. Our demand for services quadrupled from that. So it had a huge impact in the interest that experimentation has. So individuals and departments are definitely interested in it.

I think where it gets difficult is when we actually start to experiment, like when we start to put our CT proposals in front of them where they start to see, woah, there’s going to be twelve different ones, it’s going to be out at the same time. Then, you know, we’re all very busy at work, and public servants are very taxed. And they do need to then find some time to actually make room for this in their day jobs, and I recognize that that is a challenge. So we are trying to work on ways of getting around this, and I think the professor did say in the first session that JPAL is actually sending people out into governments. And something at the IIU we’ve started is this fellowship program, where I hire behavioral scientists into the IIU onto my team, and then I send them out to various departments where they’re imbedded in departments in their job. A hundred percent of their job is to design trials, run them and then help with the implementation, and I think that last point—and as I said, it was a lesson I learned early on—that implementation phase is very, very important.

We also never test anything that can’t be implemented just like right off. Like so, anything we test legal’s looked at, privacy’s looked at, the form—there’s a forms—there’s a group of people that do forms. The forms people have signed off on it, so that in the end we can say, look, this is all good to go, we know it works, here’s your, kind of, implementation package. I think that’s helped.

The other thing that’s helped is publishing our results. So we’re becoming more and more, I think, deliberate—and now publishing the results, it does take time, and by publishing, I don’t even necessarily mean from an academic perspective; I just mean getting the results out on our website to say that, look, we are testing things, this is what worked. And it sort of has the effect of saying, like, we ran this trial with this department in this particular area and this is what worked, and we recommended that this be implemented and that’s—publicly, which, I think, can also have a positive impact.

KLEIN: We structure our—the way—that’s definitely an issue. Like the whole theory of evidence-based policymaking that you prove it over here and then everyone runs and takes it up is, like—there’s a lot more steps than that to make it all work the way we want. But in our use of behavioral science, the way we’ve structured it is that we make the opportunity to work with our behavioral scientists available across agencies, but then the requests come from them. And then we vet the projects, and then if we choose the projects—because it is a behavioral science problem, they do have enough data for us to be able to run a rigorous randomized control trial et cetera—they’re interested in implementing from the beginning. So I think structuring the design and the process from the beginning such that your customer is involved as opposed to doing it in a corner and saying, hey guys, who wants it—

HARDY: Mmm hmm.

KLEIN: —helps a lot. So anything that worked basically goes to scale.

BESCHLOSS: You were talking earlier before the session started also of the cognitive issues for the policymakers, not just poor populations that we’re talking about.

GAURI: Yeah.

BESCHLOSS: I don’t know if you want—in addition to whatever else you were going to say—to address that.

GAURI: Yeah, so on that, I mean, for us in the 2015 World Development Report, in which we documented the value of behavioral economics for development policy, we did a study of 2000 World Bank staff and found that World Bank staff exhibit confirmation bias, sunk-cost bias, you know, framing bias, you know, risk—you know, all these kinds of things surprise, surprise. Colleagues in the U.K. government found the same thing with U.K. civil servants; almost identical numbers for confirmation bias. Although in some ways they were a little bit better than World Bank staff, but we don’t like to talk about that very much.

But so that opened the door, and now we’re doing things internally. On, you know, our hiring policies we have gender-parity targets. Like we’re trying to look at, you know, what we can do behaviorally to address that; you know, learning within the organization. We’re trying to address a variety of internal challenges, but that really opened the door. Because if you can sort of talk to people and say yeah, people are biased, no, you’re biased, it’s like, oh, that’s a little bit—you know?

And so I think actually this question also speaks to the fact that I think fundamentally we’re clannish; we’re all clannish beings. And so if a World Bank person writes a report, another World Bank person will pay more attention to it. If an academic writes a paper, another academic will pay attention to it.

So what I observe is that in governments they will take up evidence if it’s coming from someone else in their government, you know; like if one of them did it, you know, that’s sort of, like, just grabs attention more. And so I think that really ultimately it’s about capacity building, sort of, and we try and—you know, we try and work with partners to sort of build capacity within governments. Because, ultimately, we don’t want to be the ones coming in to whatever country and saying here’s the idea; we want them to be—have their own team, have it—have them doing it on themselves, and if they do the RCT then they’re going to be much more likely to take it up.

Q: Drago Row (ph) from EY.

A few years ago, I was advising the State Department—the New York State Department of Temporary Disability and Assistance to transform the, you know, food stamps program. It took us about four months to define the term “household,” right? So, you know, that kind of strengthened my perception that the government is always addressing problems of yesterday. So especially in today’s world of, you know, disruption, innovation, I want you to comment on the pace at which these, you know, really good schemes—you know, fruitful schemes are being implemented. If not, what can be done with some, you know, private partnerships and such?

KLEIN: Yeah, that’s a good question.

Well, never fast enough. The—but I want to—I mean, there’s a couple of things. The—there’s an impression, I think, that government is sort of—people in government are slow moving or whatever; people work very fast. There’s just a lot going on, and there’s—and the bureaucracy is not because people don’t want to make decisions; it’s because things like definitions of “household” are imbedded in, like, federal and state regulatory requirements that you can’t change without sort of a meaningful process because politically-elected folks, you know, wrote the law, you know. So there’s reasons for all this stuff.

But, of course, we’re all frustrated in government that we can’t move faster for whatever reasons, good or bad. I think the constraints on innovation within government are definitely public attention. You know, there’s definitely much more emphasis on anything that goes wrong than stuff that goes right, so that’s—and systems that may be set up for very good reasons can constrain how nimble a government can be. And procurement is a prime example of that. Procurement is a process-laden effort because we don’t want the government to misuse its funds, but at the same time that means that entities that are good at getting government contracts get the contracts, and some of the more nimble firms may not get it. It means that, you know, things like agile development are, you know, in the tech sector are harder to do because you have to spell out requirements on the front end. I’m sure all this stuff is familiar—I’m sure you’d like to work differently with government than you’re able to do because of procurement constraints.

So I just say all that to say, the stereotypes often have rings of truth to them, but often because the reasons are better than we think, and I think we’re getting better. I think being able to—I think evidence-based policymaking, which as—which requires testing, innovation, failure, rigorous measurement is growing, and we’re going to get better at it.

HARDY: And I think that’s exactly well said, Matt.

And I think the thing I will add to that is the importance of creativity and flexibility, and I think on our team whenever we approach anything, we do sort of the—kind of two buckets. So what can we do now? So what can we test right now without changing any legislation, without it taking kind of the four-month thing? OK we can test these four things, but while we’re testing these four things we definitely think that when the leg comes up for change, we need to remove this piece.

So I think a good example of that is the trial I mentioned around organ donor registration. We redesigned everything that we could within the system, which was quite a bit, but then this one thing we couldn’t remove was this wet signature that’s required. So it sounds crazy when I say it, but when you go to register you have to sign but then you take the form with you and go home. So we don’t really need the wet signature, but it’s required, and we couldn’t remove it because it’s actually in the legislation. It says a wet signature is required. So what we did is we tested within that and said, OK, we’ll keep the wet signature, but then the next time we go for a leg change, let’s just remove that line.

So I think there’s—we just need, as practitioners, to be sort of flexible with the environment in which—and the context in which we work in which, to your point, is some of it is challenging to move, but it doesn’t mean we can’t be creative in the short term.

GAURI: I’ll just add briefly that we just put up on our web page sort of profiles of, I think, ten nudge units in different governments around the world, and they have different models. Some are sort of centralized, you know, some are more networked, you know; like the sort of, you know, Danish or Dutch model where it’s sort of spread out, and some involve more universities and are privates. And so there’s different models here bringing different entities in and we do that strategically. At certain times, we’ll say, well, things are going to be stuck with this government so we’re going to work with NGOs and eventually it’ll get there, you know. So it’s sort of—you’ve got to be flexible.

BESCHLOSS: Let me thank our great speakers. I think this was a great discussion. We could go on for much longer. And thanks to the members for your great questions. Thank you. (Applause.)

(END)