Innovations for Successful Societies 



MEASURING CITIZEN EXPERIENCES:
CONDUCTING A SOCIAL AUDIT IN VIETNAM, 2009-2013

SYNOPSIS
In late 2009, following three decades of gradual economic and governance reform by Vietnam's one-party government, three organizations came together to implement a social audit across the country. The Provincial Governance and Public Administration Performance Index (PAPI)-a joint policy project of the United Nations Development Programme, the Vietnamese nongovernmental organization the Center for Community Support and Development Studies, and the Communist Party-affiliated Vietnam Fatherland Front-aimed to draw information about citizen perspectives into decision making in Vietnam. It also sought to formulate quantitative measures of provincial performance and governance. Based on public surveys, PAPI aimed to provide a reliable picture of citizen experiences with provincial government along six dimensions: participation in government at local levels, transparency, vertical accountability, control of corruption, implementation of and adherence to public administrative procedures, and public service delivery. By 2011, PAPI was able to measure governance quality in all 63 provinces in Vietnam. The survey project represented the nation's first large-scale effort to systematically gather information about citizens' experiences with their local and provincial governments. It also led some provincial governments to create action plans that would improve the services citizens received and boost the rankings of those provincial governments in the index.

Rachel Jackson drafted this case study based on interviews conducted in Hanoi in July 2014. Case published December 2014.


INTRODUCTION
    By the beginning of 2009, Vietnam's government had spent more than two decades implementing economic and public administration reforms that aimed to transition the country to a middle-income, more efficient, and more responsive state without threatening its one-party structure. The move from a centrally planned economy to a market-based one had helped transform Vietnam from one of the poorest countries in the world to one of its fastest-growing economies.1 Gross domestic product, which had grown 2.79% in 1986 (the year the economic reforms began), averaged 6.88% growth from 1987 to 2008. During the same period, GDP per capita grew from $437 in 1986 to $1,165 in 2008, measured in current US dollars.2
    The outcomes of other reforms, which aimed to improve public administration performance and increase citizen participation in government at the local level, were less easily quantifiable. In 2009, to fill that gap, a team from the United Nations Development Programme (UNDP), a Vietnamese nongovernmental organization (NGO) called the Center for Community Support and Development Studies (CECODES), and the Department of Law and Democracy within the Vietnam Fatherland Front (VFF) began work on a social audit to measure ordinary citizens' experiences with provincial and local governments. The VFF, officially affiliated with the Communist Party, was enshrined in Vietnam's constitution as the organization responsible for articulating citizen perspectives to the Vietnamese government and for overseeing the government on behalf of citizens. The project aimed to survey citizens across the country on whether they had received promised government services and on the nature of their interactions with local and provincial levels of government.
    The UNDP, CECODES, and VFF team's initiative was not Vietnam's first social audit. In 2005, the Vietnam Chamber of Commerce and Industry and the US Agency for International Development (USAID) had collaborated to survey businesses throughout the country on their interactions with provincial and local governments. The resulting data, called the Provincial Competitiveness Index (PCI), enabled researchers to rank the provinces in terms of ease of doing business. The project pioneered in Vietnam the ideas of publicly ranking provincial governments based on survey data and of measuring government performance based on the experiences of those who received government services rather than on the experiences of those who provided the services. In the years after the index was established, several provinces worked to improve their scores and thereby encourage national and foreign investment. There also were some efforts at the municipal level to measure citizen perspectives on local government.
"We noted that Vietnam had an explosion of social audits," said Jairo Acuña-Alfaro, UNDP policy adviser on public administration and anticorruption. "There was a social audit here and a social audit there, in this province or in that province, but there was not a concerted effort to try to understand how all 63 provinces were performing in public administration." 
    Building on that project, the prime minister in 2007 approved Project 30, an attempt to simplify administrative procedures across the country. The project, a collaboration with USAID, included public consultation via the project's Web site, asking citizens to provide their opinions on the necessity and reasonability of certain procedures.3
    The 2009 social audit that CECODES, the VFF, and the UNDP launched sought to measure outcomes of the first phase of Vietnam's Public Administration Reform (PAR) program. The Ministry of Home Affairs had launched the package of ambitious civil service reforms in 1998,4 and in 2008, the UNDP's policy and governance team worked with CECODES and the VFF to publish an analysis of the PAR program and recommendations for its second phase.5 (From 2009 to 2012, the UNDP funded further evaluation of the PAR program's first phase and aspects of planning for the second phase.)
    While working on the public administration reform analysis, the policy and governance team at the UNDP's Vietnam country office in Hanoi discussed with CECODES and the VFF the idea of a social audit. "From the beginning of that research, one thing that came out very strongly was the lack of a monitoring and evaluation system-not only for the overall Public Administration Reform master plan but also for overall governance dynamics in Vietnam," said Acuña-Alfaro, who joined the UNDP in Vietnam in 2007. The government had recognized the need for a monitoring system for the PAR program but "had failed to set it in motion," he said. 
    The proposed social audit would also measure the degree of transparency and citizen involvement in local government. In 1998, the Communist Party Politburo had issued the Grassroots Democracy directive, which affirmed the rights of citizens to know about the activities of their local governments, to be consulted on such issues as land-use planning, and to monitor local government activities. A response to public disaffection with the arbitrary imposition of taxes and the embezzlement of public resources by local officials, 6 the decree exactly instructed local governments in the specific ways they were expected to interact with the population.7 The subsequent 2007 Grassroots Democracy ordinance redefined Vietnam's version of democracy as "the people know, people discuss, people do, people check." Competitive, multiparty elections were not part of that definition.

THE CHALLENGE
    The UNDP team agreed to help host and coordinate the audit in conjunction with CECODES and the VFF. The UNDP's Acuña-Alfaro and policy analyst Do Thanh Huyen knew, however, that implementing a social audit across provincial governments in Vietnam posed significant political, technical, and logistical challenges.
    The Grassroots Democracy ordinance implicitly acknowledged a need to gather citizen opinions, but hierarchical decision making encouraged policy makers at all levels to prioritize the central government's agenda over citizens' needs. Even for policies set at the provincial level, legislatures (known as People's Councils) and executive bodies (People's Committees) set policy for each province. That system left little room for discretion at government levels that were closer to the citizenry, such as districts, communes, and villages. Further, because legislators were elected from a list of candidates endorsed by the Vietnam Fatherland Front, the People's Councils did not necessarily have information about voters' real views or opinions.
    "We inherited the legacy of a centrally planned economy, which means a lot of things are still decided at the center-or at least at the provincial level," said Bui Phuong Dinh, director of the Vietnam Institute of Leadership and Public Policy at Ho Chi Minh National Academy of Politics and Public Administration, which trained the country's midlevel and high-level public officials.
    There was no significant attempt at the national level to assess public opinion or systematically involve citizens in the policy-making process. Most government agencies, from the central Ministry of Home Affairs to provincial and local agencies, used self-assessment reports and internally generated statistics to evaluate performance at all levels.
     "No citizens' voices were heard during the process, which was a problem, given that the government was emphasizing the role of citizens in policy implementation and policy monitoring," said UNDP policy analyst Huyen. That lack of consultation not only ran contrary to the government's stated agenda of citizen inclusion but also thwarted effective policy evaluation. "There was a serious lack of evidence to show how average citizens received or experienced the things that the state apparatus should provide for citizens," Huyen added.
    Making the case to the Vietnamese government for a survey of citizen experiences, Acuña-Alfaro pointed to the PCI project. "'The businesses have this tool to assess the performance of the government, and you have your own self-assessments that assess how well your different departments are performing; perhaps one missing side of that triangle is citizen voices,'" Acuña-Alfaro said he told the government. "'Let's triangulate and get all of that information, but to do that, we need to start collecting data from citizens' perspectives.' That's how we tried to build the argument to the government."
    A survey also posed logistical challenges, however. Vietnam was administratively divided into 58 provinces and 5 municipalities that held the same status as provinces, and survey teams had to coordinate separately with each of those administrations. Many of the provinces lacked roads in more-rural areas, making access difficult. The surveyors had to cover the entire country within a year while maintaining consistency in data collection methodology across all locations. To do that, the team had to win the cooperation of government at the local level and guard against both deliberate and accidental interference with the survey methodology. Any perception of bias could undermine the project.
    The team also had to figure out how to translate measurement of the goals of the PAR program and the Grassroots Democracy ordinance into concrete questions the average Vietnamese citizen could understand and answer truthfully-without fear of getting into trouble with government officials. Indeed, the in-person interview format was a new approach for a project of this size in Vietnam. The PCI project had relied on mailed surveys to business owners rather than in-person interviews.

FRAMING A RESPONSE
    Acuña-Alfaro offered the services of the UNDP's Policy Advisory Team to help guide the proposed initiative. CECODES and the VFF partnered to implement the project. Together the project partners developed the Provincial Governance and Public Administration Performance Index (PAPI).
    
Finding implementing partners
    To move the project forward, the UNDP sought to identify an implementing partner with the capacity to work throughout Vietnam. Following an open bidding process, the UNDP's policy and governance team enlisted Vietnamese NGO CECODES to join the project. CECODES had the self-avowed mission of "promoting civil society, strengthening its voice and its engagement with the state."
    Although CECODES had never carried out a project as large as a survey that sampled all 63 provinces, the NGO had a crucial connection to the VFF, whose active involvement in the survey greatly increased the chance of success; and CECODES had worked with the UNDP and the VFF earlier on the PAR evaluation.
    The VFF would be the main avenue to securing provincial government buy-in and opening doors at the local level. The project fit under the VFF's constitutional mandate to express the voices of citizens to the Vietnamese government and oversee the government on behalf of citizens. The VFF also had the job of promoting the central government's grassroots democracy policy. 
    With an administrative structure covering the entire country at all levels of government and with a political mandate to represent citizens' opinions, the VFF was a natural partner for the survey project. "We joined because PAPI is related to our work, especially in the supervision of concepts like 'people know, people discuss, people do, people check,'" said Nguyen Ngoc Dinh, who headed the Department of Law and Democracy within the VFF at the time. "We joined to help CECODES implement the project and at the same time to help ourselves understand more about people's opinions and people's experiences in this area."
    
Building additional support
    Expanding political support for the project, especially at the provincial level, was also a priority. Initially, the PAPI team had reached out to the Ministry of Home Affairs, which coordinated the PAR implementation, but the ministry leadership declined to provide active support. However, Home Affairs did not oppose the project after Acuña-Alfaro described the effort as being a logical extension of existing government work rather than being a radically new approach. "[We said,] 'We are not going to change overnight what is being done in the government; that is not going to change the way the government does self-assessments, but [the survey] is going to help complement that information,'" he said.
    The PAPI team needed greater political support from the central government to guard against potential political hurdles that could arise from the survey's geographic breadth (covering the entire country) and its political depth (raising issues ranging from the national to the local level). To build legitimacy and enlist additional expertise, the team set up a National Advisory Board, which would play a role in decision making and buttress the stature of the survey work by creating an image of shared commitment and cooperation. The team recruited members of the Vietnam Fatherland Front, the Communist Party, delegates to the National Assembly, and officials from government agencies such as the Ministry of Home Affairs and the Government Inspectorate, Vietnam's national anticorruption agency. The PAPI team also invited scholars and an editor of the Communist Party magazine.
    Negotiations to bring on board members, particularly those from government and party organizations, took months. "In the beginning, it was pretty difficult to convince people to join the board, because they didn't know what it was and how credible the project was; but then, once we got some publicity, it was easier to get people to join the board," said Dang Hoang-Giang of CECODES. 
    
Designing the survey
    After discussion among members of the PAPI team and the project's National Advisory Board, the project leaders decided to measure six dimensions of provincial governance, based on the stated goals of the first PAR master plan: participation in government at local levels, transparency, vertical accountability, control of corruption, implementation of and adherence to public administrative procedures, and public service delivery. 
    The drafting of the questions prompted significant debate because of the team anticipated that a survey about governance in Vietnam would be sensitive. The team eventually decided to ask citizens about their specific experiences with services or procedures required by law or administrative code rather than solicit individual opinions about open-ended topics like transparency or government performance; it expected citizens to be more willing to answer such questions. Moreover, asking about whether a service was available or about other practical, measurable qualities would help the project convince provincial government officials of the data's usefulness and avoid accusations that the audit had a political intent. 
    "How much do we compromise by going away from the Western concept of democracy and governance but not subscribing too much to the government narrative of what they understand as governance?" Giang said, reflecting on questions the group debated in the discussions. "We tried to find a middle ground where things are useful for the Vietnamese context and the actors here, but we needed to make sure it didn't get blocked by various interested actors. We needed to constantly reassess the risk of how far we could go and what we could do-both politically and technically." The group's task was made more complicated by the fact that "governance" did not have a simple approximation in the Vietnamese language and was therefore open to interpretation.
    For example, on issues of land use, rather than asking citizens whether they thought the local government had adhered to land-use regulations or whether they thought the regulations were fair, the survey asked citizens whether they had seen the commune's land-use plans, whether they had had a chance to comment on them before publication, and whether they thought those comments were being heard.
    The team members knew that to avoid accusations of bias, their methodology had to be unimpeachable. For help they turned to US political scientist Edmund Malesky, an associate professor at Duke University. Malesky had worked on the PCI survey of businesses' experiences with provincial governments, which had a significant impact on provincial policy making in Vietnam. "PCI had become part of the Vietnamese fabric of decision making," Malesky said. "Government officials were being promoted based on how they were doing in the index, and a lot of provinces were selling themselves to investors based on it. But I started to worry that the success of the PCI was starting to define governance in Vietnam as what businesses wanted." That definition made sense on some topics, such as transparency and petty corruption, which affected both citizens and businesses, Malesky added, but didn't work on such issues as regulation. Malesky came on board to help the team design a survey that would accurately represent broader citizen opinion. 
    Acuña-Alfaro, CECODES's Giang, and the rest of the PAPI team also knew that even if they were able to gather the data, they faced an uphill battle in getting government officials to use it. Reaching the broad, intended audience of national and provincial executive and legislative governments, government-sanctioned civil society, academia, and the media required multiple dissemination strategies. Beyond the difficulty of simply getting the data into the right hands, the PAPI team also had to confront skepticism about the value of the findings. "The whole thinking was innovative five years ago because the thinking in Vietnam-at that time and even now-is that the government knows best what's good for the people. It's very much about feudalistic concepts about the king, the father, the government knows what's good for you, and people here are too ignorant," Giang said in 2014. "[PAPI] flips around the view and says, 'The people are the customers, and customers know best what's good for them.' And then the service provider needs to reorient or readdress based on customer feedback. This is a change in thinking that is quite profound, and we needed a few years to convince them of that." 

GETTING DOWN TO WORK
    Before the team members could even consider working in all of Vietnam's 63 provinces, they had to test the design and methodology, demonstrate to skeptics the value of the project, and make sure provincial governments would agree to participate. "This is a data-averse policy-making system, and we were saying, 'We're going to do this survey; we're going to try to understand through statistical terms what citizens are experiencing,'" Acuña-Alfaro said. "It was difficult to have that conversation."
    
Piloting PAPI
    To test the waters, in late 2009 the PAPI team designed and planned a pilot phase in three provinces: Phu Tho in the north, Da Nang in the south-central area, and Dong Thap in the southern Mekong delta region. "We had to say, 'Let us do this in the three provinces, and then you will see. It's very difficult to discuss in abstract terms these issues,'" Acuña-Alfaro said. "There was some reservation, but [the provincial governments] were willing to let us go ahead and do it." 
    Once the team had selected 180 respondents per province, CECODES took the lead. Giang and the rest of the CECODES leadership worked with the VFF in each province to train local VFF officials as interviewers and set schedules for interviews in the villages. In each village, the local VFF sent letters asking respondents to come to a central location on a certain day to be interviewed. Each interview lasted nearly an hour, and the interviewers found the respondents to be generally open to answering questions, often wanting to offer more information than necessary or to tell stories.8
    Presenting the results of that first pilot round to the three provinces was an early test of PAPI's viability. If the provinces rejected the survey as a useful exercise, it would be difficult to get access the following year and to maintain political support for the project. In each of the three provinces, the team held discussions with party and provincial government leaders, the provincial VFF, the heads of provincial government agencies, and media representatives. All three provinces responded positively and agreed to continue participation in PAPI during an expansion. In Phu Tho, for example, the provincial VFF chairman said the PAPI data would be a useful contribution to an ongoing discussion at the provincial party Congress. Dong Thap's VFF chairman provided similar feedback, noting the survey's ability to highlight areas for public service improvement.9 
    
Learning from the pilot and expanding PAPI
    Following the pilot experience, the team had to figure out how to expand the PAPI survey across the rest of the country as quickly as possible. "We had done only three provinces, and Vietnam has 63 provinces," Acuña-Alfaro said. "Could we really go from 3 to 63? Did we have the capacity internally, within UNDP, but also externally, from CECODES? Did they have the human resources, the enumerators, the skills, the knowledge, and the network in all the provinces? And the answer was 'No, we cannot go from 3 to 63, but we can do 30. Let's see how far we can go with 30.'" 
    The team began planning for the 30-province expanded test in 2010. Based on data from the first round, the PAPI team refined certain aspects of the survey, eliminating questions that had elicited similar answers across districts or provinces or those that citizens had trouble answering. In its review of the pilot phase, the team concluded that questions about corruption required special attention. "We think that due to the sensitivity of the questions, the answers do not reflect real respondents' opinions," the review concluded. 10 In addition, the team thought the initial questions on health care and education had not been precise enough to capture accurate responses on the issue of quality. The team also solicited questions from the National Advisory Board. 
    In its pilot phase, the project had encountered a problem common across public opinion surveys. In their initial responses to the pilots, the three provincial governments were particularly concerned about whether such a small sample of the population could accurately represent broader opinion. The 2010 internal review of the three-province pilot phase acknowledged that local-government skepticism about sampling methodology would likely be an ongoing problem for the project. "Probably they [provincial government leaders] have never seen such a process before," the review said. "Therefore, preparation needs a long lead time, patience, and diplomacy."11 (For more information on how the PAPI team dealt with this problem, see Overcoming Obstacles).
    The PAPI team debated how to best present the data to the government and the public, and especially whether to rank the provinces based on the survey data. "In the first year, we discussed a lot about how to package the information," Giang said. "The people are the ones who are naming and shaming, so it's OK. If outside experts like Transparency International or the World Bank did the naming and shaming, the information would meet much more resistance. So we thought quite a lot about the degree of naming and shaming-whether we do ranking at all. There was a lot of discussion that if we don't do ranking, people will lose incentive to see where they are, and we'll lose the interest of the media. But if we do ranking, maybe it's too much naming and shaming." After consultation with the advisory board and among the project partners, the team decided to go ahead and rank the provinces based on the data.
    Meanwhile, the VFF proposed a list of 30 provinces that the group said would be easy to work with based on personal or political relationships. The VFF representatives on the team were initially reluctant to consider other ways of choosing the 30 provinces, even though Acuña-Alfaro and the rest of the team wanted the selection to be more methodical. "At that moment, what we said here in UNDP was . . . now we need to be a bit more rigorous on how we do this selection," he said. 
    The team matched pairs of provinces with similar socioeconomic demographics and then randomly selected one from each pair as a survey site in order to arrive at their list of 30 provinces. With that list in hand, the team went back to negotiate with the VFF. "I thought at that moment, this is going to be a nightmare, the government is not going to be very receptive to the idea, they're going to start complaining. But the VFF said, 'No, that's fine, we think it will be more relevant to do it the way you are doing it,'" Acuña-Alfaro recalled.
    "We were lucky in that we could convince them to work with us-but without intervening in the content," Giang said.
    In 2010, the Spanish Agency for International Development Cooperation funded the first PAPI expansion. In 2011, the PAPI team secured continuing support from the Swiss Agency for Development and Cooperation to ensure US$1 million of funding for each subsequent annual survey. That funding covered 75% of the project's operating budget; the remaining 25% came from the UNDP.

Expanding nationwide
    The 30-province expansion revealed a number of additional problems the PAPI team had to solve before rolling out the project in all of the provinces in 2011. In both the pilot round and the initial expansion, the team relied on local VFF officials as interviewers because the team needed local counterparts who spoke the relevant Vietnamese dialect. The team found, however, that participation by VFF officials, who were affiliated with the ruling Communist Party, could intimidate interviewees and that the officials sometimes failed to follow specified surveying protocol. In 2011, the UNDP, CECODES, and VFF team members agreed instead to use the provincial student network of a Vietnamese NGO called Live & Learn to recruit about 600 university students across the country. The students worked in teams under more than 50 team leaders that CECODES recruited and trained in Hanoi in the survey methodology. Those leaders in turn trained their teams to follow the interview scripts and collect the data. (For more information about the PAPI methodology, see the text box on page 15).
    Finding the right way to invite randomly chosen interviewees also took time. "We experimented in the first few years to get the right wording," Giang recalled, "because if you say you want to interview them, they will be scared. So then we switched to 'We want to consult you and discuss with you,' and things like that. It's really nitty-gritty-down to the wording of the letter-because people get scared when asked for an interview by a team from Hanoi. They don't know anything about nongovernmental organizations. Everybody from Hanoi is from the government."
    Team members from CECODES and the VFF also developed strategies to deal with interference from local officials. Such interference ranged from attempts to direct them to the wrong villagers for interviews to the local police wanting to sit in on the interviews. In situations when they could not persuade the police officers to leave, surveyors would schedule 10 interviews at a time, so that local government officials could observe only one or two. The surveyors would then note whether local government officials were present during the interview, and that let those analyzing the data account for the interference. In some instances, local government officials confiscated the surveys, but the VFF was able to resolve each situation. "The local Fatherland Front chapters are very important in terms of giving us the legitimacy that we are not any foreign, US-based suppressive organization trying to overthrow the government," Giang said.
    Dinh of the VFF said that local governments' suspicious reactions stemmed primarily from lack of information about the exercise, particularly before the team formally rolled out PAPI across the country. "In the first period of implementation, some provinces did not understand the purpose of PAPI, so there were questions when the survey team went out," Dinh said. "When they understood later, thanks to the nationwide announcements, they felt OK with it." Dinh added that the VFF played an important role in conversations with the local government. "We would often say that the result of PAPI is a mirror for managers to look into, see the reality of their performances, and adjust their daily work."
    The teams also faced logistical challenges in traveling across the country. In one instance, the district chosen through random sampling was so remote that surveyors had to cross into Laos and back into Vietnam in order to access it. In another case, villagers were reluctant to give up a day's fishing to participate. The local PAPI team eventually prevailed on village religious leaders to persuade the chosen participants to make time for their interviews.
    In 2011, the PAPI team was able to conduct the survey for the first time in all 63 provinces, interviewing more than 14,000 Vietnamese citizens, the largest such survey ever conducted in the country. The team interviewed about 200 people per province in 55 provinces with fewer than 2 million inhabitants each, 400 people per province in provinces with populations of 2 million and 4 million each, and 600 people in Vietnam's two largest municipalities of Hanoi and Ho Chi Minh City, each of which had more than 6 million citizens. In 2011 and in subsequent years, the PAPI survey's content and process followed the same structure. Once the surveyors had collected the results, the team coded and processed the data, determining a province's indicator for each of the six dimensions to produce that year's ranked index and reports analyzing the results. 
    
Publicizing and learning from the results
    Following the National Advisory Board's sign-off, the PAPI team began to disseminate the final national and provincial reports. Those annual reports and the accompanying data were published in English and Vietnamese on the PAPI Web site. Following a national launch in Hanoi-attended by members of the media, lawmakers, senior government officials, and representatives of civil society-the team held a series of regional workshops with provincial people's committees and provincial departments of home affairs around the country, presenting their results to groups of seven to nine provincial administrations at once.
    The VFF led the workshops, and the team used the gatherings as opportunities to pinpoint gaps in policy implementation or governance that had made the differences between low- and high-ranking provinces.
    Not surprisingly, provincial officials' reactions varied significantly. "The first year, when we introduced the project and the results in workshops, public officials would stand up and say, 'In my province, people are just farmers. They don't know anything, so you should not ask them, you should ask us.' Or, 'People in my province are rich, so they are very critical, they are biased, and they just want to talk badly about government,'" Giang recalled. "So we needed a few years to convince them about the power of the data, that the data is robust, that the people are not biased, and that we are asking things they could answer."
    Beginning in 2010, the team held individual provincial diagnostics workshops when the provincial governments requested them following the regional workshops. In those diagnostic workshops, PAPI team members explained their individual results in detail to provincial government officials. "In the first years, we didn't have that many discussions, because it was new and the provinces were not used to it," Acuña-Alfaro said. "Now they are asking for-they are demanding-to know a little bit more. We are starting to receive questions like, 'What does it mean to have a score of 3.7 on transparency?' That's the exact question we want because then this is an opportunity for us to disaggregate."
    As a second dissemination strategy, the PAPI team partnered with Ho Chi Minh National Academy of Politics and Public Administration in Hanoi, the Communist Party and government-run political school that trained politicians, senior party members, and government officials. Beginning with the 2010 results, Acuña-Alfaro and the other UNDP team members commissioned academy researchers for case studies that examined in depth why provinces with similar levels of economic development and other indicators received different PAPI scores. Academy researchers would then present their findings to the provincial people's committees.
    "It was first met with some suspicion because it was unusual for a public official to listen carefully to the reality through the lens of figures and numbers," said Bui Phuong Dinh, director of the Vietnam Institute of Leadership and Public Policy at Ho Chi Minh Academy. "We were quite lucky because we are part of the central academy for training of all public officials from middle levels to high levels in the provinces and at the ministerial and central levels. So it is an advantage in leveraging the credibility and validity of the research results."
    The Ho Chi Minh Academy research teams generally concluded that the differences in PAPI scores between similarly developed provinces with comparable demographics came from the level of political commitment on the parts of provincial, district, and commune officials; the amount of funding the provinces could direct toward social welfare programs; and the level of education of public officials in the local government. Of the three, political commitment presented the biggest challenge. "If you want to boost the political commitment of the lower level, you have to first boost the political commitment of the higher levels; and then you can use the authority of the higher level to request, to encourage, and, if it is necessary, to demand similar commitment from the lower levels," Dinh said. 

OVERCOMING OBSTACLES
    The PAPI team's primary obstacle was the skepticism from government leaders at all levels both on the usefulness of citizens' opinions in policy making and on the validity of the methodology. Both of those sources of resistance affected the team's broader objective of getting central and provincial-level policy makers to use the PAPI data in their work. The team originally had hoped lawmakers on the National Advisory Board and other government officials would advocate publicly for using PAPI data, but many board members were reluctant to do so.
    When first planning the project, members of the PAPI team had anticipated difficulties in persuading government officials to adopt the survey as a tool for management, but they still spent time convincing the national and provincial governments of its usefulness. They also faced resistance to the methodology, particularly from officials who doubted that a random sample of 200 citizens covering about half of their districts could accurately reflect citizen opinion in a province.
    In 2010, team leaders presented their findings from the 30-province round to provincial government and party officials in Kon Tum, a province in central Vietnam. "The Kon Tum authorities didn't believe us," Acuña-Alfaro said. "They said, 'Kon Tum has 800,000 citizens, and you're asking only 200. Kon Tum has nine districts, and you go to only three, so how are you telling me that this is representative?'" The Kon Tum provincial government asked the UNDP to redo the survey in all nine districts, with larger samples. The UNDP refused but provided the survey and methodology instructions for the Kon Tum government, which reran the survey, sampling 750 people across all nine districts. 
    "They came back to us and said, 'We have the results. You're not going to believe it, Jairo: they are the same!'" Acuña-Alfaro recalled. "So that helped us build the credibility we needed. And based on the information, Kon Tum was the first province to issue an action plan [in response to its PAPI scores]." The story of Kon Tum eventually helped convince a number of other provinces that responses from a small sample of the population might accurately represent the overall opinions of the larger population.
    By 2013, the PAPI team was turning to Ho Chi Minh Academy researchers to produce so-called action-based research using the PAPI results, supporting provinces with policy recommendations and support in drafting action plans. As of late 2014, all three provinces with which the academy researchers worked most closely had issued formal action plans to improve their PAPI scores, and the third was in progress. The PAPI team also began presenting the project to academy students and government officials, aiming to familiarize current and future generations of Vietnamese government officials and civil servants with survey research methods. By mid 2014, three classes of 80 to 90 party and government officials had completed course work using the PAPI results.12
    Not all provinces resisted the idea of improving their PAPI scores. By the time of publication of the 2013 survey results in April 2014, 12 of the 20 lowest-ranking provinces in the previous year's survey had requested and held individual diagnostic workshops with the PAPI team. Of those 12 provinces, 4 had issued formal directives or action plans.
    PAPI secured some traction at the highest levels of the central leadership. In March 2012, the minister of planning and investment and the United Nations signed the One Plan 2012-2016, which set forth the common framework of priorities for all UN agencies operating in Vietnam and which replaced the previous plan that had run from 2006 to 2010. The plan set three focus areas in which the UN agencies would support the Vietnamese government: "to achieve inclusive, equitable and sustainable growth; access to quality essential services and social protection; and enhanced governance and participation."13
    The UN used PAPI indicators in the plan to set 2010 baselines and targets for 2016. For example, the 2016 goal for political, social, professional, and government-sanctioned civil society organizations was that they be able to "participate effectively in policy discussions and decision-making processes for the benefit of the most vulnerable and disadvantaged groups."14 To translate that broad outcome into specific objectives, the UN borrowed PAPI's 2010 metrics-for example, that 6 (20%) of 30 provinces had performed well in terms of engaging citizens on policy development, implementation, and monitoring, which meant they had achieved a 7.5 out of 10 in those categories in the PAPI assessment. It set the target for 2016 that 20 (32%) of 63 provincial governments would move to that category in the PAPI results. The minister of planning and investment signed the agreement on behalf of the Vietnamese government.
    In a December 2012 speech at the annual Anti-Corruption Dialogue between the Vietnamese government and international donor agencies, Deputy Prime Minister Nguyen Xuan Phuc acknowledged the utility of the PAPI data. "We can clearly recognize the important role of local governments at different levels when analyzing variances in the rankings of provinces in the Provincial Competitiveness Index and the Provincial Governance and Public Administration Performance Index," he said.
     The PAPI team also made gradual progress in convincing members of the National Assembly to incorporate the data into their work. In 2013, team members held an after-hours voluntary briefing on that year's annual report, which was attended by 120 of the 500 members of the legislative body.
    However, even though the PAPI project originally aimed to support planning for the second phase of the Ministry of Home Affairs' Public Administration Reform master plan and even though the coordinator of PAR sat on the National Advisory Board, the ministry was reluctant to use PAPI data in its work. Dinh Xuan Hoa, director of the Public Administration Reform department at the Ministry of Home Affairs and who sat on the PAPI advisory board, argued that PAPI focused on certain dimensions that were irrelevant to correctly evaluating implementation of the PAR plan. Hoa, who had the authority to decide whether the ministry would take the data into account, argued that the index was misnamed and applied more to implementation of the Grassroots Democracy ordinance, which was outside his domain. "Other than public procedure reforms and the public services, the four other [dimensions] involve implementation of grassroots democracy in Vietnam," he said. "From the beginning of PAPI, I told them the name is not suitable; it should be called Index on the Implementation of Grassroots Democracy."
    Instead, in 2012, the ministry rolled out its own Public Administration Reform Index, which ranked the 19 central government ministries and 63 provincial administrations on their progress in implementing the Public Administration Reform program. Acuña-Alfaro viewed the ministry's move in a favorable light. "The existence of the PAR Index from the Ministry of Home Affairs, we strongly believe, is nothing less than the result of pressure that PAPI has put on the ministry to have its own indicator system," he said. "They have gone with something that is not really scientifically robust, but at least it is moving in that direction."
    The provincial rankings in the PAR Index differed significantly from PAPI's and were based primarily on self-assessment reports and secondarily on unreleased public opinion data the ministry had collected from interviews with ministry and provincial government employees, businesses, and citizens, according to Hoa. Speaking on the accuracy and legitimacy of the index, Hoa pointed to the Ministry of Home Affairs' own ranking: 12th out of 19 central government ministries in 2012. "Of course we feel embarrassed by the results, and we have to do better," he said.
    Unlike the Ministry of Home Affairs, the Government Inspectorate incorporated the PAPI data into its internal reports and encouraged provincial governments to pay attention to their corruption scores. In 2012, the inspectorate representative on the National Advisory Board requested that PAPI help gather data about petty bribes citizens had to pay, particularly in the health and education sectors. "The government inspectorate told us, 'All your questions on corruption are very interesting, but we already know that people have to pay,'" Acuña-Alfaro said. "'What we don't know is how much they are paying. Can you help quantify those amounts?'" That year, the PAPI annual report included a chapter on petty bribes, known in Vietnam as "envelope payments," across the country, garnering significant media attention. The inspectorate also formally incorporated PAPI into an internal circular on monitoring and evaluating corruption.
    In 2012, the project faced an additional unexpected hurdle: uneven political commitment on the parts of VFF leaders. Although the team had relied heavily on the middle ranks of the VFF to win local governments' permission to conduct the survey, it also sometimes needed VFF intervention at the provincial level. In 2012, three provinces threatened to pull out of the survey, and the head of the VFF declined to intervene. The crisis was averted only after the VFF official moved to another position and was replaced. "It wasn't an ideological or even a strategic choice," Giang said. "Someone took office who had different interests, and then he moved on and somebody replaced him. Luckily, we could better convince him to the support the project. The system here is not monolithic, and it's very unpredictable in terms of how it moves."
    The general apathy toward PAPI among VFF's top leadership illustrated to the team a weakness in the project design: Though support of the VFF middle ranks generally made gathering the data possible, the leadership was not prepared to take an active role in strongly advocating for the project, likely because of concerns about its political sensitivity. At the same time, the seal of the VFF kept another political organization from taking that role.
    In mid 2014, the VFF leadership had not publicly advocated for the government to formally adopt PAPI as a benchmark for provincial government performance. Though the group had helped in the survey work itself, the chairman of the VFF did not mention PAPI in his annual address to the National Assembly, and the VFF did not formally incorporate PAPI data into its provincial and local activities.


ASSESSING RESULTS
    By mid 2014, the PAPI team had carried out three rounds of the full survey in all 63 provinces and was preparing for a fourth. Thirteen provinces had issued directives or other formal responses to their PAPI results, and an additional 11 had held individualized provincial diagnostics workshops either with Ho Chi Minh Academy or on their own. The PAPI team and members of the advisory board generally agreed that it was too early to tell what the effects of those responses would be. A 2014 midterm review of PAPI said, "Implementation of these decisions has been limited by resource constraints and political will."15
    Other countries in the region began to take notice of PAPI. By 2014, both Indonesia and Myanmar had used PAPI as a model for local government assessments. A number of international NGOs operating in Vietnam also began using the PAPI provincial rankings when considering where to allocate projects, which had the potential to change provincial incentives to respond to their scores over time.
    PAPI and its methodology helped pave the way for more-extensive use of survey data and citizen input in Vietnamese politics and governance. Many ministries did not use PAPI data directly, but some acknowledged the need for similar data by creating their own indexes. PAPI also inspired the Judicial Provincial Index, a survey-based ranking of civil justice across Vietnam, created by the UNDP, CECODES, and the Vietnam Lawyers' Association (itself a member organization of the VFF).
    The 2014 midterm review of PAPI by external consultants hired by the UNDP and the Swiss Agency for Development and Cooperation found that the project had "made a substantial and growing contribution to understanding of good governance and public administration reform in Vietnam." The review noted in particular, "Giving citizens the chance to provide feedback to local government and services on their own is groundbreaking in Vietnam."16 The report also emphasized the PAPI project's role in providing the first independent, national database on such issues as public participation, corruption, and service delivery. It concluded, however, that the project's success was mixed in terms of getting its intended audience of provincial governments and central government ministries to use the data in concrete ways. Several factors contributed to that problem, including annual reports that were overly technical and difficult to understand, and an ineffective media outreach strategy.
    Despite its successes, the PAPI project had not yet gripped-at least as of mid 2014-the psyches of Vietnam's government leaders or the broader population. The midterm review focused mostly on the overly technical and complicated nature of the reports and media briefings as a significant reason for the lack of excitement.
    A number of scholarly publications by Vietnamese and international researchers used the data to discuss service delivery, corruption and transparency, and land-use rights. In that way, the survey data enabled Vietnamese society to take some first steps toward holding government accountable.
    Several members of the project team expressed frustration with media articles that often focused solely on PAPI's corruption data rather than on the other five dimensions the project measured. Some team members were surprised that the media failed to highlight figures indicating that citizen participation in elections was far lower than the government-reported 97 to 99%. Media organizations in Vietnam were carefully watched by the central government, which also either owned or controlled all local print and broadcast outlets.
    Even in provinces that had conducted diagnostic workshops, the reviewers found low levels of awareness about PAPI among many provincial officials, pointing to the PAPI teams' ongoing trouble in effectively disseminating the project data. "If you do a large-scale survey of people's opinions but we don't hear about it, how does that accomplish your purpose?" one provincial official was quoted asking in the report, referring to his and his colleagues' lack of knowledge about PAPI.17

REFLECTIONS
    Through the Provincial Governance and Public Administration Performance Index, the UNDP, CECODES, and the Vietnam Fatherland Front created a tool whereby Vietnamese citizens could give feedback directly to their local and provincial governments. That mechanism was otherwise missing from Vietnam's government structure. After overcoming immense logistical challenges, the project was the largest-scale collection of citizen perspectives on governance to date in the country.
    The partnership with CECODES and the Vietnam Fatherland Front was essential for the launch of PAPI both in providing political cover for the project at a national level and in opening doors with provincial and local governments. The VFF's involvement also helped PAPI fly under the radar of those who might have found it politically controversial during its vulnerable early stages. Later, as PAPI became more prominent, the project became more difficult to shut down, because doing so would run counter to the government's espoused values of soliciting citizen feedback and monitoring.
    Several members of the PAPI team and National Advisory Board expressed concern that the lack of a "political champion"-a reliable, respected, and prominent political figure to publicly call for the broader use of PAPI data in Vietnamese governance-allowed national and provincial governments to shelve much of the data rather than apply it in policy making.
    Others contrasted PAPI's limited effectiveness with the success of the Provincial Competitiveness Index, which allowed researchers to rank the provinces in terms of ease of doing business. PCI data had a unique usefulness because provincial governments could use high scores to attract investment.
    "The image for PCI is that if your PCI score is low, it means it is not easy to do business with you," said Dinh of Ho Chi Minh Academy. "That is a very successful PR campaign for PCI because it links the PCI scores to investors' expectations, which in turn could affect a province's foreign direct investment flow."
    Unlike the PCI, the PAPI findings at the time of writing offered no inherent incentive for governments to make difficult changes. PAPI, Dinh added, had not yet made that explicit connection to the consequences of low PAPI scores for provincial governments. "To have a successful PR campaign for PAPI, you must identify the cost of the attention of PAPI. What will they lose? What will they not be able to achieve? What opportunities will they miss?"
    Still, Dinh said he was hopeful that PAPI data would gain credence and influence as time passed. "Right now, I can't say that if your provincial PAPI score is low, you are to blame, but after two or three years, if your province's PAPI score is low, you will be to blame. We need that transition period."
    Malesky agreed that that he did not expect provincial governance to change quickly in response to PAPI. "Governance moves slowly. It's hard to rapidly change your governance. And even if you do, it's hard to get a quick response from your people. You can come in and be a really innovative leader, and it can [still] take citizens a few years to figure it out."
    Despite some ongoing challenges, PAPI filled an important need in providing accurate data on governance and service delivery. The project had the potential to be a valuable tool for provincial governance in Vietnam by providing an accurate picture of citizens' opinions and needs. "A government like Vietnam has a hard time monitoring itself," Malesky said. "On one hand, if you go out as a government surveyor, it's not clear citizens are going to be honest with you because they're worried about what they can say; but on the other hand, you need to know what's happening in local government because problems in localities can pose threats to stability."







Innovations for Successful Societies makes its case studies and other publications available to all at no cost, under the guidelines of the Terms of Use listed below. The ISS Web repository is intended to serve as an idea bank, enabling practitioners and scholars to evaluate the pros and cons of different reform strategies and weigh the effects of context. ISS welcomes readers' feedback, including suggestions of additional topics and questions to be considered, corrections, and how case studies are being used: iss@princeton.edu. 

Terms of Use

Before using any materials downloaded from the Innovations for Successful Societies website, users must read and accept the terms on which we make these items available. The terms constitute a legal agreement between any person who seeks to use information available at successfulsocieties.princeton.edu and Princeton University.
In downloading or otherwise employing this information, users indicate that:
a. They understand that the materials downloaded from the website are protected under United States Copyright Law (Title 17, United States Code).  
b. They will use the material only for educational, scholarly, and other noncommercial purposes.   
c. They will not sell, transfer, assign, license, lease, or otherwise convey any portion of this information to any third party. Republication or display on a third party's website requires the express written permission of the Princeton University Innovations for Successful Societies program or the Princeton University Library.
d. They understand that the quotes used in the case study reflect the interviewees' personal points of view. Although all efforts have been made to ensure the accuracy of the information collected, Princeton University does not warrant the accuracy, completeness, timeliness, or other characteristics of any material available online.
e. They acknowledge that the content and/or format of the archive and the site may be revised, updated or otherwise modified from time to time. 
f. They accept that access to and use of the archive are at their own risk. They shall not hold Princeton University liable for any loss or damages resulting from the use of information in the archive. Princeton University assumes no liability for any errors or omissions with respect to the functioning of the archive.
g. In all publications, presentations or other communications that incorporate or otherwise rely on information from this archive, they will acknowledge that such information was obtained through the Innovations for Successful Societies website. Our status (and that of any identified contributors) as the authors of material must always be acknowledged and a full credit given as follows:
Author(s) or Editor(s) if listed, Year of publication, Full title, Innovations for Successful Societies, Princeton University, http://successfulsocieties.princeton.edu/






Innovations for Successful Societies (ISS) is a joint program of Princeton University's Woodrow Wilson School of Public & International Affairs and the Bobst Center for Peace & Justice. The Woodrow Wilson School prepares students for careers in public service and supports scholarly research on policy and governance. The mission of the Bobst Center for Peace & Justice is to advance the cause of peace and justice through mutual understanding and respect for all ethnic traditions and religious faiths, both within countries and across national borders. 
References
1 "Country Profile: Vietnam," Library of Congress Federal Research Division, December 2005; http://lcweb2.loc.gov/frd/cs/profiles/Vietnam.pdf.
2 "Vietnam," World Bank; http://data.worldbank.org/country/vietnam.
3 Matthew G. Schwarz, "Project 30: A Revolution in Vietnamese Governance?" Brookings East Asia Commentary, Brookings Institution, September 2010; http://www.brookings.edu/research/papers/2010/09/vietnam-schwarz.
4 For further information, see David Hausman, "Policy Leaps and Implementation Obstacles: Civil Service Reform in Vietnam, 1998-2009," Innovations for Successful Societies, 2010; http://successfulsocieties.princeton.edu/publications/policy-leaps-and-implementation-obstacles-civil-service-reform-vietnam-1998-2009. 
5 Jairo Acuña-Alfaro (editor), Reforming Public Administration in Viet Nam: Current Situation and Recommendations, The National Political Publishing House, Hanoi, Vietnam, 223 November 2009; http://www.vn.undp.org/content/vietnam/en/home/library/democratic_governance/reforming-public-administration-in-viet-nam--current-situation-a/.
6 Garry Rodan and Caroline Hughes, The Politics of Accountability in Southeast Asia: The Dominance of Moral Ideologies, Oxford Studies in Democratization, Oxford University Press, United Kingdom, 2014, p 98.
7 Garry Rodan and Caroline Hughes, p 100.
8 PAPI Pilot Final Report, 2009, p 19, http://papi.vn/documents/documents-and-data/2010/en/2009PAPI_Pilot_Final_Report_EN.pdf.
9 PAPI Pilot Final Report, 2009, p 43.
10 PAPI Pilot Final Report, 2009. 
11 Ibid.
12 PAPI Mid-Term Review.
13 "One Plan 2012-2016," United Nations; http://www.un.org.vn/en/publications/one-un-documents/doc_view/278-the-one-plan-2012-2016-between-the-government-of-the-socialist-republic-of-viet-nam-and-the-united-nations-in-viet-nam.html.
14 Ibid.
15 PAPI Mid-Term Review.
16 Ibid.
17 Ibid.
18 Jairo Acuña-Alfaro and Edmund J. Malesky, "PAPI's Sampling Methodology: An empirical exploration to alternative approaches," September 2014; http://papi.vn/documents/documents-and-data/2013/en/PAPI_Sampling_Memo_Final_ENG.pdf.
---------------

----------------------------------------

---------------

----------------------------------------




Michael Scharff
Innovations for Successful Societies





2

(c) 2012, Trustees of Princeton University  
Terms of use and citation format appear at the end of this document and at www.princeton.edu/successfulsocieties










ISS is a joint program of the Woodrow Wilson School of Public and International Affairs and the Bobst Center for Peace and Justice. Terms of use and citation format appear at the end of this document and at http://successfulsocieties.princeton.edu/about/terms-conditions. ISS invites readers to share feedback and information on how these cases are being used: iss@princeton.edu. (c) 2014, Trustees of Princeton University




Rachel Jackson
Innovations for Successful Societies





17

ISS invites readers to share feedback and information on how these case studies are being used: iss@princeton.edu.
(c) 2014, Trustees of Princeton University







ISS is a joint program of the Woodrow Wilson School of Public and International Affairs and the Bobst Center for Peace and Justice: http://successfulsocieties.princeton.edu. ISS invites readers to share feedback and information on how these cases are being used: iss@princeton.edu. 




Rushda Majeed		Innovations for Successful Societies
Innovations for Successful Societies





19
(c) 2011, Trustees of Princeton University