MAKING A SMART CITY A FAIRER CITY: CHICAGO'S TECHNOLOGISTS ADDRESS ISSUES OF PRIVACY, ETHICS, AND EQUITY, 2011-2018 SYNOPSIS In 2011, voters in Chicago elected Rahm Emanuel, a 51-year-old former Chicago congressman, as their new mayor. Emanuel inherited a city on the upswing after years of decline but still marked by high rates of crime and poverty, racial segregation, and public distrust in government. The Emanuel administration hoped to harness the city's trove of digital data to improve Chicagoans' health, safety, and quality of life. During the next several years, Chief Data Officer Brett Goldstein and his successor Tom Schenk led innovative uses of city data, ranging from crisis management to the statistical targeting of restaurant inspections and pest extermination. As their teams took on more-sophisticated projects that predicted lead-poisoning risks and Escherichia coli outbreaks and created a citywide network of ambient sensors, the two faced new concerns about normative issues like privacy, ethics, and equity. By 2018, Chicago had won acclaim as a smarter city, but was it a fairer city? This case study discusses some of the approaches the city developed to address those challenges and manage the societal implications of cutting-edge technologies. Gabriel Kuris drafted this case study based on interviews he and Steven S. Strauss, Lecturer and John L. Weinberg/Goldman Sachs & Co. Visiting Professor at Princeton University, conducted in Chicago during July 2018. Case published September 2018. INTRODUCTION In 2011, Chicago's longest-serving mayor, Richard Daley, retired after 22 years at the helm of America's third-most populous city. He left behind a city revived from postindustrial decline, with a robust economy and a thriving downtown but persistent problems of crime, poverty, and racial disparity.1 In his place, voters elected Rahm Emanuel, who had served Barack Obama as chief of staff during the first two years of the Obama presidency. A 51-year-old former congressman from Chicago nicknamed "Rahmbo" for his combative political style, Emanuel promised a fresh approach to his 2.7 million constituents, including a new focus on data. In a transition plan released days before his inauguration, Emanuel pledged to "open up our government to the public with unprecedented access to data and information" in order "to make services more effective and equitable." The document promised that Chicagoans would be able to "interpret and use City data in ways that most help the public."2 "Mayor Emanuel came into office with a data-driven approach to policy making and government," said Danielle DuMerer, who had helped the city launch a basic online data portal in 2010. Emanuel "saw firsthand, with the 2008 elections, the transformative use of data," said Sean Thornton, a Chicago urban-policy journalist. In Obama's White House as well as his Chicago-based presidential campaign, decision makers had developed insights from the emerging science of data analytics, which employed computer-automated algorithms to draw useful inferences from patterns and trends in large datasets. Obama's campaign had used data analytics to pinpoint and motivate voters,3 and his presidency included data-based innovations in services ranging from health care to national defense4-along with a few technological missteps.5 During his first months in office, Emanuel issued an executive order that required the release of city data online. He appointed Brett Goldstein as the city's first chief data officer (CDO), a post that was rare in US cities but that a few a large corporations and nonprofit institutions had created in their own ranks. Goldstein had left a successful software engineering career to join the Chicago Police Department, where he founded the predictive analytics unit. (For more on Chicago's predictive policing efforts, see text box 1.) Emanuel wanted Goldstein to expand those efforts across all city departments. "The mayor had campaigned on this," explained Goldstein, who recalled Emanuel's telling him, "We want Chicago in the lead in data." During the next several years, Goldstein and his successor Tom Schenk played key roles in developing cutting-edge uses of city data, including map-based data visualization tools, statistical targeting of restaurant inspections and rodent extermination, early predictions of lead-poisoning risks and E. coli (Escherichia) outbreaks, and a citywide network of ambient sensors. By 2018, such projects had earned Chicago acclaim as a smart city,6 but they also raised new questions about normative issues like privacy, ethics, and equity. Chicago's technology leaders faced trade-offs as they worked to improve government outcomes while protecting citizens and responding to citizens' needs and concerns. THE CHALLENGE In many ways, Chicago was well positioned for leadership in data-based policy making. Emanuel benefited from a structurally powerful mayorship and key political allies at the city, county, state, and federal levels. The city was densely populated, centralized, and contained within one county, and it had fewer regional coordination challenges than sprawling, multipolar cities like New York, Los Angeles, and Houston. Chicago had a core of skyscrapers where its namesake river met Lake Michigan, and it was surrounded by a street grid and a robust public transit system that linked 77 close-knit neighborhoods. Long considered the hub of inland America, Chicago had developed a diverse economy with a sizable high-technology sector, along with interwoven academic and civic institutions. "Chicago has a deep bench of universities, nonprofits, and philanthropic organizations that are willing to make partnerships to get things done," said Thornton. Scholars at the University of Chicago had helped pioneer the use of quantitative methods in social science and public policy. By 2011 the university had leading programs in urban informatics-the study of urban data and its applications-and had created the Urban Center for Computation and Data, the Center for Data Science and Public Policy, and Urban Labs. Other local research institutions included Northwestern University, the University of Illinois at Chicago, DePaul University, and Northern Illinois University, as well as the federally funded Argonne National Laboratory. Many of the researchers at those institutions were trying to understand how cities like Chicago worked and how they could work better. "Cities are, by default, very distributed systems," said Alex Engler, a University of Chicago lecturer who directed a graduate program in computational analysis and public policy. Every day in Chicago, people, places, and things collided in countless ways. Those interactions formed a galaxy of interdependent systems related to traffic, crime, disease, the economy, population shifts, and cycles in service usage. Records that had tracked such systems for decades became gold mines for data scientists, but technical, political, and administrative hurdles often made usage of that data impractical. Because many city agencies had their own individual procedures and processes for record keeping, it was sometimes difficult to reconcile differences between datasets. Even information as straightforward as a postal address could be written in a variety of ways, using different abbreviations, and had to be encoded into consistent formats suitable to computerized analysis. "Certainly, data harmonization across different platforms is difficult," Engler said. "It's a truism in data science that 90% of the work is data munging [reformatting] and data cleaning." In addition, some government record keepers' data-ownership policies conflicted with the policies of others, or they were reluctant to lose exclusive control of their data because of concerns about legal risks and liabilities or database errors and inconsistencies. And many lacked the time and resources to provide the documentation others needed to interpret their data. Beyond the practical challenges, however, were thorny normative concerns. The harvesting and sharing of open data could erode citizen privacy or data security. Advanced data analytics could lead to Big Brother-type surveillance mechanisms or other authoritarian intrusions into citizens' private lives, unfairly scrutinizing citizens based on opaque computer algorithms. Data analytics also could reinforce latent biases in public policy because the city did not collect data uniformly, and some neighborhoods received more surveillance, attention, or resources than others. Finally, miscalibrations or misconceptions in data analyses could have disastrous consequences. For instance, overlooked assumptions in an early computer model used in rebalancing fire coverage in New York in the late 1960s had contributed to catastrophic fires that devastated underserved neighborhoods.7 Finally, the Emanuel administration confronted long-held perceptions of city government elitism as well as city government neglect in its efforts to harness data to better serve all citizens. Chicago was one of the country's most segregated cities8 and had a storied history of public corruption and abuse of authority.9 FRAMING A RESPONSE At first, Goldstein served directly under Emanuel and was responsible for the collection, oversight, and usage of all Chicago city data. Working with DuMerer, he quickly assembled a staff that DuMerer called "small but scrappy." In 2012, Emanuel promoted Goldstein to chief information officer and commissioner of the Department of Innovation and Technology (DoIT), which the Daley administration had made the city's centralized information technology office, and his team moved with him. A year later, Goldstein hired Tom Schenk, a data scientist working in the neighboring state of Iowa, as head of analytics. The staff welcomed these changes. "In an operational office, the CDO can be less burdened by the politics of the mayor's office," Thornton said. Goldstein had planned to work in city government for only two years, and he trained his deputy, Brenna Berman, a former IBM consultant, to take his place. Berman and her colleagues helped the mayor's office release a technology plan in 2013 that drew on input from dozens of local experts, including several cited in this case study. The plan established five "foundational strategies": (1) next-generation infrastructure; (2) every community a smart community; (3) efficient, effective, and open government; (4) civic innovation; and (5) technology-sector growth.10 To serve those aims, the city data team made several strategic choices. First, they met frequently with a range of city officials to understand the challenges they faced in delivering services and the data they could make available. DuMerer stressed the importance of "working with every department and taking time to understand their work and the information being requested of them, so that ideally we were enabling staff to be more efficient." "You can imagine sticky notes on a wall," Schenk said, describing his listening and brainstorming sessions with agency leaders. "It had to be formal at first, but over time, we got better at doing the conversations informally." Several important projects grew out of those discussions, including ideas for better targeting restaurant inspections and predicting lead-poisoning risks. Second, the team invited collaboration. Civic, academic, and private-sector partnerships served as a force multiplier for the dozen or so members of the city data office. Schenk explained: "Largely, the role of the CDO is to be a bridge between all the actors out there: the civic tech community, academia, and other offices and government entities. If you can bring all those folks together, you will be a successful data officer." The team favored open-source data and software, which could be used and redistributed freely in order to lower barriers to collaboration and enable people outside government to independently pursue projects that used the data. Engler said of the city data office: "They have made their projects open-source; they have picked up open-source projects and implemented them; they have taken their open-data strategy very seriously; and they have engaged with the community. Did this mean working on the coolest, sexiest tech projects? No! But I think it was so smart, and they have improved outcomes by working with the strong civic tech and data science community." Third, the team proceeded cautiously, with an early emphasis on relatively easy, uncontroversial, foolproof applications of data analytics. That careful approach reduced the chance of embarrassing or detrimental failures or breaches of data security. "The majority of data analytics use cases that Chicago has focused on so far have arguably fewer barrier to implementation because they avoid PII [personally identifiable information]", Thornton said. This more deliberate process also enabled the city data office to write better code and design modular, multipurpose software architecture. "Our approach was always to iterate slowly," Schenk said, "building more and more reusable components, to make our work a little easier over time." The team also decided to work within existing programs and infrastructure. That was a practical choice in a city with a tight budget, a stable population, a fully built-out cityscape, established institutions, and a preservation-minded civic culture. It also marked a philosophical counterpoint to more-radical smart-city projects-like Toronto's Sidewalk Labs or the planned development of the city of Songdo in South Korea-that sought to build data-driven urban infrastructure from scratch.11 "Chicago has a model of incremental progress," said Engler. "It's not trying to be a smart city" in the usual sense of the phrase. But it was still a closely-watched model. Other cities could adapt and implement Chicago's incremental innovations more easily than those with technologies that required new infrastructure. Similarly, Chicago's data office preferred working with preexisting datasets, as opposed to gathering new data. "Eighty percent of the projects used data we already had," Schenk said. "We weren't looking to come up with new data, although sometimes we did work with national data, state-level data, and Twitter data." GETTING DOWN TO WORK After Goldstein became Chicago's CDO in 2011, he focused on building a team, instituting office protocols, releasing open data, and developing practical tools to assist city agencies. He also began building relationships with other city offices and nongovernmental partners and laid the groundwork for longer-term data analytics projects. After Berman succeeded Goldstein in 2013, she, Schenk, and DuMerer revamped the city data portal, launched several data analytics projects, and worked with external partners on experimental technologies such as a citywide network of smart sensors. (DuMerer succeeded Berman in 2017, and Schenk left office in June 2018.) Demonstrating value to earn support Goldstein and DuMerer first bolstered and relaunched the city data portal to fulfill Emanuel's campaign pledge and open-data directive.12 They chose to work directly with Socrata, a Seattle-based software company that built open-data platforms for public-sector clients. The city released 400 city datasets, many of them programmed to receive automatic updates. "It was the biggest [crime] dataset [Socrata] had ever seen," Goldstein said, adding that the data dump "became a global story that put us on the map." The data included decades of crime statistics that no major city had released before. Police officials were initially reluctant to make all this information public, but they trusted Goldstein, a former police officer, and he responded to the concerns they raised during meetings and calls. "Data is what data is," Goldstein recalled telling the police superintendent. "People were already getting the data; they find ways." In the past, media reporters and civic activists had relied on persistent phone calls and requests under the Freedom of Information Act (FOIA) to access government information, which tied up department resources. "A lot of this information would have been released through FOIA, which would have taken multiple people time to put together," said DuMerer. Making the data publicly available reduced the drain on staff time. Moreover, DuMerer said, "With open data, by demonstrating the value early on through initial releases, staff came around to seeing the benefit of publication." For example, local programmers used the data released on the portal to develop popular smartphone applications for the tracking of bus, street sweeper, and snowplow schedules. With the announcement that the city would host a major NATO summit, scheduled for May 2012, Goldstein's team soon had a chance to make a major contribution. Politically sensitive meetings often became flashpoints. The city government worried about threats to public safety like violent demonstrations or terror attacks. Referring to the 1968 Democratic National Convention in Chicago, which brought widely televised mass protests and police crackdowns, Goldstein called the NATO summit the "biggest event in the city since 1968-and 1968 didn't go too well." The question was whether the team could help city agencies use data to intervene constructively if needed. During 10 hectic months, Goldstein's team developed a tool it called WindyGrid, playing off Chicago's nickname, the "Windy City." WindyGrid increased situational awareness among those managing the event by its creation of an online dashboard that layered up-to-date data maps on top of an open-source data platform called MongoDB [and later on Plenario, which had been developed by computer scientists at the University of Chicago].13 Being able to visually integrate real-time data from surveillance systems and other sources helped the officials who were managing the summit respond to conditions more rapidly and tactically. For example, they could catch potential crises in real time by tracking calls to 911 and 311 (the city hotlines for emergency and nonemergency services, respectively). WindyGrid demonstrated the concrete advantages of data-driven decision making, which earned Goldstein's team more political support. The city later released a public version of WindyGrid, called OpenGrid, accessible through its open-data portal.14 Building checks and safeguards From the beginning, Chicago's data office sought to "promote transparency while still providing safeguards," DuMerer said. Whenever possible, the office avoided the use of personally identifiable information, often incidental to the Big Picture focus of data analytics. Schenk noted that unlike state or federal governments, "cities don't have much personal information; cities tend to manage properties and locations more than they manage people." City interactions with citizens were often anonymized. For example, Chicago was more concerned with traffic flows and public health trends than the personal identities of drivers or patients. The office abstracted, anonymized, or aggregated any personal information to protect privacy. For example, when it received data that included the specific addresses of reported crimes, it reduced this information to the city-block level, with little impact on the accuracy of crime maps. The office set policies and protocols to minimize privacy risks and security risks before releasing data for public use. "We put the datasets through a governance process with multiple checks to make sure we're thinking about and catching any potential issues," DuMerer said. Goldstein's team established the protocols in consultation with city lawyers and technologists. Schenk's team worked with city employees who managed the original datasets to ensure technical accuracy. The chief information security officer (CISO), who typically had a background in computer science and cybersecurity, reviewed each proposed data release for security and privacy issues. And city lawyers and the mayor's office conducted a final review. The CISO, like the CDO, was accountable to the chief information officer and, ultimately, the mayor's office. However, because the CISO was independent of the data office, the position served an important role as a check and balance, according to Schenk. "The CISO's job is to set the rules and set up a fence," he said. "The CDO's job is to maximize the area within that fence in order to make use of data without violating the rules. . . . At one point, those roles are going to be in conflict, and they will have to make a choice about whether to value innovation over security or security over innovation." Chicago's municipal code gave the chief information officer authority to enter into data-sharing agreements-critical for sensitive data like police records. Agreements with vendors included data protection terms that obligated vendors to use due diligence, integrate security into software architecture, follow established frameworks, adhere to industry standards, and undergo periodic security assessments. Cultivating civic partnerships Early on, Goldstein reached out to Chicago's homegrown civic tech community of volunteers interested in using technology to serve the public good by developing open-source websites and smartphone applications. Civic tech volunteers first met at local OpenGov meet-ups that began in 2009, then at Chi Hack Night, which started in 2011 as a weekly open meeting in downtown Chicago at which dozens of participants shared dinner, ideas, and expertise and "hacked" solutions to city problems. Other cities had similar gatherings, but few were as well organized and well attended. (For more on Chicago's civic tech movement, see Appendix 1.) The civic tech community had a symbiotic relationship with the city government, even though it was independent of government control. Chicago's open-data portal was a key catalyst. "The release of data by the city was the spark that got things going," said Chi Hack Night cofounder Derek Eder, a civic technology entrepreneur. "The attraction is the data," confirmed Daniel O'Neil, a software and media entrepreneur who cofounded the original OpenGov gatherings. "There were gorgeous datasets coming from the city." Civic tech leaders reached out to the mayor's office with suggestions for improvements to the open-data portal and found the Emanuel administration responsive and supportive of their efforts. "It's common for people in government to feel defensive when people ask questions about the data," Eder said. "But they saw us as citizens using our skills to voluntarily create something of public benefit." Eder gave "almost all the credit" to Schenk, a frequent Chi Hack Night attendee. Schenk saw the civic tech community as "no different from the Rotary Club"-a twenty-first-century manifestation of Chicago's strong civic fabric of clubs, associations, and good-government groups. He added, "What we can give Chi Hack Night are our attention and appreciation-a dialogue with the city." In turn, the community provided volunteer labor, expertise, ideas, and feedback. The city often promoted and adopted apps developed by civic tech volunteers. The Smart Chicago Collaborative-a nonprofit led by O'Neil and funded by the MacArthur Foundation and the Chicago Community Trust in partnership with the city-was another key partner. Smart Chicago served as a bridge between the city, academic institutions, civic institutions, and everyday Chicagoans-particularly those in underserved communities who lacked equal access to technology and public services. The collaborative's activities included projects, publications, and conferences around themes like digital inclusion and data tools to measure things like civic engagement and citywide disparities. One of Smart Chicago's innovations was the Civic User Testing Group, which started in 2013 and conducted user testing of civic tech apps and websites by way of citizens in public libraries in disadvantaged neighborhoods.15 Smart Chicago compensated testers for their time with $25 prepaid debit cards. The city used the group for user feedback on its design for OpenGrid, as well as its relaunch of the city data portal in 2017, with a new focus on user-friendliness. Making use of data analytics Schenk's team based its data analytics projects on problems pointed out by city agencies where data existed with foreseeable technical applications. One early example involved the efficient allocation of inspectors, first demonstrated through a project on better targeting restaurant inspections after media complaints about delays by the city public health department. The project required extensive testing and adjustments based on feedback received in meetings with the health commissioner, but eventually, computer algorithms could predict which restaurants were most likely to fall short of city health requirements, thereby saving the time and expense of purely random inspections. The restaurant-inspection program generated public attention and demonstrated to other city departments the potential of data analytics. Other department leaders came to Schenk's team with analogous problems like efficiently targeting rodent extermination, which inspired further projects. A more complicated program aimed at predicting outbreaks of a waterborne infection caused by E. coli bacteria started as a project at Chi Hack Night, prompted by citizen complaints about beach closures. Modeling the outbreaks turned out to be a complicated statistical problem that microbiologists had been researching for years. "After a year and a half of working together [with the civic tech community], we were able to create a better algorithm" for the statistical model, Schenk said. "We even put out a research paper coauthored with members of the community, forthcoming in a journal article." During test runs in 2017, the new algorithm tripled the accuracy of the previous model that Chicago had used for predicting outbreaks related to E. coli, thereby showing the value of external partnerships.16 Volunteers put in most of the work to develop the algorithm, which the city then tested and fine-tuned. "With the E. coli project, we had over a thousand hours contributed by people outside the city government," Schenk said. "We probably would not have finished the project without that. Likewise, I ended up hiring a data scientist who worked on a pro bono basis on that project, so it was a great way to identify talent within the community." Taking on lead poisoning Perhaps the team's most ambitious data analytics research project was Lead Safe, a program to detect high risks of lead poisoning-before children could become exposed to the neurotoxin by ingesting lead paint dust from unrehabilitated houses built before the city banned lead paint in 1978. Because of the high risks of lead poisoning citywide, Chicago required periodic blood tests of every child from the age of one through age six. When elevated blood levels were found, the Department of Public Health investigated sources of exposure-and had the legal authority to force a home inspection and remediation. However, the tests triggered inspection only after the detection of lead levels high enough to risk the stunting of children's mental development. The department instead wanted a proactive tool that would catch threats before babies grew old enough to crawl around and ingest lead dust, so it collaborated with University of Chicago researchers and Schenk's team to develop a data analytic model that would predict children's risk of exposure to lead paint. Summer fellows of the Data Science for Social Good program at the university's Center for Data Science and Public Policy began work on the project in 2013, and it launched in 2018. After examining more than 2,000 variables, the team found a subset that helped predict a child's likelihood of lead poisoning in Chicago. For example, dwellings likely to have lead paint tended to be older, of low value, unrenovated, cited for other code violations by the department of buildings, and located near other dwellings found to have lead paint. Of more than 1 million addresses registered in the city, the team found the thousand dwellings most likely to have lead paint. However, the public health department lacked the resources to inspect every high-risk home, each of which could take up to six hours. Moreover, the department found early on that many parents found direct government interventions intrusive and alarming-especially because subsidized remediation options were not available in all cases. After testing various approaches, the department learned that new parents trusted their health-care providers as intermediaries, so the city data office developed an online software program restricted to registered health-care providers like hospitals and clinics. During regular checkups of pregnant women or meetings with new parents, the health-care providers asked a series of questions about important indicators of lead-poisoning risk and submitted the data to the city electronically through an electronic-medical-record platform. Within seconds, the Lead Safe program produced a risk score and follow-up instructions so the provider could advise the woman or the parents about risks and options and then recommend an inspection as necessary. "We built this as a free service for hospitals because doctors can have conversations with their patients that we can't have," Schenk said. The department determined that health-care providers' personal relationships with their patients positioned health-care providers to better balance concerns about privacy and public health than city workers could. Supporting the Array of Things The work of Chicago's data office was not limited to software. One of the team's long-term projects was to help the city modernize street lighting by replacing incandescent street lamps with a "smart" network of light emitting diodes (LEDs) connected by an electronic management system to improve reliability, safety, and energy efficiency.17 In 2011, city technology officers planning the project and developing Chicago's technology plan met with Charlie Catlett, a computer scientist at Argonne National Laboratory, a federally funded research facility operated by the University of Chicago. Catlett realized that city workers who were upgrading streetlights could install experimental, low-cost sensors that would provide localized data on air quality, noise, climate, weather, and other factors. City officials embraced the idea as the kind of "next-generation infrastructure" they envisioned. DuMerer said the Emanuel administration wanted Chicago to serve as a "beta-testing" laboratory for emerging technologies: "We put a stake in the ground saying we are open to . . . using the city as a platform for research." For the next two years, Catlett honed his idea, and his research center hosted a series of interdisciplinary workshops to explore the potential of a distributed network of smart sensors in an urban environment. The workshops drew on the expertise of scientists, engineers, urbanists, and technologists. Catlett's initial proposal did not win federal grants in 2014, but he used research funds from Argonne and the University of Chicago to develop a prototype proof of concept that helped the proposal win funding the following year. The proposal included in-kind cost sharing from Argonne, the university, and the city, which agreed to mount the sensors for free. The project was called the Array of Things because it provided a massive array of data collected by a distributed network of computerized devices collectively referred to as the Internet of Things. Its purpose was to serve as a low-cost, open-source fitness tracker for the city by providing granular-level data related to indicators of neighborhood health and activity. The devices, called nodes, each contained experimental sensors and a processor powered by the streetlights to which they were mounted. The processors interpreted the data received by the nodes and sent the findings over the cellular network to a centralized program, where it could be reviewed by scientists and researchers and ultimately released to the public. By August 2018, the city had installed 100 of 500 planned nodes citywide, and most of the sensors were still undergoing testing and calibration. For example, the nodes contained low-grade cameras, but the cameras were not yet activated even though the city had already started collecting data related to climate and air quality and publicly releasing it online via Plenario. OVERCOMING OBSTACLES In 2014, while Catlett was still seeking funding for the Array of Things, a media story said that the system's devices might feature sensors capable of detecting and counting unique wireless addresses in their vicinity, enabling his project to access information people kept on their cellphones. The first part of the statement was right. Given the near ubiquity of smartphones, the wireless addresses were a rough proxy for the number of people, so a sensor at a busy pedestrian intersection could gauge foot traffic, eliminating the need for laborious manual counts. The second part of the statement was wrong, however. There were a number of privacy protections in place. But the report fueled anxiety. The Chicago Tribune, the city's largest daily newspaper, published an article that discussed concerns among some privacy experts that the kinds of WiFi/Bluetooth-connected device counters, the kinds of sensors used by the Array of Things, could constitute an invasion of citizens' privacy, although the experts quoted had little information about the specific project.18 "People didn't like that [the WiFir/Bluetooth-connected sensors]; it crossed some sort of line," said Rob Mitchum, who led communications for the Array of Things at the University of Chicago. Catlett said the article mischaracterized the project, and he defended the proposed sensors as common in commercial use. However, h dropped the sensors from the proposal partly for technical reasons and partly because he saw how easily misunderstandings could develop. Instead, the project relied on cameras to collect data for tallying people or objects. Cameras in public spaces were legal, uncontroversial, and increasingly common. Indeed, the Chicago Police Department had reportedly installed more than 30,000 closed-circuit video cameras citywide.19 "When the story broke . . . , we got on our heels because we weren't thinking about press messaging yet," conceded Schenk. "This was crisis management, and we got it contained and controlled and moved on." Although the public controversy was short-lived, it was an early warning that the Array of Things raised more normative concerns than previous data analytics projects had. Schenk said the privacy storm prompted both his team and Catlett's team to put more forethought into community engagement and "got them to focus on things like privacy policies and governance." "Early on, security and privacy were top of mind for this project," agreed DuMerer. Because the Array of Things was unprecedented in scope, Catlett found no comparable models for privacy policies. He chanced upon warning notices posted by public cameras in Sydney that provided a website link with public reports and audits and privacy protocols. "That caused me to think about the concept in privacy policy that we would publish all the data we're pulling," Catlett said, "so that people know we're not doing anything secret." The project team built the Array of Things to minimize risks to privacy and security. The nodes had their own processors capable of analyzing raw data before transmission. The data itself would not be stored-except for periodic snapshots used in system calibration. For example, the device might analyze camera feeds to count vehicular traffic or examine acoustic feeds to determine ambient noise levels and then transmit its findings, but it would neither save nor transmit raw images or recordings. This structure not only cut data storage and transmission costs but also eliminated the chance of incidental capture of personally identifiable information like license plate numbers or private conversations. The University of Chicago owned and managed all data captured by the project, although the institution had an agreement with the city to make reviewed data open and available, something it had proposed from the beginning. The university had experience in the warehousing of sensitive data and had sophisticated legal and technical capacity for data protection. And the university's institutional review board ensured compliance with strict federal standards for ethics and security in academic research. To review proposed policy changes and ethics questions, Catlett and the city set up an executive committee that met quarterly. The Department of Innovation and Technology and Catlett's research center-the Urban Center for Computation and Data, a joint project of the University of Chicago and Argonne National Laboratory-were co-chairs. Project bylaws gave the city veto power and the authority to shut down the project at any time. The project director and the city's chief information officer also met biweekly. Two advisory boards provided specialized expertise for the oversight committee. A privacy and security group led by Von Welch, director of Indiana University's Center for Cybersecurity Research, governed privacy policies and ensured data protection. The group reviewed proposed sensors added to the nodes to minimize privacy and security risks. For example, if project leaders contemplated an experimental acoustic sensor to detect construction noise or birdsongs, the group would make sure its design and use prevented the incidental monitoring or recording of private conversations. A scientific review group of scholars and scientists reviewed requests for unreleased data for valid research purposes. Catlett and the city data team crafted a privacy policy in 2015, and refined it after a workshop with relevant experts in early 2016. Then the team spent six months soliciting public feedback. Smart Chicago led a series of public meetings in libraries and community centers in neighborhoods where the first sensors would be placed, and it also set up a website to collect feedback online and by e-mail. Smart Chicago also used a Web-based document review tool to enable advocacy groups and concerned citizens to submit comments and edits.20 The project team released a final policy on the Array of Things website and included provision for a six-month public comment period prior to any substantial changes to the project.21 Denise Riedl, who managed Array of Things public engagement for Smart Chicago, said the experience showed the importance of "meeting people where they are." She found that low-technology approaches like posting and distributing flyers were effective tools for raising awareness in underserved neighborhoods. The group hosted evening information sessions in public libraries, churches, and community centers, including holding a bilingual session in the mostly Hispanic neighborhood of Pilsen, with translators and Spanish-language materials. Some of the meeting participants were initially skeptical of the project but became generally supportive once it was explained, according to the final report released by Smart Chicago.22 "There were certainly some questions about privacy, but actual response from the community was mostly positive," Riedl said. "There was almost a 50/50 breakdown of technical and logistical questions versus normative issues. . . . People were interested in the use cases and potential-but in the technology, too." Project leaders answered all questions received in person or online. ASSESSING RESULTS The data-driven approaches pioneered by Chicago's data team would take years to gauge fully. "The projects have long stretches with very incremental progress," Thornton said. "After some early wins, you gain some momentum, you gain some partners, some funders. It's very much a long game." O'Neil gave the city credit for releasing open data but said the city's data analytics work was similar to previous generations' efforts to optimize service delivery. Early results showed measurable gains, however. "We're targeting efficiency, and there's always some sort of metric," said Schenk. "With food inspections, we're finding critical violations 25% faster. Our predictions of West Nile virus are 80% accurate and a week ahead." The city's data projects also led to technology transfers and spin-offs. Other cities and government agencies adapted the open-source software used for improving restaurant inspections and other policies. As more and more cities established the position of chief data officer, Schenk helped share knowledge with colleagues in other cities and discussed common approaches and concerns. With support from Harvard Kennedy School's Ash Center for Democratic Governance and Innovation and the Laura and John Arnold Foundation, Schenk started a professional network of CDOs called the Civic Analytics Network.23 Schenk likened the network to a trade association, saying it enabled members to "use collective power to shape industry." For example, the network could advocate for better data policies or use collective bargaining power with vendors. The data office's strong and productive partnerships showed the success of outreach efforts to the civic tech community, academia, the private sector, and other government offices. Schenk's "work to integrate the city with the community is laudable," said Engler. "He came into Chi Hack Night and gave a presentation about the work they were doing, and he got a rare, standing ovation from the civic tech community." However, even Schenk saw shortfalls in his office's broader public engagement efforts. "This is an area where we started off pretty weak but got better," he conceded, citing increasing engagement with everyday Chicagoans through social media and increased usage of the data portal after a user-centric redesign in 2017. In 2018, the office hired a design director to bring a user-oriented perspective to its projects and services. Besides the short-lived uproar over an early proposal for the Array of Things, the city's data projects provoked no significant scandals or controversies. The public quiescence was a relief for the city as the Emanuel administration, as well as the technology sector, faced several controversies.24 As the city's data team undertook increasingly ambitious and sophisticated data analytics projects like Lead Safe, it made uneasy ethical trade-offs. "How do you get to the outcome you're looking for-the reduction of exposure to lead paint-while also respecting the privacy of the residents you're trying to protect?" DuMerer asked. "It is very personal. . . . It is your home, your children." She believed the city was right to work with health-care providers to encourage pregnant women and parents of young children to take appropriate steps, including working with the city. Engler, who was not involved with the project, was also confident the city had struck the right balance: "There are a lot of scenarios where you could sell me 110% on the ethical issues and the privacy issues, but we can't send newborns back to houses with lead paint in them. . . . The outcomes are just so damning, and the level of intrusion isn't high." He said the level of intrusion was relatively low, given the stakes. The Array of Things drew a variety of responses. Some critics said that it showed an overabundance of caution. O'Neil suggested that rather than nitpick about potential privacy concerns, the developers of the Array of Things should turn the project into a tool run by-and responsive to-the people of Chicago. "An object like this could actually be controlled by the people-a community resource instead of a police resource," he said. For example, its cameras and sensors could help resolve cases of police shootings and unsolved murders, among other social justice concerns. Indeed, Smart Chicago found in its public engagement meetings that citizens more often expressed disappointment that the Array of Things would not help enforce law and order than they expressed concern over its intrusiveness. "I wish we were more explicit about what the project doesn't do and what isn't technically possible right now, so we can manage expectations and avoid confusion," Riedl said. DuMerer carefully distinguished the Array of Things as a research project and not a public safety tool but said the project's data would ultimately help address some of Chicago's equity issues. For example, the Department of Water Management planned for the project to immediately identify standing water in flood-prone marginal communities on the city's West Side without having to wait for citizen complaints. The Department of Public Health was interested in air quality monitoring so it could correct environmental injustice. Thornton noted that an array of hundreds of real-time sensors would aid in the understanding of ways localized environmental issues like air and sound pollution affected nearby residents. Previously, the city had relied on a handful of local monitoring stations-typically managed by the state or federal government because of the high cost of preexisting sensors. "Social determinants of health have become a huge focus in the health-care sector," said Thornton. As the nodes got outfitted with more and better sensors and more-sophisticated processors, project supporters said the flow of data would give city leaders-and residents-an increasingly detailed picture of their city. "The likelihood that this data leads to questions we don't have answers to is high," said Engler, "especially in a data-rich city like Chicago, where there are many other data sources this data could be integrated with." Eder was cautiously optimistic that the Array of Things could prove useful even if it originated as a scientific research project rather than a response to a policy problem. "Plenty of amazing things in our society started that way," he said. "Nobody asked for the Internet, and now we use it every day." However, he added, the Array of Things project could do more to serve public interest. "I would love to find ways to empower more citizen science or advocacy," he said. "The Array of Things could do that, but it seems to be more about setting up an infrastructure that can be used by the city. The data may be made public, but it seems like the primary users will be the city and the university." O'Neil criticized the Array of Things more sharply. "In the main, the Array of Things is a useless toy vanity project of the richest university in the region," he said. "It does nothing to help people. The [project leaders] have failed in their community objectives. They do not publish the data. They do not meaningfully engage with people. They avoid all the tough decisions that could actually reduce violence or improve relationships with people who worried every day about dying." Engler stressed that the impact of technological projects should be gauged against a baseline of inaction, which reinforced existing inequities. "Without collecting information, services are slanted toward people with the resources to be involved," he said. "If you're not tracking problems methodically, the likelihood that outcomes would get distributed equitably in the first place is near zero." REFLECTIONS Although the data analytics work initiated by Chicago's chief data officers Brett Goldstein and Tom Schenk was still in early stages by 2018, several factors of success were already clear. Both Goldstein and Schenk credited their office's success to having a strong team with executive support from the mayor's office. "You need a strong executive sponsor," Goldstein said. "If you're nested in a branch with little authority or purview, it's problematic." Partnerships with private, academic, and civic institutions brought unique assets to the table-beyond material resources. Such partnerships were less constrained by rules and bureaucratic protocols like having to preserve documentation and comply with requests based on the federal Freedom of Information Act. They brought new potential for technology transfers and spin-offs to other sectors and other cities. And they had higher levels of public trust in a diverse city like Chicago, which helped the city make programs like Lead Safe and the Array of Things less intrusive. As local journalist Sean Thornton said about the Array of Things: "When people think it's the government driving this thing, it sounds very Big Brother; but if it's coming from Argonne [National Laboratory], that's another story. If you look at these nodes, they're measuring things like sound and air temperature and pollution. It's science nerd stuff. There's nothing Big Brother about it!" Chicago's data office also avoided controversy by integrating privacy concerns into the architecture of its projects. For example, the team worked with open data whenever possible, and it anonymized and aggregated confidential data to make it less sensitive. The team headed by Charlie Catlett, a computer scientist at Argonne National Laboratory, had designed the Array of Things to avoid collecting or storing personally identifiable information. "Don't collect data you can't secure," University of Chicago data scientist Alex Engler warned data officers. Daniel O'Neil, who led the Smart Chicago Collaborative, advised cities considering implementation of data analytics projects to listen carefully to public input to make sure projects served the public interest rather than merely the interests of government officials. "Try to solve real problems by having methods to discover what the real problems are," O'Neil said. "Create systems to listen to people in honest ways." O'Neil conceded the difficulty of soliciting input from citizens who lacked technical expertise or advocacy tools, but felt that investment in genuine public engagement would result in more-relevant and more-effective projects that responded to popular concerns. "We have to build and invest in methods for listening." Schenk, from his experience in working with other city data officers through the Civic Analytics Network, cautioned that what worked in Chicago wouldn't necessarily work elsewhere. Even within the United States, cities varied in their laws, government structures, and civic cultures. "Privacy is very contextual to the constituency," he said. In another city, the Array of Things might provoke more wariness. On the other hand, Schenk admired the strengths he saw in other cities such as interagency collaboration in New Orleans, strong data-warehousing plans in Boston, comprehensive public policies in San Francisco, and a unique separation between open-data and smart-city programs in Seattle. Denise Riedl, who led Smart Chicago's public engagement efforts, argued that Chicago's data strategy also depended on a robust local tradition of civic engagement. "One thing I love about Chicago is that people feel they have the right to ask questions," she said. "There's an active civic culture. People like public meetings." Based on her experience in running public engagement meetings for the Array of Things, Riedl encouraged a more participatory approach to urban-technology design. "Innovators don't have to be separate from everyone else," she said. "Innovation can and should be decentralized, informed by everyone." Engler agreed: "If you come in and say, 'I'm here with all my great tech skills, and I'm going to solve problems,' the likelihood that you solve anything is spectacularly low. . . . Cities should build in stakeholders from the very beginning." He cited the motto of civic technology pioneer Laurenellen McCann, who advocated for "community leadership and stewardship" in the application of public-interest technology: "Build with, not for."25 APPENDIX 1: CHICAGO'S CIVIC TECHNOLOGY MOVEMENT Civic tech grew out of the open-data and open-source movements of the early 2000s, which promoted transparency, accessibility, and collaboration in technology development. Supporters included social activists and computer engineers who said that participatory, grassroots approaches would solve community problems better than would the top-down methods they perceived in the private, public, and academic sectors. "There was a catalyzing moment when folks like myself thought, 'I'm a [software] developer; I can code; I can work with data; I can do good with this data,'" recalled Derek Eder, who worked with other local programmers to create Chicago Lobbyists in 2011. The website took lobbyist disclosures newly released on Chicago's open-data portal and presented the data in attractive, user-friendly, and useful ways.26 The site caught the eye of city leaders and local media.27 Civic tech proponents often called themselves "hackers" in a positive sense, drawing on technical skills, available resources, and group interaction to create clever solutions to specific problems. Gatherings like civic hackathons and Chi Hack Night evoked the collegial, informal, and freewheeling spirit of a dorm-room bull session in which participants considered and refined one another's ideas without passing judgment or hogging credit. Chi Hack Night, which Eder cofounded in 2011, provided a weekly gathering point for those interested in civic tech. "Having a consistent place over time gave us the ability to grow a community," Eder said. "People knew where to go, and we had the space to work on things and build up institutional knowledge . . . It was a creative time when suddenly there was a lot of newly available data, and we saw a lot of opportunities to do things with it-all volunteer and self-started. . . . In the early days, we were just fumbling around to make something cool. But as we learned more about the city, I got a real education on how the city of Chicago works, the underlying problems, and places of leverage." For example, one Chi Hack Night participant expressed concern that her landlord didn't provide recycling, even though the city required owners of multiunit apartment buildings to do so. Fellow volunteers helped her create a website through which users could report whether their buildings lacked recycling and could take actions like sending letters to city officials.28 The site mapped user-submitted violations into a clear picture of widespread underenforcement. The issue drew media attention, and the government passed a tougher recycling ordinance. The woman testified before the city council in favor of the new legislation, using the data collected by her website as evidence. Eder said: "It's pretty awesome to have a problem, build an app, get a law passed. It doesn't always happen that way. . . . [But if] technology can apply a useful lever, . . . make an impact, provoke a government response-that's a powerful thing." In another instance, a community organizer from the city's historically underserved South Side presented at a Chi Hack Night an issue involving toxic dust that was blowing from nearby open stockpiles of petroleum coke. A local civic tech nonprofit called Public Lab helped set up sensors and drones to collect evidence of the hazard. And Chi Hack Night volunteers built a tool to monitor weather station data and notify residents' phones when winds were likely to spread the dust to their homes. Under public pressure, the city passed an ordinance to remove the piles.29 Despite some success stories, the wider impact of the civic tech movement was less clear. Although gatherings like Chi Hack Night were open to everyone and diverse in ideas, participants represented mainly tech enthusiasts rather than members of Chicago's disparate constituencies. Daniel O'Neil, who led the Smart Chicago Collaborative, called Chi Hack Night a "great career-development meet-up for people in technology to build their résumés." He lamented that Chicago's civic tech scene had not grown beyond a club of like-minded people into a broader, more open, and interconnected network. "We built a great community, but we failed to create an ecosystem," he said. Eder conceded that the community was still evolving. "In the past year, we have focused on inclusion and diversity," he said, "making sure the people presenting and organizing are diverse." He contended that despite shortcomings, civic tech had helped Chicago's citizens advocate for better government: "You're not going to make Facebook on a Tuesday night, but you can make a simple website that tells a compelling story and that puts data out there." Innovations for Successful Societies makes its case studies and other publications available to all at no cost, under the guidelines of the Terms of Use listed below. The ISS Web repository is intended to serve as an idea bank, enabling practitioners and scholars to evaluate the pros and cons of different reform strategies and weigh the effects of context. ISS welcomes readers' feedback, including suggestions of additional topics and questions to be considered, corrections, and how case studies are being used: iss@princeton.edu. Terms of Use Before using any materials downloaded from the Innovations for Successful Societies website, users must read and accept the terms on which we make these items available. The terms constitute a legal agreement between any person who seeks to use information available at successfulsocieties.princeton.edu and Princeton University. In downloading or otherwise employing this information, users indicate that: a. They understand that the materials downloaded from the website are protected under United States Copyright Law (Title 17, United States Code). This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. b. They will use the material only for educational, scholarly, and other noncommercial purposes. c. They will not sell, transfer, assign, license, lease, or otherwise convey any portion of this information to any third party. Republication or display on a third party's website requires the express written permission of the Princeton University Innovations for Successful Societies program or the Princeton University Library. d. They understand that the quotes used in the case study reflect the interviewees' personal points of view. Although all efforts have been made to ensure the accuracy of the information collected, Princeton University does not warrant the accuracy, completeness, timeliness, or other characteristics of any material available online. e. They acknowledge that the content and/or format of the archive and the site may be revised, updated or otherwise modified from time to time. f. They accept that access to and use of the archive are at their own risk. They shall not hold Princeton University liable for any loss or damages resulting from the use of information in the archive. Princeton University assumes no liability for any errors or omissions with respect to the functioning of the archive. g. In all publications, presentations or other communications that incorporate or otherwise rely on information from this archive, they will acknowledge that such information was obtained through the Innovations for Successful Societies website. Our status (and that of any identified contributors) as the authors of material must always be acknowledged and a full credit given as follows: Author(s) or Editor(s) if listed, Full title, Year of publication, Innovations for Successful Societies, Princeton University, http://successfulsocieties.princeton.edu/ (c) 2018, Trustees of Princeton University References 1 Jim Kirk et al., "The Daley Legacy, Inescapable," New York Times, September 12, 2010. Accessed at https://www.nytimes.com/2010/09/12/us/12cncweblegacy.html. 2 Chicago 2011 Transition Plan, City of Chicago, May 10, 2011. Accessed at https://www.cityofchicago.org/dam/city/depts/mayor/supp_info/chicago_2011_transition_report.pdf. 3 Sasha Issenberg, "How Obama's Team Used Big Data to Rally Voters," MIT Technology Review, December 19, 2012. Accessed at https://www.technologyreview.com/s/509026/how-obamas-team-used-big-data-to-rally-voters/. 4 Nancy Scola, "Obama, the 'big data' president," Washington Post, June 14, 2013. Accessed at https://www.washingtonpost.com/opinions/obama-the-big-data-president/2013/06/14/1d71fe2e-d391-11e2-b05f-3ea3f0e7bb5a_story.html?utm_term=.f462c5e4f727. 5 "Solyndra Scandal," Washington Post. Accessed at https://www.washingtonpost.com/politics/specialreports/solyndra-scandal/; Joshua Bleiberg and Darrell M. West, "A look back at technical issues with Healthcare.gov," Brookings Institution, April 9, 2015. Accessed at https://www.brookings.edu/blog/techtank/2015/04/09/a-look-back-at-technical-issues-with-healthcare-gov/. 6 E.g., "SmartData Platform: Solving Problems before They Start," Bloomberg Philanthropies. Accessed at https://mayorschallenge.bloomberg.org/ideas/the-chicago-smartdata-platform/; Smart Cities International Symposium & Exhibition. Accessed at http://www.smartcities-symposium.com/; Brian Buntz, "Why Chicago Is a Smart City King," IoT World Today, May 4, 2016. Accessed at http://www.iotworldtoday.com/2016/05/04/why-chicago-smart-city-king/. 7 Joe Flood, The Fires: How a Computer Formula, Big Ideas, and the Best of Intentions Burned Down New York City-and Determined the Future of Cities, Riverhead Books, 2011. 8 Bob Goldsborough, "Chicago is the 13th most segregated metro area in US, study finds," Chicago Tribune, May 25, 2018. Accessed at http://www.chicagotribune.com/classified/realestate/ct-re-0603-housing-segregation-20180525-story.html; Alana Semuels, "Chicago's Awful Divide," The Atlantic, March 28, 2018. Accessed at https://www.theatlantic.com/business/archive/2018/03/chicago-segregation-poverty/556649/. 9 Craig Wall, "Report: Chicago most corrupt city in US," ABC 7 Eyewitness News, May 15, 2018. Accessed at https://abc7chicago.com/politics/report-chicago-most-corrupt-city-in-us/3478310/; James L. Merriner, Grafters and Goo Goos: Corruption and Reform in Chicago, Southern Illinois University Press, 2004. 10 The City of Chicago Technology Plan, City of Chicago. Accessed at https://techplan.cityofchicago.org/wp-content/uploads/2013/09/cityofchicago-techplan.pdf. 11 Adam Greenfield, Against the Smart City, Verso, January 2013. 12 Chicago Data Portal website. Accessed at https://data.cityofchicago.org/. 13 Plenario website. Accessed at http://plenar.io/. 14 OpenGrid website. Accessed at http://opengrid.io/. 15 Daniel X. O'Neil, The CUTGroup, Smart Chicago Collaborative. Accessed at http://www.cutgroupbook.org/. 16 Sean Thornton, "Making Chicago's Beach Water Safer with Analytics," Ash Center for Democratic Governance and Innovation, September 12, 2017. Accessed at https://datasmart.ash.harvard.edu/news/article/making-chicagos-beach-water-safer-with-analytics-1115. 17 For more on the program, see Chicago Smart Lighting Program website. Accessed at http://chicagosmartlighting-chicago.opendata.arcgis.com/. 18 David Heinzmann, "New sensors will scoop up 'big data' on Chicago," Chicago Tribune, June 20, 2014. Accessed at http://articles.chicagotribune.com/2014-06-20/news/ct-big-data-chicago-20140621_1_cell-phone-data-big-data-sensors. 19 Timothy Williams, "Can 30,000 Cameras Help Solve Chicago's Crime Problem?" New York Times, May 26, 2018. Accessed at https://www.nytimes.com/2018/05/26/us/chicago-police-surveillance.html. 20 Madison website. Accessed at https://mymadison.io/. 21 Array of Things privacy policy, 2016. Accessed at https://arrayofthings.github.io/privacypolicy.html. 22 Denise Linn and Glynis Startz, Array of Things Civic Engagement Report, Smart Chicago Collaborative, August 2016. Accessed at https://www.slideshare.net/smartchicago/array-of-things-engagement-report-august-2016. 23 Civic Analytics Network website. Accessed at https://ash.harvard.edu/civic-analytics-network 24 Rick Perlstein, "The Sudden but Well-Deserved Fall of Rahm Emanuel," New Yorker, December 31, 2015. Accessed at https://www.newyorker.com/news/daily-comment/the-sudden-but-well-deserved-fall-of-rahm-emanuel; Avery Hartmans, "The 17 biggest tech scandals of 2017," Business Insider, December 30, 2017. Accessed at https://www.businessinsider.com/biggest-tech-scandals-2017-12. 25 Build With, Not For website. Accessed at https://buildwithnotfor.wordpress.com/. 26 Chicago Lobbyists website. Accessed at http://www.chicagolobbyists.org. 27 Chuck Sudo, "New Web Project Simplifies City's Open Lobbying Data," Chicagoist, December 7, 2011. Accessed at http://chicagoist.com/2011/12/07/new_web_project_simplifies_citys_op.php. 28 My Building Doesn't Recycle website. Accessed at http://mybuildingdoesntrecycle.com/. 29 Michael Hawthorne, "Petcoke piles gone, but another dangerous pollutant discovered in the air," Chicago Tribune, February 21, 2017. Accessed at http://www.chicagotribune.com/news/local/breaking/ct-manganese-pollution-chicago-met-20170218-story.html. ISS is program of the Woodrow Wilson School of Public and International Affairs: successfulsocieties.princeton.edu. ISS invites readers to share feedback and information on how these cases are being used: iss@princeton.edu. (c) 2018, Trustees of Princeton University. This case study is made available under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Gabriel Kuris Innovations for Successful Societies (c)2018, Trustees of Princeton University Terms of use and citation format appear at the end of this document and at successfulsocieties.princeton.edu/about/terms-conditions.