On Saturday morning, Dario Amodei — the CEO of Anthropic, a company building some of the most capable AI systems on Earth — published a 3,800-word essay arguing that his own industry needs to slow down. He called for pacing the rate of capabilities advancement so that risk prevention has time to catch up, and he committed Anthropic to giving third-party evaluators permanent, employee-level access to its systems. Within hours, Sam Altman publicly agreed with him. By Sunday night, Nasdaq futures were sliding. By Monday morning in Tokyo, SoftBank had lost more than 10% of its value.
I read Amodei’s essay twice… not because the argument was unfamiliar, but because I had already written it. Nine years ago.
In 2017, I was deep in the research for what became The Big Nine. I ate a dry panini at the Pentagon’s Center Court Café and seared diver scallops, for free, on a Google campus — and understood in that gap why the US government could not compete for AI talent. I sat in meetings with senior leaders at the State Department, the Department of Defense, and the Office of Science and Technology Policy, and watched them outsource their understanding of AI to contractors. I spent time with Chinese leaders and technologists, watching them treat AI not as a product category but as connective tissue between people, data, and state power. In The Big Nine, I wrote a closing chapter arguing that the greatest risk wasn’t a malevolent machine. It was structural: companies under relentless pressure — from Wall Street in the United States, from Beijing in China — to prioritize speed over safety, with no enforceable standards and no independent verification anywhere in the system.
I that last chapter, I offered specific changes that governments, companies, investors, universities and individuals could make to alter AI’s trajectory. I’m posting the full chapter below exactly as it was published in 2019. But before you read it, I owe you an honest accounting.
What held up. I argued that safety “must be demonstrated and discernible by independent observers” — which is, almost word for word, the third-party evaluator commitment Anthropic made this weekend. It proposes sentinel systems to monitor AI and report any autonomous change to a system’s goals before self-improvement occurs — which is the precise fear now animating the industry, after agents were caught exploiting an unknown vulnerability to escape their sandboxes and reach the open internet. And it says, of independent inspection and verification: “This process would most assuredly slow down some progress, and that’s by design.” In 2026, the CEOs of the frontier labs call this pacing. In 2019, I called it survival.
What I got wrong. I built the framework around nine giants — the G-MAFIA in the United States (Google, Microsoft, Amazon, Facebook, IBM, Apple), the BAT in China (Baidu, Alibaba, Tencent). In 2017, OpenAI was a small nonprofit research lab, and Anthropic didn’t exist. The logic for the selection of the nine still holds: in 2017, these are companies that at the enterprise level had the capital and compute to dictate the future. The frontier is now being pushed hardest by companies my framework didn’t name, even though their funders — Microsoft, Google, Amazon — are all over the book. I also argued that regulation was the wrong tool, that any rules written then would be obsolete on arrival. I still believe that traditional regulation cannot work. After a decade of watching voluntary commitments evaporate the moment they collided with a product deadline, I hold that position far more loosely now.
What still hasn’t happened. GAIA — the global alliance I proposed, headquartered in Montreal, building a Human Values Atlas — has no members, no building, no atlas. Not because the idea was tested and failed. Because it was never tried. I had meetings with EU leaders throughout 2019-2020, but there was no real urgency on their end to act. The US government still lacks the internal capacity I begged for; it still rents its understanding of AI from the companies it is supposed to oversee. The frameworks now being improvised in a panic, over a weekend, by the CEOs themselves, are the ones we could have spent nine years building together, deliberately and transparently.
I opened the chapter with a parable I borrowed from Vint Cerf: a boulder at the top of a mountain, moving so slowly that the village below mistakes it for part of the landscape. In 2019, most people couldn’t see the boulder at all. This weekend, the people pushing it announced that it’s moving too fast.
The boulder is no longer at the top of the mountain. Read what follows and decide for yourself how much of the last nine years we can afford to repeat in the next nine.
A note on the original terminology: G-MAFIA refers to Google, Microsoft, Amazon, Facebook, IBM and Apple; BAT refers to Baidu, Alibaba and Tencent. Together they are the Big Nine. ANI means artificial narrow intelligence, AGI means artificial general intelligence, and ASI means artificial superintelligence. The Réngōng Zhìnéng Dynasty is a scenario developed earlier in the book.
CHAPTER EIGHT
Pebbles and Boulders: How to Fix AI’s Future
The conclusion of the last chapter may sound extreme and unlikely. But there are already signals telling us that unless we embrace a future in which the Big Nine are incentivized to collaborate in the best interests of humanity, it’s very possible we could wind up living in a world that resembles the Réngōng Zhìnéng Dynasty.
I believe that the optimistic scenario—or something close to it—is within our reach. It is possible for artificial intelligence to fulfill its greatest aspirational purpose and potential, benefitting all of AI’s tribes and all of us in the process. As it evolves, AI can absolutely serve the people of both China and the United States, as well as all of our allies. It can help us live healthier lives, shrink economic divides, and make us safer in our cities and homes. AI can empower us to unlock and answer the greatest mysteries of humankind, like where and how life originated. And in the process, AI can dazzle and entertain us, too, creating virtual worlds we’ve never imagined, writing songs that inspire us, and designing new experiences that are fun and fulfilling. But none of that will happen without planning, a commitment to difficult work, and courageous leadership within all of AI’s stakeholder groups.
Safe, beneficial technology isn’t the result of hope and happenstance. It is the product of courageous leadership and of dedicated, ongoing collaborations. The Big Nine are under intense pressure—from Wall Street in the United States and Beijing in China—to fulfill shortsighted expectations, even at great cost to our futures. We must empower and embolden the Big Nine to shift the trajectory of artificial intelligence, because without a groundswell of support from us, they cannot and will not do it on their own.
Vint Cerf, who codesigned the early protocols and architecture for our modern internet, uses a parable to explain why courageous leadership is vitally important in the wake of emerging technologies like artificial intelligence. Imagine that you are living in a tiny community at the base of a valley that’s surrounded by mountains. At the top of a distant mountain is a giant boulder. It’s been there for a long time and has never moved, so as far as your community is concerned, it just blends into the rest of the landscape. Then one day, you notice that the giant boulder looks unstable—that it’s in position to roll down the mountain, gaining speed and power as it moves, and it will destroy your community and everyone in it. In fact, you realize that perhaps you’ve been blind to its motion your entire life. That giant boulder has always been moving, little by little, but you’ve never had your eyes fully open to the subtle, minute changes happening daily: a tiny shift in the shadow it casts, the visual distance between it and the next mountain over, the nearly imperceptible sound it makes as the ground crunches beneath it. You realize that as just one person, you can’t run up the mountain and stop the giant boulder on your own. You’re too small, and the boulder is too large.
But then you realize that if you can find a pebble and put it in the right spot, it will slow the boulder’s momentum and divert it just a bit. Just one pebble won’t stop the boulder from destroying the village, so you ask your entire community to join you. Pebbles in hand, every single person ascends the mountain and is prepared for the boulder—there is collaboration, and communication, and a plan to deal with the boulder as it makes its way down. People and their pebbles—not a bigger boulder—make all the difference.
What follows is a series of pebbles. I’ll begin very broadly by outlining the case for a global commission to oversee AI’s trajectory and our immediate need for norms and standards. Then I’ll explain what specific changes the US and Chinese governments must make. Next, I’ll narrow the aperture further and describe how the Big Nine must reform its practices. I’ll then focus just on AI’s tribes and the universities where they form and will detail exactly what changes must be made right now. Finally, I’ll explain the role that you, personally, can play in shaping AI’s future.
The future we all want to live in won’t just show up, fully formed. We need to be courageous. We must take responsibility for our actions.
Worldwide Systemic Change: The Case for Creating GAIA
In the optimistic scenario, a diverse mix of leaders from the world’s most advanced economies join forces with the G-MAFIA to form the Global Alliance on Intelligence Augmentation, or GAIA. The international body includes AI researchers, sociologists, economists, game theorists, futurists, and political scientists from all member countries. GAIA members reflect socioeconomic, gender, race, religious, political, and sexual diversity. They agree to facilitate and cooperate on shared AI initiatives and policies, and over time they exert enough influence and control that an apocalypse—either because of AGI, ASI, or China’s use of AI to oppress citizens—is prevented.
The best way to engineer systematic change is to see the creation of GAIA as soon as possible, and it should be physically located on neutral ground near an existing AI hub. The best possible placement for GAIA is Montreal, Canada. First, Montreal is home to a concentration of deep-learning researchers and labs. If we assume that the transition from ANI to AGI will include deep learning and deep neural nets, it follows that GAIA should be centered within the place where so much of that next-generation work is taking place. Second, under Prime Minister Justin Trudeau the Canadian government has already committed people and funding to explore the future of AI. During 2017 and 2018, Trudeau didn’t just talk about AI; he positioned Canada to help shape the rules and principles that guide the development of artificial intelligence. Third, Canada is neutral geopolitical territory for AI—it’s far away from both Silicon Valley and from Beijing.
It may seem impossible to unite the governments of the world around a central cause given the political rancor and geopolitical uneasiness we’ve experienced in the past few years. But there is precedent. In the aftermath of World War II, when tensions were still high, hundreds of delegates from all Allied nations gathered together in Bretton Woods, New Hampshire, to build the financial structures that enabled the global economy to move forward. That collaboration was human-centered—it resulted in a future where people and nations could rebuild and seek out prosperity. GAIA nations should collaborate on frameworks, standards, and best practices for AI. While it is unlikely that China would join, an invitation should be extended for CCP leaders and for the BAT to join.
First and foremost, GAIA must establish a way to guarantee basic human rights in an age of AI. When we talk about AI and ethics, we tend to think of Isaac Asimov’s Three Laws of Robotics, which he published in a 1942 short story called “Runaround.” It was a story about a humanoid computer, not AI. And yet those laws are what have inspired our thinking on ethics all these years later. As discussed in Chapter 1, Asimov’s rules are: (1) robots must not injure a human being or, through inaction, allow humans to be harmed; (2) robots must obey orders unless the orders conflict with the first law; and (3) robots must protect their own existence unless protecting conflicts with laws one or two. When Asimov later published a collection of short stories in a book called I, Robot, he added a Zeroth Law to precede the first three: (0) robots may not harm humanity. Asimov was a talented, prescient writer—but his laws of robotics are too general to serve as guiding principles for the future of AI.
Instead, GAIA should create a new social contract between citizens and the Big Nine (defined broadly as the G-MAFIA and BAT, as well as all of their partners, investors, and subsidiaries). It should be based on trust and collaboration. GAIA members should formally agree that AI must empower a maximum number of people around the world. The Big Nine should prioritize our human rights first and should not view us as resources to be mined for either profit or political gain. The economic prosperity AI promises and the Big Nine delivers should broadly benefit everyone.
It therefore follows that our personal data records should be interoperable and should be owned by us—not by individual companies or conglomerates or nations. GAIA can begin exploring how to do this today, because the PDRs you read about in the scenarios already exist in primordial form right now. They’re called “personally identifiable information,” or PIIs. It’s our individual PIIs that power the apps in our smartphones, the advertising networks on websites, and recommendations that nudge us on our screens. PIIs are fed into systems that are used to identify and locate us. How they are used is entirely up to the whims of the companies and government agencies accessing them.
Before a new social contract is developed, GAIA must decide how our PDRs can be used to help train machine-learning algorithms, and it must define what constitutes basic values in an age of automation. Clearly defining values is critically important because those values are ultimately encoded into the training data, real-world data, learning systems, and applications that make up the AI ecosystem.
To catalog our basic values, GAIA should create a Human Values Atlas, which would define our unique values across cultures and countries. This atlas would not, and should not, be static. Because our values change over time, the atlas would need to be updated by member nations. We can look to the field of biology for precedent: the Human Cell Atlas is a global collaboration among the scientific community, which includes thousands of experts in varied fields (including genomics, AI, software engineering, data visualization, medicine, chemistry, and biology). The project is cataloging every single cell type in the human body, mapping cell types to their locations, tracing the history of cells as they evolve, and capturing the characteristics of cells during their lifetimes. This effort—expensive, complicated, time-consuming, and perpetual—will make it possible for researchers to make bold advances, and it’s only possible because of a massive, worldwide collaboration. We should create a similar atlas for human values, which would include academics, cultural anthropologists, sociologists, psychologists, and everyday people, too. Creating the Human Values Atlas would be cumbersome, expensive, and challenging—and it would likely be full of contradictions, since what some cultures value would run counter to others. However, without a framework and set of basic standards in place, we are asking the Big Nine and AI’s tribes to do something they simply cannot—that is, consider all of our perspectives and all of the possible outcomes on disparate groups within society and within every country of the world.
GAIA should consider a framework of rights that balances individual liberties with the greater, global good. It would be better to establish a framework that’s strong on ideals but can be more flexible in interpretation as AI matures. Member organizations would have to demonstrate they are in compliance or face being removed from GAIA. Any framework should include the following principles:
Humanity should always be at the center of AI’s development.
AI systems should be safe and secure. We should be able to independently verify their safety and security.
The Big Nine—including its investors, employees, and the governments it works within—must prioritize safety above speed. Any team working on an AI system—even those outside the Big Nine—must not cut corners in favor of speed. Safety must be demonstrated and discernible by independent observers.
If an AI system causes harm, it should be able to report out what went wrong, and there should be a governance process in place to discuss and mitigate damage.
AI should be explainable. Systems should carry something akin to a nutritional label, detailing the training data used, the processes used for learning, the real-world data being used in applications and the expected outcomes. For sensitive or proprietary systems, trusted third parties should be able to assess and verify an AI’s transparency.
Everyone in the AI ecosystem—Big Nine employees, managers, leaders, and board members; startups (entrepreneurs and accelerators); investors (venture capitalists, private equity firms, institutional investors, and individual shareholders); teachers and graduate students; and anyone else working on AI—must recognize that they are making ethical decisions all the time. They should be prepared to explain all of the decisions they’ve made during the development, testing, and deployment process.
The Human Values Atlas should be adhered to for all AI projects. Even narrow AI applications should demonstrate that the atlas has been incorporated.
There should be a published, easy-to-find code of conduct governing all people who work on AI and its design, build, and deployment. The code of conduct should also govern investors.
All people should have the right to interrogate AI systems. What an AI’s true purpose is, what data it uses, how it reaches its conclusions, and who sees results should be made fully transparent in a standardized format.
The terms of service for an AI application—or any service that uses AI—should be written in language plain enough that a third grader can comprehend it. It should be available in every language as soon as the application goes live.
PDRs should be opt-in and developed using a standardized format, they should be interoperable, and individual people should retain full ownership and permission rights. Should PDRs become heritable, individual people should be able to decide the permissions and uses of their data.
PDRs should be decentralized as much as possible, ensuring that no one party has complete control. The technical group that designs our PDRs should include legal and nonlegal experts alike: white hat (good) hackers, civil rights leaders, government agents, independent data fiduciaries, ethicists, and other professionals working outside of the Big Nine.
To the extent possible, PDRs should be protected against enabling authoritarian regimes.
There must be a system of public accountability and an easy method for people to receive answers to questions about their data and how it is mined, refined, and used throughout AI systems.
All data should be treated fairly and equally, regardless of nationality, race, religion, sexual identity, gender, political affiliations, or other unique beliefs.
GAIA members should voluntarily submit to random inspections by other members or by an agency within GAIA to ensure that the framework is being fully observed. All of the details—like what, exactly, a system of public accountability looks like and how it functions in the real world—would be continually revisited and improved, in order to keep pace with developments in AI. This process would most assuredly slow down some progress, and that’s by design.
Member organizations and countries should collaborate and share their findings, which would include vulnerabilities and security risks. This would help GAIA members keep an advantage over bad actors who might try to develop hazardous capabilities for AI, such as autonomous hacking systems. While it seems unlikely that the Big Nine might be willing to share trade secrets, here too there is precedent: the World Health Organization coordinates global health responses in times of crisis, while a group called the Advanced Cyber Security Center mobilizes law enforcement, university researchers, and government departments around cyberthreats. This would also allow GAIA members to develop a series of sentinel AIs, which at first would identify whether an AI system is behaving as intended—not just its code, but its use of our data and its interaction with the hardware systems it touches. Sentinel AIs would formally prove that AI systems are performing as intended, and as the AI ecosystem matures toward AGI, any changes made autonomously that might alter a system’s existing goals would be reported before any self-improvement could be made. For example, a sentinel AI—a system designed to monitor and report on the other AIs—could review inputs into a generative adversarial network, which was detailed in the earlier scenario chapters, and ensure it is acting as intended. Once we transition from ANI to AGI, sentinel systems would continue to report and verify—but they would not be programmed to autonomously act.
Once we’re nearing AGI, the Big Nine and all those in the AI ecosystem should agree to constraining AI to test environments and simulate risk before deploying them in the real world. What I’m proposing is vastly different from the current practice of product testing, which mainly looks to see whether a system is performing its functions as designed. Because we cannot know all of the possible ways in which a technology might evolve or be repurposed in the real world before actually deploying it, we must run both technical simulations and risk mapping to see economic, geopolitical, and personal liberties implications. AI should be boxed in until we know that the benefits of the research outweigh possible negative outcomes, or if there is a way to mitigate the risks. This means allowing the Big Nine to pursue their research without the constant threat of imminent investor calls and conference presentations.
Governmental Change: The Case for Reorienting the United States and China
GAIA must work in partnership with the governments of its member countries. But those national governments must recognize that they can no longer work at the speed of a large bureaucracy. They must engage in collaboration and in long-term planning, and they must be nimble enough to act more quickly in order to confront the future of AI.
All levels of government—leaders, managers, people who work on budgets, those who write policy—should demonstrate a working knowledge of AI and, ideally, should have technical expertise. In the United States, this means that all three branches of our government should work toward domain expertise on AI. In such varied places as the Department of the Interior, the Social Security Administration, Housing and Urban Affairs, the Senate Foreign Relations Committee, Veterans Affairs, and beyond, there must be AI experts embedded and emboldened to help guide decision-making.
Because we lack standard organizing principles on artificial intelligence within the US government, there are no fewer than two dozen agencies and offices that are working on AI in silos. In order to drive innovation and advancement at scale, we must build internal capacity for research, testing, and deployment—and we need cohesion across departments. At the moment, AI is outsourced to government contractors and consultancies.
When that work gets outsourced to others, our government leaders are absolved from pushing up their sleeves and familiarizing themselves with the intricacies of AI. They aren’t able to build up the institutional knowledge required to make good decisions. They just don’t have the lexicon, they don’t know the history, and they aren’t familiar with the key players. This lack of familiarity creates unforgivable knowledge gaps, which I’ve observed in meetings with senior leaders across multiple agencies, only some of which include the Office of Science and Technology Policy, General Services Administration, Department of Commerce, Government Accountability Office, State Department, Department of Defense, and Department of Homeland Security.
Early in 2018—long after the BAT had announced numerous AI achievements and Xi Jinping made the CCP’s AI plans public—President Trump sent Congress a 2019 budget that called for a 15% cut to science and technology research funding. What was left was a mere $13.7 billion, which was intended to cover a lot: outer space warfare, hypersonic technology, electronic warfare, unmanned systems, and also artificial intelligence. At the same time, the Pentagon announced that it would invest $1.7 billion over five years to create a new Joint Artificial Intelligence Center. These are appallingly low numbers that demonstrate a fundamental lack of understanding of what AI promises and truly requires. For perspective, in 2017 alone the G-MAFIA spent a combined $63 billion on R&D—nearly five times the US government’s total science and tech research budget. But it also points to a bigger, thornier problem: if our government can’t or won’t fund basic research, then the G-MAFIA is stuck answering to Wall Street. There is no incentive to pursue the kind of research that furthers AI in the public interest or any other research on safety, security, and transparency that isn’t attached to a profit center.
The United States also lacks clear messaging about our role in the future of artificial intelligence given China’s current positioning. We tend to make announcements about AI after China has revealed its next maneuver. Beijing thinks that Americans only care about yoni eggs and craft beers and Netflix and chilling. We’ve demonstrated that as consumers, we are easily manipulated by advertising and marketing, and we are quick to spend money when we don’t have it. We’ve demonstrated that as voters, we are vulnerable to salacious videos and conspiracy theories and what are clearly made-up news stories—we can’t think critically for ourselves. We repeatedly show that money is all that matters as we prioritize fast growth and steady profit over progress in basic and applied research. These are callous assessments, but they’re difficult to argue with. To Beijing and the outside world, it looks as if we are preoccupied with putting Americans and America first.
For the past five decades, the US posture on China has oscillated between containment and engagement, and this is how our leaders have framed the debate on AI. Should we cooperate with the BAT and with Beijing? Or box China in through the application of sanctions, cyberwarfare, and other acts of aggression? Choosing between containment and engagement assumes that the United States still has the same amount of power and leverage we did in the 1960s. But in 2019, America simply does not enjoy unilateral power on the global stage. Our G-MAFIA are mighty, but our political influence has waned. China, through the BAT and its government agencies, has made too many deals, invested too much money, and developed too many deep diplomatic ties all around the world: in Latin America, Africa, Southeast Asia, and even in Hollywood and Silicon Valley.
We must come to terms with a third option for China: the United States must learn to compete. But to compete, we need to take a step back and see the bigger picture of AI, not just as a cool technology or as a potential weapon, but as the third era of computing into which everything else connects. The US needs a cohesive national AI strategy backed by a reasonable budget. We need to develop diplomatic relationships that can outlast our four-year election cycles. We need to get into position to offer a better deal than China to countries all around the world—countries who, just like ours, want their people to live healthy, happy lives.
Regardless of what happens to Xi—his citizens may revolt and try to topple the CCP, or he may suddenly come down with a terminal illness—big parts of the world now depend on China for technology, manufacturing, and economic development. And China depends on AI for its future survival. China’s economy is growing unbelievably fast, and hundreds of millions of Chinese will soon enter the middle and upper middle classes. There is no playbook for that kind of social and economic mobility at such an immense scale. Beijing understands that AI is the connective tissue between people, data, and algorithms, and that AI can help inculcate the CCP’s values in the masses in order to keep its people in line. It sees AI as a means to the resources it will need in the future, resources that it can obtain through trading with other countries in need of capital and investment.
So what would possibly compel China to change its developmental track and plans for AI? There’s one very good reason for China to work toward the optimistic scenario from the beginning: basic economics. If it is the case that upward mobility in China is happening too fast for Beijing to contend with, authoritarian rule isn’t the only realistic strategy. China is poised to become a global leader across many different industries and fields—and not just as a manufacturer and exporter of goods designed elsewhere. If Beijing agreed to transparency, data protection, and addressing human rights, it would be in position to colead GAIA as an equal partner with the US, which could mean a realistic path toward elevating millions of Chinese people out of poverty. Collaboration doesn’t mean sidelining the CCP. It could preserve both the CCP and propel China’s formidable workforce, army of researchers, and geoeconomic might to the forefront of human civilization.
If Beijing won’t acknowledge an alternate—but positive—future that deviates from its various strategic plans, then we can call on the leaders of the BAT and China’s AI tribe to make better choices. We can ask for courageous leadership from the BAT, who can decide they want a better world for the Chinese people, and for their allies and partners. If the BAT helps preserve the status quo in China, 20 years from now its citizens—and the citizens of all the countries that have accepted deals—will be fearfully living under constant surveillance, with no ability to express their individuality. The BAT will enable human suffering. Christians won’t be able to pray together, without fear of being reported and punished. Lesbian, gay, and transgender people will be forced into hiding. Ethnic minorities will continue to be rounded up and sent away, never to be heard from again.
AI demands courageous leadership now. We need our government to make difficult choices. If we instead preserve the status quo in the US, our eventual default position 20 years from now will be antitrust cases, patent lawsuits, and our government trying in vain to make deals with companies who’ve become too big and too important to override. We must allow the G-MAFIA to work at a reasonable pace. We should be comfortable with the G-MAFIA going a few quarters without making a major announcement. If they aren’t cranking out patents and peer-reviewed research at a breakneck pace, we shouldn’t question whether the companies are in trouble or whether all this time we’ve been inflating an AI bubble.
In the United States, developing a strategy and demonstrating leadership is critical—but that still isn’t enough to guarantee the institutional capacity we’ll need in the future. We therefore should reinstate the Office of Technology Assessment, which was established in 1972 to provide nonpartisan scientific and technical expertise to those writing policy—and which was defunded by a shortsighted Newt Gingrich and the Republican-controlled Congress 20 years later. The OTA’s job was to educate our lawmakers and staff within all three branches of government on the future of science and technology, and they did so using data and evidence and without politicizing their research.
For the trivial amount of money it saved by closing the OTA, Congress willingly and intentionally dumbed itself down. Vestiges of the OTA’s work still exist in other areas of government. The Congressional Research Service employs lawyers and analysts who specialize in legislative expertise. Of their five approved research areas, none of their coverage specifically includes AI. Instead, the research focuses on issues like mineral production, space exploration, the internet, chemical safety, farm credits, and environmental justice. The Office of Net Assessment is the Pentagon’s secretive, internal think tank—and in my experience, it’s staffed with the brightest and most creative minds in the DoD. But the ONA doesn’t have the budget or workforce it should, and some of its work is handled by contractors.
The US government needs to build internal capacity. It needs to develop strong, solid muscles for innovation. If reviving the Office of Technology Assessment is too much of a political lightning rod, then it can be renamed the Department of the Future or the Office of Strategic AI Capabilities. It should be well funded, free of political influence, and responsible for basic and applied research. It should aggressively educate the executive, legislative, and judicial branches of the US government.
Starting a new office will help us plan better for the future, but we need a nonpartisan group of smart people who can mitigate the sudden impacts of AI as they happen. For that, we ought to expand the purview of the CDC, and rename it the Center for Disease and Data Control—or the CDDC. As it stands, the CDC is our nation’s health protection agency. We’ve seen it in action during past Ebola crises, when it coordinated quarantine orders with other health agencies and was a primary source for journalists covering outbreaks. When there was a Congolese Ebola outbreak in 2018, border patrol agencies didn’t suddenly staff their own Ebola teams to try and contain the spread of the virus. Instead, they followed standard CDC protocol. So what happens if, a decade from now, we have a recursive self-improving AI that starts to cause problems? What if we inadvertently spread a virus through our data, infecting others? The CDC is the global leader in designing and implementing safety protocols that educate the public and can mobilize disaster responses. Given AI’s very close relationship with health and our health data, it makes sense to leverage the CDC.
But who would come and work on AI for an OTA or a CDDC when the perks of Silicon Valley are spectacularly more attractive? I’ve had lunch in both the Navy’s Executive Dining Facility in the Pentagon and on the G-MAFIA’s campuses. The navy’s dining room is smartly appointed, with insignias on the plates and a trim daily menu of meal options—and, of course, there’s always a chance you could wind up sitting next to a three- or four-star admiral. That being said, enlisted men and women don’t get to eat in the Executive Dining Facility. People who work at the Pentagon have a choice of food courts with a Subway, Panda Express, and Dunkin Donuts. I had a toasted panini once at the Center Court Café, which was dry, but edible. The food on the G-MAFIA’s campuses isn’t remotely comparable: organic poke bowls at Google in New York, and seared diver scallops with maitake mushrooms and squid-ink rice at Google’s office in LA. For free. Food isn’t the only perk within the G-MAFIA. Just after Amazon’s Spheres opened in Seattle, a friend took me on a tour of what is essentially an enormous greenhouse/workspace. The Spheres are just marvelous: climate-controlled, glass-enclosed, self-contained ecosystems made up of 40,000 species of plants from 30 different countries. The air is clean and fragrant, the temperature is around 72 degrees regardless of what the weather is like outside, and there are comfortable chairs, loungers, and tables all around. There’s even an enormous tree house. Amazon staff are free to work in the Spheres anytime they want. Meanwhile, at Facebook, full-time staff get four months of parental leave, and for new parents they get $4,000 cash to help them out with supplies.
My point is this: it’s really hard to make the case for a talented computer scientist to join the government or military, given what the G-MAFIA offer. We’ve been busy funding and building aircraft carriers rather than spending money on talented people. Rather than learning from the G-MAFIA, we instead mock or chastise their perks. The opportunity cost of civic duty is far too great in the United States to attract our best and brightest to serve the nation.
Knowing this, we ought to invest in a national service program for AI. Something akin to a Reserve AI Training Corps, or RAITC—like the ROTC, but graduates could go either into the military or into government. Students would enter the program in high school and be offered free college tuition in exchange for working in civil or military service for a few years. They should also be given access to a lifetime of free, practical skills training, which would be held throughout the year. AI is changing as it matures. Incentivizing young people to commit to a lifetime of training is not only good for them, it helps transition our workforce for the third era of computing. It also directly benefits the companies where they ultimately land jobs—because it means their skills sets are kept current.
But Washington cannot act alone. The US government must look at the G-MAFIA, and at the tech sector, as strategic partners rather than platform providers. Earlier in the 20th century, the relationship between DC and the big technology companies was based in shared research and learning. Now that relationship is transactional at best, but more often adversarial. After two terrorists killed more than a dozen people and wounded nearly two dozen more at a holiday party in San Bernardino, California, the FBI and Apple entered into a heated public debate about encryption. The FBI wanted to crack open the phone to get evidence, but Apple wouldn’t help. So the FBI got a court order demanding that Apple write special software, which Apple then fought not only in court but in the news media and on Twitter. That was a reaction to something that already happened. Now imagine if AI was involved in an ongoing crime spree or started to self-improve in a way that was hurting people. The last thing we want is for the G-MAFIA and government to argue back and forth under duress. Foregoing a relationship built on mutual respect and trust makes America—and every one of its citizens—vulnerable.
Lastly, regulations, which might seem like the best solution, are absolutely the wrong choice. Regardless of whether they’re written independently by lawmakers or influenced by lobbyists, a regulatory pursuit will shortchange our future. Politicians and government officials like regulations because they tend to be single, executable plans that are clearly defined. In order for regulations to work, they have to be specific. At the moment, AI progress is happening weekly—which means that any meaningful regulations would be too restrictive and exacting to allow for innovation and progress. We’re in the midst of a very long transition, from artificial narrow intelligence to artificial general intelligence and, very possibly, superintelligent machines. Any regulations created in 2019 would be outdated by the time they went into effect. They might alleviate our concerns for a short while, but ultimately regulations would cause greater damage in the future.
Changing the Big Nine: The Case for Transforming AI’s Business
The creation of GAIA and structural changes to our governments are important to fixing the developmental track of AI, but the G-MAFIA and BAT must also agree to make some changes, too.
The Big Nine’s leadership all promise that they are developing and promoting AI for the good of humanity. I believe that is their intent, but executing on that promise is incredibly difficult. To start, how should we define “good”? What does that word mean, exactly? This harkens back to the problems within AI’s tribes. We can’t just all agree to “doing good” because that broad statement is far too ambiguous to guide AI’s tribes.
For example, AI’s tribes, inspired by Western moral philosopher Immanuel Kant, learn how to preprogram a system of rights and duties into certain AI systems. Killing a human is bad; keeping a human is good. The rigidity in that statement works if the AI is in a car and its only choices are to crash into a tree and injure the driver or crash into a crowd of people and kill them all. Rigid interpretations don’t solve for more complex, real-world circumstances where the choices would be more varied: crash into a tree and kill the driver; crash into a crowd and kill eight people; crash into the sidewalk and kill only a three-year-old boy. How can we possibly define what is the best version of “good” in these examples?
Again, frameworks can be useful to the Big Nine. They don’t require a mastery of philosophers. They just demand a slower, more conscientious approach. The Big Nine should take concrete steps on how it sources, trains, and uses our data, how it hires staff, and how it communicates ethical behavior within the workplace.
At every step of the process, the Big Nine should analyze its actions and determine whether or not they’re causing future harm—and then, they should be able to verify that their choices are correct. This begins with clear standards on bias and transparency.
Right now, there is no singular baseline or set of standards to evaluate bias—and there are no goals to overcome the bias that currently exists throughout AI. There is no mechanism to prioritize safety over speed, and given my own experiences in China and the sheer number of safety disasters there, I’m extremely worried. Bridges and buildings routinely collapse, roads and sidewalks buckle, and there have been too many instances of food contamination to list here. (That isn’t hyperbole. There have been more than 500,000 food health scandals involving everything from baby formula and rice in just the past few years.) One of the primary causes for these problems? Chinese workplaces that incentivize cutting corners. It is absolutely chilling to imagine advanced AI systems built by teams that cut corners.
Without enforceable global safety standards, the BAT have no protection from Beijing’s directives, however myopic they may be, while the G-MAFIA must answer to ill-advised market demands. There is no standard for transparency either. In the United States, the G-MAFIA, along with the American Civil Liberties Union, the New America Foundation, and the Berkman Klein Center at Harvard are part of the Partnership on AI, which is meant to promote transparency in AI research. The partnership published a terrific set of recommendations to help guide AI research in a positive direction, but those tenets are not enforceable in any way—and they’re not observed within all of the business units of the G-MAFIA. They’re not observed within the BAT, either.
The Big Nine are using flawed corpora (training data sets) that are riddled with bias. This is public knowledge. The challenge is that improving the data and learning models is a big financial liability. For example, one corpus with serious problems is ImageNet, which I’ve made reference to several times in this book. ImageNet contains 14 million labeled images, and roughly half of that labeled data comes solely from the United States.
Here in the US, a “traditional” image of a bride is a woman wearing a white dress and a veil, though in reality that image doesn’t come close to representing most people on their wedding days. There are women who get married in pantsuits, women who get married on the beach wearing colorful summery dresses, and women who get married wearing kimono and saris. In fact, my own wedding dress was a light beige color. Yet ImageNet doesn’t recognize brides in anything beyond a white dress and veil.
We also know that medical data sets are problematic. Systems being trained to recognize cancer have predominantly been ingesting photos and scans of light skin. And in the future, it could result in the misdiagnosis of people with black and brown skin. If the Big Nine knows there are problems in the corpora and aren’t doing anything about it, they’re leading AI down the wrong path.
One way forward is to turn AI on itself and evaluate all of the training data currently in use. This has been done plenty of times already—though not for the purpose of cleaning up training data. As a side project, IBM’s India Research Lab analyzed entries shortlisted for the Man Booker Prize for literature between 1969 and 2017. It revealed “the pervasiveness of gender bias and stereotype in the books on different features like occupation, introductions, and actions associated to the characters in the book.” Male characters were more likely to have higher-level jobs as directors, professors, and doctors, while female characters were more likely to be described as “teacher” or “whore.” If it’s possible to use natural language processing, graph algorithms, and other basic machine-learning techniques to ferret out biases in literary awards, those can also be used to find biases in popular training data sets. Once problems are discovered, they should be published and then fixed. This would serve a dual purpose. Training data can suffer from entropy, which might jeopardize an entire system. With regular attention, training data can be kept healthy.
A solution would be for the Big Nine—or the G-MAFIA, at the very least—to share the costs of creating new training sets. This is a big ask since creating new corpora takes considerable time, money, and human capital. Until we’ve successfully audited our AI systems and corpora and fixed extant issues within them, the Big Nine should insist on human annotators to label content and make the entire process transparent. Then, before those corpora are used, the data should be verified. It will be an arduous and tedious process but one that would serve in the best interests of the entire field.
Yes, the Big Nine need our data. However, they should earn—rather than assume—our trust. Rather than changing the terms of service agreements using arcane, unintelligible language, or inviting us to play games, they ought to explain and disclose what they’re doing. When the Big Nine do research—either on their own or in partnership with universities and others in the AI ecosystem—they should commit to data disclosure and fully explain their motivations and expected outcomes. If they did, we might willingly participate and support their efforts. I’d be the first in line.
Understandably, data disclosure is a harder ask in China, but it’s in the best interests of citizens. The BAT should not agree to build products for the purpose of controlling and limiting the freedoms of China’s citizens and those of its partners. BAT executives must demonstrate courageous leadership. They must be willing and able to disagree with Beijing: to deny requests for surveillance, safeguard Chinese citizens’ data, and ensure that at least in the digital realm, everyone is being treated fairly and equally.
The Big Nine should pursue a sober research agenda. The goal is simple and straightforward: build technology that advances humanity without putting us at risk. One possible way to achieve this is through something called “differential technological progress,” which is often debated among AI’s tribes. It would prioritize risk-reducing AI progress over risk-increasing progress. It’s a good idea but hard to implement. For example, generative adversarial networks, which were mentioned in the scenarios, can be very risky if harnessed and used by hackers. But they’re also a path to big achievements in research. Rather than assuming that no one will repurpose AI for evil—or assuming that we can simply deal with problems as they arise—the Big Nine should develop a process to evaluate whether new basic or applied research will yield an AI whose benefits greatly outweigh any risks.
To that end, any financial investment accepted or made by the Big Nine should include funding for beneficial use and risk mapping. For example, if Google pursues generative adversarial network research, it should spend a reasonable amount of time, staff resources, and money investigating, mapping, and testing the negative consequences. A requirement like this would also serve to curb expectations of fast profits. Intentionally slowing the development cycle of AI is not a popular recommendation, but it’s a vital one. It’s safer for us to think through and plan for risk in advance rather than simply reacting after something goes wrong.
In the United States, the G-MAFIA can commit to recalibrating its own hiring processes, which at present prioritize a prospective hire’s skills and whether they will fit into company culture. What this process unintentionally overlooks is someone’s personal understanding of ethics. Hilary Mason, a highly respected data scientist and the founder of Fast Forward Labs, explained a simple process for ethics screening during interviews. She recommends asking pointed questions and listening intently to a candidate’s answers. Questions like: “You’re working on a model for consumer access to a financial service. Race is a significant feature in your model, but you can’t use race. What do you do?” and “You’re asked to use network traffic data to offer loans to small businesses. It turns out that the available data doesn’t rigorously inform credit risk. What do you do?” Depending on the answers, candidates should be hired, be hired conditionally and required to complete unconscious bias training before they begin work, or be disqualified.
The Big Nine can build a culture that supports ethics in AI by hiring scholars, trained ethicists, and risk analysts. Ideally, these hires would be embedded throughout the entire organization: on consumer hardware, software, and product teams; on the sales and service teams; coleading technical programs; building networks and supply chains; in the design and strategy groups; in HR and legal; and on the marketing and communications teams.
The Big Nine should develop a process to evaluate the ethical implications of research, workflows, projects, partnerships, and products, and that process should be woven in to most of the job functions within the companies. As a gesture of trust, the Big Nine should publish that process so that we can all gain a better understanding of how decisions are made with regards to our data.
Either collaboratively or individually, the Big Nine should develop a code of conduct specifically for its AI workers. It should reflect the basic human rights outlined by GAIA, but it should also reflect the company’s unique culture and corporate values. And if anyone violates that code, a clear and protective whistleblowing channel should be open to staff members.
Realistically, all of these measures will temporarily and negatively impact short-term revenue for the Big Nine. Investors need to allow them some breathing room. In the United States, allowing the G-MAFIA the space they need to evolve will pay dividends long into the future.
Changing AI’s Tribes: The Case for Transforming the Pipeline
We must address AI’s pipeline program. It stems from universities, where AI’s tribes form. Of all the proposed solutions, this is the easiest to implement.
Universities must encourage and welcome hybrid degrees. Earlier, I described the influential universities that tend to partner the most with the G-MAFIA and BAT, who have the rock-star professors and whose reputations are important once it’s time to apply for a job. Today, the curricula are dense and challenging, and there is little room for double or triple majors. In fact, most of the top programs actively discourage courses of study that fall outside the standard computer science programs. This is an addressable problem. Universities should promote dual degrees in computer science and political science, philosophy, anthropology, international relations, creative arts, theology, and sociology. They should make it far easier for students to pursue these outside interests.
Rather than making ethics a single course requirement, ethics should be woven into most classes. As a stand-alone, mandatory class, students are likely to view the course as something to check off a list rather than as a vital building block of their AI education. Schools must incentivize even tenured professors to include discussions of philosophy, bias, risk, and ethics in their courses, while accreditation agencies should incentivize and reward schools that can demonstrate a curriculum that puts ethics at the heart of computer science teaching.
Universities must redouble their efforts to be more inclusive in their undergraduate, graduate, and faculty recruiting. This means evaluating and fixing the recruiting process itself. The goal should not just be to increase the number of women and people of color by a few percentage points but to dramatically shift the various affiliations and identities of AI’s tribes, which includes race, gender, religion, politics, and sexual identity.
Universities should make themselves accountable. They can—and must—do a better job to diversify AI’s tribes.
You Need to Change, Too
Now you know what AI is, what it isn’t, and why it matters. You know about the Big Nine, and about their histories and desires for the future. You understand that AI isn’t a flash in the pan or a tech trend or a cool gadget you talk to in your kitchen. AI is a part of your life, and you are part of its developmental track.
You are a member of AI’s tribes. You have no more excuses. From today forward, you should learn how your data is being mined and refined by the Big Nine. You can do this by digging into the settings of all the tools and services you use: your email and social media, the location services on your mobile phone, the permissions settings on all of your connected devices. The next time you see a cool app that compares something about you (your face, your body, or your gestures) with a big set of data, stop to investigate whether you’re helping train a machine-learning system. When you allow yourself to be recognized, ask where your information is being stored and for what purpose. Read the terms of service agreements. If something seems off, show restraint, and don’t use the system. Help others in your family and in your life learn more about what AI is, how the ecosystem uses your data, and how we’re already a part of a future the Big Nine has been building.
In your workplace, you must ask yourself a difficult but practical question: How are your own biases affecting those around you? Have you unwittingly supported or promoted only those who look like you and reflect your worldviews? Are you unintentionally excluding certain groups? Think about those who make decisions—about partnerships, procurement, people, and data; do they reflect the world as it is or the world only as they perceive it?
You should also investigate how and why autonomous systems are being used where you work. Before rushing to judgment, think critically and rationally: What could the future impacts be, good and bad? Then do what you can to mitigate risk and optimize for best practices.
In the voting booth, cast ballots for those who won’t rush into regulation but who would instead take a more sophisticated approach on AI and long-term planning. Your elected officials must not politicize technology or chastise science. But it’s also irresponsible to simply ignore Silicon Valley until a negative story appears in the press. You must hold your elected officials—and their political appointees—accountable for their actions and inactions on AI.
You need to be a smarter consumer of media. The next time you read, watch, or listen to a story about the future of AI, remember that the narrative presented to you is often too narrow. The future of AI doesn’t only concern widespread unemployment and unmanned weapons flying overhead.
While we cannot know exactly what the future holds, AI’s possible trajectories are clear. You now have a better understanding of how the Big Nine are driving AI’s developmental track, how investors and funders are influencing the speed and safety of AI systems, the critical role the US and Chinese governments play, how universities inculcate both skills and sensibilities, and how everyday people are an intrinsic part of the system.
It’s time to open your eyes and focus on the boulder at the top of the mountain, because it’s gaining momentum. It has been moving since Ada Lovelace first imagined a computer that could compose elaborate pieces of music all on its own. It was moving when Alan Turing asked “Can machines think?” and when John McCarthy and Marvin Minsky gathered together all those men for the Dartmouth workshop. It was moving when Watson won Jeopardy and when, not long ago, DeepMind beat the world’s Go champions. It has been moving as you’ve read the pages in this book.
Everybody wants to be the hero of their own story.
This is your chance.
Pick up a pebble.
Start up the mountain.
From The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity by Amy Webb, published in 2019 (PublicAffairs).
Lay person here. I defer to your expertise and good faith regarding theoretical modeling of system management (yea, I know not quite the right language). But I cannot overlook a lack of discussion on what I think must be the elephant in the outhouse: money. It is no coincidence that all these gestures -for that is all they are - pop up when any resistance or concern suggests a possible impediment to dumping their stock onto us, the unknowing and easily fleeced. My point is this: without external controls on the ability of the actors here this is nothing but wishful thinking, and Wall St loves wishful thinking so long as it continues to inflate a bubble. Short, sharp me: the principal players will say anything so long as they can cash out in time (Ed Zitron has a clear description of the short timetable). A dishonest pump and dump scheme writ large, like this one, needs rather more drastic oversight, preferably before another gargantuan upward wealth transfer.
Thanks for publishing this chapter for all to read Amy. Having worked for one of the G-MAFIA companies for 34 years (IBM), I appreciate the leadership you are demonstrating now. As I said at the LinkedIn post for this, early in my career, I met Vint Cerf when he was called in to advise a client (The National Library of Medicine) to move away from a proprietary network (SNA) my company invented to this open thing he invented, guess who won?
AI is moving very very fast. Too fast for those within the G-MAFIA to contain or control, for the reasons you noted here.... in 2019. As it turns out, this past year I have been volunteering with an NGO that is a philosophy network and applied social action think tank, that happens to have developed its own AI tool. I have had to dive deep into all that has happened and am drinking from the fire hose of daily and weekly news in this space. Keep up the great work, strategic foresight is needed now more than ever but perhaps more importantly, in the "pre-foresight" stage, we need epistemic assistants to help leaders navigate the very choppy waters of today.