Meet Anthropic CEO Dario Amodei’s handpicked – Business News
Come with them if you wish to reside.
Anthropic’s CEO Dario Amodei says he has discovered an outdoors workforce that may regulate AI and make sure it doesn’t destroy the world — and it simply so occurs to have deep ties to his previous housemate and the woke Effective Altruism motion that’s already been concerned in a single main tech meltdown.
In a bombshell essay titled “We Must Pace the Frontier,” Amodei argued that main AI corporations ought to decide to permitting “embedded third-party evaluators” from a nonprofit referred to as Model Evaluation and Threat Research, or METR, that will keep “employee-like access” to supervise security efforts and guarantee superior AI fashions don’t flip Terminator and wipe out humanity.
The would-be John Connors at METR purport to be absolutely unbiased, however the group traces its roots immediately back to key figures in Effective Altruism, whose Silicon Valley-based devotees as soon as included disgraced crypto boss Sam Bankman-Fried, one of Anthropic’s largest early buyers.
Anthropic co-founder and CEO Dario Amodei argues for third-party AI reviewers from a nonprofit referred to as Model Evaluation and Threat Research. REUTERS
Effective Altruism, a philosophy that’s grow to be standard with Silicon Valley masters of the universe, advocates for utilizing motive — and fortunes amassed by tech’s best and brightest — to do essentially the most good for humanity over the long time period.
In the previous, it has been used to justify making as a lot money as attainable with the thought you can donate more later to do more good – although the motion suffered a huge black eye when one of its most well-known proponents, Bankman-Fried, noticed his crypto empire collapse amid fraud fees that despatched him to jail.
Deep EA ties
The overlap between METR and Anthropic is rife with potential conflicts of curiosity, in line with critics.
METR spun off from an Effective Altruism-aligned tech incubator referred to as Alignment Research Center (ARC), which is headed by AI researcher Paul Christiano. He was Amodei’s housemate, coworker and analysis collaborator within the 2010s once they had been each at OpenAI — Anthropic’s largest rival.
Ajeya Cotra is a member of technical employees at METR. METR
Christiano additionally served as one of the primary 5 trustees of Anthropic’s Long-Term Benefit Trust.
One staffer on METR is Ajeya Cotra, who’s married to Christiano and has been a main organizer within the EA world.
The founder and CEO of the group is Beth Barnes, who was employed by Christiano and was concerned within the EA motion in school.
Chief scientist Hjalmar Wijk and staffers Megan Kinniment and Pip Arnott all got here from the identical analysis cluster on the University of Oxford — a main feeder for EA.
When reached by cellphone, a METR official acknowledged that the nonprofit has a lot of overlap with Effective Altruism, however insisted its staff have a various array of ideological viewpoints. They’re required to reveal conflicts of curiosity, corresponding to staff holding equity in corporations they’re tapped to analyze.
“We don’t think any small group should have authority over what happens with AI, and our goal is to get information from inside these companies into the public domain – external investigators can help bring more information to light,” a METR spokesperson mentioned in a assertion.
Beth Barnes is the CEO of METR. METR
“Our funders have no say in the projects we work on and we do not accept any funding from AI companies or their employees.”
Amodei, who has pledged to provide away 80% of his personal wealth, has deep ties to EA. His sister Daniela married Holden Karnofsky, the founder of two main EA nonprofits.
METR’s involvement is more likely to be a nonstarter for the Trump administration, which has had a testy relationship with Anthropic over the previous 12 months and has additionally expressed heavy skepticism of so-called AI “doomers” who argue the technology is on the precipice of catastrophe.
“METR is essentially Anthropic in another form,” a source conversant in the White House’s pondering instructed The Post on Tuesday.
Amodei, left, speaks with Salesforce CEO Marc Benioff during the keynote handle at Salesforce’s Dreamforce convention on the Moscone Center on Tuesday. Getty Images
Multimillion-dollar fundraising
Effective Altruism’s obsessed adherents imagine in funneling their wealth towards the objectives they view as most important to humanity, together with climate change and strict AI regulation.
Since its founding, METR and its associates have acquired funding from EA kingpins like Democratic megadonor Dustin Moskovitz and Skype cofounder Jaan Tallinn — each of whom are Anthropic buyers.
In August, METR introduced it had raised $71 million in new funding from outdoors buyers over the earlier six months.
Tallinn’s Survival and Flourishing Fund, a philanthropy group, has earmarked up to $752,000 in funding to help METR since its founding, together with $324,000 usually money and a $428,000 matching pledge, in line with public data.
Elsewhere, Moskovitz’s Coefficient Giving – previously often known as Open Philanthropy earlier than a hasty rebranding final 12 months – gave $1,515,000 to METR’s incubator, the ARC, in 2022. METR was initially often known as ARC Evals earlier than spinning off as an unbiased nonprofit and altering to its present title in 2023.
METR staffer Pip Arnott is pictured. METR
METR doesn’t take any compensation from AI labs for its work, nor does it take donations from executives or staff of AI corporations, the METR official mentioned. In the case of Moskovitz’s donation to ARC in 2022, the official mentioned the funding was firewalled and never used for METR’s operations.
That’s hardly reassuruing to critics.
“Saying you want to be audited by the ‘independent’ people at METR is a complete joke,” mentioned Perry Metzger, chairman of Alliance for the Future, a Washington, DC-based AI coverage group. “It’s people who’re funded by your buddies. It’s fully ridiculous.
“If you went out tomorrow and you’re a big bank and you had your audits done by an organization consisting entirely of your own former employees, which was funded by the same investors that are investing in you, do you know what the feds would do with you?” he added. “It would be positively medieval.”
Amodei’s shock proposal revealed final Saturday morning got here simply days after Jacob Coxon, an Anthropic researcher who beforehand labored at OpenAI, abruptly give up the firm whereas warning that each AI giants had been “gambling with our lives” by pursuing progress with out correct oversight.
Without safeguards in place, Amodei warned the web might be overtaken by AI bots within six to 12 months, “potentially causing hundreds of billions of dollars in damage.”
‘Stop pretending’
The doomsday warnings acquired sharp pushback in some corners of the tech and political world, with President Trump asserting that they had been a “hoax” and smaller AI startups arguing that leaders like Anthropic and OpenAI had been simply making an attempt to rewrite the law to benefit themselves.
Hjalmar Wijk is chief scientist at METR. He at present works at “trying to reduce worst-case risks from future models” of totally different AI fashions, in line with his LinkedIn profile. METR
Among the skeptics was former White House AI czar David Sacks, who referred to as on Amodei to “stop pretending METR is independent when it is interviewed with Anthropic’s investors and staff.”
“Most of all, stop pretending the motivation to slow down is purely altruistic,” he added. “You face massive product-liability exposure if your products enable a truly damaging cyberattack.”
It’s additionally unclear if the AI industry’s push to police itself will move muster with US regulators.
Federal Trade Commission Chairman Andrew Ferguson warned that “everyone should be deeply suspicious” of any request to grant AI corporations an antitrust exemption to coordinate security efforts.
“I will say generally, though, that if companies are simultaneously coming to Washington and asking for a host of regulations and an antitrust exemption, all of my alarm bells go off,” Ferguson mentioned during a Tuesday occasion at Georgetown University.
Jasmine Dhaliwal is a coverage staffer at METR. METR
Under Amodei’s proposals, the third-party evaluators – which some have likened to nuclear inspectors – would “verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes,” in line with his essay.
Megan Kinniment Kinniment, a member of METR’s technical employees, beforehand held a sequence of positions on the Center on Long-Term Risk in London, the place she centered on how “malevolent actors could increase the likelihood of poor AI outcomes,” in line with her LinkedIn. METR
“Anthropic is unilaterally committing to this step now,” Amodei wrote. “We intend this to be part of a broader push to redouble efforts on our safety and alignment work.”
OpenAI’s Sam Altman mentioned he agreed with Amodei concerning the “need to pace the frontier” – including that “committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” He made no point out of METR.
Anthropic declined to remark.
Additional reporting by Jared Downing and Georgia Worrell
