SFF-2024 S-Process Recommendations Announcement
In the second half of this year (2024), twelve people participated as “Recommenders” in a single round of a grant-recommendation process for the Funders Jaan Tallinn and Survival and Flourishing .Com. The round featured three tracks: the Main Track, the Fairness Track, and the Freedom Track. The following Recommenders in this round agreed to have their identities made public:
- Scott Alexander (Main Track)
- Benjamin Todd (Main Track)
- Deger Turan (Main Track)
- Nell Watson (Freedom Track)
- Zvi Mowshowitz (Freedom Track)
- Irene Solaiman (Fairness Track)
How final grant amounts were determined: The “S-Process”
We call the recommendation process used in this grant round the “S-Process”, for “Simulation Process”, because it involves allowing the Recommenders and Funders to simulate a large number of counterfactual delegation scenarios using a table of marginal utility functions. Recommenders specified marginal utility functions for funding each application, and adjusted those functions through discussions with each other as the round progressed. Similarly, Funders specified and adjusted different utility functions for deferring to each Recommender. In this round, the process also allowed the Funders to make some final adjustments to decide on their final intended grant amounts.
As part of the process this round, we used Recommenders’ inputs to create a synthetic mean recommender. This feature acts as a simulated Recommender, using an equal weighted average of all the Recommenders’ utility functions to generate grant recommendations.
We also introduced a couple of changes to the process that impacted both the direction and scope of funding. Two specialized tracks, the Fairness Track and the Freedom Track, ran with three Recommenders each, in parallel with the Main Track which had six Recommenders. All guaranteed eligible applications (applications who received Speculation Grant funding) were qualified to be evaluated by Recommenders in all tracks (Main, Freedom, and Fairness). As a result, in the final recommendations table there are three columns showing the recommendations of the three different tracks, as well as a column showing the Synthetic Mean Recommender’s allocation. In the “Total Funding Recommended” column is the total funding recommendation an org received, which is the sum of the recommendations from the three tracks and the Synthetic Mean Recommender.
The S-Process app is still being developed for broader use.
Total funding amounts
The total funding expected to be distributed in association with this round is $19.34MM, in excess of our $5MM-$15MM estimate. This includes $18.5MM which was recommended in the round, as well as $0.85M in funding distributed previously through Speculation Grants, in excess of the amounts recommended in the round. (Please note that due to rounding, subtotals may not precisely add up to the total figure.)
The total funding expected to be distributed in association with this round, broken down by source:
- Jaan Tallinn: $16.58MM ($17.43MM)†
- $2.44MM recommended, already distributed through Speculation Grants
- $14.15MM recommended, remaining to be distributed
- ($0.85M in excess of recommendations, already distributed through Speculation Grants)
- SFC: $1.91M
- $1.13M recommended, already distributed through Speculation Grants
- $0.78M recommended, remaining to be distributed
The total funding recommended in this round, broken down by track:
- Main Track: $11.34MM recommended (in excess of our $3MM-$10MM estimate)
- Fairness Track: $2.26MM recommended (within our $1M-$3MM estimate)
- Freedom Track: $3.26MM recommended (in excess of our $1M-$3MM estimate)
- Synthetic Mean Recommender: $1.64M recommended
† The total in parentheses reflects additional funding that was distributed in cases where the amount of the Speculation Grant awarded to an organization exceeded the eventual recommendation amount for that organization. This additional funding is not included in the figure for the total funding recommended in this round, but is included in the amount of funding expected to be distributed in total.
Final recommendations
Most of the final endorsed recommendations of this round of the S-Process are listed below. These numbers have resulted from numerical inputs from both Funders and Recommenders, which represented estimates of the marginal utility of granting to each organization. Note that:
- Some of the grants below might not happen if they are logistically difficult or time-consuming for the Funders (Jaan Tallinn and SFC) to finalize for some reason.
- Some additional grants might also appear on this list later as more details about them become clear.
(Please note that due to rounding, the funding amounts recommended by each track for an organization may not precisely add up to the total funding recommended for that organization.)
| Source | Organization | Main Track Recommendation | Freedom Track Recommendation | Fairness Track Recommendation | Synthetic Mean Recommender Recommendation | Total Funding Recommended | Receiving Charity | Purpose |
|---|---|---|---|---|---|---|---|---|
| Jaan Tallinn | AGI Inherent Non-Safety | $0 | $135,000 | $0 | $0 | $135,000 ($10,000)† | Players Philanthropy Fund | General support of AGI Inherent Non-Safety |
| Jaan Tallinn | AI Futures Project | $71,000 | $352,000 | $0 | $82,000 | $505,000 ($190,000)† | Epistea, z.s. | General support of AI Futures Project |
| Jaan Tallinn | AI Objectives Institute | $0 | $0 | $419,000 | $119,000 | $538,000 | AI Objectives Institute | General support |
| Jaan Tallinn | AI Risk Mitigation Fund | $530,000 | $0 | $0 | $0 | $530,000 ($20,000)† | Centre For Effective Altruism Usa Inc. | General support of AI Risk Mitigation Fund |
| Jaan Tallinn | AI Safety Info | $20,000 | $4,000 | $0 | $23,000 | $47,000 ($60,000)† | Ashgro Inc. | General support of AI Safety Info |
| Jaan Tallinn | AI Standards Lab and Holtman Systems Research | $12,000 | $170,000 | $0 | $1,000 | $184,000 ($20,000)† | Players Philanthropy Fund | General support of AI Standards Lab and Holtman Systems Research |
| Jaan Tallinn | AI: Futures and Responsibility Programme | $0 | $0 | $233,000 | $47,000 | $280,000 ($20,000)† | University of Cambridge | General support of AI: Futures and Responsibility Programme |
| Jaan Tallinn | Alignment of Complex Systems Research Group (ACS) | $0 | $12,000 | $0 | $52,000 | $64,000 ($40,000)† | Epistea z.s. | General support of Alignment of Complex Systems Research Group (ACS) |
| Jaan Tallinn | Alignment Research Center | $142,000 | $16,000 | $0 | $38,000 | $197,000 ($50,000)† | Alignment Research Center | General support |
| Jaan Tallinn | Alliance to Feed the Earth in Disasters (ALLFED) | $0 | $65,000 | $0 | $49,000 | $113,000 ($10,000)† | ALLFED Institute | General support |
| Jaan Tallinn | Apart Research | $294,000 | $0 | $0 | $0 | $294,000 ($20,000)† | Ashgro Inc. | General support of Apart Research |
| Jaan Tallinn | Apollo Research | $224,000 | $0 | $0 | $26,000 | $251,000 ($10,000)† | Rethink Priorities | General support of Apollo Research |
| Jaan Tallinn | Balsa Policy Institute Inc. | $0 | $33,000 | $0 | $0 | $33,000 ($10,000)† | Balsa Policy Institute Inc. | General support |
| Jaan Tallinn | Basis Research Institute | $0 | $0 | $296,000 | $0 | $296,000 | Basis Research Institute | General support |
| Jaan Tallinn | BERI-DMIP Collaboration | $22,000 | $0 | $0 | $44,000 | $66,000 ($10,000)† | Berkeley Existential Risk Initiative | General support of BERI-DMIP Collaboration |
| Jaan Tallinn | Berkeley Existential Risk Initiative | $0 | $85,000 | $0 | $41,000 | $126,000 ($20,000)† | Berkeley Existential Risk Initiative | General support |
| Jaan Tallinn | Blueprint Biosecurity | $0 | $78,000 | $0 | $0 | $78,000 ($15,000)† | Blueprint Biosecurity | General support |
| Jaan Tallinn | Brown University AI Governance Lab | $0 | $0 | $326,000 | $0 | $326,000 | Brown University | General support of Brown University AI Governance Lab |
| SFC | Center for AI Policy | $185,000 | $64,000 | $0 | $39,000 | $288,000 ($10,000)† | Center for AI Policy, Inc. | General support |
| SFC | Center for AI Safety Action Fund, Inc. | $1,508,000 | $42,000 | $0 | $71,000 | $1,621,000 ($1,120,000)† | Center for AI Safety Action Fund, Inc. | General support |
| Jaan Tallinn | Center for AI Safety, Inc. | $1,050,000 | $19,000 | $0 | $77,000 | $1,146,000 ($50,000)† | Center for AI Safety, Inc. | General support |
| Jaan Tallinn | Center for Law and AI Risk | $62,000 | $52,000 | $0 | $20,000 | $134,000 ($20,000)† | Pledged | General support of Center for Law and AI Risk |
| Jaan Tallinn | CeSIA | $11,000 | $20,000 | $0 | $17,000 | $48,000 ($20,000)† | EffiSciences | General support of CeSIA |
| Jaan Tallinn | Charter Cities Institute** | $0 | $119,000 | $0 | $0 | $119,000 ($20,000)† | Charter Cities Institute | General support |
| Jaan Tallinn | Convergence: AI Clarity | $0 | $48,000 | $0 | $19,000 | $67,000 ($20,000)† | Convergence Analysis | General support of Convergence: AI Clarity |
| Jaan Tallinn | Cornell University | $0 | $0 | $365,000 | $0 | $365,000 | Cornell University | General support |
| Jaan Tallinn | Decode Research | $86,000 | $0 | $0 | $47,000 | $133,000 ($20,000)† | Ashgro Inc. | General support of Decode Research |
| Jaan Tallinn | Effective Institutions Project | $0 | $145,000 | $0 | $0 | $145,000 ($210,000)† | Effective Institutions Project Inc. | General support |
| Jaan Tallinn | Eisenstat Research Directions | $0 | $103,000 | $0 | $0 | $103,000 ($20,000)† | Machine Intelligence Research Institute Inc. & Ashgro Inc.‡ | General support of Eisenstat Research Directions |
| Jaan Tallinn | Encode Justice | $0 | $21,000 | $0 | $29,000 | $50,000 ($10,000)† | March On Foundation | General support of Encode Justice |
| Jaan Tallinn | FAR AI, Inc. | $313,000 | $11,000 | $21,000 | $37,000 | $383,000 ($20,000)† | FAR AI, Inc. | General support |
| Jaan Tallinn | Foundation for American Innovation | $388,000 | $66,000 | $0 | $64,000 | $518,000 ($25,000)† | Lincoln Network, Inc. | General support of Foundation for American Innovation |
| SFC | Foxglove | $0 | $0 | $5,000 | $0 | $5,000 | Foxglove Legal CIC | General support |
| Jaan Tallinn | Global Catastrophic Risk Institute | $0 | $53,000 | $0 | $0 | $53,000 ($10,000)† | Social and Environmental Entrepreneurs | General support of Global Catastrophic Risk Institute |
| Jaan Tallinn | Global Shield | $39,000 | $353,000 | $0 | $32,000 | $424,000 ($20,000)† | Social and Environmental Entrepreneurs | General support of Global Shield |
| Jaan Tallinn | Good Ancestors Policy | $171,000 | $0 | $0 | $0 | $171,000 ($10,000)† | Good Ancestors Policy Ltd. | General support |
| Jaan Tallinn | History Lab | $0 | $0 | $0 | $27,000 | $27,000 | The Trustees of Columbia University in the City of New York | General support of History Lab |
| Jaan Tallinn | Impact Academy Limited | $72,000 | $0 | $0 | $0 | $72,000 ($10,000)† | Stiftelsen Impact Academy | General support of Impact Academy Limited |
| Jaan Tallinn | Institute for AI Policy and Strategy | $32,000 | $113,000 | $0 | $38,000 | $183,000 ($30,000)† | Rethink Priorities | General support of Institute for AI Policy and Strategy |
| Jaan Tallinn | JUSTICE | $0 | $0 | $258,000 | $0 | $258,000 | JUSTICE | General support |
| Jaan Tallinn | Langsikt - Centre for Long-Term Policy | $194,000 | $4,000 | $0 | $2,000 | $199,000 ($10,000)† | Senter for langsiktig politikk AS | General support of Langsikt - Centre for Long-Term Policy |
| Jaan Tallinn | Legal Advocates for Safe Science and Technology, Inc. | $266,000 | $0 | $0 | $11,000 | $278,000 ($10,000)† | Legal Advocates for Safe Science and Technology, Inc. | General support |
| Jaan Tallinn | Legal Safety Lab | $29,000 | $0 | $0 | $0 | $29,000 ($20,000)† | Legal Advocates for Safe Science and Technology Inc. | General support of Legal Safety Lab |
| Jaan Tallinn | Longview Philanthropy | $522,000 | $0 | $0 | $0 | $522,000 ($10,000)† | Longview Philanthropy USA Inc. | General support |
| Jaan Tallinn | Machine Intelligence and Normative Theory lab, ANU | $4,000 | $0 | $8,000 | $20,000 | $32,000 ($480,000)† | Australian National University | General support of Machine Intelligence and Normative Theory lab, ANU |
| Jaan Tallinn | Machine Intelligence Research Institute | $0 | $54,000 | $0 | $0 | $54,000 ($40,000)† | Machine Intelligence Research Institute Inc. | General support |
| Jaan Tallinn | Machine Learning for Socio-technical Systems Lab | $0 | $0 | $163,000 | $0 | $163,000 | The University of Rhode Island Foundation & Alumni Engagement | General support of Machine Learning for Socio-technical Systems Lab |
| Jaan Tallinn | Macrostrategy Research Initiative | $154,000 | $0 | $0 | $0 | $154,000 ($170,000)† | Lightcone Infrastructure Inc. | General support of Macrostrategy Research Initiative |
| Jaan Tallinn | Manifund | $90,000 | $0 | $0 | $0 | $90,000 ($60,000)† | Manifold for Charity | General support |
| Jaan Tallinn | Mathematical Metaphysics Institute | $0 | $53,000 | $0 | $0 | $53,000 ($260,000)† | Mathematical Metaphysics Institute | General support |
| Jaan Tallinn | MATS Research | $639,000 | $0 | $0 | $0 | $639,000 ($55,000)† | MATS Research, Inc. | General support |
| Jaan Tallinn | Meaning Alignment Institute | $0 | $145,000 | $0 | $0 | $145,000 ($20,000)† | Hack Foundation | General support of Meaning Alignment Institute |
| Jaan Tallinn | Model Evaluation and Threat Research | $166,000 | $2,000 | $0 | $36,000 | $204,000 ($20,000)† | Model Evaluation and Threat Research, Inc. | General support |
| Jaan Tallinn | MSEP Project | $0 | $165,000 | $0 | $1,000 | $166,000 ($85,000)† | Science and Technology Futures, Inc. | General support of MSEP Project |
| Jaan Tallinn | Oxford China Policy Lab | $81,000 | $0 | $109,000 | $29,000 | $219,000 ($10,000)† | Berkeley Existential Risk Initiative | General support of Oxford China Policy Lab |
| Jaan Tallinn | Palisade Research | $586,000 | $0 | $0 | $0 | $586,000 ($230,000)† | Palisade Research Inc. | General support |
| Jaan Tallinn | Palladium Magazine | $0 | $42,000 | $0 | $0 | $42,000 ($100,000)† | American Governance Foundation Inc. | General support of Palladium Magazine |
| Jaan Tallinn | Pour Demain | $0 | $0 | $0 | $70,000 | $70,000 ($10,000)† | Pour Demain | General support |
| Jaan Tallinn | Psychosecurity Ethics @ EURAIO | $0 | $38,000 | $0 | $0 | $38,000 ($19,720)† | Players Philanthropy Fund | General support of Psychosecurity Ethics @ EURAIO |
| Jaan Tallinn | Rethink Priorities | $99,000 | $0 | $0 | $0 | $99,000 ($25,000)† | Rethink Priorities | General support |
| Jaan Tallinn | Safe AI Forum | $545,000 | $0 | $0 | $36,000 | $580,000 ($40,000)† | FAR AI, Inc. | General support of Safe AI Forum |
| Jaan Tallinn | SaferAI | $23,000 | $0 | $0 | $72,000 | $95,000 ($25,000)† | SaferAI | General support |
| Jaan Tallinn | SecureBio | $134,000 | $99,000 | $0 | $22,000 | $254,000 ($30,000)† | SecureBio, Inc. | General support |
| Jaan Tallinn | SecureDNA | $100,000 | $3,000 | $0 | $25,000 | $128,000 ($20,000)† | SecureBio, Inc. | General support of SecureDNA |
| Jaan Tallinn | Sentinel | $130,000 | $0 | $0 | $1,000 | $131,000 ($10,000)† | Epistea, z. s. | General support of Sentinel |
| Jaan Tallinn | Simon Institute for Longterm Governance | $344,000 | $0 | $0 | $20,000 | $364,000 ($20,000)† | Simon Institute for Longterm Governance | General support |
| Jaan Tallinn | Simplex | $5,000 | $0 | $0 | $69,000 | $74,000 ($45,000)† | Epistea, z.s. | General support of Simplex |
| Jaan Tallinn | Talos Network gUG | $0 | $64,000 | $0 | $59,000 | $123,000 ($20,000)† | Talos Network gUG | General support |
| Jaan Tallinn | Tarbell Fellowship | $507,000 | $3,000 | $4,000 | $6,000 | $520,000 ($10,000)† | Players Philanthropy Fund | General support of Tarbell Fellowship |
| Jaan Tallinn | Technology Strategy Roleplay | $0 | $9,000 | $0 | $0 | $9,000 ($49,256)† | Technology Strategy Roleplay | General support |
| Jaan Tallinn | The AI Policy Institute | $0 | $142,000 | $0 | $0 | $142,000 ($10,000)† | Hack Foundation | General support of The AI Policy Institute |
| Jaan Tallinn | The Centre for Long-Term Resilience (CLTR) | $1,037,000 | $8,000 | $10,000 | $28,000 | $1,083,000 ($110,000)† | Founders Pledge, Inc. | General support of The Centre for Long-Term Resilience (CLTR) |
| Jaan Tallinn | The Foresight Institute | $16,000 | $35,000 | $0 | $0 | $51,000 ($10,000)† | The Foresight Institute | General support |
| Jaan Tallinn | The Future Society, Inc. | $0 | $0 | $0 | $97,000 | $97,000 ($50,000)† | The Future Society Inc. | General support |
| Jaan Tallinn | The Midas Project | $31,000 | $0 | $0 | $0 | $31,000 ($10,000)† | The Midas Project, Inc. | General support |
| Jaan Tallinn | The Quantified Uncertainty Research Institute | $62,000 | $0 | $0 | $0 | $62,000 ($10,000)† | Rethink Priorities | General support of The Quantified Uncertainty Research Institute |
| Jaan Tallinn | Timaeus | $343,000 | $0 | $0 | $0 | $343,000 ($40,000)† | Ashgro Inc. | General support of Timaeus |
| Jaan Tallinn | Topos Institute | $0 | $213,000 | $0 | $0 | $213,000 ($120,000)† | Topos Institute | General support |
| Jaan Tallinn | Vanderbilt University | $0 | $0 | $38,000 | $0 | $38,000 | The Vanderbilt University | General support |
† Parentheses indicate a Speculation Grant awarded to certain grantees, prior to the completion of the S-Process round. If the Speculation Grant amount was higher than the S-Process recommendation, it subsumed the S-Process recommendation, so only the Speculation Grant amount is meant to be granted. If the Speculation Grant was less, it was deducted from the S-Process recommendation, so the total granted is meant to be the amount of the S-Process recommendation.
‡ This indicates that a grantee has an agreement with multiple receiving charities, and the grant will be divided equally between those organizations.
“*” Asterisks demarcate grants that were “controversial” in that one or more Recommenders wished to publicly disendorse the grant. Multiple asterisks correspond to multiple disendorsements.
Note that the S-process system is designed to generally favor funding things that at least one Recommender is excited to fund, rather than things that every Recommender is excited to fund. As such, the grant recommendations above do not especially represent the “average” opinion of the group in any sense.
Freedoms compatible with the S-Process
The S-Process is designed to allow a lot of freedom and autonomy for the Recommenders and the Funders. Funders retain the right and freedom to make and/or recommend grants that the S-Process, as a whole, did not endorse, based on information the Funders learn from the S-Process. As well, the Recommenders and Funders are free to communicate with each other as much as they want outside of the S-Process structure. As such, while the S-Process might be useful in helping the Funders learn about grant-making opportunities and opinions, the Funders need not be limited by the outputs of the S-Process in order for it to function.
Sharing of Recommender input documents
Each Recommender’s own inputs to the process (both initial and final versions) are free for that Recommender to share privately or publicly in any form and for any use-case, as long as they anonymize any other Recommenders who have requested anonymity, and respect the privacy of the organizations who applied to the round. In particular, the S-Process does not require Recommenders to avoid sharing their own (appropriately anonymized) inputs, even when sharing some documents might make it somewhat easier in principle for an adversary to reverse-engineer the inputs of other Recommenders. The spirit of the S-Process is, however, to discourage this sort of intentionally adversarial de-anonymization, as it is disrespectful of the Recommenders who contributed their time and effort under the conditions of anonymity.