Survival and Flourishing
.Fund (SFF)

SFF-2023-H2 S-Process Recommendations Announcement

In the second half of this year (2023-H2), six people participated as “Recommenders” in a single round of a grant-recommendation process for funders Jaan Tallinn, Future of Life Institute, and Survival and Flourishing .Com. The following Recommenders in this round agreed to have their identities made public:

How final grant amounts were determined: The “S-Process”

We call the recommendation process used in this grant round the “S-Process”, for “Simulation Process”, because it involves allowing the Recommenders and funders to simulate a large number of counterfactual delegation scenarios. In each simulation, Recommenders specify a marginal value function for funding each application, and an algorithm calculates a table of grant recommendations by taking turns distributing funding recommendations from each Recommender in succession, using their marginal value functions to prioritize. The Recommenders then discuss their evaluations and update the simulation with their new opinions, using approval voting to prioritize discussion topics, until the end of the last meeting when their inputs are finalized. Similarly, funders specify and adjust different value functions for deferring to each Recommender. In this round, the process also allowed the funders to make some final adjustments to decide on their final intended grant amounts.

The S-Process app is still being developed for broader use.

Total funding amounts

The total funding expected to be distributed in association with this round is $21.29MM, in excess of our $9MM-$21MM estimate. This total includes $9.62MM in grants recommended by Lightspeed Grants, which funder Jaan Tallinn requested to incorporate in our announcement. Also included in the $21.29MM total is $2.51MM in funding distributed previously through speculation grants and a unilateral grant from Jaan Tallinn to MIT, in excess of the amounts recommended by SFF’s S-Process round. SFF’s S-Process itself combined with Lightspeed Grants recommended a total distribution of $18.78MM. Without Lightspeed Grants, the total funding recommended by this round was $9.16MM, at the low end of our expected estimate. (Please note that due to rounding, subtotals may not precisely add up to the total figure.)

The funding from each source is broken down as follows:

† These totals reflect additional funding that was distributed in cases where the amount of the Speculation Grant awarded to an organization exceeded the eventual recommendation amount for that organization. This additional funding is not included in the figure for the total funding recommended in this round, but is included in the amount of funding expected to be distributed in total.

‡ These totals reflect funding that was recommended by Lightspeed Grants, but was subsumed by Speculation Grants already distributed by either SFF or Survival and Flourishing .Com (SFC). For the total funding recommended in this round, the subsumed funding is accounted for as being recommended by Lightspeed Grants. For the amount of funding expected to be distributed in total, the subsumed funding is accounted for as being distributed by either SFF or SFC.

Final recommendations

Most of the final endorsed recommendations of this round of the S-Process are listed below. These numbers have resulted from numerical inputs from both funders and Recommenders, which represented estimates of the marginal utility of granting to each organization. Note that:

  1. Some of the grants below might not happen if they are logistically difficult or time-consuming for the funder to finalize for some reason.
  2. Some additional grants might also appear on this list later as more details about them become clear.

SFF’s and Lightspeed Grants’ (LSG) recommendation processes ran concurrently, but with different groups of Recommenders. Jaan Tallinn was the largest funder in both the LSG and SFF rounds, and LSG and SFF had similar applicant pools and funders this round, so LSG grant amounts were decided prior to SFF grant amounts, and used as inputs to the SFF round to avoid accidentally over-funding applicants who applied to both sources. I.e., SFF amounts were decided after LSG amounts, targeting a total recommendation amount determined by SFF grant Recommenders and Funders. To make it easier to see grand totals at a glance, we are including LSG totals alongside the SFF totals in the table below:

Organization Amount (Jaan Tallinn via LSG) Amount (FLI) Amount (Jaan Tallinn via SFF) Amount (Jaan Tallinn via SFC) Receiving Charity Purpose
80,000 Hours $12,000 ($0)‡ $0 $0 $0 Effective Ventures Foundation (UK) General Support of 80,000 Hours
Acs Base Rate Forecast $27,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Agency Foundations Hackathon $9,000 ($9,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Agrawal AI Alignment Coordination $51,700 ($48,700)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
AI Forensics $0 $0 $178,000 $0 AI Forensics General Support
AI Impacts $0 $162,000 $179,000 ($72,000)† $0 Machine Intelligence Research Institute General Support of AI Impacts
AI Safety Aachen $3,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
AI Workshop (stai.uk) $10,000 ($5,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
AI X-Risk comms survey $2,400 ($2,400)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Alignment of Complex Systems Research Group (ACS) $231,500 ($185,500)‡ $0 $12,000 $0 Epistea, z.s. General Support of ACS
Association for Long Term Existence and Resilience (ALTER) $339,900 ($316,900)‡ $0 $0 $0 Association for Long Term Existence and Resilience (ALTER) General Support
Amodo Design Ltd. $41,000 ($35,000)‡ $0 $0 $0 N/A General Support
Apollo Research $277,169 ($239,500)‡ $0 $490,000 ($490,000)† $115,331 N/A General Support
Arb Research $133,900 ($92,900)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
BAISC $15,800 ($14,800)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
BERI-SRL Collaboration $0 $0 $52,000 $0 Berkeley Existential Risk Initiative General Support of BERI-SRL Collaboration
Berkeley Existential Risk Initiative $0 $0 $60,000 $0 Berkeley Existential Risk Initiative General Support
Bhatt AI Safety Research $40,000 ($40,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Bilski War Strategy Study $30,000 ($30,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Bloom’s Trajectory Models Study $52,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Bos Catalyze Impact $30,000 ($30,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Campaign for Responsible AI $250,000 ($250,000)‡ $0 $0 $0 Campaign for Responsible AI General Support
Campos Safer AI Project $325,800 ($144,800)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Carey AI Safety Research $191,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Carroll Developmental Interpretability $50,900 ($31,900)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Cavendish Labs Co. $88,000 ($0)‡ $0 $0 $0 Cavendish Labs, Co. General Support
Center for AI Safety Action Fund $22,125 ¶ ($0)‡ $0 $0 $397,875 ($600,000)† N/A General Support
Center for AI Safety, Inc. $16,154 ($0)‡ $0 $893,000 $0 Center for Artificial Intelligence Safety, Inc. General Support
Centre for the Governance of AI (GovAI) $0 $0 $5,000 $0 Effective Ventures Foundation General Support of GovAI
Chaudhry AI-Bio Policy $1,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Colbourn AGI Moratorium $30,000 ($30,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Community Health and Special Projects team at the Centre for Effective Altruism $113,000 ($0)‡ $0 $0 $0 Effective Ventures Foundation USA, Inc. General Support of Community Health and Special Projects team at the Centre for Effective Altruism
Coordination Project $0 $0 $29,000 $0 Center for Applied Rationality General Support of Coordination Project
Csendes Cavendish Scholarship $5,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Dupuis Cyborg Cognition $87,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Earth Law Center $0 $0 $24,000 $0 Earth Law Center General Support
Effective Altruism Sweden Rationality Research Project $126,600 ($60,600)‡ $0 $0 $0 Effektiv Altruism Sverige (Effective Altruism Sweden) General Support of Effective Altruism Sweden Rationality Research Project
EffiSciences $10,000 ($10,000)‡ $0 $0 $0 EffiSciences General Support
Elmore AI Moratorium Movement $88,900 ($69,900)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
ENAIS $55,000 ($55,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Epistea $229,000 ($40,000)‡ $0 $0 $0 Epistea, z. s. General Support
FAR AI, Inc. $472,577 ($411,600)‡ $0 $404,000 $0 FAR AI, Inc. General Support
Farnik Interpretability $60,000 ($60,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Filan AI Risk Podcast $2,200 ($1,200)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Fønix $0 $0 $378,000 $0 Effektiv Altruism Sverige General Support of Fønix
Foundation for American Innovation $0 $0 $189,000 $0 Lincoln Network (dba The Foundation for American Innovation) General Support
Fox Agency Research $32,500 ($31,500)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Freestyle Research $96,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Gleave Alignment Summit $187,500 ($134,500)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Global Shield $82,700 ($82,700)‡ $0 $0 0 Social and Environmental Entrepreneurs General Support of Global Shield
Greene Lab Cognitive Oversight $13,800 ($13,800)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Haddad Automating Experts $186,700 ($82,700)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Hayman MLAB Oxford $13,700 ($6,700)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Hebbar Cognition Theory $269,200 ($203,200)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Hiremath Interpretability Tools $1,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Howe RL Safety Research $5,900 ($4,900)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Hu Alignment Upskilling $10,600 ($9,600)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Jha Agent Foundations Research $100,400 ($85,400)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Jones Cybersecurity for AI $32,500 ($14,500)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Jose Interpretability Research $50,300 ($47,300)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Khan AI Debate Oversight $20,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Krachun AI Safety Research $9,900 ($8,900)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Lantern Bioworks $72,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Legal Priorities Project $302,900 ($199,900)‡ $0 $0 $0 Legal Priorities, Inc. General Support
Leveraging ESG for AIS $30,056 ($30,056)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Levoso Algorithm Interpretability $86,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Lightcone Infrastructure $0 $0 $410,000 ($250,000)† $0 Center for Applied Rationality General Support of Lightcone Infrastructure
LLM Safety Research $44,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Long-Term Future Fund $197,659 ($69,600)‡ $0 $564,000 $0 Effective Ventures Foundation USA General Support of Long-Term Future Fund
Marks Runway Crowdfunding $13,300 ($8,300)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Massachusetts Institute of Technology (Tegmark group) $0 $0 $223,000 ($1,000,000)§ $0 Massachusetts Institute of Technology General Support
MATS London Ltd $282,300 ($185,300)‡ $0 $0 $0 Berkeley Existential Risk Initiative General Support of MATS London Ltd
Miles AI Safety Info $97,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Monson Energy Surface Extraction $62,800 ($40,800)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Nabeshima Interpretability Study $88,900 ($69,900)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Newberry Bostrom Assistant Hire $59,200 ($57,200)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Online Team at the Centre for Effective Altruism $126,000 ($0)‡ $0 $0 $0 Effective Ventures Foundation USA, Inc. General Support of the Online Team at the Centre for Effective Altruism
OPIS Guide and Advocacy $15,000 ($15,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Orthogonal $493,000 ¶ ($438,000)‡ $0 $0 ($275,000)† $0 Ashgro Inc General Support of Orthogonal
Oxford China Policy Lab $0 $252,000 $1,000 ($13,645)† $0 Berkeley Existential Risk Initiative General Support of Oxford China Policy Lab
Palisade Research $401,627 ($195,500)‡ $0 $164,000 $0 Palisade Research General Support
PauseAI $20,000 ($20,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Payor Alignment Research $266,000 ($210,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Pope Alignment Research $10,800 ($9,800)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS) $0 $0 $186,000 $0 Epistea, z. s. General Support of PIBBSS
Raffel Alignment Methods Review $111,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Schnabel AI Safety Litigation $404,200 ($220,200)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Schons AI Pharma Regulation $40,400 ($30,400)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
SecureBio $0 $0 $548,000 $0 SecureBio, Inc. General Support
SecureDNA $32,930 ($0)‡ $0 $803,000 $0 SecureDNA Stiftung General Support
SERI ML Alignment Theory Scholars Program $297,818 ¶ ($0)‡ $0 $48,000 ($885,000)† $0 Berkeley Existential Risk Initiative General Support of SERI ML Alignment Theory Scholars Program
Shai Interpretability Research $75,500 ($57,500)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Shard Theory Model Internals $116,500 ($96,500)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Short Women in AI Safety $52,400 ($30,400)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
SLT Summit organizers $0 $0 $20,000 ($20,000)† $0 Center for Applied Rationality General Support of SLT Summit organizers
Stampy’s AI Safety Info $99,000 ($75,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Technologies for Pandemic Defense $0 $0 $86,000 $0 Convergence Analysis General Support of Technologies for Pandemic Defense
The AI Governance & Strategy team within Rethink Priorities $0 $0 $402,000 ($175,000)† $0 Rethink Priorities General Support of The AI Governance & Strategy team within Rethink Priorities
The Australian Responsible Autonomous Agents Group $0 $0 $83,000 $0 Federation University Australia General Support of the Australian Responsible Autonomous Agents Group
The Center for Election Science $0 $0 $96,000 $0 The Center for Election Science General Support
The Centre for Long-Term Resilience (Alpenglow Group Limited) $226,500 ($226,500)‡ $769,000 $60,000 ($577,000)† $0 Founders Pledge General Support of The Centre for Long-Term Resilience (Alpenglow Group Limited)
The Events team at the Centre for Effective Altruism $166,000 ($0)‡ $0 $0 $0 Effective Ventures Foundation USA, Inc. General Support for The Events team at the Centre for Effective Altruism
The Future Society, Inc. $0 $365,000 $11,000 $0 The Future Society, Inc. General Support
The Intrinsic Perspective $0 $0 $24,000 $0 SciArt Initiative, Inc. General Support of The Intrinsic Perspective
The Rethink Priorities Existential Security Team (XST) $0 $0 $49,000 $0 Rethink Priorities General Support of The Rethink Priorities Existential Security Team (XST)
Thibodeau Alignment Research $88,000 ($71,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Timaeus $29,400 ($0)‡ $0 $426,000 $0 Ashgro Inc General Support of Timaeus
Topos Institute $0 $0 $4,000 ($70,000)† $0 Topos Institute General Support
Van Nostrand Paradigms $19,300 ($16,300)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Van Nostrand Stimulant $22,500 ($22,500)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Van Nostrand Vegan Nutrition $60,000 ($25,000)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Vinelight $46,000 ($0)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Vipra AI Trade Safety Assessment $54,900 ($35,900)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Vymenets Translations $7,300 ($4,300)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Wang AI Safety Research $244,500 ($233,500)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Ward AI Safety Outreach $24,600 ($18,600)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Wiebe Replication Crisis $16,800 ($15,800)‡ $0 $0 $0 TBD via Lightspeed Grant announcement TBD via Lightspeed Grant announcement
Wytham Abbey $76,000 ¶ ($0)‡ $0 $0 ($275,000)† $0 Effective Ventures Foundation General Support of Wytham Abbey

† Parentheses indicate a speculation grant awarded to certain grantees, prior to the completion of the S-Process round. If the speculation grant amount was higher than the S-Process recommendation, it subsumed the S-Process recommendation, so only the speculation grant amount is meant to be granted. If the speculation grant was less, it was deducted from the S-Process recommendation, so the total granted is meant to be the amount of the S-Process recommendation.

‡ Parentheses marked by a ‡ indicate Lightspeed Grants Tranche 1 grant amounts, awarded to Lightspeed Grant recipients. The total amount of each Lightspeed Grant is found left of the ‡ amount. Lightspeed Tranche 2 grant amounts can be found by subtracting the total LSG amount from the ‡ parenthetical Tranche 1 amount.

§ Jaan Tallinn unilaterally awarded $1,000,000 in funding to Massachusetts Institute of Technology (Tegmark group) prior to the completion of the S-Process round. Since this grant amount was higher than the S-Process recommendation, it subsumed the S-Process recommendation, so only the amount in parentheses is meant to be granted.

¶ The ¶ symbol indicates that a speculation grant was awarded to this grantee, the value of which can be found in a separate column of the same row in parentheses marked by †. The speculation grant subsumed both the Lightspeed Grants Tranche 2 recommendation and the S-Process recommendation so only the speculation grant amount and the Lightspeed Grants Tranche 1 recommendation are meant to be granted. Therefore, the total amount meant to be granted to this grantee is the speculation grant amount found in parentheses marked by † plus the Lightspeed Grants Tranche 1 grant amount found in parentheses marked by a ‡ .

“*” Asterisks demarcate grants that were “controversial” in that one or more recommenders wished to publicly disendorse the grant. Multiple asterisks correspond to multiple disendorsements. (There were no asterisks this round.)

Note that the S-Process system is designed to generally favor funding things that at least one recommender is excited to fund, rather than things that every recommender is excited to fund. As such, the grant recommendations above do not especially represent the “average” opinion of the group in any sense.

Also note that additional grants may appear on this announcement as SFF continues to confirm their details.

Freedoms compatible with the S-Process

The S-Process is designed to allow a lot of freedom and autonomy for the Recommenders and the funders. Funders retain the right and freedom to make and/or recommend grants that the S-Process, as a whole, did not endorse, based on information the funders learn from the S-Process. As well, the Recommenders and funders are free to communicate with each other as much as they want outside of the S-Process structure. As such, while the S-Process might be useful in helping the funders learn about grant-making opportunities and opinions, the funders need not be limited by the outputs of the S-Process in order for it to function.

Sharing of recommender input documents

Each Recommender’s own inputs to the process (both initial and final versions) are free for that Recommender to share privately or publicly in any form and for any use-case, as long as they anonymize any other Recommenders who have requested anonymity, and respect the privacy of the organizations who applied to the round. In particular, the S-Process does not require Recommenders to avoid sharing their own (appropriately anonymized) inputs, even when sharing some documents might make it somewhat easier in principle for an adversary to reverse-engineer the inputs of other Recommenders. The spirit of the S-Process is, however, to discourage this sort of intentionally adversarial de-anonymization, as it is disrespectful of the Recommenders who contributed their time and effort under the conditions of anonymity.