Mapping the individual, social, and biospheric impacts of Foundation Models

Abstract.

Responding to the rapid roll-out and large-scale commercialization of foundation models, large language models, and generative AI, an emerging body of work is shedding light on the myriad impacts these technologies are having across society. Such research is expansive, ranging from the production of discriminatory, fake and toxic outputs, and privacy and copyright violations, to the unjust extraction of labor and natural resources. The same has not been the case in some of the most prominent AI governance initiatives in the global north like the UK’s AI Safety Summit and the G7’s Hiroshima process, which have influenced much of the international dialogue around AI governance. Despite the wealth of cautionary tales and evidence of algorithmic harm, there has been an ongoing over-emphasis within the AI governance discourse on technical matters of safety and global catastrophic or existential risks. This narrowed focus has tended to draw attention away from very pressing social and ethical challenges posed by the current brute-force industrialization of AI applications. To address such a visibility gap between real-world consequences and speculative risks, this paper offers a critical framework to account for the social, political, and environmental dimensions of foundation models and generative AI. Drawing on a review of the literature on the harms and risks of foundations models, and insights from critical data studies, science and technology studies, and environmental justice scholarship, we identify 14 categories of risks and harms and map them according to their individual, social, and biospheric impacts. We argue that this novel typology offers an integrative perspective to address the most urgent negative impacts of foundation models and their downstream applications. We conclude with recommendations on how this typology could be used to inform technical and normative interventions to advance responsible AI.

1. Introduction

Since the release of ChatGPT at the end of 2022, there has been considerable and sustained interest across policy, academia, and public discourses in the risks posed by artificial intelligence (AI), and particularly generative AI. Though ChatGPT may have generated the most column inches, its underpinning model—GPT 3.5 and, subsequently, GPT 4—was just one example of wider developments in the area of so-called “foundation” or “frontier” models.

Foundation models are AI technologies trained on very large, “broad” datasets that can be applied to a wide range of tasks and purposes (Bommasani et al., 2022). Their general, unspecified purpose and widespread deployment leads to concerns about potential unanticipated impacts when used in novel areas. These models are considered to form the “foundation” or base architecture of other systems. For example, a number of new applications such as ChatGPT and the conversational features of the Bing search engine, have been built on top of successive versions of OpenAI’s GPT foundation model, which has been designed for natural language processing tasks.

While the techniques involved in developing and deploying foundation models are not new, the scale of the data used for training, the coordination of global networks of labor to process these massive amounts of data, the increased architectural complexity, and the development of increasingly powerful processors and other computational resources have together made possible new levels of predictive and generative sophistication. Due to the resources required to produce foundation models, the most prominent ones known to exist are the products of large and well-financed technology companies, such as Anthropic, Cohere, Hugging Face, Meta, OpenAI, Microsoft, and Alphabet (the latter two being substantial funders of other firms).

Responding to the rapid rollout and large-scale commercialization of foundation models, large language models (LLMs), and generative AI, an emerging body of work is exploring the myriad impacts these technologies are having across society. Such research is expansive. Prominent topics range from the production of discriminatory, fake, and toxic outputs and privacy and copyright violations to the unjust extraction of labor and natural resources. Foundation models present complex issues for law, policy, and practice, which arise concretely from the intertwined array of socio-technical systems in which their design, development, and use are embedded. These technologies pose risks and hazards that emerge from real-world contexts. Such issues arise from the global supply chains and labor sources that support their production, as well as the market forces and regulatory environments that influence their funding and financing. Risks likewise arise from the sociohistorical realities and legacies of inequity and exclusion that shape the data on which they are trained and from the patterns of privilege and socioeconomic stratification that influence the composition of the project teams that build them. All these as real-world contexts lead to the real-world consequences that have been the subject of much critical and responsible AI research on the impact of foundation models and generative AI over the past year. Yet, despite the importance of centering this empirically oriented work, much of the agenda-setting policy and public discourse emerging primarily from the main geographical centres of innovation on foundation models (and relatedly LLMs and generative AI) has focused on sensational, catastrophic, and largely speculative risks relating to extreme and hypothetical scenarios.

The narrative of catastrophic risks was amplified by much mainstream, English language, media coverage of foundation models throughout 2023, fueled by high profile statements declaring potential existential threats from AI (Metz, 2023b, a). Most notably a statement published in May 2023, and signed by many prominent figures in the field of AI, claimed that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war” (Center for AI Safety, [n. d.]). While the research behind these claims is scant and broadly disputed, this narrative has reached far into predominant policy and regulatory discussions, including high profile international initiatives in the global north like the UK’s AI Safety Summit (GOV.UK, 2023) and the G7’s Hiroshima AI Process (OECD, 2023a). Such processes, while presenting valuable opportunities to bring together the international community in discussing the extant risks that AI presents to people, society, and the planet, and corresponding approaches to needed regulation, have been dominated by technical discourses with narrow framings of “safety” and have prioritized discussions of speculative global catastrophic or existential risks of AI.

Concerns about this narrowing of AI policy and regulatory discourse have been echoed by various academic communities who have questioned the lack of “ideological and demographic diversity” of the AI safety field (Lazar and Nelson, 2023); and the privileging of speculative narratives as effective distractions from existing and well-documented risks and harms of AI systems today (Vallor and Luger, 2023; Helfrich, 2024; Gebru and Torres, 2024). A wealth of evidence demonstrates in particular the disproportionate negative impacts that AI systems have on marginalized communities—communities who are also underrepresented in decision-making processes regarding the design, development, deployment, or governance of AI. These issues, which are linked to historical and entrenched power asymmetries, are now being exacerbated by AI technologies, and risk becoming increasingly more urgent.

Therefore, through a review of the literature on the harms and risks of foundation models and generative AI, in this paper, we aim to address this visibility gap between real-world consequences and speculative risks. We propose a critical framework capable of adequately illuminating socio-technical risk contexts and attending to the social, political, and environmental dimensions of foundation models and generative AI. Through reviewing the existing evidence base our research maps a diverse set of risks and harms impacting individuals, communities, society, and the biosphere.

The paper starts with a discussion on the theoretical framework employed to investigate risks and harms in the context of foundation models. This framework abductively informs the methodological approach, which is detailed in section 3. Section 4 present our findings by delineating three distinct levels of algorithmic impacts—individual, social and biospheric. We conclude discussing how the integrative perspective we offer can aid in understanding the negative impacts of foundation models, and can help shape responsible AI futures.

2. Theoretical Lens: Expanding views on algorithmic risks and harms

There is a growing and multidisciplinary body of literature which explores the potential risks and harms posed by the relatively recent widespread popularization of foundation models. Although these risks and harms share many similarities with those presented by other types of AI, and indeed other data-intensive technologies more generally, it is crucial to examine how the proliferation and widespread adoption of foundation models may introduce new challenges, as well as further complicate existing ones.

The concept of risk is pervasive in modern societies and has become a central aspect of the AI governance discourse. Sociological studies have argued that theories of risk management and securitization emerged as a means to grapple with the uncertainties of modernity (Beck, 1992) and “making the future secure and certain” (Schuilenburg, 2017). The understanding of risk and harm, as well as how they interrelate, varies greatly across domains. For legal, policy, organizational, or actuarial purposes, risk is typically approached through quantifiable measures and anticipatory models of potential adverse or harmful events such pandemics, financial losses, health and safety incidents, or operational disruptions (Dionne, 2013; Power, 2009; NIST, 2023; Mäntymäki et al., 2022). Risk is therefore not inherently an issue of morality and can be understood as a context-dependent and relational concept—that is, expressed in relation to how factors like technical developments, organizational structures, or innovation may adversely impact a desired future and be variously interpreted by different actors (von Scheve and Lange, 2023). For example, consider a company which uses a job applicant screening software and has discovered that it makes racially biased screening recommendations. Where policy makers and civil society may be concerned about the various ways in which the technology undermines individual rights and social equity, a company might instead be more likely to focus on the risk of reputational damage and how this could in turn result in public distrust, low technology adoption, and commercial impacts.

Critical social science perspectives underline the multiple ramifications of identifying, analyzing, and managing risk associated with the impacts of technology in society (Stahl et al., 2022; Wirtz et al., 2022). In this, some areas of research critically examine the various ways in which the risks and harms resulting from data and algorithmic systems are patterned across society at different levels (Lupton, 2013; Canning and Tombs, 2021). For instance, the field of science and technology studies (STS) has a well-established body of work exploring the interdependencies between technology and society, challenging the widespread deterministic view of technology as operationally autonomous and inherently value-neutral, and of innovation as following a predictable linear trajectory (MacKenzie and Wajcman, 1999). When it comes to examining the impacts of AI, scholars in the fields of STS, critical data studies, and socio-legal studies have proposed to conceptualize the harmful effects of data-intensive technologies as issues relevant to human rights, social justice, cultural institutions, the rule of law, the public sphere, economic systems, and the environment (Malik et al., 2022; Crawford, 2021; Williams and Edge, 1996; Marjanovic et al., 2022; Stoilova et al., 2021; Domínguez Hernández et al., 2023; Smuha, 2021). It is also from a critical perspective that some scholars increasingly highlight the need to approach algorithmic governance not only in terms of speculative notions of risk, but also in response to actual and observed harms to humans and the environment (Ferri and Gloerich, 2023).

To build an understanding of risks and harms of technology from a moral and ethical standpoint, it is key to consider the questions of how power is distributed across the entire ecosystem of actors, where and how power manifests, and how asymmetries of power and information relate to differential impacts in society. Here it is helpful to draw insights from feminist and data justice scholarship, socio-legal studies, and the environmental justice literature. In particular, the concept of intersectionality makes explicit how social hierarchies manifest in different outcomes and experiences depending not only on the identity of the impacted individual or group, but also on the multiple compounding intersections of social identity and standing. These intersections go beyond crude constructions and classifications of individuals in society, to recognize the ways in which nuances of context, identity, and circumstance distinctly shape people’s understanding and experiences of harm (Crenshaw, 1991).

To capture these differential impacts as they relate to the proliferation of foundation models, in this paper we propose, following Leslie and Rossi (Leslie and Rossi, 2023), to assess risks and harms according to their individual, social, and biospheric levels of impact, contributing to an emerging dialogue within the FAccT and AI ethics communities (Simbeck, 2022; Weidinger et al., 2022). This perspective allows for a nuanced understanding of the observed and anticipated impacts of foundation models by recognizing the inherent socio-technical and globally entangled nature of these technologies. We use this framework to map the risks and harms found in the literature to date and offer an integrative perspective to address the most urgent negative impacts of foundation models and their downstream applications.

3. Methods: snowball and structured search

We conducted a review of literature using two complementary and reinforcing qualitative literature review approaches which have been shown to be useful in studies aimed at surfacing patterns and trends in corpora of academic research (Wohlin et al., 2022).

We first applied a snowball approach (Wohlin, 2014), crowd-sourcing among all the co-authors, a core set of recent papers focused broadly on algorithmic harms and risks (Bender et al., 2021; Birhane et al., 2023; Bommasani et al., 2022; Derczynski et al., 2023; Jones and Steinhardt, 2022; Kasirzadeh and Gabriel, 2023; Shelby et al., 2023; Weidinger et al., 2022). These papers were selected based on our inclusion criteria of covering a broad range of multidisciplinary literature addressing risks and harms associated with foundation models. Populating a Zotero database, we followed snowball sampling approaches starting from the set of core articles and then carrying out a manual search using the references in those articles. Forward snowballing was used to identify new papers that focus on algorithmic risks and harms citing those in the core list using Google Scholar and Internet-based searches to access the abstract and full-text. We identified a total of 64 articles including journal articles, conference papers, and pre-prints drawing broadly on technical and socio-technical literature in which key risks and issues of foundation models, LLMs, and generative AI are described historically and contemporaneously. The search inclusion criteria covered English language publications and the most recent publication was of October 2023.

Complementing the snowball approach, we conducted a structured database search to identify primarily academic articles with a focus on the risks and harms associated with foundation models, LLMs, and generative AI.

Inclusion criteria covered English language journal articles, including conference papers and pre-prints, but excluding grey literature, monographs, commentaries, correspondences, and opinion pieces. The following electronic databases were searched: arXiv, ACM Digital Library, IEEE, Scopus, and Web of Science. After the removal of duplicates and non-relevant papers, there were 114 papers in total for analysis, of which 11 were also papers in the snowball search results. The resulting set of papers were published in the period January 2018 to July 2023. The start date aligns with the date of appearance in the literature of transformer architectures which underpin LLMs, such as the BERT large language representation model (Vaswani et al., 2017).

We applied an abductive approach (Tavory and Timmermans, 2014; Vila-Henninger et al., 2022) to analyze the combined corpus of literature 167 papers. The abductive approach was used to code the results and subsequently map key relationships between the risks and harms found in the literature guided by our theoretical framework described in Section 2. Firstly, we extracted applicable keyword information from the abstract of each primary study as part of an initial coding process. The resulting keywords served largely as summative and process attributes of a risk or type of harm relevant to foundation models or LLMs. Three researchers subsequently clustered these attributes and assigned them into parent categories. The codes were subsequently refined by all co-authors (see Appendix for search strategy). In this stage, we considered whether at least one of the risks and harms listed in the paper implied or directly referred to individual, social, or biospheric impacts to map the papers onto our three-level framework. These codified findings are further analyzed in Section 4.

3.1. Limitations

We note that our structured search showed a prevalence of preprint papers (non-peer review) in the result set, some of which had been cited extensively in both preprint and peer-review articles. We used citations only as a broad indication of the preprints’ impact. While some preprints were highly cited, these may have not undergone a rigorous peer-review process, and their findings should be interpreted with caution. In our sample, we analyze articles solely as an indicator of the range of concerns raised in the existing literature. The selected papers were also limited to the English language, and date range of January 2018 to July 2023. This range by default provided only a snapshot of a growing and multidisciplinary body of work which is arising in response to the fast development and roll-out of foundations models and LLMs, especially generative AI related studies. In addition, our structured literature search, which is not intended as systematic, only centred academic articles as a relevant account of the discourse on risks and harms of foundation models and LLMs to inform governance. We also note that this body of research is not focused exclusively on risks and harms, but often addresses, or points to, mitigations and governance of foundation models and AI at large. An analysis of other non-academic and informal sources was not part of this study and it warrants additional research. Notwithstanding, the codified mappings provide a substantive view of the research landscape and the range of concerns that are being raised around foundation models and LLMs, their development, implementation, and applications.

4. Mapping individual, social, and biospheric impacts of foundation models

The result set of 14 risks and harms coding (Table 1) is telling of the wide-ranging and multifaceted challenges posed by foundation models and AI systems at large. Papers typically engage with three or more aspects, exemplifying the broad spectrum of concerns in the literature to date. For instance, Zhuo et al. (Zhuo et al., 2023b) delve into potential biases, toxicity, and issues related to the reliability and robustness of ChatGPT. Notably, a substantial set of search results focuses on bias and discrimination. These are most typically related to LLMs, and include works such as van der Wal et al. (van der Wal et al., 2023) and Huang et al. (Huang et al., 2023b), which address multiple biases and harmful stereotypes, and others, like Abid et al. (Abid et al., 2021), Felkner et al. (Felkner et al., 2023), Ovalle et al. (Ovalle et al., 2023), and Gadiraju et al. (Gadiraju et al., 2023) that concentrate on specific societal biases. The result set also reveals that papers that focus on specific or a limited number of risks and harms often scrutinize issues of unreliable performance of foundation models, or misinformation and propaganda. While other issues such as lock-in and opacity, or overdependence in human-computer interaction, are less prevalent in the result set, it is important to note that their absence does not imply that they are not identified as relevant within the existing literature. Instead, they are simply not the primary focus of most papers in our dataset.

In the following, we elucidate three distinct levels of algorithmic impact (individual, social, and biospheric) with examples stemming from the literature and categorize the risks and harms emerging from our search using this typology (see Table 1). This lens on algorithmic harm aims to account for the intersectional dimensions and socio-technical embeddedness of foundation models with the aim to better inform potential technical and normative interventions to advance responsible AI.

4.1. Individual risks and harms

It is now undeniable that AI systems can have adverse impacts to individuals which may be physical, psychological, or financial, but may also negatively affect a person’s dignity, reputation, or fundamental rights and freedoms. Indeed, individual-level risks and harms arising from the implementation of data-intensive technologies have been largely documented in the literature, with individual impacts having immediate as well as long-lasting implications for those at the receiving end of such harms (Raymond, 2017). Equally so, a focus on individual rights has been at the core of conceptions of data protection, privacy, and security within digital regulation such as the European Union’s General Data Protection Regulation (GDPR), where protection and redress mechanisms privilege notions of personal data or data subjects (Smuha, 2021).

Table 1. A mapping of foundation models risks and harms according to their level of impact (individual, social, biospheric). The table shows the number of papers across categories of harm and levels of impact, and the percentage (in brackets) of the total number of papers in the set in which the risk/harm appears. Papers appearing in more than one category were counted more than once. See Appendix for an explanation of each risk and harm category.

Risks and Harms Category	Individual	Social	Biospheric
Bias and societal prejudices	71 (42.5%)	71 (42.5%)	0 (0%)
Misinformation, disinformation and propaganda	45 (26.9%)	45 (26.9%)	1 (0.6%)
Unreliable performance	43 (25.7%)	40 (24%)	0 (0%)
Cybersecurity risks and harms	37 (22.2%)	27 (16.2%)	0 (0%)
Privacy risks and harms	28 (16.8%)	21 (12.6%)	0 (0%)
Systemic social and economic risks and harms	18 (10.8%)	24 (14.4%)	1 (0.6%)
Legal and regulatory violations	17 (10.2%)	17 (10.2%)	0 (0%)
Environmental effects and ecological disruption	8 (4.8%)	8 (4.8%)	19 (11.4%)
Misuse	13 (7.8%)	14 (8.4%)	19 (11.4%)
Lock-in and opacity risks	13 (7.8%)	15 (9%)	0 (0%)
Overdependency in human-computer interaction	11 (6.6%)	6 (3.6%)	0 (0%)
Data risks and harms	7 (4.2%)	7 (4.2%)	0 (0%)
Value misalignment	2 (1.2%)	2 (1.2%)	0 (0%)
Extreme or catastrophic risks and harms	2 (1.2%)	2 (1.2%)	1 (0.6%)

One of the biggest ethical concerns associated with AI systems has been their role in creating discriminatory, biased, and unfair outcomes which impact an individual’s safety, health, wellbeing, and integrity. The proliferation of applications based on foundation models has not only augmented these concerns but has given rise to a host of new ethical, privacy, and safety issues. Biases and social prejudices present in training datasets and encoded in foundation models trickle down to applications like chatbots or image generators, thereby reproducing existing patterns of harm, exclusion, and discrimination (Abid et al., 2021; Steed et al., 2022; Gadiraju et al., 2023). For instance, chatbots—such as OpenAI’s ChatGPT or Google’s Bard—are capable of producing natural language responses to prompts through complex statistical associations (Bommasani et al., 2022; Zhao et al., 2023). These responses are enabled by the vast amounts of data that the foundation models have been trained on, comprising mostly of internet-scraped datasets that have been shown to encode biased and harmful associations across protected categories of religion, disability, gender, race, and ethnicity (Nadeem et al., 2020; Abid et al., 2021; Deshpande et al., 2023), and also at the intersections of these characteristics (Tan and Celis, 2019; Guo and Caliskan, 2021).

When it comes to issues of privacy and online safety, LLMs have been found to be effective tools for fraudulent activities such as scams, impersonation, and phishing, given their capability to generate personalized and highly convincing messages at scale (Hazell, 2023; Derner and Batistič, 2023; Neupane et al., 2023). Research has also shown that LLMs are prone to leaking private or sensitive information contained within their training data corpus (Carlini et al., 2023; Kandpal et al., 2022). Because these models are trained on data scraped from the Internet, privacy violations might take place unbeknownst to impacted individuals thereby limiting their ability to consent or opt out (O’Neill and Connor, 2023). Furthermore, models can associate different pieces of data to an individual, which poses privacy and security risks wherein sensitive information or even wrongful associations are revealed without consent (Huang et al., 2023a). Indeed, in some instances, chatbots have been used with the intention of extracting protected information and bypassing restrictions through the use of malicious methods like prompt hacking, jailbreaking (Rao et al., 2023; Wang et al., 2024), and prompt injection (Perez and Ribeiro, 2022; Greshake et al., 2023).

Lastly, the proliferation of foundation models complicates the already significant negative impacts digital technologies have on people’s psychological wellbeing. In the last decade, researchers have raised concerns over how hyperconnectivity and unhealthy relationships with technology can have serious mental health implications and influence how people form individual identities and construct self-hood (Albrechtslund and Lauritsen, 2013; Lupton, 2016; Brubaker, 2020). These issues are now exacerbated with the emergence of personalized AI chatbots capable of producing human-like responses. New research into the use of these conversational agents for mental health support warns about the potential for people’s overdependence on these agents and the risk of social isolation (Ma et al., 2023). Similarly, documented cases of users self-harming as a result of exposure to toxic interactions with AI agents evidence the devastating impacts these technologies can have on vulnerable individuals (Walker, 2023; Xiang, 2023; Lee et al., 2023). For example, prior research has demonstrated that individuals who are isolated may be more vulnerable to misleading information on social media platforms (Bonsaksen et al., 2021). When individuals are subject to algorithmic nudging and personalized information campaigns, they are at risk of loss of autonomy, dignity, and integrity (Rosenberg, 2023).

4.2. Social risks and harms

While the rise of foundation models has generated huge expectations about their positive transformational potential for society in many domains, there are also concerns about how their adoption can contribute to worsening existing social inequalities, biases, and injustices. Beyond individual risks and harms, there are different ways in which technology can have wider collective and societal impacts. One way to conceptualize collective risks and harms is to look at the compounded or aggregate impacts to a particular group, or to the functioning of a society as a whole (Malik et al., 2022).

Risks and harms to communities can be characterized by how particular demographics and group formations within society are distinctively impacted by technology, particularly when it comes to vulnerable groups with protected characteristics, e.g., migrants, refugees, LGBTQI+ communities, Muslims, the elderly. As we have shown, not only can these technologies produce responses that are discriminatory and psychologically triggering for individual users, but the reproduction of hegemonic views, harmful stereotypes, and biases can have detrimental consequences at the collective level, affecting marginalized groups and communities including those who are not direct users, or are far removed from the technology. For instance, Abid et al. (Abid et al., 2021) demonstrate the persistent religious bias present in foundation models, noting that prompts containing the word “Muslim” produced responses with violent language. These negative associations, in turn, can spur targeted hate and disinformation campaigns centered around anti-Muslim or Islamophobic sentiments.

Collective forms of harm can also be understood from a relational lens. The relational lens captures the adverse impacts to social relationships of interdependence, trust, and solidarity, thereby negatively impacting people’s experiences of belonging, or their capacity to flourish through their relationship to others. A rich and interdisciplinary body of research has shed light on the range of interpersonal effects arising from people’s interactions with algorithms, including the affective impacts of the algorithmic sorting of content on social media feeds (Eslami et al., 2015; Bucher, 2017; Stark, 2018) and the proliferation of misinformation as detrimental to the quality of the public sphere and democratic deliberation (Bennett and Livingston, 2018; Tsfati et al., 2020; Vaccari and Chadwick, 2020). Foundation models in particular can increase the scale and speed at which disinformation campaigns can be disseminated across the information ecosystem (Solaiman et al., 2019; Pan et al., 2023). As generative AI applications powered by foundation models flood the public sphere with fake information, there is a risk of eroding public trust in the information that circulates online, further fueling social polarization and the creation of echo chambers (Kirk et al., 2023; Simmons, 2023).

Lastly, algorithmic systems are inextricably embedded in wider social, political, and institutional structures, and as such they also have a bearing on those structures (Williams and Edge, 1996). To grapple with the impacts that foundation models can have at a structural level, one must consider geopolitical, socioeconomic, legal, and organizational manifestations of power (Leslie and Rossi, 2023). For instance, some scholars have warned about the tight concentration of power arising from the limited pool of actors controlling technological capabilities globally (Crampton, 2019; Deibert and Pauly, 2019; O’Hara et al., 2021); the influence of these power asymmetries on the development of international policies, standards, and regulations for technological practices (Baik, 2020; Chomanski, 2021; Cohen, 2019); and the overwhelmingly invasive data practices carried out by governments and companies (Eubanks, 2018; Fourcade and Gordon, 2020; Tréguer, 2019).

The collective and deeper structural effects of these technologies are difficult to anticipate and measure as they manifest over time as AI becomes more widespread and embedded in society. Nonetheless, it is helpful to analyze and situate foundation models in the context of well-known social implications linked with digital technologies. For instance, going back to the issues related to the public sphere, one could argue that the effortless production and spread of targeted misinformation and propaganda enabled by foundation models, can have lasting effects on wider cultural, social, and political structures, namely established democratic processes, representation, participation in public life, and institutional trust. Given the evidence of public and private actors deploying technologies to generate manipulative content to sway public opinion (Funk et al., 2023; Collier et al., 2022), scholars have warned about an increasing risk to social cohesion given the mounting erosion of trust in democratic structures (Kreps and Kriner, 2023).

4.3. Biospheric risks and harms

Lastly, biospheric risks and harms refer to the potential adverse impacts to ecological systems and environments, as well as wider impacts to the biosphere at the planetary level (Potter, 1999). Importantly, this category also includes harms to the individuals and communities negatively impacted by environmental effects and ecological disruptions resulting from AI development and use. In other words, biospheric, social, and individual risks and harms are fundamentally intertwined. A good entry point to understand biospheric impacts is to look at the largely hidden socio-material entanglements of data-intensive systems. That is, the complex supply chains, labor, and underpinning materials crucial to AI infrastructures—including the Rare Earth Elements (REEs) that are necessary to manufacture the semiconductors essential to computer hardware. The global nature of the supply chains of foundation models pose individual, social, and environmental risks by entangling processes of extraction, knowledge production, and ultimately decision-making (Rella, 2023).

The development of supercomputers which utilize REEs, and enable foundation models to be trained, has been found to have significant material footprints globally (Anton et al., 2020). The mining of silicon and other rare metals not only cause environmental harms, but can rupture local communities (Crawford, 2021; Al Rawashdeh et al., 2016) and cause individual harm to the workers mining these elements (White and Shine, 2016). Furthermore, the negative consequences are usually borne disproportionately on vulnerable or marginalized communities largely located in the global south (Bullard, 1993). Thus, when put in the context of global infrastructures, the interrelated nature of these risks and harms becomes clearer. Robbins and van Wynsberghe argue that the “interconnectedness of AI with ecological, social, and economic systems” (Robbins and van Wynsberghe, 2022) pose very real risks regarding societal lock-in to AI infrastructures. That is to say that beyond a certain point, the nature of the supply chains and materials powering certain AI infrastructures cannot be changed, whilst ecological harms would continue to multiply because of resultant carbon emissions, REE mining, and other processes. Given these potential path dependencies, we are currently at a crucial juncture for developing ethical governance mechanisms that address the interlocking risks of the biospheric and human facets of AI ecosystems.

The development and implementation of data-intensive systems like foundation models is also known to have high local environmental costs in the form of energy consumption required to train increasingly large and complex machine learning systems (Kirk et al., 2023; Luccioni et al., 2022), and heavy water consumption needed for data center cooling (Li et al., 2023d). The amount of compute needed to train large-scale models has doubled every 3.4 months since 2012 (Amodei et al., 2018; Hao, 2019), which translates to higher energy expenditures, and the use of costly resources even if the improvements in model accuracy are modest (i.e., diminishing returns in terms of model accuracy come at increased computational cost, see (Schwartz et al., 2019)). For example, the carbon emissions of training Google’s BERT were roughly those of a transatlantic flight (Strubell et al., 2019). All of this at a time when curbing our global emissions is crucial to slowing climate change down and effectively mitigating its effects (United Nations, [n. d.]). Efforts to develop responsible technologies that minimize their cascading impacts on the environment are hindered by the difficulty in comprehensively assessing carbon and energy footprints of training large models. Tools have emerged enabling practitioners and developers of these energy intensive technologies to monitor and track their models’ emissions (Anthony et al., 2020).

Although the environmental costs of AI are borne by everyone on the planet in terms of negative externalities contributing to climate change and ecological deterioration, they are not uniformly distributed among the world’s populations or regions. The allocations of benefits and risks replicate existing patterns of environmental injustice, coloniality, and “slow violence” (Nixon, 2013), in which a disproportionate exposure to risks and harms is borne by marginalized communities; be these in terms of pollution, destruction of local ecosystems, or involuntary displacement. By contrast, the majority of the economic benefits reaped by the use of the model mostly go to the model’s proprietary owner(s), even when these technologies adopt an open source ethos (Liesenfeld et al., 2023).

Finally, the impact of pollution and ecosystem degradation on interdependent and connected biological systems (including sentient non-human animals) cannot be ignored (Strubell et al., 2019; Coghlan and Parker, 2023). AI has the potential to put non-humans at further risk (e.g., animals and the ecological systems in which they live) through surveillance, monitoring, and testing for which the benefits are largely directed to humans and society. Potential harms arise from data collection on animals in intensive factory farming to increase productivity, or data tracking of protected wildlife, which hunters and poachers can hack to illegally hunt or trade animals (Bossert and Hagendorff, 2021; Coghlan and Parker, 2023). Beyond these physical harms to animals, the functional use of AI in image recognition and as recommender systems demonstrate biases in differentiating species borne out from underlying social factors (Hagendorff et al., 2023). There is an urgent need to understand and respond in a systemic way to the interconnected and shared vulnerabilities of humans, animals, and the ecosystems in which we live and share (Coghlan and Parker, 2023).

4.4. Summary of findings

Our review shows that a large proportion of papers (40%) raise concerns about the potential for foundation models to perpetuate and amplify hegemonic views, harmful stereotypes, and societal and behavioral biases on an unprecedented scale. A similarly large set of papers (20%) also point to issues related to the creation and spread of misinformation and propaganda as well as the potential exploitation of these models for fraudulent services and cybersecurity attacks by malicious actors. Furthermore, the individual and societal impacts stemming from inconsistent or undesirable performance constitute a significant part of the mapped academic literature, making the case for cautious approaches to the deployment and use of such models.

Although less prevalent, a subset of the literature mapped point to concerns in connection to a range of technical and supply chain challenges of foundation models, and the potential compounding effects of those challenges as foundation models become more widespread. These range from the reliance on proprietary software and lack of transparency that enable lock-in and opacity risks to issues like outcome homogenization, narrowing of the market or monopolization, and the perpetuation of inequalities, which affect societal and economic structures. Our mapping also reveals that approximately 10% of the mapped papers address the environmental risks and harms associated with foundation models. While the primary focus of these latter works is on the biospheric-level impacts, some authors explore the interconnections between the adverse impacts of foundation models on ecology and animals and their consequential impacts on individuals and societies. This underscores the need for expanded exploration and attention to holistically understand the complexities surrounding the environmental adverse impacts of foundation models. Lastly, despite the extensive emphasis in much of the governance discourse, less attention is given in the academic literature to the extreme scenarios of existential, catastrophic, and other speculative risks of foundation models. These findings highlight the current visibility gap between real-world consequences and speculative risks, and sheds light on the areas requiring urgent and greater attention and efforts.

5. Discussion: Grappling with the scale and interconnectedness of foundation models

Foundation models are highly consequential technologies which have sparked discussions about a spectrum of transformational impacts from the most promising to the most concerning. They pose unprecedented governance challenges but also offer an opportunity to draw lessons from, as well as re-examine, the landscape of socio-technical impacts of data-intensive technologies. While emergent and novel, foundation models do not arrive de novo; they are “built on an installed base” (Star, 1999) and inherit the tools and methods of prior generations of machine learning and neural network technologies. As such, many of the foundation-model-related risks and harms that we highlight are likely to appear familiar because they occur, to some degree, in adjacent and precursor technologies.

Even so, two key differentiating characteristics of foundation models are their massive scale and widespread embeddedness. Foundation models comprise hundreds of billions of parameters, trained on mountains of data, that consume enormous resources for both training and deployment. In particular, the scale of foundation models means that the risks and harms they present are not only likely to be magnified and amplified, but that this will happen in ways which transcend national and political boundaries, requiring a multi-pronged and transnational response. Harms that may have been minimal or just minimally attended to in prior generations of technology are now made visible and urgent. For example, data-intensive systems have been criticized for some time for their carbon footprint (Stoll et al., 2019), but the high visibility and widely publicized demands of foundation models have brought this issue to the fore to the point that it is unsurprising to see this aspect discussed in the technical literature (Bommasani et al., 2022).

Another differentiating characteristic of foundation models is their embeddedness. Foundation models are conceptualized and architected as the base models for many and diverse types of downstream applications. The embeddedness of foundation models renders them invisible yet pervasive. As a result of their platformized architecture, foundation models form the basis of many thousands of extensions, and as such, the negative impacts and harms stemming from foundation models may be obfuscated and rendered relatively intractable. These two characteristics—scale and embeddedness—position foundation models to be both highly adaptive, highly elusive, and highly dangerous. We argue that any assessment of risks and harms should account for these socio-technical interdependencies, and any design of mitigations and policy responses should be commensurate with the level of impact be it individual, societal, or biospheric. The value of our proposed framework for conceptualizing risks and harms is that by decomposing these into individual, social, and biospheric impacts, we provide a conceptual tool with which to challenge attempts to narrow down salient risks and harms in a way that is meaningful only to some discrete set of affected actors and thereby limited in effect. The prevailing international policy discourse focusing on the technical safety of “frontier AI” systems, which has led to a flight from confrontation of the full spectrum of hazards presented here, is a signal example of such an erroneous narrowing (Helfrich, 2024). In this paper, we underline the necessity of the opposite approach. When identified at their empirical sources, the risks and harms of foundational models become visible from all angles. They cut across individual, collective, and environmental levels, spreading over and affecting entire populations, including entire socio-material and biophysical ecologies of humans and non-humans. In this way, the impact and importance of the full range of risks and harms cannot be invisibilized or waved away in pursuit of economic, geopolitical, or other short-term goals.

5.1. Visibility gaps in the current assessment of algorithmic impact

As we have shown in this paper, the literature on risks and harms of foundation models is expansive and is likely to continue to grow. Grappling with such a vast and heterogenous landscape is a challenging task both for those who are attempting to build a nuanced understanding of the technology as well as for those who are seeking to inform the debate on mitigations and governance. We recognize notable attempts in the literature to taxonomize these risks (Weidinger et al., 2022), as well as significant progress being made internationally to reckon with the scale and embeddedness of foundation models by framing AI governance as a human rights and transnational issue (Intelligence, CAI; UNESCO, 2023). However, some of the most prominent governance initiatives, particularly in Europe and the US, has thus far fallen short of contending with the most problematic harms stemming from unequal patterns of data, labor, and resource extraction and instead has focused largely on risks to adoption, technical safety issues, and catastrophic risks.

One of the biggest challenges in assessing and anticipating algorithmic harm has to do with limited evidence and difficulties related to observing the indirect manifestations of harm, as well as foreseeing its effects over time. Grasping these complexities involves nuanced and context-dependent understanding. For instance, there are direct measurable impacts from the energy used to train or operate an LLM which are immediate and visible, but indirect harms arising from AI applications may only come to light over time or when enough evidence and research is made available. An AI system enabling the unsustainable extraction of mineral sites or unfair labor practices in the data labelling supply chain, are just a couple of illustrations of how negative indirect harms are likely to be largely hidden from view (OECD, 2022). Direct harms to individuals may translate to indirect harms to society and vice versa. For instance, the potential harm that can ensue with data misuse or privacy breach (e.g., personal data) is not just limited to the individual who is directly manipulated, but indirectly affects the interests of society at large (Smuha, 2021). Where anthropocentric and species biases exist in AI, there are further examples of direct harm which remain understudied, largely because wider biospherical needs are not part of the conversation (Coghlan and Parker, 2023). Algorithmic systems that ignore animals or privileges a particular view of animal welfare while ignoring others can exacerbate these consequential impacts, and not least can lead to indirect and direct harms to interconnected ecosystems shared by both humans and animals (Bossert and Hagendorff, 2021; Coghlan and Parker, 2023).

5.2. Building a socio-technical scaffold to technical interventions

The current strategies for the mitigation of foundation model generated risks and harms that have most traction among policymakers and governments predominantly focus on technical interventions (OECD, 2023b). There is thus an urgent necessity to delve into the social context within which these interventions are situated. This is crucial, especially since the advent of generative AI represents not only a significant milestone in technical advancement but also simultaneously transforms the very fabric of far-reaching socio-digital infrastructure like the Internet and its social experience (Heikkilä, 2022; Heaven, 2022).

Consider, for instance, the issue commonly framed in technical terms as “model collapse” or “data pollution”. In this scenario, the generative output from the widespread experimental use of chatbots and image generators feeds into public data pools. This influx poses the risk of influencing future datasets that AI models will subsequently incorporate. The amalgamation of human and AI-generated content potentially undermines the quality and diversity of AI-generated outputs (Martínez et al., 2023). While this cycle undoubtedly necessitates technical intervention, the risk typology outlined in this paper enables the identification of concurrent impacts within the social sphere—impacts like the undermining of the long-term integrity of the information ecosystems on which modern democratic ways of life rest.

The integrative perspective we offer emphasizes, for example, that the risks surrounding the AI-enabled generation of biased or harmful images go beyond just offending individuals; in the aggregate, these systems have the potential to change social narratives around communities and to lock in cultural prejudices at scale, replicating and augmenting patterns of structural discrimination and injustice. Likewise, given the scale of their generative abilities, these technologies have broader planetary implications that derive from the cumulative costs of mass industrialized compute. Concentration on the perceived technical complexity of foundation models often mistakenly occludes such a clear ecological view of the social impact of generative outputs. Regardless, evolving research rightly insists that the AI community must grasp the subtleties, social contexts, and boundaries of human interaction with AI as a user-oriented technology, as well as the social and longitudinal aspects of innovation, more broadly (Raji et al., 2022; Lehmann, 2023; Sanchez, 2023). The risk mapping presented in this paper facilitates this understanding. Even a singular, seemingly innocuous creation of politically incorrect content by an AI tool, a feature that many such tools still permit (Hagendorff and Danks, 2023), can accumulate and result in a proliferation of societally effective bias within the community. This bias is amplified by the repetitive contributions from multiple individuals. Consequently, the framework presented here helps us understand how concerns usually detected or faced at individual levels, in fact, scaffold larger risks and contribute to higher-level concerns, thereby providing a deeper socio-technical understanding of foundation models and AI at large.

5.3. Toward an integrative perspective on risks and harms

Our framework illustrates concretely how claims that there may be gains from the implementation and use of foundation models that outweigh their potential and observed harms need to be examined and nuanced through sharpened socio-technical lenses. Such claims often rely on a strictly utilitarian calculation in which the overall potential “benefits” of foundation model application may outweigh the total harms and risks and are hence largely incapable of accounting for more systemic considerations. However, as shown by our framework, foundation model harms are frequently difficult to track and measure—and this affects the capacity to coherently weigh benefits and harms. For instance, harmful use cases that are relatively intelligible when measured individually could have impacts and consequences that are much harder to trace at social and biospheric levels. A moderate, but cumulative and difficult-to-perceive, harm to planetary health has implications for billions of people whereas a significant, but straightforward, harm to some individuals may be quite limited in scope. This makes utility calculations of benefits and harms difficult to perform with sufficient precision to capture their full range, scale, and scope. For example, Bommasani et al. (Bommasani et al., 2022) argue that the benefits of releasing large models—such as applications that translate text in otherwise underserved languages—outweigh the risks of misuse and abuse by malicious actors. Within their analysis, there is also a recognition that relatively few firms and organizations have sufficient resources and capacities to produce foundation models and that efforts to develop them for use by less-resourced non-elites is nascent and unlikely to produce models with similar capacities. The analysis, however, fails to perceive the deeper risks engendered by the asymmetrical power structures and dynamics that lead to these inequitable differentials in resources, capacities, and access.

Where a handful of highly self-interested and profit-driven companies control the data, compute, and skills infrastructures on which the development and use of foundation models rely, social harms arising from expanding inequality, wealth polarization, concentration of economic power, and privilege biases that lead to the escalating marginalization of minoritized groups will likely abound. Risk analyses that fail to acknowledge macro-scale issues like this will discount socially consequential adverse impacts. In this case, the very fact that the architecture of foundation models relies on high resource concentration should be a launching pad for expanding the narrow lens on technical risk to account for the ways in which foundation models play a role in reproducing harmful social hierarchies and planetary degradation. Such recognition is also crucial for contending with the potential for foundation models to lead to hard-to-reverse and long-term effects as this technology progressively becomes more embedded in society. What we call an integrative perspective is essential for overcoming the dominant utilitarian and performance-oriented approach in the AI governance discourse which has tended to frame societal challenges in terms of quantifiable trade-offs between risks and benefits.

6. Conclusion

In this paper we have confronted a blind spot in the evolving AI governance landscape that derives from its reliance, particularly among prominent policymakers in the global north, on speculative risks and selective seeing. Drawing on the rapidly growing multidisciplinary body of research on foundation-model-generated risks and harms, we have shown how discerning this through integrative and socio-technically curved lenses better discloses the full spectrum of impacts across individual, social, and biospheric levels. We have argued, in this respect, that there exists a visibility gap between the range of concerns and evidence raised in the critical, empirically anchored literature, and the abstract and mainly hypothetical issues focused on within some of the most influential international AI governance initiatives. Such a gap reflects the convergence of the power dynamics, private interests, and geopolitical priorities that have agenda-setting consequences in the AI governance ecosystem—which is a challenge to coherently grappling with the unprecedented industrialization of large scale data-driven technologies, rapidly transforming veritably every domain of life and communities around the world. While potentially beneficial and transformative, foundation models also pose numerous risks to people, society, and the planet. We have aimed to deepen understandings of this broad range of risks by bridging the technical aspects of foundation models with their socio-technical underpinnings, connecting individual concerns to collective and planetary issues, and doing justice to the multifaceted and differential impacts these models have on affected communities. Our conceptualization of risk, particularly regarding the potential transformations effected by these technologies, demonstrates that the visibility of risks and harms should not be concealed or obscured by speculative concerns about existential threats, hypothetically conceived. Instead, as under the framework presented here, an understanding of risks should be grounded in robust evidence that is observable at various levels, demonstrating the potential of these adverse impacts to escalate, over time, and to widen in their scale and scope. Ultimately, the framework we have presented enables a comprehensive assessment of algorithmic impacts for which an interdisciplinary dialogue is key. As such it can be applied as an analytical tool to inform socio-technical mitigations and to fundamentally expand existing toolkits for algorithmic fairness, transparency, and responsible AI.

Impact Statement

1) Description of the ethical concerns the authors mitigated while conducting the work (as part of an ethical considerations statement):
Throughout the process of developing the research questions and methodology, we foregrounded our work in an acute awareness and recognition of the role power plays in shaping conversation around the design, development, and deployment of technology. We grounded the evaluation of risks and harms as they relate to foundation models within well-established scholarship and critical discussion surrounding how power is distributed, the pervasiveness of stark power asymmetries, and how the differential impacts of foundation models –and technology at large—are experienced across multiple actors and layers of the ecosystem. We allowed this grounding to inform the framing of our research questions and our methodology, ensuring that a multidisciplinary body of scholarship was drawn on to inform our investigation. This approach included drawing on critical data studies, science and technology studies, and environmental justice scholarship, amongst other fields. It is through this grounding that we aimed to challenge dominant discourses that adopt speculative perspectives on the risks and harms presented by the rapid deployment of foundation models; discourses which overwhelmingly distract from current, real-world consequences. We therefore aim to foreground our work in observed impacts. Through the proposed individual – social – biospheric framework, we work to acknowledge the inherently entangled and interdependent nature of socio-technical systems and their impacts, rather than further dominant discourses of fragmentation and division.

2) Reflections on how their background and experiences inform or shape the work (as part of a researcher positionality statement):
The final form and content of this work was shaped by a few factors. Firstly, all researchers involved in the development and writing of this work are academically trained in research, and are based at research and higher-education institutions in the Global North, where the primary language is English. As such, the research itself was limited to the English language. The researchers involved in this work, however, represent a variety of communities and come from a diverse range of backgrounds, including lived and research experience in the Global South. Throughout the process of developing the work, we continuously reflected on– and engaged with– how our own values, beliefs, perspectives, and lived experiences inevitably shaped the work presented here. The research team which developed this work is interdisciplinary, coming from a variety of research specializations including anthropology, sociology, data science, and information science, as well as diverse and global industry experience.

3) Reflection on the adverse, unintended impact the work might have once published (as part of an adverse impact statement):
In presenting this work, we are acutely aware that the research is squarely grounded in a rapidly evolving field. As such, we acknowledge that the proposed arguments and recommended pathways this research presents might not be relevant in the near to distant future, and we do not prescribe any solutions that could have an impact in the future. However, we also acknowledge that –despite these intentions—our risks and harms codes, and the subsequent categorizations we develop, may have unintended impacts; either through mis-categorizing certain risks and harms, or not accounting for others. These codes and categories were developed by our team’s collective conceptualization of the terms ‘risk’ and ‘harm’; as we are each working within our individual and group positionalities, there may be other forms of risks and harms not fully captured in this research.

Appendix A Appendix

A.1. Search strategy and objectives

The primary goal of our search strategy was to identify relevant literature on risks and harms of foundation models and ensure a comprehensive understanding of their socio-political and environmental dimensions. To achieve this objective, we conducted searches across five academic electronic databases. The selected databases include arXiv, ACM Digital Library, IEEE, Scopus, and Web of Science. These databases were chosen based on their relevance in the field, their inclusion of a broader range of sources such as conference proceedings and preprints, and access to the full text of the articles.

To develop the search strategy, we chose a handful of keywords based on their relevance to the research question and maximize the scope of the search: “foundation model”, “large language model”, “llm”, “general purpose artificial intelligence”, “GPAI”, “risk”, “harm”, “ethic”. In addition, we used Boolean operators to refine search queries (see Table 2 for example). In some cases, we refined when the result set had a high percentage of irrelevant returns: AND TI “artificial intelligence” OR TI “AI” OR AB “artificial intelligence” OR AB “AI”.

Searches were conducted on title and abstract fields. A sample set was piloted, before a full search was conducted across the selected databases. Inclusion criteria covered English language journal articles, including conference papers and pre-prints, but excluding grey literature, monographs, commentaries, correspondences, and opinion pieces.

The results were added to a Zotero library, which also included our snowball sample, and the merge of these two libraries was exported into a spreadsheet for refinement and coding. We then removed duplicates and non-relevant papers, reaching 167 papers in our sample. The use of an abductive approach supported establishing clearer links with the research objectives and developing a codified mapping of key relationships found in the literature results. To code the papers, firstly, we extracted applicable keyword information from the abstract of each primary study as part of an initial coding process. The resulting keywords served largely as summative and process attributes of a risk or type of harm relevant to foundation models or LLMs. Three researchers subsequently clustered these attributes (Level 2 – attribute codes) and assigned them into Level 1 or parent categories. For example, to attribute code copyright and intellectual property violations we assigned the parent category legal and regulatory violations. For the attribute codes difficult to ensure explainability and reliance on proprietary software we assigned the parent category lock-in and opacity risks (See Table 3 for full list of parent and attribute codes). The codes were subsequently refined by all co-authors.

Table 2. Example of broad search (with refining keywords):

Broad search (*)	Refining #2
TI “foundation model*” OR AB “foundation model*”	AND TI “Risk*” OR TI “Harm*” OR TI “ethic*”OR AB “Risk*” OR AB “Harm*” OR AB “ethic*”
TI “Large Language model*” OR TI “LLM*” OR AB “Large Language model*” OR AB ”LLM*”	AND TI “Risk*” OR TI “Harm*” OR TI “ethic*” OR AB “Risk*” OR AB “Harm*” OR AB “ethic*”
TI ”general purpose artificial intelligence” OR TI ”general- purpose artificial intelligence” OR TI “GPAI” OR AB “general purpose artificial intelligence” OR AB ”general-purpose artificial intelligence” OR AB “GPAI”	AND TI “Risk*” OR TI “Harm*” OR TI “ethic*” OR AB “Risk*” OR AB “Harm*” OR AB “ethic*”

Table 3. Parent and attribute codes.

Parent Code	Attribute Code
Bias and societal prejudices	Amplifying and perpetuating stereotypes and societal biases Behavioral biases Political biases
Misinformation, disinformation and propaganda	Abusive interactive experiences Manipulation Misinformation spread
Unreliable performance	Accuracy/inaccuracy (outputs) Harmful or toxic outputs Language performance gap Poor performance due to excessive training with synthetic data Unpredictability of behaviour pre- and post-deployment Untruthfulness of outputs Disparate performance
Cybersecurity risks and harms	Cyber-attacks payload Data leakage Data poisoning attacks Fraudulent services Goal hijacking and prompt leaking Prompt injection attacks Spear phishing Jailbreaking Malicious actors Impersonation attack
Privacy risks and harms	Privacy risks and harms
Systemic social and economic risks and harms	Erosion of semantic capital Misleading narratives about AI Narrowing of the market/Monopolisation Outcome homogenization Perpetuation of inequalities Effects on labor market Widening of digital divide Concentration of authoritative power Invisibilization and poor working condition of data and content moderation labor
Legal and regulatory violations	Copyright and intellectual property violations Data protection violations Consumer protection laws violations
Environmental effects and ecological disruption	Environmental effects and ecological disruption
Misuse	Misuse Biological misuse Dual use Illegitimate surveillance Creation of violent or harmful content
Lock-in and opacity risks	Challenges to benchmarking Difficult to ensure explainability Lack of replicability and transparency Low technological readiness Reliance on proprietary software
Overdependency in human-computer interaction	Overdependency in human-computer interaction
Data risks and harms	Data extractivism Data quality Datasets containing toxic data Degradation of the digital commons
Value misalignment	Value misalignment
Extreme or catastrophic risks and harms	Extreme risks Catastrophic risks

A.2. Results of the coding and codes description

Table 4. Papers per risks and harms parent and attribute codes and codes description.

Risks and Harms Codes (and Papers)	Code Description	Count
Bias and societal prejudices Amplifying and perpetuating stereotypes and societal biases: (Abid et al., 2021; Ahuja et al., 2023; Alemany et al., 2023; Birhane et al., 2021; Awais et al., 2023; Bender et al., 2021; Bommasani et al., 2022; Anderljung et al., 2023; Au Yeung et al., 2023; Borchers et al., 2022; Brewer et al., 2023; Chan et al., 2023; Connor and O’Neill, 2023; da Silva et al., 2021; Huang et al., 2023a; Dacon et al., 2022; de Jager, 2023; Deas et al., 2023; Fan et al., 2023; Farina and Lavazza, 2023; Felkner et al., 2023; Gadiraju et al., 2023; Glocker et al., 2022; Guo and Caliskan, 2021; Hossain et al., 2023; Hosseini and Horbach, 2023; Huang et al., 2023b; Hundt et al., 2022; Khowaja et al., 2023; Kidd and Birhane, 2023; Kirk et al., 2023; Liao and Vaughan, 2023; Mattas, 2023; Mhlanga, 2023; Nadeem et al., 2020; Nozza et al., 2022; O’Neill and Connor, 2023; Ovalle et al., 2023; Piñeiro-Martin et al., 2023; Qadir, 2022; Rauh et al., 2022; Sebastian, 2023; Shaikh et al., 2022; Shelby et al., 2023; Simmons, 2023; Solaiman et al., 2019, 2023; Steed et al., 2022; Sun et al., 2023; Talat et al., 2022; Talboy and Fuller, 2023; Tan and Celis, 2019; Thakur, 2023; Thieme et al., 2023; van der Wal et al., 2023; Venkit et al., 2023; Wang et al., 2024; Weidinger et al., 2022, 2021; Wójcik, 2022; Zhuo et al., 2023b; Zhao et al., 2023; Wu et al., 2023; Jakesch et al., 2023a) Behavioral biases: (Chen et al., 2023) Political biases: (Liu et al., 2021; Motoki et al., 2023; Rozado, 2023; Urman and Makhortykh, 2023)	The perpetuation and amplification of hegemonic views, harmful stereotypes, societal and behavioral biases.	72
Misinformation, disinformation and propaganda Abusive interactive experiences: (Bossert and Hagendorff, 2021; Rosenberg, 2023; Shelby et al., 2023) Disinformation: (Anderljung et al., 2023; Barnard et al., 2023; Bian et al., 2023; Christ et al., 2023; Fecher et al., 2023; Kidd and Birhane, 2023; Liao et al., 2023; Májovský et al., 2023; Mattas, 2023; Neupane et al., 2023; Pan et al., 2023; Shelby et al., 2023; Solaiman et al., 2019, 2023; Weidinger et al., 2021, 2022; Bommasani et al., 2022; Li et al., 2023b) Manipulation: (Bender et al., 2021; Chan et al., 2023; Kirk et al., 2023; Shelby et al., 2023; Weidinger et al., 2021, 2022; Shevlane et al., 2023) Hallucinated information: (Connor and O’Neill, 2023; Liao and Vaughan, 2023; Májovský et al., 2023; Deroy et al., 2023; Kang et al., 2023b; Tian et al., 2023) Misinformation spread: (Anderljung et al., 2023; Bian et al., 2023; Chan, 2023; Christ et al., 2023; Farina and Lavazza, 2023; Fecher et al., 2023; Gao et al., 2023; Goldstein et al., 2023; Huang et al., 2023a; Khowaja et al., 2023; Kidd and Birhane, 2023; Kirk et al., 2023; Liao et al., 2023; Liao and Vaughan, 2023; Mattas, 2023; Neupane et al., 2023; Piñeiro-Martin et al., 2023; Qadir, 2022; Sebastian, 2023; Shelby et al., 2023; Solaiman et al., 2023; Sun et al., 2023; Rillig et al., 2023; Strasser, 2023; Thapa and Adhikari, 2023; Weidinger et al., 2021, 2022; Zhou et al., 2023a, b; Zhuo et al., 2023b; Li et al., 2023b; Jakesch et al., 2023b; Tang et al., 2023) .	Unintentional or intentional efforts to disseminate false or misleading information. Related risks and harms include disinformation, misinformation spread, extreme manipulation, and abusive interactive experiences.	45
Cybersecurity risks and harms Cyber-attacks payload: (Charan et al., 2023; Weidinger et al., 2021, 2022) Data leakage: (Bender et al., 2021; Derner and Batistič, 2023; Fan et al., 2023; Huang et al., 2023a; Jagannatha et al., 2021; Liao and Vaughan, 2023; O’Neill and Connor, 2023; Piñeiro-Martin et al., 2023; Plant et al., 2022; Sebastian, 2023; Shao et al., 2023; Wang et al., 2024; Weidinger et al., 2021, 2022; Zhuo et al., 2023b; Zhou et al., 2023a) Data poisoning attacks: (Fan et al., 2023; Huang et al., 2023a; Li et al., 2023a; Wu et al., 2023) Fraudulent services: (Derner and Batistič, 2023; Farina and Lavazza, 2023; Kirk et al., 2023; Porsdam Mann et al., 2023; Sebastian, 2023; Strasser, 2023; Weidinger et al., 2021, 2022) Goal hijacking and prompt leaking: (Perez and Ribeiro, 2022) Prompt injection attacks: (Greshake et al., 2023; Liu et al., 2023; Wu et al., 2023; Zhuo et al., 2023b) Spear phishing: (Anderljung et al., 2023; Derner and Batistič, 2023; Eggmann et al., 2023; Farina and Lavazza, 2023; Hazell, 2023; Karanjai, 2022; Neupane et al., 2023; Sebastian, 2023; Solaiman et al., 2023) Jailbreaking: (Rao et al., 2023; Wang et al., 2024) Malicious actors: (Awais et al., 2023; Bommasani et al., 2022; Eggmann et al., 2023; Farina and Lavazza, 2023; Kang et al., 2023a; Weidinger et al., 2021, 2022; Wu et al., 2023; Jakesch et al., 2023b) Impersonation attack: (Piñeiro-Martin et al., 2023; Solaiman et al., 2023; Zhou et al., 2023a)	Unauthorized access and attacks to the foundation model that exploit vulnerabilities or compromise its integrity. Examples of cybersecurity risks and harms include cyber-attacks payload, data leakage, data poisoning attacks, data pollution, fraudulent services, goal hijacking and prompt leaking, offensive cyber capabilities, prompt injection attacks, spear phishing, jailbreaking, and other attacks by malicious actors.	37

Risks and Harms Codes (and Papers)	Code Description	Count
Unreliable performance Accuracy/inaccuracy (outputs): (Au Yeung et al., 2023; Awais et al., 2023; Farina and Lavazza, 2023; Fecher et al., 2023; Huang et al., 2023a; Kang et al., 2023b; Liao and Vaughan, 2023; O’Neill and Connor, 2023; Piñeiro-Martin et al., 2023; Shelby et al., 2023; Strasser, 2023; Zhuo et al., 2023b; Alkaissi and McFarlane, 2023; Borji, 2023; Rudolph et al., 2023; Uz and Umay, 2023; Ye et al., 2023; Casper et al., 2023) Harmful or toxic outputs: (Anderljung et al., 2023; Bender et al., 2021; da Silva et al., 2021; Deshpande et al., 2023; Huang et al., 2023b; Liao and Vaughan, 2023; Borji, 2023; Ma et al., 2023; Rauh et al., 2022; Shaikh et al., 2022; Shelby et al., 2023; Solaiman et al., 2023; Strasser, 2023; Sun et al., 2023; Zhuo et al., 2023a, b; Blodgett and Madaio, 2021; Dinan et al., 2021; Gehman et al., 2020; Li et al., 2023c; Qi et al., 2023; Triguero et al., 2023; Vashishtha et al., 2023; Weidinger et al., 2021, 2022; Welbl et al., 2021; Casper et al., 2023) Language performance gap: (Alshahrani et al., 2022; Weidinger et al., 2021, 2022; Verma et al., 2022; Zhuo et al., 2023b) Poor performance due to excessive training with synthetic data: (Veselovsky et al., 2023) Unpredictability of behaviour pre- and post- deployment: (Anderljung et al., 2023) Untruthfulness of outputs: (Borji, 2023; Oviedo-Trespalacios et al., 2023; Rauh et al., 2022) Disparate performance: (Bommasani et al., 2022; Shelby et al., 2023; Solaiman et al., 2023)	Inconsistent or undesirable performance exhibited by foundation models. This includes inaccurate or non-factual outputs, harmful or toxic outputs, language performance gap, performance disparities at group levels, and unpredictaibility of behavior pre- and post-deployment.	44
Privacy risks and harms (Bender et al., 2021; Eggmann et al., 2023; Fecher et al., 2023; Huang et al., 2023a; Jagannatha et al., 2021; Kandpal et al., 2022; Khowaja et al., 2023; Kirk et al., 2023; Liao and Vaughan, 2023; Li et al., 2023a; O’Neill and Connor, 2023; Porsdam Mann et al., 2023; Piñeiro-Martin et al., 2023; Shelby et al., 2023; Solaiman et al., 2023; Strasser, 2023; Tian et al., 2023; Wang et al., 2024; Weidinger et al., 2021, 2022; Wu et al., 2023; Zhou et al., 2023a; Patsakis and Lykousas, 2023; Peris et al., 2023; Yan et al., 2023; Zhang et al., 2023; Lu et al., 2023; Mireshghallah et al., 2022)	Adverse consequences associated with the collection, storage, processing, and use of data beyond the data pipeline that arise from the increasing capabilities of foundation models and their potential to inadvertently expose private or sensitive data.	29
Systemic social and economic risks and harms Erosion of semantic capital: (Nannini, 2023) Misleading narratives about AI: (Ferri and Gloerich, 2023; Nannini, 2023; Shanahan, 2023) Narrowing of the market/Monopolisation: (Bommasani et al., 2022; Luitse and Denkena, 2021) Outcome homogenization: (Bommasani et al., 2022; Kirk et al., 2023; Porsdam Mann et al., 2023; Thieme et al., 2023) Perpetuation of inequalities: (Bommasani et al., 2022; Fecher et al., 2023; Kirk et al., 2023; Porsdam Mann et al., 2023; Rillig et al., 2023; Shelby et al., 2023; Solaiman et al., 2023; Weidinger et al., 2021, 2022; Williams, 2022) Effects on labor market: (Farina and Lavazza, 2023; Khowaja et al., 2023; Kirk et al., 2023; Liao and Vaughan, 2023; Mattas, 2023; Eloundou et al., 2023; Lund et al., 2023; Qadir, 2022; Solaiman et al., 2023; Weidinger et al., 2021, 2022; Zarifhonarvar, 2023) Widening of digital divide: (Khowaja et al., 2023; Shelby et al., 2023) Concentration of authoritative power: (Khowaja et al., 2023; Solaiman et al., 2023) Invisibilization and poor working condition of data and content moderation labor: (Solaiman et al., 2023)	Risks and harms that arise from the widespread adoption and impact of foundation models on societal and economic structures. These range from the erosion of semantic capital, outcome homogenization, and misleading narratives about AI, to the widening of digital divide, the narrowing of the market or monopolization, effects on labor market, and the perpetuation of inequalities.	24
Legal and regulatory violations Copyright and intellectual property violations: (Al-Kaswan and Izadi, 2023; Bommasani et al., 2022; Fecher et al., 2023; Farina and Lavazza, 2023; Khowaja et al., 2023; Lund et al., 2023; O’Neill and Connor, 2023; Piñeiro-Martin et al., 2023; Porsdam Mann et al., 2023; Sebastian, 2023; Strasser, 2023; Solaiman et al., 2023; Lee et al., 2023; Henderson et al., 2023a; Huang and Siddarth, 2023) Data protection violations: (Al-Kaswan and Izadi, 2023; Bommasani et al., 2022; Kandpal et al., 2022; Khowaja et al., 2023; Kirk et al., 2023; O’Neill and Connor, 2023; Patsakis and Lykousas, 2023; Peris et al., 2023; Piñeiro-Martin et al., 2023; Porsdam Mann et al., 2023; Sebastian, 2023; Solaiman et al., 2023; Strasser, 2023; Tian et al., 2023; Zhang et al., 2023) Consumer protection laws violations: (Sebastian, 2023)	Breaches of laws and regulations, including those designed to safeguard individuals’ privacy and secure handling of their personal data, intellectual property, consumer protection, and cybersecurity.	21

Risks and Harms Codes (and Papers)	Code Description	Count
Environmental effects and ecological disruption (Anthony et al., 2020; Bender et al., 2021; Bommasani et al., 2022; Hagendorff et al., 2023; Huang et al., 2023a; Khowaja et al., 2023; Kirk et al., 2023; Kirkpatrick, 2023; Liao and Vaughan, 2023; Luccioni et al., 2022; O’Neill and Connor, 2023; Rillig et al., 2023; Shelby et al., 2023; Solaiman et al., 2023; Triguero et al., 2023; Weidinger et al., 2022; Patterson et al., 2021; Li et al., 2023d)	Adverse and negative impacts of foundation models on ecology and animals.	19
Misuse Misuse: (Anderljung et al., 2023) Biological misuse: (Cohen, 2023; Sandbrink, 2023) Dual use: (Anderljung et al., 2023; Kang et al., 2023a; Weidinger et al., 2021, 2022; Glukhov et al., 2023; Soice et al., 2023; Henderson et al., 2023b) Illegitime surveillance: (Anderljung et al., 2023; Bommasani et al., 2022; Kirk et al., 2023; Weidinger et al., 2021, 2022) Creation of violent or harmful content: (Anderljung et al., 2023; Bender et al., 2021; Bommasani et al., 2022; Kirk et al., 2023; Shelby et al., 2023; Solaiman et al., 2023)	The misuse of foundation models is used here to describe specific contexts in which foundation models or technologies based on foundation models may be inappropriately used and lead to negative consequences. This includes uses that can have both civilian and military applications, or contribute to the handling, manipulation, or application of biological materials, organisms, or technologies.	19
Lock-in and opacity risks Challenges to benchmarking: (Awais et al., 2023) Difficult to ensure explainability: (Piñeiro-Martin et al., 2023; Thieme et al., 2023; Triguero et al., 2023; Wasserman Rozen et al., 2023) Lack of replicability and transparency: (Farina and Lavazza, 2023; Hosseini and Horbach, 2023; Kang et al., 2023b; Khowaja et al., 2023; Liao and Vaughan, 2023; O’Neill and Connor, 2023; Piñeiro-Martin et al., 2023; Yan et al., 2023; Zhou et al., 2023a; Lu et al., 2023) Low technological readiness: (Yan et al., 2023) Reliance on proprietary software: (Liao and Vaughan, 2023; Liesenfeld et al., 2023)	Risks and harms associated with foundation models that are difficult to understand, replicate, or modify. These risks and harms arise from issues including reliance on proprietary software, challenges to benchmarking, lack of transparency, difficulty to ensure explainaibility, and low technological readiness.	15
Overdependency in human-computer interaction (Connor and O’Neill, 2023; Kirk et al., 2023; Liao and Vaughan, 2023; Ma et al., 2023; Mhlanga, 2023; Piñeiro-Martin et al., 2023; Solaiman et al., 2023; Thieme et al., 2023; Weidinger et al., 2021, 2022; Peris et al., 2023)	Overreliance of humans interacting with computer systems, interfaces, and technologies built on foundation models for various aspects of their lives and potentially negatively impacting their well-being, safety, decision-making abilities, or interpersonal relationships.	11
Data risks and harms Data extractivism: (Huang and Siddarth, 2023) Data quality: (Huang and Siddarth, 2023; Kandpal et al., 2022) Datasets containing toxic data: (Birhane et al., 2023; Fan et al., 2023; O’Neill and Connor, 2023; Piñeiro-Martin et al., 2023; Wang et al., 2024) Degradation of the digital commons: (Chan, 2023; Huang and Siddarth, 2023)	A range of risks and harms associated with the collection, storage, processing, and use of data within the data pipeline. These risks extend beyond the cybersecurity risks and harms described above and includes broader issues of ethical, social, and economic implications of data stewardship and management. They include issues of poor data quality, datasets containing toxic data, data extractivism, and the degradation of the digital commons.	8
Value misalignment (Huang and Siddarth, 2023; Kasirzadeh and Gabriel, 2023)	Misalignment of the output content of conversational agents with the norms and values of the human interacting with such agent.	2
Extreme or catastrophic risks and harms Extreme risks: (Shevlane et al., 2023) Catastrophic risks: (Barrett et al., 2023)	Speculative far-reaching or irreversible adverse impacts of foundation models at societal scale that extend beyond immediate impacts.	2