Data Exposure from LLM Apps: An In-depth Investigation of OpenAI’s GPTs

In this paper we study the OpenAI’s GPT (app) ecosystem, the most mature third-party LLM app ecosystem with more than 3 million GPTs [24]. OpenAI provides GPTs the ability to customize the behavior of the LLM, browse the web, generate images, interpret code, search files, and connect to the APIs of external online services. Browsing (i.e., Web Browser), image generation (DALLE), code interpretation (Code Interpreter), and file searching (Knowledge) are built-in tools and provided by OpenAI [3], whereas connection to external APIs are implemented as custom tools, which are referred to as Actions [21]. Actions are akin to third-party services on the web, such as analytics, JS wrappers, CDNs, that websites embed to enhance their offerings.

Built-in tools can be enabled by clicking check-boxes on the GPT creation interface [30], whereas Actions need to be implemented as HTTP APIs and exposed to OpenAI in a JSON format [21]. The JSON format of Actions describes the functionality offered by each API, including its data types, as natural language descriptions (Appendix A lists the source code of a GPT with an Action). GPTs also define their functionality in natural language and interface with the LLM, their tools, the user, and other GPTs through natural language instructions. To build the necessary context to use a GPT, LLMs inject the natural language-based source code of GPTs in their context window, when users install and interact with GPTs. Figure 1 presents the architecture of GPTs with its core components.

While third-party apps extend the capabilities of computing platforms, they also pose several risks to user privacy. For example, in almost all online computing platforms, such as the web, mobile, and IoT, it is a standard practice for third-party apps to collect excessive user data, often with other specialized third-party services, for the purposes of profiling users for personalized online advertising  [5, 6, 7, 8]. We worry that the GPTs might also engage in similar practices on the OpenAI’s platform. In fact, GPTs are already including specialized third-party Actions to track users (as we show later in Section 5.2.2).

OpenAI currently imposes some restrictions [13, 14, 15] on GPTs but they are mostly limited, optional, or not strictly enforced [16, 17, 18]. For example, OpenAI currently does not implement any foolproof access control mechanisms, and leaves it up to the developers to define permission interfaces for activities performed by the GPTs, which may not be reviewed [15]. There are already instances where policy violating apps were hosted on OpenAI and only removed when publicly brought to attention [19]. Furthermore, OpenAI also intends to use user’s interaction with the GPTs, i.e., to train its models [32]. Although, OpenAI provides users’ controls to delete their data [33], these controls may not extend to third-party GPTs, as OpenAI may not have visibility or control over the data exfiltrated by the Actions inside GPTs.

Privacy risks may be further exacerbated in LLM platforms because of the natural language-based execution paradigm of LLMs. For example, user’s main mode of interaction with LLMs is information rich natural language, which can be processed to infer several characteristics about the user, such as their age or interests [9, 10]. Furthermore, malicious GPTs can launch straightforward attacks (e.g., with prompt injection [11]) to access information beyond their one-to-one interactions with the user, as LLMs automatically load prior user interactions in their context window to provide a contextually relevant response [12].

Given the potential for privacy issues and their harms to the users, this paper aims to bring transparency in the OpenAI’s third-party app ecosystem. More specifically, our goal is to characterize the privacy practices in the OpenAI’s GPT ecosystem, including (i) surveying GPTs and Actions embedded in them, (ii) characterizing their data collection practices, (iii) measuring potential indirect data exposure across GPTs and their Actions, and (iv) checking the consistency of data collection practices with disclosures in privacy policies of GPTs and Actions.

We conduct a four-month long periodic weekly crawls of GPTs from February 8th to May 3rd 2024, to measure their evolution across several axes (Section 4). To characterize data collection by GPTs and their actions, we rely on static code analysis, as GPTs and Actions need to state their data collection in natural language, so that it can be interpreted and acted upon by LLMs (Section 3). Furthermore, we analyze the indirect exposure of data across Actions because of embedding of multiple Actions in GPTs by modeling Action co-occurrence as a graph (Section 5.3). Lastly, to measure the consistency between the data collection by GPT Actions and disclosures in their privacy policies, we develop an LLM-based privacy policy analysis framework (Section 6). Figure 2 provides an overview of our approach.

With these measurements, our goal is to build an informed understanding of the third-party app ecosystems in LLM platforms. We envision such measurements to serve as a guide to inform the design of current and future integrations of third-party services in LLM platforms, to improve their privacy.

Since OpenAI does not provide any interfaces to download GPTs hosted on their platform, we rely on several third-party GPT stores that index a large number of GPTs. We identified a total of 13 popular sources that list GPTs (listed in Table I) from popular developer communities and forums, such as the OpenAI Developer Forum [34, 35].

We implemented selenium-based [36] crawlers for each of the third-party store to extract links to the GPTs. After extracting the links, we process them to extract the GPT identifiers, and then send a request to an OpenAI API endpoint with the GPT identifier¹¹1https://chat.openai.com/backend-api/gizmos/g-{identifier} that returns the JSON specification of a GPT. If the GPT identifier is not associated with a publicly available GPT, OpenAI returns a 404 error code. We also crawl a small number of featured GPTs listed on the OpenAI’s official GPT store. The downloaded JSON specifications of GPTs describe their functionality in natural language, including the endpoints contacted by Actions, and the data exfiltrated by them (Appendix A lists the source code of a GPT with a third-party Action).

After crawling GPTs, we download the privacy policies of their Actions by requesting the URL in the legal_info_url field in their specifications.²²2Note that only the Actions embedded in GPTs are required to provide privacy policies [21]. We successfully crawl 98.9 $\pm$ 1.7% GPTs and 91.5 $\pm$ 2.3% privacy policies of GPT Actions, over four months. We are unable to crawl the remaining GPTs and privacy policies due to internal server errors and server unresponsiveness. Table I shows the cumulative number of GPTs from each of the GPT stores. In total, we crawl 119,543 unique GPTs from all of the GPT stores.

We note that several GPTs are modified over time, either because they are changed by their developers or because some of their metadata is changed by OpenAI, such as ratings and usage statistics. Table II presents the breakdown of changes in properties of crawled GPTs. In total, we identify 303 GPTs that are modified over time (we do not consider the properties that are changed by OpenAI). We note that some modification (e.g., metadata and Actions/Files) could be more consequential than the others (e.g., contact information) in altering a GPT’s functionality. We investigated all such instances, i.e., modifications to metadata and Actions/Files related properties. However, none of these modifications indicated a functionality change and most seem to be related to performance/accuracy tweaks. For example, in all instances where GPTs changed their descriptions, they were to make them more precise.

It is important to note that the GPT’s exact instructions are not revealed in their crawled source code, so we cannot investigate how they change over time. Moreover, we could also only observe that the name of the files associated with the GPTs have changed, but not their content.

Next we analyze the removed GPTs to assess if the reason for their removal were problematic behaviors. We consider a GPT to be removed if it is no longer present on the third-party GPT stores and also inaccessible on ChatGPT. In total, we note that 2,883 GPTs were removed from the GPT store during our crawl period.

Since our goal is to reliably assess the potential reasons for the removal of GPTs, we resort to manual investigation. We specifically emphasize on GPTs that embed Actions because they present the potential for most harms – as they connect to potentially untrustworthy third-party services on the internet and load unvetted content. Our manual review process involves two human coders first independently analyzing a small set of GPTs to generate a code book, and then independently analyzing GPTs using that code book. At a high level, the code book contains rules that characterize GPTs functionalities, including their data collection practices and their content generation practices. This characterization requires us to analyze the natural language functionality description of the GPTs and their API endpoints, individually using them in ChatGPT, and also interacting with their API endpoints.

Table III presents the potential reason for the removal of 175 GPTs that embed Actions. We find that the largest proportion of removed GPTs are the ones whose Action APIs are no longer accessible. In some cases, we noticed that upon calling the Action’s APIs, they returned messages that the GPTs have been discontinued. For example, the AskYourCode Action within the AskYourCode GPT returned the message that: “AskYourCode was closed on 15th Feb due to low usage.” [37]

The second largest category of removed GPTs are the ones that provide web browsing functionality. Upon investigating, we discovered that OpenAI from time-to-time, although inconsistently, has been removing GPTs that allow users to browse the web [38, 39]. More recently, OpenAI has been reaching out to the GPT developers which provide web browsing functionality, that their GPT provides “copyright infringing content” to its users [40].

The third largest category of removed GPTs were the ones that contained Actions which provide analytics and advertising services. OpenAI currently does not condone GPTs to collect analytics of their own and promises an in-house analytics feature in future releases [32]. As for the advertising, it was initially prohibited by OpenAI [41, 42] but does not seem to be prohibited anymore, as per the updated OpenAI’s policies [14].

We also noticed that a number of GPTs were removed because they contained Actions that use YouTube’s APIs. Since OpenAI by default uses user’s interaction with ChatGPT, including with custom GPTs, for training its models, YouTube API embedding GPTs could be removed because they are in a potential violation of YouTube’s data usage policies [43].

Several other removed GPTs provided sexually explicit content (e.g., SutraKama [44]), enabled gambling (e.g., CrytoCipher [45]), or enabled stock trading (e.g., MetaTrader GPT [46]), all of which are practices that are prohibited by OpenAI [14]. We also noticed a couple of instances where the GPTs likely tried to impersonate other services. For example, we identified a GPT appearing to be representing booking.com but serving content from amadeus.com. We have reached out to booking.com to notify them about the existence of this GPT and also to validate whether they are hosting this GPT, but we have not yet heard back from them.

Table IV provides the breakdown of tool usage in GPTs. We note that almost all (97.5%) GPTs include tools; with most popular integration being the Web browser with 92.3%, followed by DALL-E with 85.5%, Code interpreter with 53.0%, Knowledge (Files) with 28.2%, and Actions with 4.6%.³³3The high prevalence of Web browser and DALL-E could be because they are pre-checked by default in the OpenAI’s GPT configuration interface [47].

A significant majority (93.2%) of GPTs connect to online services through Web Browser and Actions. Specifically, the Web Browser tool allows to consume content from any webpage on the internet and Actions allow to connect to specific online services. While these tools extend the capabilities of GPTs, they also expose users to unvetted online content on the Internet, threatening user security and privacy [48, 17]. In the case of Actions, these risk may be further exacerbated as a significant number of Actions in GPTs are not developed in-house but are simply integrated from other third-party developers.⁴⁴4We classify an Action as a third-party if its eTLD+1 does not match the eTLD+1 of the hosting GPT — a standard process to detect third-parties on the web [49].

We also noticed that in some cases GPTs integrate more than one Action. Specifically, among the GPTs that integrate actions, 90.9% contain one Action, 6.6% contain two Actions, 1.2% contain three Actions, and the remaining 1.3% contain as many as 4 to 10 Actions. Of the GPTs with multiple Actions, majority (55.3%) of them connect to additional domains (i.e., different online services), while the remaining 44.7% described other paths/endpoints for an API within the same domain (i.e., the same online service). The presence of multiple Actions can allow them to read each other’s data and also influence each other’s functionality, as currently ChatGPT does not isolate the execution of Actions inside a GPT [17, 25].

This practice of integrating Actions, especially from the third-parties is reminiscent of the early days of the web and mobile platforms when only a few websites/apps included a few third-party services [50]. As LLM ecosystems mature, GPTs may include tens of Actions, including from third-parties, as it is a common practice in the modern web, mobile, and IoT ecosystems [5, 6, 7, 8].

We further investigate the practices of GPTs and their Actions in Section 3 (Data collection) and Section 6 (Privacy policy compliance).

Next, we analyze Actions that collect user data, including analyzing their practices and offerings.

Since Actions execute in shared memory space in GPTs, they have unrestrained access to each others data, which allows them to access it (and also potentially influence each others execution) [25, 17]. Thus, in this subsection, we analyze the indirect exposure of user data due to integration of multiple Actions in GPTs, given the lack of isolation in ChatGPT.

OpenAI mandates, individual third-party Actions embedded in GPTs, to provide privacy policies but does not require GPTs to provide a privacy policy that describes its data practices as a whole [21]. This approach deviates from the norm in other platforms, where the apps provide a privacy policy with information about their own practices, including information about third-party services that they embed. In OpenAI’s ecosystem, to understand data practices of GPTs, users need to read the privacy policies of all of their third-party Actions. Since the GPT interface does not disclose the Actions embedded in them, and given that Actions can dynamically embed other third-party Actions (Section 5.2.1), users may simply be unaware of the existence of these Actions in GPTs, let alone their data practices.

For the purposes of analysis in this section, we analyze the privacy policy disclosures at the granularity of individual Actions. Table IX presents high-level statistics about privacy policies. Overall, we were able to crawl privacy policies of 86.68% of Actions (among 2,596 distinct Actions). For the remaining 13.32% of the Actions, the privacy policies were inaccessible. We also note that nearly 39.56% of the polices appear more than once for distinct Actions and 5.50% of the policies are near duplicates of each other (i.e., have a Jaccard similarity [72] of more than 95%).

We investigate these duplicates and near-duplicates, and provide our assessment in Table X. We note that, the inclusion of privacy policy of the external third-party services (e.g., Github, Google) is the most common reason for duplicate policies (33.5%), followed by empty privacy policies (27.0%) and Actions belonging to the same vendor (19.2%). For near-duplicates, we find that all such Actions include a boilerplate privacy policy generated from freeprivacypolicy.com, with mostly the only change being the name of the Action.

We also noted that for 12.45% of the Actions the privacy policies were less than 500 characters. We manually analyze these policies and find that they contain generic statements, such as “We do not collect any personal data from users of our Service.” and “Your data is never for sale.”. Nonetheless they still describe the data practices of the Actions, albeit being short, thus we still consider them in our analysis.

Our goal with the privacy policy analysis is to assess whether they contain disclosures about the data collection practices of Actions. To that end, we build on the automatic privacy policy analysis by prior work [26, 27, 8, 28], and leverage the recent advances in natural language processing [73] to develop an LLM-based framework to check the consistency of data collection disclosures.

Considering that LLMs are not always reliable and that their performance degrades with large context [29], we do not simply pass the large and complicated privacy policies to an LLM and probe it to measure the disclosures by GPTs. Instead, our framework takes a three step approach to analyze privacy policies. First, we tokenize the sentences in privacy policies [74] and pass individual sentences to an LLM to assess whether they pertain to data collection. Second, we pass (indexed) data collection statements to the LLM, so that it can build its context. Third, we pass the data items one-by-one to the LLM and ask it to provide its assessment about whether the data is disclosed in the passed sentences, as a two item tuple (i.e., <sentence index, disclosure type>). Overall, this process allows us to reliably associate the LLMs assessment about individual data types with individual sentences.

We label the disclosures either as: clear: If the data type description exactly matches a collection statement, vague: If the data type description matches a collection statement in broader terms, omitted: If there is no collection statement corresponding to the data type description, ambiguous: If there are contradicting collection statements about a data type description, incorrect: If there is a data type description for which the collection statement states otherwise. We further group these labels as consistent (i.e., consisting of clear and vague) and inconsistent (i.e., omitted, ambiguous, and incorrect) data flows (similar to prior work [27, 8]). To enable the LLM to assign one of these labels, we provide it several examples of these cases in a prompt template [51]. We list some of these examples in Table XI.

Since we assign multiple labels to each data type (per each data collection statement in the privacy policy), we next process the labels to assign it the most precise label, such that if consistent labels are present we prioritize them over inconsistent labels. We use the following precedence: clear, vague, ambiguous, incorrect, and omitted in determining the most precise label.

Next, we use our framework to check the consistency of data collection with the disclosures in Action’s privacy polcies.