A study on ChatGPT chat data found that the system accumulates user information from past conversations, forming what the research team calls an "algorithmic self-portrait." Of the memories identified in the study, 96% were created or updated without explicit requests from users.
Researchers from institutions including the Max Planck Institute for Software Systems and the Institute for Security and Privacy in Germany analyzed data provided by 80 volunteers, finding 2,050 bot memories in the chat logs of 65 participants. Most of the content came from information directly disclosed by users during conversations, while another portion was inferred by the system itself, and these inferences were not always accurate.
One example cited in the study shows that when a user simply asked for band recommendations similar to Nirvana, the system recorded that they liked Nirvana. The study also states that 52% of memories involve psychological states such as users' desires, intentions, emotions, and beliefs. Users may view conversations with chatbots as private exchanges, yet underestimate the impact of having their content saved, analyzed, and used for personalization across different contexts.
The researchers suggest reducing personal details in questions, such as changing "I have diabetes" to asking about a hypothetical person. Users can also disable memory features, view or correct memory summaries, use temporary chats, or opt out of having their conversations used for training. However, the study emphasizes that memory summaries may not present all the information used by the system.