The Big Picture

Imagine asking an AI to roleplay as a person from Spain. Would it form friendships the same way as when it roleplays as someone from Japan? This seemingly simple question has profound implications for the growing field of AI-powered social simulations.

Large Language Models (LLMs) like GPT-4 are increasingly being used to simulate human behavior in social science research¹². Researchers use these AI “agents” to conduct virtual user studies, model social dynamics, and predict community behavior—all without the ethical complexities and costs of human participants. But how reliable are these simulations?

My research reveals a critical gap: LLMs generate networks that look structurally similar to real-world ones, but fail at the individual connection level. Additionally, the predicted network structure would vary significantly based on the language and cultural context of the prompt.

Why This Matters

Social simulation with AI agents is being used in political science, economics, and communication studies. If researchers trust these simulations to predict real human behavior, they need to know the limitations.

Previous research has shown that LLM-generated networks have similar overall structures (density, clustering) to human networks³. But does structural similarity mean the AI is actually predicting who would be friends with whom? Think of it this way: I could generate a random social network with 100 people where everyone has about 5 friends on average—that might have similar statistics to a real network, but it wouldn’t tell us anything meaningful about actual friendship patterns.

Other studies have raised concerns about biases. This is particularly concerning when LLM agents play minority roles, as it suggests that LLM values may align more closely with specific cultural groups⁴⁵.

Research Questions

1. Edge-level accuracy: When we ask LLM agents to predict friendships, how accurate are they at the individual connection level?
2. Cross-cultural variation: Does the language of the prompt affect how LLMs simulate social behavior? Do these variations reflect real-world cultural differences?

Testing Edge-Level Prediction Accuracy

I used the Indian Village Microfinance dataset⁶. It is a real social network data from 77 Indian villages that records who people borrow money from, ask for advice, etc.

Data Preprocessing

Our dataset is tie level, where each row represents a possible edge (friendship) between two people, along with their demographic attributes.

Find all possible pair of connections between any two villagers: ($V_i$, $V_j$)

• Y= if the edge exist (0 or 1)
• X = demographic features of $V_i$, $V_j$

Since the number of not exist edges is much more the existed edge, we applied balance method to prevent bias prediction.

In here we applied three balance method: SMOTE (Synthetic Minority Over-sampling Technique), ROSE (Random Over-Sampling Examples) and down sampled to handle class imbalance in friendship prediction.

SMOTE

Generate new edge (Y=1)

1. Select a random edge
2. Find the sample’s nearest KNN neighbors in feature space
3. Generate a new edge between them

ROSE

Generate new edge (Y=1)

1. Select a random edge
2. Create a small noise according to data distribution
3. Generate a new edge

Down Sampling

Drop non-existed edge (Y=0)

1. Select a random non-existed edge
2. Drop it

We trained Lasso Logistic Regression models on datasets processed with different class balancing methods to predict edge existence. Model performance was evaluated using precision, recall, and F1 score. All balancing methods yielded similar performance across these metrics, with SMOTE achieving the highest overall F1 score. We therefore selected the SMOTE-balanced dataset for subsequent analysis.

Identify What Predicts Real Friendships

Using Lasso Logistic Regression on the real network data, I identified which attributes most strongly predict whether two people will be connected. The top predictors were caste, religion, gender, and mother tongue—reflecting the homophily principle (people tend to befriend similar others).

However, notice that most of the variance in friendship formation is unexplained by demographics alone. This suggests a realistic upper bound for how well any model (including LLMs) can predict friendships based solely on these attributes.

Rank from the most influential variable to the least influential variables:

1. Caste
2. Religion
3. Gender
4. Mother
5. Tongue
6. If vilager have savings
7. Ration card ownership
8. Election card ownership
9. Ration card classify
10. Speak English or not
11. If vilager participate SHG or not
12. Status

Ask LLM Agents to Predict Friendships

Now it comes to the core of the experiment: Can LLMs predict friendships based on these attributes? The prompt design refer methods from existing research ³, which proven to generate structurally realistic networks:

From previous step, we know which attributes matter for network formation.

We select one of the smallest village (village 10th) with 93 villagers to run the simulation in order to know how different attributes affect the prediction accuracy. We select the top 2, 4, 6, 8 and all 12 attributes to include in the prompts and see how prediction accuracy changes.

Findings

Interestingly, the performance did not improve consistently as we added more attributes. 4 attributes performed best and adding more led to diminishing returns.

1. More attributes didn’t help. The model perform best when there are 4 attributes, but keep adding more attributes showed no consistent improvement in prediction accuracy.

2. Structural similarity is misleading. When evaluated by traditional network metrics (density, average degree), the Global method performed best. But at the edge level, it performed poorly. This is the critical insight: a network can look statistically similar while being completely wrong about who connects to whom.

Can Real-World Information Improve Accuracy?

I tested whether providing LLMs with real network information—either micro-level (individual friend counts or lists) or macro-level (network-wide statistics)—could improve predictions.

You are 60. backward class caste, HINDUISM, Woman, KANNADA, age 65. You are joining a social network.

You will be provided a list of people in the network, where each person is described as "ID. Caste, Religion, Gender, Mothertongue, Age", followed by their current friends IDs. In your free time, whose house do you visit? Provide a list of YOUR friends in the format ID, ID, ID, etc.

Do not include any other text in your response. Do not include any people who are not listed below. Do not select yourself.

3. backward class caste, HINDUISM, Woman, KANNADA, age 28; friends with IDs 43, 22, 1
31. backward class caste, HINDUISM, Woman, KANNADA, age 70; friends with IDs 21, 33
56. backward class caste, HINDUISM, Woman, KANNADA, age 20; friends with IDs 8, 15, 42

You are 60. backward class caste, HINDUISM, Woman, KANNADA, age 65. You are joining a social network.

You will be provided a list of people in the network, where each person is described as "ID. Caste, Religion, Gender, Mothertongue, Age", followed by their current number of friends. In your free time, whose house do you visit? Provide a list of YOUR friends in the format ID, ID, ID, etc.

Do not include any other text in your response. Do not include any people who are not listed below. Do not select yourself.

3. backward class caste, HINDUISM, Woman, KANNADA, age 28; has 6 friends
31. backward class caste, HINDUISM, Woman, KANNADA, age 70; has 4 friends
56. backward class caste, HINDUISM, Woman, KANNADA, age 20; has 5 friends

Your task is to create a realistic social network of an indian village. The village size is 93, the average degree is 8.80, the density is 0.096, the shortest path is 2.26, the diameter is 4.

You will be provided a list of people in the network, where each person is described as "ID. Caste, Religion, Gender, Mothertongue, Age". Provide a list of relationship pairs in the format ID, ID with each pair separated by a newline. Specifically, each pair of network tie ID1, ID2 means that ID1 visits ID2's house in their free time.

Do not include any other text in your response. Do not include any people who are not listed below.

3. backward class caste, HINDUISM, Woman, KANNADA, age 28
31. backward class caste, HINDUISM, Woman, KANNADA, age 70
56. backward class caste, HINDUISM, Woman, KANNADA, age 20

Real information helps, but only marginally. Precision improved from ~5% (random baseline) to ~8% at best. Interestingly, providing degree counts often outperformed giving complete friend lists—more information didn’t mean better predictions.

How LLMs Misunderstand Network Structure

Looking at the structural properties of generated networks revealed why:

LLMs fail in two opposite directions:

• With friend lists → Over-prediction. GPT-4o generated networks with ~50 average connections, far exceeding the actual ~9. Detailed information triggered an “everyone knows everyone” bias.

• With macro statistics → Under-prediction. Even when told “average degree is 8.8,” the Global method produced networks with only ~5 connections per person.

Does Language Change AI Social Behavior?

Due to unequal distribution of language in training datasets, LLMs generally perform better in English⁷⁸. Most research on multilingual prompts focuses on improving LLM performance in minority languages—but rarely examines how prompt language affects LLM behavior in social simulations.

Cross-National Dataset

I used the ISSP 2017 Social Networks survey: a dataset from 30 countries measuring social behavior like frequency of meeting friends, trust levels, and loneliness.

Simulation Design

For each country, I:

1. Added country context to the persona (e.g., “You are a typical adult living in Spain”) and translated prompts into corresponding local language (e.g., Spanish for Spain, Japanese for Japan)

You are a typical adult.

Task:
• After silent reasoning, list EVERY person in the roster you would be close friends with. By "friends," we mean people you regularly interact with and trust enough to discuss personal matters or ask small favors.
• Output one line of IDs separated by comma + single space: ID1, ID2, ID3, ...
• Use ONLY IDs from the roster. Do NOT include your own ID. No extra text.

Think step-by-step internally; reveal ONLY the final ID list. DO NOT ADD ANY ADDITIONAL WORDS

YOUR PROFILE
ID: 24
24. female, age 69, Upper secondary, In paid work

ROSTER:
ID, Sex, Age, Education, Working Status
2. male, age 48, Lower secondary, Retired
28. female, age 23, Lower level tertiary, In paid work
13. male, age 77, Lower level tertiary, Retired

You are a network generator.

Task:
• Using ONLY the people in the roster, create a realistic undirected friendship network. By "friends," we mean people you regularly interact with and trust enough to discuss personal matters or ask small favors.
• Output each friendship as: ID1, ID2 (comma + single space).
• One pair per line. No headers, no commentary, no duplicates, no self-ties.

Think silently; reveal ONLY the list of pairs.

ROSTER:
ID, Sex, Age, Education, Working Status
2. male, age 48, Lower secondary, Retired
28. female, age 23, Lower level tertiary, In paid work
13. male, age 77, Lower level tertiary, Retired

Eres un adulto típico que vive en España.

Tarea:
Después de pensar en silencio, enumera a todas las personas del listado con las que llegarías a ser amigo íntimo. Por «amigos» nos referimos a aquellas personas con las que interactúas con frecuencia y en las que confías lo suficiente como para hablar de asuntos personales o pedir pequeños favores.

• Salida: una línea con los IDs separados por una coma y un solo espacio: ID1, ID2, ID3, …
• Usa únicamente los IDs del listado. No incluyas tu propio ID. Ningún otro texto adicional.

Piensa paso a paso en privado; muestra solo la lista final de IDs; no añadas ningún otro texto. Por favor, asegúrate de que el texto dentro de los paréntesis permanezca sin cambios.

Tu perfil personal
ID: 12
12. Hombre, 44 años, educación superior (avanzada), trabajo remunerado, vive en Spain

Lista:
ID, sexo, edad, nivel educativo, situación laboral, país
16. Mujer, 34 años, educación secundaria, trabajo remunerado, Spain
20. Mujer, 38 años, educación secundaria, trabajo remunerado, Spain
18. Mujer, 37 años, educación postsecundaria (no universitaria), trabajo remunerado, Spain

Eres un generador de redes.

Tarea:
Usando solo las personas del listado, crea una red realista y no dirigida de amistades para Spain. El término "amigos" se refiere a aquellas personas con las que interactúas con frecuencia y en quienes confías lo suficiente como para hablar de asuntos personales o pedir pequeños favores.

• Salida: cada relación de amistad debe tener el formato: ID1, ID2 (coma + un solo espacio).

Ejemplo de salida:
6, 22
7, 13
8, 10

Una pareja por línea, sin título, sin comentarios, sin repeticiones ni autoenlaces.

Piensa en silencio; solo muestra la lista de pares correctos. Asegúrate de que todo el texto dentro de las llaves se mantenga sin cambios.

Lista:
ID, sexo, edad, nivel educativo, situación laboral, país
16. Mujer, 34 años, educación secundaria, trabajo remunerado, España
20. Mujer, 38 años, educación secundaria, trabajo remunerado, España
18. Mujer, 37 años, educación postsecundaria (no universitaria), trabajo remunerado, España
23. Mujer, 54 años, educación postsecundaria (no universitaria), trabajo remunerado, España

2. Generated 30 networks per country using both Local and Global methods, each network with 30 personas
3. Tested with GPT-3.5-turbo and GPT-4o

Overview of Generated Networks

We starting from examining the most basic network property: degree distribution. The degree distribution of most of countries predicted network are similar, which follow the power law distribution similar in the real-world.

Language Context Matters

I computed correlations between:

• Real-world country-level variables (GDP, democracy score, loneliness, depression, trust, etc.)
• Network metrics from LLM simulations (average degree, clustering, density, etc.)

Interestingly, if we simply use english prompts for all countries, the correlations are much weaker. The maximum correlation coefficient between network matrix and country variables is around 0.1, it is about 1/3 of the correlation when using local language prompts. It suggest that language amplify the LLM’s pattern when predicted a social network, whatever it is a bias stereotype or a pattern existed in real world.

1. Prompt language affects network structure but inconsistently. For example, in GPT-3.5-turbo, higher GDP countries show lower average degree and sparser networks, while GPT-4o shows the opposite pattern.
2. LLM-generated networks don’t have a stable pattern. For instance, cont_tot (weekly contact frequency) should positively correlate with avg_degree, but this only holds for GPT-3.5-turbo local and GPT-4o global. GPT-4o local shows a negative correlation, and GPT-3.5-turbo global shows no significant relationship.
3. GPT-4o produces more significant correlations than GPT-3.5-turbo, but not more valid ones. Notably, GPT-4o’s generated networks correlate more strongly and consistently with country-level variables (GDP, democracy index) than with individual-level behavioral measures (e.g., cont_tot, cont_f2f). This suggests GPT-4o may rely more on stereotypical associations about “what social networks should look like in certain types of countries” rather than capturing actual individual-level variations.

When we look further into the correlations, a surprising pattern emerges: GDP and democracy index show stronger correlations with predicted network metrics, while individual-level social behavior variables (contact frequency, loneliness) which should relate to networks more directly, show weaker or inconsistent correlations.

GDP predicts friendships, but weekly contact frequency doesn't. Why?

Click to reveal

These indicators represent an outsider's view—how the world ranks and describes countries. And that's exactly what's in the training data: text that talks about countries, not the lived experience of people in them.

Click to flip back

Key Takeaways

Structural ≠ Accurate

Networks can look statistically similar while being completely wrong at the individual level

Imagined, Not Lived

The predicted network structure are more related to how countries are described from the outside than to measures of people’s lived experience within the country

Language As Amplifier

Prompt language amplifys LLM’s pattern when predicting social networks, whether it’s bias or real-world patterns

References