AI Towns and Agent Based Modeling

How LLMs could transform economic forecasting

I have recently been thinking about old ideas that could be revived based on the recent progress in AI and LLMs. Recently, I wrote about the Semantic Web¹. Another similar area is agent-based modeling. Analogous to the semantic web, this technique has underdelivered and not become mainstream in economics².

Agent based modeling (“ABM”), or agent-based simulation, refers to statistical models predicated on programmable computer agents that interact with each other and an environment based on a series of rules specified at the agent-level. Typically, the emergent behavior that is then observed at the population level is of interest. The point is that it is difficult to predict population-level emergent behavior from even a handful of simple rules, so it is easier to simulate it³.  

One challenge with agent-based simulation has always been in encoding the preferences and rules by which agents would behave, and then empirically proving such preferences and responses are empirically accurate in the first place. Simulating actual human behavior, even in a narrow space, is surprisingly difficult: sure, you only want to simulate agents in the stock market – but how do you know that what was on the news that day or what the agent had for lunch does not impact its behavior? By explicitly hardcoding preferences, you are making explicit and non-empirical assumptions about which part of human context matter to a decision. Probably the technique has only gone “mainstream” in video games: the AI in titles such as Roller Coaster Tycoon and SimCity can be thought of as predicated on this approach.

LLMs present an opportunity to transform the technique because they can power agents that contain realistic context and knowledge windows orders of magnitude larger than traditional ABMs without explicit preference hardcoding. Just as large language models predict “the next word”, this capability can be used to predict “the next action” based on an agents’ knowledge, individual memory, and action history. That knowledge, memory, and action history can be trained on public data sources and real observational human data.  

What is particularly promising about this approach is that there exists the possibility we can simulate close-enough human behavior at the agent level and then recover population-level phenomena that we do not know about yet. This could revolutionize social sciences and economic forecasting because the cost of experimentation becomes very cheap and because experiments can easily be replayed in various configurations. The key question is whether the individual agents properly simulate actual people (the non-empirical assumptions objection to traditional ABMs). One way to test this is to run simulations and see if we can recover known population-level effects from experiments⁴. If a sufficient number of these known effects replicated, it would be reasonable to trust new latent population-level effects we discover⁵.

The key to this is, of course, data. Beyond providing basic contextual knowledge data to the agents (ie. simulating what most people “know”), you would also want to provide historical behavioral data that the agent could replicate (for instance, real historically observed web browsing patterns, purchasing behavior, particular product preferences, etc.). You would want then agents to adopt the “behavioral persona” of people in the population in proportion to the number of such people exist. So, a successful implementation of this approach requires a variety of datasets, some of which are already widely used in training large models and some of which are not today.

If this works, it would be a major advance in economic forecasting and microeconomics. If the simulation accurately simulates population-level effects, rather than taking a “top down” econometrics or statistical modeling perspective to timeseries, we could take a “bottoms up” approach to forecasting. Forecasters could easily simulate different scenarios and see how the population of agents would react and what the aggregate outcome would be. This also massively simplifies hierarchal timeseries forecasting, as you get all the complexity “for free” and can just query your simulated result. Obviously, the accuracy of such simulations hinges on how accurately the agents have “internalized” the preferences, likely actions, and reactions of actual people. The complexity of human behavior is likely impossible to capture with a finite number of explicitly encoded rules, rather, it seems far more likely to be able to benefit from emergent behavior that follows from enough training data.

Some promising projects I am watching in this space include:

·      a16z Infrastructure team’s AI Town

·      Simulcra – and the associated Stanford paper

·      John Horton’s paper that explicitly aims to replicate known behavioral results

1.

2. A great summary of some reasons behind this is here.

3. In Bayesian statistics, there is an analogous concept where certain posterior distributions are easier calculation by simulation than by integration.

4. John Horton explicitly takes this approach. The other key question is whether agents might be learning of expected effects by ingesting knowledge about known economic phenomena. One would ideally exclude this knowledge.

5. As mentioned in (4), there is always the risk that the agents “know about” these results from the corpus of data that they have been trained on. The real hope is that we can recover effects that have not been explicitly taught to the agents but are emergent from their preferences and correspond with experiment research in the real world. Ideally, such economic knowledge would be explicitly excluded from the LLM training data.