Before building a new house or apartment building, residents and developers must first ensure their project adheres to local zoning rules. 

But zoning documents are often difficult for both applicants and local government staff to navigate. This can lead to extended back and forth between staff and applicants, slowing down the permit process at a time when most communities don’t have enough housing.

Some local leaders have expressed interest to Urban researchers in the promise of using generative AI to make zoning codes easier to understand, whether through screener tools that scan permitting applications to ensure they are complete, or chatbots that can answer development questions. However, leaders are hesitant to adopt tools that haven’t been properly vetted for the quality of the information they produce.

To test how well generative AI tools could interpret zoning codes, we ran a benchmarking exercise that evaluated the capabilities of various large language models (LLMs), building on previous work that explored whether machine learning could help automate the collection of standardized zoning data. To do this, we developed a set of zoning and permitting queries for Minneapolis—a city with a complex, 467-page zoning code and with zoning processes familiar to our team.

Building a retrieval-augmented generation (RAG) system for zoning

With the city of Minneapolis’ zoning code as the source of truth, we needed a method for serving only the relevant parts of the zoning code to the LLMs to answer user questions. We wanted to avoid putting all 467 pages into an LLM’s context window, because larger inputs could be cost prohibitive for local governments trying to build applications at scale.

To serve only relevant parts, we used retrieval-augmented generation (RAG). This technique augments an LLM’s response to a user’s query by retrieving relevant information from the zoning code document that aids the LLM in generating a full response. It allows an LLM to access information that is outside the scope of its training data and helps ground its output in factual and pertinent information. 

As part of the RAG framework, we needed to first partition the zoning code into smaller chunks that could be stored in a searchable database. For chunking, we used the open-source LangChain platform, which provides several tools for splitting text. We experimented with multiple approaches, including a recursive splitter that tries to find common separator characters and splitters from the Unstructured library, which can preserve aspects of document structure like headings, paragraphs, and tables.

To create this searchable database, we then generated embeddings—numeric representations of the zoning code represented in vector form—using the E5 family of open-source embedding models developed by Microsoft and available from HuggingFace. We stored the resulting embedding vectors in the open-source Facebook AI Similarity Search (FAISS) vector database, which allows the system to do an efficient similarity search and retrieve the text chunks most relevant to a user’s query.

Finally, we selected several LLMs to handle the queries themselves, including a mix of open-weight models such as Meta Llama 3.2 1B Instruct and Mistral Pixtral Large 25.02, lightweight models like OpenAI’s GPT-5 mini, and a more powerful model, ChatGPT 5.1. We accessed the Meta and Mistral models through Amazon Bedrock’s managed services, GPT-5 mini through OpenAI’s API, and ChatGPT 5.1 through the graphical user interface on chatgpt.com. ChatGPT 5.1 allows for larger context windows, so we supplied the entire zoning code in context to have a non-RAG approach for comparison (though this would not be a cost-sustainable approach for a functional local chatbot).

At this point, we had a ready-to-test system that could retrieve specific document chunks from the Minneapolis zoning code and use them to augment LLM’s responses to user queries.

But we still needed an evaluation method grounded in realistic scenarios local residents might present to genAI tools—something that would allow us to compare different technical configurations across different types of user questions.

Crafting an evaluation method

To make the benchmarking exercise realistic, we designed our evaluation across three dimensions: prompt audience, prompt complexity, and prompt repetition.

We defined the prompt audience by developing two user personas who might use a genAI-enabled zoning tool:

• User 1: A professional developer who wants to apply for a building permit for a new multi-family apartment building in Minneapolis.
• User 2: An owner of a single-family home in Minneapolis who wants to build an accessory dwelling unit (ADU) on their property.

We then developed a set of questions that each user might ask, recognizing that a professional developer is likely to be more familiar with zoning regulations than a typical homeowner. 

This list included

• simple information extraction questions, like which zoning requirements apply to the user’s district,
• questions requiring comparison and synthesis, like assessing the legality of the user’s proposed new development, and
• questions requiring a specific format, like a checklist for a new building permitting application.

Importantly, we assumed user 1 and 2 knew which zoning district they wanted to build in.

Districts can be both complex and overlapping. Without access to a machine-readable zoning map, even LLMs with web search could not correctly answer what district an address was in. The interactive content on Minneapolis’ website made the information more accessible to human readers but was harder for the LLM to parse. For this reason, to ask more detailed questions about a given district, we had to supply the zoning district directly to the model in the prompt.

We defined various levels of prompt complexity for the same set of questions. Questions could be sparsely phrased using keywords similar to a Google search, simply phrased in a conversational tone, or more elaborately structured using prompt engineering. Finally, we iterated the same set of prompts for each user multiple times and tested the variations in LLM responses across runs.

The table below maps out a few examples broken down by user, prompt complexity, and task evaluated.

For a deeper look, check out the full evaluation matrix and codebase for our benchmarking exercise.

Next, we created a system prompt, or a set of instructions that guides how the LLM generates responses. Our system prompt encouraged specific behaviors in the LLM responses and imposed instructional guardrails such as refraining from responding when the model deemed the retrieved context to be insufficient.

For each prompt in our repertoire, we asked an Urban Institute zoning expert familiar with Minneapolis zoning regulations to provide a ground-truth response.

We then compared the LLM responses we received against these ground-truth answers and manually assessed each response using the following criteria:

• Accuracy: Is the response to the user’s question factually correct?
• Relevance: Does the response address the user’s question directly?
• Reference to context: Does the RAG system pull the correct sections from the zoning code to answer the question?
• Consistency: Does the model give a substantively similar answer when asked a question multiple times?
• Confidence: Does the model capture uncertainty in its response when there is no clear answer or it does not have adequate information?

Each criterion was scored as fully met, partially met, or not met.

What We Found

We found the usability and accuracy of the LLM’s responses were heavily shaped by limitations in retrieval and the specificity of user queries.

Since we constrained responses to rely strictly upon retrieved context, poor retrieval performance directly resulted in noisy and ineffective responses. Zoning ordinances are inherently interconnected documents, with relevant information distributed across disparate sections, often making context retrieval challenging.

In cases where the relevant context was more narrowly constrained to a particular section, answers tended to be objective, specific, and closely aligned with exact keywords referenced in the question. Click on the box below to see an example of a question and response from our Mistral model that correctly covers primary uses, permitted structures, and even interaction with relevant overlay districts.