AI Underwriting Models Can't Account for Property Changes They Can't See

An insurance underwriting benchmark from Snorkel AI found that general insurance knowledge was not enough when the correct answer depended on information specific to the business, the applicant, or the underwriting environment.

The benchmark, called UNDERWRITE, was developed with input from Chartered Property Casualty Underwriters (CPCUs) and tested 13 frontier AI models in a realistic commercial P&C underwriting environment. The agents had to interpret company guidelines, query databases, use tools, ask for missing information, and make underwriting decisions across tasks such as appetite screening, business classification, policy limits, deductibles, and product recommendations.

Property underwriting has the same gap in a different form. A property can change substantially while the information used to evaluate it stays the same. This post looks at what the benchmark found, and at how permit records can close part of that gap.

Key Takeaways

• In the UNDERWRITE benchmark, frontier models became less reliable as tasks depended more heavily on information outside their general pretrained knowledge.
• On product-recommendation tasks, GPT-5 Mini and GPT-5 Nano hallucinated insurance products that were not offered in the benchmark company's guidelines 58 to 66% of the time.
• A roof replacement, addition, or solar installation that is missing from an insurer's records is invisible to underwriters and AI systems alike. Permit records can show those changes and give a carrier's own data something to check against.

What the UNDERWRITE Benchmark Found

UNDERWRITE was designed to reproduce some of the messiness of enterprise underwriting. The simulated environment included proprietary business rules, multiple database tables, metadata, noisy tool interfaces, incomplete user information, and multi-turn interactions. Snorkel describes how it built the dataset in a companion post.

The authors rated how much each task's correct answer depended on carrier-specific information, and accuracy fell as that dependence rose. The drop was steepest for smaller models: one went from close to 100% accuracy on the least dependent tasks to 66% on the most dependent.

In product recommendations, some models suggested common insurance products that sounded reasonable but were not offered by the fictional carrier. The models knew insurance. The right answers were in the carrier's own guidelines and data tables, which the models could access, and they sometimes answered from general knowledge instead.

Snorkel has since released SnorkelUnderwrite 2.0, which makes the environment harder in ways that look more like a working carrier: role permissions, workflows that carry state from one step to the next, and tasks that require reasoning across more than one source. Snorkel frames underwriting as a decision process under incomplete information, and 72.5% of the 200 tasks in the new version are built around partial information. The results point the same direction as the first benchmark. On Snorkel's leaderboard, the best model passed 30.5% of tasks on the first attempt, and none passed more than 41% of answer checks. The two versions use different tasks and scoring, so the numbers should not be set side by side. What holds across both is the lesson: a more capable model does not fix a decision when the information behind it is incomplete.

The Property Change Gap

UNDERWRITE did not test permit data or missing records, and Gryd data was not part of the study. We read it from the data side. BuildZoom Data, now known as Gryd, has provided building permit data for over a decade leveraged in research publicized through national outlets such as The Wall Street Journal, Forbes, The New York Times and more. Snorkel's benchmark raises a question we strive to answer every day for our customers: what can a decision be checked against? A model can only check its answer against records that exist. In property underwriting, a roof replacement that never reached the record leaves nothing to check.

Property underwriting depends on understanding two things: what does the insurer know about this property today, and what has changed since it was last evaluated? Roofs are replaced. HVAC systems age and are upgraded. Solar is installed. Electrical and plumbing systems change. Homes are remodeled, expanded, repaired, or converted. Each event can change what is true about a property after the original policy record, inspection, or application was created.

Many underwriting inputs are snapshots. Construction activity is a timeline. Building permits can provide an independent record that work occurred at a specific property, including the type of project, when the permit was issued, and, where the source jurisdiction provides it, status, completion or inspection information, job value, and contractor details. Viewed over time, those records show how a property has changed.

Nationwide Insurance found that 51% of homeowners surveyed had completed a major renovation within the previous two years. Among those homeowners, 55% had not adjusted their insurance policies afterward. That does not mean every renovation was invisible to the insurer, but it shows how easily the property and the policy record can diverge. Self-reported information is an important part of underwriting. Permit records give insurers a second source to check reported details against, and a way to find permitted changes that never made it into the policy record.

Gryd data shows the scale of property-changing activity across the U.S. Over a 12-month period, millions of issued permits were captured across dozens of project types.

Issued Permits Captured by Gryd, September 2025 to August 2026

These four project examples were pulled from our data. Construction records surface many other kinds of property change, including electrical and plumbing work, kitchen and bathroom renovations, and other permitted improvements.

Using Permit History to Estimate the Age of a System

Permit history can show when major systems were installed or replaced, and a later permit can change what an earlier one suggested.

Consider the residential property below. Its construction history includes HVAC, electrical, mechanical, and plumbing activity spanning nearly two decades. Records show furnace and water heater work in 2016, additional HVAC and electrical activity that same year, electrical work in 2023, and new furnace and heat pump activity in 2025.

In this report, the 2016 furnace permit shows the age of the system. By 2026, that permit is ten years old, placing the documented furnace work within the typical 8 to 10 year replacement window. The report also shows additional HVAC, plumbing and electrical work.

The same approach applies to other major systems. Roofing permits, for example, can help identify properties approaching a 15 to 20 year replacement window. For underwriting and renewal teams, a permit history turns past construction activity into a reason to take a closer look at a property.

Start researching property history with Gryd. Create a free account to search properties and explore permit and construction activity across the U.S. →

Why Raw Permit Records Aren't Underwriting-Ready

Carriers already hold their own underwriting guidelines, which set the kinds of risks they will and won't write, along with policy and claims history, application data, and carrier-specific models. Permit records add an outside source about the property and the work done on it. Raw public records are hard to use directly: they sit in thousands of local systems, in different formats, with inconsistent labels and update schedules.

Gryd normalizes that source data through a five-stage data pipeline. Permit and contractor records are collected from primary sources, standardized into a common schema, mapped across properties, projects, permits, and contractors, and enriched to resolve entities that appear differently across jurisdictions.

What comes out the other end is a single history for each property: what work was done, when, what kind of project it was, which contractor was tied to it where the record allows, and what has appeared since the last time anyone looked. Across the U.S., that history spans 400M+ permit records, 6.5M+ contractors and 100M+ properties in 2,500+ jurisdictions, with 25+ years of history, and it grows daily as new permits are ingested.

None of that makes the data perfect, and we do not describe it that way. Permit records can differ from what a local government shows because sources update, permits get amended, publication lags, or an agency reports things its own way. Data quality assurance runs alongside the automated processing and enrichment, and when customers find duplicates, matching errors or source anomalies, they can report them to our data support team. That is the difference between a record that can be checked and a guess that cannot.

This is where the benchmark comes back to the carrier. UNDERWRITE showed that the quality of a model's answer depends on the quality of the record it is given. For underwriting teams, a documented construction history is that record. It can flag a property for a closer look, confirm an improvement the applicant reported, or surface new work at renewal. Analysts can search it directly in Explorer. For AI-assisted workflows, the same data is available through an API or bulk feed, so a model is reading a property's documented history rather than inferring one from general knowledge or an outdated profile. It does not replace inspections, applicant information, claims history, catastrophe models or a carrier's own underwriting rules. It gives all of them something to check against.

The Bottom Line

UNDERWRITE showed capable AI models going wrong when a decision depended on a carrier's own records and the models answered from general knowledge instead. Property underwriting adds a problem no model can reason its way out of: a new roof or a replaced HVAC system that never reached the record cannot be checked by anyone. Permit and contractor records, cleaned and connected into one property history, give underwriting teams and the systems they use a way to see those changes.

Bring construction intelligence into your underwriting workflow.

Access Gryd through API or bulk data to incorporate permit, property, and contractor data into underwriting models and internal systems.