Copilot on a messy semantic model: what breaks, and what it costs to fix

Copilot will answer a question about your data. Whether the answer is right is decided long before anyone types a prompt, in the naming conventions and descriptions of your semantic model.

The five defects that produce wrong answers

• Synonymous columns: CustomerName, Cust_Nm and Account all present in one model. The model picks one, and it is not always the one you meant.
• Missing descriptions: an ambiguous measure with no description gives the language model nothing to disambiguate with. It guesses, fluently.
• Ambiguous relationships: two date relationships where one is inactive and unspecified. Time intelligence then answers a different question than the one asked.
• Measures built in DAX instead of the semantic layer, so the same concept exists twice with different filters.
• No verified answers: without a set of questions with known correct values, you cannot tell improvement from variation in phrasing.

The hardening work, in the order that pays back

1  Name hygiene            Merge synonyms, one name per concept        ~2 days
2  Descriptions            Every measure: what it means, when to use    ~3 days
3  Relationships           Fix ambiguity, mark the active path          ~1 day
4  Definition alignment    Measures read from the metric layer          ~4 days
5  Verified answer set     40 questions with expected values, in CI     ~3 days

Steps one to three are what most teams buy. Step four is the one that stops the model inventing a second version of a number you have already reconciled. Step five is what lets you keep a promise: after any change, you can say whether accuracy moved.

What to promise, honestly

On a well-described model with a documented metric layer, 80 to 90 per cent of natural-language questions in our tests get the right answer. On a model with synonym columns and no descriptions, we have measured well under half, and the failures are confident. That spread is not a Copilot problem. It is the model quality you already had, now visible to people who cannot inspect it.

So the honest version of the sales conversation is: we will harden the model, measure accuracy on your data against 40 questions you supply, and publish the number internally. If it comes back at 60 per cent, the next euro goes into definitions, not into licences.

Natural-language BI does not create a data-quality problem. It removes the layer that was hiding it.