PowerLeap Logo

Copilot Studio Friction

Get notified when this record changes

One email when the status or the fix changes — double opt-in, no tracking, unsubscribe in every email.

Agent makes up answers or uses outdated information

Mitigatedsince 8 July 2026

Last verified

Details & related

Assessment

Confidence
CorroboratedMultiple independent sources describe the same behaviour.
Severity
Degrading
Typical time lost
Hours

Identification

Knowledge & groundingLLM stepsAll channels

Verification & changes

  1. Verified

    Doc check by human: Dossier sections A9/K1 and source threads re-read during seed migration; hallucination reports and restriction guidance unchanged.

  2. Change

    Provisionally approved by the Product Owner; external LLM quality review pending.

  3. Change

    Initial record created from the seed dossier (migration wave 1).

Related

Are you in the right place?

  • Your agent answers fluently, but some answers are wrong or invented.
  • The wrong details do not appear anywhere in your knowledge source.
  • Or the agent cites information that is clearly outdated.
  • It answers even when your documents contain nothing on the question.
  • The tone stays confident; there is no error and no warning.

If instead the agent answers nothing at all → see No answer, no error: silent "ContentFiltered". If instead correct documents exist but are skipped → see Your PDF files are ignored or not answered. If instead the outdated answers match old file versions → see New or updated files don't show up in answers. If instead Teams serves an old agent version → see Teams answers with an old version of your agent.

What's happening

The language model behind your agent is built to produce an answer, not to stay silent. When your documents leave a gap, the model fills it with its general world knowledge. That knowledge can be outdated or plain wrong for your organization. If web search or fallback behavior is active, outside content mixes into answers too. Think of a keen new colleague who never says "I don't know" and improvises instead. Nothing is technically broken: generative answers behave as designed. The fix is to narrow what the agent may use.

For technicians
  • Community threads, documented as of 2026-07-08, describe agents surfacing outdated vendor details absent from the grounding documents.
  • The recommended controls: restrict responses to your knowledge, disable web and fallback sources, and tighten instructions.
  • Practitioners additionally route high-stakes interactions through structured topics instead of free generation.

How to fix it

Solution 1

Official workaround

Restrict answers to your knowledge

  1. In Copilot Studio, open "Settings", then "Generative AI".
  2. Turn on the option that restricts responses to your knowledge sources ("Restrict responses to knowledge").
  3. Turn off web search and any general-knowledge fallback options.

✅ You should now see: the agent declining questions your documents cannot answer.

  1. Retest in the test pane with a question your documents cannot answer.

Solution 2

Community workaround

Tighten the agent's instructions

  1. Add an instruction that the agent must answer only from the provided sources.
  2. Tell the agent to admit openly when the sources contain no answer.
  3. Test both changes with questions inside and outside your content.

✅ You should now see: honest "no answer" behavior instead of invented details.

Solution 3

Community workaround

Build high-stakes paths as structured topics

  1. List the interactions where a wrong answer causes real damage.
  2. Build each of those as a structured topic with fixed messages.
  3. Leave open, low-stakes questions to generative answers.

✅ You should now see: critical paths returning identical, curated wording every time.

Check that it worked

Ask three questions your documents answer and three they don't, in the test pane. Expected: correct cited answers for the first three, and an honest decline for the rest.

If it didn't work

Prevent it next time

  • Decide per use case whether wrong-but-fluent answers are acceptable before you enable open generation.
  • Route high-stakes paths through structured topics from the start.
  • Review restriction settings after every change to knowledge or instructions.

Evidence

  • Community threadlearn.microsoft.com

    A maker reports the agent surfacing outdated vendor information absent from the HR documents, with knowledge restriction and tighter instructions as the fix.

  • MVP blogteam400.ai

    Practitioners describe generative AI producing confident-sounding wrong answers and recommend structured topics for high-stakes interactions.

  • Official documentationlearn.microsoft.com

    The generative answers node documentation covers configuring which sources answers may draw from.