[Copilot Studio] Tried Generating Insights from Aggregated Numbers: Also Includes Settings to Block Responses Without Basis

Introduction

Hello, I'm Keima.

In KPI reports, it's common to want not just a list of numbers, but also comments (insights) that indicate "where things are growing" and "what to pay attention to."
Since these insights are written text, they're something we'd like to leave to generative AI.
However, when you have generative AI comment on numbers you've provided, there's a concern that it might fabricate numbers you didn't provide, or make comparisons that don't match the facts.

This article covers methods for generating insights from aggregated numbers and techniques to keep those insights grounded in facts, with hands-on verification conducted as of June 2026.
The approach centers on leaving calculations to deterministic means and having generative AI handle only the "interpretation of numbers."
I hope this serves as a reference for those who want to add accurate comments to numerical reports.

This article is the 8th installment in a series on building agents with Copilot Studio.
The series as a whole aims to build an agent that handles "collection → aggregation → charts → insights → documentation" end-to-end, and this article covers the "insights" portion.

Target audience: Those who want to generate comparison comments and key points from numbers in Copilot Studio, in a way that stays true to the facts

Series Article List

In the previous installment (Part 7), we got as far as displaying KPI charts inside a chat.
This time, we'll generate written comparison comments and key points (insights) from those numbers and charts.

1. What We're Doing This Time

We'll generate insights from aggregated numbers.
The key is to separate the calculation of numbers from the interpretation of numbers.

• Have generative AI handle only "interpreting the provided numbers"
• Apply techniques to keep it grounded in facts (pass pre-calculated numbers / fix numbers via instructions)
• Verify with real hands-on testing that insights are generated from the KPIs of three companies
• As an advanced topic, also try the setting that blocks responses that don't use tools or knowledge

For verification, we'll use the same fictional SaaS companies from previous installments (CloudNova / StreamForge / Datapeak) and their KPIs (fictional values).

2. Insights Are Interpretation, Not Calculation

In previous installments, calculations such as totals, averages, and year-over-year figures were handled by deterministic means rather than generative AI.
Insights are the work of reading those pre-calculated numbers and writing text about "where things are growing" and "what to pay attention to."
This is interpretation, not calculation, which is where generative AI excels.

In other words, we divide the roles as follows.

With this division of roles, the numbers themselves remain accurate while only the comments are generated by AI.
The remaining challenge is to suppress generative AI from "fabricating numbers that weren't provided" or "making comparisons that don't match the facts," after which insights can be delegated with confidence.

3. Techniques for Keeping Insights Grounded in Facts

When passing numbers to a prompt and asking for insights, there are two techniques for keeping the output grounded in facts.

By passing pre-calculated numbers (Technique 1) and instructing it to use only those numbers (Technique 2), we confine generative AI's job to interpretation alone.
In addition, Copilot Studio has a setting to "block responses that don't use tools or knowledge," but it serves a different purpose and is covered in Chapter 5 as an advanced topic.

3.1 Pass Pre-Calculated Numbers

The first technique is to pass the aggregation results (pre-calculated numbers) from previous steps.
If you ask generative AI all at once to "aggregate this Excel file and comment on it," calculation and interpretation get mixed together, creating room for numerical errors.
Complete the calculations in a prior step and pass only the "finalized numbers" in the insights step.
This way, all generative AI needs to do is interpret.

3.2 Fix Numbers via Instructions

The second technique is to explicitly state in the instructions (prompt) to "use only the provided numbers."
You can control generative AI's behavior through custom instructions and prompts.

When asking for insights, structure the instructions as follows, for example.

Please comment based only on the following aggregated values. Do not calculate new numbers or bring in numbers not listed here.
- CloudNova: 2025 ARR 1,950M JPY (YoY +20.0%), NRR 120%, Operating Margin 13.5%
- StreamForge: 2025 ARR 1,300M JPY (YoY +10.2%), NRR 109%, Operating Margin 1.0%
- Datapeak: 2025 ARR 2,810M JPY (YoY +12.4%), NRR 128%, Operating Margin 19.0%
Please structure the output in two parts: "Key Points for Each Company" and "What Can Be Said from Comparing All Three Companies."

By embedding the numbers directly in the message body and constraining it to "not bring in numbers not listed here," we confine generative AI's job to interpretation.

4. Generating Insights with Real Hands-On Testing

We'll put the techniques covered so far into practice using the integrated agent "KPI Report Creation Assistant," which handles KPIs for all three companies together.

4.1 Verify Agent Settings

Before generating insights, confirm two things.

First, make sure the generative AI orchestration is set to "Generative" (Settings → Generative AI → Orchestration).
This allows the agent to independently select and use tools and knowledge as needed.

Generative orchestration enabled. "Yes, use available tools and knowledge as appropriate to dynamically respond" is selected

Second, confirm that the deterministic aggregation tool built in previous installments is incorporated.
The tool list includes "KPIScriptAgg Quarterly Aggregation (Office Script · Deterministic)," forming the foundation for leaving calculations to this tool and having generative AI handle only interpretation (Technique 1).

Tool configuration. "KPIScriptAgg Quarterly Aggregation (Office Script · Deterministic)" for deterministic aggregation is incorporated. A prompt tool for Word mail merge is also included

Note that for the insights in this installment, numbers are passed directly in the prompt, so knowledge and tools are not used.
Therefore, "Allow ungrounded responses" (the setting covered in Chapter 5) is left on.

4.2 Pass Aggregated Values and Request Insights

In the test panel chat, send the instruction text from Section 3.2 as-is.
The constraint to "base responses only on the provided aggregated values," the finalized values from the prior aggregation step (2025 ARR, YoY, NRR, and operating margin for CloudNova / StreamForge / Datapeak), and the output structure of "Key Points for Each Company" and "What Can Be Said from Comparing All Three Companies" are all combined into a single message.

The actual message sent is as follows.

Please comment based only on the following aggregated values. Do not calculate new numbers or bring in numbers not listed here.
- CloudNova: 2025 ARR 1,950M JPY (YoY +20.0%), NRR 120%, Operating Margin 13.5%
- StreamForge: 2025 ARR 1,300M JPY (YoY +10.2%), NRR 109%, Operating Margin 1.0%
- Datapeak: 2025 ARR 2,810M JPY (YoY +12.4%), NRR 128%, Operating Margin 19.0%
Please structure the output in two parts: "Key Points for Each Company" and "What Can Be Said from Comparing All Three Companies."

Requesting insights in the test panel. The instruction "Please comment based only on the following aggregated values..." is accompanied by the finalized aggregated values and output structure

4.3 Reviewing the Generated Insights

Upon sending, key points for each company along with a comparison table and comments across all three companies were returned in a single response.

Agent response. Key points per company and cross-company comparison table and comments were generated in line with the provided aggregated values

In this verification, the returned comments were consistent with the provided aggregated values (ARR, NRR, and operating margin for CloudNova / StreamForge / Datapeak), and no numbers that were not provided were introduced.
However, since generative AI responses vary with each response, the content may differ even with the same request.

As shown here, by completing calculations in a prior step, the insights step can be delegated to generative AI while maintaining a reasonable level of numerical accuracy in the comments.
It's safer to verify the output multiple times before publishing.

5. (Advanced) Setting to Block Ungrounded Responses

Through Chapter 4, we passed numbers directly in the prompt and kept the output grounded in facts using Techniques 1 and 2.
Here, as an advanced topic, we'll cover the "Allow ungrounded responses" setting in Copilot Studio.
This is a setting that blocks responses generated solely without using tools or knowledge, and it takes effect when configured to pass numbers via tool outputs or knowledge.

This setting is found under "Settings" → "Generative AI" → "Knowledge".
When turned off, it can block responses generated in a turn where no knowledge or tools were used.
Note that this setting is used with agents that have generative orchestration enabled.

This setting requires that the agent has generative orchestration turned on.

Source: Knowledge sources summary | Microsoft Learn

"Allow ungrounded responses" under "Settings" → "Generative AI" → "Knowledge." When turned off, responses that don't use knowledge sources or tools (including responses that only reference conversation context) are blocked

When turned off, what gets blocked is "responses in a turn where neither knowledge nor tools were used."
Turns where the agent attempts to answer relying only on conversation history or general knowledge without calling knowledge or tools are blocked, and the fallback topic triggers.

When you turn off this setting, the agent blocks any response generated in a turn where it didn't use a knowledge source or tool. This condition means that if the agent decides to answer a question directly from the conversation history or its general knowledge, without calling a knowledge source or tool, the response is blocked and the fallback topic triggers.

Source: Knowledge sources summary | Microsoft Learn

5.1 Relationship Between How Numbers Are Passed and This Setting

The key point is that the determination is based on "whether tools or knowledge were used in that turn."
The following diagram organizes the combinations of how numbers are "passed" and this setting.

A turn where numbers are only written directly in the prompt body, as in Chapter 4, is considered to be looking at only "conversation context" and is not treated as having used tools or knowledge.
Therefore, turning this setting off will block the insights themselves (which is why it was left on in Chapter 4 — the path to the lower left in the diagram).
On the other hand, if aggregated values are passed as output from a prior tool (flow or code interpreter) or as knowledge, the insights turn is treated as grounded in tools/knowledge and will not be blocked even when off (the path on the right side of the diagram).

However, turning this setting off does not mean generative AI will stop using general knowledge entirely.

Turning the Allow ungrounded responses setting off doesn't guarantee that the agent never uses general knowledge.

Source: Knowledge sources summary | Microsoft Learn

Therefore, rather than relying on this setting alone, use it in combination with Techniques 1 and 2 from Chapter 4.

5.2 Observing Responses Being Blocked in Practice

We verified with a minimal agent configuration what actually happens to responses in a turn that uses neither tools nor knowledge when this setting is off.
The steps were as follows.

• Create a new agent with no tools or knowledge
• Turn "Allow ungrounded responses" off (the state shown in the screenshot above)
• Also turn "Use web information" off (if left on, general questions can be answered based on web search and won't be blocked)
• Send a general question ("What is the tallest mountain in Japan?") in a state where there are no source materials to ground any answer

The result was as shown below — the generated-only response was blocked and switched to a fallback message.

Response when no tools, knowledge, or web search can be used. "I couldn't find information to help answer this" was displayed, the generated response was blocked, and it switched to the fallback "I'm sorry, I couldn't understand your inquiry"

With the same question, turning "Allow ungrounded responses" back on results in the agent answering from general knowledge: "The tallest mountain in Japan is Mt. Fuji."

Same agent, same question, with "Allow ungrounded responses" turned back on. Without using any tools or knowledge, it answers from general knowledge: "It's Mt. Fuji"

The same logic applies to generating insights.
If aggregated values are passed as tool outputs or knowledge, insights will pass through even with this setting off, and only answers that don't go through the data will be blocked.
Since the procedure in Chapter 4 involves writing directly in the prompt, this setting was left on.

6. Summary

Since insights are interpretation rather than calculation, they are a step that can be delegated to generative AI.
To keep comments grounded in facts, the basics are to pass pre-calculated numbers (Technique 1) and fix them via instructions so only those numbers are used (Technique 2). These two techniques are what keeps the output from straying in this installment's procedure.
For those who want to impose even stricter constraints, there is a method of passing numbers as tool outputs or knowledge and then turning off "Allow ungrounded responses" to block generation that isn't based on data (Chapter 5).

By ensuring numerical accuracy in the prior deterministic aggregation step and having generative AI handle only interpretation, you can maintain the credibility of reports even when automating through the insights stage.
Starting from the next installment, we'll move on to putting the numbers, charts, and insights built so far into Word and PowerPoint documents.

References

• Use prompt modification to provide custom instructions to your agent | Microsoft Learn
• Add the code interpreter capability to your agent | Microsoft Learn