Best Practices for Choosing Between and Combining Amazon Quick Dataset Q&A, Topic Q&A, and Dashboard Q&A

I'm Ishikawa from the Cloud Business Division. In Amazon Quick, even for the same operation of "asking in natural language," three methods are provided: Dataset Q&A, Topic Q&A, and Dashboard Q&A, and their internal mechanisms are completely different. Confusing them leads to incidents such as "the aggregation numbers don't match" or "I can't tell where the source numbers are."

This article assumes an environment where structured data on the scale of hundreds of millions of rows in Redshift / Athena coexists with unstructured documents such as PowerPoint / Excel, and organizes best practices for technically choosing between and combining the three methods.

Conclusion First

1. Answer numbers (sales and inventory aggregations) using only the structured route. Redshift/Athena "certified data mart" → QuickSight dataset → (Dataset Q&A / Topic Q&A / Dashboard Q&A). Since this is on the scale of hundreds of millions of rows, make direct query the main player and limit SPICE to aggregated marts.
2. Use PowerPoint/Excel as a knowledge base (RAG) that returns "context." Do not treat it as the source of truth for numbers. In particular, do not put numeric tables or aggregation tables from Excel into RAG; structure them and route them to the Dataset Q&A side (RAG cannot perform deterministic aggregation).
3. Divide roles as follows: Quick Chat (chat agent) = everyday inquiries, Deep Research (research) = generation of analysis reports with citations. Do not leave important KPIs to "All data"; instead, scope them to certified resources with task-specific custom agents.
4. The three methods are technically different things:
   • Dataset Q&A = text-to-SQL (dynamically generates SQL)
   • Topic Q&A = NLQ using a semantic model approach (goes through a pre-curated metadata layer)
   • Dashboard Q&A = NLQ against pre-configured visuals (already aggregated)
     ※ Knowledge base = RAG (semantic search + generation, for unstructured documents)

1. What Each One Is — The Technical Identity of the 3 Methods

This answers the question, "Dataset Q&A is text-to-SQL. So what are Topic and the knowledge base?" Even though all of them are "asking in natural language," their internal mechanisms are completely different. Confusing them causes "the aggregation doesn't match" incidents.

Key point: The knowledge base is RAG (Retrieval-Augmented Generation). Documents are indexed (Quick Index), chunks are retrieved via semantic search, and an LLM generates the answer. Because it deals with unstructured documents such as PowerPoint/Excel/PDF, it cannot be used for aggregation. It merely returns "the value as written in the document."

Key point: Dataset / Topic / Dashboard Q&A are "SQL-based to varying degrees" and their numbers are trustworthy. Only the knowledge base is RAG, and it should never be used for numeric aggregation.

2. Which Data to Combine, by Question Pattern

2-1. Inquiries about Sales Analysis (Aggregation) — Structured Route

Example: "Net sales by business unit for last month, year over year. What are the top 10 SKUs by gross margin rate?"

The combination of data needed to answer correctly (stacked from the bottom up):

• ① Certified data mart: Put KPI definitions in the data layer, not in prompts. Define metrics such as net sales / gross margin rate / allocatable inventory / inventory turnover / sell-through rate with Redshift/Athena views or dbt models, keeping the "source of truth" in one place.
• ② Dataset: Configure RLS (row-level security) / CLS (column-level security), date granularity, and default aggregations for non-additive metrics (inventory balance, rates, unit prices, and composition ratios must not use a simple SUM).
• ③ Three kinds of Q&A: Because of the hundreds of millions of rows, direct query (Redshift/Athena) is the main player. Dataset Q&A has no row limit with direct query. Use Topic for standard KPIs and Dashboard Q&A for existing dashboards.
• ④ Quick Chat: Go through a sales agent bound only to certified resources. Check the generated SQL, filters, and granularity of the answer with Explain.

※ Point for hundreds of millions of rows: SPICE has an upper limit of 2 billion rows / 2 TB even in Enterprise. Rather than loading raw detail into SPICE as is, a two-tier approach is best: load only marts aggregated by day, SKU, etc. into SPICE for fast responses, and use direct query for drill-down at raw granularity.

2-2. Inquiries about PowerPoint / Excel (Natural Language) — Unstructured Route

Example: "What was the aim of the Q1 discount campaign? How was the strategy against Competitor A decided in the meeting minutes?"

• ① Text-centric content (glossaries, definitions, policies, procedures, reports, proposals, meeting minutes) → goes to the knowledge base (RAG).
• ②③ Adding metadata: Also attach fiscal year, department, product category, target period, campaign name, and SKU/store ID to the documents, aligning them with the structured data using common business vocabulary and IDs (this is the key to bridging).
• ④ Usage: Use Quick Chat for simple search and summarization, and Deep Research for investigations spanning multiple documents + the web + internal data.

※ The most important pitfall — numeric tables and aggregation tables in Excel
Even if "the unstructured data includes numeric tables and aggregation tables," you must not put numeric tables into RAG (the knowledge base) and have them aggregated. Because RAG is semantic search, it cannot correctly calculate "totals" or "year-over-year."
→ Ingest Excel files to be aggregated into Athena/Redshift via S3, structure them, and handle them on the Dataset Q&A side. Only Excel files that are mainly text and findings go to the knowledge base. See the flow in 3-1 for the decision criteria.

2-3. Combining the Two (Bridging)

Example: "Why did sales drop in April?" → The numbers (size of the drop) come from Dataset Q&A, while the reasons, campaigns, and sales comments are cited from the knowledge base, and then combined.

• Don't just throw it all at Quick Chat; connect structured ↔ unstructured through common IDs and metadata.
• Have answers display separately the "source of the numbers (which dataset/SQL)" and the "source of the documents (which file)."
• Put the KPI definition document into the Topic/dataset descriptions as well as the knowledge base, so the vocabulary matches.

3. How to Ensure Accurate Aggregation and How to Choose

3-1. Routing Flow: "Numbers or Context?"

3-2. Six Design Principles to Ensure Accuracy

1. Put KPI definitions in the data layer: Define them centrally in certified views / dbt models / data marts. Natural language Q&A is simply layered on top.
2. Make Topic the primary entry point for standard KPIs : Fix ambiguous terms such as "sales," "year-over-year," and "stockouts" with synonyms and default aggregations. Do not allow simple SUM on non-additive metrics (inventory balance, rates, unit prices, composition ratios).
3. Open Dataset Q&A only on certified datasets : Use curated datasets, not raw tables. For important reported values, check the SQL, WHERE, date column, and aggregation granularity with Explain.
4. Minimize the target datasets for Dashboard Q&A : Since it may reference things other than the visuals currently displayed, protect hidden columns with CLS and make RLS mandatory.
5. Split the handling of Excel into two : Files to be aggregated (detail, budgets, inventory tables) are ingested into the structured route. Meeting materials and findings go to the knowledge base. Treat numbers returned by the KB as "values stated in documents," distinct from the BI source of truth.
6. Continuously verify with a golden question set (a collection of representative questions with confirmed correct answers) : A regression test in which you prepare pairs of "representative questions" and "expected values from manually verified SQL (= ground truth / correct values)" and periodically check whether Q&A answers match the expected values. An accuracy improvement of about 48% on a ground truth benchmark has been reported, but don't over-trust it; for data with audit requirements, corroborate with the original query.

4. Quick Reference of Data Scale and Limits

4-1. Structured Data (SPICE vs direct query)

Specific to Dataset Q&A: direct query has no row limit, SPICE is within its capacity and scales up to 2 billion rows, and RLS/CLS are respected. Not supported: composite datasets (parent SPICE + child direct query) / custom SQL parameters. Enterprise Edition is required.

Scale considerations for Topic / Dashboard Q&A: Both inherit the limits of the underlying dataset (SPICE/direct query). Topic has curation operating costs, and having too many fields makes it hard to manage. Because Dashboard Q&A is based on aggregated visuals, there is no concern about directly scanning large raw data.

4-2. Limits on Uploaded Files (Divided into Two Systems: Structured and Unstructured)

※ The core of choosing: Even for the same Excel file, if you want to "aggregate" it, use SPICE import/Athena ingestion (structured); if you want to "read and reference" it, use the knowledge base (unstructured). If you push numeric tables into RAG with a 30 MB extraction limit, large tables get cut off midway and the aggregation breaks.

4-3. Timeouts and Limits for Quick Chat / Deep Research

※ Rather than "relying on fixed values" for timeouts, the proper approach is to effectively avoid them through a design that keeps heavy aggregations light with marts/SPICE and materialized views.

5. Best Practice Configuration Optimized for This Requirement

5-1. Overall Architecture

5-2. Mapping of 3 User Tiers × Recommended Features

5-3. Implementation Roadmap

1. Build certified marts — Create certified sales and inventory marts / views (the source of truth for KPI definitions) in Redshift/Athena. Hundreds of millions of rows assume direct query, with only aggregated results loaded into SPICE.
2. Certify datasets — Configure RLS/CLS, date granularity, and aggregation rules for non-additive metrics. Write field descriptions and synonyms.
3. Enable Dashboard Q&A — Provide the shortest path using existing dashboards (minimizing target datasets).
4. Prepare Topics — Create sales/inventory Topics and enter synonyms, default aggregations, and business descriptions.
5. Open Dataset Q&A — Open it only on certified datasets. Apply an Explain verification workflow and Enrichment (synonyms and aggregation rules in YAML/JSON). Existing Topic assets can be migrated into enrichment files.
6. Build the knowledge base — Sync PowerPoint/Excel/PDF via S3/SharePoint, etc. Separate Excel files for aggregation into the structured route. Prepare document metadata and ACLs.
7. Build agents — Create sales/inventory agents in Quick Chat and bind only certified Dashboards/Topics/Datasets/KBs. Instruct them to state sources, filters, and assumptions in answers.
8. Continuous verification — Periodically compare Q&A answers against verified SQL using the golden question set. For Deep Research, also consider periodic report generation with Quick Flows.

6. Anti-Patterns

Finally

Although the three methods in Amazon Quick look like "the same natural language interface," there is a decisive difference: Dataset / Topic / Dashboard Q&A are SQL-based and their numbers are trustworthy, while the knowledge base is RAG and returns context. Routing numbers to the structured route and context to the unstructured route, and bridging them with common business vocabulary and IDs, is the key to achieving both accurate aggregation and rich context.

In particular, the three points "do not put Excel numeric tables into RAG," "put KPI definitions in the data layer, not in prompts," and "continuously verify with a golden question set" directly lead to preventing aggregation incidents. I hope this article helps with design when incorporating Amazon Quick into your business operations.