VLDB SANA Slides

My name's Austin, an undergrad at Columbia's Data, Agents, Processes Lab (DAPLab) supervised by Professor Eugene Wu and mentored by Jerry Liu.

In this talk, I am going to talk about question-answering agents and how scale affects difficulty and how careful ablation can pinpoint bottlenecks for improvements.

With a simple user question as an example, “Which supplier caused the most delivery delays in our Midwest locations last quarter?”, we can explain what QA agents do.

In general, QA agents plan a series of subquestions needed to answer the question, where each subquestion may require the agent to search a knowledge base for relevant sources and analyze those sources to extract facts.
They then use those facts to answer the next subquestions.

At the smallest scale, if we tell the agent the desired states are Indiana and Illinois and give it the deliveries table, then the agent just has to correctly write SQL to compute the result.

If we asked "in the Midwest" instead, this is a 2-hop question: the agent has to plan to first find out which states are Midwest, run SQL to resolve them to Indiana and Illinois, and then find the suppliers with the most delays in those states by generating more analysis queries.

There’s a lot of progress on this type of text-to-SQL, and accuracy on benchmarks like Spider 2.0 Snow is over 90%

Now, if we had 1k+ data sources, then an agent cannot fit all those sources into context.

So it first has to search for the relevant sources before running SQL on it to get the same filters.

Recent work like Kramabench provides 1.7K sources and the current best implementation under Sonnet 3.7 reaches only 56% accuracy on that benchmark.

In general, tasks are more difficult as we scale the data lake, increase the required steps to answering the question, and increase the sources required.

In LakeQA, our ICML 2026 work, we scale this to the extreme by building a 10TB data lake of 40M files, where each task in this benchmark has about 13 reasoning steps and 7 required sources.

In LakeQA, even state-of-the-art models like Opus 4.5 get 46% and GPT 5.2 18%, but end-to-end accuracy doesn’t help us understand the root causes, where research is, and where we need more progress.

Scale degrades the quality of planning, search, and analysis, and a failure at any step can cascade.

SANA is an evaluation framework to isolate the contributions of each component by keeping other components at oracle.

Specifically, for planning, it gives you the right subquestions to answer the question. Then in each subquestion, search always retrieves the required sources relevant to the query, and analysis always extracts the correct fact from those sources.

SANA does intent-based ablation.

An agent’s intent does not necessarily result in the right answer as implementation could be a bottleneck.

For instance, even if the agent intended the right query for a source, the relevant sources might not appear from a search tool as they get pushed out of the top-k by the scale of the lake.

However, if we restrict the data lake into the gold datasets required to answer the question, then the retrieved sources will always be relevant.

Hence, if the agent had the right intent in its query, it will always get the right results.

SANA can answer “how big a component’s bottleneck is”: by taking the oracle implementation as an upper bound, and making a naive implementation like BM25 for search, then the difference between the accuracy acts as a proxy for how big a component’s bottleneck is.

You can also judge how good your implementation of a component is by calculating how far it is from the oracle and how much better than naive it is.

The experiments we ran were on Kramabench and a 10k subset of LakeQA. We are mainly evaluating on accuracy and the percentage of required datasets the agent retrieved and analyzed.
We run this on gpt 5 mini and gpt 5.4 nano, relatively small models because due to scale, frontier models are cost-prohibitive.

In this talk, we are mainly talking about LakeQA-mini, accuracy, and findings on gpt-5-mini.

First we do a per-component ablation, for instance on search: keeping plan and analyze oracle, we change the search modes around from the naive BM25 to the standard hybrid search to oracle.

Here, standard means an off-the-shelf implementation, standing in for whatever your implementation is.

On this, we found out that search and analysis are major bottlenecks.

With y-axis as accuracy and x-axis as the components, where hollow is naive and fully-opaque is oracle, the difference between naive and oracle of planning is 9.6, with search at 13.3, and analysis at 18.5.

Planning looks like a smaller bottleneck because agents do not follow 44% of their own plans.

Now, when we go through end-to-end mode ablation: for instance, naive and standard all the way.

We found out that we are still far away from the oracle. We can see that the Standard is slightly better than Naive (1.5%), and there is still a huge gap between Standard and Oracle (18.5%).

So there are still improvements that can be made to the implementation.

Some key takeaways from this paper:
(i). Careful measurement is important, so isolation is necessary as end-to-end scoring can conflate improvement contribution.
(ii). Bottlenecks are non-generalizable: they are specific to the tasks, benchmark, scale, and models; LakeQA is bottlenecked by analysis and search, while Kramabench is only bottlenecked by analysis.

We open-sourced SANA so you can:
- Identify bottlenecks in any question-answering benchmark.
- Evaluate the effectiveness of your tools for planning, search, and analysis in question-answering.

Appendix: Conversion of benchmark tasks to oracle tools.
Each subquestion in a benchmark task can be represented as a node with the source, subquestion, and intermediate answer.
(i) Extracting the source sequence gives the golden datalake for oracle search.
(ii) Extracting the subquestions into subquestions that do not leak intermediate facts or dataset names gives the oracle plan.
(iii) Execution records containing the SQL/code on the source to give the intermediate answer can be made to the oracle data analysis.

Appendix: Accuracy delta for each component on LakeQA-mini, relative to naive.
Search and data analysis are relatively big bottlenecks for both models. On planning, gpt-5-mini gains more accuracy as it follows plans better.

Preloaded: We give the agent the gold datasets at the start. This also ablates the agent's reasoning to declare what to search for and knowing when to stop searching for more datasets. Smaller models struggle much more with these bottlenecks.

Appendix: Accuracy delta for each component on Kramabench, relative to naive.
Due to smaller scale, search is not a big bottleneck for either models.
Planning and data analysis are relatively big bottlenecks on gpt-5.4-nano, while only data analysis is a big bottleneck on gpt-5-mini.

Appendix: Accuracy comparison on LakeQA 20-tasks subset, including the latest gpt-5.6-sol and gpt-5.6-luna.
Newer models achieve higher accuracy.

Appendix: Table comparison on LakeQA 20-tasks subset, including the latest gpt-5.6-sol and gpt-5.6-luna.
Newer models spend less turns per task while achieving much higher accuracy.

Appendix: Search comparison on LakeQA 20-tasks subset, including web search.
Web search achieves lower accuracy as retrieved sources are usually not tables and hence cannot have SQL/code run over them.