The story about how Claude's Web Fetch tool doesn't necessarily return the latest information
This page has been translated by machine translation. View original
Claude's Web Fetch tool sometimes returns data using cache, so don't over-trust AI research results as the latest information! That's what this article is about.
Background
I wanted to try out the Data table block (a table block that can be placed in messages) recently added to Slack's Block Kit, so I had Claude research the details of the specifications and how to implement it.
I see — it seems there are constraints such as a maximum of 101 rows and a maximum total of 10,000 characters.
However, I unfortunately hit the usage limit midway through my research, so I decided to ask ChatGPT to continue. That's when an unexpected pushback came back.
Surprisingly, the information Claude had provided was wrong. When I checked Slack's official documentation myself, ChatGPT's claim of "the limits are 201 rows and 20,000 characters" turned out to be correct.
Had Claude mistakenly referenced an outdated page?
However, even checking the links in the response showed that Claude was looking at the correct page, and no matter how many times I had it re-investigate with different models, it stubbornly insisted that "the official documentation clearly states the limit is 101 rows and 10,000 characters."
Cause
Upon investigation, the most likely cause of this phenomenon is the Web Fetch tool's cache.
Checking Anthropic's official documentation, it states that the Web Fetch tool automatically caches the content of fetched pages, and the content returned does not necessarily match the latest version at the URL[1].
The API provides a way to bypass the cache by specifying "use_cache": false, but I was unable to find a way to specify this on claude.ai (the official website chat).
On the other hand, ChatGPT's Web search tool defaults to live fetching, with explicit specification required to limit it to cache[2]. The reason ChatGPT was able to retrieve more up-to-date information this time may be due to this specification.
Countermeasures
If you want to research the latest information with Claude, where possible it would be reliable to specify "use_cache": false via the API to bypass the cache. Alternatively, as in this case, having multiple AI tools conduct a review also seems like a good approach.
Also, though it may be a common suggestion, I was reminded that the safest approach for truly important figures is to verify them with your own eyes. Outdated specification information could cause bugs during implementation and similar situations, so please be careful everyone.
