Study finds AI agents struggle to extract pricing data from B2B websites due to technical and structural barriers
Research indicates that AI agents frequently fail to extract pricing data from B2B websites due to technical barriers, complex page structures, and access restrictions.
1. AI Agent Challenges on B2B Websites
A study conducted by Kevin Indig and David Kaufman of Siteline analyzed how AI agents interact with 100 B2B websites to retrieve information on pricing, integrations, and security. While most sites are generally accessible, the research identified a significant "breaking point" regarding pricing. AI agents frequently struggle to extract pricing data directly from vendor websites, often defaulting to third-party sources like directories and editorial blogs. This failure occurs even when pricing information is publicly available on the vendor's site.
2. Primary Causes of Agent Failure
The research categorized the reasons agents fail to retrieve first-party data into three distinct modes: opacity, machine-readability, and access friction. Pricing opacity refers to vague or undisclosed pricing, which forces agents to look elsewhere. Machine-readability issues arise when data is present but difficult for agents to parse due to complex page structures, JavaScript-heavy rendering, or the use of non-crawlable elements like PDFs and calculators. Access friction involves technical barriers such as rate limits, blocking, or slow page performance, which significantly increase the cost and time required for an agent to successfully retrieve information.
3. Strategies for Agent-Proofing Websites
To ensure AI agents can reliably extract and cite information directly from a brand's website, the study recommends several technical and content-based optimizations. Companies should disclose clear, text-based pricing for all self-serve tiers and maintain a single, canonical URL for pricing information. To improve machine-readability, businesses should utilize crawlable HTML and implement schema.org Product and Offer markup. Finally, site owners should ensure that AI crawlers are permitted in robots.txt files and that pricing pages are optimized for speed and accessibility to minimize the likelihood of agents defaulting to third-party aggregators.