网易首页 > 网易号 > 正文 申请入驻

Trustworthy data is critical infrastructure for AI agents.

0
分享至



Author:Aurora / iDIYuan

Excutive Editor:Louis Lu / iDIYuan

Singapore’s Marina Bay Sands became the center of Asia’s latest AI frenzy as SuperAI drew tens of thousands of founders, investors and technologists from more than 150 countries, alongside more than 1,500 technology companies.

The event featured exchanges of ideas among figures including Balaji Srinivasan, MIT professor Max Tegmark, and renowned analyst Benedict Evans, as well as the 36-hour NEXT Hackathon and the Genesis startup competition, which offered up to US$2.3 million in funding. Across the venue, conversations centered on large language models, AI agents and the race to turn generative AI into products that can work reliably in the real world.

Yet a reality rarely brought into the spotlight is becoming a new bottleneck for the industry: AI models are getting smarter at a rapid pace. Their ability to access accurate, real-time information from the open internet, however, is not improving nearly as quickly — and in some cases, it is becoming harder.

Agents that perform well in laboratory environments can see their performance deteriorate once they are deployed on the live web at scale. The problem often does not lie in the models themselves, but in the increasingly sophisticated defenses being deployed across the web — from anti-scraping mechanisms and dynamic verification to content designed specifically to detect, restrict or mislead automated agents. In the process, the public web is gradually shifting from an open source of information into an increasingly adversarial environment for AI.

At this year’s conference, Ariel Shulman, chief product officer of Bright Data, a public-data infrastructure provider, put it bluntly in his keynote: “The web has learned to lie.”

While the industry’s attention remains focused on model parameters, compute scale, and reasoning capabilities, Shulman believes a more fundamental problem is being overlooked: giving AI agents reliable, real-time access to the internet — and ensuring the data they retrieve is accurate, trustworthy and continuously available.



Bright Data CPO Ariel Shulman speaking at SuperAI. Source: SuperAI

Speaking with iDIYuan after the conference, Shulman said the problem goes beyond websites learning how to block or mislead AI agents. The useful life of online data is also getting shorter, in some cases shrinking to just hours or even minutes.

At the same time, tighter compliance requirements and growing legal risks are raising the barriers for companies seeking access to public data. Without authentic, stable, and compliant data pipelines, even frontier models can be constrained by the weakest layer in the stack: their inputs.

That is turning access to trustworthy, real-time data into a new competitive battleground for the AI industry.

When the Internet Learns to Lie on Purpose,What Will Happen to AI?

In Ridley Scott’s 1982 science-fiction classic Blade Runner, the Tyrell Corporation creates replicants so lifelike that appearance alone is no longer enough to distinguish them from humans. Identifying them requires the Voight-Kampff test, which looks for subtle differences in emotional response and behavior.



Film: Blade Runner. Source: thegallery.gr

More than forty years later, a similar scenario is emerging on the open web.Major platforms are increasingly acting as testers themselves. Instead of simply blocking IP addresses, they are beginning to actively identify whether the visitor behind a request is a human user or an AI agent.

Once identified as an agent, however, the response may not be a simple access-denied page. The agent can instead be routed into a deliberately distorted version of the web — one where prices are synthetic, inventory is inaccurate, page content has been altered and search results are selectively misleading. From the agent’s perspective, the request appears to have succeeded. In reality, it has been fed corrupted inputs. That shift prompted a stark assessment from Ariel Shulman during his SuperAI presentation:

“The web has learned to lie.”

As traffic generated by LLM training and AI agents grows, Shulman argues that the relationship between the internet and AI has evolved from the old contest of “blocking versus bypassing” into a much more complex offensive-and-defensive battle. In the past, websites mainly relied on CAPTCHAs, rate limits, and IP blocking. Today, more advanced defenses are emerging, including honeypots designed to trap automated traffic and deception systems built specifically to identify, misdirect or manipulate AI agents.



Honeypot system. Source: Bright Data

For human users, these tactics may be largely invisible. But for agents that rely on web data for reasoning and decision-making, once a bad input enters the workflow, every subsequent analysis, judgment, and even action may be built on a false foundation.

Shulman described such AI-targeted “data poisoning” as a growing operational risk. He gave the example of a non-technical CEO who used Vibe Coding to develop an application that pulled hotel-pricing data from the web. The agent returned a rate of $399 and warned that inventory was running low. The actual page, however, showed rooms available for $199, with plenty of inventory remaining. The application had not failed to retrieve data. It had successfully retrieved the wrong data.

At the engineering level, this targeted deception of AI agents leads to an extremely dangerous consequence: garbage in, garbage out. Once an LLM receives precisely poisoned data, its subsequent reasoning, analysis, and even business decisions are all built on the wrong foundation.

Shulman said the bigger problem is that agents respond to deception very differently from humans. People bring intuition, context and accumulated experience to what they see online. If a hotel rate suddenly doubles within minutes, or an airline site shows inventory that makes little sense, a human user is likely to become suspicious and check another source.

AI agents, however, lack this intuitive brake. Under today’s largely deterministic workflows, an agent may simply follow its instructions: log the wrong price, propagate it across thousands of database records and, in the worst case, trigger downstream systems that send incorrect offers or pricing information to real customers at scale.

Once agents begin to participate deeply in enterprise operations, data quality stops being just an engineering problem and becomes a business risk. For AI, the greatest threat is no longer being unable to obtain data, but obtaining data that looks authentic while actually being wrong.

Breaking Through the Challenges of Deception, Freshness, and Compliance: Three Questions AI Data Infrastructure Must Answer

Founded in 2014 and headquartered in Israel, Bright Data is a major provider of public-web data infrastructure. Its role in the AI stack is less about training models than about keeping them connected to fresh, real-world information at scale.



Bright Data booth at SuperAI. Source: Photo by iDIYuan on site

Rather than a traditional web-scraping company, Bright Data increasingly positions itself as part of the data infrastructure layer for AI. Its technology stack helps LLMs, AI agents, and enterprise applications access fresh public-web data at scale, while navigating reliability and compliance requirements. As AI becomes increasingly multimodal, that data layer is also expanding beyond text to include images, video and other forms of web content.

As part of the data pipeline behind the AI ecosystem, Bright Data currently supports more than 100 million AI agent interactions every day and provides public-data services to more than 20,000 organizations worldwide, including Fortune 500 companies, leading universities, and fast-growing technology companies. According to the Bright Data Blog, the company also says 14 of the world’s 20 leading LLM labs use its data infrastructure.

As the web becomes more hostile to AI traffic, Bright Data’s approach is not to simply overpower those defenses, but to make automated access look more like ordinary human browsing. Shulman told iDIYuan that the company runs real browser environments in the cloud, replicating normal browsing, clicking and interaction patterns so that AI systems receive the same page responses and data a human user would see.

“If you have the technology to look like a real person, you will get real person information,” he said.

Underpinning that system is a network of more than 150 million compliant residential IPs across 195 countries and regions. With this infrastructure, Bright Data says it can collect more than 2.5 PB of data from the public internet every day while maintaining its compliance requirements, providing continuously updated data for LLM training, AI agents, and enterprise applications.



IP addresses across 195 countries. Source: Bright Data website

If authenticity determines whether AI can trust the data it sees, freshness determines whether that data is still useful. In Shulman’s view, “Data quality and data freshness are two sides of the same coin.” Data that is no longer fresh, he explained, is no longer quality data.

Consider an e-commerce customer-service bot or a logistics agent. If it is working with last month’s prices, inventory, or delivery information, the information may be accurate but still useless for decisions being made today. Shulman’s point is that access alone is not enough. For data to remain valuable to AI systems, it also has to stay current.

Bright Data’s public-data decay map illustrates just how quickly that window can close. Fast-moving information such as social-media trends, e-commerce prices and product inventory can become stale within hours, and often within a day. After that, its usefulness to an AI system can deteriorate quickly.



Shelf life of different types of data. Source: Bright Data

Shulman gave the example of an AI agent that helps users compare prices globally. If it relies on prices and inventory data from 12 hours earlier, it may ultimately recommend a product that has already increased in price or even sold out. When the user reaches the checkout page, the real price may be completely different from the result provided by the agent. For the user, this is not only a poor experience; it directly weakens trust in the AI system.

As agents move deeper into search, procurement, customer service, and operations, the data challenge is shifting from access to real-time access. For companies deploying AI in the real world, securing a continuous supply of trustworthy, up-to-date data may prove harder than simply scaling model parameters.

If authenticity and freshness determine whether data is useful, compliance determines whether a data business can operate over the long term. As demand for data from LLMs and AI agents continues to grow, the debate over public data collection is gradually shifting from a technical issue to a legal one.

Disputes over access rights, platform controls and the ownership of public data have intensified in recent years, making compliance an increasingly important competitive advantage for AI infrastructure providers.

On compliance, Shulman told iDIYuan Bright Data follows a simple rule: it only collects data that is publicly accessible on the web.

In practice, that means information that can be viewed without logging into an account or paying for a subscription. “If you can see that, that’s probably good,” Shulman said. Data behind personal accounts, internal enterprise systems, bank accounts or other non-public sources falls outside that boundary. That approach has also been tested in court. Shulman said Bright Data has faced lawsuits in the U.S. from Meta and X over its collection of publicly available web data. The company’s position, he said, was that it accessed only public information, not private data — and that the rulings in those cases supported that distinction.



Judgment related to Meta’s case against Bright Data. Source: courtlistener.com

In January 2024, the U.S. District Court for the Northern District of California granted summary judgment to Bright Data on Meta’s breach-of-contract claim, finding that the company had scraped only publicly available data while logged out of Meta’s services and, on those facts, had not violated Meta’s terms. The ruling was specific to the contractual claims and facts before the court; it did not declare public-web scraping universally lawful.

For Bright Data, however, it was an important validation of its approach to collecting public data. “That is really important,” Shulman said.

More broadly, the case shows that the legal boundaries around public-web data can depend heavily on how the data is collected, what information is accessed and the contractual terms involved. As AI companies consume more public data, those distinctions are becoming increasingly important.

MCP Is Only the Starting Point: Data Services Compete on Scalability and Engineering Efficiency

After solving the problems of data authenticity, freshness, and compliance, the next challenge for AI engineers is how to enable agents to connect efficiently to external tools and data services. Over the past year, Anthropic’s Model Context Protocol, or MCP, has rapidly become a major focus in the industry. As more model providers and development platforms add support, MCP is becoming an important connection layer in the agent ecosystem.

Shulman told iDIYuan that Bright Data released its MCP server about a year ago. In his description, an MCP server is essentially the connection layer between Bright Data’s services and AI: it gives an agent a list of the available Bright Data tools, tells it what those tools can do, and shows it how to use them.

He gave a simple example. If an AI agent says it needs to search Google, the MCP server can tell it that Bright Data has a SERP search tool and provide the syntax and instructions needed to call it. The agent can then use that information to complete the task automatically.

In Ariel’s view, MCP’s greatest value is that it makes it much easier for an agent to discover and use external tools. It is especially useful at the prototype and proof-of-concept stages, when developers are still working out how an application should obtain data and which tools it needs.



Relationship between Bright Data’s data infrastructure and the layers above and below it. Source: Bright Data

However, Shulman also pointed out the limits of MCP at scale. If an agent needs to find the price of a single product, MCP can tell it how to do so. But if that task expands to one million products, he said, using MCP becomes inefficient because “there is a lot of overhead.” In large-scale production workloads, those additional layers can translate into unnecessary latency and cost.

Shulman described what he sees as the most practical path:

1. Start with MCP to understand how Bright Data’s tools work and help the AI agent discover a workable way to complete the task;

2. Once the workflow is clear and the application needs to scale, switch to more direct methods such as direct APIs, and eventually stop using MCP for the production workload.

In other words, MCP helps developers “find the road,” while direct APIs turn that road into a highway. As agents move from demos into production, the industry’s focus is also shifting from “can it be achieved?” to “can it run at scale?” Engineering efficiency will ultimately become a key factor in determining whether a product succeeds or fails.

This pursuit of engineering efficiency and scalability is reflected not only in external data services, but also in how this data-infrastructure company organizes itself. As a data-infrastructure provider, Bright Data must process massive volumes of data requests every day while continuously responding to changes in the internet environment. That business model means that its competitiveness comes not only from the technology itself, but also from its ability to iterate continuously and deliver quickly.

Shulman told iDIYuan that Bright Data, with a lean team of roughly 500 employees, generates hundreds of millions of dollars in revenue and has remained profitable since 2017. In his view, corporate development does not depend entirely on capital investment; what matters more is whether a company can build an efficient engineering system.

“We have a very unique way of working,” Shulman said. For example, as long as engineers meet the required standards, they can push code directly into the production environment. After a new feature goes live, the system first exposes it to roughly 1% of users and continuously monitors its performance. If it runs stably, the rollout is gradually expanded to 5%, then 10%, and eventually to all users.

This continuous-delivery mechanism enables Bright Data to maintain an extremely high iteration frequency. Shulman revealed that the company averages around 60 software releases per day. In his view, data infrastructure is fundamentally a long-term engineering competition. The real advantage comes not from token volume or access to capital, but from building systems that can iterate quickly, improve continuously and respond to change.

From “Bright Data” to Its China Brand: Bright Data’s Localization Strategy in China

No matter how technology architectures evolve, infrastructure ultimately has to prove its value in specific markets and industrial ecosystems. For the data-services industry, this means not only exporting technology, but also sustaining long-term localized operations.

At a time when some multinational companies are scaling back their operations in China, Bright Data has chosen another path: continuing to invest in the Chinese market under its localized brand and seeking to further combine its global data-infrastructure capabilities with the needs of China’s domestic AI industry.



Bright Data China website: bright.cn. Source: Bright Data

In the interview, Shulman placed particular emphasis on Bright Data’s position in China. “We have a very strong presence in China,” he said.

To support localized operations, Bright Data has established offices in Beijing, Shanghai, and Shenzhen, with R&D, technical-support, and customer-service teams. Public recruitment information shows that its China team covers multiple roles including solutions architects, customer success managers, and senior sales managers, spanning the full chain from technical consulting to long-term customer operations.

For domestic AI companies, this means gaining not only access to a global data-service capability, but also a service system capable of providing local support. Whether the issue involves complex data-collection requirements, network-scheduling problems, or technical failures in production environments, customers can obtain more timely responses and support through local teams.

Bright Data is also continuing to expand its local partnership ecosystem in China. In April 2026, it partnered with Data Trading Network to bring its public-data capabilities to the domestic market. The datasets can be updated on daily, weekly and monthly cycles, helping companies lower the cost of sourcing and maintaining data.

The company has also teamed up with DuoPlus Cloud Phone to explore how data services and cloud-device infrastructure can support cross-border expansion. On the developer side, Bright Data has co-hosted technical events with communities including DataFun, where its China engineering team has shared practical experience with MCP, agent tool calling and data-infrastructure development.



Chinese edition of a Bright Data report. Source: Bright Data

Its local partnerships have also helped Bright Data move beyond simple market expansion and gradually integrate into the local AI ecosystem. In Shulman’s view, Chinese companies targeting global markets face many of the same needs as their peers elsewhere, including access to global data for model training. But once they focus on the domestic market, localization becomes far more important.

Shulman revealed that for data-collection needs involving local retailers in different countries, the company’s AI agents can automatically generate customized scraping solutions, extract multilingual and multimodal structured content from massive amounts of public data, and directly serve localized business decisions such as market analysis and competitive monitoring.

Over the past two years, Bright Data’s AI customer base in Asia-Pacific has doubled annually and now accounts for nearly one-third of the company’s global AI customers. The region represents more than 20% of global AI spending, while China and India together make up roughly 60% of Bright Data’s AI customers in Asia-Pacific. This regional presence allows Bright Data to provide more targeted local data services to companies across markets including Japan, Southeast Asia, India and China.

The Next Generation of Data-Infrastructure Competition Is Already Quietly Unfolding

At the end of the interview, Shulman turned to what he sees as the next phase of the AI industry. “I think one day every company will have its own AI agents,” he said.

A human analyst might have needed to open only five or ten links to complete a research task. An AI agent, by contrast, could scan dozens or even hundreds of websites to produce a more comprehensive answer. That means “more data to handle,” Shulman said — and potentially a need to rethink the infrastructure that supports it.



Ariel Shulman discussing Bright Data’s vision. Source: SuperAI

While the industry remains focused on model parameters, reasoning capabilities, and investment in compute, another, more fundamental layer of competition is already quietly unfolding. As more agents connect to the internet and complete tasks autonomously, data is no longer just fuel for training. It is becoming a real-time input that keeps AI systems running. Authenticity determines whether an agent can form an accurate understanding; freshness determines whether its decisions remain effective; compliance determines whether this capability can exist over the long term; and the engineering system determines whether the entire system can operate at scale. As Shulman put it, once the internet learns to “lie,” trustworthy data will become new infrastructure for the agent era. The next phase of the AI race may be less about pushing model capability alone and more about building the data systems that allow agents to operate reliably in the real world. Algorithms shape how AI reasons; data infrastructure shapes what it can see, understand and act on. The model may be the brain and the agent the executor, but neither is particularly useful without a reliable connection to the outside world. As AI agents increasingly browse the web, make decisions and act on behalf of users, that underlying data layer is likely to become only more important.

特别声明:以上内容(如有图片或视频亦包括在内)为自媒体平台“网易号”用户上传并发布,本平台仅提供信息存储服务。

Notice: The content above (including the pictures and videos if any) is uploaded and posted by a user of NetEase Hao, which is a social media platform and only provides information storage services.

相关推荐
热点推荐
真被马斯克说中?全球争抢的不是芯片,而是中国20万一台的变压器

真被马斯克说中?全球争抢的不是芯片,而是中国20万一台的变压器

小莜读史
2026-11-07 17:58:29
9人得分入账,辽宁男篮杯赛迎开门红,鄢手骐复出17+9,姜宇星16+6+6全能

9人得分入账,辽宁男篮杯赛迎开门红,鄢手骐复出17+9,姜宇星16+6+6全能

替补席看球
2026-11-18 21:51:48
大意外!卞相壹难挡白棋攻势,李钦诚抢占先机夺三星杯赛点!

大意外!卞相壹难挡白棋攻势,李钦诚抢占先机夺三星杯赛点!

L76号
2026-11-18 14:58:57
到了高中,你会发现:考700分需要顶级天赋,考600分需要一般天赋,考550分不需要天赋

到了高中,你会发现:考700分需要顶级天赋,考600分需要一般天赋,考550分不需要天赋

好爸育儿
2026-11-13 08:33:19
每体:莱万温暖致意冲场波兰球迷,除合影之外还送出球衣

每体:莱万温暖致意冲场波兰球迷,除合影之外还送出球衣

懂球帝
2026-11-18 21:50:11
1927年,毛泽东被捕后,在被押解去执行死刑的途中,他突然对一名团丁说:“老哥,我腿疼,可能伤筋了!”

1927年,毛泽东被捕后,在被押解去执行死刑的途中,他突然对一名团丁说:“老哥,我腿疼,可能伤筋了!”

扶苏聊历史
2026-11-06 18:18:35
张子宇一生之敌?日本归化中锋单场59分33板太恐怖 但性别引发争议

张子宇一生之敌?日本归化中锋单场59分33板太恐怖 但性别引发争议

大嘴爵爷侃球
2026-11-18 10:00:33
罗纳尔多选历史最佳阵:C罗、梅西和自己搭档三叉戟,门将布冯

罗纳尔多选历史最佳阵:C罗、梅西和自己搭档三叉戟,门将布冯

懂球帝
2026-11-18 17:18:05
多所高校要求师生在论文发表前完成备案,否则不纳入科研成果

多所高校要求师生在论文发表前完成备案,否则不纳入科研成果

澎湃新闻
2026-11-18 12:14:07
交锋15连胜!尼克斯逆转猛龙 布伦森31分关键3+1唐斯16+16+8

交锋15连胜!尼克斯逆转猛龙 布伦森31分关键3+1唐斯16+16+8

醉卧浮生
2026-11-18 10:57:07
新加坡总理黄循财已到中国深圳!APEC4天访华行程曝光

新加坡总理黄循财已到中国深圳!APEC4天访华行程曝光

新加坡万事通
2026-11-18 20:05:06
中国与日本有着不共戴天之仇,为何还有140多万华人居住在日本?

中国与日本有着不共戴天之仇,为何还有140多万华人居住在日本?

文史达观
2026-11-18 06:45:19
又是0,央5直接弃播,杨瀚森,真是太烧心了……

又是0,央5直接弃播,杨瀚森,真是太烧心了……

体育新角度
2026-11-18 21:51:45
全民拒接陌生电话,运营商、快递员、疾控中心为此付了多少成本?

全民拒接陌生电话,运营商、快递员、疾控中心为此付了多少成本?

小鹿姐姐情感说
2026-11-18 03:24:23
为什么格莱姆斯状态这么猛?必须要感谢三个人

为什么格莱姆斯状态这么猛?必须要感谢三个人

球毛鬼胎
2026-11-18 18:45:13
迪拜首富之女重病赴华寻求中医,中医扎五根银针,开口便让他们傻眼

迪拜首富之女重病赴华寻求中医,中医扎五根银针,开口便让他们傻眼

萧竹轻语
2025-10-13 11:40:11
发现中国一个奇怪的现象:只要有公婆帮扶的家庭,儿媳都忙着回公婆家;而公婆不帮的家庭,儿媳早已断联!

发现中国一个奇怪的现象:只要有公婆帮扶的家庭,儿媳都忙着回公婆家;而公婆不帮的家庭,儿媳早已断联!

心理观察局
2026-08-01 07:10:30
黄埔一楼盘打6折拍卖大批车位,仅需3万起

黄埔一楼盘打6折拍卖大批车位,仅需3万起

老覃讲历史
2026-11-18 14:32:17
老黄事件后,美开始全面封堵追踪芯片、包括云算力出租,并建圈子

老黄事件后,美开始全面封堵追踪芯片、包括云算力出租,并建圈子

邵旭峰域
2026-11-18 13:40:14
除夕夜晚上,亲戚轮番挖苦我,我没闹直接取消二叔的升职,他们懵了

除夕夜晚上,亲戚轮番挖苦我,我没闹直接取消二叔的升职,他们懵了

红豆讲堂
2025-05-05 11:25:03
2026-11-18 22:43:00
数据猿DataYuan incentive-icons
数据猿DataYuan
数据智能产业创新服务媒体
3059文章数 623关注度
往期回顾 全部

教育要闻

2026年留学趋势复盘:国内教育与国际教育生源争夺战升级

头条要闻

追上特朗普专机 马斯克又来了

头条要闻

追上特朗普专机 马斯克又来了

体育要闻

“中超卡里乌斯”,成了大连的夺冠英雄

娱乐要闻

主持人张泉灵被曝猛料,公司成老赖

财经要闻

黄仁勋留下的高分成绩单,能延续吗?

科技要闻

离职后首份声明!黄仁勋认错愿回归

汽车要闻

全新红旗H9最有意思的地方,是它开始有自己的豪华逻辑了

态度原创

手机
游戏
艺术
亲子
公开课

手机要闻

华为引爆阔直板手机!友商纷纷下场,但很可能会集体踏空?

PC Gamer试玩GTA6百小时:双主角出色但任务重复

艺术要闻

深邃宁静的风景油画,奥地利著名风景画家弗舍尔

亲子要闻

当婴配市场进入存量博弈,有机赛道如何破局突围?

公开课

李玫瑾:为什么性格比能力更重要?

无障碍浏览 进入关怀版