"The legislation seeks to hold AI companies to account for illicit scraping and failing to comply with publishers' "no crawling" specifications in their robots.txt files, which are notoriously difficult to enforce - non compliance poses pretty devastating consequences for publishers. Tollbit's latest report, out this week, highlights the sprawling ecosystem of third-party web scrapers that has sprung up - built to feed AI and enterprise developers, often operating around paywalls, bypassing web controls and fueling legal and technical conflicts."
""Ultimately, we believe this to be an absolutely critical issue to the future of the free and open internet, which is funded by an ad-supported model, largely. No industry can survive by giving away its goods for free," said Michael Hahn, evp and general counsel, at the IAB and IAB Tech Lab. Hahn cited recent industrywide increases in AI bot traffic to publishers' sites, and decreases in human traffic as more people use AI search tools to find information, leading to fewer clicks to websites."
The IAB proposed legislation aims to stop AI bots from freely harvesting online content and to hold AI companies accountable for illicit scraping and ignoring publishers' robots.txt "no crawling" directives. Robots.txt rules are difficult to enforce, and widespread noncompliance can cause severe harm to publishers' ad-supported revenue models. Tollbit documents a sprawling ecosystem of nearly 40 third-party scraper APIs built to feed AI and enterprise developers that operate around paywalls and bypass web controls. The proposal cannot compel compliance as a trade body measure, but it could elevate legal and reputational risks and challenge AI firms' fair-use defenses and claims of unjust enrichment.
Read at Digiday
Unable to calculate read time
Collection
[
|
...
]