AI brokers run on file programs utilizing commonplace instruments to navigate directories and browse file paths.
The problem, nonetheless, is that there’s a lot of enterprise knowledge in object storage programs, notably Amazon S3. Object shops serve knowledge via API calls, not file paths. Bridging that hole has required a separate file system layer alongside S3, duplicated knowledge and sync pipelines to maintain each aligned.
The rise of agentic AI makes that problem even more durable, and it was affecting Amazon’s personal means to get issues carried out. Engineering groups at AWS utilizing instruments like Kiro and Claude Code stored working into the identical drawback: Brokers defaulted to native file instruments, however the knowledge was in S3. Downloading it domestically labored till the agent’s context window compacted and the session state was misplaced.
Amazon’s reply is S3 Recordsdata, which mounts any S3 bucket straight into an agent’s native atmosphere with a single command. The information stays in S3, with no migration required. Below the hood, AWS connects its Elastic File System (EFS) know-how to S3 to ship full file system semantics, not a workaround. S3 Recordsdata is offered now in most AWS Areas.
“By making knowledge in S3 instantly accessible, as if it is a part of the native file system, we discovered that we had a very huge acceleration with the flexibility of issues like Kiro and Claude Code to have the ability to work with that knowledge,” Andy Warfield, VP and distinguished engineer at AWS, instructed VentureBeat.
The distinction between file and object storage and why it issues
S3 was constructed for sturdiness, scale and API-based entry on the object stage. These properties made it the default storage layer for enterprise knowledge. However in addition they created a basic incompatibility with the file-based instruments that builders and brokers rely upon.
“S3 is just not a file system, and it would not have file semantics on an entire bunch of fronts,” Warfield mentioned. “You possibly can’t do a transfer, an atomic transfer of an object, and there aren’t truly directories in S3.”
Earlier makes an attempt to bridge that hole relied on FUSE (Filesystems in USErspace), a software program layer that lets builders mount a customized file system in person area with out altering the underlying storage. Instruments like AWS’s personal Mount Level, Google’s gcsfuse and Microsoft’s blobfuse2 all used FUSE-based drivers to make their respective object shops appear like a file system.
Warfield famous that the issue is that these object shops nonetheless weren’t file programs. These drivers both faked file habits by stuffing additional metadata into buckets, which broke the article API view, or they refused file operations that the article retailer could not assist.
S3 Recordsdata takes a unique structure solely. AWS is connecting its EFS (Elastic File System) know-how on to S3, presenting a full native file system layer whereas maintaining S3 because the system of document. Each the file system API and the S3 object API stay accessible concurrently towards the identical knowledge.
How S3 Recordsdata accelerates agentic AI
Earlier than S3 Recordsdata, an agent working with object knowledge needed to be explicitly instructed to obtain recordsdata earlier than utilizing instruments. That created a session state drawback. As brokers compacted their context home windows, the document of what had been downloaded domestically was usually misplaced.
“I’d discover myself having to remind the agent that the info was accessible domestically,” Warfield mentioned.
Warfield walked via the before-and-after for a typical agent activity involving log evaluation. He defined {that a} developer was utilizing Kiro or Claude Code to work with log knowledge, within the object solely case they would want to inform the agent the place the log recordsdata are positioned and to go and obtain them. Whereas if the logs are instantly mountable on the native file system, the developer can merely establish that the logs are at a selected path, and the agent instantly has entry to undergo them.
For multi-agent pipelines, a number of brokers can entry the identical mounted bucket concurrently. AWS says hundreds of compute assets can connect with a single S3 file system on the similar time, with combination learn throughput reaching a number of terabytes per second — figures VentureBeat was not in a position to independently confirm.
Shared state throughout brokers works via commonplace file system conventions: subdirectories, notes recordsdata and shared mission directories that any agent within the pipeline can learn and write. Warfield described AWS engineering groups utilizing this sample internally, with brokers logging investigation notes and activity summaries into shared mission directories.
For groups constructing RAG pipelines on prime of shared agent content material, S3 Vectors — launched at AWS re:Invent in December 2024 — layers on prime for similarity search and retrieval-augmented technology towards that very same knowledge.
What analysts say: this isn’t only a higher FUSE
AWS is positioning S3 Recordsdata towards FUSE-based file entry from Azure Blob NFS and Google Cloud Storage FUSE. For AI workloads, the significant distinction is just not primarily efficiency.
“S3 Recordsdata eliminates the info shuffle between object and file storage, turning S3 right into a shared, low-latency working area with out copying knowledge,” Jeff Vogel, analyst at Gartner, instructed VentureBeat. “The file system turns into a view, not one other dataset.”
With FUSE-based approaches, every agent maintains its personal native view of the info. When a number of brokers work concurrently, these views can doubtlessly fall out of sync.
“It eliminates a complete class of failure modes together with unexplained coaching/inference failures attributable to stale metadata, that are notoriously troublesome to debug,” Vogel mentioned. “FUSE-based options externalize complexity and points to the person.”
The agent-level implications go additional nonetheless. The architectural argument issues lower than what it unlocks in observe.
“For agentic AI, which thinks by way of recordsdata, paths, and native scripts, that is the lacking hyperlink,” Dave McCarthy, analyst at IDC, instructed VentureBeat. “It permits an AI agent to deal with an exabyte-scale bucket as its personal native onerous drive, enabling a stage of autonomous operational pace that was beforehand bottled up by API overhead related to approaches like FUSE.”
Past the agent workflow, McCarthy sees S3 Recordsdata as a broader inflection level for a way enterprises use their knowledge.
“The launch of S3 Recordsdata is not simply S3 with a brand new interface; it is the elimination of the ultimate friction level between large knowledge lakes and autonomous AI,” he mentioned. “By converging file and object entry with S3, they’re opening the door to extra use circumstances with much less remodeling.”
What this implies for enterprises
For enterprise groups which have been sustaining a separate file system alongside S3 to assist file-based purposes or agent workloads, that structure is now pointless.
For enterprise groups consolidating AI infrastructure on S3, the sensible shift is concrete: S3 stops being the vacation spot for agent output and turns into the atmosphere the place agent work occurs.
“All of those API modifications that you just’re seeing out of the storage groups come from firsthand work and buyer expertise utilizing brokers to work with knowledge,” Warfield mentioned. “We’re actually singularly centered on eradicating any friction and making these interactions go in addition to they’ll.”
