On March 17, 2025, Google Cloud unveiled a groundbreaking feature—the hierarchical namespace (HNS) in Cloud Storage—designed to elevate the efficiency of AI and machine learning (ML) tasks by transforming data management.

AI and ML endeavors often require meticulous checkpointing to save ongoing model states. Traditional flat storage systems can turn this process into a laborious nightmare, assigning a unique rewrite or delete task to each object during folder renames. Enter HNS: a revolutionary approach that allows atomic folder-level operations, making checkpointing processes up to 20 times faster and far more reliable, as revealed in recent Google benchmarking.

The new RenameFolder API is the powerhouse behind this leap in performance, enabling speedy metadata-only operations that dramatically reduce administrative overhead. According to real-world data, the impact is substantial. For example, AssemblyAI experienced a mind-blowing 10-fold increase in throughput when leveraging HNS alongside Cloud Storage FUSE, culminating in an astonishing 15 times faster model training.

But that's not all! HNS takes performance to the next level by optimizing storage layouts, allowing higher queries per second (QPS) for reading and writing operations. In environments with extensive clusters, synchronized I/O can often bottleneck workflows, but hierarchical namespace buckets effectively eliminate these hurdles by delivering up to eight times the QPS of their flat counterparts.

As Jason Stevens, Senior Director of Engineering at Google, aptly stated: "Google Cloud Storage (GCS) Hierarchical Namespace (HNS) accelerates storage workloads that rely on filesystem semantics, enhancing overall efficiency for AI projects. With speeds up to 20 times faster during checkpointing and QPS improvements of up to 8 times, HNS is a key player in maximizing GPU and TPU usage for AI/ML pipelines."

For those eager to harness this revolutionary feature, be advised: HNS must be activated during the bucket creation process—it’s not retrofittable. Using the gcloud CLI, simply execute the command: gcloud storage buckets create with the -enable-hierarchical-namespace flag, ensuring your bucket is primed for the future of AI and ML. Alternatively, in the Google Cloud Console, navigate through the Cloud Storage menu and select 'Create bucket,' remembering to enable HNS in the Advanced settings.

Once activated, expect a seamless transition to enhanced AI and machine learning capabilities, complete with filesystem-like organization, atomic renames, and skyrocketed read and write throughput.