Technology

LinkedIn Unveils Revolutionary Northguard and Xinfra: The Future of Scalable Log Storage

2025-06-25

Author: William

LinkedIn’s Game-Changing Announcement

In a groundbreaking move, LinkedIn has officially launched Northguard and Xinfra, two innovative infrastructure systems aimed at tackling the complexities of log storage and messaging as they hit massive heights.

The Shift from Kafka to New Horizons

After a remarkable 15 years of relying on Apache Kafka, the platform they pioneered, the tech giant has recognized the growing challenges posed by skyrocketing data demands. As their user base has expanded to over 1.2 billion members and their daily data influx surpasses tens of petabytes, the limitations of Kafka became apparent, prompting the pursuit of more robust alternatives.

Introducing Northguard: The Future of Log Storage

Northguard is more than just a log storage system; it’s engineered for unparalleled scalability and operational efficiency. Unlike Kafka, which struggled under the weight of LinkedIn’s vast operations, Northguard boasts cutting-edge features such as decentralized metadata management and dynamic log striping. This means enhanced durability and the elimination of common scaling problems that have plagued traditional Kafka setups.

Smarter Load Balancing and Resource Management

One of Northguard's standout features is its automatic load balancing through advanced segment-level replication. By utilizing a fault-tolerant, sharded system based on the Raft consensus protocol, Northguard provides precise control over data placement, replication, and retention.

Xinfra: Bridging the Gap Between Worlds

To complement Northguard, LinkedIn has also introduced Xinfra, a virtualized Pub/Sub layer that simplifies the transition from Kafka to Northguard. It allows seamless interaction with virtual topics that stretch across both environments, ensuring that topic migration is virtually invisible to users.

Driving Innovation with Advanced Features

Despite being newly announced, Xinfra has already integrated with over 90% of LinkedIn’s applications, facilitating the migration of thousands of critical topics from Kafka to Northguard without any downtime. It leverages a robust backend, employing MySQL, Vitess, and Couchbase to deliver low-latency caching and comprehensive consumer group management.

Looking Ahead: The Future of LinkedIn's Infrastructure

As LinkedIn forges ahead with the deployment of Northguard and Xinfra, they’re committed to enhancing these systems. Future features will likely include auto-scaling capabilities for topics based on traffic demands, and improved fault tolerance for virtualized operations.

This exciting new chapter not only redefines LinkedIn's infrastructure but also signals a broader shift in how tech giants manage their data workloads in a rapidly evolving digital landscape.