LinkedIn introduces Northguard and Xinfra to replace Kafka for scalable log storage

LinkedIn today launched Northguard and Xinfra, two new internal infrastructure systems designed to address the growing challenges of log storage and messaging at massive scale.

The offerings mark a significant evolution from the company’s longstanding use of Apache Kafka, the distributed event streaming platform it originally developed and open-sourced. Though LinkedIn has used Kafka for 15 years, the Microsoft Corp.-owned company says that as use cases have become more demanding, it became increasingly difficult to scale up and operate Kafka, hence the need for new alternatives.

The first offering, Northguard, is a new log storage system engineered for scalability, high throughput and operational efficiency. Unlike Kafka, which began to show limitations as LinkedIn scaled to more than 1.2 billion members and tens of petabytes of data daily, Northguard offers architectural improvements.

They include decentralized metadata management, dynamic log striping and enhanced durability. Northguard eliminates common scaling and load balancing issues seen in traditional Kafka deployments by sharding data and metadata and using lightweight units of replication known as segments.

Northguard is designed to automatically balance loads through segment-level replication to address resource skew challenges that arose from Kafka’s partition-based replication model. It does so by storing metadata in a fault-tolerant, sharded system backed by the Raft consensus protocol, with support for advanced storage policies that allow fine-grained control over data placement, replication and retention.

To support infrastructure-wide virtualization and simplify transitions between Kafka and Northguard, LinkedIn also unveiled Xinfra, a virtualized Pub/Sub layer that abstracts physical cluster boundaries. Xinfra allows producers and consumers to interact with virtual topics spanning both Kafka and Northguard clusters. The idea is to make topic migration transparent, with Xinfra supporting features such as dual writes during migration, epoch-based ordering and unified client application programming interfaces.

Though formally announced only today, Xinfra clients are already powering more than 90% of LinkedIn’s applications, allowing the company to migrate thousands of topics, including mission-critical workloads, from Kafka to Northguard without downtime. Xinfra’s metadata service supports operations across virtual clusters, using MySQL and Vitess for storage and Couchbase for low-latency caching. It also handles consumer group management through Apache ZooKeeper.

The introduction today is just the beginning for both offerings, with LinkedIn writing in a blog post that “looking ahead, our focus will be on driving even greater adoption of Northguard and Xinfra, adding features such as auto-scaling topics based on traffic growth and enhancing fault tolerance for virtualized topic operations.”

Image: SiliconANGLE/Reve