Revolutionizing AI Processing with JetStream I/O Cards

AI chip startup d-Matrix is making waves in the tech world by introducing its groundbreaking JetStream I/O cards, defined to scale AI operations across multiple servers and even entire racks with minimal performance hitches. This innovation is set to redefine how AI models are distributed, paving the way for larger and more complex computations.

A Closer Look at JetStream's Capabilities

At first glance, the JetStream I/O cards may appear as just another standard PCIe 5.0 Network Interface Card (NIC), boasting dual ports at 200 Gb/s or a powerful single port at 400 Gb/s, all while utilizing conventional Ethernet technology. But a deeper dive reveals that d-Matrix has crafted these NICs using field programmable gate arrays (FPGAs) to achieve the remarkable feat of reducing network latency to just two microseconds—an impressive improvement over off-the-shelf alternatives.

Power Meets Performance with Corsair AI Accelerators

When paired with up to eight of d-Matrix's formidable 600-watt Corsair AI accelerators, these NICs create a formidable architecture capable of monumental processing power. Each Corsair unit delivers an astonishing 2.4 petaFLOPS using MXINT8 data type and an incredible 9.6 petaFLOPS with MXINT4 precision. This setup ensures an optimal performance landscape for AI-powered workloads.

A Memory Hierarchy Like No Other

Diving into the memory structure, d-Matrix's design integrates high-speed SRAM and spacious LPDDR5 memory, typically found in top-tier laptops. Each Corsair card boasts 2GB of ultra-fast SRAM with a staggering speed of 150 TB/s, alongside 256GB of LPDDR5 capable of 400 GB/s. To put this into perspective, a single Nvidia B200 only provides 180GB of HBM3e and 8TB/s of bandwidth.

Flexible Memory Options for AI Models

With memory demands varied, d-Matrix offers customers the flexibility to choose between speed and cost when running their models. For instance, while 2TB of LPDDR5 allows for handling multi-trillion parameter models at 4-bit precision, d-Matrix recommends that for maximum speed, running models entirely in SRAM is essential.

Scalable Performance Across Many Racks

Boasting 16GB of on-chip memory per node, d-Matrix ensures efficient multi-billion parameter model performance. With a configuration of eight nodes per rack, their systems can tackle models up to 200 billion parameters when using MXINT4 precision, and d-Matrix is preparing for further scaling capacity across multiple racks.

The Future of AI Inference is Here

The JetStream I/O cards strategically employ tensor, expert, data, and pipeline parallelism, resulting in high-performance compute clusters. This approach is akin to an assembly line, processing model information systematically across each node.

Competing in the High-Performance Arena

In the landscape of AI chip startups, d-Matrix faces competition from notable players like Groq and Cerebras, who have also embraced distributed models to achieve outstanding inference performance. While Groq's approach utilized 576 processing units to tackle complex models, d-Matrix claims its design achieves generation latencies as low as 2 ms per token.

Looking Ahead: A Stellar Roadmap for Innovation

While d-Matrix is pushing boundaries with JetStream, industry giants like Nvidia and AMD are also upping their game. d-Matrix is preparing to release its upcoming Raptor family of accelerators, promising significant enhancements including 3D stacked SRAM and advanced networking capabilities, ultimately transforming rack-scale computations.

The Up-and-Coming JetStream I/O Cards

Currently sampling to customers, d-Matrix anticipates full production of JetStream I/O cards to ramp up by year’s end, solidifying its position in the competitive AI chip arena. Brace yourselves—transformative changes are just on the horizon!