AI Firms Gain 40% More Inference Efficiency With Microsoft's New Liquid-Cooled Maia 300

Microsoft has officially unveiled the Azure Maia 300, a custom-built artificial intelligence accelerator designed to support internal and partner workloads at a multi-gigawatt scale, according to Neowin. The company stated that this new silicon will gradually become its primary source of inference capacity, aiming to challenge established competitors in the high-performance computing market.

The Maia 300 is optimised specifically for inference tasks, with Microsoft estimating it can process 40% more tokens per megawatt than leading GPU-based systems, likely referencing Nvidia products, according to SiliconANGLE. This efficiency gain is critical for organisations managing large-scale AI deployments, where power consumption and thermal management are significant operational constraints.

What we know

The Maia 300 features a complex multichiplet architecture comprising four computing chiplets fabricated using Taiwan Semiconductor Manufacturing Co.’s latest two-nanometre process node. These computing units are paired with eight HBM4 memory modules, providing a combined capacity of 288 gigabytes. This memory amount matches that of Nvidia’s flagship Rubin accelerator, although Microsoft notes the Maia 300 has approximately 20% less memory bandwidth.

To mitigate the performance penalties associated with data movement, Microsoft employed a dataflow architecture in the Maia 300 design. The system aims to minimise the constant shifting of data between computing and memory circuits by keeping information in an internal SRAM cache whenever possible, according to SiliconANGLE. This approach reduces latency and improves overall processing throughput for AI models.

When accessing high-bandwidth memory is unavoidable, the Maia 300 directs data to physically nearby memory stacks rather than those on the opposite end of the chip. Additionally, the accelerator compresses data into lightweight formats such as MXFP4. Microsoft executives Rani Borkar and Saurabh Dighe explained that these narrower representations reduce memory footprint and data movement, enabling support for larger models and longer contexts within the same power envelope.

The accelerator’s core components are supported by a device control chiplet that manages power flow, collects telemetry, and handles other supporting tasks. Two additional auxiliary chiplets oversee the movement of data in and out of the accelerator. Microsoft intends to deploy these units in trays of four accelerators, which are then installed in liquid-cooled racks containing a total of 96 accelerators.

Defender's checklist

• Monitor vendor announcements for custom silicon that may alter existing hardware compatibility requirements.
• Review data centre power and cooling capacity to ensure readiness for high-density, liquid-cooled rack deployments.

-