Compression Native Architecture

A different way to build AI

Compression Native Architecture, or CNA, has been one of our greatest achievements here at Forlais.

We started with a problem. Training and running large models takes substantial computation, memory and energy, all of which come at a cost. GPUs and storage are major overheads. As models grow, the resources needed to build and use them become an increasingly important part of the problem.

We started by looking at the model itself, how it learns, how it represents what it has learned and the mathematics behind its calculations.

That brought us to floating point arithmetic. Many numbers and calculation results cannot be represented exactly at the precision available, so they are rounded. The numerical detail beyond that precision is not retained.

Those small differences are carried through later calculations and into the values produced during training. This is the rounding noise we wanted to address. Increasing the precision stores more detail, but it also means using more bits for each value.

We wanted to change that dependency at the source. With CNA, we removed the reliance on repeated floating point approximation and developed a floatless approach to model computation.

CNA brings compression into the architecture from the beginning. It changes the starting point for how we build a model.

What CNA means

CNA stands for Compression Native Architecture.

In a conventional neural model, training produces weights. These are the numerical values that a model learns. A common approach to making a model smaller is to compress those weights after training.

With CNA, compression is part of how the model is built from the outset. The architecture and the way the model learns are developed together, with compression built into the foundation.

CNA uses its own architecture and learning approach in place of a conventional transformer. Forlais changes how the model is formed and how it computes, including the arithmetic it uses.

Why we removed floating point

Floating point arithmetic represents many numbers approximately. There is a limit to the precision it stores, so values that cannot be represented exactly are rounded.

Those small numerical differences are a source of rounding noise. Across a sequence of calculations, they affect the precision of the result. Using more precision also requires more bits to represent each value.

CNA removes floating point arithmetic from its model computation. We remove that source of rounding noise at the arithmetic level.

This is part of the architecture itself. It changes the way the model calculates and represents what it learns.

CNA at Forlais Research

CNA is part of Forlais Research and our work on how intelligence is represented.

We develop model architecture, learning and computation together. The way a model represents what it learns matters alongside the resources needed to build and run it.

Forlais is sharing the purpose of CNA and its use of compression from the outset. These give a clear explanation of the work while keeping how it is done private.

The detailed architecture, learning process and implementation remain within our research.