COMPACT TERNARY AI
MATRIX is a family of compact artificial intelligence models developed by TERNARY. We are studying how much practical capability can be reached with a smaller, more resource-efficient model when the architecture, the training data and the intended use are deliberately kept under control.
The first complete model is MATRIX EST — a language model aimed at Estonian and at local use. It is the first measurable basis from which we develop the MATRIX architecture, its runtime layer and later specialised models.
[ The MATRIX architecture ]
MATRIX starts from the question of whether part of the tasks given to artificial intelligence can be solved with a smaller model without losing the practical capability that is needed. The aim is not to build the smallest possible model, but to find the smallest model that is sufficient for the task.
The MATRIX architecture is built on the principle of ternary weights:
[-1, 0, +1]
Using three weight values makes it possible to reduce the precision needed to represent the model, and opens the way to smaller memory use and simpler computation. The actual benefit still has to be measured together with the quality of the model, how well it trains and how fast it runs.
In MATRIX we treat the architecture, the training, the model format and the runtime as one technical whole. Ternary representation is not simply a quantisation trick applied to a finished model afterwards.
architecture → training → model format → runtime → hardware
[ MATRIX EST ]
MATRIX EST is the first complete MATRIX model and the project's main test platform. The first version is focused on Estonian, on local use and on practical tasks.
The aim is not only to generate Estonian text. The model is being developed so that it can understand natural Estonian, follow instructions, solve the tasks that suit it, work with texts and documents, and tell the difference between something it knows and a situation where the information needs checking.
MATRIX EST also lets us measure which tasks a model of this size class is sufficient for, and when a larger or separately specialised model is needed.
- 01Estonian
- 02Following instructions
- 03Problem solving
- 04Knowledge of Estonia
- 05Programming
- 06Factual reliability
- 07Working with documents
[ The model ]
The current MATRIX EST configuration belongs to the class of models with roughly 300 million parameters. It is a development and measurement platform, not a final product specification.
- Parameters
- 293.4M
- Model dimension
- 1024
- Layers
- 22
- Feed-forward dimension
- 2730
- Vocabulary
- 8,000 tokens
- Context
- 1,024 tokens
- Architecture
- MATRIX
The size and configuration of the next model variants will be decided from measurements, and depending on the task they may be either smaller or larger.
[ Data and training ]
For a small model the quality of the training data matters especially. The MATRIX EST corpus is not treated as a large pile of text: the material is cleaned, structured and balanced according to the capabilities expected of the model.
The training material covers general language, Estonian content, following instructions, problem solving, knowledge of Estonia, programming and working with documents. The origin of the data sources and their licences are documented separately.
The model is trained in stages, and intermediate versions are evaluated against fixed tests. The aim is not the highest possible number of training steps, but measurable improvement in language, in following instructions and in solving tasks.
[ Where it stands ]
MATRIX EST is in active development. The current figures describe the technical validation of the model version in progress, and they are updated along with the next training stages.
- 5.0303
- Hard validation CE
- 5.023
- Reference value
Hard validation CE and the reference value are control measures for development and reproducibility, not quality scores aimed at users.
[ Runtime and specialisation ]
The aim of MATRIX does not end with training a model. For local use the model has to run efficiently after training as well, which is why we are developing the MATRIX model format and runtime layer for CPU and GPU use in parallel.
The same technical basis has to make it possible to build smaller specialised models for particular tasks later. A model for programming, for processing documents, for an organisation's internal knowledge or for one field of work does not need to carry the capability of a whole general model.
The aim is to use the smallest model for each task whose capability is sufficient for that work.
[ Development stages ]
- Studying the ternary architecture, memory, computation and trainability. The benefit of three weight values has to be measured together with the quality of the model, how well it trains and how fast it runs.01
Architecture
Developing the MATRIX ternary architecture and its trainability.
- The first complete MATRIX model and the project's main test platform. The focus is Estonian, local use and practical tasks.02
MATRIX EST
Training and evaluating the first complete Estonian model.
- For local use the model has to run efficiently after training as well. For that we are developing MATRIX's own model format and runtime layer.03
Runtime
The model format and optimisation for CPU and GPU.
- A model for programming, for processing documents, for an organisation's internal knowledge or for one field of work does not need to carry the capability of a whole general model.04
Specialists
Smaller models for particular tasks.
- MATRIX is not only one model. On a shared technical basis, models specialised for different languages and fields can be built.05
Languages
Extending the same technical basis to other languages.
[ The next stage ]
The next goal for MATRIX EST is a complete base version that can be compared against the same evaluation set throughout the rest of the development.
The work ahead is focused on expanding and balancing the training data, training the model, evaluating its capabilities, and developing the MATRIX runtime and model format further.
Further decisions about the size, the specialisation and the architecture of the model will be made from measurements.
The aim of MATRIX is not to replace every large language model with one small model. The aim is a technical basis on which the smallest sufficiently capable model can be used for a task, bringing in a larger model or an external data source only when it is actually needed.
Small. Local. Specialised. Controllable.
[ Other projects ]
[ HÄÄL ]
HÄÄL is a working Estonian text-to-speech (TTS) solution built around a small, resource-efficient model. Development is focused on the balance between speech quality, stability, voice adaptation and low computational cost. It can be used on a server, and locally where the hardware allows.
[ View project ] — HÄÄL
[ KUULA ]
KUULA is the follow-on project to HÄÄL and an Estonian speech-to-text (STT) solution. Its task is to turn speech into text and to form the input layer of a system where speech recognition, a language model and speech synthesis work together. We are building a connecting layer between them, so that the chain KUULA → LLM → HÄÄL can later be used in different applications.
[ View project ] — KUULA