laya.cpp: Optimized laya near-instant decision making

After seeing u/Nandakishor_ml’s post introducing Laya, I wanted to see how fast it could run in a standalone C++ implementation.

Credit to u/Nandakishor_ml for the architecture, training and open-source release. My contribution is the inference implementation: laya.cpp, built on ggml with custom CUDA kernels.

It supports all three checkpoints—English, multilingual and typed-decisions—with native tokenization, model execution and output formatting. There’s also an HTTP server with a JEV-compatible endpoint. No Python or PyTorch is required for inference.

Some English-model results on an RTX PRO 6000 Blackwell, capped at 450 W:

| Batch | Python BF16 | C++ BF16 | Python FP32 | C++ FP32 |
|---|---:|---:|---:|---:|
| 1 | 149 | 366 | 148 | 342 |
| 2 | 268 | 586 | 202 | 421 |
| 4 | 460 | 761 | 233 | 437 |
| 8 | 663 | 810 | 232 | 386 |

These are questions per second over a fixed 250-question corpus containing choices, scores and booleans. Each precision has paired Python/C++ timings with alternating execution order. Loading and JSON transport are excluded. The README has the full three-model results.

Most of the optimization came from removing unnecessary conversions and copies, fusing operations while preserving rounding, and improving attention memory access.

The code is MIT-licensed. BF16 currently needs the documented CUDA 13.0/cuBLAS 13.1.0 build profile.

Implemented using Codex Astra.