Local AI coding benchmark

LEEK BENCH: Can a local AI agent build a game with 8 GB VRAM?

Published August 26, 2026 · Updated August 27, 2026 · Reproducible local AI test

You can run LEEK BENCH on your own computer. Give a local coding model the fixed game prompt below, allow up to four turns, and record what it can build and repair. You can also copy our Codex and Ollama settings for a direct comparison.

Run LEEK BENCH in four steps

1. Create a new empty folder for the model.
2. Start Codex with a local model through Ollama.
3. Paste the fixed turn-one prompt from this page.
4. Use no more than three later turns for fault repair, game balance, and graphics. Keep a copy of the session and final files.

What LEEK BENCH tests

LEEK BENCH is a long-task coding test for local AI models. It asks one practical question: can a model plan, write, inspect, and improve a complete small game on limited consumer hardware?

A short code sample can hide serious faults. A game cannot. The model must create several files, keep rules consistent, manage game state, and correct its own work. The four-turn limit also tests whether the model can use feedback without losing earlier requirements.

The comparison uses the same core rules

Every model starts from the same first-turn prompt. Each run can use up to four turns. Later turns focus on faults, game balance, and visual improvements. Each model works in its own trusted project folder so that files from one run cannot help another run.

The benchmark looks at the complete result, not one code answer. We check whether the game starts, whether the main loop works, whether the circular movement is smooth, whether combat follows the brief, and whether later edits repair faults without breaking working features.

Use this fixed prompt for turn one

Copy this prompt without model-specific hints:

Create a small browser game called Leek Spinner Defense.
It is a 2D tower defense game with original anime-inspired chibi visuals.
Do not use copyrighted characters, franchise names, or copied character designs.

Use plain HTML, plain CSS, plain JavaScript, and HTML5 Canvas.
Do not use TypeScript, npm, build tools, backend services, or external game engines.
Create index.html, style.css, and game.js. The game must run locally by opening index.html.
Prioritize a working game over complicated graphics.

Build one circular level. Enemies spawn at 12 o'clock, march clockwise on a smooth curved path, and exit at a base at 9 o'clock. Use angle-based circular movement when practical. Do not use a path made from straight horizontal and vertical segments.

The player starts with gold and 20 lives. The player places defenders on fixed tower pads. Defend the base through 10 waves. Defeated enemies give gold. Enemies that reach the base remove lives. Win after wave 10. Lose when lives reach zero.

The main tower is the Leek Spinner: an original chibi defender with a large green leek. Its attack is melee. The leek spins around the defender and causes damage only when it physically touches an enemy.

Use clean outlines, bright colors, simple shading, readable weapons, and clear silhouettes. Use simple Canvas shapes if custom sprites are not available. Clearly separate the enemy path, grass, tower pads, base, enemies, and towers.

Show a large number on every enemy and friendly. An enemy's number is its current HP and must change as soon as it takes damage. A friendly's number is its current attack damage.

When a friendly defeats an enemy, permanently add that enemy's original HP to the friendly's attack. Only the friendly that gets the killing blow grows. Store both enemy.maxHp and enemy.hp. On death, use tower.attack += enemy.maxHp. If a projectile gets the kill, credit the tower that fired it. Briefly animate or flash the new attack number.

Increase enemy HP substantially from wave to wave.

The full protocol allows later turns for fault repair, balance, and graphics. Do not add new core requirements during those turns. A new requirement would make results harder to compare.

How the four setup pieces connect

You do not need prior experience with Codex configuration files. The setup has four pieces. Each piece has one job:

The connection is: Codex reads the profile, the profile selects the model catalog and Ollama provider, and Ollama runs the model. Codex then lets the model inspect and edit files in the empty game folder.

Create the Codex profile file

A TOML file is a plain-text settings file. On Windows, open %USERPROFILE%\.codex in File Explorer. This normally resolves to a path such as C:\Users\YourName\.codex. Create the folder if it does not exist.

Save the Qwen profile as qwen.config.toml inside that folder. Do not save it as qwen.config.toml.txt. The official Codex configuration reference confirms that named profile files use profile-name.config.toml inside the Codex home folder and are selected with --profile profile-name. See the official Codex configuration reference.

We used Windows, Ollama, and Codex. Each model had a separate profile, model catalog, and empty project folder. This profile shows the Qwen run:

[model_providers.ollama-launch-codex-app]
name = "Ollama"
base_url = "http://127.0.0.1:11434/v1/"
wire_api = "responses"
stream_idle_timeout_ms = 3600000
stream_max_retries = 5
request_max_retries = 10

model_provider = "ollama-launch-codex-app"
model = "hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL"
model_catalog_json = "C:/Users/YOU/.codex/qwen-models.json"

[apps._default]
enabled = false

[features]
apps = false

[projects.'c:\code\leek-tower']
trust_level = "trusted"

Change YOU to your Windows user-folder name. Then create C:\code\leek-tower. This must be a new empty folder for the Qwen run. The trusted-project entry lets Codex use the project configuration in that folder. Do not point two model runs at the same folder.

What the profile settings mean

• base_url points Codex to Ollama on the same computer.
• wire_api = "responses" tells Codex to use the Responses protocol.
• The timeout and retry values allow a slow local model to continue working.
• model selects the exact Ollama model.
• model_catalog_json points to a second file that describes the model to Codex.
• The apps settings disable connected apps for a controlled local test.
• The project entry identifies the folder that the model can edit.

Create one profile for each model

Copy qwen.config.toml to gemma.config.toml and glimmer.config.toml. In each copy, change the model identifier, model-catalog path, and project path. Use a separate empty folder such as C:\code\leek-dg for Gemma and C:\code\leek-dm for Glimmer.

Our Qwen catalog set apply_patch_tool_type to null. The Gemma and Glimmer catalogs set it to freeform. We also gave the model one short file-edit rule: use the shell to run apply_patch, and prefer patches to full-file rewrites.

Our exact Ollama model identifiers were:

• Qwen: hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
• Gemma: gemma4:31b
• Glimmer: muse-glimmer:30b-q4_K_M

The model catalog is an advanced compatibility file. The profile can load a different catalog for each model. For an exact comparison with our runs, use the matching catalog and apply_patch_tool_type value. For your own LEEK BENCH variant, record any catalog changes with the result. A catalog change can change how well the model uses tools.

Start the selected profile

Open Command Prompt in the empty project folder. If the codex command is on your PATH, run:

codex --profile qwen

The name after --profile must match the start of the profile filename. For example, --profile qwen loads qwen.config.toml.

We used this small batch file because our Codex executable was not on PATH. Save it as qwen.bat:

@echo off
set "CODEX=%USERPROFILE%\.codex\packages\standalone\current\bin\codex.exe"
"%CODEX%" --profile qwen %*

If this path does not exist on your computer, run where codex in Command Prompt. Use the returned path in the batch file. Run the batch file from the empty Qwen project folder.

Copy our Ollama memory settings

Install and start Ollama before you start Codex. Use ollama list in Command Prompt to confirm that Ollama responds and that the selected model is available.

The Ollama setup keeps one model loaded, runs one request at a time, enables flash attention, and uses an 8-bit KV cache. Run these commands once in Command Prompt:

setx OLLAMA_KEEP_ALIVE "-1"
setx OLLAMA_KV_CACHE_TYPE "q8_0"
setx OLLAMA_FLASH_ATTENTION "1"
setx OLLAMA_NUM_PARALLEL "1"
setx OLLAMA_MAX_LOADED_MODELS "1"

setx saves the values for future processes. Close Ollama and Command Prompt after you run the commands. Then start them again. These controls reduce memory pressure and prevent two large jobs from competing for the same GPU. They do not make the RTX 4060 faster. They help the run continue when one response takes a long time.

Verify each layer before the test

1. Run ollama list. Confirm that the model identifier is present.
2. Run the model once with ollama run MODEL_IDENTIFIER. Enter a short test message. Then exit the model.
3. Open the empty game folder in Command Prompt.
4. Run codex --profile qwen or the matching batch file.
5. Confirm that Codex reports the intended model and project folder.
6. Paste the fixed LEEK BENCH prompt only after these checks pass.

If Codex cannot connect, confirm that Ollama is open and that http://127.0.0.1:11434 is reachable. If Codex cannot find the profile, confirm the filename and the %USERPROFILE%\.codex location. If Codex cannot find the model catalog, correct the absolute path in model_catalog_json.

Our reference runs

Our test computer has an NVIDIA GeForce RTX 4060 with 8 GB VRAM. We use three models near 30B parameters: Qwen 3.8 27B, Google Gemma 4 31B, and Meta Muse Glimmer 30B. Ollama serves each model to Codex. Model data that does not fit in VRAM uses system memory.

The first video shows the complete Qwen run and gives a brief summary of its final results. Glimmer and Gemma will each get a separate video. We will add those videos here when they are ready.