Private local inference guide

How to host Qwen locally on Windows and use it from any device

Published August 6, 2026 · Tested with Qwen 3.5 9B

Turn a high-RAM Windows PC into your personal AI server. Buy one powerful HiRAM inference box. Put it somewhere with power and Ethernet. Use Qwen from every laptop, desktop, and application you already own.

Ollama runs the model, LiteLLM supplies an authenticated OpenAI-compatible endpoint, and Tailscale makes it available to your own devices without placing it on the public internet.

The private inference architecture

1. Ollama loads Qwen and listens only on the computer.
2. LiteLLM Proxy presents one authenticated, OpenAI-compatible API and maps a stable alias to the installed model.
3. Tailscale Serve terminates HTTPS and exposes LiteLLM only inside your tailnet. This guide does not enable Tailscale Funnel.

Keeping both application servers on 127.0.0.1 prevents ordinary LAN or public access. Remote clients enter through the authenticated tailnet and still need the LiteLLM master key.

Prerequisites

• Windows 11 with Ollama installed and a Qwen model already downloaded.
• Tailscale installed and signed into the same tailnet as the client devices.
• PowerShell running as Administrator for Tailscale Serve configuration.
• Enough RAM or VRAM for the exact model and quantization you select.

1. Verify Ollama before adding a router

ollama list

Invoke-RestMethod http://127.0.0.1:11434/api/tags

Test the exact Qwen 3.5 model directly. This example uses the model from our experiment; substitute an installed identifier when needed:

$body = @{
    model = "hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M"
    messages = @(
        @{
            role = "user"
            content = "Reply with exactly: no thinking"
        }
    )
    think = $false
    stream = $false
} | ConvertTo-Json -Depth 6

Invoke-RestMethod `
    -Uri "http://127.0.0.1:11434/api/chat" `
    -Method Post `
    -ContentType "application/json" `
    -Body $body

2. Install a pinned LiteLLM Proxy

winget install --id=astral-sh.uv -e
uv tool install --python 3.12 "litellm[proxy]==1.90.3"

The explicit version makes the tested environment reproducible. Review a later LiteLLM release before changing the pin.

3. Configure a stable model alias

Create a neutral local directory and configuration file:

New-Item -ItemType Directory -Force C:\local-ai-router
notepad C:\local-ai-router\config.yaml

Use this configuration:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M
      api_base: http://127.0.0.1:11434

Generate a long random master key with a password manager, place it in the process environment, and start LiteLLM on loopback:

$env:LITELLM_MASTER_KEY = "sk-replace-with-a-long-random-secret"

litellm `
  --config C:\local-ai-router\config.yaml `
  --host 127.0.0.1 `
  --port 4000

4. Test the authenticated local endpoint

Open a second PowerShell window with the same master-key environment variable, then check health:

Invoke-RestMethod http://127.0.0.1:4000/health/liveliness

Send an OpenAI-compatible chat request through the model alias:

$body = @{
    model = "local-qwen"
    messages = @(
        @{
            role = "user"
            content = "Reply with exactly: router works"
        }
    )
    reasoning_effort = "none"
    stream = $false
    max_tokens = 40
} | ConvertTo-Json -Depth 6

Invoke-RestMethod `
    -Uri "http://127.0.0.1:4000/v1/chat/completions" `
    -Method Post `
    -Headers @{ Authorization = "Bearer $env:LITELLM_MASTER_KEY" } `
    -ContentType "application/json" `
    -Body $body

5. Put authenticated inference on your private tailnet

Expose LiteLLM—not the unauthenticated Ollama port—through Tailscale Serve:

& "C:\Program Files\Tailscale\tailscale.exe" serve reset

& "C:\Program Files\Tailscale\tailscale.exe" `
    serve --bg --https=443 http://127.0.0.1:4000

& "C:\Program Files\Tailscale\tailscale.exe" serve status
& "C:\Program Files\Tailscale\tailscale.exe" ip -4

Tailscale prints the private HTTPS hostname. A client must belong to the permitted tailnet and send the LiteLLM bearer token. Use tailscale serve off when the endpoint should no longer be available.

The two internal services remain bound to loopback:

• 127.0.0.1:11434 — Ollama, not exposed directly to the tailnet.
• 127.0.0.1:4000 — LiteLLM, exposed through private Tailscale HTTPS.

6. Test from another tailnet device

On another authorized device, use the same OpenAI chat-completions body and point the request at the private hostname printed by Tailscale:

$headers = @{
    Authorization = "Bearer sk-your-router-key"
}

Invoke-RestMethod `
    -Uri "https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions" `
    -Method Post `
    -Headers $headers `
    -ContentType "application/json" `
    -Body $body

OpenAI-compatible applications can use that HTTPS origin as their base URL and the LiteLLM key as their API key. They do not need bespoke Ollama support.

Diagnose one layer at a time

This is why the direct Ollama request remains in the guide: it is a diagnostic boundary, not the endpoint clients should use.

Qwen 3.5, Qwen 3.6, and the Qwen 3.8 release

Qwen 3.8 release-day checklist

1. Confirm the official model card, license, parameter count, and context limit.
2. Confirm an official or reputable Ollama/GGUF artifact and record its immutable digest.
3. Measure download size, idle memory, prompt processing, generation memory, and tokens per second.
4. Test thinking controls, tools, images, long context, and the LiteLLM OpenAI-compatible route.
5. Update this article's tested-version label and modification date without changing its URL.

Security checklist

• Bind Ollama and LiteLLM to 127.0.0.1, not every network interface.
• Expose LiteLLM through Tailscale Serve; do not enable Funnel for this private endpoint.
• Require a long LiteLLM master key and rotate it if it is copied into logs or shell history.
• Use tailnet access controls to restrict which users and devices can reach the host.
• Run the services under a dedicated non-administrator account for persistent operation.
• Pin software versions and review updates before applying them to an inference host.

Primary references

• Ollama Qwen 3.5 library
• Ollama Qwen 3.6 library
• LiteLLM Proxy documentation
• Tailscale Serve command documentation