How you actually add AI to a web app: run your own model on your server, or call a hosted API. Both are just another HTTP service.
← / → to navigate · EN/VI switch below
Your app already talks HTTP (Flask, curl, nginx). AI is just one more thing it can call over the same wire — the only question is who runs the model.
asks your app a question
/ask — forwards to a model
same HTTP call either way — only the address (and who pays) changes
There are exactly two shapes. Everything in this topic is a detail of one of these.
You download a model file and serve it yourself with llama.cpp's llama-server. It becomes a local HTTP endpoint that answers chat requests. You own it, it stays on your machine.
Someone else runs the models. You send a request to a URL (e.g. OpenRouter) with an API key and pay per token. No hardware, no setup — just an HTTP call to a provider.
AI is not a new kind of technology for your app. It is just another service you call over HTTP — like your database, like any third-party API. You send a message, it sends text back.
The one decision that shapes everything: do you run the model, or does someone else?
It's not a technical difference — it's an economics-and-control difference. The code on your side is almost identical.
The model runs on your VPS. Your prompts and data never leave your machine. No per-token fees — you pay for the server and the electricity.
Wins on privacy, control, and running cost. Costs you in setup, hardware, and scale.
The model runs on the provider's GPUs. Your prompts cross the internet to reach it. You pay per token, only when you use it, and you can swap models any time.
Wins on scale, effort, and variety. Costs you in per-token fees and data leaving your box.
Most real products sit somewhere in the middle — or pick a different end for different features. You move along this axis as your needs change.
Privacy, control, fixed cost. Your data never leaves the machine. You own the failure too.
Scale, variety, zero setup. Pick any model, pay as you go. Your prompts travel to the provider.
the code is the same — only the endpoint and the bill differ
Same request on your side. These five rows are the whole decision.
| Dimension | 🏠 Self-host | ☁️ Hosted |
|---|---|---|
| Who runs the model | You — on your VPS | The provider — on their GPUs |
| Where your data goes | Stays on your machine | Crosses the internet to them |
| Upfront / fixed cost | Server + GPU/RAM you must buy | None — no hardware |
| Per-use cost | None per request (you pay the server) | Pay per token, only when used |
| Model choice & scale | Limited by your hardware | Hundreds of models, elastic scale |
Whatever you call — local or hosted — the request and response always have this shape. That's why the code barely changes.
A list of messages — each with a role (system / user / assistant) and text. This is the whole prompt.
A POST to a /v1/chat/completions URL, with a JSON body. The same verb and shape you've used with curl all course.
A single JSON answer — or a stream where tokens arrive one by one. Either way: text out, that's the whole job.
What a model really is, how you serve it, and why model size is the whole game.
No magic. A trained model is a file of numbers (its weights). You load it, it runs, and it exposes an HTTP endpoint.
The model is a single file — often in GGUF format for llama.cpp. Download it and it's yours. 8B ≈ a few GB on disk.
llama-server reads the file into RAM/VRAM, then listens on a port and answers requests. It behaves like a tiny server.
The endpoint is OpenAI-compatible — the same /v1/chat/completions shape as every hosted provider. So your code is identical.
Grab the llama-server binary, make it runnable, and start it pointed at your model file. It now answers on :8080.
Two things fight over your machine's memory: the model itself, and the context (the conversation it must remember). Size sets both.
An 8B model needs roughly 6–8 GB of RAM/VRAM just to load. A 70B model needs far more — most small VPSes simply can't hold it.
Every token of conversation costs memory. Bigger --ctx-size means longer conversations but more RAM. You tune this to fit the model.
On a fixed-amount machine you can't have everything. This is the daily tension of running a local model.
Load the largest model your RAM allows. Smarter answers, but the model can only see a small window of the conversation.
Good for one-shot Q&A on modest hardware.
Pick a smaller model so there's room for a long conversation, a big file, or a long document to process.
Good for processing documents or long chat threads.
Rough numbers for 4-bit models. Your exact figures vary by model and quantization — this is the shape to plan around.
| Model size | Rough RAM/VRAM to load | Typical fit |
|---|---|---|
| ~7–8B | ≈ 6–8 GB | A 16 GB VPS, comfortably, with room for context |
| ~13B | ≈ 10–12 GB | A 16 GB VPS, tighter — context shrinks |
| ~30–34B | ≈ 20–24 GB | A 32 GB box, or a GPU with that much VRAM |
| ~70B | ≈ 40–48 GB | Multi-GPU or a big-VRAM machine — not a small VPS |
One request in, one answer out. It speaks the OpenAI-compatible shape — so local and hosted look identical.
The body is a list of messages plus a few knobs. model names the model, messages is the whole conversation, stream asks for tokens one-by-one.
{
"model": "llama-3.1-8b",
"messages": [
{"role": "user", "content": "Explain what a reverse proxy is, in one sentence."}
],
"max_tokens": 120,
"temperature": 0.7,
"stream": true
}That's the entire request. roles are system / user / assistant; content is the text.
A POST to the local endpoint. The -d body is exactly the JSON above — nothing more, nothing less.
# local llama-server, OpenAI-compatible
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama-3.1-8b",
"messages": [{"role": "user", "content": "Hello there"}]
}'Same verb, same path, same body you'd send to any provider — only the host is 127.0.0.1.
The stream flag is the difference between a chat that waits and a chat that types. Same request, different response framing.
The server thinks the whole answer, then returns it as one JSON object. Simple, but the user waits and sees nothing until it's done.
Fine for short, one-shot answers.
The server sends the answer as a series of small chunks, each with the next few tokens. Your UI paints them as they arrive — the typing effect.
The default feel of every modern chat app.
With stream: true, the response is a sequence of data: lines. Each carries the next slice; [DONE] marks the end.
Skip the hardware. One key opens a gateway to hundreds of models from many providers.
OpenRouter is a router in front of lots of providers. You call one URL; it forwards to whichever model you named.
The model field is the whole switch — openai/gpt-4o-mini, anthropic/claude, a local model, and more, all in one field.
A single Bearer token works across every model on the platform. No per-provider keys to juggle in your app.
You're billed per token, per model, only when you send a request. Some smaller models are even free. No server to run.
Identical shape to the local call — same body, same /v1/chat/completions path. Two things change: the host, and the Authorization header.
# one key, any model — pick the model in the body
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello"}]
}'Bearer token in the header; the model chosen in the body. That's the whole difference from local.
Line up the two calls. The JSON body is word-for-word the same — only the URL and the key differ.
POST to 127.0.0.1:8080/v1/chat/completions. No auth header needed — it's your own machine. You pay in hardware.
http://127.0.0.1:8080/v1/chat/completions
POST to openrouter.ai/api/v1/chat/completions with a Bearer key. No hardware — you pay per token.
https://openrouter.ai/api/v1/chat/completions
It's not 'better' than self-host — it's better for these situations.
Traffic spikes, many users, or you want to try ten models this week. You'd never buy ten GPUs for that — but a key handles it.
For general-purpose features where prompts don't contain sensitive user data, sending them to a provider is a fine trade for zero ops.
No model to download, no RAM to size, no server to babysit. Create a key, send a request, ship. You can self-host later.
Because the request shape is identical, a feature built on OpenRouter can point at your own llama-server tomorrow — just change the URL and key.
Your Flask app becomes a forwarder: it holds the key, sends the messages, and returns the text.
A few lines: read the user's message, POST it to the model (local or OpenRouter), return the text. You'll use AI to build the real thing — this is the shape.
# one endpoint that forwards the user's message to the model
@app.route("/ask", methods=["POST"])
def ask():
prompt = request.json.get("message")
r = requests.post(MODEL_URL, # local /v1 OR OpenRouter /v1
headers={"Authorization": "Bearer " + MODEL_KEY},
json={"model": "openai/gpt-4o-mini",
"messages": [{"role": "user", "content": prompt}]})
reply = r.json()["choices"][0]["message"]["content"]
return {"reply": reply}MODEL_URL + MODEL_KEY are the only two things that differ between local and hosted. Everything else is the same.
Follow a question from the user to the answer. Your app holds the key and never exposes it to the browser.
POST /ask
holds the key, builds the messages, forwards
key stays server-side
…and the model's text streams back up the same path, to the user.
Before the code, choose these. They're cheap to get right and expensive to regret.
Match the model to the RAM you have. If you only have 8 GB, an 8B 4-bit model with modest context is your sweet spot — not a 70B dream.
Set --ctx-size (or the request's limit) to how much you actually need. Too small drops history; too big eats memory for nothing.
Turn streaming on for chat — it feels alive and hides the wait. Keep it off for short, machine-parsed answers where one JSON is simpler.
Your app doesn't talk to 'AI'. It talks to a URL. That URL can be your own llama-server or a provider — and nothing else in your app changes.
sends a message
self-host for privacy and control · a provider for scale and variety — and the code barely moves between them
These are real and important — but they're not your next step, and they don't change how you call the service.
Attention, layers, how the model actually computes. Fascinating, but irrelevant to wiring it into your app.
Training a model further on your own data. Powerful, but a whole project of its own — prompt and API first.
The math behind 4-bit vs 8-bit. You'll just pick a GGUF file that fits your RAM — the why is optional.
Squeezing tokens-per-second out of a GPU. An optimization topic, far beyond a first integration.
You don't need to understand the model to use it. You need to understand the contract: send messages over HTTP, receive text. Own it for privacy, rent it for scale — and the code barely moves.
You now see AI as what it is: another HTTP service. Self-host with llama-server or rent from OpenRouter — and forward it through your Flask app.