Deploying your RAG Pipeline

Welcome Back!

So you came back for seconds, huh? Well, that was a great choice because now you’ll get to see how we can set up our RAG kit for production deployment. But that’s not all we’re gonna do. We will be creating and setting up a PostHog account for analytics as well as getting an OpenRouter Account created so we can utilize the API for our open-weight cloud models.

If you haven’t read part one, you can check it out here:

(Click on the title to get to the technical article if you prefer to read 🤓)

For those who are caught up, let’s get started!

Setting up Evals

Now we need to address our grounding. We need a set of evals to make sure our pipeline is up to par during new iterations. So what we want to do is create a brand new folder for evals. Our tree structure should look similar to this:

rag-starter-kit/ ├── core/ ├── app/ ├── evals/ │ ├── questions.json │ └── run.ts ├── docker-compose.yml └── rag.config.ts

*NOTE: the app folder is he front end of the project if you decided to run the SvelteKit CLI. That’s where your frontend will live. But I will leave that up to the reader to decide how they want to proceed with their frontend + UI.

We need to create a single JSON file to hold all of our premade questions. Here is a little example of the questions we will have:

[ { "question": "What does Digital Dopamine charge for a RAG integration?", "mustCiteSource": "digital-dopamine.md", "mustContain": ["$"], "mustRefuse": false }, { "question": "What's the capital of France?", "mustCiteSource": null, "mustContain": ["I don't have that information"], "mustRefuse": true } ]

So the path will be evals/questions.json at the repo root. And if you check the full repo, you can see the full list of questions i have for my use case, but you can create whatever questions you want, so long as they align with grounding and it’s a fixed list of (question, expected-source, expected-answer-shape) tuples drawn from your sample docs. The point of a fixed set is that re-running it is the only way you can trust an improvement.

A good rule of thumb:

* 5–10 in-scope questions answerable only from your docs.

* 3–5 out-of-scope questions that should trigger the refusal string.

Now we will need a script to run our evals in the first place. Since we are using TypeScript, we will create a new file called run.ts and the path will be rag-starter-kit/evals/run.ts . The full typescript file is below and you can see the order of operations that happen when running this file:

// evals/run.ts import questions from './questions.json'; import cfg from '../core/config'; import { getCollection } from '../core/store/vectorStore'; import { retrieve } from '../core/retrieve/retriever'; import { answer } from '../core/generate/chain'; type Result = { q: string; pass: boolean; why: string }; async function run() { const results: Result[] = []; const collection = await getCollection(cfg); for (const q of questions) { const chunks = await retrieve(collection, q.question); const { text, citations } = await answer(cfg, q.question, chunks); let pass = true; const whys: string[] = []; if (q.mustRefuse) { pass = text.includes("I don't have that information"); if (!pass) whys.push('expected refusal, got answer'); } else { for (const term of q.mustContain ?? []) { if (!text.includes(term)) { pass = false; whys.push(`missing "${term}"`); } } if (q.mustCiteSource) { const cited = citations?.some((c) => c.source.includes(q.mustCiteSource)); if (!cited) { pass = false; whys.push(`did not cite ${q.mustCiteSource}`); } } } results.push({ q: q.question, pass, why: whys.join('; ') || 'ok' }); } const score = results.filter((r) => r.pass).length / results.length; console.table(results); console.log(`\nGrounded-answer score: ${(score * 100).toFixed(1)}%`); } run();

We then just need to add the eval script to our package.json file (just don’t add the // comment I included next to the updated line since JSON doesn’t allow comments and the file would fail to parse):

"scripts": { "dev": "tsx core/index.ts", "typecheck": "tsc --noEmit", "test": "echo \"Error: no test specified\" && exit 1", "eval": "tsx evals/run.ts" // <- Added line },

NOTE: Re-run this on every change

Every pipeline tweak (new chunk size, new prompt wording, new model) gets pnpm run eval before you trust it, a score that goes down is how you catch regressions; the game plan explicitly calls this “regression-catching, same philosophy as Forge.” This file is also your proof artifact for the README/case study.

Now it’s time to wire in PostHog

Got that HOG In Me

PostHog is a fantastic product that has a generous free tier. They allow you to get up to a million requests before you're charged for anything, but it’s dynamic depending on the events you are monitoring. You don’t even have to add a credit card if you only wire in a single project. However, if you end up creating multiple projects, they will require you to add a credit card before you can actually finalize the setup for the 2nd project. Still, you aren’t charged a dime until a given tool’s own monthly allowance runs out (1M analytics events, 5K replays, 1,500 survey responses, 100K error exceptions, etc.). I mean, on their front page, they claim that 97% of companies use them for free, which is absolutely insane.

And they even have a general measurement tool to get a decent guesstimate on how much you can get before you start getting charged.

As you can see here, identified events and anonymous events share the same 1M/month free allowance; There’s no separate bucket for each one and where they actually differ is cost once you’re past the free tier: anonymous events are up to ~4x cheaper, which is why you’d lean on them for logged-out traffic. That’s the lever the calculator is showing you, not two separate free pools.

That’s just analytics though, all of the other tabs you see in the screenshot above ALSO allow you to measure the extent of their free limits. So you can pretty much use this premium product for free or very cheaply if you know how to manage the data you’re trying to capture.

And to just give a quick overview of the tool as a whole, which is open-source too, is best said below:

PostHog is the open-source platform for building self-driving products

PostHog provides every tool you need to build a successful product, and captures all the context agents need to proactively diagnose problems, uncover opportunities, and ship fixes:

* Self-driving mode: Turn signals in your product data (errors, rage clicks, failed queries, and more) into researched reports and pull requests you review and merge.

* Product analytics: Autocapture or manually instrument event-based analytics to understand user behavior and analyze data with visualization or SQL.

* Web analytics: Monitor web traffic and user sessions with a GA-like dashboard. Easily monitor conversion, web vitals, and revenue.

* Session replays: Watch real user sessions of interactions with your website or mobile app to diagnose issues and understand user behavior.

* Feature flags: Safely roll out features to select users or cohorts with feature flags.

* Experiments: Test changes and measure their statistical impact on goal metrics. Set up experiments with no-code too.

* Error tracking: Track errors, get alerts, and resolve issues to improve your product.

* Logs: Ingest, search, and analyze log data alongside the rest of your product data.

* Surveys: Ask anything with our collection of no-code survey templates, or build custom surveys with our survey builder.

* Data warehouse: Sync data from external tools like Stripe, Hubspot, your data warehouse, and more. Query it alongside your product data.

* Data pipelines: Run custom filters and transformations on your incoming data. Send it to 25+ tools or any webhook in real time or batch export large amounts to your warehouse.

* AI observability: Capture traces, generations, latency, and cost for your LLM-powered app.

* Workflows: Create workflows that automate actions or send messages to your users.

You can steer it all from Slack, web, desktop (PostHog Desktop), or your own editor via the MCP.

Best of all, all of this is free to use with a generous monthly free tier for each tool. Get started by signing up for PostHog Cloud US or PostHog Cloud EU.

This was pulled straight from their GitHub repo, and you can go through the README for more details → https://github.com/PostHog/posthog.

But let’s move on to how we will be adding it to our project. The snippet below is the location the docs suggest, but it isn’t where this project ends up initializing PostHog. That lands in hooks.client.ts, which we’ll get to next.

// app/src/routes/+layout.svelte

The setup above is slightly different from the docs, but we’ll initialize PostHog inside hooks.client.ts instead. It’s where handleError lives (so error capture and init share a file), it reads $env/dynamic/public at runtime rather than baking the token into the bundle at build time, which matters when one Docker image ships to several clients. It also runs before the app mounts, so nothing captures an event before PostHog is ready.

import { dev } from "$app/environment"; import type { HandleClientError } from "@sveltejs/kit"; import { env } from "$env/dynamic/public"; import posthog from "posthog-js"; const POSTHOG_TOKEN = env.PUBLIC_POSTHOG_PROJECT_TOKEN; const POSTHOG_HOST = env.PUBLIC_POSTHOG_HOST; export function init() { if (!POSTHOG_TOKEN || !POSTHOG_HOST) { if (dev) { const missingVariable = !POSTHOG_TOKEN ? "PUBLIC_POSTHOG_PROJECT_TOKEN" : "PUBLIC_POSTHOG_HOST"; throw new Error( `${missingVariable} variable required by PostHog is missing ` + "or un-configured, this causes events to be silently missed. " + `This error stops appearing once ${missingVariable} is configured`, ); } return; } posthog.init(POSTHOG_TOKEN, { api_host: POSTHOG_HOST, capture_exceptions: { capture_unhandled_errors: true, capture_unhandled_rejections: true, capture_console_errors: false, }, }); } export const handleError: HandleClientError = ({ error, status, message, }) => { if (POSTHOG_TOKEN && POSTHOG_HOST) { posthog.captureException(error); } return { message, status }; };

You’ll need your PostHog project token (the phc_... value) and host once you create your profile and set up your project. Down the line we’ll be capturing a custom event on every chat answer in the /api/chat endpoint (server-side via the PostHog node SDK, or client-side in the widget): question asked, answer returned, citation count, latency, refusal-vs-answered. In the code as written today, though, the wired-up events are chat_question_submitted (from the widget) and knowledge_base_reindexed (from the admin page). /api/chat itself has no PostHog call yet, so note that this would be an update later on.

* PostHog JS: https://posthog.com/docs/libraries/js

* PostHog node SDK: https://posthog.com/docs/libraries/node

* SvelteKit env vars: https://svelte.dev/docs/kit/$env-static-public

The idea is that events become the client dashboard

The events you emit here are the exact numbers a small-business owner will see: “40 questions asked this week, 95% answered from your docs, average answer time 1.2s, 1 out-of-scope question”. It’s the target state once we wire the full event set into /api/chat, not what’s there on day one.

Then we will need to add posthog into our project. Which is as simple as:

cd app && pnpm add posthog-js

There is also a PostHog wizard that you have an option to use, and it will run through your entire project to do the full implementation for you. And it’s pretty damn accurate. I’m talking all the way from proper imports to actually adding your POSTHOG public key into your env file and making sure that the variable is only used throughout the project AND if you didn’t have an env file before this was run (you def should have lol but…) it will create one for you and make sure it gets into .gitignore.

PostHog also has a Python SDK along with many other languages, so just in case you used Python or a different language to build the RAG pipeline, PostHog most likely has an SDK for the language you’re using.

Now that we have PostHog set up, let’s get OpenRouter added to our rag.config.ts file. Below is what the full file should look like (more or less):

export type FolderSource = { type: 'folder'; path: string; glob: string }; export type WebSource = { type: 'web'; urls: string[] }; export type GDriveSource = { type: 'gdrive'; folderId: string }; export type Source = FolderSource | WebSource | GDriveSource; export type LLMProvider = | { provider: 'ollama'; model: string; baseUrl: string; numCtx: number; numPredict: number; keepAlive: string | number; temperature: number; think?: boolean; } | { provider: 'openrouter'; model: string; apiKey: string; baseUrl: string }; export default { sources: [ { type: 'folder', path: './data', glob: '**/*.{md,txt}' }, // { type: 'web', urls: ['https://example.com/docs'] }, // { type: 'gdrive', folderId: '...' }, ] as Source[], chunking: { chunkSize: 500, chunkOverlap: 50 }, embeddings: { provider: 'ollama', model: 'nomic-embed-text', baseUrl: process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434', }, vectorStore: { provider: 'chroma', url: process.env.CHROMA_URL ?? 'http://localhost:8000', collection: 'rag-kit', }, llm: { active: 'ollama' as 'ollama' | 'openrouter', ollama: { provider: 'ollama', model: 'gemma3:4b', baseUrl: process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434', numCtx: 8192, numPredict: 512, keepAlive: '30m', think: false, temperature: 0.1, } satisfies LLMProvider, openrouter: { provider: 'openrouter', model: 'z-ai/glm-5.2', // comment out/un-comment models when switching. // model: 'moonshotai/kimi-k3', // model: 'google/gemma-4-31b', // model: 'minimax/minimax-m3', apiKey: process.env.OPENROUTER_API_KEY ?? '', baseUrl: 'https://openrouter.ai/api/v1', } satisfies LLMProvider, }, };

You might notice 5 additional Ollama settings that haven’t been mentioned previously. I made these needed improvements to my pipeline and each one kills a specific waste:

* numPredict: 512 - hard cap on generated tokens. When uncapped, the same one-sentence question ran 3,200+ tokens when testing one time; at ~10 t/s that’s a five-minute wait (with gemma4: 12b). Biggest single win for perceived speed.

* think: false - suppresses the hidden reasoning pass on thinking models, which can be thousands of tokens you never see. More speed, less memory pressure.

* numCtx: 8192 - context window set to 8192, and the main memory knob (the KV cache scales with it). The retrieval prompt here is ~650 tokens, so 8192 is already generous; Meaning without this limit, Ollama would just use the model’s max context window, which ate up my memory, no matter which smaller model I used. For example running ollama ps while using OpenCode and Ornith 1.5 9b, you can see that while this model is only 6.6 GB in size, it’s still taking up its max context window:context_screenshot.png

* keepAlive: '30m' - keeps the 12B model (~8 GB) resident instead of re-reading it off disk between questions. Saves the reload on every follow-up.

* temperature: 0.1 - near-determinism. Left at the model’s own default (often 1), the same question answers on one run and refuses on the next; grounded extraction can’t have that.

We don’t want to replace Ollama, which defeats the “switch without code changes” goal. With active, both providers are always configured; switching is a one-line edit. A client who never wants cloud just leaves active: 'ollama' forever and the OpenRouter block is inert (no API key, no calls). A client who wants cloud flips it and sets the env var. But now we need to update our buildChain function.

answer() and generate() both call buildChain(cfg) and pass cfg: Pick. They don’t need to change. The only file that reads cfg.llm fields directly is core/generate/chain.ts, so that’s the one place the branching lives.

Below is the full core/generate/chain.ts after this step. It’s the Phase 3 file (prompt + buildNumberedContext + answer + generate) with two additions:

* An activeLLM() resolver at the top. The only place that knows about llm.active.

* buildChain() now branches: OpenRouter uses the OpenAI JS client wrapped in a RunnableLambda so it has the same .invoke({ numberedContext, question }) contract as the Ollama path. Both branches return a LangChain Runnable. answer() and generate() see no difference.

NOTE: This is the version from Part 1 plus the OpenRouter branch. In my live repo the prompt also reads the refusal from core/generate/refusal.ts via REFUSAL instead of hardcoding the string, and the ChatOllama call passes the extra Ollama settings which are both shown here. This was added during my attempt to include a dedicated Judge (Like Jev, but with Tev1 or Nimble since they are open-weight decision models), but I still need to work on that more. (I have to post screenshots of hain.)

import { ChatOllama } from '@langchain/ollama'; import { ChatPromptTemplate } from '@langchain/core/prompts'; import { ChatPromptValue } from '@langchain/core/prompt_values'; import { RunnableLambda } from '@langchain/core/runnables'; import { basename } from 'node:path'; import type { RagConfig } from '../config.js'; import type { RetrievedChunk } from '../retrieve/retriever.js'; import { REFUSAL } from './refusal.js'; import OpenAI from 'openai'; // REFUSAL is interpolated at module load rather than passed as a // variable, so this prompt and the answerability gate cannot drift // apart. It must not contain braces, which ChatPromptTemplate would // read as a placeholder. const PROMPT = ChatPromptTemplate.fromTemplate(` Rules: 1. If the context contains the answer, even if the question's wording differs from the source wording (synonyms, paraphrases, related terms like "implementation" vs "integration", "cost" vs "charge", "price" vs "pricing"), answer it and cite every claim with [N]. 2. Refuse ONLY when no context item is relevant to the question's subject at all. Do not refuse because the question is terse or omits its subject: a bare question like "how much do they charge?" is answered by the context item covering pricing for one of the organization's services, give that answer and name the service it applies to, using the wording of the context item. When refusing is warranted, respond ONLY with: "${REFUSAL}" 3. Use the closest matching source. Refer to products, services and figures using exactly the terms the context uses, never infer or invent a name the context does not state. Do not combine unrelated facts to fabricate an answer. 4. Every claim MUST end with a citation [N] referring to the numbered context item it comes from. 5. Answer in at most three short sentences. Do not restate the context or add preamble. Context (numbered): {numberedContext} Question: {question} Answer `); export interface Citation { n: number; source: string; excerpt?: string; } export interface AnswerResult { text: string; citations: Citation[]; } const CITE_RE = /\[(\d+)\]/g; // Show just the filename, not a path function formatSource(source: string | undefined | null): string { if (!source) return 'unknown'; if (source.startsWith('http://') || source.startsWith('https://')) { return source; } return basename(source); } export function buildNumberedContext( chunks: RetrievedChunk[], ): { numberedContext: string; citationMap: Map } { const citationMap = new Map(); const parts: string[] = []; chunks.forEach((c, i) => { const n = i + 1; parts.push(`[${n}] ${c.pageContent}`); citationMap.set(n, { n, source: formatSource(c.metadata?.source as string), excerpt: c.pageContent.slice(0, 120), }); }); return { numberedContext: parts.join('\n\n'), citationMap }; } type LLMConfig = RagConfig['llm']; function activeLLM(llm: LLMConfig) { return llm.active === 'openrouter' ? llm.openrouter : llm.ollama; } function buildChain(cfg: Pick) { const llm = activeLLM(cfg.llm); if (llm.provider === 'openrouter') { // Guard: fail loudly if cloud is active but the key is missing. // Saves you (or a client) from a mysterious 401 an hour into // debugging. if (!llm.apiKey) { throw new Error( "OPENROUTER_API_KEY is not set but llm.active is " + "'openrouter'. Add it to .env (repo root) and app/.env, " + "or set llm.active back to 'ollama'.", ); } const client = new OpenAI({ apiKey: llm.apiKey, baseURL: llm.baseUrl, }); // Wrap the OpenAI call in a RunnableLambda so it has the same // .invoke({ numberedContext, question }) contract as the Ollama // pipe below. const model = RunnableLambda.from( async (input: ChatPromptValue) => { const messages = input .toChatMessages() .map( ( m, ): OpenAI.Chat.Completions.ChatCompletionMessageParam => { const t = m.getType(); if (t === 'human') { return { role: 'user', content: m.content as string }; } if (t === 'ai') { return { role: 'assistant', content: m.content as string }; } return { role: 'system', content: m.content as string }; }, ); const res = await client.chat.completions.create({ model: llm.model, messages, }); return { content: res.choices[0]?.message?.content ?? '' }; }, ); return PROMPT.pipe(model); } const model = new ChatOllama({ model: llm.model, baseUrl: llm.baseUrl, numCtx: llm.numCtx, numPredict: llm.numPredict, keepAlive: llm.keepAlive, temperature: llm.temperature, ...(llm.think === undefined ? {} : { think: llm.think }), }); return PROMPT.pipe(model); } export async function generate( cfg: Pick, question: string, chunks: RetrievedChunk[], ): Promise { const chain = buildChain(cfg); const { numberedContext } = buildNumberedContext(chunks); const response = await chain.invoke({ numberedContext, question, }); return typeof response === 'string' ? response : (response as { content?: string }).content ?? String(response); } export async function answer( cfg: Pick, question: string, chunks: RetrievedChunk[], ): Promise { const { numberedContext, citationMap } = buildNumberedContext(chunks); const chain = buildChain(cfg); const response = await chain.invoke({ numberedContext, question }); const raw = typeof response === 'string' ? response : (response as { content?: string }).content ?? String(response); const claimed = new Set(); let m: RegExpExecArray | null = CITE_RE.exec(raw); while (m !== null) { claimed.add(Number(m[1])); m = CITE_RE.exec(raw); } const validNs = [...claimed].filter((n) => citationMap.has(n)); const validSet = new Set(validNs); const text = raw.replace(CITE_RE, (full, n) => validSet.has(Number(n)) ? full : '', ); const citations = validNs .sort((a, b) => a - b) .map((n) => citationMap.get(n)!) .filter(Boolean); return { text, citations }; }

buildChain is the single seam. Whether it builds an Ollama-backed PROMPT.pipe(model) or an OpenRouter-backed one, the thing it returns has to have the same .invoke({ numberedContext, question }) contract. The OpenRouter branch above wraps client.chat.completions.create(...) in RunnableLambda.from(...) so both branches are Runnables. Regarding new imports, the OpenAI JS client is the only new one.

OpenRouter is set, and now we need to get Docker situated.

Docker Deploy

This will be based on me getting Docker set up on my first go-round, so make sure to double-check your local environment if you get any errors (for example, you might have your code in a different root level than I do or maybe using a different port).

A few notes about the version I worked on, docker-compose.yml pins chromadb/chroma:1.5.9, mounts chroma-data:/data, has a /api/v2/heartbeat healthcheck, and restart: unless-stopped. The full file is below:

services: chroma: image: chromadb/chroma:1.5.9 container_name: rag-starter-kit-chroma-1 ports: - "8000:8000" volumes: - chroma-data:/data healthcheck: test: - CMD - bash - -c - "exec 3

/dev/tcp/localhost/8000 && printf 'GET /api/v2/heartbeat HTTP/1.0\\r\\nHost: localhost\\r\\n\\r\\n' >&3 && head -n1 <&3 | grep -q '200'" interval: 10s timeout: 5s retries: 5 start_period: 15s restart: unless-stopped ollama: image: ollama/ollama:latest container_name: rag-starter-kit-ollama-1 ports: - "11435:11434" volumes: - ollama-data:/root/.ollama restart: unless-stopped app: build: context: . dockerfile: app/Dockerfile container_name: rag-starter-kit-app-1 environment: - OPENROUTER_API_KEY=${OPENROUTER_API_KEY:-} - ADMIN_TOKEN=${ADMIN_TOKEN:-} - PUBLIC_POSTHOG_PROJECT_TOKEN=${PUBLIC_POSTHOG_PROJECT_TOKEN:-} - PUBLIC_POSTHOG_HOST=${PUBLIC_POSTHOG_HOST:-} - CHROMA_URL=http://chroma:8000 - OLLAMA_BASE_URL=http://ollama:11434 # RAG_DATA_DIR overrides the import.meta.url-derived path, which points # into the build output in the bundled image; /data holds the COPY'd docs. - RAG_DATA_DIR=/ ports: - "3000:3000" depends_on: chroma: condition: service_healthy ollama: condition: service_started restart: unless-stopped volumes: chroma-data: ollama-data:

The build context must be the repo root, not app/. The SvelteKit app imports $core/*, your pipeline in core/ via the kit.alias. If the build context is app/, core/ is outside the Docker build and pnpm run build fails with Cannot find module '$core/...'. Setting context: . + dockerfile: app/Dockerfile brings core/ into the build while keeping the Dockerfile in app/.

SvelteKit’s adapter-node output is a standalone Node server in build/. The Dockerfile builds that and runs it. So we need to get the file created with the following:

FROM node:22-alpine WORKDIR /docker_root RUN corepack enable && corepack prepare pnpm@10 --activate COPY package.json pnpm-lock.yaml ./ RUN pnpm install --frozen-lockfile WORKDIR /docker_root/app COPY app/package.json app/pnpm-lock.yaml app/pnpm-workspace.yaml ./ RUN pnpm install --frozen-lockfile WORKDIR /docker_root COPY app/ ./app/ COPY core/ ./core/ COPY rag.config.ts ./ COPY data/ /data/ WORKDIR /docker_root/app RUN pnpm run build EXPOSE 3000 CMD ["node", "build"]

core/ directory has no package.json of its own. It’s just .ts files that get compiled as part of the app’s build via the include: ["../core/**/*.ts"] in app/tsconfig.json. But the Dockerfile still installs two manifests: the root one (because core/ imports @langchain/*, chromadb, openai, and friends from the root node_modules) and then the app’s. So the install steps cover both, and the copy step afterward brings in everything (including core/) for the build.

We also need a .dockerignore at the root level /rag-starter-kit/.dockerignore):

After adding this, the build context drops from ~874MB to a few MB (just your source + manifests) and builds get dramatically faster. The .env exclusion is critical, it prevents accidentally baking secrets into the image. The evals / docs / *.pdf exclusions keep the image lean (they're not needed at runtime).

The Dockerfile’s WORKDIR /docker_root/app + COPY app/package.json ... ./ means pnpm runs in /app, so pnpm run build builds the app directly. The ollama/ollama image is a fresh Ollama with no models. After docker compose up, you need to pull the chat + embedding models into the container once:

docker compose exec ollama ollama pull gemma3:4b docker compose exec ollama ollama pull nomic-embed-text

These persist in the ollama-data volume, so it’s a one-time setup. Any future run of docker compose up reuses them. To make sure we are ready to test our deploy, lets run through some checks.

We need to create a 2nd .env file

We’ll need two .env files with the same variables in each file. The root env feeds the CLI/eval harness, core/config.ts loads it with import 'dotenv/config' (there’s no --env-file flag; dotenv is the thing doing the loading), and the app/ env feeds the SvelteKit client via $env/dynamic/public in hooks.client.ts, plus plain process.env in the server routes. PUBLIC_ vars are public by definition, so they’re never read through $env/static/private.

File 1: .env at the repo root (next to docker-compose.yml):

OPENROUTER_API_KEY={key} # from https://openrouter.ai/keys PUBLIC_POSTHOG_PROJECT_TOKEN={key} # from PostHog project settings PUBLIC_POSTHOG_HOST=https://us.i.posthog.com # or your self-hosted URL ADMIN_TOKEN={key} # a secret you invent; gates the /admin re-index endpoint

File 2: app/.env (same three lines, copy-paste):

OPENROUTER_API_KEY={key} # from https://openrouter.ai/keys PUBLIC_POSTHOG_PROJECT_TOKEN={key} # from PostHog project settings PUBLIC_POSTHOG_HOST=https://us.i.posthog.com # or your self-hosted URL ADMIN_TOKEN={key} # a secret you invent; gates the /admin re-index endpoint

ADMIN_TOKEN is optional for the container to start (compose defaults it to empty), but the compiled container runs in production mode, so the /admin re-index endpoint returns 403 unless it’s set. Your local cd app && pnpm dev doesn’t need it (dev mode allows reindexing), but a fresh Docker deploy relies on /admin, so set it to any secret string if you want the reindex step below to work.

Both must be in .gitignore.

Add a .dockerignore at the repo root

As mentioned earlier, we need to add .dockerignore. Without it, Docker ships ~874MB of context to the build daemon on every up --build.

From the repo root:

docker compose down -v && docker compose up --build -d

* down -v wipes containers + volumes (True cold start. The path a client hits on first deploy).

* up --build rebuilds the app image (picks up any rag.config.ts / code changes) and starts everything.

* -d detaches so you get your shell back.

You should see three containers running:

docker compose ps

NAME STATUS PORTS rag-starter-kit-chroma-1 Up (healthy) 8000 rag-starter-kit-ollama-1 Up 11434 rag-starter-kit-app-1 Up 3000

If app exits, check docker compose logs app. The most common cause is a missing CHROMA_URL/OLLAMA_BASE_URL env var or a rag.config.ts that still hardcodes localhost. (The app service has no healthcheck defined, so you won’t see Up (unhealthy) on it; it’s either Up or gone.)

Pull models into the Ollama container (one-time only)

The ollama/ollama image ships with no models. Pull them now and they persist in the ollama-data volume, so you only do this once per cold wipe:

docker compose exec ollama ollama pull gemma4:12b # Or gemma3:4b docker compose exec ollama ollama pull nomic-embed-text

The first pull takes a bit (gemma4:12b is about 8 GB). Subsequent docker compose up reuses them and doesn’t re-pull.

Verify the pulls worked. docker compose exec ollama ollama list should show both models. If a model is missing, the app's first chat/retrieval call will return a 500 with a "model not found" error from Ollama.

Re-index the docs into the fresh Chroma

docker compose down -v wiped the chroma-data volume, so the vector store is empty. Trigger a re-index from the admin UI:

NOTE: On the local-dev side: the dev server is cd app && pnpm dev, not pnpm run dev at the repo root. The root script runs the core/index.ts CLI. This is the containerized app at 3000 either way.)

* Open http://localhost:3000/admin (make sure your docker container is still running)

* Click the re-index button

* Wait for it to report the chunk count

build/ is the compiled SvelteKit server, not a reindex entrypoint. The admin page’s re-index button calls the same runIngestion() the CLI uses, so that’s the way to trigger it here.

Verify the app works

* Open http://localhost:3000, you should see the demo chat page, exactly as you would see it i ran within /app with pnpm run dev.

* Ask an in-scope question from your evals/questions.json → should get a grounded answer with citations.

* Ask an out-of-scope question → should get the refusal string.

Some docs for deeper dives into some of the concepts covered:

* SvelteKit adapter-node: https://svelte.dev/docs/kit/adapter-node

* Docker Compose depends_on conditions: https://docs.docker.com/compose/compose-file/05-services/#depends_on

* Ollama Docker: https://github.com/ollama/ollama/blob/main/docs/docker.md

* Docker Compose: https://docs.docker.com/compose/

* .dockerignore reference: https://docs.docker.com/build/concepts/context/#dockerignore-file

Set Sails

That about wraps up the Docker setup, and now you should be good to use your favorite deployment platform to ship this to production. Railway is a platform I use myself, and their Docker quick start can be found here → https://docs.railway.com/quick-start#deploying-your-project---from-a-docker-image.

Hopefully this quick part 2 helped you deploy your kit so you can show it off to potential clients. I’m always open to improving my technical writing, so let me know if you think there are gaps in my steps or if any of them don’t make sense.

If you want to keep up with my work or want to connect as peers, check out my social links below and give me a follow!

* 🦋 Bluesky

* 📸 Instagram

* ▶️ Youtube

* 💻 Github

* 👾 Discord