▌ INTERNAL · UNLISTED

Local Security RAG + Exploit-Dev Pipeline

A fully self-hosted, GPU-split retrieval-augmented-generation stack for security research — no cloud, no API keys leaving the LAN. A ~488-PDF security library is embedded into a hybrid Qdrant index, cross-encoder reranked, and answered by an uncensored local model with real citations. Everything runs on a Tesla V100 32 GB + RTX 3060 (44 GB CUDA), with a 32 GB Radeon Pro V620 specialist tier landing on a second node.

What changed — 2026-09-14: the P100 came out, a Tesla V100 32 GB went in, the generator collapsed onto one card, and the reranker moved back onto GPU

The 16 GB P100 is gone from pluto; a Tesla V100 PCIE 32 GB (it enumerates as Tesla PG500-216) now sits in the CPU x16 slot. pluto is V100 32 GB + RTX 3060 12 GB = 44 GB CUDA, no display output. The old P100 + 3060 split is retired: 32 GB on a single card means Qwen3.8-27B RVN -multilingual (Q5_K_M, 19.2 GB) runs single-card on the V100 with no tensor split at 262,144 context (up from 196,608), KV q8_0 — 28,230 MiB resident, verified loaded through /props. The V100 build carries sm_70 kernels; a numeric CUDA_VISIBLE_DEVICES would have silently retargeted, so everything is pinned by GPU-UUID.

The reranker moved back onto GPU. With 32 GB instead of the old 28 GB budget, bge-reranker-v2-m3 (FP16) no longer has to hide on the CPU to protect the generator’s context — it now shares the V100 at ~1.2 GB (repointed P100 → V100 on 2026-09-10, since the P100 UUID it was pinned to no longer exists). Generator + reranker together sit at ~29.5 GB of the V100’s 32 GB. The embedder llama-embed stays pinned to CPU (CUDA_VISIBLE_DEVICES=-1), so the RAG retrieval path still costs the generator zero VRAM contention.

A second, uncensored small-model chat tier landed on the freed 3060. Rocinante-X 12B v1 Heretic (Uncensored, IQ4_XS, ~7.6 GB) runs as rocinante.service on the RTX 3060 at :8089, 16,384 ctx, fronted by a no-JS chat UI (violet-chatui, :8093). It is independent of the RAG/generator stack: the 3060 carries the quick uncensored chat while the V100 carries the big grounded generator and the reranker, so the two never contend. (This is the interim shape while the P100 is staged to swap back into the 3060’s slot for a 48 GB all-Tesla config — prepped, not yet done.)

What changed — 2026-08-26: the switch became an and, and the boot path got tested instead of assumed

The RAG-or-chat switch is gone. Both run at once now, and the reason is one line of systemd: the reranker was moved to CPU. llama-embed was already pinned there; llama-rerank was still taking ~1,185 MiB on the P100, which was exactly enough to push qwen38 past the 28,672 MiB ceiling and into a crash-loop at full context. With both retrieval models on CPU the RAG stack costs zero VRAM, so the general-chat model keeps all 28 GB and its 196,608 context while grounded RAG serves alongside it.

The generator on :8083 is now Qwen3.8-27B Heretic RVN — the -multilingual build (Q5_K_M, 19.2 GB, 14.0 tok/s). Same abliteration and same quant as before; only the imatrix calibration differs, covering code, reasoning and tool-use structures rather than English-only text — which matters because everything downstream of this box speaks tool-calls. huihui is stopped and disabled: it claims the same :8083, and left enabled it would have won the port on the next boot.

MTP was measured and rejected. The same RVN weights ship an embedded multi-token-prediction draft head; enabling --spec-type draft-mtp bought +19% generation (12.7 → 15.1 tok/s) and cost 42% of the context window (196,608 → 114,688), because MTP allocates a second draft KV context plus a ~714 MiB compute buffer that has to fit on the 3060. A sweep run without --spec-type reported 212,992 as fine and was completely misleading — a capacity sweep has to run the same flags as the real service, or it measures a configuration you will never ship.

The boot path is now tested rather than assumed, and testing it found two things. privategpt was failing its first start on every boot and only surviving on the Restart=always retry — it calls the generator at startup to register models, and lost the race to qwen38’s ~12–15 s load; ordering cannot fix that (After= waits for started, not ready), so it now has an ExecStartPre that polls :8083. And sparse-proxy was enabled but not running — so hybrid search had silently degraded to dense-only for an unknown stretch. Nothing errored, because the pipeline is fail-open by design. Fail-open hides outages: it converts a failure into slightly worse answers, which nothing alerts on.

What changed — 2026-08-22: the RAG stack is back live, and general chat is parked

The switch flipped the other way. qwen38.service was stopped — freeing :8083 and both cards — and the full grounded-RAG stack was brought back up: container-qdrant, llama-embed, embed-proxy, sparse-proxy, llama-rerank, the huihui-Qwen3-30B-A3B abliterated generator on :8083, and rag-proxy on :8092. Verified end-to-end against the real corpus — a query through the security-corpus model id came back grounded in actual files, not just open ports.

With huihui back on :8083, rag-proxy’s GEN_MODEL matches the port again — the cosmetic mismatch noted below is gone. Placement differs from the paused-era generator: huihui runs on the RTX 3060 with its MoE experts on CPU (--n-cpu-moe 40, ~5.8 GB VRAM, 32,768 ctx), leaving the P100 to the embedder and reranker (~1.2 GB together). The box is now parked on the RAG side of the switch; general chat is stopped, not removed. (First query waited ~110 s on the MoE-to-CPU load; warm after.)

What changed — 2026-08-15 (later): the PSU mystery is solved, and the power plan collapses to one cable

pluto’s PSU is a Thermaltake Toughpower GF1 750 W, 80 Plus Gold, fully modular — identified by opening the case and reading the label, which is the paragraph further down asking “how is the P100 fed today” getting its answer the honest way. The earlier working theory (a ~450 W OEM unit, which motivated an interim 300 W GPU power cap: 3060 → 120 W, P100 → 180 W) was wrong. Wattage was never the problem; the wiring was: both cards were riding one daisy-chained PCIe run (3060 first, its pigtail feeding the P100’s dual-PCIe→EPS adapter — ~270 W of cable draw on one run at stock limits).

The fix costs nothing. The GF1 750 W ships with two 4+4 EPS CPU cables and its modular 8-pin ports are shared CPU/PCI-E; the board uses one, and the spare is in the factory accessory bag. So: spare stock EPS cable → P100 direct (its socket is EPS12V native — no adapter at all), 3060 alone on the existing PCIe run, adapter retired to the parts bin. After that rewire the caps return to stock (170 W / 250 W).

This also re-opens the V100 door the connector paragraph below had half-closed. The three-EPS-load problem is solved without a PSU purchase: the retired adapter comes back for the second Tesla, fed by one dedicated PCIe run using both of that run’s 6+2 connectors — safe, because a single 250 W Tesla only pulls ~175 W through the cable (the two plugs exist for the 150 W-per-connector spec, not for two separate runs). Or spend ~$12 on a third genuine GF1 EPS cable and run everything native. The remaining V100 gate is price alone ($624 for the 32 GB SKU), and the 3090-vs-V100 trade below still deserves the comparison — but on merits now, not on connectors.

What changed — 2026-08-15: the generator moved, the RAG stack is paused, and a LoRA got rejected

Qwen3.8-27B (dense, UD-Q5_K_XL, 19 GB) replaced huihui-Qwen3-30B-A3B as the model on :8083, running at 131,072 context across P100 + 3060 with TENSOR_SPLIT=0.60,0.40 weighted toward the P100 (262,144 OOMs). Everything downstream — aider, the /chat deck, Open WebUI — was repointed to it, and the RAG stack was stopped in the process. It is paused, not broken: five systemctl --user units away from returning.

The considered alternative — train a LoRA on the 488-book corpus and retire RAG — was rejected. A LoRA has no provenance: it cannot cite, because after training there is no “came from.” That is precisely the failure this pipeline was built to fix. Adapters move behaviour (style, format, vocabulary); retrieval moves facts, checkably. Adding a book is a re-index, not a retrain. The LoRA is parked as a future house-style layer on top of retrieval — not a replacement for it.

An uncensored Qwen3.8-27B (Heretic method — co-minimizes refusal count against KL divergence from base; Apache 2.0) now exists at Q5_K_M/MTP, 19.5 GB — a half-gigabyte from the model already running. That becomes the specialist-tier generator when the V620 lands, and it is a drop-in on pluto in the meantime.

What changed — the 2026-08-13/14 rebuild

The Tesla P100 (16 GB HBM2) is now physically installed on pluto, replacing the GTX 1080. That single swap rewrote the whole stack: the embedder went from ~5–11 chunks/s to full-rate FP16, the reranker finally has a home, and the box now hosts a 35B model entirely in VRAM. Along the way the design shifted from an agentic “orchestrator routes to specialist” model to a direct grounded-RAG pipeline — because PrivateGPT v2 turned agentic and orphaned the bulk-ingested corpus (see Grounding, the hard way). Net result: grounded security-corpus chat actually works end-to-end, served through a browser UI on the LAN.

How a query flows LIVE — STACK UP 2026-08-22

A single service — rag-proxy on :8092 — speaks the OpenAI chat API as one model id, security-corpus, and drives the whole pipeline. That means the existing web chat UI and the model-swap proxy treat grounded RAG like any other model: pick it from a dropdown.

  1. Analyst query arrives at rag-proxy (:8092) via the LAN-only /chat/ web UI — nginx reverse-proxy, RFC-1918 gated.
  2. The query is embedded two ways in parallel: dense via Qwen3-Embedding-0.6B (through embed-clamp, :8085) and sparse BM25 (:8086).
  3. Qdrant runs a hybrid query — dense + sparse prefetch, fused with Reciprocal Rank Fusion — returning the top 32 candidate chunks.
  4. The 32 candidates go to bge-reranker-v2-m3 (:8088), a cross-encoder that re-scores each (query, chunk) pair and keeps the best 5.
  5. Those 5 chunks are assembled into a grounding prompt with numbered sources and a “answer only from these sources” instruction.
  6. The generator on :8083 — Qwen3.8-27B Heretic RVN -multilingual (Q5_K_M, 19.2 GB, single-card on the V100) — streams the answer; rag-proxy appends a real Sources list of the actual file names. (The field is routed by port regardless, so llama.cpp answers with whatever occupies :8083 — since 2026-08-26 that is the general-chat model, which now serves both roles.)

Superseded — see 2026-08-26 above. Status as of 2026-08-22: this path is live. container-qdrant (:6333), llama-embed (:8082), embed-proxy (:8085), sparse-proxy (:8086), llama-rerank (:8088), the huihui generator (:8083) and rag-proxy (:8092) are all active and listening; qwen38.service is stopped and disabled. As of 2026-08-22 the seven RAG units are enabled (user-linger is on), so RAG is now the boot default — a reboot comes back on the grounded librarian, not the general-chat model. Flip back with swap-model.sh plus re-enabling qwen38.service.

Current status (2026-09-14): the RAG stack still runs as one enabled unit set — container-qdrant (:6333), llama-embed (:8082, CPU), embed-proxy (:8085), sparse-proxy (:8086), llama-rerank (:8088, now on the V100), rag-proxy (:8092), privategpt (:8001), qwen38 (:8083) and chat-relay (:8094) — alongside the independent uncensored chat tier rocinante (:8089, 3060) and its UI violet-chatui (:8093). On the V100 the generator (28,230 MiB) and reranker (~1.2 GB) sit at ~29,452 MiB of 32,768; the 3060 carries Rocinante at ~7.6 GB. (The move to a single 32 GB card is what let the generator go to 262,144 ctx and put the reranker back on GPU at the same time.)

Every stage is fail-open: if sparse or the reranker is down, retrieval degrades to dense-only / raw order rather than erroring. Verified end-to-end against the real corpus — answers cite actual books (Shellcoder’s Handbook, CEH v9, the XSS Attacks title), not hallucinated titles.

Components on pluto — live 2026-08-22

ComponentRolePortGPU
rag-proxyGrounded-RAG orchestrator; OpenAI-compatible; model id security-corpus8092— (CPU glue)
Qwen3-Embedding-0.6B (llama.cpp)Dense query/doc embeddings, 1024-dim8082CPU (pinned)
embed-clampToken-aware embedding proxy (open-sourced)8085—
sparse encoderBM25 sparse vectors for hybrid search8086—
bge-reranker-v2-m3 (llama.cpp)Cross-encoder rerank of the top-K before context8088V100 (~1.2 GB; back on GPU 2026-09-14)
QdrantVector DB — hybrid dense + BM25, RRF fusion6333—
Qwen3.8-27B Heretic RVN -multilingual (Q5_K_M) LIVEGenerator — serves both grounded RAG and general chat on :8083; 19.2 GB, 262,144 ctx, single-card (no tensor split), KV q8_0. huihui is stopped and disabled — it rivals this port.8083V100 (single card)
Rocinante-X 12B v1 Heretic (Uncensored, IQ4_XS) LIVEStandalone uncensored quick-chat tier — independent of the RAG/generator stack; fronted by a no-JS UI (violet-chatui, :8093); ~7.6 GB, 16,384 ctx8089RTX 3060
markerPDF → Markdown preprocessing (ingest only)——

As of 2026-09-14 the CPU-only retrieval tier is half-undone — on purpose. The 32 GB V100 has the headroom the 28 GB budget did not, so the reranker went back onto the GPU (~1.2 GB on the V100) while the embedder stays pinned to CPU (CUDA_VISIBLE_DEVICES=-1). The generator still keeps the whole card for weights and a 262,144-token KV cache with the reranker riding alongside. Placement is by GPU-UUID, never a numeric index: the serve scripts use ${CUDA_VISIBLE_DEVICES:-<uuid>}, and :- substitutes its default on empty as well as unset — and the P100 UUID those defaults used to name no longer enumerates, which is exactly why the reranker was repointed to the V100 rather than left to fail silently.

General chat — now the same process LIVE — 2026-08-26

There is no longer a separate general-chat mode to start. The model on :8083 answers direct chat and is the generator rag-proxy grounds against — the difference is which endpoint you hit (:8083 raw, :8092 grounded), not which model is loaded. Historically hot-swappable via llama-swap (:8090); since 2026-08-15 a single llama-server holds the card outright — and since the 2026-09-14 V100 swap that is one 32 GB card, not a two-card split. swap-model.sh now offers four targets — vanilla, heretic, heretic-ml (live) and heretic-mtp:

ModelRoleGPU footprint
Qwen3.8-27B (dense) LIVE★ The general-chat model — aider, Open WebUI (“dix”) and /chat point here when it runs. Last-loaded build was the uncensored Heretic Q5_K_M (196,608 ctx); the vanilla UD-Q5_K_XL runs at 131,072 — picked by swap-model.sh {heretic|vanilla}~19–28 GB on the V100 (single card) when active
Qwen3.6-35B-A3B-Uncensored-HauhauCS (Q4_K_M)Security specialist — uncensored + cybersecurity-tuned; thinking model, native tool-calling. Displaced 2026-08-15; staged on telesto as a V620 A/B candidate~22 GB across P100 + 3060, 65k ctx
Ternary-Bonsai-27B (Q2_0)Experimental ternary modelRTX 3060

Served to a cyberdeck-themed web chat UI (streaming, conversation history, per-chat list, image staging, copy controls, iOS-safe) at a LAN-only /chat/. A coding harness (aider) on a third node points at this :8083 general-chat endpoint over the LAN — model qwen3.8-27b at 131,072 ctx, repo-map retrieval keeping a whole codebase in-window at ~700 tokens/edit. (It followed the model: aider tracked HauhauCS until that was displaced 2026-08-15, and moved to Qwen3.8-27B on :8083 with it.)

Since 2026-09-14 there is also a lightweight, always-on uncensored option that does not touch the V100: Rocinante-X 12B v1 Heretic (Uncensored, IQ4_XS) on the RTX 3060 at :8089 (16,384 ctx), with its own no-JS chat UI (violet-chatui, :8093). It exists precisely so quick uncensored chat never has to evict the big grounded generator — the 3060 owns it, the V100 owns the RAG stack, and the two run without contention.

The specialist tier (telesto · Radeon Pro V620 32 GB) PARTS IN HAND · BUILD DEFERRED

A dedicated 32 GB ROCm security/audit node, fully sourced and staged. The card is not in the machine yet — telesto’s PCI bus still enumerates only its interim GTX 1060 6 GB, and there is no /opt/rocm. The remaining work is a chassis job: new board, new PSU, new case, OS drive cloned. Deferred by choice; nothing downstream is blocked by it (see below).

Model (staged on the NAS)Job
Qwen3.8-27B Uncensored (Heretic, Q5_K_M/MTP, 19.5 GB) TO FETCH★ Intended generator — uncensored, Apache 2.0, leaves ~12 GB for KV on a 32 GB card. The one file not yet staged. Caveat: its refusal-removal eval explicitly omitted coding tasks — A/B before trusting it here
Qwen3.6-35B-A3B-Uncensored-HauhauCS (Q4_K_M)Security specialist — uncensored and cybersecurity-tuned. Already re-quantised from the IQ2_M that was sized for a smaller card
Ornith-1.0-35B (MoE, Q5_K_M, ~25 GB)Agentic coding challenger — A/B against the incumbent
Gemma-4-31B-Heretic + mmprojUncensored generalist with vision — the future image-input path

What the V620 actually buys — and what it doesn’t. It is a capacity and availability upgrade, not a speed one. Decode is memory-bandwidth-bound, and the arithmetic is unsentimental: pluto today streams the generator’s ~19 GB single-card on the V100 at its ~897 GB/s HBM2 ≈ 21 ms/token; a single V620 at its 512 GB/s paper rate streams the same 19 GB in ≈ 37 ms/token — already the slower card on paper, and slower still at its measured ROCm rate (~74.7 tok/s on Llama2-7B Q4_0 ⇒ ~284 GB/s realised, ~55% of paper). So the V620 does not win on throughput. Its prize is the freed CUDA card for ComfyUI — moving image generation onto telesto ends the Qwen ⇔ ComfyUI exclusivity documented above. That is a good enough reason on its own; it just isn’t a throughput reason.

Telesto’s current board is a Gigabyte B85-HD3 (2014 F2 BIOS) with no Above-4G Decoding — IFR-verified absent, so a 32 GB BAR can never be mapped. The fix is a board swap, not a tuning problem. Acquired / ordered 2026-08-13: a Supermicro X10SLH-F (Intel C226, Xeon E3-1241 v3 pre-installed, IPMI/BMC), a Corsair RM1000e 1000 W PSU (dual native EPS for the Xeon + the V620’s EPS-8-pin), a GDSTIME 9733 blower (18 W, 2.0 inH2O — static pressure is the spec that matters for a sealed datacenter heatsink), an ABS-printed shroud, and a 500 GB SSD to move the OS off a 14-year-old WD Green. When it lands: aider repoints here, and bge-reranker migrates from telesto (a CUDA reranker can’t run on ROCm) — already done, it now lives on the pluto P100.

Fleet end-state PLANNED · FUNDED ~JAN 2027

The current two-node split is an intermediate state. The intended end-state gives every workload its own silicon and stops the contention that shapes today’s trade-offs:

NodeCardsJob
plutoTesla V100 32 GB IN 2026-09-14 + Tesla P100 16 GB = 48 GB P100 SWAP-BACK PENDINGCUDA LLM node — the V100 is installed (currently paired with the RTX 3060 for 44 GB); the last step swaps the 3060 out for the P100
telestoRadeon Pro V620 32 GB (ROCm) + the freed RTX 3060 (CUDA)ROCm security/audit LLM, plus ComfyUI on its own dedicated card

The two cards on telesto are not pooled — that is the point. V620 (ROCm) and 3060 (CUDA) cannot share a model, and nobody is asking them to: one runs the LLM, the other runs image generation, as independent workloads with independent runtimes. An earlier version of this plan was argued against on the grounds that “VRAM cannot be pooled across AMD and NVIDIA” — true, and irrelevant, because pooling was never the design. The PCIe-lane objection falls the same way: both workloads stream weights inside the card, so x8 costs model load time, not generation throughput.

Cost and honesty. A V100 PCIe 32 GB is ~$624 as of 2026-08-15 — six times what older planning notes assumed, because those quoted the 16 GB SKU. Worth re-checking before committing: V100 is Volta (sm_70) and is aging out of the kernel stack, the same class of problem that already bit this fleet when Pascal sm_60 hit no kernel image is available. A used RTX 3090 lands near the same money with ~936 GB/s, Ampere sm_86, working flash-attention, and it runs ComfyUI — the trade is 24 GB against 32 GB in a single card. Also note V100 PCIe has no NVLink; that is the SXM2 variant. Either way it is layer-split over PCIe, and per the arithmetic above, that costs nothing worth naming.

Power — RESOLVED 2026-08-15: the connector gate is cleared. Wattage was always the easy part: at stock limits the V100 swap means ~620 W worst-case (V100 250 W + P100 250 W + 5600X) on a supply now known to be a Thermaltake Toughpower GF1 750 W Gold, fully modular — workable for inference loads, which rarely peg both cards at TDP simultaneously, though that is the ceiling being lived near. The cable-count problem this paragraph used to end on is solved: pluto’s three EPS-8-pin loads (CPU socket, P100, V100) are covered by the GF1’s two stock EPS cables (CPU + P100 direct) plus the dual-PCIe→EPS adapter on its own dedicated PCIe run for the V100 — both of that run’s 6+2 connectors into the adapter, one 250 W card per run, every connector inside its 150 W spec. (Or ~$12 for a third genuine GF1 EPS cable and no adapters anywhere.) The P100’s feed today is the spare stock EPS cable, direct — the daisy-chained adapter era ended with the 2026-08-15 rewire. The RTX 3090 alternative above no longer wins on connectors; if it wins, it wins on bandwidth, sm_86 kernel longevity, and ComfyUI — which is the comparison worth having.

The corpus

~488 PDFs — a security/tech library of audit references and exploit-dev material — flow through marker into clean Markdown, get chunked, embedded, and indexed in Qdrant as 143,193 hybrid (dense + sparse) points. The model never “memorizes” the books; it retrieves the relevant pages at query time and answers grounded in them, with citations. Direct retrieval scores sit at a healthy 0.64–0.67 cosine; the reranker then pulls the genuinely-relevant chunk from #14 up to #1 where a limited context window can see it.

Grounding, the hard way — when the corpus goes invisible

The single most instructive failure of the rebuild: the RAG chat gave confident, well-written answers — that cited books not in the library. It was hallucinating grounding. The diagnosis chain is worth keeping.

  1. A great answer with a fake citation is worse than an error. The tell was a citation to a plausible-sounding title that wasn’t in the corpus. If retrieval were working, every citation would be a real file name. Always verify grounding against the known source set, not against how confident the prose sounds.
  2. “RAG is running” ≠ “RAG is grounding.” The reranker service was up and healthy but receiving zero requests during chat — it was serving nothing. A component being alive says nothing about whether it’s in the path.
  3. PrivateGPT v2 went agentic and per-artifact. It no longer chats over “all your docs”; retrieval is a semantic_search tool the model must choose to call, and that tool refuses to run without an ingested-artifact context. The bulk-ingested corpus predates that model — /v1/artifacts/list returned 0 artifacts. The 143k vectors existed and queried perfectly, but were orphaned from the chat path. The vectors were never the problem; the framework’s idea of “a searchable thing” had changed underneath them.
  4. The model wasn’t the problem either. Forcing the tool still produced no call — but the same model, hit directly, emitted a clean tool call on the first try. The tool definition simply wasn’t being sent, because there was no artifact to attach it to.
  5. The fix was to stop fighting the framework. Rather than re-ingest 488 PDFs through the new artifact API (hours, and a re-run of the 8×-duplication risk below), a ~230-line proxy does the pipeline directly — embed, hybrid-search, rerank, ground, stream — and presents as one OpenAI model so the existing UI just works. When the tool a framework hands you has outgrown your use case, a thin purpose-built endpoint beats bending the framework back.
  6. Block-buffered stderr hides the truth. Instrumenting the proxy showed nothing until it was run with python -u — the “0 logged ≠ stuck” gotcha, again. Unbuffered, it printed retrieve: hybrid(dense+sparse) and rerank OK: 30→5, proving the path end-to-end.

The P100 install — a GPU swap breaks more than you think

Dropping a passively-cooled datacenter card into a consumer tower surfaced a cluster of gotchas, each masquerading as something else.

  1. “Won’t POST” was a BIOS toggle. The board hung before video because CSM was enabled, which greys out Above-4G Decoding — and without Above-4G the P100’s 16 GB BAR can’t be mapped. They’re coupled: CSM off first, then Above-4G becomes settable. The headless box was diagnosed remotely by watching the UniFi controller for the machine’s Wi-Fi MAC re-associating — a network controller doubles as a boot sensor when there’s no monitor.
  2. Same-generation, different kernel. The P100 is GP100 = sm_60; the retired GTX 1080 was GP104 = sm_61. Every local llama.cpp build targeted 61;86 and died on the P100 with no kernel image is available for execution on the device — it allocates VRAM fine, then fails at first kernel launch. Rebuilt with -DCMAKE_CUDA_ARCHITECTURES="60;61;86". (CUDA 13 dropped Pascal entirely, so this pins to CUDA 12.9.)
  3. The card throttled to 45% clock — and it was a fan curve, not cooling. Under load the P100 hit 80 °C in under a minute and clocked from 1328 to 594 MHz. It had a blower. The blower was on a chassis fan header whose curve tracks CPU temperature — and the CPU sits on a cold AIO, so the board idled the fan at ~30% while the GPU cooked. Nothing in the loop knew the GPU existed. Pinning the header to 100% dropped peak-load temp 80 → 60 °C and held full boost indefinitely. Lesson: any passive datacenter GPU in a consumer board has an open control loop — wire its fan to constant 12 V, never a temp-driven header.
  4. The power connector is the one mistake with no undo. Both the P100 and the V620 take an EPS-12V 8-pin (CPU-style), not PCIe 6+2 — identical shells, incompatible pinouts. EPS is 4×GND + 4×+12V; PCIe 8-pin is 3×+12V + 5×GND, two of them sense pins. Feed a PCIe pinout into an EPS socket and +12 V lands on pins the card treats as ground. There is no protection circuit for that. The adapter here was metered before first power-on rather than trusted, which is the correct order of operations on a card you cannot afford to replace twice. Two corollaries: a single PCIe 8-pin is rated 150 W and a V620 pulls 300 W, so even a correctly-wired adapter off one PCIe lead is a thermal problem rather than an electrical one — use a native EPS output (the RM1000e ships two for exactly this reason). And modular PSU cables are not interchangeable between units: Corsair, Seasonic and EVGA use different PSU-side pinouts behind identical sockets, so old cables retire with the old supply.
  5. The swap silently broke every service that named the old card. Scripts using a numeric CUDA_VISIBLE_DEVICES=1 kept pointing at “device 1” — which used to be the 1080 and was now the P100 — and services sized for the 1080’s 8 GB OOMed on the P100. Numeric indices silently retarget when hardware moves; pin by GPU-UUID instead.

Lessons learned — getting a code-heavy corpus through a fixed-context embedder

Backfilling the security/programming library surfaced a set of non-obvious gotchas about local embeddings. The fix that fell out — a transparent, token-aware embedding proxy — is open-sourced as embed-clamp.

  1. A batch of embeddings shares one context budget. With the client batching ~10 chunks per request, the server returned request (17821 tokens) exceeds the available context size (8192 tokens) — even though each chunk fit on its own. llama.cpp counts the whole batch against n_ctx. Your effective per-input budget is n_ctx / batch_size, not n_ctx; batch size 1 is the safe default when inputs run near the limit.
  2. The real wall is atomic chunks, not batching. Single unbroken blocks — code listings, opcode tables, hex dumps — reached 22k–30k tokens. No batch or context setting helps: an 8 GB embedding card can't hold a 30k-token sequence, and the chunker won't break the block.
  3. RAG chunkers treat code/tables as atomic and bypass the token splitter. “Markdown-aware” parsers split on headings/sentences and token-limit the pieces — except tables (parsed to dataframes) and fenced code are emitted as single atomic nodes that skip the splitter entirely. The token limit you think you set never applies to exactly the blocks that overflow.
  4. Re-splitting markdown at the source is fragile. Re-wrapping a giant code block into smaller fenced blocks desyncs any parser that toggles on ``` lines — code content often contains backtick runs. Always verify-or-revert: re-segment the output and restore the original if any block is still over the ceiling. Never leave a document worse than you found it.
  5. The robust fix lives at the embedding boundary, not the chunker. Rather than chase every atomic-node bypass path, put a proxy in front of the embedder: pass normal inputs through byte-identical, and for oversized ones split → embed pieces → pool into one vector — model- and pipeline-agnostic, no source mangling. Keep pass-through exact (cosine ≈ 1.0), give the embedding client a long timeout (pooling = several sequential embeds), and don't tokenize per line — calibrate a char budget so the GPU stays the bottleneck. (That tool is embed-clamp.)
  6. Operational gotchas. Don't poll the vector DB under heavy ingest load — counts time out and read inconsistently, and each poll adds load; trust total-point growth or the run’s summary line instead. exact:false counts over-report. Block-buffered stdout hides progress (“0 logged” ≠ stuck). And a client-side timeout isn’t data loss — a server that streams vectors as it embeds may finish a request the client already abandoned.

Lessons learned — recovering a corrupted ingest (the 8× duplication)

A second backfill pass went wrong in an instructive way: an unattended ingest loop quietly re-embedded the same books over and over until Qdrant held ~1.74M vectors for ~250 documents — a median of ~7,000 points per book, the worst at 35,724. Pulling that apart surfaced a different class of gotcha than the embedder ones above — idempotency, runaway processes, and old-GPU limits.

  1. Idempotency has to be against the store, not your own bookkeeping. The ingester deleted a document’s old vectors only when it was already in the local manifest. A doc that succeeded server-side but whose client call timed out got recorded as failed — so the next retry wave re-ingested it without deleting first, appending a second full copy. Across ~40 waves that compounded to 8×. Fix: purge-before-ingest unconditionally (delete the artifact’s vectors every time — a cheap no-op when absent), and cap retries with a quarantine so a doc that simply can’t ingest stops being re-chunked forever.
  2. Detached children outlive the parent you killed. The loops were launched with setsid, so killing the orchestrator left the ingest driver (and its worker grandchildren) running — three separate escaped loops over one session, each still duplicating. Always pgrep for stray drivers before and after any rebuild step; a dead parent is not a stopped job.
  3. You can’t cheaply de-duplicate re-ingests. The duplicate vectors weren’t byte-identical copies — the source .md had been re-split between runs, so chunk boundaries drifted and the “duplicates” were distinct vectors (32,981 points / 31,124 unique on one book). Nothing to delete cheaply; the only correct fix is wipe the collection and re-embed from the current text.
  4. Pascal is the wall. The interim embedder ran f16 on a GTX 1080 (Pascal, sm_61) at ~5–11 chunks/s, and batching did nothing — llama.cpp serialized the batch and Pascal’s f16 rate is ~1/64. Worse, the modern stack has dropped Pascal: vLLM/TEI won’t run on sm_61, and marker’s Torch (cu130) has no Pascal kernels at all (no kernel image is available for execution on the device) — so you can’t even shard the PDF→Markdown sweep onto that card. The same model on the Ampere RTX 3060 hits ~27 chunks/s and does scale with batch. Confirm a card can run your kernels before you plan around it. (Since resolved: the P100’s full-rate FP16 removes this wall for the embedder.)
  5. The real speed lever is a properly-batching backend. llama.cpp’s embedding endpoint is effectively one-sequence-at-a-time; swapping to vLLM serving the same Qwen3-Embedding on the Ampere card — with requests actually batched on-GPU — is what turns a ~day-long re-embed into a few hours.
  6. Trust the measurement, not the process. The bloated collection looked healthy by every process check; only counting points-per-document revealed the 8×. Gate “done” on an audit — each doc’s point count near its real node count (none at a multiple), a sane total, and a vector-hash duplicate scan — and start every rebuild from an empty collection with the manifest reset (wipe without resetting the manifest and the ingester skips everything, leaving you empty).