Zum Hauptinhalt springen
tsecurity.de LIVE
Echtzeit-Radar & Feeds
Alle RSS Feeds
👥 Community & Social
Sichere ProgrammierungFliproom: a room changeover is a content problem(21.09.2026 um 04:10 Uhr)
Sichere ProgrammierungNova Adiutrix: My Second Agent Built My First Project's To-Do List(21.09.2026 um 04:11 Uhr)
IT Security ToolsAntiphishing v35456910988(21.09.2026 um 02:35 Uhr)
IT Security Toolsbrave-browser v1.98.12(21.09.2026 um 03:35 Uhr)
Sichere ProgrammierungFliproom: a room changeover is a content problem(21.09.2026 um 04:10 Uhr)
Sichere ProgrammierungNova Adiutrix: My Second Agent Built My First Project's To-Do List(21.09.2026 um 04:11 Uhr)
IT Security ToolsAntiphishing v35456910988(21.09.2026 um 02:35 Uhr)
IT Security Toolsbrave-browser v1.98.12(21.09.2026 um 03:35 Uhr)
Intelligence View
⚡ tsecurity.de Intelligence

How we built a hybrid FTS5 + embedding search for code — and why you need both

Reagiere als Erste:r — dein Feedback zählt!

How we built a hybrid FTS5 + embedding search for code — and why you need both

srclight is a deep code indexing MCP server — it gives AI agents understanding of your codebase (symbol search, call graphs, git blame, semantic search) in a single pip install.

When you're building AI coding assistants, you need search that works two ways:

  1. Keyword search — I know the function name, find it now
  2. Semantic search — find code that "handles authentication" without knowing the exact term

Most tools pick one. We built both.

The problem with pure keyword search

FTS5 is great for exact matches. But code has naming conventions: calculateTotalPrice, calculate_total_price, CalculateTotalPrice. A single FTS5 index can't handle all of these well.

And sometimes you don't know the name at all. You want to find "code that validates user input" — that's a concept, not a keyword.

The problem with pure embedding search

Embeddings are great for meaning. But they struggle with:

  • Exact symbol names (searching for handleAuth should find handleAuth)
  • Substring matches (searching for parse should find parseJSON)
  • Short queries (embeddings need context)
  • Naming conventions

Our solution: 4 indexes + RRF fusion

We built three FTS5 indexes, each tuned differently:

1. Symbol names index (unicode61 tokenizer)

Splits on case changes and underscores:

calculateTotalPrice → calculate, Total, Price
handle_user_auth → handle, user, auth

This catches CamelCase, snake_case, and any convention developers throw at it.

2. Source content index (trigram tokenizer)

Indexes every 3-character substring. This catches substring matches even inside words.

3. Docstrings index (porter stemmer)

Stems words to their roots: "running, ran, runner → run". This makes docstring search actually useful.

4. Embeddings (via Ollama)

Semantic vectors for meaning-based matching. We use qwen3-embedding (4096 dims) or nomic-embed-text (768 dims).

The secret sauce: Reciprocal Rank Fusion

Here's how we combine them. We run each query against all 4 indexes, get ranked results, then merge using RRF:

RRF_score(d) = Σ 1 / (k + rank(d))

where k = 60 (standard constant).

A result appearing at rank 1 in FTS5 and rank 2 in embeddings gets:

  • FTS5: 1 / (60 + 1) = 0.0164
  • Embeddings: 1 / (60 + 2) = 0.0161
  • Total: 0.0325

A result at rank 10 in embeddings only gets: 1 / (60 + 10) = 0.0143

This means exact matches can still win even if embeddings also match — and vice versa. You get the best of both worlds.

But wait, there's more

We also built:

  • GPU vector cache: Embeddings loaded to VRAM once (~300ms cold), then ~3ms per query via CuPy
  • Incremental indexing: Only re-index changed symbols (tracked via content hash)
  • Git intelligence: Query "what changed recently?" → git blame, hotspots, uncommitted WIP
  • Multi-repo workspaces: SQLite ATTACH+UNION across 10+ repos

Why not just use Elasticsearch?

We wanted something that installs in one command:

pip install srclight
srclight index --embed qwen3-embedding
srclight serve

No JVM, no Docker, no Redis, no cloud. Your code never leaves your machine.

Results

We index 13 repos (45K symbols) in a workspace. Claude Code goes from ~20 tool calls per task to about 6 — because it can just ask "who calls this?" instead of grepping 10 times.

The hybrid search is the key. Keyword matches for precision, embeddings for recall. RRF fusion brings them together.

What search challenges are you running into with AI coding assistants? Drop a comment — I'd love to hear what's blocking you.

Ähnliche Beiträge
🔍 Verwandte News

Auch interessante Nachrichten How we built a hybrid FTS5 + embedding search for code — and why you need both

Thematisch verwandte Begriffe: built, hybrid, FTS5, embedding · 6 Treffer

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Laden...

Beiträge werden geladen ...

Laden...

Videos werden geladen ...

Zum Aktualisieren ziehen
ZERO-DAY CVE-2026-93977 | A vulnerability was determined in code-projects Assessment Management 1.…
Advisory →
TTS Reader • tsecurity.de Voice
tsecurity.de Icon
tsecurity.de App
Offline-Lesen, Eilmeldungen & 0ms Ladezeit

Installiere tsecurity.de direkt auf deinen Home-Bildschirm für das ultimative Vollbild-Magazinerlebnis ohne Browser-Leisten.

Nächster Beitrag
Themen-Radar & Intelligence Matrix
Echtzeit-Taxonomie nach Angriffsvektoren & Plattformen

tsecurity.de Live Threat Radar

🔴 LIVE RADAR
MONITORING
AKTIV
CVE-DATENBANK
LIVE
🔍
Community Radar & Live Chat
Sentinel Bot online • Live-Stream
Dein Cluster: Security Explorer
Match:
lädt…
Verbindung zum Community-Stream wird aufgebaut...
Bearbeitungsmodus — Senden überschreibt deine Nachricht
Community-Puls — was gerade passiert
lädt…
Aktivitäten deiner Analysten
lädt…
Neues Thema oder Eilmeldung einreichen

Reiche interessante Links, Zero-Days oder Debatten ein. Die Community entscheidet per Upvote über die Veröffentlichung.

Heiß diskutierte Einreichungen
🔖 Gespeicherte Artikel
📂 Keine gespeicherten Artikel vorhanden.
Zurück Ziehen Vor
Links: vorheriger Artikel Rechts: nächster Artikel unten: schließen
News NIS-2 Frühwarnung Tier-1 Intel ⏱️ 3 Min vor 10 Min
Artikeldaten werden geladen...

Zurück: vorheriger Vor: nächster
↗ Original-Quelle
Social Reaktionen Deine Reaktion zählt
Einstufung & Relevanz-Poll 0 Stimmen
In sozialen Netzwerken teilen 1-Klick