Show Dev: Self-reinforcing K-pop data pipeline using Spring Boot and pgvector (Built on OCI Free Tier)

Hi everyone,

I'm a backend developer based in Seoul. I built k-cosmos, an interactive web-based 3D music space that maps K-pop tracks based on 768-dimensional vector embeddings.

The main reason I had to build this from scratch is that there's no clean, structured K-pop metadata or emotional tag dataset available anywhere.

How the pipeline grows itself

It runs on an autonomous background sync cycle. First, the system ingests tracks and uses an LLM to analyze the mood and aesthetic. Then, the AI reverse-engineers low-latency search keywords based on that analysis. These keywords are absorbed back into the database to fuel the next day's ingestion scheduler, allowing the system to expand its data footprint without human intervention.

Architectural decisions under hard constraints

Since I am running everything on the OCI free tier with around 4,000 tracks, I had to resolve several performance bottlenecks at the database and thread layer.

1. Preventing HikariCP starvation
   Waiting for slow external network I/O (Gemini API, Wikipedia) inside a DB transaction is a severe anti-pattern that leads to connection pool exhaustion. I decoupled the transaction boundaries using TransactionTemplate into three tight phases:

Phase 1 (Short TX): Claim the target track via FOR UPDATE SKIP LOCKED and immediately flip the status to PROCESSING to isolate rows for worker concurrency. Commit and release the connection.

Phase 2 (Zero TX): Perform the heavy external network I/O and embedding generation while holding zero active DB connections.

Phase 3 (Short TX): Open a short transaction to persist the final structured entity data.

The entire flow runs over Java 21 Virtual Threads to minimize scheduling overhead during I/O wait states.

1. Diversity-enforced vector search
   Raw pgvector cosine similarity search caused an echo chamber where a single popular artist's massive discography monopolized the recommendation coordinate space. To guarantee diverse digging exploration, I moved a 2-stage window function directly into the database layer rather than post-processing in application memory:
   

WITH candidates AS (

    SELECT *, embedding <=> CAST(:embedding AS vector) AS distance

    FROM cosmos_tracks

    WHERE status = 'COMPLETED' AND cluster_id = :clusterId AND id NOT IN (:excludeIds)

    ORDER BY embedding <=> CAST(:embedding AS vector) LIMIT :poolSize

),

diversified AS (

    SELECT *, ROW_NUMBER() OVER (PARTITION BY artist ORDER BY distance) AS artist_rank

    FROM candidates

)

SELECT * FROM diversified WHERE artist_rank <= :maxPerArtist ORDER BY distance LIMIT :limit

This preserves index efficiency while strictly scattering artist density with a single roundtrip.

1. Defensive budgeting
   The self-reinforcing loop naturally causes exponential data growth, which easily threatens free-tier LLM quotas. To control this, the engine flattens and randomizes the entire task grid (Collections.shuffle()) every midnight, distributing budget queries evenly across all latent moods before hitting the daily hard ceiling.

I deliberately chose a Thymeleaf SSR hybrid architecture to keep the deployment unit single and maintain high operational visibility (P6Spy, Actuator) instead of splitting into a separate SPA.

Live Project: https://cosmos.codeghost.cloud/

I'm very happy to discuss any architectural or design decisions. Let me know your thoughts or hit me with any questions!