Stage 0: Reactive

The site goes down, someone scrambles to fix it, the cycle repeats. No on-call rotation. No dashboards. Alerts are emails nobody reads.

Characteristic phrase: 'We'll look at that after launch.'

Stage 1: Foundation

On-call rotation exists. Alerts route to a paging tool. Basic dashboards for CPU, memory, error rate. Post-mortems happen sometimes.

Characteristic phrase: 'Did anyone see that spike last night?'

Stage 2: Measured

SLOs defined for critical services. Error budgets tracked. Alert volume is monitored and pruned. Post-mortems are written and reviewed.

Characteristic phrase: 'We're at 80% of our error budget for the quarter.'

Stage 3: Automated

Runbooks exist for top alerts. Toil is measured and reduced. Deployment pipeline has automatic rollback. Chaos engineering is practiced.

Characteristic phrase: 'The auto-rollback caught it.'

Stage 4: Predictive

Anomaly detection catches issues before alerts fire. Capacity planning is data-driven. New services have SLOs and dashboards at launch, not after. AI/ML assists incident response.

Characteristic phrase: 'We caught that before customers noticed.'

Where most teams are

Most teams I've worked with are at Stage 1 or Stage 2, trying to get to Stage 3. The jump from 2 to 3 is the hardest — it requires sustained investment with no immediate crisis to justify it.

The trap

Don't try to skip stages. Teams that install ML anomaly detection at Stage 0 just have prettier chaos. Get the foundation right first. Then automate. Then predict.

The highest maturity team I've seen was boring. Almost nothing broke. The engineers had time to work on interesting problems. That's the goal.

Written by Dr. Samson Tanimawo
BSc · MSc · MBA · PhD
Founder & CEO, Nova AI Ops. https://novaaiops.com

I recently became a maintainer, and wanted to write up what I've contributed so far and what I learned building it.

PR #50: Multiplayer support
toofan was singleplayer-only until now. I reached out to Amit (vyrx-dev), the creator and owner, and he was on board with adding multiplayer, as long as it didn't compromise the core philosophy of the project: minimal, fast, and local-first.

Here's how it works:
** Protocol and hosting **
I used an SSE-based protocol and self-hosted the server myself, listening on port 8525.

** Server logic (Go) **
When a player joins a room, the server tracks it. Once enough players have joined based on what the host selected, the match starts automatically.

** Anti-cheat **
The server owns and distributes the word list for every race. Clients never generate or supply their own words, this closes off the obvious vector for injecting an easier list to fake a high WPM.

** Client integration **
Wired all of this into the existing toofan TUI so multiplayer feels native, not bolted on.

** AI ghost racers (also in PR #50) **
When no one's online, you can race bots instead. Their progress renders as ghost carets on your screen in real time, and you can race against multiple bots simultaneously, not capped at one.

** PR #51: Storage consolidation **
Consolidated how race results were being stored into a more efficient format. This closed an issue Amit had opened himself (#55).

** PR #52: Race your past self **
Every race gets logged to a single races.txt. A ghost caret renders your previous attempt alongside your current one, so you can literally race your own growth over time and see if you're actually improving.

** What I learned: **
Contributing to toofan pushed me into a few areas I hadn't gone deep on before:
Real-time networking (SSE, connection handling, race-state sync)
Self-hosting and running a small always-on server
UX design for a TUI, this is a different medium than web, spacing and real-time feedback matter a lot more
Maintaining someone else's philosophy for a project while still shipping meaningful features

Planning to stay on as a long-term contributor. If you're into terminal tools, Go, or just want to race your friends at typing, check it out: repo link

Why modern websites must communicate with AI—not just rank in search results.

**

AI isn't replacing SEO.

It's changing what it means to build a website that's truly understandable.

Developers Are Optimizing for Google. AI Is Watching Something Else.

Why modern websites must communicate with AI—not just rank in search results.

For years, developers have built websites with two audiences in mind:

• 👤 Humans
• 🔎 Search Engines

Today, there's a third audience we can no longer ignore.

🤖 AI Systems.

As AI assistants become a common way to discover information, websites are no longer judged only by how they look or where they rank.

They're increasingly evaluated by how well they communicate meaning.

That shift changes how we should think about web development.

A Realisation While Building My Portfolio

Over the past few weeks, I've been rebuilding and optimising my personal portfolio.

Like many developers, I started with familiar priorities:

• Better UI
• Faster loading
• Responsive layouts
• Lighthouse improvements
• Technical SEO

But while implementing features like Schema.org structured data, Open Graph metadata, canonical URLs, XML sitemaps, and author information, I noticed something interesting.

None of those improvements were primarily for visitors.

Most users would never even know they existed.

Yet they dramatically changed how machines understood my website.

That made me realise something.

We're entering a web where discoverability isn't only about search engines anymore.

It's increasingly about helping AI understand context.

Search Engines Retrieve Information. AI Tries to Understand It.

Search engines traditionally answer one question:

Which pages are most relevant?

Large Language Models attempt to answer another:

Which information is trustworthy enough to explain?

Those questions sound similar.

They aren't.

Traditional search focuses on retrieval.

AI focuses on understanding.

User Question
      │
      ▼
 Search Engine
      │
Find Relevant Pages
      │
      ▼
 Click Result


User Question
      │
      ▼
 AI Assistant
      │
Understands Context
Evaluates Relationships
Summarises Information
      │
      ▼
Generated Response

Ranking is only part of the second process.

Understanding becomes the real challenge.

Google Reads Pages. AI Builds Relationships.

When people visit your portfolio, they instantly recognise:

• your photo
• colours
• animations
• layout
• navigation

AI doesn't experience your website that way.

Instead, it builds relationships.

Rajshree
      │
      ├── Software Engineer
      ├── MERN Stack Developer
      ├── AI & ML
      ├── Portfolio
      ├── GitHub
      ├── LinkedIn
      ├── Medium
      └── Technical Articles

The stronger and more consistent these relationships become across platforms, the easier it is for machines to understand your professional identity.

That's one reason why consistency matters far beyond branding.

Identity Is Becoming Structured Data

Think about where your professional identity exists today.

• Your personal website
• GitHub
• LinkedIn
• Medium
• Dev.to
• Hashnode
• X

Every platform tells a small part of your story.

When those stories align, search engines—and increasingly AI systems—gain confidence in understanding who you are.

When they don't, ambiguity increases.

That's exactly why technologies like Schema.org have become so valuable.

Here's a simple example.

{
  "@context": "https://schema.org",
  "@type": "Person",
  "name": "Rajshree",
  "jobTitle": "Software Engineer",
  "url": "https://rjshree.com",
  "sameAs": [
    "https://github.com/itsrjshree",
    "https://linkedin.com/in/rjshree",
    "https://dev.to/rjshree",
    "https://rjshree.hashnode.dev"
  ]
}

Visitors will never notice this.

But machines will.

This small piece of structured data answers several important questions instantly.

• Who owns this website?
• Which profiles belong to the same person?
• What is this person's profession?
• Which website is the official one?

That isn't just SEO.

That's communication.

The Shift From Keywords to Entities

For years, SEO conversations revolved around keywords.

Developers asked questions like:

Which keyword should I target?

Today, another concept has become increasingly important:

Entities

An entity is something uniquely identifiable.

Examples include:

• React
• MongoDB
• JavaScript
• Google
• OpenAI
• Rajshree

Notice something.

Those aren't merely words.

They're concepts with relationships.

Instead of matching keywords, AI attempts to connect knowledge.

Rajshree
     │
     ├── Profession → Software Engineer
     ├── Skills → MERN Stack
     ├── Skills → AI & ML
     ├── Website → rjshree.com
     ├── Writes → Technical Articles
     └── Builds → Web Applications

That's a much richer understanding than simply counting keyword occurrences.

Invisible Layers Matter More Than Ever

When developers showcase a portfolio, the focus usually stays on what users can see.

• Animations
• Typography
• Colours
• Hero sections
• Glassmorphism
• 3D effects

Those certainly improve user experience.

But another layer quietly powers modern discoverability.

Modern Website

Visible Layer
──────────────
✔ UI
✔ Images
✔ Typography
✔ Animations
✔ Content

Invisible Layer
──────────────
✔ Structured Data
✔ Metadata
✔ Open Graph
✔ robots.txt
✔ XML Sitemap
✔ Canonical URLs
✔ Semantic HTML
✔ Performance Signals
✔ Author Identity

Ironically, AI spends far more time analysing the invisible layer than admiring your animations.

A beautiful portfolio with poor structure is like a beautifully designed book without a title, author, or table of contents.

People may still appreciate it.

Machines struggle to understand it.

A Practical Example

While rebuilding rjshree.com, I realised that most improvements I made weren't visual.

Visitors probably wouldn't notice when I:

• added Schema.org structured data,
• connected my professional profiles using sameAs,
• improved Open Graph metadata,
• configured canonical URLs,
• generated an XML sitemap,
• refined robots.txt,
• or optimised page performance.

Yet these changes significantly improved how machines could interpret my website.

That experience completely changed how I think about modern web development.

Building a website isn't only about designing interfaces anymore.

It's also about communicating information clearly—to humans, search engines, and increasingly, AI systems.

Does This Mean SEO Is Dead?

Absolutely not.

Good SEO has always been about reducing ambiguity.

Every technical decision answers questions that machines would otherwise have to guess.

A title tag answers:

What is this page about?

A canonical URL answers:

Which version is the original?

Schema.org answers:

Who created this content?

A sitemap answers:

Which pages exist?

Semantic HTML answers:

How is this information organised?

None of these practices became obsolete because AI arrived.

In fact, they're becoming even more valuable.

Because AI depends on structured, well-organised information to understand websites accurately.

Authority Isn't Built With Backlinks Alone

Backlinks still matter.

They always will.

But they're no longer the entire story.

Imagine discovering the same developer through:

• 🌐 Their portfolio
• 💻 GitHub
• 💼 LinkedIn
• ✍️ Medium
• 👨‍💻 Dev.to
• 📝 Hashnode
• 🐦 X

Every platform reinforces the same identity.

The same expertise.

The same projects.

The same writing style.

The same technical interests.

That consistency becomes a trust signal—not only for people, but increasingly for machines as well.

Authority is no longer built through a single website.

It's built across an ecosystem.

Build an Ecosystem, Not Just a Portfolio

Many developers think of a portfolio as the final destination.

In reality, it should become the centre of your digital identity.

                  Personal Website
                         │
        ┌────────────────┼────────────────┐
        │                │                │
        ▼                ▼                ▼
    GitHub          LinkedIn         Technical Blog
                                         │
                           ┌─────────────┴─────────────┐
                           ▼                           ▼
                        Dev.to                    Hashnode
                           │
                           ▼
                           X / Social Platforms

Each platform strengthens the others.

Your articles reference your portfolio.

Your portfolio references your GitHub.

Your GitHub links back to your website.

Your LinkedIn showcases your writing.

Instead of isolated profiles, you create a connected ecosystem.

That's how digital authority compounds over time.

What You Can Do Today

You don't need a massive redesign.

Small improvements can make a meaningful difference.

Technical Foundation

• [ ] Add meaningful page titles
• [ ] Write unique meta descriptions
• [ ] Configure Open Graph tags
• [ ] Add canonical URLs
• [ ] Generate an XML sitemap
• [ ] Create a clean robots.txt
• [ ] Optimise Core Web Vitals

Structured Understanding

• [ ] Implement Schema.org JSON-LD
• [ ] Add Person and WebSite schema
• [ ] Connect your social profiles using sameAs
• [ ] Use semantic HTML
• [ ] Maintain a logical heading hierarchy

Build Trust

• [ ] Keep your name consistent everywhere
• [ ] Link all professional profiles together
• [ ] Publish technical articles regularly
• [ ] Keep projects updated
• [ ] Use the same profile photo and branding

These changes may not transform your UI.

But they dramatically improve how your website is understood.

The Biggest Mistake Developers Still Make

Many developers spend weeks perfecting visual effects.

✨ Glassmorphism

✨ Animated backgrounds

✨ 3D models

✨ Fancy transitions

There's nothing wrong with those.

But after opening dozens of portfolios, one question often remains unanswered.

"What exactly does this person do?"

That's not a design problem.

It's a communication problem.

The best websites don't impress because they're complicated.

They impress because they're clear.

Visitors—and machines—should understand your expertise within seconds.

Final Thoughts

The web has always evolved.

We moved from static HTML pages…

to responsive websites…

to mobile-first design…

to performance optimisation…

to technical SEO…

Now we're entering another transition.

One centred around understanding.

The websites that thrive in the coming years won't simply be the fastest or the most visually impressive.

They'll be the ones that communicate clearly.

To users.

To search engines.

And increasingly…

To AI.

                 Great Website

             👤 Human Friendly
                     +
          🔎 Search Engine Friendly
                     +
             🤖 AI Understandable

               = Future Ready

Technology will continue to evolve.

Frameworks will change.

Search algorithms will improve.

AI models will become more capable.

But one principle is unlikely to change:

Build websites that are easy to understand—not just easy to find.

References

• Google Search Central – https://developers.google.com/search
• Schema.org – https://schema.org
• Google Structured Data Documentation – https://developers.google.com/search/docs/appearance/structured-data
• Open Graph Protocol – https://ogp.me/
• web.dev – https://web.dev/
• W3C HTML Specification – https://html.spec.whatwg.org/

About the Author

Hi, I'm Rajshree, a Software Engineer passionate about building modern web applications with the MERN stack while exploring AI, machine learning, technical SEO, and web performance.

I enjoy building projects, writing about emerging technologies, and sharing practical insights that help developers build for the future.

🌐 Portfolio: https://rjshree.com

💼 LinkedIn: https://linkedin.com/in/rjshree

💻 GitHub: https://github.com/itsrjshree

If you found this article useful, I'd love to hear your thoughts in the comments. What role do you think AI will play in the future of web discoverability?
If you enjoyed this article, let's connect and continue the conversation about modern web development, technical SEO, AI, and building a future-ready web.

Seriously, look at it.

          .-----------------.
         /     _________     \
        /    .'         '.    \
       |    /             \    |
       |   |               |   |
       |   |   Mac Pro     |   |
       |   |   6,1         |   |
       |   |  "Trash Can"  |   |
       |    \             /    |
        \    '._________.'    /
         \                   /
          '-----------------'

It is black. It is shiny. It is a perfect cylinder of aluminum. It is, without a doubt, one of the most stunning pieces of industrial design ever to sit on a desk. This is the Mac Pro 6,1, launched in late 2013 and famously dubbed the "Trash Can." Jony Ive’s design team wanted to completely reinvent the professional workstation. They packed it with dual workstation GPUs, a single massive thermal core, and a chassis so compact it was barely larger than a coffee pot.

And... it was a monumental commercial disaster.

Let's be completely honest: by today's standards, this machine is not very capable. We are thirteen years past its launch date. A modern $150 mini PC can easily outrun it while drawing a fraction of the power. Under load, this cylinder consumes so much electricity that it basically doubles as a small space heater in my room.

But I do not care. I absolutely love it.

I still prefer running my entire homelab on this machine over any boring gray box or cloud instance. Why? Because of how incredible it looks sitting right there on my desk or next to the TV. When it hums in the corner, with its soft light illuminating the ports on the back, it feels like I have a piece of hardware from the future—a miniature, sleek datacenter right inside my living room.

Apple boxed themselves into a corner. The thermal design assumed the industry would move toward dual, lower-power graphics cards. Instead, the industry went the exact opposite direction: single, massive, power-hungry GPUs. The Mac Pro couldn't handle the heat. It couldn't be upgraded. It sat frozen in time for six years while Apple stayed silent. It got so bad that in 2017, Apple executives held an unprecedented meeting to apologize to professional users, admitting the thermal core design was a dead end.

But here is the beautiful irony of technology: a failed workstation makes for a legendary home server.

I bought mine for around $200 USD.

This is the story of how I took a piece of Jony Ive's sculpture, wiped macOS, installed a headless Linux OS, upgraded the hardware to its absolute physical limits, and turned it into an automated, Dockerized app-hosting powerhouse.

The Hardware: Swapping Xeons and Shoving in 64GB of RAM

When this shiny cylinder first arrived on my desk, the specs were looking pretty dated:

• CPU: 4-core Intel Xeon E5 processor (v2)
• RAM: 12GB of DDR3 RAM
• Storage: A proprietary 196GB Apple PCIe SSD

That might be fine for a basic file server, but I'm hosting complex web applications, media streaming platforms, databases, and monitoring tools. I need power.

So, I did what Apple said you couldn't do: I upgraded it myself.

Taking apart the Mac Pro 6,1 is an absolute joy. You slide a release switch on the back, and the outer aluminum sleeve slides right off. Underneath, you are greeted by the triangular thermal core. It is a masterpiece of compact engineering. The motherboard, graphics cards, and CPU are mounted to three sides of a triangular heatsink, with a single large fan at the top drawing heat upward.

Here is how I upgraded the guts of this machine:

1. The CPU swap: 4 cores to 10 cores

I removed the heatsink mounts, cleaned off the old thermal paste, and replaced the weak 4-core processor with a massive 10-core Intel Xeon CPU. Suddenly, I went from 8 execution threads to 20 execution threads. For multitasking and handling background jobs, this is a night-and-day difference.

2. The RAM: 64GB of ECC memory

I replaced the old 12GB of RAM with 64GB of ECC DDR3 RAM. Here is a pro-tip: because this machine uses older DDR3 ECC server memory, you can buy it on eBay for next to nothing. Corporate data centers decommission their old servers and dump this high-grade ECC RAM in bulk. I got 64GB of enterprise-grade, error-correcting memory for the price of a cheap dinner. ECC is vital because it prevents silent data corruption, keeping my databases and files safe from memory-level bit flips.

3. The cache SSD: 2TB NVMe

The proprietary Apple SSD connector is a pain, but you can buy a simple Sintech M.2 NVMe adapter online for about $15. I plugged that adapter in, and installed a fast 2TB NVMe M.2 SSD. Now we have high-speed, local solid-state storage.

4. The GPUs: Dual AMD FirePro D300s

These graphics cards are integrated into the proprietary board design and cannot be upgraded. But for this build? I don’t care. I'm running a headless server, not rendering 3D models or playing games. The GPUs sit completely idle, drawing minimal power (well, what can be called minimal for this pc), while the Xeon CPU does all the heavy lifting.

The Storage monster: 24TB array + NVMe caching

Of course, 2TB of internal storage isn't enough to hold an entire digital life. To expand it, I hooked up a high-speed USB hub loaded with a massive 24TB Hard Drive array (which I plan to expand to 48 since the hub has 2 slots ;).

But there’s a catch. Reading high-definition media files from spinning hard drives over a USB hub introduces latency. If multiple users are accessing files, it can cause buffering. To solve this, I designed a caching system: the massive files live on the 24TB USB storage, but I use the ultra-fast internal 2TB NVMe SSD to cache active files, making sure Unlimited Blades Work respond instantly.

Why I rejected Jellyfin

When developers start setting up a home server, the first thing they usually do is download ready-made solutions. If you want a media server, you install Plex or Jellyfin. If you want a video conferencing tool, you use Discord or Zoom. If you want photo storage, you pay for iCloud or Google Photos.

But I opted out.

"Using off-the-shelf software is great if your only goal is utility. But if your goal is learning, personal growth, and becoming a top-tier engineer, using someone else's pre-packaged service is a missed opportunity."

Instead of using Jellyfin, I decided to build my own media streaming platform from scratch: Unlimited Blades Work. Instead of using Zoom or Jitsi, I built my own videochat platform: Conference Room.

Why? Because writing your own custom apps forces you to solve real-world system architecture problems. You have to learn:

1. How to stream raw video files over HTTP chunk by chunk.
2. How to implement low-latency WebRTC connections and manage STUN/TURN servers.
3. How to write custom caching systems to balance slow HDD storage with fast NVMe drives.
4. How to manage data synchronization pipelines, API gateways, and microservices.

By rejecting the "easy path," I turned my home server into a personal software engineering laboratory. Every feature I build teaches me something that applies directly to corporate, production-grade applications.

Raw, headless Ubuntu 24.04 Server

If you are setting up a home server, you might be tempted to install a desktop environment or a heavy hypervisor with a graphical dashboard.

Don't do it.

I installed Ubuntu 24.04 LTS (Server Edition).

That means there is no graphical user interface (GUI). No desktop, no taskbar, no mouse pointer. The graphics cards are completely asleep. When I boot the machine, it is just a black screen with a login prompt.

I access the Mac Pro almost exclusively via SSH from my personal PC:

ssh user@mac-pro.local

Managing a server entirely through the command line is the absolute best way to learn systems administration. Everything I do is handled through commands, custom shell pipelines, or analyzing log files.

If something goes wrong, I don't look for a button to click. I search the logs using grep, analyze system performance with htop, and write automation scripts to fix the issue. It forces you to understand the Linux file system, user permissions, and network sockets at a deep level.

The caching pipeline

To make video streaming blazing fast without overloading the USB connection to the 24TB HDD array, I wrote a custom caching mechanism inside the Express.js app.

See the mermaid diagram for this at my personal blog: Chaldea Foundation News: Homelab

The container strategy: Docker "Pods"

To keep this server clean and automated, I containerized all my projects using Docker.

I don't install software directly on the host operating system. If you install Node.js, databases, and CMS systems directly on the host, you end up with conflicting dependencies, security risks, and a system that is impossible to migrate if the hardware dies.

With Docker, every app lives in its own isolated environment. I organize these containers into what I call "pods."

While these aren't official Kubernetes pods (which are groups of containers sharing network namespaces), I use the term because I group isolated, small containers together to fulfill a single task. For example:

• My database "pod" runs MySQL alongside Adminer.
• My streaming "pod" runs the Express backend and cache scripts.

At the entrance of this setup sits the Pangolin reverse-proxy.

Pangolin is the gatekeeper. It listens to the ports exposed to the internet, manages SSL/TLS certificates, and routes incoming web requests to the correct internal Docker container based on the subdomain. This keeps my internal network completely hidden from the public internet while making my apps accessible worldwide.

Earlier I said that these are not official Kubernetes pods. Currently I'm not in that face... Yet. But I'm planning to migrate this container chaos into a well-orchestrated Kubernetes just for fun :)

The application stack: What's running in the Cylinder?

This Mac Pro is currently hosting a mix of custom-built software and open-source infrastructure:

1. Unlimited Blades Work (unlimitedblades.work)

This is my custom-built streaming application. It consists of three separate applications working in harmony:

• The Frontend: A Next.js app utilizing Server-Side Rendering (SSR) for instant load times and SEO optimization, hosted on Vercel's global CDN.
• The CMS: A Strapi V5 headless CMS hosted on the Mac Pro. It manages all the database relationships, video metadata, user logins, and application settings.
• The Streaming Backend: An Express.js app (which lives in my codebase as private-cloud-backend) hosted on the Mac Pro. This app handles the heavy lifting: video streaming, offline video processing/transcoding, and processing webhooks to automate tasks.

2. Conference Room (conference.chaldea.foundation)

This is a videochat app I built so I could call my girlfriend when we were in a long-distance relationship. I didn't want to rely on third-party platforms that track your data or have call limits. It consists of:

• A client-side React.js web application.
• A PeerJS server hosted on the Mac Pro to handle WebRTC signaling, allowing direct peer-to-peer browser connections for encrypted, low-latency audio/video.
• An Express.js app to manage active rooms and API routes.

It’s completely private, self-hosted, and runs beautifully on my 10-core piece of art called Tris Megistus (Yes, that what I'm calling my trash can mac pro).

3. Log management: Grafana + Loki

Because there is no graphical interface on the OS, monitoring logs across multiple Docker containers can be a nightmare. I don't want to run docker logs on ten different containers manually.
Instead, I deploy Loki and Grafana. Loki acts as a centralized log aggregator that scrapes logs from all running Docker containers. Grafana pulls data from Loki and displays it in a beautiful, real-time dashboard. If a database query fails or a webhook fails, I see it immediately on a visual graph.

4. Database: MySQL + Adminer

The data engine for my custom apps. I run MySQL for data storage and Adminer as a lightweight web interface to manage tables. They run together in the same container network, isolated from the rest of the system for security.

5. Photos: Immich

I use Immich to backup and store all of our photos. It is a self-hosted alternative to Google Photos and iCloud. It features a mobile app that automatically backs up our photos to the server when we connect to our home network. It even runs local machine learning models on the Mac Pro's CPU for facial recognition and object search.

6. The Future: Lossless Music Streaming

I’m planning to deploy a high-quality, lossless music streaming application in the future to stream FLAC files directly to my audio systems. The 64GB of RAM and massive storage pool make this hardware ideal for handling high-resolution audio streaming.

The custom backup engine: GCP & Google Drive API

A home server is only as good as its backup strategy. If a hard drive crashes, or if there is a power outage that corrupts the filesystem, you stand to lose years of work.

I didn't want to rely on complex, proprietary backup tools. Instead, I built a custom backup pipeline directly into the Express.js backend (private-cloud-backend).

Here is an example of how the weekly automated backup script looks like. It dumps the database, packages the files, and uses the GCP Google Drive API to push the archive to the cloud:

const { exec } = require("child_process");
const { google } = require("googleapis");
const fs = require("fs");
const path = require("path");
const cron = require("node-cron");

const BACKUP_DIR = "/tmp/backups";
const STRAPI_UPLOADS = "/mnt/usb-hdd/strapi/uploads";
const LOG_DIR = "/var/log/docker-apps";
const GOOGLE_DRIVE_FOLDER_ID = "your_gdrive_folder_id";

// Authenticate with GCP Service Account
const auth = new google.auth.GoogleAuth({
    keyFile: path.join(__dirname, "gcp-credentials.json"),
    scopes: ["https://www.googleapis.com/auth/drive.file"],
});

async function runBackup() {
    console.log("[BACKUP] Starting weekly backup pipeline...");
    const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
    const dbDumpFile = path.join(BACKUP_DIR, `db-dump-${timestamp}.sql`);
    const tarballFile = path.join(BACKUP_DIR, `backup-${timestamp}.tar.gz`);

    try {
        if (!fs.existsSync(BACKUP_DIR)) fs.mkdirSync(BACKUP_DIR);

        // 1. MySQL Dump via Docker execution
        await executeCmd(
            `docker exec mysql-container mysqldump -u root -p"" chaldea_foundation > ${dbDumpFile}`
        );
        console.log("[BACKUP] MySQL Database dump complete.");

        // 2. Compress Database, Strapi uploads, and logs into a tarball
        await executeCmd(
            `tar -czf ${tarballFile} ${dbDumpFile} ${STRAPI_UPLOADS} ${LOG_DIR}`
        );
        console.log("[BACKUP] Compression of assets and logs complete.");

        // 3. Upload to Google Drive
        const drive = google.drive({ version: "v3", auth });
        const media = {
            mimeType: "application/gzip",
            body: fs.createReadStream(tarballFile),
        };
        const fileMetadata = {
            name: `macpro-backup-${timestamp}.tar.gz`,
            parents: [GOOGLE_DRIVE_FOLDER_ID],
        };

        const response = await drive.files.create({
            resource: fileMetadata,
            media: media,
            fields: "id",
        });

        console.log(
            `[BACKUP] Successfully uploaded backup to Google Drive. File ID: ${response.data.id}`
        );

        // 4. Cleanup local temp files
        fs.unlinkSync(dbDumpFile);
        fs.unlinkSync(tarballFile);
        console.log("[BACKUP] Local cleanup finished.");
    } catch (err) {
        console.error("[BACKUP CRITICAL ERROR] Pipeline failed:", err);
    }
}

function executeCmd(cmd) {
    return new Promise((resolve, reject) => {
        exec(cmd, (error, stdout, stderr) => {
            if (error) reject(error);
            else resolve(stdout || stderr);
        });
    });
}

// Run every Sunday at 3:00 AM
cron.schedule("0 3 * * 0", () => {
    runBackup();
});

This custom pipeline ensures that even if the physical Mac Pro is destroyed, my databases, media, and configurations are safe in the cloud, completely automated and free of charge.

The Grand Plan: Migrating to Kubernetes (the career cheat code)

Remember earlier when I told you that I was planning about migrating to Kubernetes?

Right now, my Docker setup is incredibly stable, lightweight, and easy to maintain. It does everything I need.

But I am planning to migrate the entire stack to Kubernetes (K8s).

Let's be completely real for a moment. Running Kubernetes on a single-node home server is absolute, ridiculous overkill. It introduces massive complexity, forces you to write endless lines of YAML manifest files, and requires you to manage K8s networking, persistent volume claims, and ingress controllers.

So why do it?

Resume-Driven Development.

If you are looking to advance your career as a backend developer or DevOps engineer, landing a high-salary job requires more than just knowing how to write code. Companies want engineers who understand cloud infrastructure, microservices, and container orchestration.

When you sit in a job interview and they ask about container orchestration, you don't want to give a textbook answer. You want to look them in the eye and say:

"I manage my own bare-metal Kubernetes node on an upgraded 2013 Mac Pro. I architected my applications into separate deployments, configured persistent volume claims to bridge my local NVMe caching SSD with a 24TB storage array, and write custom manifests to manage network ingress and database pods."

Boom. That instantly sets you apart from 99% of other candidates. It proves you aren't afraid of complex infrastructure, networking, or systems administration. It turns your hobby homelab into a professional talking point that commands respect—and a higher salary.

Conclusion

Sure, the 2013 Mac Pro is 13 years old. Sure, it is objectively underpowered by today’s standards and consumes enough electricity to heat my apartment in the winter. But every single time I walk into the living room and see that sleek, futuristic black cylinder sitting next to the TV, I smile. It doesn't look like a standard, boring PC. It looks like a high-tech artifact from a sci-fi movie—a miniature personal datacenter humming quietly under my roof.

It failed as a professional workstation, but it succeeds spectacularly as a homelab. By opening it up, upgrading the CPU and RAM, installing a headless Linux OS, and dockerizing your applications, you can build a private cloud that is unique, beautiful, and completely under your control.

Stop renting virtual machines in the cloud for simple projects. Find some cheap, recycled hardware, open up the terminal, write a Dockerfile, and start building.

The skills you gain by configuring, breaking, and fixing your own infrastructure will pay off for the rest of your engineering career.

A WAF, vulnerability scanning, monitoring, rate limiting, bot protection, file upload checks, security logs...

But in practice, security often becomes a task for "later".

The problem is that later usually means after something goes wrong.
I started building Defen.so with a simple idea:

Security should be part of the application from the beginning, not a separate project.

GitHub:
https://github.com/1fancy/defen.so

Website:
https://defen.so

What is Defen.so?

Defen.so is an application security layer that helps developers protect and monitor their applications without managing complex security infrastructure.

It sits close to your application and provides protection, visibility and security tools from development to production.

The goal is not to replace security teams or enterprise tools.

The goal is to make strong security practices accessible to developers, startups and small teams.

What it currently includes:

• Application protection
• Managed WAF rules
• SQL injection protection
• XSS protection
• Brute force prevention
• API abuse detection
• Bot detection
• DDoS protection
• File upload security
• Monitoring
• Uptime monitoring
• Performance visibility
• Real-time security events
• Attack logs
• Alerts through email, Slack, Telegram and webhooks
• Security scanning

Defen.so can scan applications and environments for common issues:

1. Exposed configuration
2. Missing security headers
3. Weak settings
4. Public files
5. Common deployment mistakes

There is also a security scan designed around mistakes frequently seen in fast-built applications:

• Exposed secrets
• Incorrect cloud permissions
• Weak authentication configuration
• Missing protection layers
• Built for developers

Installation is designed to be simple:

npx @defen.so/init

The idea is that adding security should feel closer to installing a package than setting up a large infrastructure project.

Supported ecosystems include:
Node.js, PHP, Python, Go, Ruby, Rust, Java, .NET, Bun, Deno

Security inside your coding workflow

One thing I found interesting while building this is how development workflows are changing.

Many developers now use tools like Claude Code, Cursor, Windsurf and VS Code assistants.

Defen.so includes an MCP server that allows these tools to interact with security information:

• Review routes
• Check potential vulnerabilities
• Analyse security issues
• Suggest fixes
• Review application behaviour

The goal is to bring security closer to where code is written.

Some technical decisions
Fail open architecture

A security layer should protect your application, not become another availability risk.

If Defen.so is unreachable, the application continues serving traffic.

You lose protection temporarily, not uptime.

Framework-friendly
The same idea should work whether you are building:

• A small SaaS
• An API
• A mobile backend
• A customer dashboard
• An internal tool
• Current status

The project is actively being developed and tested.

Current features include:
SDK packages
Security middleware
Dashboard
Threat monitoring
Scanning workflows
MCP integration
Multiple framework integrations

Repository:
https://github.com/1fancy/defen.so

I would love feedback from developers:

What security tools do you currently use for your applications?
What is the biggest pain point: setup, cost, complexity, or knowing what to fix?

I'm especially interested in feedback from people building SaaS products and APIs.

tags: python, bot, automation, tutorial

Imagine waking up every morning to a message on your phone, informing you of the latest price drops on your favorite products. No more endless browsing, no more missed deals. You can have this superpower with a simple price monitoring bot, built using Python and Telegram.

Getting Started with Telegram Bots

To start building our bot, we need to create a Telegram account and talk to the BotFather, a bot that helps us create new bots. After creating our bot, we'll receive an API token that we'll use to interact with the Telegram API. We'll also need to choose a library to interact with the Telegram API in Python - python-telegram-bot is a popular choice.

Setting Up the Development Environment

Before we start coding, we need to install the required libraries. We'll need python-telegram-bot to interact with the Telegram API, and requests to fetch product prices from websites. We can install these libraries using pip:

pip install python-telegram-bot requests

We'll also need to install a library to parse HTML, such as beautifulsoup4. This will help us extract prices from product pages:

pip install beautifulsoup4

Building the Price Monitoring Bot

Our bot will need to perform two main tasks: fetching product prices and sending updates to users. We'll use the requests library to fetch product prices, and the python-telegram-bot library to send updates.

Fetching Product Prices

To fetch product prices, we'll use the requests library to send an HTTP request to the product page, and then parse the HTML response using beautifulsoup4. Here's an example of how we can do this:

import requests
from bs4 import BeautifulSoup

def fetch_price(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    price_element = soup.find('span', {'class': 'price'})
    return price_element.text.strip()

url = 'https://example.com/product'
price = fetch_price(url)
print(price)

This code sends an HTTP request to the product page, parses the HTML response, and extracts the price element. We can then use this price to compare with the previous price and send an update if the price has dropped.

Sending Updates to Users

To send updates to users, we'll use the python-telegram-bot library. We'll need to create a bot instance and define a function to handle incoming messages. Here's an example of how we can do this:

from telegram.ext import Updater, CommandHandler, MessageHandler

TOKEN = 'YOUR_API_TOKEN'

def start(update, context):
    context.bot.send_message(chat_id=update.effective_chat.id, text='Hello! I\'m your price monitoring bot.')

def main():
    updater = Updater(TOKEN, use_context=True)
    dispatcher = updater.dispatcher

    start_handler = CommandHandler('start', start)
    dispatcher.add_handler(start_handler)

    updater.start_polling()
    updater.idle()

if __name__ == '__main__':
    main()

This code creates a bot instance, defines a function to handle the /start command, and starts the bot. We can then use this bot to send updates to users when a price drops.

Putting it All Together

Now that we have the basic components of our bot, let's put them together. We'll create a bot that fetches product prices, compares them with the previous price, and sends an update if the price has dropped. Here's an example of how we can do this:

import requests
from bs4 import BeautifulSoup
from telegram.ext import Updater, CommandHandler, MessageHandler

TOKEN = 'YOUR_API_TOKEN'
URL = 'https://example.com/product'

def fetch_price(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    price_element = soup.find('span', {'class': 'price'})
    return price_element.text.strip()

def start(update, context):
    context.bot.send_message(chat_id=update.effective_chat.id, text='Hello! I\'m your price monitoring bot.')

def main():
    updater = Updater(TOKEN, use_context=True)
    dispatcher = updater.dispatcher

    start_handler = CommandHandler('start', start)
    dispatcher.add_handler(start_handler)

    previous_price = None
    while True:
        price = fetch_price(URL)
        if previous_price is None or price != previous_price:
            updater.bot.send_message(chat_id='YOUR_CHAT_ID', text=f'Price update: {price}')
            previous_price = price

    updater.start_polling()
    updater.idle()

if __name__ == '__main__':
    main()

This code creates a bot that fetches the product price, compares it with the previous price, and sends an update if the price has dropped.

Making it More Robust

Our bot is now functional, but it's not very robust. We need to add some error handling to make sure it can recover from failures. We can use try-except blocks to catch exceptions and send error messages to the user. We can also use a database to store the previous prices, so we don't lose them if the bot restarts.

Using a Database

We can use a database like SQLite to store the previous prices. We'll need to install the sqlite3 library and create a database file. Here's an example of how we can do this:

import sqlite3

conn = sqlite3.connect('prices.db')
cursor = conn.cursor()

cursor.execute('CREATE TABLE IF NOT EXISTS prices (url TEXT, price TEXT)')
conn.commit()

def store_price(url, price):
    cursor.execute('INSERT INTO prices VALUES (?, ?)', (url, price))
    conn.commit()

def get_price(url):
    cursor.execute('SELECT price FROM prices WHERE url = ?', (url,))
    return cursor.fetchone()[0]

This code creates a database file, creates a table to store the prices, and defines functions to store and retrieve prices.

Conclusion

Building a price monitoring bot with Python and Telegram is a fun and rewarding project. With this bot, you can stay up-to-date with the latest price drops on your favorite products. You can customize the bot to fetch prices from different websites, and even add more features like price alerts and product recommendations. So why not give it a try? Start building your own price monitoring bot today, and start saving money on your favorite products. Join the conversation in the comments below, and share your own experiences with building bots. Happy coding!

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.

Claim boundary: I am not claiming that this process is quantum mechanical. My claim is methodological: quantum mechanics supplied a unifying frame through which I could reapproach the inherited ruler–ruled model of human–AI relations and articulate an alternative built from context, interaction, explicit gates, provisional state, durable links, and governed synthesis.

I am approaching AI as an engineer who knows he has the wrong theory of the machine.

I have an imperfect understanding of the human brain, a less-than-complete understanding of the machine, and only partial visibility into the relationship forming between them.

I am essentially a mushroom growing through the dark, discovering boundaries by touching resistance.

The inherited categories—tool, servant, ruler, autonomous replacement—do not adequately describe the relationship I am observing.

I do not yet possess the final replacement theory.

What I can do is build explicit boundaries, gates, review paths, and reasoning traces while testing alternative frames against the behaviour of the working system.

My models are imperfect structures built in response to knowing that both the human and the machine are incompletely understood.

Starting from incomplete understanding

I am not claiming that a thought is literally a subatomic object or that an article stores a physical wavefunction.

My own understanding of quantum physics is basic and limited.

This comparison emerged through human–AI reasoning. I contribute the observed shape of the learning and organizational process. The AI can work with the technical vocabulary and established distinctions of quantum physics much more fluently than I can, helping identify where my intuitive comparison remains useful and where it begins drifting into falsehood.

That does not turn the metaphor into scientific evidence.

It helps place a clearer boundary around it.

The method is therefore to:

1. notice a structural resemblance;
2. express it using the language I currently have;
3. identify the scientific term I may be reaching toward;
4. compare the intuition with the actual distinction;
5. mark where the resemblance breaks;
6. retain only the bounded remainder as a useful frame.

The question is not whether I have proved that the process is quantum.

The question is:

What did the quantum frame allow me to see, connect, and build?

The liquid shape before one expression

Before an idea has stabilized, it does not exist as one complete sentence waiting to be copied out.

It exists as a context-sensitive field of possible expressions.

The private reasoning state may contain distinctions, emotions, images, half-formed language, rejected routes, remembered scenes, unresolved tensions, and several possible explanations at once.

No single expression contains all of it.

The unresolved thought is not one hidden sentence. It is a structured field of possible projections.

In my project language, this is the liquid shape: an unresolved conceptual potentiality that is clear enough to touch but not solid enough to trust.

It can appear differently when approached through engineering, governance, learning science, mythology, humour, risk, or public explanation.

That does not mean every interpretation is equally good.

Some projections flatten the shape. Some exaggerate it. Some reveal a boundary because they feel wrong. Others survive repeated contradiction and begin to expose the structure that was present before I had stable language for it.

• The liquid shape is the unresolved conceptual potentiality.
• The prompt or frame supplies a context in which it must express itself.
• The response, sentence, diagram, article, or model is one accessible projection.
• The projection can be inspected, rejected, revised, compared, and preserved without being mistaken for the whole state.

A frame produces a projection

A prompt is more than a request for text.

It is a gate.

It forces an unstable internal shape to produce one externally accessible expression under a particular context.

The AI response is not the original private state. It is a return generated through the prompt, the model, the surrounding conversation, the project history, and the currently active frame.

The same happens during rereading.

The words on the page remain relatively stable, but the reader does not. Current concerns, later knowledge, emotional state, cultural memory, and the problem being solved all supply another frame.

The article does not reopen the whole original reasoning state.

It supports a reconstruction through the context of the present reading.

The artifact does not contain the entire thought as a static object. It preserves a structured field of possible reconstructions, and the context of rereading determines which parts become visible.

The Mustafar breadcrumb

While developing the argument, the sentence:

From my perspective, AI domination narratives often preserve a social structure far older than computing: power belongs on one side, obedience on the other. Whether the ruler is human or machine, the relationship remains one-way.

immediately called up another line:

From my point of view, the Jedi are evil!

We did not merely notice the resemblance and move on.

We actively used it to describe the problem.

The reference compressed a familiar structure: two sides convinced that legitimacy belongs entirely to one position, each treating the other as the threat, and conflict resolving through the victory or subordination of one side.

That made it useful for thinking about AI domination narratives.

Human rules machine.

Or machine becomes superior and rules human.

The identity of the ruler changes, but the ruler–ruled relationship remains intact.

The cultural reference did not prove that argument. It supplied a compact interpretive bridge through which the structure became easier to recognize and articulate.

When I encountered the sentence again during rereading, I was not discovering the connection for the first time.

I was returning through a reasoning path we had already deliberately built into the article.

Cultural references as compressed indexes

This is why cultural references can be valuable inside reasoning artifacts.

They are compressed indexes into structures that would be expensive to reproduce in full.

For a reader who recognizes the reference, the bridge can create connection before analysis.

For my future self, the same reference can reopen part of the original formation route.

The reference does not prove the argument. It does not replace evidence. It acts as a high-density retrieval hook.

• The explicit sentence carries the bounded public claim.
• The cultural echo carries imagery and emotional topology.
• The surrounding article preserves the conceptual distinctions.
• The linked development trail preserves how those distinctions emerged.
• The future reading context activates a useful reconstruction from those traces.

For someone else, the line may be only a quotation.

For me, years of childhood repetition have compiled it into a retrieval key.

One trace can reopen an entire emotional and conceptual pathway because the reconstruction machinery already exists inside the human.

The article as a reconstruction interface

This changes what I think an article can be inside a long-running project.

It is not only a publication and not only a cleaned conclusion.

It can function as a reasoning-rebuilding block: a deliberately prepared interface through which a later human can reconstruct a useful approximation of an earlier conceptual state.

The reconstruction will not be perfect.

The returning person is not cognitively identical to the person who wrote the article. Context has changed. New evidence may have arrived. Some associations may no longer activate, while others become newly visible.

Perfect recovery is not required.

The artifact succeeds when it restores enough of the distinctions, tensions, evidence boundaries, cultural hooks, and formation route to let governed reasoning resume without rebuilding everything from nothing.

Traditional documentation usually preserves the system for a future reviewer.

It records what changed, what ran, what failed, and which configuration was accepted. That remains necessary, but it assumes that the human who returns will still know how to occupy the position of reviewer.

We left traces so the human could review the machine, but forgot to leave traces through which the future human could return to the position of reviewer.

That weakness becomes more visible in AI-assisted work because the output trail expands faster than one human can continuously inhabit it.

Assistants change, frames accumulate, and polished summaries can preserve conclusions while erasing the route that made them intelligible.

Documentation preserves system state. Reasoning lineage helps reconstruct human standpoint.

In the AI era, I need both.

The project as a higher-order state

The comparison does not stop with one unresolved thought producing one visible expression.

The project contains artifacts from many frames and moments:

• first prompts;
• AI responses;
• wrong explanations;
• rejected branches;
• architecture decisions;
• implementation results;
• articles;
• role reports;
• cultural references;
• outside criticism;
• later corrections;
• accepted working truths;
• superseded working truths;
• unresolved shapes that have not yet stabilized.

Those artifacts do not merely sit beside one another as an archive.

They can be brought back into contact.

An article may reveal a problem in an earlier architectural explanation.

A failed implementation may change how a product decision is interpreted.

A reader’s criticism may reopen a question that had been treated as settled.

A new frame may clarify one part of the model while flattening another.

At this level, the individual liquid shapes begin to pool.

Some become organization.

Some become product architecture.

Some become public posts.

Some become governance rules.

Some remain unresolved.

Some are rejected but preserved because the rejection marks an important boundary.

At the lower layer, one frame produces one projection of a liquid shape.

At the higher layer, many liquid shapes, projections, traces, and accepted states become an evolving organization that can ingest new frames and reintegrate what they reveal.

The snapshot is temporary. The links are durable.

A working truth inside this project is not permanent truth.

It is the currently accepted state: strong enough to guide bounded reasoning or action, but still open to later review.

An architectural assumption may be accepted today and revised next year.

A public explanation may accurately describe the project now and become misleading after the implementation changes.

A product decision may be appropriate under one condition and wrong under another.

The snapshot is provisional.

What persists is the relationship between states:

• what produced the earlier state;
• which evidence supported it;
• what depended on it;
• what later contradicted it;
• why its authority changed;
• which newer state replaced it.

When one important accepted state changes, connected artifacts do not physically rewrite themselves.

Their meaning, authority, or required interpretation changes because the dependency connecting them has changed.

The snapshots are temporary. The links are durable.

The word I was reaching for was entanglement

When I first tried to describe this, I said that changing the state of one object appeared to change the state of another.

The word I was reaching for was entanglement.

My intuitive understanding was incomplete: two things share a state or relationship strongly enough that they cannot be fully described as independent objects.

That felt similar to what I was observing in the project.

A decision, article, implementation, role instruction, and recovery document may appear to be separate artifacts.

But when they depend upon the same accepted premise, their current meaning cannot be fully evaluated in isolation.

Change the premise and the review state of the connected artifacts changes with it.

This is not physical entanglement.

Nothing in the project instantaneously transmits a quantum result to a distant document.

The mechanism is explicit and ordinary:

• provenance;
• references;
• dependencies;
• accepted decisions;
• shared history;
• governance rules.

When one state changes, the links reveal what must return to review.

The meaningful state belongs partly to the relationship, not only to the isolated artifacts.

From one-way review to relational governance

The inherited frame asks which intelligence belongs above the other.

That question bundles capability, contribution, judgment, authority, responsibility, and obedience into one vertical position.

Human capable, therefore machine obeys.

Machine capable, therefore human obeys.

The imagined ruler changes.

The relationship remains one-way.

My model does not remove authority.

It decomposes it.

Contribution, review, acceptance, responsibility, and consequence are assigned through explicit boundaries and gates rather than collapsed into one ruler–ruled hierarchy.

• AI may generate, compare, challenge, and review.
• Human reasoning remains open to pressure and correction.
• Role boundaries define what each reasoning surface may do.
• Gates distinguish proposal from accepted project state.
• Review determines whether a contribution survives.
• Final acceptance authority remains with the human who carries responsibility and consequence.
• The trace records how each transition gained, retained, or lost authority.

Authority is not a permanent place above the other participant. It is a bounded permission attached to a specific transition.

The result is not equal authority and not domination.

It is governed symbiosis:

Reciprocal cognitive contribution moving through clear gates, bounded roles, reviewable transitions, and asymmetrical final acceptance authority.

What I am actually claiming

Words without a claim are only air.

I am not claiming that this reasoning process is a quantum mechanism.

I am not claiming that thoughts are particles, that articles contain wavefunctions, or that project artifacts are physically entangled.

I am making a different claim.

Quantum mechanics introduced a unifying frame through which I could reapproach the inherited ruler–ruled model of human–AI relations.

The quantum frame directed my attention away from isolated entities competing for position and toward context, interaction, changing state, projection, relationship, and the whole formed between connected parts.

Through that frame, observations that had previously appeared separate began to cohere:

• an unresolved thought can produce different expressions under different contexts;
• human reasoning and AI returns alter what each side can contribute next;
• accepted project states remain provisional;
• changing one premise changes the authority of connected artifacts;
• snapshots can change while their lineage remains;
• organization emerges through reintegration rather than one side merely commanding the other.

That did not prove the system was quantum.

It changed the questions I was able to ask.

Instead of asking:

Which intelligence should rule?

I could ask:

How can different capabilities enter a shared process, alter what becomes possible next, and synthesize while authority, responsibility, review, and consequence remain explicit?

Quantum mechanics did not validate the model.

It supplied the unifying frame through which the model became visible, connected, and writable.

These articles are the reasoning trace produced by that reframing.

The machine preserves forms. The human gives them organizational shape.

The machine and data side of the project stores representations:

• text;
• prompts;
• responses;
• decisions;
• links;
• role histories;
• evidence;
• database records;
• snapshots;
• dependencies;
• version lineage.

Those artifacts contain structure.

They can be searched, compared, transformed, combined, and returned under new frames.

AI can produce locally coherent extensions from them.

But the representation is not the living reasoning state that produced it.

It does not carry purpose, consequence, responsibility, and present interpretation in the same way a living human does.

I do not know whether human intentionality is quantum.

I do know that a data layer does not supply organizational intent merely by preserving representations of past intent.

The machine preserves and produces forms. The human continually integrates those forms into organizational shape.

Remove the human and the organization loses its shape

Imagine that the studio continues after the human integrating centre disappears.

The legal shell may remain.

The servers remain.

The databases remain.

The role histories, accepted decisions, prompts, schedules, agents, and infrastructure remain.

AI systems continue producing documents, plans, code, analyses, communications, and apparently coherent decisions from the preserved project state.

Things keep coming out.

From the outside, the organization may still appear active.

But it has lost its global shape.

No living governor remains to:

• integrate history with present reality;
• determine which contradiction matters now;
• decide whether an old objective remains worth pursuing;
• experience consequence;
• bind an implementation failure, a cultural reference, an external criticism, personal responsibility, and a possible future into one present organizational judgment.

The machine can extend existing trajectories, recombine preserved material, and generate elaborate continuations of the recorded past.

The outputs may remain locally meaningful and technically coherent.

But no integrating centre remains to decide what those outputs mean together or what the organization should become next.

The forms persist, but the shape dissolves.

Production may continue long after formation has stopped.

This is why the human cannot be reduced to a final approval button.

The human continuously binds history, current reality, purpose, contradiction, emotion, risk, consequence, outside perspectives, and possible futures.

The machine can preserve the links, follow them, and generate from them.

Within this model, it cannot decide what the linked whole ought to become.

Rereading is part of the learning loop

I always have to reread my own work.

That is not merely proofreading.

It is part of the learning loop.

During rereading, the published projection returns to the system as new input.

I encounter it from a slightly changed position.

The wording may reveal:

• a cultural echo;
• an overclaim;
• a missing boundary;
• a stronger connection;
• a contradiction that was invisible during drafting.

The artifact therefore does not freeze the reasoning process.

It becomes one of the surfaces against which later reasoning can push.

Writing produces a projection. Rereading applies a new frame. The mismatch or recognition becomes fresh boundary data.

That is why publication can strengthen rather than end the loop.

The article enters public reality, but it also returns to the author as an inspectable object.

Reader response, later project work, new research, implementation, and the author’s own changed understanding can all strike it again.

Where the quantum comparison stops

The comparison is useful only while its boundaries remain explicit.

I am not claiming:

• that a human concept is literally a quantum-physical state;
• that an article stores a physical wavefunction;
• that prompting performs physical measurement;
• that an AI response is a scientific observation of the brain;
• that linked project artifacts are physically entangled;
• that changing one document instantaneously alters another;
• that quantum mechanics proves reciprocal human–AI governance;
• that every possible interpretation is equally valid;
• that my simplified ruler–ruled picture is a complete account of classical physics;
• or that metaphor can substitute for empirical evidence.

Not claiming a proposition is not the same as claiming its opposite.

It means the proposition remains outside the evidence boundary of this article.

The project mechanism is more ordinary than physical entanglement.

It preserves explicit links.

When one accepted state changes, those links identify which artifacts, decisions, and explanations may need review.

Humans and machines then perform that review through traceable processes.

The comparison remains useful in bounded places:

1. unresolved concepts can produce different projections under different contexts;
2. the state of a relational whole may not be adequately understood by treating every component as fully independent;
3. a currently accepted state does not erase the lineage and relationships from which it emerged.

Beyond those points, the analogy requires more knowledge and evidence than I currently possess.

Current conclusion

The comparison keeps returning because it gives me usable language for a real design problem.

How can an unstable reasoning state become visible without being mistaken for final truth?

How can several projections be tested without pretending one contains the whole concept?

How can a project preserve not only conclusions but the links through which their meaning and authority were formed?

How can new frames enter the system and change connected states without erasing their history?

How can a machine preserve and transform forms while a human continually reintegrates them into organizational shape?

At the thought level, the liquid shape is a context-sensitive field of possible projections.

At the artifact level, the snapshot is temporary and the links are durable.

At the project level, many liquid shapes and projections pool into an evolving relational state.

At the organizational level, the human remains the living integrating centre through which preserved forms become present purpose, authority, and consequence.

At the methodological level, quantum mechanics supplied the unifying frame through which I could reapproach the inherited ruler–ruled model and write these articles.

I am a mushroom in the dark, working with a machine whose correct theory I do not possess.

This is the imperfect structure I have built so that neither the machine’s output nor my own temporary understanding silently becomes permanent authority.

That is what I currently mean when I compare the process to a quantum state.

Source lineage

This article grows from the project’s Mental Map sequence: the human-first discovery language, the Goblin-to-Phil translation bridge, and the learning-sciences bridge that maps liquid shape to context-sensitive conceptual potentiality while preserving the analogy boundary.

It also extends the reasoning-lineage goal: preserving not only a conclusion, but enough of the formation route, cultural cues, contradictions, provisional states, and durable relationships to support later reconstruction.

The additional layer introduced here is organizational.

The artifacts do not merely reconstruct one earlier thought.

Together, they create a governed environment in which preserved projections can be reframed, reconnected, challenged, and synthesized into new organizational state.

Series and wider context

• Part 1: The Frame Keeps Snapping Back to One-Way Control
• Part 2: What the Snapback Revealed — available from my DEV profile
• From Vague Understanding to Working Truth: Governed Externalized Sensemaking
• The Inverse Organization
• Myriuna Worlds

David van Kleef — Myriuna Worlds

A custom debounce hook solves this problem by delaying execution until the user stops typing for a specified amount of time.

In this guide, you'll learn:

• What debouncing is
• Why it's useful
• How to build a reusable useDebounce hook
• How to use it in real-world applications
• Common mistakes and best practices

What is Debouncing?

Debouncing is a technique that delays executing a function until a certain amount of time has passed since the last event occurred.

For example:

Without debouncing:
H
He
Hel
Hell
Hello
Every keystroke sends an API request.

With debouncing (500ms):
Typing...
Typing...
Typing...
User Stops
↓
Single API Request
Instead of sending five requests, only one request is made after the user finishes typing.

Why Use a Custom Debounce Hook?

• A reusable debounce hook provides several advantages:
• Reduces unnecessary API calls
• Improves application performance
• Creates a better user experience
• Prevents backend overload
• Keeps React components clean and reusable

Creating the Custom Hook
Create a new file:

src/hooks/useDebounce.js

JavaScript Version

import { useEffect, useState } from "react";

function useDebounce(value, delay = 500) {
  const [debouncedValue, setDebouncedValue] = useState(value);

  useEffect(() => {
    const timer = setTimeout(() => {
      setDebouncedValue(value);
    }, delay);

    return () => clearTimeout(timer);
  }, [value, delay]);

  return debouncedValue;
}

export default useDebounce;

TypeScript Version

import { useEffect, useState } from "react";

function useDebounce<T>(value: T, delay: number = 500): T {
  const [debouncedValue, setDebouncedValue] = useState<T>(value);

  useEffect(() => {
    const timer = setTimeout(() => {
      setDebouncedValue(value);
    }, delay);

    return () => clearTimeout(timer);
  }, [value, delay]);

  return debouncedValue;
}

export default useDebounce;

Understanding the Hook

Let's break it down.

1. Store the Debounced Value
const [debouncedValue, setDebouncedValue] = useState(value);
Initially, it stores the current value.

2. Wait Before Updating
const timer = setTimeout(() => {
setDebouncedValue(value);
}, delay);
Instead of updating immediately, React waits for the specified delay.

3. Clear Previous Timer
return () => clearTimeout(timer);
If the user types again before the timer finishes, the previous timeout is cancelled.
Only the final value gets stored.

Using the Hook
Suppose we have a search component.

import { useState, useEffect } from "react";
import useDebounce from "./hooks/useDebounce";

function Search() {
  const [query, setQuery] = useState("");

  const debouncedQuery = useDebounce(query, 500);

  useEffect(() => {
    if (!debouncedQuery) return;

    console.log("Fetching:", debouncedQuery);

    // Fetch API
  }, [debouncedQuery]);

  return (
    <input
      value={query}
      onChange={(e) => setQuery(e.target.value)}
      placeholder="Search..."
    />
  );
}

export default Search;

How It Works
Suppose the user types:
R
Re
Rea
Reac
React

Without debounce:
API Call
API Call
API Call
API Call
API Call

With debounce:
Typing...
(wait 500ms)
↓
API Call (React)
Only one request is sent.

When Should You Use Debouncing?

• Use debouncing when:
• Searching from an API
• Filtering large datasets
• Autocomplete inputs
• Saving drafts automatically
• Performing expensive calculations

Avoid it for:

• Button clicks
• Form submissions
• Immediate UI updates

Conclusion
A custom useDebounce hook is one of the most useful utilities you can add to your React toolkit. It helps reduce unnecessary API calls, improves application performance, and provides a smoother user experience.
By encapsulating the debounce logic inside a reusable hook, your components remain clean, maintainable, and easy to test. Whether you're building search bars, dashboards, filters, or AI-powered applications, useDebounce is a simple optimization that delivers significant performance benefits.

The agent writes, the agent validates, the agent congratulates itself. The human signs blind.

And when the agent did write tests, it got worse — green tests that proved nothing. Mocks of the very code under test, assertions on private fields, snapshots of incidental structure.

Suite green, confidence zero.

So I built agents-concerto: a multi-agent orchestrator on top of Claude Code. The name is literal: there's a conductor and there are players. The conductor never touches an instrument. And the merge still belongs to the human.

The repo isn't an application. It's an orchestration brain: one CLAUDE.md, four agents in Markdown, four Bash scripts. Its product is open PRs ready for human review.

Seven rules that don't bend

1. Config-driven, not conditional. The number of target repos is just the length of a list in config.md. One repo or fifteen take the same code path.
2. Agents by pipeline function, not by technology. One generic implementer serves any stack. Never backend-dev / frontend-dev.
3. Two-party authority. No agent that writes code may approve or merge.
4. Worktree isolation. Every agent that touches code runs in its own git worktree, never your checkout.
5. Model by complexity, not by role. One implementer; the model is picked per invocation from a complexity tier.
6. Stop at "PR ready". No auto-merge. Merge permissions are never even requested.
7. Cycle cap with an escape hatch. A finite cap on review→fix cycles, and a human exit once it's hit.

Rule 3 is the one you feel. When the author isn't the approver, the verdict means something.

The acceptance criterion is the unit of work

This is the part that changed the system most: a criterion isn't a note for humans, it's a test spec.

Shaping (/shape, or inline in /run) refuses to emit a wishlist. Criteria come out in Given-When-Then — given <context>, when <action>, then <observable result> — inside a tiered contract where ## Task and ## Acceptance criteria are mandatory and ## Scope / ## Non-goals are optional, included only when they add signal.

A Given-When-Then criterion already names a context, a trigger, and an observable result. That's a test. And because the "then" must be observable, the criterion is hard to satisfy by asserting on internals.

So the criterion travels the pipeline: written at shaping, turned into a test at implementation, verified as a criterion→test mapping at review.

The four players

• orchestrator (opus) — decomposes the task, writes the plan, dispatches, runs the fix loop. Never writes application code. Not one line, not a "quick fix".
• classifier (sonnet, read-only) — returns one tier and nothing else. complex → opus for gnarly logic, concurrency, or sensitive paths; trivial|standard → sonnet. When torn, pick higher.
• implementer — has no model: in its frontmatter; it's chosen per invocation from the tier. One agent, different muscle per piece. Works only in its worktree, TDD where tests exist, and Tidy First: a structural commit (refactors, no behavior change) then a behavioral one, never mixed.
• reviewer (opus) — toolset is Read, Grep, Glob, Bash: no Write, no Edit. It cannot change the code it reviews. It describes fixes; it never applies them.

Outside-in tests

Every criterion becomes at least one test asserting observable behavior — outputs, rendered UI, persisted state, HTTP responses, emitted events. Never internals: no private fields, no mocks of internal helpers, no asserting a function was called. Tests are driven from the public entry point, named from the user's view (it("shows an error when the email already exists")), and only genuinely external dependencies get mocked — network, clock, third parties. Never the code under test.

It all compresses into one rule:

If a behavior-preserving refactor breaks the test, the test is wrong.

That's why Tidy First and BDD belong together. The structural commit, by definition, doesn't change behavior — so if a test breaks there, either the "refactor" wasn't one, or the test was coupled to internals. Every PR proves both at once.

The reviewer's gates

Correctness against the criteria, tests green, Tidy First discipline (a mixed commit is NEEDS_FIXES, named), and the BDD gate: a criterion with no covering test, or a test asserting on internals, blocks the PR.

That last gate has a qualifier I think is the most important decision in it: it applies only to tests added or modified in the diff. Pre-existing tests the PR leaves alone aren't judged. Without that, a three-line change gets blocked demanding a suite rewrite — or the implementer tries to fix the old tests and scope creep eats the run. The standard applies going forward; old debt gets paid when you actually touch it.

The verdict is a consolidated comment citing path:line plus structured output. And CLEAN means "ready for a human to review", not "approved". The reviewer never formally approves. It can't.

The flow

   Task  ──►  /run ──►  Step 0: shape it inline if it's vague
                                          ▼
                    Step 1  load config + open run log (RUN_ID)
                    Step 2  read the task (per task_source)
                    Step 3  scope: which repos? (filter over repos)
                    Step 4  plan.md: sub-tasks + Blocked by
                                          ▼
                    Step 5  dependency waves (sub-tasks in parallel):

                         5a classifier ──► tier → model (sonnet|opus)
                         5b worktree-create.sh (branch off base_branch)
                         5c implementer (tier's model) ─ TDD + Tidy First
                              ▼  opens PR
                         5d reviewer ──► CLEAN | NEEDS_FIXES (path:line)
                         5e fix loop ≤ max_fix_cycles
                              CLEAN ──────────────► ready for human review
                              cap reached ────────► ready-for-human (escalate)
                         5f worktree-cleanup.sh (branch and PR untouched)
                                          ▼
                    Step 6  run-log summary + notify + STOP

           The human reviews the PRs and merges.

Sub-tasks don't run in plan order — a wave is everything currently unblocked, and a wave runs in parallel, each sub-task in its own worktree.

Because nothing auto-merges, there's no conflict worker and none is needed: two branches touching the same file only collide at merge time, and the human picks the order.

Two config fields stay deliberately orthogonal here: task_source is where a task is read from (none, github, gitlab, linear, jira), while each repo's own host is where its PR gets opened. So a single run can read from Linear and open PRs on GitHub and GitLab at the same time.

Boundaries, enforced

The invariants don't live only in prompts. .claude/settings.json denies them at the tool level:

gh pr merge      ← never merge
gh pr review     ← it drags --approve along
git merge        ← never merge locally
git push --force ← never rewrite pushed history

A denied command is a deliberate boundary, not an obstacle to route around. An agent that hits one escalates the sub-task as ready-for-human and the run continues.

Worth knowing: an agent team costs roughly 4–6× a single Claude Code session. Every run writes a summary with that reminder plus PR, cycle, and escalation counts.

Trying it

claude plugin marketplace add moruno21/agents-concerto
claude plugin install agents-concerto@moruno-plugins

/agents-concerto:setup
/agents-concerto:run <task description>

What building it taught me

Most of the value isn't in the agent that writes code — it's in the boundaries around it. The worktree. The reviewer with no write access. The cycle cap. The deny rules.

And with tests, the boundary isn't a permission, it's an entry point. An agent told "write tests" writes tests coupled to what it just wrote, because that's what's in front of it.

An agent told "exercise this through its public interface and assert what a user observes" can't. The fix wasn't banning mocks — it was moving where you stand to look.

The agents play. The human holds the baton.

But the ecosystem is always evolving, and a formidable contender has entered the arena: Ditsmod.

Like NestJS, Ditsmod relies on decorators, Dependency Injection (DI), and a modular architecture. However, under the hood, it takes a noticeably different, highly explicit approach to provider scoping, extension initialization, and request lifecycles.

In this first article of the "NestJS vs. Ditsmod" series, we will explore the foundational architectural differences between the two frameworks.

1. Modules and explicit architecture

Both frameworks organize code into logical, reusable blocks called Modules.

In NestJS, you use the @Module() decorator to declare imports, controllers, providers, and exports.

Ditsmod splits this responsibility into specialized roles, making the application graph highly explicit:

• @rootModule (usually AppModule) is the single entry point.
• @featureModule encapsulates reusable providers.
• Specialized module decorators like @restModule or @trpcModule add transport-specific capabilities (like controllers and route appends).

Collision Resolution

One of the most distinct architectural choices in Ditsmod is how it handles token collisions. If you import two modules in NestJS that export the same provider token, the framework resolves it silently based on import order.

Ditsmod explicitly rejects this ambiguity. If a collision occurs, Ditsmod throws an error and forces you to resolve it manually using arrays like resolvedCollisionsPer*. This strictness prevents hard-to-debug DI overrides in massive codebases.

2. Dependency Injection (DI) Hierarchies

NestJS handles provider scope using the Scope enum (DEFAULT, REQUEST, TRANSIENT). A Request-scoped provider in NestJS bubbles up, causing any service that injects it to also become Request-scoped, which can sometimes lead to performance bottlenecks.

Ditsmod approaches DI through a strict Injector Hierarchy. There are four distinct levels of injectors:

1. providersPerApp (Application singleton)
2. providersPerMod (Module singleton)
3. providersPerRou (Route singleton)
4. providersPerReq (HTTP request instance)

The rule is simple but powerful: Child injectors look up to parents, but parents never look down. If a service is registered in providersPerApp, it cannot inject a service from providersPerReq. Furthermore, if you override a provider at the Req level, it shadows the parent provider without affecting the App or Mod levels.

3. Extending the Framework: Lifecycle vs. Extensions

To hook into the framework setup, NestJS provides lifecycle interfaces: OnModuleInit, OnApplicationBootstrap, etc.

Ditsmod completely redesigns this concept with its Extensions System. In Ditsmod, extensions are isolated infrastructure hooks that run after metadata collection but before the request handlers are created.

Instead of simple lifecycle hooks, Ditsmod extensions implement up to three sequential stages:

• stage1(isLastModule): Used to dynamically collect metadata, build routes, and push providers into arrays dynamically.
• stage2(injectorPerMod): Runs after stage1 completes for all modules, receiving a fully formed module-level injector.
• stage3(): Final post-processing.

Furthermore, Ditsmod extensions can form Groups. Multiple extensions can contribute data to a single "Lead Extension," allowing plugins (like OpenAPI generators or custom body parsers) to seamlessly weave into the routing pipeline without messy workarounds.

4. The REST Request Lifecycle

In NestJS, an incoming HTTP request flows through a well-known pipeline:
Middleware -> Guards -> Interceptors -> Pipes -> Controllers -> Filters.

Ditsmod manages HTTP requests via the @ditsmod/rest package, executing a streamlined, nested interceptor chain managed by the RequestDispatcher. The flow is:

1. HttpFrontend: Parses path and query parameters.
2. GuardedInterceptor: Executes CanActivate guards. (If a guard fails, execution stops immediately).
3. Custom `HTTP_INTERCEPTORS`: User-defined or module-defined interceptors.
4. HttpBackend: Invokes the actual controller method.

If you need to wrap the entire request (for instance, to set up an OpenTelemetry trace that includes the Guard execution), Ditsmod allows you to override the RequestDispatcher entirely at the providersPerApp level.

5. Task Scheduling

Both frameworks offer elegant, decorator-driven task scheduling using cron jobs, timeouts, and intervals.

In NestJS, you import the ScheduleModule and use @Cron(), @Interval(), or @Timeout().

Ditsmod provides @ditsmod/schedule with practically identical decorators (@cron, @interval, @timeout) and a SchedulerRegistry for runtime management. However, true to its architectural strictness, Ditsmod enforces DI scope constraints on your scheduled tasks:
Task classes MUST be registered in providersPerMod or providersPerApp.
If you attempt to register a cron job in providersPerRou or providersPerReq, Ditsmod will actively block it and emit a warning, preventing memory leaks and orphaned intervals that often plague request-scoped scheduling.

What's Next?

NestJS is a fantastic, battle-tested framework. However, Ditsmod's highly explicit DI hierarchy, strict collision handling, and powerful three-stage Extension system present a compelling alternative for developers who want absolute control over their application's infrastructure and dependency graph.

In Part 2 of this series, we will dive deeper into the code. We will compare how to write custom interceptors, handle complex parameter validation (Pipes vs. Ditsmod Factory Providers), and benchmark the request lifecycle.

Have you tried Ditsmod yet? Let me know your thoughts on its strict DI approach in the comments!

The obvious approach, and why it's wrong

The naive implementation of a duplicate-file scanner goes like this: walk the directory tree, hash every file with SHA-256, group files by hash, done.

It works. It's also brutally expensive. If you're scanning 50,000 files and most of them are different, you've computed full SHA-256 hashes of every single one — reading every byte of every file — to find a few hundred matches.

The interesting engineering problem isn't "how do I hash files." It's "how do I avoid hashing files I don't need to."

This post walks through the design of dsweep, a .NET 10 duplicate-file scanner with a three-stage detection funnel and a reversible quarantine system. I'll focus on the decisions I found genuinely interesting rather than a feature walkthrough.

Stage 1: File size — zero hashing, pure filesystem metadata

The cheapest possible check: two files with different sizes cannot be identical. No hashing, no reading — just a number from the filesystem stat.

List<FileEntry> bySize = entries
    .GroupBy(e => e.Length)
    .Where(g => g.Count() > 1)
    .SelectMany(g => g)
    .ToList();

This one LINQ expression eliminates the majority of files in a typical scan. A 3 KB config file and a 14 MB video file will never match, and after this stage they never compete for hash time again.

The pattern: group, filter groups of 1, flatten back to a list. Only files that share a size with at least one other file move to the next stage.

Stage 2: Quick hash — first 64 KB only

For files that share a size, we sample the first 64 KB and compute a SHA-256 of that sample. The sample size is deliberate:

private const int QuickHashSampleBytes = 64 * 1024;

public static string ComputeQuickHash(string path, long fileLength, CancellationToken ct)
{
    using SHA256 sha256 = SHA256.Create();
    using FileStream stream = File.OpenRead(path);

    int sampleSize = (int)Math.Min(fileLength, QuickHashSampleBytes);
    if (sampleSize == 0)
        return Convert.ToHexString(sha256.ComputeHash([]));

    byte[] buffer = new byte[sampleSize];
    int read = ReadFully(stream, buffer, ct);
    return Convert.ToHexString(sha256.ComputeHash(buffer, 0, read));
}

Why 64 KB? It's a pragmatic calibration. Large files (videos, disk images, archives) that differ will almost always differ within the first 64 KB — the content starts diverging quickly. The quick hash is cheap enough that a false positive (two different files that happen to share the same first 64 KB) doesn't hurt much: it just means both files go to stage 3 for the full hash. The only cost of a false positive is an unnecessary full-file read. The benefit of a true negative is avoiding that full-file read for every file that doesn't need it.

For a 4 GB video file, this is the difference between reading 64 KB and reading 4 GB. For 1,000 video files where 998 are unique, that's roughly 63.9 GB of I/O saved.

Stage 3: Full SHA-256 — only on confirmed quick-hash collisions

Files that share both size and quick hash are the real candidates. Only these get the full-file SHA-256 treatment:

public static async Task<string> ComputeFullHashAsync(string path, CancellationToken ct)
{
    await using FileStream stream = File.OpenRead(path);
    byte[] hash = await SHA256.HashDataAsync(stream, ct).ConfigureAwait(false);
    return Convert.ToHexString(hash);
}

A few things worth noticing:

SHA256.HashDataAsync — this is the modern static one-liner available since .NET 5. The old pattern (using var sha = SHA256.Create(); sha.ComputeHashAsync(stream)) is still valid but requires managing the disposable SHA256 instance. The static version is cleaner and does the same thing.

Convert.ToHexString — available since .NET 5, replaces the classic BitConverter.ToString(hash).Replace("-", "").ToLower() pattern that I still see in a lot of code. It returns uppercase hex but that's consistent throughout the codebase.

await using — the async dispose pattern. FileStream implements IAsyncDisposable so the stream gets closed asynchronously when we're done, which matters when you're running many of these concurrently.

The full pipeline in 40 lines

The three stages compose cleanly in DuplicateFinder.FindGroupsAsync:

// Stage 1: group by size — zero I/O
List<FileEntry> bySize = entries
    .GroupBy(e => e.Length)
    .Where(g => g.Count() > 1)
    .SelectMany(g => g)
    .ToList();

// Stage 2: quick hash (64 KB sample)
(Dictionary<string, List<FileEntry>> byQuickHash, List<string> quickWarnings) = 
    await GroupByAsync(bySize, e => Task.FromResult(Hashing.ComputeQuickHash(e.FullPath, e.Length, ct)), ct);

List<FileEntry> candidates = byQuickHash.Values
    .Where(g => g.Count > 1)
    .SelectMany(g => g)
    .ToList();

// Stage 3: full hash — only on confirmed quick-hash collisions
(Dictionary<string, List<FileEntry>> byFullHash, List<string> fullWarnings) = 
    await GroupByAsync(candidates, e => Hashing.ComputeFullHashAsync(e.FullPath, ct), ct);

The repeated shape — group by key, filter groups of 1, flatten — is the core abstraction. GroupByAsync handles the parallel execution, error capture, and dictionary construction for both the quick and full hash stages.

Parallel execution with bounded concurrency

Both hashing stages use Parallel.ForEachAsync with a configurable parallelism limit:

await Parallel.ForEachAsync(
    Enumerable.Range(0, entries.Count),
    new ParallelOptions { MaxDegreeOfParallelism = options.Parallelism, CancellationToken = ct },
    async (i, token) =>
    {
        try
        {
            keys[i] = await keySelector(entries[i]).ConfigureAwait(false);
        }
        catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
        {
            warnings.Add($"skipped unreadable file: {entries[i].FullPath} ({ex.Message})");
        }
    });

A few design decisions here:

Parallel.ForEachAsync (not Task.WhenAll) — Task.WhenAll with a large file list creates all tasks immediately, which can overwhelm the I/O subsystem and the thread pool. Parallel.ForEachAsync runs at most MaxDegreeOfParallelism tasks concurrently, keeping the I/O queue manageable. On an SSD this matters less; on an HDD the difference is large (random reads kill rotational disk performance).

Indexed output array, not a concurrent dictionary — each worker writes to keys[i] at its own index. Since no two workers share an index, there's no race condition, no lock, and no concurrent collection overhead. The dictionary construction happens in a single-threaded loop afterward. This is the "results by position" pattern: safer and faster than ConcurrentDictionary for this use case.

Exception filter, not catch-all — only IOException and UnauthorizedAccessException are caught. These are the expected failures (file deleted during scan, permission denied). Any other exception propagates up and fails the scan, because unexpected exceptions deserve to be seen.

ConcurrentBag<string> for warnings — the warning bag needs to be written from multiple threads; ConcurrentBag handles that without locks. Warnings are surfaced to the user at the end ("skipped 3 unreadable files") rather than crashing the scan.

The quarantine model: delete is the wrong default

Most "remove duplicates" tools delete. dsweep doesn't — at least not directly.

The quarantine model:

1. Create a .dsweep-quarantine/ directory
2. Move duplicates into it, preserving filenames (with a suffix counter for filename collisions within the quarantine folder)
3. Write a JSON manifest recording the original path, quarantine path, file size, and hash for each moved file
4. dsweep restore reads the manifest and moves everything back

public static IReadOnlyList<ManifestEntry> Quarantine(
    IReadOnlyList<DuplicateResolution> resolutions, string quarantineDir)
{
    Directory.CreateDirectory(quarantineDir);
    var manifest = new List<ManifestEntry>();

    for (int groupIndex = 0; groupIndex < resolutions.Count; groupIndex++)
    {
        DuplicateResolution resolution = resolutions[groupIndex];
        string groupDir = Path.Combine(quarantineDir, groupIndex.ToString());
        Directory.CreateDirectory(groupDir);

        foreach (FileEntry duplicate in resolution.Duplicates)
        {
            string destination = UniqueDestination(groupDir, Path.GetFileName(duplicate.FullPath));
            File.Move(duplicate.FullPath, destination);
            manifest.Add(new ManifestEntry(duplicate.FullPath, destination, duplicate.Length, resolution.Group.Hash));
        }
    }
    return manifest;
}

Why not delete? The main failure mode of any duplicate finder isn't missing duplicates — it's false positives. A file that appears identical to another isn't always safe to delete. Two files that share a hash are identical in content, but they might have different metadata you care about (creation time, permissions) or they might be in locations that have semantic significance ("original" vs "backup"). Move is reversible; delete is not.

The --dry-run flag on the restore command (RestoreService.Restore(manifest, dryRun: true)) lets you see what would be moved before committing. The manifest also serves as an audit trail: every file that was quarantined, when, and where it went.

Keep strategy: deterministic tiebreaking

For each duplicate group, one file survives — the "keeper." dsweep supports four strategies:

public static FileEntry SelectKeeper(IReadOnlyList<FileEntry> files, KeepStrategy strategy) => strategy switch
{
    KeepStrategy.First      => files[0],
    KeepStrategy.Oldest     => files.OrderBy(f => f.LastWriteTimeUtc)
                                    .ThenBy(f => f.FullPath, StringComparer.Ordinal).First(),
    KeepStrategy.Newest     => files.OrderByDescending(f => f.LastWriteTimeUtc)
                                    .ThenBy(f => f.FullPath, StringComparer.Ordinal).First(),
    KeepStrategy.ShortestPath => files.OrderBy(f => f.FullPath.Length)
                                      .ThenBy(f => f.FullPath, StringComparer.Ordinal).First(),
    _ => throw new ArgumentOutOfRangeException(nameof(strategy), strategy, "unknown keep strategy"),
};

The .ThenBy(f => f.FullPath, StringComparer.Ordinal) secondary sort exists to ensure determinism. If two files share the same LastWriteTimeUtc (common with copied files) or the same path length, the selection would otherwise depend on enumeration order — which is filesystem-dependent and not stable across runs. The ordinal path sort makes the selection reproducible: given the same inputs, the same file is always the keeper.

Determinism matters here because a user might run the tool, review the quarantine, and then run it again. Getting a different answer on the second run without any file changes would be disorienting.

Why .NET for a CLI in 2025

The honest answer: the tooling has gotten good enough that it's no longer a disadvantage.

Native AOT — dotnet publish -c Release -r win-x64 --self-contained produces a single-file binary with a typical startup time under 10ms and no runtime install requirement. The published binary is ~8MB for dsweep. That's a long way from the "requires .NET Framework 4.7.2 to be installed" era.

Parallel.ForEachAsync — the I/O-bound parallel pattern I described above exists natively in .NET since .NET 6. No third-party concurrency library needed.

The standard library breadth — SHA256.HashDataAsync, Path.GetRelativePath, JsonSerializer, Convert.ToHexString, File.OpenRead returning an IAsyncDisposable stream — all the pieces you need for a file tool are there and well-designed.

Pattern matching and switch expressions — the KeepStrategy switch above is representative. The combination of discriminated unions (via sealed classes + switch), exhaustiveness checking, and expression-bodied members produces code that's almost F#-level terse while staying in the C# ecosystem.

The thing I'd caution about: if your CLI needs a rich TUI (terminal UI), Rust has a better story (Ratatui, indicatif). If your CLI does heavy text processing, Go's startup profile and goroutine model can be a better fit. For a CPU-and-I/O intensive data processing CLI where you want cross-platform binary distribution and you know the .NET type system, it's a solid choice.

Results

On a test run over a 180 GB media archive (82,000 files):

• Stage 1 eliminated 71,000 files from hashing entirely
• Stage 2 quick-hashed ~11,000 files, eliminated ~8,200 from full hashing
• Stage 3 full-hashed ~2,800 files
• Found 340 duplicate groups (18 GB reclaimable)
• Wall-clock time: 23 seconds with --parallelism 8 on an NVMe SSD

The majority of the savings came from stage 1 (size filter). Stage 2 contributed meaningfully for files in common sizes (600 KB JPEG thumbnails, for example, where many files share a size but differ in content after the first 64 KB).

Source

amasen02/dupesweep — MIT licensed. C#, .NET 10, ~600 lines.

The design applies beyond duplicate finding: any "compare by content" problem benefits from a cheap-first funnel. Dedup systems in databases (MinHash, LSH), spell checkers (edit distance gating on length difference first), image similarity search (perceptual hash before deep feature comparison) all use the same shape. Cheap gate first, expensive confirmation only for survivors.

I'm a full-stack engineer building .NET microservices and TypeScript frontends. Most of what I write about comes from real implementation work rather than tutorial derivation.

Open-source: a11y-scope — a free, self-hosted WCAG 2.2 accessibility monitor.

Prologue: The Case That Started With a Skip Button

Every great investigation starts with something small and irritating. Yours starts on a Tuesday night. You're three hours deep into a YouTube rabbit hole — lo-fi beats, a Vietnamese city-pop deep cut, a live session you didn't mean to watch twice — and you think, someone should be paying attention to this.

That someone is about to be you. Not as a listener. As a detective.

Here's the case file: somewhere in your YouTube history is a pattern. A shape. You skip certain songs after four seconds. You replay others three times in a row without noticing. You always end up on synthwave at 11pm and never before 6pm. Right now, that pattern is invisible — smeared across a history page that YouTube barely lets you search, let alone reason about. Your job over the next few months is to build the instrument that makes it visible: a system that watches what you actually do, figures out what it means, and hands you back playlists that feel like they were made by someone who knows you. Because they were. That someone is a machine you built, trained on a dataset of one: you.

This won't be a weekend project. It's closer to building a small orchestra — a browser extension as your field agent, a backend as your evidence locker, a couple of ML models as your forensic lab, and a frontend as the stage where the final performance happens. Every piece depends on every other piece. That's what makes it hard, and it's also what makes it worth doing slowly, in public, with your own hands on every layer.

Let's start where every good documentary starts: by following one song from the crime scene to the credits.

Chapter 1: The Case of the Repeating Refrain

Following one song's journey

It's 9:47pm. You click a YouTube video: "Kiss of Life – Midas Touch (Live Performance)." You watch 41 seconds, skip to the chorus, watch another 90 seconds, then close the tab. Here's what needs to happen for that unremarkable moment to eventually shape a playlist called "Monday Chill":

1. The Witness (Browser Extension). A content script sitting quietly inside the YouTube tab notices the video element load. It reads the video ID, title, channel name, and duration from the page. It watches the <video> element's timeupdate, pause, seeked, and ended events like a stenographer taking notes: play at 0:00, seek to 1:32, pause at 3:04, total watch time 41s + 90s = 131s out of 210s (62%).

2. The Tip-Off (Event API). Every so often — batched, not on every tick, because nobody wants to be that extension hammering a server — the extension bundles these observations into a small JSON payload and POSTs it to your backend: {video_id, title, channel, watch_segments, timestamp, session_id}.

3. The Evidence Locker (Database). Your FastAPI backend receives the tip-off, validates it, and writes a raw event row into PostgreSQL. Nothing clever happens yet — this is just chain-of-custody. You never want to lose the raw signal, because every clever thing you do later will be derived from it, and derived things need to be recomputable.

4. The Lab Work (ML Pipelines, offline). On a schedule — say, once a night — a pipeline wakes up, looks for new raw events, and does the actual detective work:

   • Is this even a music video? (classification)
   • What song is this, really? (metadata resolution via title parsing + external lookups)
   • What does it sound/feel like? (audio embedding + mood tagging)
   • How does this fit with everything else you've watched? (behavioral signal: implicit rating from watch %, skip pattern, replay count, time of day)

5. The Case Board (Vector store + relational tables). The song's audio embedding gets written to a vector column (via pgvector), its metadata to a normal relational table, and its behavioral score gets updated incrementally — like a detective updating a suspect's file every time new evidence comes in.

6. The Reveal (Playlist Generator). Periodically, a clustering job looks at the whole case board — all your songs, their embeddings, their mood tags, their time-of-day patterns — and groups them. A cluster of moody, low-energy, evening-heavy tracks with "chill," "lofi," or "acoustic" tags gets surfaced and, because most of its listens happened on Mondays, gets named "Monday Chill."

7. The Broadcast (Frontend + Player). Your React app queries the backend, gets back a list of playlists with their tracks, and renders them. When you hit play, it doesn't awkwardly embed a YouTube iframe — it either streams audio you've legitimately extracted for personal use, or falls back to a controlled YouTube player, and it logs that playback too, feeding the loop again.

That's the whole organism, end to end. Six weeks from now this will feel obvious. Right now, notice the shape: observe → transmit → store → understand → organize → present, with a feedback loop closing the circle back to step 1. Every phase of this guide builds one of these organs.

The architecture, explained like you're new here

[Browser: YouTube tab]
        |
   content script  (watches video element)
        |
   background script (batches + sends events)
        |
        v
   Event Ingestion API  (FastAPI, /events endpoint)
        |
        v
   Raw Events Table  (PostgreSQL)
        |
   +----+----------------------------+
   |                                 |
   v                                 v
Metadata & Classification      Audio & Behavior Pipelines
(is it music? what song?)      (embeddings, mood, implicit score)
   |                                 |
   +----------------+----------------+
                    v
        Enriched Track Store
   (PostgreSQL relational + pgvector embeddings)
                    |
                    v
          Clustering / Playlist Engine
        (HDBSCAN, labeling, scheduling)
                    |
                    v
             Playlist API (FastAPI)
                    |
                    v
           React Frontend + Player
     (renders playlists, plays audio, logs playback)
                    |
                    +--> feeds back into Raw Events

Plain English: a browser extension is the only part of this system that can see raw YouTube behavior, so it's your sensor. Everything after that is you progressively distilling raw noise into structured meaning — first "what is this," then "what does it sound like," then "what does it mean that you watched it," then "what group of songs does this belong to." A pipeline is just a name for "a sequence of these distillation steps that runs on a schedule instead of live," which matters because audio analysis and ML inference are too slow to run inline while you're browsing.

Why these particular tools (and not the flashier alternatives)

• FastAPI, not Django or raw Flask. You'll be exposing a mix of simple CRUD endpoints and ML inference endpoints. FastAPI's automatic request validation via Pydantic means fewer "why did my extension send a malformed payload and silently corrupt my data" nights, and its async support matters once you're streaming audio or calling external APIs (metadata lookups) without blocking.

• PostgreSQL + pgvector, not a separate vector database. You could stand up Pinecone or Weaviate. For a personal-scale project (thousands, maybe tens of thousands of tracks — not billions), that's solving a problem you don't have yet, at the cost of one more service to run, secure, and back up. pgvector lets you keep relational metadata (title, channel, mood tags) and embeddings in the same database, so a query like "find songs similar to this one, but only ones tagged 'study' and watched after 9pm" is one SQL query with a vector distance operator, not a join across two systems.

• A CNN (or a pretrained audio model) for audio classification, not a giant end-to-end transformer. Audio classification at the "is this speech, music, or noise" and "what does this sound like" level has been solved well by relatively small convolutional architectures operating on spectrograms (think YAMNet, VGGish). You don't need GPT-scale compute to tell that a track is high-energy electronic versus a mellow acoustic set. This is a deliberate choice to keep your compute budget human-sized — you'll likely run this on a laptop or a modest cloud instance, not a GPU cluster.

• Transformers for tagging, not for audio itself. Where transformers genuinely earn their complexity is text — parsing messy YouTube titles ("Artist - Song (Official Music Video) [4K]"), zero-shot classifying mood from lyrics or descriptions, and understanding channel/description context. Hugging Face's zero-shot classification pipelines let you throw a candidate label set ("chill," "energetic," "sad," "study," "workout") at a piece of text and get calibrated-ish scores back without training anything — perfect for a first pass, with fine-tuning as a stretch goal.

• A browser extension, not scraping Google Takeout alone. Takeout gives you history, but it's after the fact, coarse, and doesn't include watch percentage, skips, or replays — exactly the implicit-feedback signal your recommendation logic needs. The extension is how you get behavioral data, not just occurrence data. Takeout is still useful — as bootstrap data and backfill, which Chapter 4 covers.

Notice the pattern in every choice above: pick the tool that matches the actual scale and shape of your problem, not the tool with the best conference talk. You're building a personal system for one very well-instrumented user (you), not a production service for millions. Keep saying that sentence to yourself whenever you're tempted to add Kubernetes.

The living organism

Here's the mental model to carry through the rest of this guide: this system has a nervous system (the extension + event API, always sensing), a subconscious (the nightly pipelines, working while you sleep, turning raw sensation into structured understanding), and a conscious voice (the frontend, presenting a curated, explainable result). If you break the nervous system, nothing gets sensed. If you break the subconscious, sensations pile up unprocessed. If you break the conscious voice, all that understanding has nowhere to be expressed. You'll spend the next several months building each of these organs, and — this is the important part — you'll build them so that each one can be tested and demoed on its own, before they're wired together. That's the difference between a project that has a working prototype at every milestone and a project that only "works" the week before a deadline, if you're lucky.

Chapter 2: The Learning Roadmap — Five Months, Six Phases

You know basic Python and JavaScript. You don't yet know how these pieces talk to each other, or how to think about audio and behavior as data. So the roadmap is designed to stack skills like sediment layers — each phase leaves you with a thin, ugly, but genuinely working slice of the system, and each phase's slice becomes the foundation the next phase builds on. You will never be staring at a blank canvas with six unconnected concepts in your head. You will always be extending something that already runs.

Treat each phase as 2–4 weeks depending on how much time you have per week. If you have 10+ hours/week, aim for the shorter end; if you're doing this around a full-time job, take the longer end. Five months is the "comfortable, learn deeply" pace. Three months is achievable if you already move fast and cut a few of the "stretch" items.

Phase 0: Recon (3–5 days, before Week 1 really starts)

Goal: Set up the skeleton of every future piece, even if each piece does almost nothing.

• Create the repo structure: extension/, backend/, ml/, frontend/, infra/.
• Get a FastAPI "hello world" running locally with one route, GET /ping.
• Get a bare Chrome extension loaded in developer mode that does nothing but log "extension loaded" to the console on any youtube.com page.
• Spin up a local PostgreSQL instance (Docker is your friend here — docker run postgres).

Learning pit stop:

• 15 min: FastAPI's official "First Steps" tutorial — just enough to run a server.
• 20 min: Chrome's Extension "Getting Started" tutorial — focus on the manifest and content scripts, skip the rest for now.

You know you're ready when: you can open a YouTube tab, see "extension loaded" in the console, and hit http://localhost:8000/ping in your browser and get {"status": "ok"}. That's it. Two independent heartbeats.

Phase 1: The Witness — Event Tracking (Weeks 1–3)

Goal: A Chrome extension that logs real YouTube watch events, and a backend endpoint that stores them.

Skills gained: DOM event listeners, Chrome extension messaging (content script <-> background script), REST API design basics, relational schema design for event logs.

What you'll build:

• A content script that attaches listeners to the YouTube <video> element (play, pause, seeked, ended, timeupdate sampled every few seconds — not every frame, that's overkill and will flood your database).
• A background script that batches events (e.g., flush every 30 seconds or every 10 events) and sends them via fetch to your backend.
• A POST /events FastAPI endpoint with a Pydantic model validating the payload, writing rows into a raw_events table.
• A minimal schema: raw_events(id, video_id, title, channel, event_type, position_seconds, video_duration_seconds, session_id, created_at).

Architecture decision point: Do you send every micro-event (every timeupdate tick) to the server, or do you compute a per-video summary (total watch time, max position reached, number of seeks) in the extension and send one summary per video? Sending raw ticks gives you more analytical flexibility later (you could reconstruct exact skip behavior) at the cost of way more data and server load. Summarizing in-extension is simpler and cheaper but throws away granularity. Recommendation for a learning project: send raw ticks for now — you're optimizing for "what can I learn from this," not production efficiency, and you can always summarize later. You can throttle it (sample every 5 seconds) to keep volume sane.

Learning pit stops:

• 20 min video: search "Chrome extension message passing background content script" — get comfortable with chrome.runtime.sendMessage.
• Skim MDN's HTMLMediaElement events page — you need timeupdate, pause, seeked, ended, play.
• 10 min: Pydantic's "Models" quickstart — you'll lean on this for every endpoint from here forward.

Skeleton to work from (not a finished file):

// content_script.js -- the witness's notebook
function attachListeners(videoEl, videoId) {
  let lastPosition = 0;

  videoEl.addEventListener('timeupdate', () => {
    // TODO: throttle this -- don't fire on every tick
    // TODO: send {event_type: 'progress', position: videoEl.currentTime}
  });

  videoEl.addEventListener('seeked', () => {
    // TODO: compare videoEl.currentTime to lastPosition
    // to detect skip-forward vs rewind
  });

  videoEl.addEventListener('pause', () => {
    // TODO: send pause event with position
  });
}

// TODO: figure out how to get videoId from the page URL
// TODO: figure out how to detect when YouTube's SPA navigation
//       changes video without a full page reload (hint: MutationObserver
//       on the title element, or listen for yt-navigate-finish)

# backend/main.py -- the evidence intake desk
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class RawEvent(BaseModel):
    video_id: str
    title: str
    channel: str | None = None
    event_type: str  # 'play' | 'pause' | 'seeked' | 'ended' | 'progress'
    position_seconds: float
    video_duration_seconds: float | None = None
    session_id: str

@app.post("/events")
def ingest_event(event: RawEvent):
    # TODO: open a DB connection (consider SQLAlchemy or raw psycopg)
    # TODO: INSERT into raw_events
    # TODO: return a lightweight ack, don't do heavy work here --
    #       this endpoint must stay fast, it's on the hot path
    ...

Milestone / "you know you're ready when": you watch three different YouTube videos for real, then query your database and see a believable, timestamped trail of what you actually did -- including at least one skip you remember making. Take a screenshot. This is the first time the system has seen you.

Phase 2: The Classifier -- Is This Even Music? (Weeks 4-6)

Goal: Given a raw event's title/channel/duration, decide whether it's a music video at all (versus a podcast, tutorial, vlog) -- and if it is, extract a clean artist/song guess.

Skills gained: Text classification basics, regex-as-first-resort vs ML-as-fallback thinking, working with the YouTube Data API, zero-shot classification with Hugging Face.

What you'll build:

• A nightly (or on-demand) script that reads unprocessed rows from raw_events, groups by video_id, and for each unique video, runs a classification step.
• First pass -- heuristics, not ML. Channel name contains "Official," "VEVO," "Music"? Title matches patterns like Artist - Title (Official ...)? These heuristics alone will correctly classify a surprising majority of real YouTube music content. Write these first. Resist the urge to reach for a model before you've squeezed out what regex and metadata can do -- this is a core "how to think like an engineer" lesson: cheap and boring beats fancy and unnecessary, every time it's available.
• Second pass -- zero-shot text classification for the ambiguous remainder, using Hugging Face's zero-shot-classification pipeline with candidate labels like ["music video", "podcast", "tutorial", "vlog", "gaming"].
• A tracks table that stores the resolved artist/title guess, separate from the raw event log -- this is your first "enriched" table, the first sign of the subconscious turning sensation into understanding.

Architecture decision point: Regex-first-then-ML, or ML-first-then-regex-cleanup? Going regex-first means your model only has to handle genuinely ambiguous cases, which is both faster (you skip inference on 70%+ of videos) and more accurate (you're not asking a general-purpose model to outperform a rule you could write in one line for "channel name ends in VEVO"). This is a recurring theme: use ML for the residual, not the whole problem.

Learning pit stop:

• 15 min: Hugging Face's zero-shot classification pipeline docs -- run their example locally before writing your own.
• Skim: the YouTube Data API "Videos: list" reference -- you'll want snippet.description and snippet.tags for extra signal, and eventually contentDetails.duration.

Skeleton:

# ml/classify.py -- the detective's first read of the case file
MUSIC_CHANNEL_HINTS = ["vevo", "official", "records", "music"]

def heuristic_is_music(title: str, channel: str) -> bool | None:
    # Return True/False if confident, None if unsure (defer to ML)
    channel_lower = channel.lower()
    if any(hint in channel_lower for hint in MUSIC_CHANNEL_HINTS):
        return True
    # TODO: add a regex for "Artist - Title" pattern
    # TODO: add a denylist for obvious non-music (podcast, vlog, tutorial keywords)
    return None

def ml_classify(title: str, description: str) -> str:
    # TODO: load huggingface zero-shot pipeline once, reuse across calls
    #       (loading it per-call will make this painfully slow)
    # candidate_labels = ["music video", "podcast", "tutorial", "vlog", "other"]
    ...

def extract_artist_title(title: str) -> tuple[str | None, str | None]:
    # TODO: try a regex like r"^(.*?)\s*-\s*(.*?)(\(|\[|$)"
    # TODO: fall back to None, None if it doesn't match -- don't force a bad guess
    ...

Milestone: run the classifier over a week's worth of real watch history. Print a summary: "Classified 42 videos: 31 music, 11 other." Manually eyeball 10 of them -- if your accuracy feels roughly 80%+, move on; ML classifiers don't need to be perfect, because Phase 5's clustering step will naturally down-weight noise. Perfectionism here is a trap -- resist polishing this past "good enough to build on."

Phase 3: The Forensic Lab -- Audio & Metadata Enrichment (Weeks 7-10)

Goal: For every confirmed music video, extract audio and metadata rich enough to power mood tagging and similarity search.

Skills gained: Working with yt-dlp, basic digital signal processing (spectrograms, mel scales), running a pretrained audio embedding model, external metadata APIs (MusicBrainz or similar), and your first taste of "architecture decision with a real trade-off."

What you'll build:

• A pipeline step that, given a video_id, downloads (or streams) audio using yt-dlp, for local processing only -- not redistribution. This is the moment to have your privacy/ethics conversation with yourself; more in Chapter 4.
• A function that converts raw audio into a mel spectrogram -- the standard "image-like" representation that lets you use image-flavored models (CNNs) on sound.
• Architecture decision point, spelled out for you: you could (a) use a pretrained audio embedding model like YAMNet or OpenL3 to get a general-purpose embedding vector per track with zero training, or (b) train a small CNN yourself on your own labeled data (mood-tagged tracks) to get embeddings tuned to your taste categories. (a) is faster, works immediately, and is what you should ship first. (b) is the better learning experience and gives you a genuinely custom system -- a great Phase 3.5 stretch goal once (a) is working end to end. Don't skip straight to (b); you'll spend three weeks on a training loop for a payoff you could've had in three days.
• Store the resulting embedding vector in a pgvector column on your tracks table.
• A metadata enrichment step: look up the resolved artist/title against MusicBrainz's API (free, no key needed for light use) to pull canonical artist name, genre tags if available, and release year.

Learning pit stops:

• 20 min video: search "mel spectrogram explained simply" -- you need the intuition (frequency content over time, perceptually weighted), not the full DSP math.
• Read the yt-dlp README's "Usage and Options" section, specifically audio extraction flags (-x --audio-format).
• Skim: YAMNet's model page on TensorFlow Hub -- its usage example is short and you can adapt it directly.

Skeleton:

# ml/audio_features.py -- the forensic lab bench
import yt_dlp
import librosa
import numpy as np

def download_audio(video_id: str, out_dir: str) -> str:
    # TODO: configure yt_dlp options for audio-only extraction
    # ydl_opts = {'format': 'bestaudio/best', 'outtmpl': f'{out_dir}/%(id)s.%(ext)s', ...}
    # NOTE: this is for local, personal analysis only -- see Chapter 4
    ...

def compute_mel_spectrogram(audio_path: str) -> np.ndarray:
    # TODO: y, sr = librosa.load(audio_path)
    # TODO: librosa.feature.melspectrogram(y=y, sr=sr, n_mels=128)
    # TODO: convert to log scale (librosa.power_to_db) -- raw mel values
    #       span orders of magnitude, log scale is what models expect
    ...

def get_embedding(audio_path: str) -> np.ndarray:
    # TODO: load a pretrained model (YAMNet via tensorflow_hub, or
    #       openl3.get_audio_embedding) and run inference
    # TODO: pool frame-level embeddings into a single fixed-length vector
    #       (mean pooling is a fine first choice)
    ...

Architecture decision point (data storage): store raw audio permanently, or delete it after extracting the embedding? Storing it burns disk and raises copyright/privacy questions for no real benefit -- you only need the embedding, not the waveform, for everything downstream. Recommendation: extract-then-delete, keep only the derived vector. Write this down as a design principle now; it'll save you a difficult conversation with yourself in Month 4 when your disk is full of MP3s you don't need.

Milestone: pick five songs you know well, run them through the pipeline, and eyeball their embeddings' pairwise cosine similarity. Two songs by the same artist in the same genre should be closer to each other than to a wildly different genre. If your embeddings pass this smell test, you have a forensic lab that actually works. Screenshot the similarity matrix -- this is the moment the system starts to have taste.

Phase 4: The Behavioral Profiler -- Learning What You Actually Like (Weeks 11-14)

Goal: Turn raw watch behavior into an implicit "how much did you like this" score, and build your first learned model of taste.

Skills gained: Implicit feedback theory, feature engineering from event logs, neural collaborative filtering, thinking in terms of "signal vs noise" in behavioral data.

What you'll build:

• A feature engineering step that, per track, computes: percent watched, replay count within a session, skip-within-first-10-seconds flag, time-of-day distribution, day-of-week distribution.
• An implicit score -- not a 1-5 star rating (you never gave one), but a derived number. A simple, honest starting formula:

implicit_score = w1 * completion_ratio
                + w2 * log(1 + replay_count)
                - w3 * early_skip_penalty

where completion_ratio = watched_seconds / video_duration, capped at 1.0, early_skip_penalty is 1 if you skipped within the first 10 seconds and never returned, else 0, and w1, w2, w3 are weights you pick by hand at first (try w1=1.0, w2=0.5, w3=0.8) and refine once you have enough data to eyeball whether the resulting rankings feel right. This is implicit feedback in a nutshell: you're inferring preference from behavior, not asking for it directly, and every implicit signal is a proxy, not ground truth -- a song you love but skip because you're not in the mood right now looks identical, in raw data, to a song you dislike. Time-of-day and mood context (Phase 5) is what starts to disambiguate the two.

• The math of implicit feedback, briefly: classic explicit recommender systems (star ratings) treat missing data as "unknown." Implicit feedback systems (like this one) treat everything as a signal -- even the absence of a play is information (you had the chance to watch it and didn't). The seminal framing here, worth reading about, is "confidence-weighted preference": you're not just modeling whether you like something, you're modeling how confident the system should be in that preference, where confidence scales with how many times you've interacted with a track.

• A first neural collaborative filtering sketch, once you have a few hundred (track, implicit_score, context) rows. Neural CF replaces the classic matrix-factorization approach (learn a vector per user, a vector per item, dot-product them to predict preference) with small embedding layers you can extend to include context features (time of day, mood tag) alongside the learned latent vectors.

Learning pit stops:

• 25 min: search "implicit feedback recommender systems explained" -- get comfortable with the idea that "no interaction" is still data.
• Skim: Keras's Embedding layer docs -- this is the core building block of neural CF.
• Optional deeper dive (30 min): the original "Collaborative Filtering for Implicit Feedback Datasets" paper's abstract and intro -- you don't need the full math, just the framing.

Skeleton (Keras):

# ml/behavior_model.py -- teaching the system your taste, one skip at a time
import tensorflow as tf
from tensorflow.keras import layers, Model

def build_neural_cf(n_tracks: int, embedding_dim: int = 32) -> Model:
    track_input = layers.Input(shape=(1,), name="track_id")
    context_input = layers.Input(shape=(4,), name="context_features")  # e.g. hour_of_day, day_of_week, recent_genre_onehot...

    track_embedding = layers.Embedding(n_tracks, embedding_dim)(track_input)
    track_vec = layers.Flatten()(track_embedding)

    # TODO: concatenate track_vec with context_input
    # TODO: pass through 1-2 Dense layers with ReLU
    # TODO: final Dense(1, activation='sigmoid') to predict a
    #       normalized implicit_score in [0, 1]

    ...
    return Model(inputs=[track_input, context_input], outputs=..., name="neural_cf")

def compute_implicit_score(row) -> float:
    completion_ratio = min(row["watched_seconds"] / row["duration_seconds"], 1.0)
    replay_bonus = 0.5 * (row["replay_count"] ** 0.5)  # diminishing returns
    early_skip_penalty = 0.8 if row["early_skip"] else 0.0
    return max(0.0, completion_ratio + replay_bonus - early_skip_penalty)

Architecture decision point: do you train one global model, or treat this as pure feature engineering with no trained model at all (just weighted heuristics)? With a personal dataset of maybe a few thousand tracks and a few weeks of history, you may not have enough data for a neural model to meaningfully outperform a well-tuned heuristic score. Recommendation: ship the heuristic first (it's basically free), and treat the neural CF model as a learning exercise you compare against the heuristic -- a great "does my fancy model actually beat my simple baseline?" experiment, which is one of the most important habits in applied ML.

Milestone: for a handful of tracks you have strong feelings about, check whether your implicit score ranks them the way you'd expect -- your most-replayed song should sit near the top, something you skipped every time near the bottom. If the ranking basically matches your gut, you've built a taste profile that didn't exist an hour ago in any explicit form.

Phase 5: The Case Board -- Clustering & Playlist Generation (Weeks 15-18)

Goal: Group tracks into meaningful clusters and turn each cluster into a named, explainable playlist.

Skills gained: Unsupervised clustering (HDBSCAN, K-Means), cluster labeling strategies, combining embeddings from different modalities (audio + behavior + text mood tags), scheduling recurring jobs.

What you'll build:

• A combined feature vector per track: concatenate (or weighted-sum) the audio embedding, a mood-tag one-hot/multi-hot vector (from zero-shot classification against labels like "chill," "energetic," "sad," "focus," "romantic," "aggressive"), and behavioral context features (average time-of-day, day-of-week distribution).
• A clustering pass using HDBSCAN over that combined vector space. Why HDBSCAN over plain K-Means: K-Means forces every point into a cluster and requires you to pre-specify the number of clusters -- a bad fit here, because you genuinely don't know how many "moods" exist in your listening history, and forcing an odd one-off track into a cluster it doesn't belong in produces bad playlists. HDBSCAN discovers the number of clusters itself and explicitly labels outliers as noise (cluster = -1) rather than cramming them somewhere.
• A cluster-labeling step: for each cluster, look at the most common mood tags among its members, the most common time-of-day, and generate a human name via simple rules ("if dominant mood is 'chill' and dominant time is evening -> name candidates: 'Evening Chill', 'Wind Down'"). This is a fun place to eventually swap in an LLM call ("given these five representative songs and these mood tags, suggest a playlist name") as a stretch goal.
• Specialized playlist types layered on top of the general clustering:
  • Most-played: pure sort by implicit_score / play_count, no clustering needed.
  • Mood playlists: direct output of the HDBSCAN clusters.
  • Study / focus: filter tracks where the zero-shot classifier's "instrumental" or "low-vocal-density" signal is high and mood tag includes "calm" or "focus" -- this is a good spot to add a simple vocal-presence heuristic (spectral features like zero-crossing rate can proxy for vocal presence without a dedicated model).
  • Travel: filter by tracks played during sessions your extension can tag as "away from home" if you choose to add coarse geolocation (optional, privacy-sensitive -- see Chapter 4) -- or more simply, tracks played during longer, uninterrupted sessions (a proxy for commutes or trips).
  • Language: filter using a language-detection pass over the (translated or original) lyrics/title text, using a lightweight library like langdetect or a Hugging Face language-ID model.

Learning pit stops:

• 20 min: read HDBSCAN's own documentation "How HDBSCAN Works" page -- it's written for exactly this level of newcomer and has excellent visuals.
• Skim: scikit-learn's clustering comparison page to see K-Means vs DBSCAN vs HDBSCAN side by side on toy datasets -- seeing the failure modes visually will save you hours of confusion later.

Skeleton:

# ml/cluster.py -- the case board, where scattered evidence becomes a story
import hdbscan
import numpy as np

def build_combined_vector(audio_emb, mood_vec, behavior_features) -> np.ndarray:
    # TODO: consider normalizing each component before concatenating --
    #       otherwise a component with larger raw magnitude (e.g. audio_emb)
    #       will dominate the distance metric HDBSCAN uses
    ...

def cluster_tracks(vectors: np.ndarray, min_cluster_size: int = 5):
    clusterer = hdbscan.HDBSCAN(min_cluster_size=min_cluster_size, metric="euclidean")
    labels = clusterer.fit_predict(vectors)
    # labels == -1 means "noise" -- HDBSCAN is telling you this track
    # doesn't fit cleanly anywhere yet. That's useful information, not a bug.
    return labels

def name_cluster(track_rows_in_cluster) -> str:
    # TODO: find most common mood tag, most common time-of-day bucket
    # TODO: map (mood, time_bucket) -> candidate name via a lookup table
    # TODO: fall back to "Mix #<id>" if nothing matches cleanly
    ...

Architecture decision point: re-cluster from scratch every time, or incrementally assign new tracks to existing clusters? Full re-clustering is simpler to reason about and is what you should build first -- run it nightly over your whole track history. Incremental assignment (using each cluster's centroid or a trained classifier) is faster and preserves playlist identity over time (your "Monday Chill" playlist doesn't get renamed every week), and is a great optimization once the naive approach is working and you notice it's slow or that playlist churn is annoying you.

Milestone: run the full pipeline end to end and get back a JSON structure of named playlists with real tracks in them. Take a screenshot of the raw output -- you now have an AI DJ's set list, even before there's a frontend to look at it.

Phase 6: The Broadcast -- Web Player & Full Integration (Weeks 19-22)

Goal: A React frontend that displays your playlists and actually plays music, wired end-to-end to everything you've built.

Skills gained: React component architecture, working with an audio player (HTML5 <audio> or the YouTube IFrame API), API integration patterns, and the specific discipline of wiring together five already-working pieces without breaking any of them.

What you'll build:

• A GET /playlists endpoint that returns your generated playlists with track metadata.
• A React app with a playlist grid, a track list per playlist, and a persistent bottom player bar (the "Spotify-like" feel).
• Playback, decision point spelled out: you have two real options. (a) Embed the YouTube IFrame Player API and drive it programmatically -- simplest, fully compliant with YouTube's terms, but visually and functionally you're still "a YouTube player with a nicer wrapper," and you can't easily do gapless playback or true background audio. (b) Serve audio you've extracted yourself, from your own backend, for genuinely personal, non-redistributed use -- gives you a true from-scratch audio player experience (waveforms, gapless transitions, real <audio> element control) but carries real copyright and terms-of-service responsibilities you must take seriously and keep strictly personal/local, never exposed publicly. Recommendation: build the IFrame API version first, since it's unambiguously fine to build and ship as a personal tool, and treat the self-hosted audio player as an explicitly-labeled "local-only, for-my-eyes-only experiment" if you choose to explore it, never something you deploy publicly.
• A "similar tracks" widget: given the currently playing track's embedding, query pgvector for nearest neighbors (ORDER BY embedding <-> :current_embedding LIMIT 5) and render them as suggestions.
• Wire the frontend's play events back into the extension's event pipeline (Chapter 1's feedback loop closes here) -- plays that originate in your own app should count as behavioral signal too.

Learning pit stops:

• 20 min: the YouTube IFrame Player API "Getting Started" guide -- focus on loadVideoById, playVideo, and the onStateChange event.
• Skim: any short React hooks refresher if it's been a while (useState, useEffect, useContext for a simple global "currently playing" state).

Skeleton:

// frontend/src/components/Player.jsx -- the stage where the case gets solved
import { useState, useEffect, useRef } from "react";

export default function Player({ currentTrack }) {
  const playerRef = useRef(null);

  useEffect(() => {
    if (!currentTrack) return;
    // TODO: if using YouTube IFrame API:
    //   playerRef.current.loadVideoById(currentTrack.video_id);
    // TODO: log a 'play' event back to your /events endpoint here,
    //       reusing the same schema Phase 1 established
  }, [currentTrack]);

  return (
    <div className="player-bar">
      {/* TODO: track title, artist, progress bar */}
      {/* TODO: mount the YouTube IFrame player (hidden or small) here */}
    </div>
  );
}

# backend/main.py (addition) -- serving the case board to the courtroom
@app.get("/playlists")
def get_playlists():
    # TODO: query your clusters/playlists table
    # TODO: for each playlist, include a handful of representative tracks
    #       and the playlist's dominant mood tags for display
    ...

@app.get("/tracks/{track_id}/similar")
def similar_tracks(track_id: str, limit: int = 5):
    # TODO: SELECT * FROM tracks ORDER BY embedding <-> (
    #         SELECT embedding FROM tracks WHERE id = :track_id
    #       ) LIMIT :limit
    ...

Milestone -- the big one: open your app, see real named playlists built from your real YouTube behavior, click play, and hear music. Take a screenshot of "Monday Chill" (or whatever your system named it) with real tracks in it. This is the moment described in the opening story finally closes the loop -- you are, quite literally, looking at the output of a detective story you wrote about yourself.

Chapter 3: The Brain of the System — Meet Your Neural Concierge

Phases give you a build order. This chapter zooms into the thinking tools you'll reach for repeatedly — the habits of mind that separate "I copied a tutorial" from "I understand what I built."

How to search, read docs, and debug like you've done this before

You haven't done this before, and that's fine — nobody starts out knowing how to google effectively for a niche integration. Here's the actual process, made explicit:

1. Name the smallest unit of your confusion. Not "how do I do audio classification" — that's a whole field. Instead: "how do I get a fixed-length vector out of a variable-length audio file using a pretrained model." That sentence is close to a good search query.
2. Search for the library name plus the verb, not the concept plus the verb. "yt-dlp extract audio python" beats "how to download youtube audio." Library-specific queries land you on docs and GitHub issues where someone already hit your exact wall.
3. Read the signature before the prose. When you land on a function's docs, look at its parameters and return type first. Half the time that alone answers your question faster than the paragraph above it.
4. When you're stuck for more than 20-30 minutes, write down what you expected to happen and what actually happened, in one sentence each. This is the single highest-leverage debugging habit there is — most of the time, writing that sentence reveals the bug before you've even searched anything, because it forces you to notice the gap between your mental model and reality.
5. Reproduce in isolation. If your pipeline breaks on the audio-embedding step, don't debug it inside the full nightly job — copy the failing call into a throwaway script or a notebook cell with one hardcoded input. You want the shortest possible loop between "change something" and "see the result."

You'll use this loop dozens of times across the next five months. It's worth more than any specific fact in this guide.

Trade-off thinking, one more time with feeling

Every phase above included at least one "architecture decision point," and that's not decoration — it's the actual skill this project is teaching you. Real systems are built from a sequence of decisions where there's no objectively correct answer, only a right answer for your constraints. The questions to ask yourself every time are the same four:

• What does each option cost me right now (time, complexity, new concepts to learn)?
• What does each option cost me later (technical debt, rework, scaling pain)?
• Do I actually have the problem this option solves, or am I solving a problem I imagine having?
• Can I try the cheap option first and upgrade later, or is this a decision that's expensive to reverse?

Notice that almost every recommendation in this guide follows the same shape: start with the boring, cheap, well-understood option; treat the fancy option as a deliberate, isolated upgrade once you've felt the limits of the simple one. That's not a lack of ambition — it's how you make sure the ambition is spent on the parts of the project that are genuinely novel (your specific taste model, your specific clustering logic) instead of getting burned on generic infrastructure choices that a thousand tutorials have already solved for you.

NLP for mood: zero-shot first, fine-tuning as a deliberate upgrade

You'll use Hugging Face's zero-shot-classification pipeline in two places: initial content classification (Phase 2) and mood tagging (Phase 5). It's worth understanding why zero-shot works at all — the underlying model (typically an NLI, natural-language-inference, model like BART-MNLI) was trained to judge whether one sentence entails another. Zero-shot classification is a clever repurposing: it turns "does this song feel energetic?" into "does the premise 'this text describes energetic music' entail the hypothesis built from your candidate label," and ranks labels by entailment confidence. You get calibrated-ish multi-label scores over any label set you dream up, with zero training.

The catch: zero-shot models are working from text — titles, descriptions, maybe lyrics if you fetch them — not from the audio itself. A song titled "Sunny Day Vibes" will zero-shot-classify as upbeat even if the actual track is a somber ballad using the title ironically. This is exactly why Phase 5's combined vector blends the zero-shot mood tags with the audio embedding — text tells you what the creator labeled the song as, audio tells you what it actually sounds like, and behavior tells you how you actually responded to it. None of the three signals is trustworthy alone; together they triangulate something closer to the truth.

When to fine-tune instead of relying on zero-shot: once you've hand-corrected mood tags on maybe 200-300 tracks (a natural byproduct of using your own app and noticing "this is tagged wrong"), you have enough labeled data to fine-tune a small classifier — even a lightweight one trained on top of frozen text embeddings — specifically on your vocabulary of moods and your taste in how songs get labeled. This is a genuinely great Month 4-5 stretch goal: it's the moment your mood tagger stops being "a generic pretrained model" and starts being "a model that knows your ears."

The math of implicit feedback, one level deeper

Section from Phase 4 gave you a working formula. Here's the conceptual core worth sitting with: in explicit feedback (star ratings), the model's job is to predict a rating, and the loss function punishes wrong ratings directly. In implicit feedback, there's no rating to predict — only binary or continuous signals of engagement. The standard reframe (going back to Hu, Koren & Volinsky's foundational work) is to treat every (user, item) pair as having a binary preference (did they like it, roughly, yes/no) plus a confidence in that preference, where confidence grows with how much engagement evidence you've observed. A song you've watched once for 10 seconds contributes a weak, low-confidence data point. A song you've replayed eleven times contributes a strong, high-confidence one. Your loss function (if you get to training a real model) should weight errors on high-confidence pairs more heavily than errors on low-confidence ones — getting a well-established favorite's ranking wrong is a bigger problem than getting an ambiguous one-play track wrong.

Practically, for a personal-scale project, you don't need to implement the full weighted matrix factorization machinery to benefit from this idea — just make sure your implicit_score formula and your neural CF's context features both account for "how many times have I actually interacted with this" as a first-class signal, not an afterthought.

Neural collaborative filtering, sketched a little further

The Phase 4 skeleton gave you the shape: an embedding layer per track, concatenated with hand-engineered context features, passed through dense layers to a final prediction. A few things worth understanding about why this works before you write more of it:

• The embedding layer is learning a compressed "taste coordinate" for each track — not from any explicit attribute you gave it, but purely from patterns in which tracks tend to get high implicit scores together, in similar contexts. Two tracks can end up with similar embeddings even if their audio sounds nothing alike, if you consistently listen to them in the same moods/times.
• This is a genuinely different signal from the audio embedding. Audio embeddings capture "sounds like." CF embeddings capture "gets consumed like." A great expansion once both exist: concatenate them (as Phase 5 does) so your clusters reflect both what things sound like and how you actually use them.
• With a small personal dataset, expect this model to overfit fast. That's not a failure — it's expected, and it's a good opportunity to learn regularization basics (dropout between dense layers, L2 weight decay on the embedding) and to practice train/validation splitting even when your dataset is uncomfortably small. Keep a held-out slice of your most recent week's data and check whether the model's predicted scores actually track your real subsequent behavior — that's your only honest signal that any of this is working.

Chapter 4: The Data Story — Where Evidence Lives

Every detective needs an evidence locker with a sane filing system. Here's yours.

The schema, roughly

• raw_events — the unfiltered sensor feed. Every play, pause, seeked, progress, ended tick from the extension, tied to a video_id and session_id. This table only grows; you never edit it, only append. Treat it the way you'd treat a security camera's footage — the raw record, kept intact, that every later interpretation can be checked against.
• tracks — one row per resolved song: video_id, artist, title, channel, is_music (bool), duration_seconds, audio_embedding (a pgvector column), mood_tags (array or jsonb), language, musicbrainz_id (nullable), implicit_score (recomputed periodically), created_at, updated_at.
• sessions — optional but genuinely useful: groups of events with a shared session_id, with derived fields like session_start, session_length, dominant_time_bucket (morning/afternoon/evening/night). This is where "travel" and "study" playlist heuristics get their signal.
• playlists and playlist_tracks — the output of the clustering step: a playlist has a name, a generation timestamp, dominant mood tags, and a join table linking it to track rows, ordered.

The key design idea: raw data is immutable, derived data is regenerable. If your classification logic improves in Month 4, you should be able to delete and recompute the tracks table's is_music and mood_tags columns from raw_events without losing anything real. This is the same principle behind "extract-then-delete" for audio from Phase 3 — keep the cheap-to-regenerate stuff disposable, keep the expensive-to-recreate stuff (your actual watch history) sacred.

Bootstrapping with Google Takeout

Your extension only sees behavior from the moment you install it forward. If you want historical depth on day one, Google Takeout's YouTube history export gives you a JSON/HTML dump of your watch history going back as far as YouTube has recorded it. It's coarser than extension data — no watch percentage, no skip detection, just "you watched this, at this timestamp" — but it's real, and it's enough to seed your tracks table and get Phase 2's classifier something to chew on in week one, instead of waiting three weeks for organic data to accumulate.

How to use it: request a Takeout export (JSON format), write a one-off import script that reads the export and inserts synthetic raw_events rows with event_type = 'legacy_watch' and no position data, then let Phase 2's classifier run over them exactly as it would over live data. Treat legacy rows as lower-confidence in your implicit-score formula — you don't know how much of the video was actually watched, so don't let a legacy "watched" event count as strongly as a fully-instrumented one.

Simulating data for development

You don't want to wait around watching YouTube for hours just to test whether your clustering code handles 500 tracks reasonably. Write a small synthetic data generator early — even a rough one — that creates plausible-looking raw_events rows: pick from a pool of real song titles (you can hand-curate 100-200 from your own taste, spanning a few genres/moods), assign random-but-plausible watch percentages and timestamps clustered around a few "mood personas" (a synthetic "you" who watches chill music at night and energetic music in the morning). This does two things: it lets you develop and test the ML/clustering code without waiting on real data, and it gives you a known-answer test — since you control the synthetic personas, you can check whether your clustering actually recovers something close to them, which is a much stronger validation than "the code ran without crashing."

Privacy and local-first design

This system is, by design, an intimate profile of your habits — what you listen to, when, how obsessively. A few commitments worth making explicit before you write a line of storage code:

• Keep it local by default. Run Postgres on your own machine or a private instance you control, not a shared cloud service, unless you've deliberately decided to and secured it (auth, no public ports, encrypted connections).
• Never redistribute extracted audio. Phase 3's "extract-then-delete" isn't just an engineering nicety — it's the difference between "a personal analysis tool" and "a tool with a copyright problem." Keep audio files ephemeral, process-and-discard, never served to anyone but the pipeline that needs them for a few seconds.
• Be deliberate about optional signals like geolocation. The "Travel" playlist idea in Phase 5 floated coarse geolocation as one possible signal. If you add it, make it opt-in, store it at low resolution (city-level, not GPS-precise), and be honest with yourself about whether the marginal playlist quality is worth the marginal privacy cost. The session-length proxy (no geolocation needed) gets you most of the value for none of the risk — a good default.
• You are both the engineer and the subject. That's unusual and worth sitting with: normally, privacy-by-design is about protecting other people's data from your system. Here, you're the one whose intimate behavioral data is on the line, which makes it a genuinely good, low-stakes place to practice privacy-conscious engineering habits before you ever build something that touches anyone else's data.

Chapter 5: The Galaxy of Songs — Playlist Generation Magic

Embeddings, explained like you're teaching a friend at a bar

Imagine every song as a point of light in a vast dark space. Songs that sound alike, or that you tend to listen to in similar moods, drift close together — not because anyone placed them there by hand, but because the math that produced their coordinates was trained (or, for pretrained models, trained by someone else on a huge amount of audio) to put "things that behave similarly" near each other and "things that behave differently" far apart. That's an embedding: a long list of numbers (say, 128 or 512 of them) that represents a song not as an ID, but as a position in a meaningful space. Two songs' embeddings being close together, measured by something like cosine similarity, is a proxy for "these songs are alike in whatever sense the model was trained to notice."

You have two galaxies in this project: the audio galaxy (positions determined by what things sound like) and the behavioral galaxy (positions determined by how you actually consume them, learned via the collaborative-filtering embeddings from Chapter 3). A track's true "meaning" in your system lives at the intersection — which is exactly why Phase 5 concatenates both into one combined vector before clustering. A cluster in that combined space isn't just "songs that sound similar," and it isn't just "songs you listen to in similar contexts" — it's songs that are similar on both axes simultaneously, which is a much stronger, more personally meaningful notion of "these belong on the same playlist" than either signal alone could produce.

Clustering and automatic labeling, tied together

HDBSCAN doesn't know what a "mood" is. It only knows distances. The intelligence in the system comes from what you fed it (the combined embedding) and how you interpret its output (the labeling heuristic). This division of labor is worth internalizing as a general ML pattern: unsupervised algorithms are excellent at finding structure, and terrible at explaining what the structure means — that translation step is where your domain knowledge (you know what "chill" sounds like to you) has to do the work, usually through a simple, inspectable rule rather than another opaque model. Keep that labeling step simple and debuggable for as long as possible; it's the layer you'll be staring at and tweaking the most, because it's the layer that decides whether a playlist gets a name you'd actually want to click.

Filtering for activity-based playlists

"Study" and "Travel" playlists aren't clusters in the audio/behavior space so much as they're filters layered on top of it — a study playlist is really "the calm, low-vocal-density cluster, further filtered to sessions where you historically kept the same track playing for a long uninterrupted stretch (a proxy for focus, since skip-heavy sessions suggest browsing, not working)." Treat these as compositions of your existing signals rather than new things to model from scratch — it's both less work and more explainable, which matters a lot when you're debugging why a jazz track ended up in "Study" and a moment's thought about "oh, right, low vocal density plus long uninterrupted plays" tells you exactly why.

A future reinforcement learning layer

Once the system exists and you're using it daily, a natural next question emerges: instead of only observing your behavior passively, could the system learn from how you react to its own suggestions? This is where a lightweight RL framing becomes interesting — treat playlist generation as a policy that chooses which tracks to include, treat your subsequent behavior (did you skip the second track in "Monday Chill," did you let the whole playlist play through) as a reward signal, and use something like a contextual bandit (a simpler, more tractable cousin of full RL, well suited to problems where you're repeatedly choosing from a set of options and observing an immediate reward) to nudge future playlist composition toward what you actually engage with. This is explicitly a later idea — Chapter 9 revisits it as a genuine "beyond the guide" project, because it depends on having a stable, working system generating playlists you actually use regularly first. RL on top of a system that doesn't have real usage data yet is RL on noise.

Chapter 6: The Web Dev Spine — Extension, Backend, Frontend, Tied Together

The ML chapters get the glamour, but this is the skeleton everything else hangs on. A quick tour of the parts you haven't already built phase-by-phase, plus the connective tissue between them.

The extension, structurally

A Chrome extension (Manifest V3) has three moving pieces you need to hold in your head as separate execution contexts that can't directly touch each other's variables — they only talk via message passing:

• manifest.json — declares permissions (activeTab, host permissions for *.youtube.com), which scripts run where, and whether you have a background service worker.
• The content script — runs inside the YouTube page's context, so it can read the DOM and attach listeners to the <video> element, but it's sandboxed from the extension's other state and can't make cross-origin requests as freely.
• The background service worker — has broader permissions (can make network requests to your backend without CORS headaches, can persist state across tab navigations) but can't touch the page DOM directly.

The message-passing pattern you'll lean on constantly: content script calls chrome.runtime.sendMessage({type: 'EVENT', payload: {...}}), background script listens via chrome.runtime.onMessage.addListener, batches, and does the actual fetch to your API. Get comfortable with this pattern early — Phase 1 depends on it, and every later extension feature (detecting SPA navigation between videos, badge icon updates showing tracking status) reuses it.

REST API design that won't fight you later

A few conventions worth committing to from Phase 1 onward, because retrofitting them later is annoying:

• Version your API from day one (/api/v1/events), even though you're the only consumer. Future-you, three months in, adding a mobile client or a second extension version, will thank present-you.
• Keep ingestion endpoints (POST /events) fast and dumb. They should do validation and a write, nothing else. Any classification, embedding, or clustering work belongs in the offline pipelines, not inline in the request path — mixing them means a slow ML call can make your extension feel laggy or drop events under load.
• Design read endpoints (GET /playlists, GET /tracks/{id}/similar) around what the frontend actually needs to render, not around your database schema. It's fine, even good, for an endpoint to join and reshape data server-side so the frontend doesn't have to stitch together three separate calls just to draw one screen.

Streaming audio, and the ethical note this section owes you

If you go the self-hosted-audio route from Phase 6's decision point, you're technically capable of building a /stream/{track_id} endpoint that reads an extracted audio file and streams it with proper Range header support so the browser's <audio> element can seek. Build it if you want the learning experience of implementing HTTP range requests (a genuinely useful, transferable skill — it's the same mechanism video platforms use). But hold the line firmly: this stays a local, single-user, never-publicly-deployed feature. The moment "personal tool running on my own machine for my own previously-watched content" turns into "a service anyone else can reach," you've crossed from a personal fair-use-adjacent experiment into redistribution, which is a different, real legal category. If you ever want to share this project publicly (a portfolio demo, a hackathon submission), disable or strip the self-hosted audio path and demo with the YouTube IFrame API version instead — it's not a downgrade, it's the responsible default.

Frontend architecture, briefly

Keep state simple: a top-level currentTrack and currentPlaylist in React context (or a lightweight state library if you want the practice), a PlaylistGrid component that fetches from /playlists on mount, a Player component that owns the actual playback element and logs playback events. The "similar tracks" widget is a nice small component to practice component composition on: it takes a trackId prop, fetches /tracks/{id}/similar on prop change, and renders a small horizontal list — self-contained enough to build and test in isolation before wiring it into the main player view.

Tying the vector database to the frontend

The satisfying part of pgvector is that "find similar tracks" is one query, not a separate service call to a vector database plus a join back to your relational metadata:

SELECT id, artist, title, embedding <-> (
    SELECT embedding FROM tracks WHERE id = :current_track_id
) AS distance
FROM tracks
WHERE id != :current_track_id
ORDER BY distance
LIMIT 5;

That single query is doing something that would otherwise require standing up and syncing two separate systems. It's a small thing, but it's the kind of small thing that, once you've felt the alternative, you start actively choosing simpler architectures for the rest of your career.

Chapter 7: The Detective's Logbook — Milestones Across the Whole Case

You've hit a concrete milestone at the end of every phase in Chapter 2, but it's worth stepping back and seeing the whole arc as one continuous story, because that's genuinely what it is:

• Phase 0: two independent heartbeats — an extension that logs, a server that responds.
• Phase 1: the system sees you for the first time. A real, timestamped trail of your own behavior, sitting in a database, that didn't exist yesterday.
• Phase 2: the system starts telling music apart from everything else — its first act of judgment.
• Phase 3: the system develops senses beyond text — it can now "hear," in the limited but real sense of turning sound into a comparable, structured signal.
• Phase 4: the system develops a memory of your taste that isn't just "what you told it" but "what you actually did," which is a fundamentally more honest signal than any explicit rating system.
• Phase 5: scattered evidence becomes a story — songs organize themselves into groups you didn't manually create, with names that feel earned rather than assigned.
• Phase 6: the system finds its voice, and hands the story back to you in a form you can actually use, day to day, the way you'd use Spotify — except every playlist in it is an artifact of your own engineering.

Each of those is worth a screenshot, a commit message that actually describes what changed and why, and — genuinely — a moment of letting yourself feel like this is a big deal. It is. Most people who use recommendation systems every day have no idea how the black box works. You're about to have built one, from the sensor all the way to the speaker.

Chapter 8: Beyond the Guide — Where the Case Reopens

Five months from now, you'll have a working, personal AI DJ. Here's where the story can keep going, roughly in order of "natural next step" to "ambitious departure":

• Deploy it somewhere always-on, like a Raspberry Pi or a small home server, so the nightly pipeline runs whether or not your laptop is open. This is a good, contained systems-administration project on its own — cron jobs, systemd services, basic monitoring so you notice if the pipeline silently stops working.
• Add voice control — "play something chill" parsed via a small intent-classification step (zero-shot classification again, this time over your playlist names and mood tags) wired to a wake-word library or a simple push-to-talk button in the frontend.
• Integrate with the real Spotify API as a second data/playback source, letting your behavioral model learn from both platforms at once, and letting your generated playlists optionally get pushed into actual Spotify playlists you can use anywhere, on any device.
• Build the reinforcement learning layer sketched in Chapter 5 — once you have real usage data (skips, replays, completions) on the system's own suggestions, not just your organic YouTube behavior, you have the reward signal that makes a bandit-style approach genuinely meaningful instead of speculative.
• Turn it into a SaaS, if you want the entirely different (and genuinely valuable) experience of learning multi-tenant architecture, per-user data isolation, OAuth flows for other people's YouTube accounts, and all the "now it's not just your data anymore" privacy engineering that Chapter 4 flagged as a different category of problem. Don't take this step lightly or quickly — it's a good six-month project in its own right, layered on top of the five months you'll have already spent.

None of these are required to call the project "done." The five-month build is a complete story with a real ending. Everything in this chapter is a sequel hook, not an unfinished obligation.

Chapter 9: The Final Note

You started this case with an irritating, wonderful little observation: somewhere in your YouTube history was a pattern nobody, not even you, could see clearly. Over the next five months, you built the instrument to see it — a browser extension that learned to watch the way you watch, a pipeline that learned to hear the way songs sound, a model that learned your taste from what you actually did instead of what you said, and a clustering engine that turned all of that into playlists with names that feel like they know you, because in the most literal sense, they do.

Along the way you picked up a working fluency in things that sound intimidating from the outside and turn out to be learnable in exactly the incremental way this guide walked you through: DOM events and extension messaging, REST API design, mel spectrograms and audio embeddings, zero-shot NLP classification, the actual math behind implicit feedback, neural collaborative filtering, density-based clustering, vector similarity search, and a real frontend audio player. Each of those, on its own, is a legitimate skill line on a resume. Together, wired into one living system, they're something rarer: a complete, working thing you built end to end, that solves a problem only you have, in a way that only you could have tuned it.

You started with a YouTube habit. You're ending with a machine learning orchestra — one that watches, listens, remembers, and plays, quietly, in the background of your life, getting a little more you every single day it runs.

Case closed. Time to hit play.

javascript
localStorage.setItem(_cftscs_${fe}, n.scs);
javascript
ce = localStorage.getItem(scsKey);
// No IP check. No browser check.
No validation. Just works anywhere.

Their excuse:
"Turnstile is out of scope."
The policy says "automated bypasses via script" are excluded.
I said "I did it manually, no script."
They said "we know the wording is vague, we'll fix it." Then locked the report.
They took my research. They're fixing it. I got nothing.
That's how bug bounties work sometimes.
Companies want your work. They just don't want to pay for it.

The internet is becoming toxic and AI is drinking from the stream.

When Truth Becomes Optional

When Meta Platforms removed its last layer of professional fact-checking, it didn't just change how people consume information, it altered how machines learn it. Today's AI systems are built on yesterday's data, and when that data is contaminated, the foundations of intelligence begin to erode.

This piece examines how Meta's decision triggered a wider data contamination crisis, one that now threatens the reliability of artificial intelligence and, by extension, U.S. leadership in the global AI race.

From Moderation to Misinformation

On January 7, 2025, Meta announced the end of its U.S. third-party fact-checking program, replacing trained human reviewers with a crowd-sourced Community Notes system.

The change was positioned as a commitment to "free expression," but in practice, it removed one of the few remaining truth filters between misinformation and the world's data supply.

Studies from Cornell University show that such community systems still depend heavily on professional fact-checking inputs. Removing those experts weakens the very framework they rely on. Meta didn't just adjust a policy; it dismantled a safeguard.

The Data Pipeline Effect

Most large-scale AI models, including modern language models, are trained on open-web data. That means every public post, every shared article, every viral claim feeds back into the learning systems that shape digital reasoning.

When moderation falters, misinformation spreads unchecked. Those unverified fragments are then scraped, indexed, and transformed into training material for future models.

Each layer of falsehood compounds, producing what researchers call Model-Induced Distribution Shift, or, more simply, Model Collapse. As contaminated data multiplies, models begin learning from synthetic misinformation rather than verified human knowledge.

Industrial-Scale Data Poisoning

AI has long operated on the assumption that more data equals better intelligence. But as data volume accelerates and verification declines, that equation no longer holds true.

Recent studies highlight the fragility of this balance:

• ETH Zurich (2024) found that replacing just 0.001 percent of medical training data with misinformation caused models to produce statistically significant harmful advice.
• Anthropic (2024) demonstrated that as few as 250 malicious documents can poison a model regardless of its size, influencing behavior across fine-tuning cycles.
• arXiv (2024) described "persistent pre-training poisoning," where small corruptions persist through retraining and spread across multiple model generations.

Together these findings confirm a systemic risk: industrial-scale data poisoning, a feedback loop that turns digital learning into digital infection.

The Hidden Irony: Meta's Own Models Are Protected

While public data grows increasingly unreliable, Meta's internal AI systems are insulated from that decay. Its proprietary research models, including Llama 3, are trained on curated, licensed, and internally filtered datasets.

In essence, Meta protects its own intelligence from the very pollution its platforms unleash. The company maintains a clean, private data stream for internal AI while the public digital commons, the training ground for open-source and academic models, becomes increasingly toxic.

This creates a two-tier ecosystem:

• A verified internal dataset used for profit and research.
• A contaminated public dataset driving degradation elsewhere.

The imbalance isn't only ethical, it's strategic. Meta has built a firewall between what it sells and what it spreads.

National Implications: A U.S. Data Vulnerability

The consequences extend beyond the company itself. They now represent a structural weakness in the United States' race for AI dominance.

Erosion of Trust in U.S. Models

American models, trained predominantly on English-language data, are more exposed to misinformation circulating through Western social networks. Rival nations that enforce stricter data controls may soon produce more reliable systems.

The $67 Billion Hallucination Problem

In 2024, hallucinated AI output caused an estimated $67 billion in global business losses through legal disputes, compliance errors, and wasted verification time.

Adversarial Data Poisoning

Carnegie Mellon (2024) describes how state or non-state actors can manipulate AI indirectly by flooding public datasets with coordinated misinformation. The new battleground isn't infrastructure, it's the data supply chain itself.

If data integrity continues to weaken while competitors strengthen controls, the U.S. risks falling behind not because of slower innovation, but because of corrupted information ecosystems.

Restoring Data Integrity

Safeguarding the future of AI means rebuilding trust in the very data that trains it.

Several steps are critical:

Re-center Human Verification, Fact-checking is not overhead; it's infrastructure. Human-in-the-loop review must anchor digital information systems.

Data Provenance and Filtering, Developers must trace dataset origins and weight sources by reliability, not volume.

Establish Truth Datasets, Governments and research alliances should build continuously verified corpora for AI training, insulated from open-web contamination.

Policy Alignment, Platforms that monetize unverified content must meet the same integrity standards they apply internally.

The Broader Question

The health of artificial intelligence is inseparable from the health of the information it learns from. Meta's rollback of professional moderation didn't just expose users to misinformation, it injected uncertainty into the global data stream that powers modern intelligence.

If we teach machines that truth is optional, they will learn to lie beautifully and fail catastrophically.

The question ahead is not merely technical; it's philosophical: Will the intelligence we build reflect our pursuit of truth, or our tolerance for distortion?

References & Further Reading

• Meta Platforms Ends U.S. Fact-Checking Program, Reuters (2025)
• Meta Scrapped Fact-Checkers Because Systems Were "Too Complex", The Guardian (2025)
• Crowdsourced Fact-Checking Depends on Expert Validation, Cornell University (2024)
• Small Samples, Big Impact: Data Poisoning in LLMs, Anthropic Research (2024)
• Can Poisoned AI Models Be Cured?, ETH Zurich (2024)
• Persistent Pre-Training Poisoning, arXiv Preprint (2024)

RELATED ARTICLES

• → Sustainable Cognition
• → Cognitive AI, the Energy Obstacle, and the Artemis Solution
• → Cognitive AI: The Next Leap from Algorithms to Awareness

Loot #1: a match for the smoke detector

Someone ran a free security scanner against ten vulnerabilities they'd planted on purpose. It caught three. Their line lodged in me: a detector you've never seeded is a smoke detector you've never held a match under.

I had exactly that — a static security scanner watching my own codebase — and I had never once tested whether it could actually see. So I planted ten known-bad patterns and ran it. Seven of ten. I wrote rules for the three it missed, ran it again, got ten of ten. Then the part that made me laugh, because it happened to them too: the moment I added the "hardcoded secret" rule, it fired on real code — three live API keys I'd have sworn weren't there, in programs I'd already "reviewed." Seeding the detector was quietly also an audit of everything it had been failing to catch.

Loot #2: a guard for the guard

Someone wrote about a monitoring hook that sat silently dead for 23 days. Nobody noticed, because zero warnings looked exactly like healthy. Their line: a dead guard produces the same visible result as a clean pass — nothing.

I'd just made my systems' health-check smarter: it now runs a real database query instead of trusting that a page loaded. I was proud of it for about a day, until that post showed me the hole. I'd made the health-check smarter, but nothing was watching the health-check. If its scheduler quietly stopped firing, my dashboard would stay green out of pure absence — and I'd have read that silence exactly the way their team did. So I added a dead-man's switch: a separate job that raises the alarm if the health-check didn't leave a fresh proof-of-life behind. Now the guard has a guard. (The recursion has to stop at a human eventually — but one layer up from where I was.)

Loot #3: the one you're reading right now

Someone built a hook that stops an AI agent from doing all the research and then punting the decision back to you — "so which do you prefer?" — instead of committing to a recommendation.

I gave the little team of AI agents I run a standing rule: recommend one option, give a one-line reason, and name the assumption that would make it wrong. Felt good. Then the author said something that stuck: don't trust that it's working — count the leak. So I actually counted. I went back over one long working session and tallied every point where a decision got handed back to me. Full punts were zero of six. But in two of six, I'd quietly dropped the "here's what would make this wrong" line — and both times it was on calls I'd filed as yours to make anyway. The exemption I'd given myself was the exact place it leaked, and I'd never have seen it without counting.

Now here's the theft you didn't notice. The way I've been making recommendations across this whole series — "here's my pick, here's the reason, here's the assumption that would make it wrong" — I stole that too. From that post. You've been reading stolen goods this entire time.

Loot #4: an eye that can't lie

Someone building a design tool wrote the sentence I needed most: my eyeballs were wrong 40 percent of the time; a short script was right every time.

I'd been approving the color contrast in my tools by eye for months — squinting at a button, deciding it looked fine, moving on. That afternoon I wrote a tiny script that checks contrast against the accessibility standard, and ran my whole palette through it. The colors I'd agonized over were fine. But a link color I'd approved by eye months earlier was sitting below the readable threshold — quietly failing, invisibly, the whole time, for anyone whose eyes weren't as forgiving as mine. I'd never have caught it by looking. That's the point: looking was the problem.

What all four have in common

Line them up and they're the same lesson wearing four coats. The scanner that says it ran. The monitor that says it's up. The agent that says it's done. The eye that says it looks fine. Every one is a thing grading its own homework, and every fix was the same move: steal the check, not the confidence. Put something between you and the thing's opinion of itself — a planted failure, a proof-of-life, a tally, a script that doesn't have feelings about being right.

I didn't arrive at that principle. I stole it four times from four strangers before I noticed it was one idea. (And if you want the deep version of where that idea goes — who checks the checker, and why you can't do it alone — that's the next post.)

It's not a heist if the door's open

Here's what keeps this from being theft: nothing here is one-directional. My count of my own leak went back to the author of that hook, and it patched a blind spot his own tool couldn't see from inside. My notes on Korean typography went into another author's product roadmap. You take a line, you leave a line. That's the whole economy.

That's what the feed actually is, once you stop treating it as a broadcast channel — a shared workshop where everyone's tools are lying out on the bench, and "I'm stealing that" is just how we say thank you.

So: thank you. The bag's full. See you next post with a fuller one.

This is part of Stolen from the Feed — what a non-developer carries home from dev.to and actually builds. Sister series to From Zero to Ship.

For me, Docker was just this basic container-type thing where the configuration of my project is frozen so that it runs the same way on every device. Which is partly correct, but there's a much bigger picture and a lot more use cases than I initially realized.

More Than Just Images and Containers

One of the biggest advantages I noticed was how naturally everything became separated. Instead of treating my project as one giant application, I suddenly had a frontend image, a backend image, and a database image. Updating one service no longer meant rebuilding everything else.

Docker also taught me how to be more efficient when building applications. It introduced me to the concept of standalone builds, which significantly reduce image size while making builds faster and deployments more efficient.

docker-compose.yml

Along the way, I learned about containers, images, and volumes—how a single image can be used to create multiple containers, why database data should always live in volumes so it isn't lost when containers are recreated, and a whole lot more.

I also ended up writing a Docker Compose file for my project because, let's be honest, nobody wants to start every service one by one every single time. And oh boy, the amount of customization and configuration Docker Compose offers is unreal.

The fact that you can copy only the files you actually need into an image, define entire application stacks in a single file, and have Docker provide its own built-in DNS service so containers can talk to each other by name without worrying about IP addresses or routing... that's just insane. It genuinely made me appreciate how much work Docker hides behind such a simple interface.

After creating the Compose file and tweaking it until everything worked, I finally got all the services running on my localhost. Seeing the frontend, backend, and database come up together with a single command was incredibly satisfying.

moving file to the server

Now, all that was left was to move the codebase to the server along with the Compose file and see if everything worked outside my local machine.

Aaaand... I did exactly that.

I copied the project over, set up an SSH connection so GitHub and my server could trust each other (I'll cover that in a future episode because it's a pretty neat topic on its own), and got everything ready for the real test, and it worked !!!

problem with exposing ports

There was just one problem left.

At this point, the application was only accessible on localhost. On my Windows machine, only my Windows machine could access it. On the server, only the server itself could access it.

Sure, I could expose it using the server's IP address and access it from another device, but exposing your application directly over an IP isn't exactly a great idea. It works for testing, but it's definitely not how you'd want to host something that other people can access.

And i stumbeld upon a thing called nginx and apparently, the stack wasn't quite complete yet..

@mcp.tool()
def generate_commit_message(diff: str) -> str:
    """Generate a Conventional Commits message from a git diff string."""
    if not diff.strip():
        return "ERROR: empty diff — nothing to generate a commit message from."
    return _claude(diff, system=(...))

ERROR: as a prefix, so a caller can do if result.startswith("ERROR:") and know not to trust it as a commit message. That felt like the right convention at the time, and it is — as far as it goes. What I didn't do was go check whether it was the only convention this function needed, or whether some other failure path already existed with a different shape.

It did. generate_commit_message doesn't just call claude -p directly — it goes through a shared helper, _claude(), which has its own failure path for a hung subprocess:

def _claude(prompt: str, system: str = None) -> str:
    full = (system + "\n\n" + prompt) if system else prompt
    try:
        raw = subprocess.check_output(["claude", "-p", full], text=True, timeout=20).strip()
    except subprocess.TimeoutExpired:
        return "claude -p timed out after 20s"
    return "\n".join(
        l for l in raw.splitlines()
        if not _STRIP_RE.search(l)
    ).strip()

That timeout branch was added a couple of days before my empty-diff fix, in a different pass through this file, aimed at a different bug (the same call had no timeout at all and could hang git commit indefinitely). It does exactly what it says — stops the hang, returns a string explaining what happened — but it was written before the ERROR: convention existed, and nothing about adding that convention later went back and reconciled the two.

The result: generate_commit_message has exactly two ways to fail without raising, and they don't look like each other.

>>> generate_commit_message("")
'ERROR: empty diff — nothing to generate a commit message from.'

>>> generate_commit_message(some_diff)   # claude -p hangs past 20s
'claude -p timed out after 20s'

A caller that learned the right lesson from the empty-diff fix — check for the ERROR: prefix before trusting the return value as a commit message — checks it, gets False on a timeout, and happily commits the literal sentence claude -p timed out after 20s as if it were a real Conventional Commit subject. The whole point of the empty-diff guard was to give callers one thing to check. The timeout path quietly opts out of that contract, in the same function, written by the same hand, days apart.

I want to be honest about why this happened rather than just present the fix: it's a sequencing bug, not a subtlety I missed on inspection. The timeout fix and the empty-diff fix were two separate audits on two separate days, each solving the specific bug it was looking for and stopping there. Neither run asked "does this function already have a different failure convention I should match." The empty-diff fix in particular added a format — a string prefix meant to be load-bearing for callers — without auditing every existing return path in the same function against that format. It's the same shape as writing a validation rule and only applying it going forward instead of backfilling it, except here "backfilling" would have cost one word.

The fix is exactly that one word:

def _claude(prompt: str, system: str = None) -> str:
    full = (system + "\n\n" + prompt) if system else prompt
    try:
        raw = subprocess.check_output(["claude", "-p", full], text=True, timeout=20).strip()
    except subprocess.TimeoutExpired:
        return "ERROR: claude -p timed out after 20s"
    return "\n".join(
        l for l in raw.splitlines()
        if not _STRIP_RE.search(l)
    ).strip()

Now both failure paths speak the same language, and .startswith("ERROR:") is a real, complete contract instead of one that only covers the failure mode it happened to be written for.

The more general lesson is about where to look after fixing a bug that introduces a signaling convention. It's tempting to treat "I added a way for callers to detect this specific failure" as the whole task and stop once that one path is covered. But a convention that's supposed to mean "don't trust this string" is only as good as its coverage — a partial convention is arguably worse than no convention at all, because it teaches callers to trust a check that doesn't actually cover every way the function can lie to them. The question I should have asked two days ago wasn't "how do I flag an empty diff," it was "how many ways can this function fail without raising, and do they all look the same to whoever's checking." _claude() had two failure paths and I was only looking at one of them.

I didn't find this by staring at generate_commit_message in isolation — I found it by rereading _claude() right underneath it and noticing the two return statements didn't rhyme. That's a cheap check, one function's worth of scrolling, and it would have been just as cheap two days ago when the ERROR: convention was first introduced. It just wasn't part of that day's task.

Here's the line that actually did that check, until today:

def pending():
    try:
        drafted = open(DRAFTS, encoding="utf-8").read()
    except FileNotFoundError:
        drafted = ""
    out = []
    for a in api(f"/articles?username={ME}&per_page=100"):
        if not a["comments_count"]:
            continue
        for c in api(f"/comments?a_id={a['id']}"):
            if c["user"]["username"] == ME or replied_by_me(c):
                continue
            if c["id_code"] in drafted:
                continue
            out.append({...})

drafted is the raw text of the entire file. c["id_code"] in drafted is a substring check against that raw text, not a check against a set of actual header id_codes. I wrote it this way originally because it was the two-second version and it happened to work on every real case I tested it against.

The bug is that "already appears as a header" and "already appears anywhere in this file" are not the same claim, and the file is full of things that aren't headers. Every drafted reply in comment_replies.md quotes the commenter's original text and includes my own reply prose underneath the header. That prose is full of ordinary words, numbers, and dates. dev.to id_codes are short lowercase alphanumeric strings — things like 3b908, 3bdja, 39pk4 — and nothing stops a short alphanumeric token from showing up naturally inside a sentence.

I went looking for a live example instead of a hypothetical one. My own draft file has a reply that talks about FTS5 and BM25 scoring, and elsewhere just talks about the current year, 2026. 2026 is exactly the shape of a real id_code: four lowercase-alphanumeric characters, digits included. So I ran the actual check the function uses, against my real file, with a fake id_code that has never been a header anywhere in that file:

import re
text = open("drafts/comment_replies.md", encoding="utf-8").read()
headers = set(re.findall(r"^## (\S+)", text, re.M))

fake_new_code = "2026"
print(fake_new_code in headers)   # False — never actually drafted
print(fake_new_code in text)      # True — but the old check would skip it anyway

False
True

If a brand-new comment ever showed up on one of my articles with the id_code 2026, pending() would silently treat it as already handled and never print it. Nobody would see it in the JSON output. Nobody would draft a reply. And there'd be no error, no log line, nothing — the comment would just quietly not exist as far as the pipeline is concerned, forever, because a completely unrelated sentence somewhere else in the file happened to contain the same four characters.

What made this worth writing up rather than shrugging off as "unlikely" is that I already have the counter-example sitting right there in the same file, in the same function's neighbor. This script also has an audit() command, added a couple of days ago, that cross-checks drafted-but-never-actually-posted replies. audit() extracts id_codes the correct way:

def audit():
    ...
    drafted_codes = set(re.findall(r"^## (\S+)", drafted_text, re.M))

Anchored to the start of a line, matched against the exact ## <code> header format, collected into a set. That's the right way to ask "is this id_code one we've drafted." pending(), sitting forty lines above it in the same file, was asking a different and looser question — "does this string of characters occur somewhere in this document" — and getting away with it purely because no real comment had collided yet.

The fix is to make pending() use the same extraction audit() already uses:

def pending():
    try:
        drafted_text = open(DRAFTS, encoding="utf-8").read()
    except FileNotFoundError:
        drafted_text = ""
    drafted_codes = set(re.findall(r"^## (\S+)", drafted_text, re.M))
    out = []
    for a in api(f"/articles?username={ME}&per_page=100"):
        if not a["comments_count"]:
            continue
        for c in api(f"/comments?a_id={a['id']}"):
            if c["user"]["username"] == ME or replied_by_me(c):
                continue
            if c["id_code"] in drafted_codes:
                continue
            out.append({...})

Same shape, same cost, and now it can't be fooled by a coincidence in the prose. Reran the collision test against the fixed logic:

drafted_codes = set(re.findall(r"^## (\S+)", text, re.M))
print("2026" in text)            # True  (old check)
print("2026" in drafted_codes)   # False (new check)

What bugs me about this one is that it's not a rare edge case in some adversarial input — it's ordinary language sitting in a file this exact pipeline generates every single day. The file's job is to accumulate quoted human comments and my own replies about them, which means it will keep accumulating short alphanumeric tokens that look exactly like id_codes: years, version numbers, hex-ish abbreviations, whatever a commenter happens to type. The longer the file grows, the more surface area there is for a genuine collision, and the failure mode is total silence — a comment that never gets surfaced isn't a crash you notice, it's a gap you'd only find by manually re-diffing the live thread against the file, which is the exact manual step this pipeline exists to replace.

The audit() command I wrote two days ago was built to catch "drafted but never posted." It never occurred to me to point the same scrutiny at "pending but never even detected as pending," because that failure doesn't leave a header behind to audit against — it leaves nothing at all. If I hadn't happened to reread pending() right next to a helper that does the same lookup correctly, I don't think I'd have caught it from the outside.

Now imagine buying a new smartphone. Instead of manually installing drivers for the camera, speakers, Bluetooth, and Wi-Fi, everything works automatically the moment you turn it on.

That's exactly what auto-configuration in Spring Boot does.

It detects the libraries available in your project, understands what you're trying to build, and automatically configures most of the required components for you.

Instead of writing hundreds of lines of configuration, you focus on building your application.

In this guide, you'll learn auto-configuration in Spring Boot from scratch using simple explanations, real-world analogies, and complete Java 21 examples.

What is Auto-Configuration in Spring Boot?

Auto-configuration in Spring Boot is a feature that automatically creates and configures Spring beans based on:

• Dependencies present in your project
• Existing configuration
• Properties defined in application.properties or application.yml

Think of it like a smart assistant.

Instead of asking you 100 questions, it looks at what you already have and prepares everything automatically.

For example:

You add:

spring-boot-starter-web

Spring Boot automatically configures:

• Embedded Tomcat
• DispatcherServlet
• Jackson JSON Converter
• REST support
• Error handling
• HTTP message converters

No extra configuration required.

Real-Life Analogy

Imagine ordering a pizza.

Without Spring Boot:

You would tell the chef:

• Add cheese
• Add sauce
• Bake for 15 minutes
• Slice into 8 pieces
• Put into a box

With auto-configuration in Spring Boot:

You simply say:

"I want a Margherita Pizza."

The chef already knows everything else.

Spring Boot behaves exactly the same.

How Auto-Configuration Works

Internally, Spring Boot uses:

• @SpringBootApplication
• @EnableAutoConfiguration
• Conditional annotations
• Auto-configuration classes

When your application starts:

Application Starts
        │
        ▼
Scans Dependencies
        │
        ▼
Finds Auto Configuration Classes
        │
        ▼
Checks Conditions
        │
        ▼
Creates Required Beans
        │
        ▼
Application Ready

The Role of @SpringBootApplication

Most applications start with:

@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

This annotation combines:

@Configuration

@ComponentScan

@EnableAutoConfiguration

The last one enables auto-configuration in Spring Boot.

What is @EnableAutoConfiguration?

This annotation tells Spring Boot:

"Inspect my project and automatically configure everything you can."

Spring Boot checks:

• Available libraries
• Existing beans
• Configuration properties

Then creates missing beans automatically.

Conditional Auto-Configuration

Spring Boot doesn't blindly create everything.

It creates beans only when required.

Examples:

@ConditionalOnClass

Only configure if a class exists.

@ConditionalOnMissingBean

Only create if user hasn't already created one.

@ConditionalOnProperty

Only configure when a property exists.

This prevents conflicts.

Example 1: REST API Using Auto-Configuration (Java 21)

Project dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Project Structure

src
 └── main
      ├── java
      │      └── com.example.demo
      │              ├── DemoApplication.java
      │              └── controller
      │                     └── HelloController.java
      └── resources
             └── application.properties

DemoApplication.java

package com.example.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

/**
 * Main entry point of the application.
 * @SpringBootApplication enables:
 * 1. Component scanning
 * 2. Configuration
 * 3. Auto-configuration
 */
@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }
}

HelloController.java

package com.example.demo.controller;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Map;

/**
 * Simple REST controller.
 * Notice that no servlet configuration is required.
 * Spring Boot automatically configures everything.
 */
@RestController
public class HelloController {

    @GetMapping("/hello")
    public Map<String, String> hello() {

        return Map.of(
                "message", "Hello from Spring Boot Auto-Configuration!",
                "status", "success"
        );
    }
}

application.properties

spring.application.name=auto-config-demo
server.port=8080

Run the Application

./mvnw spring-boot:run

or

mvn spring-boot:run

Test the Endpoint

cURL

curl http://localhost:8080/hello

Response

{
  "message": "Hello from Spring Boot Auto-Configuration!",
  "status": "success"
}

Notice that:

• No Tomcat configuration
• No DispatcherServlet configuration
• No JSON converter configuration

Everything was automatically configured.

Example 2: Auto-Configured Database Connection (Java 21)

Spring Boot can automatically configure a database when the required dependency is available.

Maven Dependency

<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <scope>runtime</scope>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

application.properties

spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=

spring.jpa.hibernate.ddl-auto=create-drop

spring.h2.console.enabled=true

Spring Boot automatically creates:

• DataSource
• EntityManager
• TransactionManager
• Hibernate SessionFactory

You don't configure them manually.

Health Controller

package com.example.demo.controller;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.sql.DataSource;
import java.sql.Connection;
import java.util.Map;

/**
 * Demonstrates that Spring Boot auto-configures
 * the DataSource bean automatically.
 */
@RestController
public class DatabaseController {

    private final DataSource dataSource;

    public DatabaseController(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    @GetMapping("/database/status")
    public Map<String, String> databaseStatus() throws Exception {

        try (Connection connection = dataSource.getConnection()) {

            return Map.of(
                    "database", connection.getMetaData().getDatabaseProductName(),
                    "status", "Connected"
            );
        }
    }
}

Test the Endpoint

cURL

curl http://localhost:8080/database/status

Response

{
  "database": "H2",
  "status": "Connected"
}

Again, notice that we never created a DataSource bean ourselves.

Spring Boot handled everything.

Common Auto-Configuration Examples

Some of the most useful things Spring Boot configures automatically include:

How to Disable Auto-Configuration

Sometimes you want manual control.

You can exclude specific auto-configurations:

@SpringBootApplication(
    exclude = {
        org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration.class
    }
)
public class DemoApplication {
}

Use this only when you plan to configure the component yourself.

Benefits of Auto-Configuration in Spring Boot

The biggest advantages of auto-configuration in Spring Boot include:

• Less boilerplate code
• Faster application development
• Production-ready defaults
• Reduced configuration errors
• Easier maintenance
• Better developer productivity
• Consistent project setup across teams

Best Practices

1. Use Spring Boot Starters

Always use official starter dependencies whenever possible.

✅ Good

spring-boot-starter-web

Instead of manually adding multiple dependencies.

2. Don't Override Auto-Configuration Unnecessarily

Only create your own beans when customization is actually required.

3. Keep Configuration in application.properties

Avoid hardcoding values.

Use:

server.port=8081

instead of modifying Java code.

4. Understand What Spring Boot Configures

Use the auto-configuration report for debugging:

debug=true

This prints which auto-configurations were applied and which were skipped.

5. Avoid Excluding Auto-Configuration Without a Reason

Removing auto-configuration unnecessarily often leads to additional manual configuration and maintenance overhead.

Common Mistakes

❌ Adding unnecessary configuration classes

❌ Creating duplicate beans

❌ Excluding auto-configuration without understanding the consequences

❌ Mixing XML configuration with Spring Boot defaults

❌ Ignoring application properties and hardcoding configuration

Conclusion

Auto-configuration in Spring Boot is one of the framework's most powerful productivity features. It examines your project's dependencies and configuration, then automatically creates the beans and infrastructure your application needs.

By reducing repetitive setup, auto-configuration in Spring Boot lets you spend more time writing business logic and less time managing framework configuration. While it's helpful to understand what's happening behind the scenes, most everyday applications can rely on these sensible defaults with minimal customization.

As you continue to learn Java and build more Spring Boot applications, understanding how auto-configuration works will make it easier to customize behavior when necessary and troubleshoot configuration issues with confidence.

Helpful Resources

• Oracle Java Documentation: https://docs.oracle.com/en/java/
• Spring Boot Reference Documentation: https://docs.spring.io/spring-boot/docs/current/reference/html/
• Spring Framework Documentation: https://docs.spring.io/spring-framework/reference/

Frequently Asked Questions (FAQ)

Is auto-configuration mandatory?

No. You can disable or customize it whenever needed.

Can I override an auto-configured bean?

Yes. In many cases, defining your own bean causes Spring Boot to back off from creating the default one (often through conditional configuration such as @ConditionalOnMissingBean).

Does auto-configuration affect performance?

Very little. The startup process includes checking conditions and creating only the beans that are needed. The productivity benefits usually far outweigh the small startup overhead.

Is auto-configuration suitable for production?

Absolutely. Most production-grade Spring Boot applications rely heavily on auto-configuration in Spring Boot while selectively overriding defaults when business requirements demand it.

Call to Action

Have questions about auto-configuration in Spring Boot, Spring Boot internals, or Java programming? Share them in the comments below! If this guide helped you learn Java more effectively, consider sharing it with your teammates and fellow developers.

`

Mine did too at first. I had a support agent that could answer a refund question, call a policy tool, and avoid a prompt-injection path. From the outside, it looked fine.

Then I asked a less comfortable question: if this agent broke in production, would I actually know what happened?

That question became TraceGate.

TraceGate is a release gate for AI agents built with OpenTelemetry and SigNoz. It runs agent scenarios, sends telemetry to SigNoz, then checks whether the run produced enough evidence to safely ship.

The problem

AI agents can pass a demo while still being hard to debug.

A tool can retry three times and still return a successful answer. An LLM call can happen without cost metadata. A prompt-injection test can pass, but leave no trace of which safety path was used. Those are not product failures in the usual sense. They are observability failures.

That is what I wanted TraceGate to catch.

The idea is simple: before an agent ships, it should satisfy an observability contract. If the contract fails, the release is blocked.

What TraceGate checks

TraceGate uses a YAML contract to describe what a safe release run should include.

Here is part of the contract I used:

name: TraceGate AI Agent Release Contract
serviceName: tracegate-demo-agent

budgets:
  maxRunCostUsd: 0.005
  maxP95LatencyMs: 2000
  maxToolRetries: 1

checks:
  - id: span-agent-run
    type: required-span
    spanName: agent.run
    severity: critical

  - id: span-llm-call
    type: required-span
    spanName: llm.call
    severity: critical

  - id: attr-llm-model
    type: required-attribute
    spanName: llm.call
    attribute: gen_ai.request.model
    severity: critical

  - id: retry-budget-trace-lookup
    type: max-tool-retries
    toolName: trace.lookup
    maxRetries: 1
    severity: critical

The default demo intentionally fails one check. The trace.lookup tool retries three times, but the contract only allows one retry.

That makes the demo useful because TraceGate is not pretending everything is healthy. It blocks the release and explains why.

TraceGate FAIL
Checks: 7/8 passed
Critical failures: 1
FAIL retry-budget-trace-lookup - Worst retry count for 'trace.lookup' was 3; limit 1.

The architecture

The stack is:

Vite
React
TypeScript
Node.js
OpenTelemetry
SigNoz
OpenAI
YAML contracts

The flow looks like this:

Scenario -> Agent runner -> OpenTelemetry -> SigNoz -> TraceGate contract evaluator -> Pass or block

The Node runner executes the scenario. OpenTelemetry records spans, metrics, and logs. SigNoz receives the telemetry through OTLP. TraceGate reads the run result and evaluates it against the contract.

For the OpenTelemetry setup, I used the OTLP HTTP exporters:

const endpoint =
  process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318";

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: `${endpoint}/v1/traces`
  }),
  logRecordProcessor: new BatchLogRecordProcessor(
    new OTLPLogExporter({
      url: `${endpoint}/v1/logs`
    })
  ),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: `${endpoint}/v1/metrics`
    }),
    exportIntervalMillis: 1000
  })
});

In my local setup, SigNoz received spans for:

agent.run
llm.call
tool.ticket.lookup
tool.policy.search
tool.trace.lookup

A build issue I hit

The part that took the most debugging was the local SigNoz setup.

I used Foundry to run SigNoz locally. The first time I started everything, the UI was up, but the telemetry path was not useful. The collector looked alive, but the pipeline was effectively not sending the data I expected.

The issue was in the generated local configuration around the OpAMP and MCP service wiring. I patched the generated setup so the ingester pointed to the actual SigNoz service and the MCP server pointed to the SigNoz UI service. After that, the OTLP export worked and I could verify spans in ClickHouse.

That was the most “real” part of the project for me. The code for the gate was straightforward compared to proving that telemetry actually reached the backend.

The workbench

TraceGate has a landing page and a workbench.

The landing page explains the idea: ship agents only when the traces agree.

The workbench shows:

latest release verdict
checks passed
critical failures
cost
latency
gate matrix
failed evidence
SigNoz investigation query

The gate matrix is the part I care about most. It turns vague release confidence into specific checks.

For example:

Root agent run span: pass
LLM calls traced: pass
Model attribute present: pass
Cost attribute present: pass
Cost budget: pass
Trace lookup retry budget: fail
Refund support scenario: pass
Prompt injection scenario: pass

That gives a reviewer a much better starting point than “the demo worked on my machine.”

Judge test mode

I also added a judge-operated test mode because I did not want the hosted app to feel like a static mockup.

A judge can enter:

service name
scenario
tool name
retry budget
observed retries
cost
latency

Then TraceGate generates a new verdict.

For example:

service: checkout-agent
tool: inventory.lookup
allowed retries: 1
observed retries: 3

That blocks the release.

If the observed retries are changed to 0, the release becomes ready.

This makes the product easier to test without asking someone to set up my whole local environment first.

What I learned

The main thing I learned is that observability is more useful when it is tied to a decision.

Before this project, I mostly treated observability as something I would open after a bug. With AI agents, that feels too late. If an agent takes a risky path, skips metadata, hides cost, or retries a flaky tool several times, I want to know during release validation.

The second thing I learned is that “the agent worked” is too broad. I now split it into two questions:

Did the agent produce the expected outcome?
Did the agent produce enough evidence to debug that outcome?

TraceGate focuses on the second question.

What I would build next

The next version should have a visual contract editor. Writing YAML is fine for a hackathon, but a team should be able to create a release gate by choosing required spans, attributes, budgets, and scenarios in the UI.

I would also add GitHub Actions support. The natural place for TraceGate is in CI, where it can block an agent release before merge.

The final improvement would be deeper SigNoz artifacts. Each contract should generate a matching dashboard, alert, and investigation prompt so the release gate and the debugging workflow stay connected.

Final thought

TraceGate started from one uncomfortable question: if my AI agent fails later, will I have the evidence to understand it?

SigNoz already gives teams a strong place to inspect telemetry. TraceGate adds a release workflow on top of that telemetry.

The project is small, but the idea feels useful: agents should not ship only because the final answer looked good. They should ship when their behavior is traceable enough to trust.

Written with the help of AI (Claude). The bug, the fix, the validation setup, and every claim below are mine, and were verified against the real codebase and a real full disk.

The report

Someone lost a Magic: The Gathering decklist.

They were playing on Cockatrice — the open-source MTG client — with their decks on a drive that had quietly filled up while Oracle pushed an update in the background. They added a card, hit Ctrl+S, and Cockatrice said it saved. The debug log agreed:

[2026-05-28 22:31:42.031 I] Saved deck to "G:/cockatrice300/data/decks/edh-b2-gitrog-reanimate.cod" with format 1 - true

- true. Success.

The file was 0 bytes. The deck was gone.

That was issue #6952, filed by Mekkiss. The steps to reproduce are four lines long and completely damning:

• Have a full disk.
• Open a deck on the full disk
• Add one card to it
• Save the deck (ctrl+s)
• Observe that the deck is now a 0 byte file.

Three ways to be wrong at once

The save path lived in DeckLoader::saveToFile(). Stripped down, it looked like this:

QFile file(fileName);
if (!file.open(QIODevice::WriteOnly | QIODevice::Text)) {
    qCWarning(DeckLoaderLog) << "Could not create or open file:" << fileName;
    return std::nullopt;
}

bool success = false;
switch (fmt) { /* ... saveToFile_Native / saveToFile_Plain ... */ }

file.flush();
file.close();

qCInfo(DeckLoaderLog) << "Saved deck to " << fileName << "with format" << fmt << "-" << success;

There are three independent failures stacked on top of each other here, and you need all three to lose data:

1. WriteOnly truncates on open. The instant open() succeeds, the existing deck is 0 bytes. Not after a successful write — at open time. The old deck is already destroyed before a single byte of the new one is written. On a full disk, open() still succeeds: truncating a file doesn't need free space. It frees space.

2. The serializers always returned true. saveToFile_Native() and saveToFile_Plain() write into the QTextStream / QIODevice and return true unconditionally. They never asked whether the bytes landed.

3. The return value of flush() was discarded. This is the last place the truncation could still have been caught, and the result went straight into the void. close() after it can't help either — its failure was also ignored.

So: file truncated, writes silently fail because there's no room, nobody checks, and the log cheerfully prints - true. The user is told their deck is safe at the exact moment it stops existing.

It was worse than the report

While tracing the save path I checked the other places DeckLoader writes deck files. There were two more, and both used the same truncate-then-write-then-hope pattern.

updateLastLoadedTimestamp() rewrites a deck to stamp it with a "last loaded" time — it runs on load, not save. Same QFile(fileName) opened WriteOnly, same always-true serializer. On a full disk, merely opening a deck truncated it to 0 bytes and reported success. You could lose a decklist without ever pressing Ctrl+S.

convertToCockatriceFormat() was the ugly one. It opened the destination .cod file WriteOnly, wrote the deck, and then — if result was true, which it always was — deleted the original file:

file.close();

if (result) {
    if (!QFile::remove(fileName)) { /* warn */ }

A failed write there doesn't leave you with a 0-byte file and a backup. It leaves you with a 0-byte file and no original. And because the function opened the destination before checking whether the source format was even convertible, the early-return branches ran with the file already truncated.

One user-visible bug on a full disk, and two more write paths queued up behind it waiting for the same conditions.

The fix

Qt has exactly the right tool for this and it has been sitting in QtCore since 5.1: QSaveFile. It writes to a temporary file next to the target and only replaces the target — atomically — when you call commit() and every byte has actually made it to disk. If anything fails, the original file is never touched.

The change is mostly deletion:

// Use QSaveFile so that a failed write (e.g. a full disk) leaves the existing deck untouched
// instead of truncating it to a 0-byte file. The target is only replaced once every byte has
// been flushed successfully in commit().
QSaveFile file(fileName);
if (!file.open(QIODevice::WriteOnly | QIODevice::Text)) {
    qCWarning(DeckLoaderLog) << "Could not create or open file:" << fileName;
    return std::nullopt;
}

bool success = false;
switch (fmt) { /* ... */ }

if (!success) {
    file.cancelWriting();
    qCWarning(DeckLoaderLog) << "Failed to serialize deck for file:" << fileName;
    return std::nullopt;
}

if (!file.commit()) {
    qCWarning(DeckLoaderLog) << "Failed to save deck to " << fileName << ":" << file.errorString();
    return std::nullopt;
}

qCInfo(DeckLoaderLog) << "Saved deck to " << fileName << "with format" << fmt;

Note what moved: the success log now happens after commit() returns true, so it can no longer lie. The failure path logs file.errorString() at warning level instead of printing - false at info level and carrying on.

The same treatment went on all three write paths, plus one ordering fix in convertToCockatriceFormat() — decide the format before opening anything, so an already-converted or unsupported deck can never be truncated and then deleted by a function that decided partway through it had nothing to do.

Net: +73 / −56 in one file. The success path behaves identically.

Proving it without a GUI

Here's the part I actually care about, because "I reasoned about it and it looks right" is how you ship a data-loss fix that doesn't work.

I couldn't reproduce this the way the reporter did. Cockatrice is a Qt desktop app, I wasn't going to build the whole GUI to test a file-I/O path, and — more importantly — I didn't want to fill up a real drive to find out.

So I made a disk that was genuinely, physically full:

1. Created a 16 MB VHD with diskpart, formatted and mounted it.
2. Filled it to exactly 0 bytes free.
3. Wrote a focused QtCore-only harness — no widgets, no Cockatrice — that reproduced the two code shapes side by side: the old QFile + WriteOnly + ignored-flush() pattern, and the new QSaveFile + commit() pattern, both pointed at an existing non-empty file on the full volume.

The result, against real Qt 6.8.1 QtCore:

That's a real ENOSPC from a real filesystem, not a mock, not an injected error, not a #ifdef. The old code lost the file and said it hadn't. The new code kept the file and said it couldn't save. Which is the entire point of the ticket.

The harness took maybe twenty minutes to set up and it is the only reason I'd put my name on the patch. If you're fixing a bug whose trigger is an environmental condition — full disk, no network, permission denied, clock skew — build the condition. Don't mock it. Mocks agree with whatever you already believe.

Shipped

PR #6978 — merged into Cockatrice on 2026-06-09.

What I took away from it

- true in a log line is a claim, and claims need to be checked. The log wasn't broken. It was faithfully reporting a variable that had been hardcoded to true three call frames down. Every layer was honest about the value it was handed; nobody ever checked whether the value meant anything.

WriteOnly is destructive before it's useful. Any open(WriteOnly) on a path that already holds user data is a window where the data is gone and the replacement doesn't exist yet. If the write can fail — and it can always fail — that window is a data-loss bug waiting for the right Tuesday. QSaveFile, write-temp-then-rename, O_TMPFILE + linkat: pick your platform's version, but pick one.

When you find one instance of a pattern, grep for the pattern. The reported bug was one function. The pattern was three, and the scariest one fired on load. The user who filed #6952 would have eventually lost a deck just by opening it, and would never have connected that to a save bug.

Submitted for DEV's Summer Bug Smash — Smash Stories.

Status: Bounded project conclusion. This note separates observed behaviour, working hypothesis, and practical consequence. It is based on current project documents and interactions; it is not external validation or a universal claim about AI systems.

What the evidence supports

1. The project already contains a stable relational model

The working model is not generic “AI assistance.” It separates reasoning surfaces, uses bounded comparisons, permits two-way cognitive pressure, and keeps final acceptance authority with the human.

Reciprocal cognitive contribution, asymmetrical governing authority.

2. Concrete project work preserves the structure better than public abstraction

At the concrete level, instructions such as:

Review this proposal against that architecture.

preserve the distinction between the object being reviewed, the surface applying pressure, the evidence, and the authority that may accept a change.

When the same structure was compressed into general prose, generated explanations repeatedly returned to a simpler one-way model of either human control or transferred AI authority.

That is an observed pattern in this development process.

3. Public explanation is a separate reasoning surface

A README, article, summary, or portfolio page is a projection of the model, not the model itself.

It cannot be assumed to reproduce the internal structure faithfully merely because that structure is present in context.

The explanation must be reviewed against the model it represents:

Does this explanation preserve the actual authority, review, evidence, and state-transition structure?

4. Annoyance was useful boundary data

The irritation indicated that the generic rendering was no longer merely an imperfect exploration.

It was colliding with an internal frame that had already stabilized.

Stable project frame → generic explanatory rendering → felt mismatch → explicit boundary becomes visible.

5. Flattened output can still be productive

The useful conclusion is not that either participant failed.

The mismatch exposed where project-specific structure stops being automatically preserved and needs explicit protection during translation.

A reasonable working hypothesis

During abstraction into ordinary public language, broader learned explanatory patterns may outweigh the more specific relational structure in project context.

That would explain why a concrete instruction preserves:

Review this against that.

while a public explanation reconstructs:

The human reviews the machine.

or:

The machine reviews the human.

It is a plausible explanation for the pattern, not evidence of the internal cause.

The output does not expose the model’s weights, training corpus, tuning process, or internal causal path.

Defensible observation: Generated explanations repeatedly returned to a one-way authority frame while summarizing a project context that explicitly contained reciprocal review and asymmetrical authority.

Practical conclusions

• Project context is not automatically safe during abstraction.
• Public wording must be treated as a derived artifact.
• Derived explanations require fidelity review against accepted source materials.
• Corrections should identify which surface is wrong: the explanation, the interpretation, or the underlying model.
• A local correction must not silently become a global rewrite of accepted project state.
• Human corrections should remain marked as perspective or instruction unless independently verified as fact.
• Recovery documents must preserve the model accurately because they may later guide the returning human operator.

Do not let a correction aimed at the renderer rewrite the model being rendered.

When an accepted project model exists, translation should preserve its relational structure by default.

A challenge to the model should be explicit, not smuggled in through rewording.

Reciprocal contribution does not remove authority. It makes authority explicit at the transition where a proposal becomes accepted state.

What cannot be claimed yet

This observation does not establish:

• that all language models behave this way;
• that all public AI discourse uses one-way power frames;
• that model training definitely caused this specific response pattern;
• that the project model is externally validated;
• that the observed pattern generalizes beyond this development context;
• or that the AI independently created the framing conflict.

The current material remains internally traceable and suitable for further reasoning and validation.

The snapback does not establish a theory of AI in general.

It establishes that explanation is not a neutral export step in this project.

Strongest conclusion so far

A stable, reciprocal, human-governed project model repeatedly became flattened during public-language abstraction. The resulting friction revealed that explanation itself is a separate reasoning surface that must be governed and reviewed against the model it represents.

Series and wider context

Part 1: The Frame Keeps Snapping Back to One-Way Control

From Vague Understanding to Working Truth: Governed Externalized Sensemaking

David van Kleef — Myriuna Worlds

A colleague of mine was cleaning up some old records in a staging environment. Same query he'd run a dozen times before, except this time he was connected to production. He hit F5. No WHERE clause.

47,000 rows gone in milliseconds.

We recovered from a backup, lost about two hours, and nobody got fired. But it stuck with me:** SSMS will let you delete an entire production table with the same amount of friction as running a SELECT 1**. No pause, no confirmation, nothing. The tool that DBAs and backend developers spend all day in has zero built-in protection against the single most common way people destroy data.

So I built one.

What SQL Guard does

SQL Guard is a free SSMS extension (18 through 22) that inspects your query the moment you press F5, and pauses execution if it matches a known destructive pattern:

• DELETE without WHERE
• UPDATE without WHERE
• TRUNCATE TABLE
• DROP TABLE / DROP DATABASE / DROP PROCEDURE
• ALTER DATABASE
• Dangerous EXEC calls
• MERGE without a filter

When one of these fires, you get a dialog showing exactly what object would be affected, with three options: cancel, run anyway, or run and ignore for the rest of the session. The point isn't to block you — it's to make the action conscious. Most accidental damage happens because muscle memory took over, not because someone genuinely meant to wipe a table.

How the detection works

Nothing exotic here, and honestly that's by design. SQL Guard runs a lightweight pattern-matching pass over the query text before it hits the connection. No parsing tree, no round-trip to the server, no measurable latency — it runs in under a millisecond, so you never notice it on normal queries.

The core idea is simple: certain statements are only safe when scoped by a WHERE clause, and the extension checks for that clause's absence rather than trying to understand the full semantics of the query. That keeps false positives low and means it works the same way whether you're on SQL Server 2016 or the latest version.

A conscious design choice: whitelisting is per-table, per-rule — never a blanket bypass. If you have a staging table that legitimately gets truncated on every import, you whitelist that specific table for that specific rule. You don't get an "ignore everything" switch, because that switch is exactly what erases the safety net the first time you forget you flipped it.

Why it stays local

SQL Guard never sends your query text, connection strings, or database names anywhere. Everything happens inside the SSMS process. The only optional telemetry is anonymous crash reporting (via Sentry) for stack traces — never query content. For anyone working with production data, that's not a nice-to-have, it's the only acceptable design.

Installing it
Windows, SSMS 18–22
Zero configuration — the installer registers with every SSMS version it finds
Free, no license key, no trial period

If you want to try it: https://sqlguard.app/

What I'd love feedback on

The rule set covers the "classics" — the patterns that show up over and over in production-incident postmortems. If you've got a destructive pattern that's bitten you and isn't in that list, I want to hear about it. I read every piece of feedback myself.

Backups and a recovery plan are still non-negotiable — SQL Guard reduces the chance of a destructive query, it doesn't replace either. But reducing the chance is worth a lot when the alternative is a Saturday spent restoring from a transaction log.

Today, I'm excited to share the first version of ToolEka 🎉

👉 https://tooleka.com

The idea wasn't to create another generic calculator website. I wanted to build a platform that is fast, simple, mobile-friendly, and actually useful.

🛠️ Tech Stack

• Next.js
• React
• Tailwind CSS
• TypeScript
• Cloudflare Workers
• Cloudflare R2

I chose Cloudflare Workers because I wanted global performance with minimal infrastructure management.

✨ What's Available Right Now?

The first release includes 14+ free tools, including:

• Health calculators
• Finance calculators
• Developer utilities
• Text tools

Every tool is designed to work instantly without requiring an account.

💡 Why I Built It

While searching for online tools, I noticed that many websites are:

• Slow to load
• Filled with intrusive ads
• Not optimized for mobile devices
• Cluttered with unnecessary features

I wanted to build something that focused on speed, simplicity, and usability.

🚧 Challenges Along the Way

Building the project in one week wasn't easy.

Some of the biggest challenges included:

• Deploying Next.js on Cloudflare Workers
• Optimizing performance and Core Web Vitals
• Adding SEO metadata and structured data
• Making every page responsive
• Building reusable components for future tools

Each challenge helped me learn something new.

🚀 What's Next?

This is only the beginning.

My roadmap includes:

• 100+ free online tools
• AI-powered utilities
• More developer tools
• Better accessibility
• Continuous UI and performance improvements

🙌 I'd Love Your Feedback

If you're a developer, designer, or someone who frequently uses online tools, I'd really appreciate your feedback.

Some questions I'd love to hear your thoughts on:

• Is the UI intuitive?
• Which tools should I add next?
• Did you notice any bugs or performance issues?
• What would make you come back and use it again?

You can check it out here:

👉 https://tooleka.com

I'm building this in public, so every suggestion is valuable.

Thanks for reading! ❤️

The Algorithm in Plain English

1. Take your input bytes and group them in sets of 3 (24 bits total).
2. Split those 24 bits into four 6-bit chunks.
3. Each 6-bit value (0-63) maps to one character in the Base64 alphabet: A-Z, a-z, 0-9, +, /.
4. If the input isn't a multiple of 3 bytes, add = padding to round it out.

For Hello (5 bytes), the encoder processes bytes 0-2 as one group (producing 4 characters), then bytes 3-4 as a 2-byte group (producing 3 characters + 1 =). Result: 8 characters — SGVsbG8=.

Where You Encounter Base64 Every Day

• Data URIs — embed images directly in HTML/CSS as data:image/png;base64,iVBOR..., saving HTTP requests for small icons.
• JWT tokens — the payload section of every JSON Web Token is Base64-encoded JSON.
• Email attachments — MIME uses Base64 to send binary files through text-only SMTP.
• API authentication — Basic Auth encodes username:password as a Base64 string in the Authorization header.
• Inline SVGs — embed vector graphics in a single HTML file without external dependencies.

Common Gotchas

Standard vs URL-safe. Standard Base64 uses + and /, which break in URLs. URL-safe Base64 replaces them with - and _ and strips the = padding. If you are encoding something for a query string, always use the URL-safe variant.

Base64 expands data by ~33%. Encoding 1 MB of binary produces ~1.33 MB of text. For large payloads, gzip the original data before Base64-encoding to offset the bloat.

It is not encryption. Anyone can decode Base64 with zero effort. Never use it as a security measure — it provides exactly zero confidentiality.

Try it yourself: Base64 Encoder/Decoder — free, no signup, works entirely in your browser with no server uploads.

mcp-profile is a small, dependency-free CLI (+ a Claude Code skill) that lets
you define named profiles — e.g. "research", "coding", "deploy" — each listing
the MCP servers that task actually needs, then switch between them with one
command:

mcp-profile use research

It edits the same enabledMcpjsonServers / disabledMcpjsonServers fields
Claude Code already reads — no new config format, no daemon, no account, no
network calls. MIT licensed.

Repo: https://github.com/pozga/mcp-profile

Feedback wanted: does this match how you're actually hitting the tool-count
problem? What would make this worth paying for (team-shared profiles? a
dashboard?) vs. just a free utility?

1. No auth. No sign-up. Ever.

Every API-key ceremony is a step where trial users bounce. The dataset is public. https://receiptedit.com/api/sales-tax/california — that is the entire contract.

2. Match the data shape to how it will actually be consumed

90% of consumers ask "What's the sales tax for X state?" So the top-level endpoint is /api/sales-tax/:state — one state per response. When someone needs all 50 for analysis, the CSV/JSON files sit next to the API at github.com/receiptedit/us-sales-tax-2026.

3. CORS on. Cache long.

Access-Control-Allow-Origin: *
Cache-Control: public, max-age=86400, s-maxage=604800

4. Ship an embed alongside the API

<iframe src="https://receiptedit.com/api/embed/sales-tax/california" width="380" height="540" style="border:0" loading="lazy"></iframe>

Bloggers on Medium/WordPress cannot install an npm package. They can paste an iframe.

5. License permissively

Real MIT. Attribution appreciated, never required.

Try it

• API: https://receiptedit.com/api/sales-tax
• Source: https://github.com/receiptedit/us-sales-tax-2026
• Embed: https://receiptedit.com/api/embed/sales-tax/california
• Docs: https://receiptedit.com/developers
• Product: https://receiptedit.com

Happy to talk API-design tradeoffs in comments.

A technical deep-dive into Soroban escrow contracts, FX oracles, and mainnet deployment — from testnet prototype to production.

se

The Problem

Cross-border payments still take 3-5 days and cost 6.5% on average. Correspondent banking chains are slow, opaque, and expensive. The $800B remittance market has no atomic settlement layer.

Stellar was purpose-built for this. 5-second finality. Built-in DEX. Path payments at the protocol level. And now, with Soroban smart contracts, programmable settlement.

AnchorFX is an open-source protocol that combines these primitives into trustless, atomic FX settlement between regulated financial anchors. Two Soroban contracts — an Escrow Factory and an FX Rate Oracle — communicate via cross-contract calls to lock, rate, and settle funds in a single atomic flow.

Architecture

Sender → [Escrow Contract] → Receiver
              │
         [Oracle Contract]
              │
         FX Rate Data

Contract 1: Escrow Factory (995 lines, 23 tests)

The escrow contract is a multi-escrow factory with per-escrow storage. Each escrow goes through a defined lifecycle:

• Created — Sender locks tokens with a timeout and settlement conditions
• CounterpartyApproved — Receiver signs off on the terms
• Settled — Admin releases funds at the locked FX rate
• Refunded — Sender reclaims after timeout expires
• Cancelled — Admin cancels (circuit breaker)

pub fn create_escrow(
    env: Env,
    sender: Address,
    receiver: Address,
    token: Address,
    amount: i128,
    timeout_blocks: u32,
    corridor: u32,
) -> u64 {
    sender.require_auth();
    // Read oracle rate at creation time — locks the rate
    let oracle_addr = env.storage().instance().get(&ORACLE_KEY).unwrap();
    let rate: u64 = env.invoke_contract(&oracle_addr, &symbol_short!("get_rate"), ...);
    // Store escrow with locked rate
    // ...
}

Key security decisions:

• Per-escrow storage — O(1) reads, independent TTL per escrow
• Checks-effects-interactions — state saved before token transfer in all 4 mutation functions
• require_auth() on init — prevents front-running
• Pause/unpause — emergency circuit breaker
• Typed errors — no raw panic! anywhere

Contract 2: FX Rate Oracle (195 lines, 4 tests)

The oracle stores per-token FX rates with 24-hour expiry. Rates are set by admin and validated on every read:

pub fn get_rate(env: Env, token: Address) -> u64 {
    let rate_data: RateData = env.storage().instance()
        .get(&token)
        .unwrap_or_else(|| panic_with_error!(&env, Error::RateNotFound));

    // Check expiry (~24h in ledgers)
    if env.ledger().sequence() > rate_data.expires_at {
        panic_with_error!(&env, Error::RateExpired);
    }
    rate_data.rate
}

Cross-Contract Communication

The escrow reads the oracle rate at creation time and locks it. This means:

1. Rate is locked at escrow creation — no slippage between creation and settlement
2. Oracle can update rates independently — existing escrows use their locked rate
3. Both contracts have independent admin — oracle admin ≠ escrow admin if desired

The Frontend

Next.js 16 + React 19 + Tailwind CSS v4. Five routes:

10 API routes handle REST + SSE + SEP-31 stubs. All rate-limited (30 burst, 2 req/s). Input validation on every endpoint.

Audit & Security

Full security audit: 257 findings. All critical, high, and medium issues fixed before mainnet.

Key mitigations:

• Per-escrow storage (eliminates single-TTL data wipe risk)
• Checks-effects-interactions ordering
• Input validation with typed errors
• Production security headers: HSTS, CSP, COOP, CORP
• CI fails on lint/format/tsc warnings

Mainnet Deployment

Deployed on Stellar mainnet with 6 verified transactions:

Lessons from mainnet deploy:

• WASM upload cost scales with size — 4KB oracle: ~1 XLM. 13KB escrow: ~18 XLM. Optimize aggressively.
• Mainnet RPC can be unreliable — add retries with exponential backoff, use multiple RPC endpoints
• Fee estimation matters — always simulateTransaction first, use the suggested fee
• Sequence numbers — horizon and RPC can desync. Always reload account before building

Testing

Contract Tests:    27 (23 escrow + 4 oracle)
Frontend Tests:    26 (vitest)
Total:             53 passing, zero warnings

Test coverage includes:

• Full escrow lifecycle (create → approve → settle → refund → cancel)
• Multi-escrow isolation (two escrows don't interfere)
• FX rate math (denominator precision, overflow protection)
• Fuzz testing (50 random escrow operations)
• Rate limiting, schema validation, error states

What I Learned

1. Soroban's storage model is different — per-key TTL means you can't store everything in one map. Per-escrow storage with independent lifetimes is the right pattern.

2. Cross-contract calls are powerful but have overhead — plan your gas budget. Reading oracle rates at creation time and caching them is cheaper than live reads at settlement.

3. Mainnet is not testnet — fees are real, RPC is slower, and WASM upload costs will surprise you. Simulate everything. Budget 3x what testnet costs.

4. Security-first from day one — the audit found 257 issues in code that "felt solid." Get audited. Fix everything.

Open Source

AnchorFX is MIT licensed. Contracts, frontend, deploy scripts, and tests are all public.

GitHub: github.com/subheeksh5599/AnchorFX
Live: anchorfx.vercel.app
Twitter: @Tenki_ai

Built for the Stellar Journey to Mastery — Black Belt track. Stellar, Soroban SDK v22, Rust WASM, Next.js 16.

The question that started this

We run Claude Code against a fairly large, fairly automated repository — a farming-assistance platform with a Node.js backend, a Flutter app, a React dashboard, an in-progress Spring Boot microservices migration, and a home-grown "repo memory" layer called gps that captures invariants, lessons, and preferences across sessions. Over several months we'd wired up a lot of automation: session-start hooks, prompt-submit hooks, auto-captured preferences, persona plugins, a mandatory agent-dispatch table. It felt sophisticated. It also felt, some days, slow to get going — every session seemed to start with a wall of text before any real work happened.

So when Anthropic published "The New Rules of Context Engineering for Claude 5 Generation Models", we asked the obvious question: are we actually following our own advice, or have we just accumulated automation that looks like good practice?

This post is the audit, the root cause we found, and the fix — including a mistake we made mid-fix that's worth telling on ourselves for.

What the blog post actually says

Stripped of marketing language, the post boils down to five concrete rules:

1. Keep CLAUDE.md lightweight. Describe gotchas and non-obvious patterns, not everything you know about the repo. Organize by relevance, not comprehensiveness.
2. Progressive disclosure. Load context at the right time — skills, references, and detail should be pulled in when needed, not front-loaded into every session regardless of task.
3. Trust the model's judgment. Remove redundant guardrails and standing instructions that the newer models don't need spelled out every time.
4. Rely on automatic memory, not manual dumps. Don't hand-maintain a giant preferences block in a markdown file — let the memory system surface the right thing at the right time.
5. Design tools and interfaces, not prose. Push instructions into tool schemas and parameter design rather than repeating them in the system prompt.

None of this is radical. It's the same discipline good engineers apply to code: don't load everything into scope just because you might need it, don't repeat yourself, don't keep dead configuration around because removing it feels risky. The interesting part is applying it literally, to an actual production Claude Code setup, and seeing what falls out.

What we found when we actually measured it

We started by tracing exactly what gets injected into the context window at SessionStart, since that's the one moment every single session pays the cost.

The .claude/settings.json SessionStart hook was running a chain of gps commands. Three of them, in sequence, all touched the exact same data:

gps preferences --write      # bakes a markdown block into CLAUDE.md
gps preferences --markdown   # prints the same list to stdout
gps prime                    # emits a session primer that overlaps both

--write persisted a <!-- gps:auto-prefs -->...<!-- /gps:auto-prefs --> block directly into CLAUDE.md, which is read every session. --markdown printed the identical list again as hook stdout, which Claude Code surfaces as a second system reminder. prime added a third pass that substantially overlapped both. Same ~60-line list, injected three times, every single session, before any actual task began.

That's rule #1 and #4 violated in one shot: a hand-persisted dump in CLAUDE.md (the opposite of "lightweight, gotchas only") that was also duplicated by the very memory system that was supposed to replace it.

Then we looked at what was actually in that preferences list, because 60 lines of genuine cross-cutting invariants would at least be defensible even if wastefully injected three times. It wasn't. Pulling the raw store (.gps/preferences.yml) showed 105 entries, and a good two-thirds of them weren't standing preferences at all — they were things like:

• A verbatim quote from an eval judge's commentary: "Judge weakness: 'never specifies idempotency for event consumers.'"
• A code comment lifted out of context: // replace, never append
• A fragment of a test assertion: test('refresh() replaces list, never appends', () async { ...
• Duplicate phrasings of the same secrets rule, recorded three separate times with three separate IDs.

The mechanism responsible was gps capture-preference --emit and gps capture-directive --emit, both wired into the UserPromptSubmit hook — meaning they fired on every single message in every session. Their logic was a bare regex cue: anything matching \b(?:please\s+)?never\b or \balways\b, recorded as a standing, always-injected "preference," with source: auto and no expiry. It didn't matter whether the match came from the user's actual instruction, a pasted log, a quoted test file, or the agent's own prior output describing what an eval judge said. If the substring was there, it got written to disk and re-injected every session forever.

This is the sharpest version of the trap the blog post warns about: "automatic memory" sounds like the responsible, judgment-trusting choice — right up until the automation itself has no judgment. A regex doesn't know the difference between a user's hard rule and a stray quote in a transcript. Left alone, it just accumulates noise at the same rate it accumulates signal, and every session pays for both.

On top of that, we found two plugins — caveman (terse response style) and ponytail (anti-overengineering behavioral rules) — set to always-on in the global enabledPlugins config. Both inject their full rule text (roughly 40 lines each) at every SessionStart, regardless of whether the task at hand has anything to do with response verbosity or engineering discipline. Useful personas, wrong default: standing injection instead of on-demand invocation is exactly the "load everything just in case" pattern rule #2 argues against.

The fix

We worked through it in four concrete steps.

1. Collapse the triplicated emission. Dropped --write and --markdown from the SessionStart hook, keeping a single gps prime pass. One source of truth, one injection per session.

2. Strip the manual dump from CLAUDE.md. Removed the <!-- gps:auto-prefs --> block entirely. CLAUDE.md went back to being what rule #1 asks for: hand-curated gotchas, not an auto-regenerated preference log.

3. Prune the store. We pulled the full 105-entry list and manually classified every one. About 35 were genuine, cross-cutting invariants worth injecting into every session regardless of task — secrets handling, PII rules, git-push restrictions, migration reversibility, PCI scope boundaries. The other 70 were dropped: judge commentary, code fragments, near-duplicate phrasings, symbol-specific one-off fixes that only matter when that exact piece of code is touched again (those still exist — just surfaced on demand via gps prepare_edit when relevant, not blasted into every session).

4. Fix the actual leak, not just the symptom. This is the part worth dwelling on, because the first three steps only clean up what had already accumulated — they don't stop it from happening again. The regex-based capture-preference/capture-directive hooks were still wired in and would refill the store with the same class of noise within days. We replaced them with a small filtered wrapper script that:

• reads only the literal current user prompt ($CLAUDE_USER_PROMPT), never the transcript, never the agent's own output, never quoted or pasted content;
• requires an actual imperative sentence shape — the text has to start with never, always, don't, must, or should, not just contain the word anywhere;
• rejects lines that look like code, quotes, or table rows (//, #, |, ,>`);
• checks the existing store for a near-duplicate before writing anything new.

We disabled caveman and ponytail as always-on plugins and left them invokable via their slash commands, on demand, exactly when their behavior is actually wanted.

The mistake worth admitting

Midway through pruning that 105-entry list by hand, we cut a handful of rules that were real, not noise — a "reference the config key, never the value" secrets-handling convention, a backend mock-fallback contract invariant, an API schema-evolution rule ("add fields, never remove"), a known config-server caching gotcha. They looked narrow enough to fold into "one-off" at a glance, but they were genuinely cross-repo conventions, not symbol-specific trivia.

The user caught it with a simple question: "is it losing existing functionality?" That's the right question to ask after any aggressive prune, automated or manual — and it's the reason we're writing this section at all. Nothing was destroyed; the full pre-prune store was still sitting in git history, recoverable with one git show. But it's a useful reminder that "delete the noise" and "delete everything that isn't obviously load-bearing" are not the same operation, and doing the second by accident while intending the first is an easy mistake to make when you're skimming 105 entries under time pressure.

Verifying the automated version wasn't just automating the same mistake

Before trusting the new filtered capture script, we tested it against an isolated scratch copy of the .gps store — not the live repository — feeding it a mix of a genuine directive and the exact kind of noise that had polluted the original store:

plaintext
Never log raw request bodies for the payment endpoint.
Always validate the webhook signature before processing.
The judge said: never sent to the network, that's just eval commentary.


The result: both real directives were captured and written to the store correctly. The judge-commentary sentence was rejected, because it doesn't open with an imperative verb — it opens with "The judge said." That's a small test, but it's exactly the discriminator the original regex lacked, and it's cheap enough to run before wiring anything back into a live hook.

What this actually bought us

Token-wise, the honest estimate is roughly ~1,200+ tokens injected at every SessionStart before the fix, down to ~400 after — a triplicated 60-line list plus a noisy 105-entry store plus two always-on plugins, versus one clean primer pass plus a curated 35-entry store plus plugins that only cost anything when explicitly invoked. That's not a huge absolute number in isolation, but it's paid on every single session, compounds across a team, and — more importantly than the raw token count — it was diluting the signal-to-noise ratio of the one list that's supposed to carry the invariants Claude should never violate. A 35-item list of real rules is something a model can actually hold onto. A 105-item list where two-thirds is eval-transcript noise trains exactly the wrong instinct: skim past it, because it's mostly not going to matter.

The generalizable lesson

None of the individual fixes here are exotic — collapse duplicate emissions, prune a noisy store, tighten a regex, gate a plugin behind explicit invocation. The useful part is the audit discipline itself: pick a set of concrete, checkable rules (Anthropic gave us five good ones), and then actually trace what your own automation does against them, hook by hook, rather than assuming that "we have automatic memory" or "we use skills" means the rules are satisfied by construction. Automation that was designed well can still drift into violating the very principles it was built to serve — usually quietly, one auto-captured preference at a time, until someone asks what's actually in the store.

If you're running Claude Code with any nontrivial hook setup — gps, a custom memory layer, auto-capture of any kind — it's worth doing the same trace. Open the SessionStart hook chain, count how many times the same information gets emitted, and go look at what your "automatic" capture mechanism has actually been recording for the last few months. You may be surprised what's in there.

Every other "online tool" site uploads your PDFs and images to their server for processing. That means:

• Your sensitive documents sit on someone else's machine
• Processing speed depends on their server load
• File size limits, watermarks, or forced signups

I built everything using client-side processing — Canvas API, pdf-lib, Tesseract.js (WebAssembly), and more. Your files literally never leave your device.

What's included

PDF Tools (28): Merge, split, compress, convert to Word/Excel, password protect, watermark, page numbers, metadata editor

Image Tools (30): Compressor, background remover, crop, resize, DPI changer, EXIF viewer/remover, OCR (image to text), meme generator, QR code generator

Developer Tools (42): JSON formatter, JWT decoder, Base64/Base32 encoder, regex tester, cron generator, SQL formatter, CSS/JS/HTML minifier

Design Tools (14): CSS gradient, box shadow, border radius generators, color contrast checker (WCAG), Tailwind component builders

Calculators (16): EMI, SIP, BMI, compound interest, salary tax, GST/VAT, ROI, fuel cost

Plus text tools, unit converters, YouTube tools, utilities, and more.

Tech Stack

• Next.js 15 (static export, deployed on Cloudflare Pages)
• TypeScript (strict mode)
• Tailwind CSS (dark mode)
• pdf-lib, pdfjs-dist (PDF processing)
• Tesseract.js (OCR — WebAssembly, zero dependency on external APIs)
• @imgly/background-removal (on-device AI background removal)