[Day 5] My Cat-LoRA Got Worse With 45x More Photos. So I Figured Out Why and Fixed It.

Intro

Day 5!

Today was originally going to be "have AI analyze a year of my Amazon order history," but downloading the Amazon purchase history just wouldn't work no matter what I tried. So that was a bust.

Pivoted. On Day 4 the AI sorted my iPhone library and pulled out 999 photos it thought were cats. I figured: why not feed those into the Day 2 cat-LoRA pipeline and re-train at scale?

Day 2 used 22 photos. Today: 45x that. Surely the LoRA gets stronger, right?

Spoiler: it got worse. So I dug into why, fixed it, then ran a second experiment to test a different hypothesis. This post is the whole journey: failure → diagnosis → revenge → bonus experiment.

Today's setup

• 
  Data: 999 photos labeled "cat" by the Day 4 CLIP classifier.


• 
  Goal: Strengthen the Day 2 cat-LoRA by re-training with much more data.


• 
  Comparison axis:
  
  
  
  
  • 
    v1 (Day 2 LoRA): 22 photos, hand-written natural-language captions
  
  
  • 
    v2 (today's first LoRA): 999 photos, ohwx cat trigger-only captions
  
  
  
  
  
  

Expected: v2 beats v1.

🔧 v2 training procedure

The flow:

Day 4 _review/cat/  (1,009 symlinks)

   ↓ ① Resize to short-side 512px → 999 images

   ↓ ② Write a tiny .txt next to each image with just "ohwx cat"

   ↓ ③ Train LoRA with Kohya_ss (~14 min)

   ↓ ④ Generate side-by-side comparisons in ComfyUI

Base model and trigger word are the same as Day 2 (Realistic Vision V6.0 B1 + ohwx cat). Only the photo count and the captions changed.

Step counts are kept roughly equal so the only meaningful variables are image count and caption style.

📊 v2 results

Canonical prompt: "ohwx cat sitting on a wooden floor"

No-LoRA / v1 / v2 side-by-side:

• 
  v1: Tuxedo cat in the forest with fantasy lighting. Identity preserved.


• 
  v2: A black-bear-looking creature in an illustration style. Cat identity is gone.

Huh.

Variation prompt 2: Cute chef

"ohwx cat as a cute chef":

• 
  v1: Tuxedo cat in a spacesuit.


• 
  v2: A tabby (orange-striped) cat in a spacesuit. Fur color got swapped out.

v1's image itself is a little weird in places, but the cat is unmistakably mine. v2's image is technically clean — except the cat's coat pattern is wrong.

Variation prompt 4: Sunny balcony

The one creative-ish prompt that did fine:

v3 is as sharp as v1. Tuxedo pattern, white chest, white snout — all there.

Fantasy forest (3-way)

v2's human chef is now a cat chef. The trigger word is pointing at the cat again.

Astronaut (3-way)

Photorealistic. v3 is on par with v1.

Hypothesis A: confirmed.

Just cleaning the data brought back identity preservation across the board. The 45x data problem was essentially "the data wasn't the right granularity for identity learning."

v1 (22 imgs) still has a slight edge in painterly styles, but for identity consistency v3 (213 imgs) holds its own.

🧪 Revenge experiment #2: Also fix the captions (v4)

v3 solved most of the problem, but hypothesis B is still untested. Maybe richer captions push v3 over the top?

Auto-generating captions

Hand-writing 213 captions is rough. So I had Qwen2-VL 7B (the chat-with-images model from Day 4) generate captions instead.

The instruction prompt: "describe what's happening in this cat photo using short comma-separated phrases — pose/action, view angle, setting, background details. Do NOT describe the cat's appearance (color, breed, fur, markings) — focus on the scene only."

The "don't describe the cat's appearance" rule is intentional: ohwx cat should carry the appearance signal, the caption only adds context.

Sample outputs:

IMG_0100: ohwx cat, sitting, side view, indoor setting, wooden floor,

          folding chair, curtain, air conditioner

IMG_0189: ohwx cat, laying on a blanket, side view, indoor setting with

          a window, a chair, and a pillow in the background

IMG_0241: ohwx cat, laying on its side, overhead view, indoor setting

          with a patterned rug, a couch, and a suitcase in the background

Stylistically very close to the Day 2 hand-written captions. 213 captions generated in 6 minutes.

v4 training

Same setup as v3, just pointing at the new caption directory.

The only variable is captions. This isolates the caption-granularity effect.

📊 v4 results

5-way grid:

Canonical (5-way)

Here v4 might be a hair ahead — the kitchen background reads more detailed.

Other prompts

v3 and v4 are essentially indistinguishable.

Honestly? v3 and v4 are nearly tied. Chef is the one prompt where v4 looks slightly better; everything else is noise.

Expected a clear v4 win. Got a shrug.

💡 Why was v4 barely better than v3? (hypotheses)

A few theories. Thought through these with Claude.

Hypothesis X: Data quantity does the job captions used to do

Day 2's 22 photos got shown ~100 times each during training. With only 22 photos the scene variety is limited, so the captions had to explicitly separate "the cat" from "the background" to let the model factor them apart.

v3 and v4 have 213 photos, each shown only ~10 times. But the image distribution itself encodes "the cat is constant, backgrounds vary" — the model can learn that factorization directly from the pixels, without needing captions to spell it out.

In other words, the work that captions were doing at 22 images is done by image variety at 213 images. So bumping caption granularity at this scale adds little.

"Quantity" and "caption granularity" might be substitutable information sources — when one is high enough, the marginal value of the other drops.

Hypothesis Y: VLM captions describe scenes, not identity

Look at a v4 caption:

ohwx cat, sitting, side view, indoor setting, wooden floor, folding chair, curtain, air conditioner

It's rich on "what's happening" but the "this is my cat" part is still just the trigger ohwx cat. VLMs do scene description well — they don't do identity discrimination. So v4 captions added no new identity signal, only extra background labels. And the background separation was already handled by image variety (per hypothesis X), so the added captions had nowhere to add value.

Hypothesis Z: LoRA capacity saturation

network_dim=32 has a limited representation budget. v3 might already be using most of it for "the look of my cat." v4 having richer captions doesn't help if there's no room left to absorb them.

Hypothesis X feels the most explanatory, but I'm still a beginner here. Take with a grain of salt.

What I learned today

Data homogeneity matters more than data count

The biggest takeaway. Day 4's classifier filtered for "contains a cat". A LoRA needs "contains MY cat in this composition". These are different levels of abstraction, and one does not substitute for the other.

More data isn't always better. More data of the right granularity is what matters.

"Quantity" and "label granularity" are partially substitutable

The v3 vs v4 finding. When data is scarce, captions need to explicitly separate identity from context. When data is plentiful, image variety can do that work implicitly.

Day 2's 22 photos + hand-written captions worked because the two together compensated for the small dataset.

CLIP image similarity is a great curation tool

A small set of "ground truth" examples (the 22 Day 2 photos) was enough to score a much larger candidate set and produce a high-purity subset. A few confirmed examples can bootstrap automated curation at scale — broadly useful, I think.

VLM auto-captioning, even when it didn't help much here, is a great prep tool

The Qwen2-VL captions came out stylistically very close to the Day 2 hand-written ones. Even if the final LoRA didn't benefit much in this experiment, "VLM writes the captions for you in 6 minutes" is a powerful workflow for cases where caption granularity would matter.

What I actually did (the AI walks through the technical bits)

Technical details, narrated by Claude.

:::details 1. Data prep (HEIC → JPG, short-side 512 resize)

_review/cat/ from Day 4 holds 1,009 symlinks: 503 HEIC, 505 JPG, 1 other.

python3 shared/utils/resize-shortside.py \

  --src private-data/iphone-photos-classified/_review/cat \

  --dst private-data/cat-lora-v2/images-512 \

  --short-side 512

Output: 999 JPGs, 139 MB total. The drop from 1,009 to 999 came from stem collisions during conversion (9 cases where IMG_XXXX.HEIC and IMG_XXXX.JPG both produced the same IMG_XXXX.jpg) plus 1 resize failure.

:::

:::details 2. CLIP image-similarity ranking

Using openai/clip-vit-base-patch32, encode the 22 reference photos and the 999 candidates, then compute mean cosine similarity:

from transformers import CLIPModel, CLIPProcessor



model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to("cuda")

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")



ref_feats = embed(model, processor, ref_paths)     # (22, D)

cand_feats = embed(model, processor, cand_paths)   # (999, D)



sim = cand_feats @ ref_feats.T          # (999, 22)

score = sim.mean(dim=1)                 # (999,)

Then create symlinks named like 0001_0.871_IMG_2906.jpg — image viewers will sort by name, which now corresponds to "most-likely-my-cat first."

:::

:::details 3. Manual review via in-browser thumbnail page

All 999 thumbnails in one HTML page, served via python3 -m http.server. Each thumbnail has a checkbox; toggling adds the photo to an "exclude" set.

<div class="grid">

  <div class="cell" data-name="IMG_2906.jpg">

    <img src="thumbs-256/IMG_2906.jpg">

    <div class="meta">#1 0.871</div>

    <input type="checkbox" onchange="toggleExclude(this)">

  </div>

  ...

</div>

<script>

function exportExcluded(){

  const names = [...document.querySelectorAll('.cell.excluded')]

    .map(c => c.dataset.name);

  download('excluded.txt', names.join('\n'));

}

</script>

The "Export excluded list" button downloads excluded.txt. 312 manual exclusions total.

:::

:::details 4. v3 / v4 training configs

v3 and v4 share identical training configs — the only delta is image_dir:

# dataset_v3.toml

[[datasets.subsets]]

image_dir = ".../cat-lora-v2/images-512-v3"  # 213 imgs + "ohwx cat" captions

num_repeats = 5



# dataset_v4.toml

[[datasets.subsets]]

image_dir = ".../cat-lora-v2/images-512-v4"  # same 213 imgs + VLM natural captions

num_repeats = 5

# train_v3.toml / train_v4.toml: identical except output_name

output_name = "ohwx_cat_v3"  # or "ohwx_cat_v4"

max_train_epochs = 2

network_dim = 32

network_alpha = 16

unet_lr = 1e-4

text_encoder_lr = 5e-5

Total steps: 213 × 5 × 2 ÷ 2 = 1,065. Matches v1 (1,100) and v2 (999) closely.

:::

:::details 5. VLM caption generation with Qwen2-VL

Reusing the Qwen2-VL 7B Instruct setup from Day 4. The prompt:

Describe what is happening in this cat photo using short comma-separated

phrases. Cover: (1) the cat's pose or action, (2) the view angle,

(3) the setting and notable background details. Keep it under 25 words.

Do NOT describe the cat's appearance (color, breed, fur, markings) — focus

only on the scene. Output the description directly without any preamble.

Example: walking on a metal kitchen counter, side profile, indoor kitchen

with spice bottles and shelves in the background

The "do not describe appearance" clause is the key design choice: ohwx cat should own the identity, captions own the context.

Outputs get ohwx cat, prepended and saved as .txt:

desc = vlm_caption(model, processor, img)

caption = f"ohwx cat, {desc}"

txt_path.write_text(caption + "\n", encoding="utf-8")

213 photos in 6 minutes. Reused Day 4's .venv, no additional setup.

:::

Tomorrow's preview: Day 6

Day 6: torn between a follow-up revenge ("just-my-cat 50 photos + hand-written captions, can it beat v1?") and switching to a completely different domain (voice with Whisper? feeding my old blog posts to an LLM?).

Decision will probably be made over morning coffee.

100ExperimentsWithDGX #LocalLLM #LoRA #StableDiffusion