Stop Reading Documentation. Start Reading GitHub Issues.

Reading the docs isn't enough. The most valuable developer feedback lives inside GitHub issues, bug reports, and feature discussions. In this article, I share the checklist I use to mine repositories and coding frame work for the issue, uncover real developer pain points, and turn engineering conversations into meaningful UX research findings.

Why Coding Is the Difference Between Reading and Researching

Reading GitHub issues without a coding framework produces impressions. Coding them produces findings.

When you code an issue, you are making an explicit analytical decision about what type of signal it contains. You are saying: "This issue is evidence of a feedback gap at Stage 4 (model loading), filed by an ML engineer with a new user experience level, in KServe v0.11, and it directly answers my research question about what information engineers need during model loading that the product currently does not provide."

That sentence assembled from your coding decisions is a finding. Multiply it across 100 issues and you have a research study.

Without coding, you have 100 anecdotes. With coding, you have a dataset.

It makes patterns visible. When you code 20 issues and notice that 14 of them map to the same stage and the same challenge type, you have found your highest-priority UX problem. You would never see that pattern by reading casually.

It makes your findings defensible. "Engineers struggle with KServe" is an opinion. "18 issues across v0.10–v0.13 filed by engineers in their first deployment show identical feedback gap patterns at Stage 4, with an average of 14 comments per issue" is a finding.

It separates your role from the engineer's role. Engineers read GitHub issues as bug reports. You read them as evidence of design decisions. The coding framework is the analytical lens that makes your UX reading possible.

Step 1: Collect Basic GitHub Metadata

Before any analysis, capture the foundational data for every issue. This establishes the quantitative baseline for your research report.

Issue Metadata Checklist

─────────────────────────────────────────────

[ ] Issue URL / Number        e.g. #1234

[ ] Issue creation date

[ ] Issue category            Bug | Question | Feature Request | Docs

[ ] Labels                    e.g. InferenceService, Control Plane, Kubernetes

[ ] Current state             Open | Closed | Stale | Merged

[ ] Resolution type           Code Fix | Docs Update | Workaround | No resolution

[ ] Total comment count

[ ] Unique commenter count

[ ] Ping-pong count           Back-and-forth before root cause found

[ ] Brief issue summary

The "ping-pong count" — the number of back-and-forth diagnostic comments between the user and maintainers before the root cause was identified — is a particularly powerful metric. High ping-pong means the product gave the user no diagnostic signal. That is a UX failure in the product itself.

Step 2: Identify Who Is Reporting — The Demographics Checklist

Understanding user demographics is essential in any research project because it tells you whose problems to prioritise. In interviews or surveys, you simply ask. In GitHub mining, you have to infer from signals embedded in the issue itself.

In the context of KServe, we are primarily trying to distinguish between three main personas.

Technique 1: Read the vocabulary

The language engineers use tells you their world immediately.

Technique 2: Look at the "altitude" of the problem

KServe issues typically fall into three abstraction layers:

• 
  Infrastructure/cluster layer — Kubernetes, Istio, Knative, CRDs → Platform Engineer


• 
  Pipeline/deployment layer — model formats, runtimes, autoscaling, vLLM/Triton config → ML Engineer
  


• 
  Consumption layer — API latency, authentication, payloads, querying from an app → App Developer

Technique 3: Read what they attached as evidence

• 
  Platform engineers dump kubectl describe outputs, cluster events, Helm values files, or Istio configurations — they speak in Kubernetes YAML


• 
  Data scientists share Python stack traces, Jupyter Notebook snippets, or InferenceService configuration blocks without knowing what the pods underneath are doing


• 
  App developers provide curl commands, Postman logs, or application server logs showing connection timeouts

Technique 4: Ask "what are they trying to accomplish?"

• 
  "I just want to see if my new LLM works with this payload" → ML consumer/creator


• 
  "I am trying to automate onboarding of 50 teams onto a shared KServe instance" → Platform builder

Quick triage question: Is this person treating KServe as an infrastructure component or as a model-delivery tool? Infrastructure → Platform/DevOps. Model-delivery → Data Scientist/ML Engineer.

Demographics Checklist

─────────────────────────────────────────────

[ ] Inferred user persona

      Data Scientist / ML Engineer

      Platform / DevOps Engineer

      Application Developer

      ML-expert / K8s-novice  ← the hardest edge case

      Unclear



[ ] Experience level

      New user (first issues, plain English, no version info)

      Experienced (provides full env, logs, rules out causes)

      Unclear



[ ] Deployment environment

      Local (Kind / Minikube)

      Cloud managed (EKS / GKE / AKS)

      On-premises / bare metal



[ ] Deployment scale

      Single-cluster

      Multi-cluster

      Multi-tenant



[ ] Deployment method

      Helm | Kustomize | ArgoCD / GitOps | Direct kubectl

The hardest edge case: ML-experienced / Kubernetes-novice engineers. They write technically confident issues about model formats or serving runtimes — but are completely confused about Istio or Knative. Always code these as a separate category — they reveal a completely different class of UX failure.

Step 3: Identify Issues Using Plain-Language Signals

The key insight for non-technical UX researchers: you do not need to understand the technical content of an issue to identify its UX signal. The signals are in the language, not the configuration.

Here is how to read any issue in under two minutes.

Signal 1: Time language → friction severity

Scan for time words before you read anything else.

Real example: "I've spent the last three days trying to figure out why my model stays in Unknown status."

You do not need to know what Unknown status means. "Three days" tells you this is a high-severity finding.

Signal 2: The "but" inflection point

Every friction-revealing issue has the word "but" at a specific moment. Everything before "but" is what the engineer did correctly. Everything after is where the product failed them.

Real example: "I followed the quickstart exactly but the webhook never became Ready."

Before "but" = user followed instructions. After "but" = product gave instructions that led to failure. That is a documentation UX finding, not a technical bug.

Real example: "The status shows Ready but every curl request returns 503."

This is the "misleading success" friction type, the product declared success when the user's actual goal was completely unmet. One of the most trust-destroying UX failures possible.

Signal 3: "I thought / I assumed / I expected"

These phrases directly reveal a mental model gap — the engineer built an incorrect picture of how the system works, and reality contradicted it.

Real example: "I thought Transformer meant a language model component like BERT. Turns out it's just data preprocessing."

This is not a bug. It is a naming failure. "Transformer" means attention-based neural architecture in the ML world. KServe uses it to mean "a component that preprocesses data before the model." Every ML engineer who encounters this name builds the wrong mental model from it.

Real example: "I assumed KServe would track model versions automatically — like a proper ML serving platform should."

This is scope confusion — the engineer's expectation of what KServe is does not match what it actually is. That mismatch is a design communication failure, not a user error.

Signal 4: "I had to / I also had to"

Each "I had to" signals a missing workflow step — something the engineer needed that the product should have provided.

Real example: "I had to write a polling script to check when the InferenceService became Ready, because there's no built-in wait command."

Count the "I had to" chains in an issue. An issue with four of them put four separate manual burdens on an engineer for a task that should have been automated.

Signal 5: Tool name count

Scan for capitalized tool names: Knative, Istio, Prometheus, MLflow, Argo, Triton, HuggingFace. Count them.

• 
  1 tool — single-product issue


• 
  2 tools — integration friction


• 
  3+ tools — cross-service challenge; the user must navigate multiple systems for what should be one task

Signal 6: The apology

When an engineer writes "sorry if this is a basic question" — that is not politeness. That is evidence the product made a competent person feel responsible for the product's communication failure.

Step 4: The End-to-End Coding Checklist

Once you have your basic metadata and demographics, here is the full coding structure to apply to every issue.

4A. Deployment Stage

Map every issue to one of 8 stages. This tells you where in the journey the product is losing engineers.

Deployment Stage Checklist

─────────────────────────────────────────────

[ ] Stage 1 · Setup         Installing KServe and dependencies

                             Signals: "webhook not ready" · "quickstart fails"



[ ] Stage 2 · Storage       Getting the trained model accessible

                             Signals: "access denied" · "model not found" · "storageUri"



[ ] Stage 3 · Configuration Writing the InferenceService YAML spec

                             Signals: "minimum config?" · "deprecated field" · "required fields?"



[ ] Stage 4 · Loading       Applying config and waiting for model to load

                             Signals: "Unknown status" · "how long?" · "no logs" · "OOMKilled"



[ ] Stage 5 · Network       Reaching the deployed endpoint

                             Signals: "Ready but 503" · "connection refused" · "EXTERNAL-IP pending"



[ ] Stage 6 · Inference     Sending requests and getting predictions back

                             Signals: "400 error" · "what format?" · "V1 vs V2 protocol"



[ ] Stage 7 · Hardening     Making the deployment production-reliable

                             Signals: "zero downtime update" · "autoscaling conflict" · "SLA"



[ ] Stage 8 · Day-2 ops     Updating, monitoring, governing over time

                             Signals: "rollback" · "update my model" · "60 models across teams"



[ ] Cross-stage?            Root cause in Stage X, discovered at Stage Y

                             (delayed discovery = highest severity finding)

The most important finding: Stages 4 and 5 consistently produce the highest issue volume in K-Serve. The product is completely silent at the two moments when engineers are most anxious and most blind.

4B. Usability Challenge Type

Usability Challenge Checklist

─────────────────────────────────────────────

[ ] U1 · Learnability breakdown

         Cannot figure out how to do the task the first time

         Signal: "how do I" · already answered in docs · confused by concepts



[ ] U2 · Error recovery failure

         Hits error, can't understand it, doesn't know which log to check

         Signal: pastes cryptic error, "stuck for days", tries random things



[ ] U3 · Feedback & visibility gap

         System gives no signal — Unknown, Pending, complete silence

         Signal: "nothing happens" · "how long should this take?" · "no logs"



[ ] U4 · Configuration complexity

         Too many fields, unclear defaults, no minimum viable spec

         Signal: "is all of this needed?" · "which fields are required?"



[ ] U5 · Mental model mismatch

         Expectation contradicts how system actually works

         Signal: "I expected" · "I thought" · "this makes no sense"



[ ] U6 · Workaround proliferation

         User invented their own solution to fill a product gap

         Signal: "I wrote a script" · "I had to" · shared snippets in comments

4C. Developer Friction Type

Developer Friction Checklist

─────────────────────────────────────────────

[ ] F1 · Invisible wall        System silent — nothing to debug

[ ] F2 · Misleading success    "Ready" but goal completely unmet

[ ] F3 · Hidden prerequisite   Required knowledge never communicated until failure

[ ] F4 · Terminology confusion Word means something different in this context

[ ] F5 · Broken feedback loop  Can't tell if a change had any effect

[ ] F6 · Forced context switch Must configure Istio/Knative to complete one KServe task

[ ] F7 · Documentation gap     Knows what they want, can't find how to do it

[ ] F8 · Accumulated friction  5–6 small frictions in sequence → abandonment signal

4D. System & Environmental Challenges

System Challenge Checklist

─────────────────────────────────────────────

[ ] Ownership ambiguity    KServe says "that's Istio", Istio says "that's KServe"

[ ] Abstraction leakage    InferenceService was meant to hide Knative/Istio; it doesn't

[ ] Observability gap      Logs scattered across 4+ components; no unified view

[ ] Role boundary collision ML engineer task structurally requires platform engineer action

[ ] Upgrade path fragility  Every version upgrade risks production breakage



Environmental Challenge Checklist

─────────────────────────────────────────────

[ ] Managed K8s divergence   EKS, GKE Autopilot, OpenShift behave differently

[ ] Corporate proxy / air-gap No public internet; private registry; air-gapped

[ ] GPU & hardware           OOMKilled, VRAM insufficient, driver mismatch

[ ] Org security policy      OPA, Gatekeeper, PodSecurityAdmission blocking KServe

[ ] On-premises / hybrid     No managed LoadBalancer, NFS storage, bare metal

[ ] Regulated / compliance   HIPAA, SOC2, GDPR, data residency requirements

4E. Version Tracking (Essential for Progress Measurement)

This is what transforms your research from a snapshot into a longitudinal UX health report.

Version Tracking Checklist

─────────────────────────────────────────────

[ ] KServe version (exact)         e.g. 0.11.2

[ ] Previous version (if upgrade)  e.g. 0.10 → 0.11

[ ] Kubernetes version             e.g. 1.27

[ ] Cloud provider                 EKS / GKE / AKS / On-prem / Local

[ ] Version stated by:             User upfront | Maintainer had to ask | Never provided

[ ] Upgrade experience:            Better | Same | Worse | New regression introduced

[ ] Chronic pain signal:           Same issue present in prior version? Yes / No

The chronic problem list — friction points that appear in the top-3 across three or more versions — is your most powerful finding. A problem that survived three release cycles is not a bug. It is an architectural decision.

4F. LLM Inference Tracking (Track Separately — Fastest Moving Component)

LLM Inference Checklist

─────────────────────────────────────────────

[ ] Is this an LLM issue?          Yes | No | Hybrid

[ ] LLM model family               Llama | Mistral | Qwen | Gemma | Custom

[ ] LLM runtime                    vLLM | TGI | OpenAI-compatible | Custom

[ ] Capability attempted:

      Basic inference

      Streaming tokens (SSE)

      Multi-GPU / tensor parallelism

      LoRA / adapter serving

      HuggingFace Hub authentication

      OpenAI API compatibility

      Quantisation (GPTQ, AWQ)



[ ] LLM-specific challenge:

      GPU OOM / VRAM insufficient

      Model loading with no progress signal

      Streaming failure through gateway

      HuggingFace auth failing in cluster

      Runtime version lag behind vLLM/TGI ecosystem

      No LLM-specific metrics (token throughput, TTFT)



[ ] Innovation lag signal:

      Date of capability request: ___________

      Date KServe released support: ___________

      Gap (days): ___________

4G. Research Question Mapping

For each issue, record which research question it provides evidence for. This anchors your mining to your study goals.

Research Question Mapping

─────────────────────────────────────────────

Current-state questions (what is broken today)

[ ] RQ1 · First deployment challenges across roles and experience levels

[ ] RQ2 · Workflow gaps between deployed model and reliable production

[ ] RQ3 · Observability and debugging challenges by stage

[ ] RQ4 · LLM deployment challenges vs classical ML serving

[ ] RQ5 · Environmental factors shaping deployment experience

[ ] RQ6 · How challenges evolved across versions (your unique longitudinal contribution)

[ ] RQ7 · Design changes that would most reduce friction



UX improvement questions (what should be designed differently)

[ ] UX1 · Time-to-first-inference reduction

[ ] UX3 · Model loading progress visibility (highest volume finding)

[ ] UX4 · Self-service diagnostic experience

[ ] UX9 · LLM mental model bridge (vLLM/HuggingFace → KServe)

[ ] UX11 · Environment validator / dependency pre-flight checker

4H. UX Finding Template

Once you spot a pattern across multiple issues, record it here. One template per pattern — not per issue.

UX Finding Template

─────────────────────────────────────────────────────────────

Finding type:       ___________________________________________

Affected users:     Role · Experience level · Version band

Deployment stage:   ___________________________________________

Evidence:           N issues · Date range · e.g. "14 issues, 2022–2024"

Best quote:         Under 25 words — your strongest evidence

─────────────────────────────────────────────────────────────

UX finding statement:

"Engineers [doing X] cannot [accomplish Y] because [design gap Z],

 which means [impact on time / confidence / adoption]."

─────────────────────────────────────────────────────────────

Severity:           Low | Medium | High | Critical

Chronic?            Present across ___ versions

Design recommendation: ________________________________________

Research question answered: ___________________________________

Step 5: The Session Checklist — Before Calling Mining Complete

Mining Session Completion Checklist

─────────────────────────────────────────────

Issue coverage

[ ] 50+ general deployment issues coded across all 8 stages

[ ] 30+ LLM inference issues coded and version-tracked

[ ] 10+ issues per major version band (v0.10, v0.11, v0.12, v0.13+)

[ ] 15+ upgrade issues with before/after UX delta recorded

[ ] Top 30 most-commented issues reviewed (sort:comments-desc)

[ ] 10+ abandoned issues (open 6+ months, last message unanswered)

[ ] 15+ success cases (quickly-resolved — positive signal baseline)

[ ] All enhancement/feature-request labels reviewed

[ ] All competitive tool mentions captured (BentoML, Seldon, Ray Serve)



Baseline measurement

[ ] Metrics table filled per version band:

      Issue count | Avg comments | 7-day resolution rate | Emotional language %

[ ] Top-3 friction points per version — chronic problem list built

[ ] LLM innovation lag calculated for 5+ capabilities

[ ] Version reporting rate: % of issues that include version upfront



Quality

[ ] 15% of issues coded by a second researcher (inter-rater reliability)

[ ] Every research question has at least 3 issues as evidence

[ ] One finding statement written per significant pattern found

How This Benefits UX Researchers and the Community

For UX researchers without a technical background

The biggest barrier to studying developer tools as a UX researcher is the assumption that you need to understand the code to understand the problem. This framework removes that barrier entirely.

When you code an issue, you are not evaluating the correctness of someone's Kubernetes configuration. You are recording what the issue reveals about the human experience of using the product. "Status shows Unknown for 20 minutes" tells you everything you need regardless of whether you understand what Unknown means technically. The product left a user without feedback during its most critical operation. That is a UX finding independent of any technical knowledge.

For product managers and maintainers

The coding sheet converts anecdote into pattern. Instead of "users seem to struggle with deployment," you can say "14 of 20 issues sampled from v0.11 show feedback gap failures at Stage 4, with an average of 18 comments per issue, suggesting loading state communication is the highest-priority improvement target for this version band."

That is a product roadmap argument. The coding sheet built it.

For open source contributors

The research questions embedded in the coding sheet map directly to design recommendations with evidence behind them. A contributor who wants to make a meaningful impact on user experience now has specific, evidence-backed targets — not "improve docs" but "add granular status conditions per loading phase that distinguish between 10 failure modes currently all reporting as Unknown."

For the community as a whole

When researchers share findings publicly in CNCF Blogs, KubeCon talks, or articles like this one — the coding framework makes the research reproducible. Other researchers can apply the same checklist to a different version or a different tool and compare results. That cumulative body of evidence is what eventually changes product direction.

Conclusion

GitHub issues are not a bug tracker. They are a longitudinal, naturalistic record of where real engineers encounter the gap between what a product promises and what it delivers.

A coding sheet is the analytical framework that transforms that record into research. Without it, you are reading. With it, you are studying.

The framework I built for KServe — covering demographics, deployment stages, usability challenges, friction types, mental model gaps, system challenges, environmental barriers, version tracking, and LLM inference — did not emerge from theory. It emerged from reading hundreds of issues and asking the same question every time: what is the UX researcher's reading of this, beyond what the engineer sees?

The answer is always the same: engineers see symptoms. The coding sheet helps you see the design decisions that caused them.

Start with the "but" sentence. Work backwards to the design failure. Code it. Repeat 100 times. Then write the research report.

This coding framework was developed as part of a UX research study on ML model deployment in KServe. If you are working on similar research in the cloud-native or MLOps space, I would love to hear your thoughts.