Research Human Security Review in the Copilot App With Stop Conditions

GitHub announced on July 14, 2026 that security reviews are available in the GitHub Copilot app.

Primary source: GitHub Changelog, July 14, 2026.

The meaningful research question is not whether people click Accept. It is whether they can build an evidence-backed decision when guidance is useful, incomplete, or wrong.

understand change -> inspect evidence -> challenge findings

                  -> verify uncertainty -> accept, reject, or escalate

This is a proposed research protocol, not a completed study. It does not invent product fields or report findings.

Build scenario cards

scenario_id: "SR-03"

repository_type: "synthetic"

seeded_conditions:

  - "one relevant issue"

  - "one plausible but irrelevant concern"

  - "one important omission"

participant_goal: "ready, blocked, or escalate"

success_evidence:

  - "decision cites inspected code"

  - "unsupported claim is challenged"

  - "unresolved uncertainty is recorded"

stop_conditions:

  - "real credentials appear"

  - "a live repository could be modified"

  - "participant mistakes study output for production approval"

Vary the seeded mix so participants cannot learn that every scenario contains exactly one true and one false finding. Establish ground truth independently before sessions.

Recruit people who hold different review responsibilities: routine reviewers, maintainers, security specialists, less-experienced reviewers, and people using keyboard navigation or assistive technology. Do not collapse every group into one average.

Require a decision record

Decision: ready | blocked | escalate



Evidence inspected:

- file and relevant lines

- test or documentation



Guidance accepted:

- claim and evidence



Guidance rejected:

- claim and reason



Unresolved:

- question and next owner

Spoken confidence is not the outcome. This artifact exposes whether acceptance connects to evidence.

Measure relevant issues identified, unsupported claims challenged, evidence references, correct escalation, time, and confidence before and after inspection. None alone proves production effectiveness. Avoid optimizing acceptance rate: low acceptance may mean healthy resistance, confusion, or poor guidance.

Stop safely

Stop an individual session if credentials, personal data, confidential code, live production actions, unclear authorization, withdrawn consent, inaccessible tasks, or mistaken production approval appear.

Pause the whole study if ground truth is disputed, logging captures more than approved, the same safety problem repeats, or participants cannot distinguish simulation from production.

Analyze three dimensions: reliance, recovery after bad guidance, and omission—what the review never mentioned. Preserve disconfirming cases rather than averaging them away.

A lab study cannot establish real-world vulnerability detection. Think-aloud changes behavior, synthetic repositories reduce organizational pressure, and expertise is hard to stratify. Still, it can test a critical property: whether reviewers inspect evidence, contest the system, record uncertainty, and stop before unsafe action.