The first objection a CISO raises to almost any AI analytics pitch is some version of the same question: “so now I need to put another agent on every developer’s laptop?” It’s a fair question — security teams have spent years fighting endpoint sprawl, and a new mandatory install is a real cost even when the tool behind it is useful. For Olakai’s AI coding tool detection, the answer is no. Nothing runs on a developer’s machine. The entire signal comes from pull request data that already exists in GitHub, Bitbucket, or GitLab, read through the same organization-level credential used to power the AI Impact Dashboard.
That’s a meaningfully different trust posture than most AI monitoring tools, and it’s worth walking through exactly how the detection actually works — not as a black box, but as a stack of specific, checkable signals, with an honest fallback for the cases none of them catch.
Three signals, applied in order
Every merged pull request is checked against three detection methods. First, bot author detection: was the PR opened by a known bot account — dependabot, renovate, devin-ai, copilot-swe-agent, claude-code, sweep-ai, snyk-bot, and others recognized by GitHub’s own `user.type === “Bot”` flag? Second, commit co-author trailers: do any commits in the PR carry a Co-authored-by: line matching a known AI tool pattern, covering GitHub Copilot, Cursor, Claude Code, Devin, Amazon Q, and Gemini? Third, PR title and body markers: does the PR’s own text contain a recognizable phrase — “Generated by Cursor,” “claude-code,” “Copilot Workspace,” “Created by Devin,” and similar strings that AI tools leave behind by convention?
If any one of those three signals fires, the PR is classified AI-assisted — and a single PR can be attributed to more than one tool if different commits carry different signals, which happens more than you’d expect on PRs where a human picks up and finishes AI-started work. A PR earns the stronger label of fully agentic specifically when a bot account opened it: the AI created the branch, wrote the code, and opened the PR itself, without a human author in the loop at all. That’s a real distinction, not a marketing one — a fully agentic PR is a different governance conversation than one where a human developer used AI as a fast collaborator.
What happens when none of the three signals fire
Not every AI-assisted PR leaves a clean marker. A developer might paste AI-generated code into a normal commit with no trailer, no bot account, and no mention in the PR body. For that gray area, Olakai runs an LLM classifier — Claude Haiku 4.5 — that reads the PR title, body, and a sample of the diff, and assigns a confidence score. Critically, the PR is only marked AI-assisted through this path at 60% confidence or higher; anything below that threshold is treated as “not detected,” specifically so ambiguous cases don’t get counted and inflate the AI-assisted total.
That’s a deliberate design decision worth sitting with for a second: the system is built to under-count in ambiguous cases rather than over-claim. Most vendors pitching AI-coding-tool ROI have every incentive to inflate the “AI is helping” number — a bigger adoption percentage is a better story. Olakai’s classifier does the opposite by default, which is exactly the kind of choice that should show up in an honest AI coding tool ROI metric instead of a vanity one.
Multi-provider integrity, and where the fidelity differs
Three source-control providers feed the same underlying pull-request table: GitHub, the production integration with the most mature detection signal set; Bitbucket Cloud, currently in beta; and GitLab, covering both SaaS and self-managed instances, using the same signal types plus GitLab-specific Duo markers. Every row is uniquely keyed by account, provider, repository, and PR number, so an organization running both GitHub and Bitbucket at once never gets a collided or double-counted PR — the two providers’ data sits cleanly side by side in the same analytics.
Fidelity isn’t identical across providers, and Olakai says so rather than presenting every provider as equivalent. Bitbucket and GitLab have no first-class “review” object the way GitHub does, so review rounds and approvals are derived from each platform’s activity stream instead of native review submissions — directionally correct, but lower-fidelity than GitHub’s numbers. Any cross-provider cycle-time comparison should carry that caveat rather than treating a GitHub PR and a Bitbucket PR as measured on perfectly identical instruments.
Whose PR is this, exactly?
Detecting AI usage is only half the problem; attributing it to the right developer is the other half, and it’s less glamorous but just as important. Olakai resolves author identity through a four-step priority order: first, the commit email from a commit whose author matches the PR opener, excluding GitHub’s own noreply addresses; if that’s unavailable, the email from the PR’s first commit; failing that, the email on the GitHub user profile; and as a last resort, a constructed noreply fallback built from the GitHub login. That chain exists because real organizations have messy Git configuration — different emails on different machines, corporate SSO aliases, personal accounts used for a first commit — and per-developer ROI attribution is worthless if it silently drops or misattributes a chunk of activity because someone’s `git config` didn’t match their Olakai account exactly.
The same pull-request pipeline also captures signals beyond raw detection — PR size classification from XS to XL by lines added (used to flag AI PRs that are suspiciously always large, which the documentation itself calls a potential sign of rubber-stamping), issue linkage via Fixes #N or Closes #N references, first-pass approval rate, and a test-file ratio that the product is explicit about not being perfect: it’s based on file count, not line count, so a PR with one large test file and many small source files reads as low-coverage even when it isn’t. Publishing that limitation alongside the metric is the same pattern as the LLM classifier’s 60% confidence floor — under-claim rather than over-claim.
Put together, the detection layer is the foundation everything else in Olakai’s AI monitoring is built on — the PR mix, the cycle-time comparisons, the productivity score, all of it depends on getting “was this PR actually AI-assisted, and by which tool” right at the source, without asking a single developer to install anything. For a CISO evaluating the tool, that’s the actual pitch: the analytics run on data your organization already generates and already controls access to, not on a new agent asking for a new set of permissions.
Want to see what this detection stack finds in your own repositories before rolling it out further? Talk to an Expert.