Skip to content
Loading iBrothers Group
Back to top
The method

How the Checks Work

A check, not a claim

Claim vs Install

Where a vendor's claim meets a real install: the same twelve checks on every agent.

Most directories repeat what a vendor says. We install every agent on a throwaway account we own, write down what it actually asks for beside what its documentation says it needs, and publish the method for every check so you can judge it yourself.

  • 12 Checks on every agent
  • 3 Observed on a throwaway account
  • 3 Required to be listed at all
List your agent See the twelve checks
The method

The Twelve Checks

From install to profile, each with the method a reviewer follows, and what a reader is told when it passes and when it does not. The ones marked Observed happen on the throwaway account.

02  Vendor site & domain

Check WHOIS for the registration date, load the site over HTTPS, and confirm the footer or terms name the same entity as the registry lookup.

  • When it passes: Domain since :year; site live

  • When it does not: Site did not load, or names a different entity

03  Product is live
Required to be listed

Install the agent or reach its trial without talking to sales. A waiting list is recorded as such rather than failed, but it is not eligible to be listed.

  • When it passes: :state

  • When it does not: No installable product or trial found

04  Permissions declared

The vendor's own documentation must list the OAuth scopes, API keys, roles or file paths the agent needs. Each one goes on the permission worksheet as declared.

  • When it passes: Declared: :count permissions

  • When it does not: The documentation does not list the permissions it needs

05  Permissions observed
Observed Required to be listed

Install the agent on a throwaway account owned by iBrothers, seeded with dummy data. Record every scope the consent screen actually asks for and compare it, scope by scope, with the declared list. A scope requested but not declared cannot pass.

  • When it passes: Requested :count; :match

  • When it does not: Could not be observed, or requested permissions it did not declare

06  Data flow stated

The vendor must state where data goes, how long it is kept, whether it is used for training, and which sub-processors see it. We check the statement exists, not that it is true.

  • When it passes: Data-flow statement present

  • When it does not: No statement of where data goes or how long it is kept

07  Autonomy & approval
Observed

On the throwaway account, give the agent a task and watch: does it act without confirmation, or is there an approval step before anything is sent, written or paid?

  • When it passes: :observed

  • When it does not: Autonomy could not be observed

08  Off switch
Observed

Revoke the agent's access, stop it, and request deletion of its data. Count the steps, and confirm the access is actually gone from the provider's side afterwards.

  • When it passes: Revocable in :steps steps; verified

  • When it does not: Access could not be revoked cleanly

09  AI provenance stated

The profile must state which models are used, from which provider, whether they are the vendor's own or an API, and how the agent is evaluated. We check the questions are answered, not that the answers are true.

  • When it passes: Provenance stated (content not verified)

  • When it does not: Models, providers or evaluation not stated

10  Privacy, terms & security page

A privacy policy and terms must be present, dated, and name the entity. A claimed certification must link to the issuer or a letter, not to a logo.

  • When it passes: :summary

  • When it does not: Policies missing, undated, or name no entity

11  Pricing public

A pricing page must be reachable without contacting sales. "Contact us" is recorded as not public rather than failed.

  • When it passes: Pricing public

  • When it does not: Pricing not public

12  Support & incident path

There must be a named support channel, and a documented process for incidents and for rolling back a bad release.

  • When it passes: Support channel and incident process stated

  • When it does not: No support channel or incident process found

Never checked: accuracy, "best", return on investment, "hallucination-free". Those stay the vendor's own claims, shown with their evidence or marked Unverified.

The rules

Three Fixed Rules

The throwaway-account rule

  • Every agent is installed on an account iBrothers owns.
  • Seeded with dummy data, and nothing else.
  • Never a customer's account, never a venture's.
  • Access revoked afterwards; the account wiped before the next review.

Why it is free

  • The moment a vendor pays, the register works for the vendor.
  • No listing fee, no commission, no featured place.
  • We do not host or run any agent.
  • A paid audit elsewhere is never counted as a check here.

What "Verified" means

  • A named person looked at one thing, on one date, on one version.
  • Not that the agent is safe, accurate or worth buying.
  • Behaviour can change with any release.
  • So every profile shows the date, and we re-check every six months.
Plain English

Permissions, Plainly

Every permission, written the same way. Every scope an agent asks for is described in the same words on every profile, with how much it lets an agent do. These are the ones we have wording for so far.

High risk

20 scopes

Medium risk

12 scopes

Low risk

6 scopes

Google

11 scopes · 7 high, 3 medium, 1 low

  • https://mail.google.com/ High risk

    Full access to the mailbox, including permanent deletion

    Almost no agent needs this; ask why.

  • calendar.events Medium risk

    Can create, change and delete events on your calendars

  • calendar.readonly Medium risk

    Can see every event on your calendars, with its details

  • contacts.readonly Medium risk

    Can read your contacts

  • drive High risk

    Can read, change and delete every file in your Drive

  • drive.file Low risk

    Can see and edit only the files it creates or you open with it

  • drive.readonly High risk

    Can read every file in your Drive

  • gmail.compose High risk

    Can write drafts and send email as you

  • gmail.modify High risk

    Can read, label, archive and bin email

  • gmail.readonly High risk

    Can read every email and attachment in the mailbox

  • gmail.send High risk

    Can send email as you

Microsoft

9 scopes · 5 high, 3 medium, 1 low

  • Calendars.Read Medium risk

    Can see every event on your calendars

  • Calendars.ReadWrite Medium risk

    Can create, change and delete calendar events

  • Files.Read.All High risk

    Can read every file you can open in OneDrive and SharePoint

  • Files.ReadWrite.All High risk

    Can read, change and delete every file you can open in OneDrive and SharePoint

  • Mail.Read High risk

    Can read every email in the mailbox

  • Mail.ReadWrite High risk

    Can read, change and delete email

  • Mail.Send High risk

    Can send email as you

  • offline_access Medium risk

    Keeps its access while you are not using it

  • User.Read Low risk

    Can read your basic profile: name, email address and photo

AWS

6 scopes · 3 high, 2 medium, 1 low

  • AdministratorAccess High risk

    Can do anything in the account, including deleting it

    No agent should need this; treat it as a refusal to scope.

  • ce:GetCostAndUsage Low risk

    Can read the account's cost and usage reports

  • iam:PassRole High risk

    Can hand an IAM role to a service - a common route to wider access

  • ReadOnlyAccess High risk

    Can read almost every resource and setting in the account

  • s3:GetObject Medium risk

    Can read objects in the S3 buckets it is given

  • s3:PutObject Medium risk

    Can write objects to the S3 buckets it is given

GitHub

6 scopes · 3 high, 1 medium, 2 low

  • admin:repo_hook High risk

    Can add and remove webhooks on your repositories

  • public_repo Medium risk

    Can read and write your public repositories

  • read:org Low risk

    Can see which organisations and teams you belong to

  • read:user Low risk

    Can read your GitHub profile

  • repo High risk

    Can read and write every repository you can reach, private ones included

    A GitHub App limited to chosen repositories asks for far less.

  • workflow High risk

    Can change GitHub Actions workflows, and so what runs in CI

Slack

6 scopes · 2 high, 3 medium, 1 low

  • channels:history Medium risk

    Can read messages in the public channels it is added to

  • chat:write Medium risk

    Can post messages

  • files:read Medium risk

    Can read files shared in the channels it can see

  • groups:history High risk

    Can read messages in the private channels it is added to

  • im:history High risk

    Can read direct messages it is part of

  • users:read Low risk

    Can see the people in the workspace

For vendors

Build an Agent?

Put it through the same twelve. Free, now and later. You see the profile before anyone else does.

List your agent

Free, now and later

Join the agent marketplace

A few details and a verified address is all it takes. You write your profile afterwards, and nothing is shown to anyone until our team has reviewed it. How the checks work

I am registering as

One organisation per domain: if yours is already registered, you ask to join it.

Sellers: an address on your website's domain. Buyers: a personal address lets you browse and save; a work one lets you contact sellers.

At least 8 characters, with letters and numbers.

Already registered? Sign in

Get in touch

Ask about the agent register

Questions about the register, an agent on it, or the group? Write, and a person will answer.

Your message is kept in our admin panel and answered by email. Nothing is sent to anyone else.

New agents in the categories you follow

A monthly note listing what was added and re-checked. Nothing else.

For the newsletter only, nothing else. How we handle it