Satya Nadella Says Companies Should Own Their AI Quality Tests

Satya Nadella’s October 26 20VC interview gives marketing teams a reason to keep the thinking behind an AI approval. A record of what passed, what failed, and why gives the next tool a standard grounded in the company’s own work.

The Microsoft chairman and CEO describes a future in which each company has:

“a private eval benchmark, which is their core IP.”

An evaluation benchmark is a set of tasks and standards used to judge the work a model produces. Nadella’s argument, around 12 minutes into the released interview, is that the knowledge a business develops while using AI should retain value for that business. Models would compete to perform better against its own test.

For marketers, the practical asset is the reason a draft was accepted. A finished campaign shows a new tool the result. The decision record helps explain what made that result acceptable.

Keep the reasons behind the edits

In the interview passage, Nadella discusses who benefits from feedback. Nadella discusses even a thumbs-up or thumbs-down as something that creates value. He argues that more of the value produced through using AI should remain with the firm, and says that knowledge can sit in context or tests rather than only inside the model itself.

That is a strategic argument from an enterprise AI seller. It doesn’t establish that a provider uses confidential customer material for training, and it doesn’t announce a new Microsoft product or data policy. His description of core intellectual property is also no substitute for checking ownership and confidentiality terms.

The marketing application is ours. Consider a team deciding whether a product email is ready. A tidy draft might still describe a feature the product doesn’t offer. A more cautious version might be accurate but fail to explain why the buyer needs it. Keeping the approved wording alone loses the distinction between those decisions.

A useful review record preserves the source behind a claim and the reason a reviewer changed it. When the team tries another model, those examples give it something specific to assess. The question becomes whether the new output resolves known problems in real work.

Our coverage of Basis’s AI measurement gap raised the difficulty of counting automation without establishing its full return. A company’s quality test addresses a narrower question: whether the output meets its requirements. Passing that test still doesn’t prove the campaign produced a commercial gain.

The standard has to come from people

Machine learning engineer Hamel Husain and computer science researcher Shreya Shankar give the work a practical starting point in their joint guide to AI evaluation. They recommend reviewing actual outputs before asking an agent to propose what has gone wrong.

Their instruction for that early review is short:

“Keep this first pass manual.”

Their concern is that early automated suggestions can steer what the reviewer notices. The team needs room to discover problems that its first checklist didn’t anticipate. Their guide also recommends revising criteria when real examples expose a gap or reviewers disagree.

This is practitioner guidance, not a measured promise that keeping approval notes will increase sales. Both authors sell related teaching and a book. Its useful contribution is explaining how a company begins to turn an experienced person’s judgment into something colleagues can inspect.

That work has an owner. In marketing, the person accountable for the product claim or customer message needs time to explain a rejection. Otherwise, a new test can simply automate an old misunderstanding. Our earlier Anthropic marketing-operations story put the remaining human work in checking the numbers. The same responsibility applies to defining what a correct answer must accomplish.

A useful test can change

When the team changes models, keep the rejected examples alongside the work that passed. The original brief and the reviewer’s reason explain what the new tool needs to handle. A polished portfolio of final outputs alone won’t show where the previous model failed.

Revisit the examples when the offer or audience changes. A test built around outdated positioning can reward work the company has stopped wanting. The test should travel with the team through a vendor change, along with the reasons its standards changed. Before approving the new tool, check its results against those known failures as well as the finished work.