How to Audit Whether Your Analytics AI Is Really Private (5-Question Checklist)

Every analytics tool now has an AI feature, and every AI feature now has a privacy page. The pages read well. They are also, in the main, written to be reassuring rather than falsifiable — long on "enterprise-grade" and short on who exactly runs the model and what happens to the request.

You do not need a legal background to test that. You need five questions asked in the right order, because the order matters: the first one can invalidate the answers to all the others. Send them by email, ask for written answers, and keep the replies. That exchange is also, conveniently, the evidence your accountability file needs.

Question 1: Who owns the company running the inference?

Not who sells you the product — who operates the machine the model runs on. These are often different companies, and the second one is the one that matters.

The reason is jurisdictional. The US CLOUD Act (2018) obligates any provider subject to US jurisdiction to disclose data in its possession or control regardless of where in the world that data is stored. FISA Section 702 adds a separate foreign-intelligence collection regime aimed at non-US persons. Both follow the corporate parent. Neither is affected by which region you selected in a dropdown.

So if the answer names a US-headquartered group, you are in residency territory, not sovereignty territory — and you should stop and evaluate that exposure before spending time on the rest of the checklist. We wrote this distinction up in more detail in Residency Is Not Sovereignty.

Question 2: Where does inference run, exclusively?

The load-bearing word is exclusively. Plenty of providers can process in Europe. Fewer will commit that they process nowhere else, and the gap between those two statements is where incidents live.

Two follow-ups are worth asking. First, does the endpoint you are actually calling pin a region, or is it a global endpoint that may route by capacity? Google's Vertex AI, for instance, supports EU region pinning, but its global endpoint gives no control over where processing happens — same vendor, two very different answers. Second, are failover and overflow capacity in the same region? A region-pinned steady state with an unpinned failover is not EU-only.

It is also worth knowing the shape of the market. US providers have built real EU options: OpenAI offers EU data residency configured per project with in-region zero retention, Azure OpenAI offers an EU Data Boundary deployment type. Anthropic's first-party API has no EU residency option at all — EU processing goes through AWS Bedrock EU regions or Google Vertex AI EU instead. These are genuine improvements over sending data to Virginia. None of them changes the answer to question one.

Question 3: What is retained, and for how long?

Ask separately about three things, because vendors often answer one and let you assume the rest: the prompt content, the model output, and the metadata around the call.

A credible zero-retention answer names its exceptions. A provider that says "we retain nothing, ever, under any circumstance" is either not operating the infrastructure or not describing it accurately — real systems keep something when something breaks. What you want is the boundary written down: which condition triggers retention, what is kept, for how long, and who can read it.

Metadata is the part people forget. Counting tokens for billing is legitimate and necessary. Storing the text of every question your team asked, indefinitely, in order to "improve the experience" is a different thing wearing the same word.

Question 4: Does anything train a model?

This has a clean version and a soft version, and you want the clean one. Clean: customer inputs and outputs are never used to train, fine-tune or improve any model, with no action required from you. Soft: "we do not train on your data" followed, several paragraphs later, by a carve-out for aggregated, de-identified or telemetry data.

Check where the commitment lives. If it is only in a blog post or a FAQ, it is a statement of current intent. If it is in the data processing agreement, it is an obligation. And if the commitment depends on you having toggled something in an admin panel, then your privacy posture depends on configuration — which means it depends on whoever configures it next.

Question 5: Can you leave — and can you reproduce what happened?

This is the question almost nobody asks, and it is the one that decides whether your compliance file still means anything in twelve months.

A closed API model can change under you silently. The version you assessed, red-teamed and documented in your risk assessment may not be the version answering your queries next quarter, and you will have no way to prove otherwise. An open-weight model under a permissive licence can be pinned to an exact version, inspected, red-teamed and reproduced — and if the host disappoints you, the same weights are available from other hosts, so switching is an infrastructure decision rather than a rebuild.

Portability is usually filed under procurement. It belongs in the privacy review, because reproducibility is what turns a claim into evidence.

What a good answer sounds like, and what an evasive one sounds like

You will rarely get a flat refusal. You will get an answer that addresses a nearby question instead. This table is a rough decoder.

One general rule: an answer that is easy to check is worth more than an answer that is impressive. "Scaleway SAS, Paris, France" can be verified in an afternoon. "Leading European infrastructure" cannot be verified at all.

A worked example: running the checklist on ourselves

A checklist you never apply to yourself is a sales document. So here are our own five answers, for Seal AI, the private AI layer inside SealMetrics.

1. Ownership. Inference runs on Scaleway Generative APIs. Scaleway is a French company whose parent is the Iliad group, with no US ownership, and it states explicitly that its AI services are not subject to extraterritorial laws such as the American CLOUD Act. It is listed as an Article 28 processor in our subprocessor list — Scaleway SAS, Paris, France, purpose LLM inference.

2. Location. Paris, France, only. Because the data never leaves the EU and the processor has no US parent, GDPR Chapter V — the international-transfer chapter — is not triggered at all. No Standard Contractual Clauses, no Transfer Impact Assessment, no dependence on the EU-US Data Privacy Framework surviving its next hearing.

3. Retention. Scaleway applies Zero Data Retention by default, with one documented exception: on a severe service error, the failing request may be kept for up to two weeks for root-cause analysis. On our side, the metering layer persists only token counters — organisation, model, input and output token counts — for quota and billing. Prompt and response content is never persisted by it. Transport is TLS 1.2 or above with certificate verification.

4. Training. Nothing. No training on customer data at the host, and the model is a static set of published weights, so there is no mechanism by which your questions could influence it.

5. Exit. The model is gpt-oss-120b, open weights under the Apache 2.0 licence. The exact version can be pinned, inspected and reproduced, and the same weights are served by other EU-sovereign hosts — OVHcloud AI Endpoints, IONOS AI Model Hub with processing confined to Germany, Nebius in Finland and France, though Nebius is US-listed, so governance deserves as much attention there as geography. We chose our host; we are not locked to it.

Two honest footnotes, since a checklist that only produces flattering answers is not a checklist. Scaleway holds ISO/IEC 27001:2022 and HDS, granted in July 2024; its SecNumCloud qualification with ANSSI has been in progress since January 2025 and is not granted — we say "in qualification" and nothing stronger. And customers who prefer a different model can configure their own OpenAI, Anthropic, Gemini or DeepSeek key, in which case their prompts go to that provider under that provider's terms. That is a deliberate, per-customer choice, kept clearly separate from Seal AI — which stays the default precisely so that privacy does not depend on configuration.

What to do with the answers

Score them the way a regulator would: not on tone, but on whether each claim is written down somewhere binding and checkable. A vendor who answers question one with a named entity and a named parent has told you more in one line than a page of "enterprise-grade privacy" ever will. A vendor who cannot answer it at all has told you something too.

If you want the full technical write-up behind our own answers, it is in the Seal AI architecture documentation, and the market audit that led to the model choice is in the model selection report.