Switch AI Providers Without Losing Your Workflows: A Practical Exit Test

Image: Michael Jasmund on Unsplash.

Portability is something you demonstrate, not something an export button promises. A real exit test exports your assistant's state, imports it somewhere clean, connects a second model provider, reruns the same tasks and records what moved, what needed adapting and what behaved differently.

Run it before you need it.

Prices change.

Policies change.

Models get retired.

The best time to find out you are locked in is during a calm afternoon, not an outage. This guide is part of the AI harness engineering guide.

Define what leaving a provider means

"Switching" can mean three different things. Decide which one you are testing.

Scope	What changes	Typical effort
Model switch	The model behind the harness	Adapter and prompt tuning
Connector switch	A tool or data service, such as a different calendar	New tool contract and credentials
Platform migration	The whole assistant moves to different software	Export, import, rebuild and retest everything

Then write success criteria before you start:

• all portable state arrives intact,
• data still flows only to allowed destinations,
• the evaluation suite passes at an acceptable rate on the new setup,
• old credentials are revoked.

Inventory portable and provider-specific state

Sort everything the assistant depends on.

State	Portable?	Notes
Your documents	Yes	Keep originals in files you own
Memory records	Yes, if exportable with sources	See the memory guide
Prompts and instructions	Yes	Store as plain text under version control
Workflow definitions	Depends	Portable if written as code or open formats
Tool contracts	Yes	Inputs, outputs and scopes in plain text
Evaluation cases	Yes	The single most useful thing to carry over
Search indexes and embeddings	Rebuild	Usually specific to one embedding model
Provider-hosted memory or files	Depends	Check whether an export exists and what it includes
Credentials and OAuth grants	Never copy	Re-authorise in the new setup

Mark anything you cannot export as unknown. That row is your lock-in.

Create a readable export contract

An export should be understandable without the software that produced it.

export/
  manifest.json        schema version, date, counts, SHA-256 per file
  memory.json          records with ids, sources, scopes, status
  prompts/             one file per instruction set
  workflows/           definitions in a documented format
  tools.md             every tool contract
  evals/               the test cases and their expected outcomes

Leave secrets out. Leave caches and indexes out; they get rebuilt. Record a checksum for each file so you can prove nothing changed in transit.

Adapt the model and tool interfaces

Model providers differ in ways that matter to a harness:

how tool calls are requested and returned,
structured output support,
context length,
error formats and rate limits,
what data they retain and for how long.

Put a thin adapter between your harness and each provider, so the rest of the system speaks one internal format. The quickstart uses this idea: its loop expects either { tool, args } or { answer, sources }, whatever produced it.

When the new provider lacks a capability, fail visibly. Never quietly weaken a permission or route sensitive data somewhere the old setup did not send it. Check the new provider's data policy for the exact product and endpoint you will use; the local vs cloud guide lists the questions.

Run the same workflows before and after

This is the test. Everything else is preparation.

1. Baseline. Run your evaluation suite on the current setup. Record results, cost and latency, with versions and dates.
2. Import. Load the export into a clean environment.
3. Re-index. Rebuild search indexes with the new configuration.
4. Rerun. Run the identical suite on the new setup, several times per case.
5. Compare. Report per case: same, better, worse or broken.

Expect differences. Models are not interchangeable, and a prompt tuned for one may underperform on another. The goal is not identical behaviour; it is knowing exactly what changed. The evaluation guide covers repeat runs and honest reporting.

Check final state, not messages. If a workflow drafts replies, inspect the drafts on both setups.

Revoke old access and plan rollback

Once the new setup passes:

• revoke the old provider's API keys and connector grants,
• confirm the old credentials no longer work,
• delete or export any data left with the old provider, according to its documented process,
• keep the old export and baseline results for rollback.

Rollback should be a decision you can make in minutes: point the adapter back at the old provider, re-authorise, rerun the suite. Never resurrect revoked credentials as a shortcut. Re-authorise deliberately.

Remember the limit of every rollback. It restores your system, not the outside world. Messages sent during the migration stay sent.

Publish the real limits of portability

Write a short record and keep it with your exports:

Item	Result
Moved unchanged	e.g. documents, memory, prompts, tool contracts
Needed adaptation	e.g. tool-call format, one prompt
Behaved differently	e.g. briefs longer; two boundary cases changed
Could not move	e.g. provider-hosted conversation history
Date and versions	Both setups

Retest when either provider changes models, formats or policies. For disaster recovery rather than migration, see the backup and restore runbook.

Frequently asked questions

Is an export file enough to avoid lock-in?

No. You have avoided lock-in when the export imports cleanly and your tasks pass on the new setup.

Will every model behave the same after a swap?

No. Expect differences in quality, format, tool use and cost. Measure them with the same test suite.

Can provider-hosted memory always be exported?

Not always, and not always with sources and dates. Check before you depend on it.

How does migration differ from backup and restore?

Migration moves to a different provider or platform. Backup and restore return the same system to a known state after a failure.