20 Questions to Ask in Every WealthTech AI Demo
Do not walk into a wealthtech AI demo with rose-colored glasses. Every tool looks impressive in a controlled presentation. Vendors invest significant resources in their demo environments. The data is clean, the workflow is choreographed, and the edge cases your firm would actually encounter are nowhere to be seen.
The difference between a good technology decision and an expensive mistake is what you ask when the scripted presentation ends.
Before the demo begins, prepare your own requirements list. Do not let the vendor set the agenda. Document your firm’s specific use cases, the client workflows where you need improvement, and the compliance constraints your technology must navigate. The universe of viable tools compresses quickly when you lead with your operational needs rather than their feature list.
The framework below is organized in five levels, from governance fundamentals to roadmap accountability. Work through them in order. If a vendor cannot answer the questions in Level 1, the answers in Level 2 become irrelevant.
Level 1: Governance First, Features Second
A vendor who leads with flashy features before establishing its governance and compliance framework has already told you something important. Questions to ask first:
1. What data do you capture from our interactions with your platform, and where does that data go?
2. How does your platform handle personally identifiable information and client financial data? What controls prevent that data from being used to train your models or your software team?
3. How long do you retain client data, and what happens to it if we terminate our contract?
4. What compliance certifications or third-party audits has your AI system undergone in the past twelve months?
Firms that get these answers upfront are protecting themselves under the 2024 Regulation S-P amendments, which impose documented vendor oversight obligations on every software tool that touches client data. The compliance deadline for smaller RIAs is June 2026.
Level 2: Architecture Transparency
This is where you find out whether the AI is real, proprietary, or borrowed. A vendor who becomes evasive at this level is signaling that the marketing narrative differs from the technical reality.
5. What AI models power the features you are demonstrating today? Are they proprietary, licensed, or built on a third-party API such as OpenAI or Anthropic?
6. What training data did you use to build or fine-tune the model? Does it include any client data from existing customers?
7. Where does data processing occur? On your infrastructure, on a third-party cloud, or on our servers?
8. How does your system handle hallucinations and bias? What validation exists before an AI output reaches a user?
9. What temperature or confidence settings govern the model’s responses, and can we configure those settings for our compliance environment?
10. Has your AI system ever produced incorrect output that affected client communications or recommendations? How was that identified and resolved?
Level 3: The “Turn It Off” Test
This is the most revealing category. The vendor’s response to these four questions tells you more than any feature demonstration.
11. If you turned off the AI component entirely, what functionality would remain?
12. Can you show us a side-by-side comparison of a workflow completed with the AI active versus without it?
13. What can this system accomplish that a rules-based engine could not?
14. Where is the human in the loop before an AI output reaches a client account or a client communication?
Advisor360’s research found that 93% of advisors want final human review authority over any AI-influenced output.1 If the vendor’s answer to question 14 is vague, that gap between your firm’s expectations and the vendor’s design is something you need to resolve before signing a contract, not after deploying the platform.
Level 4: Evidence and Measurability
A vendor who conflates platform-level metrics with AI-specific metrics is obscuring more than they are revealing. You need data on what the AI does, not what the platform costs or how many users it has.
15. What performance data do you have on the AI component specifically, not the platform as a whole?
16. What is the error rate for the AI’s outputs, and how do you measure it?
17. Has your AI’s performance been validated by a third party? Can you share that assessment?
18. Can you connect us with a reference client who has deployed this AI feature in production for at least twelve months? Not a pilot. Production.
Level 5: Roadmap vs. Reality
These questions are not unreasonable. They are what any serious buyer asks before committing to a multi-year technology contract. The vendor who takes offense is telling you something about how they plan their demonstrations.
19. Of the capabilities demonstrated today, which are in production, which are in beta, and which are planned for future release? Please be specific.
20. What is contractually guaranteed versus aspirational? What happens to our pricing and contract terms if the roadmap features we saw today are delayed by twelve months or more?
The Bottom Line
If a vendor cannot answer these questions clearly and specifically, that inability is itself an answer. It tells you either that the AI is less mature than the demo suggested, or that the sales team is operating ahead of what the product team can actually support.
Neither situation produces a good technology outcome for your firm. The firms that build this evaluation discipline now will make better AI purchasing decisions and carry more internal credibility into the next round of adoption.
