How Should Investment Firms Evaluate AI Vendors Before Deployment?

AI vendors can be hard for investment firms to evaluate, especially when product capabilities aren't clearly tied to the firm’s intended use. An AI vendor demonstration can make a compelling case for their product; that doesn’t mean the product will succeed in the complex, regulated, and unique environments in which financial firms operate.

To succeed, firms must evaluate AI tools against the workflow they intend to deploy. They should ask targeted questions about whether an AI tool’s capabilities perform under relevant conditions, whether its handling of firm data is acceptable, and whether the firm can retain operational control as conditions change, including changes to the tools themselves.

Firms should ask targeted questions when deciding where an external capability belongs in the business. They also need to determine how much they are willing to rely on an AI tool, who will operate it, and what information they will allow it to process. This article outlines how investment firms can successfully evaluate vendors so they can deploy their tools successfully.

Why is a demonstration only the starting point?

A demonstration shows what a product can do under particular conditions. Those conditions may differ from the firm’s daily work. An AI research assistant vendor may demonstrate the product’s capabilities using a well-structured public filing, while analysts routinely work with documents whose terminology or formatting is less consistent.

Those differences impact what conclusions a buyer can draw about the success of the tool. For example, while an AI-generated summary is useful evidence of a tool’s capabilities, it does not establish that the tool consistently preserves material facts or appropriately handles ambiguous source information. Nor does it show how much effort an analyst must spend checking the output before using it.

The evaluation should therefore begin with the work the firm wants to improve. If analysts already spend significant time reviewing filings, the relevant question is whether the assistant reduces that burden without creating additional, potentially more complicated tasks for verification.

Firms must align each AI tool capability with one of their own defined business needs before they can make a confident purchasing decision. They should also list any remaining uncertainties; doing so will help them test against those uncertainties before approving broader use.

What is the firm actually buying?

Before comparing providers, an investment firm should define the role it expects the AI capability to play. This way, product selection and service responsibility are part of the same discussion.

For a research assistant, an initial role might be straightforward: summarize approved public documents for analyst review. Analysts remain responsible for interpreting the information; the tool does not distribute its output externally or initiate investment actions.

That boundary gives the evaluation a practical reference point. The firm can assess the vendor based on its ability to support this workflow; they do not need to measure every capability in the provider’s product roadmap.

The International Organization of Securities Commissions’ guidance on AI and machine learning addresses this operating relationship. Its September 2021 report calls for firms to clearly understand their reliance on third-party providers and to establish agreements that clarify outsourced functions and provider responsibilities. While it does not impose a universal rule directly on investment firms, the report offers nonbinding guidance for securities regulators to consider within their own frameworks.

Once the firm defines what it is buying, it can assess whether the provider can support that role on acceptable terms.

Three tests before deployment

Financial firms can conduct a useful evaluation without it being exhaustive. Above all, it needs to resolve the uncertainties most relevant to the deployment they have in mind. Three tests can serve as a good starting point.

1. Does it perform in the intended workflow?

Firms should establish performance criteria before they begin evaluating a product. Using the AI research assistant example, acceptable performance might mean that summaries will both faithfully represent material facts and direct analysts back to the relevant source passages. Firms should avoid potential pitfalls, like favoring one tool over another simply because it is faster or its writing style is better.

The UK Information Commissioner’s Office takes a similar approach in its AI procurement toolkit. It recommends deciding the acceptable level of accuracy before procurement and conducting due diligence to confirm that the system meets it. The toolkit concerns UK personal-data protection; its procurement reasoning is useful more broadly, but its legal expectations should not be treated as applying universally to US investment firms.

A limited pilot can help the firm evaluate performance on work representative of the intended workflow. The documents should reflect the intended scope, including examples that expose the tool’s limitations. Testing only the easiest material may leave the firm’s decision makers with little understanding of how the product behaves when an analyst needs serious help.

The pilot should also examine the human work around the system. If analysts must repeatedly reconstruct a summary from the original filing, the apparent time savings during routine use may not translate into real business results. On the other hand, a tool with modest capabilities during routine tasks may still offer value if its limitations are clear and its output is easy to verify.

2. Are the data boundaries acceptable?

Firms must evaluate how new AI tools handle data and determine which data each tool is appropriate to manage. A product can perform well and still be unsuitable for the information the firm intends to provide.

Consider the difference between summarizing public filings and processing internal research. The second use introduces confidential data to the third-party AI system, and therefore to the AI tool provider. The firm needs a dependable account of how the information will be handled before considering adoption.

The firm must also determine whether the provider retains submitted information or may use it to improve or train models. It should establish whether other providers may receive or process that data. Those answers should be specific to the product configuration and contractual terms the firm will actually use.

Firms can still proceed with tests before all these details are clear. For example, a firm can limit the pilot to approved public material while completing its evaluation. That way, they retain the opportunity to learn about tools even as they explore these areas of concern. But they should ultimately draw clear lines about which types of data are appropriate for each tool they consider.

3. Can the firm retain operational control?

A firm has operational control when it can govern its use of a service and respond when the service behaves unexpectedly, even without assuming complete control of the underlying technology. Control becomes more important as a tool gains access or authority. An assistant that retrieves approved documents may require more limited permissions than a tool that modifies records or sends information outside the firm.

Any expansion should prompt a review of what a firm’s team members can observe and interrupt. While the technology provider remains responsible for the service commitments it has accepted, the investment firm must determine what it can control and how that impacts its desired use cases for the tool.

What happens when the service changes or fails?

Software evolves, and providers regularly push updates to clients. Each firm needs a way to learn about relevant changes and decide whether they impact their existing use cases. Firms should discuss change-notification processes at the start of the provider relationship to reduce the risk of surprises or disruptions later.

Firms should also consider interruption and exit scenarios when evaluating a tool. For example, analysts should be able to continue essential work if the tool becomes unavailable. If the relationship ends, the firm should be able to have its information returned or deleted by the provider, and be able to revoke the provider’s access to future data.

Firms should evaluate whether acceptable operating conditions can be maintained after adoption, including during foreseeable disruptions. The relationship will only succeed if it involves clear communication and a workable plan for responding to material change.

Start each deployment decision with finding the right fit

Product capability and operational suitability are two different considerations within an AI vendor evaluation. A firm can recognize a tool’s promise while concluding that its initial use should remain narrow, or that it should not use the tool at all.

A careful AI vendor evaluation gives leaders a basis for deciding the right scope for a new tool, or whether a new tool is appropriate at all. This process of vendor due diligence can become a practical foundation for adoption.

How Option One Technologies can help

Option One Technologies helps investment firms assess and strengthen the environments in which emerging AI capabilities will operate. Our consulting and implementation support helps firms determine the deployments that are right for them. Our cloud services, cybersecurity, and managed IT help firms establish appropriate operating boundaries for those deployments. Contact one of our experts to learn more.

Executive FAQs

How should an investment firm proceed after an AI vendor’s demonstration?

A demonstration is only a starting point. A successful demonstration supports further evaluation: once it’s over, the firm should test representative work against acceptance criteria it has defined beforehand. It should also understand the human review required to use the output responsibly.

Should an investment firm run a pilot before deploying an AI tool?

A limited pilot can help an investment firm establish suitability, particularly when a product’s performance depends on the firm’s documents or workflows. The pilot should have a defined purpose and an approved data scope. Firms should be cautious: pilots can quickly and informally expand into a more consequential use before they have formally reviewed the evidence and controls for those use cases.

Who should own AI vendor evaluation?

Clear ownership prevents product enthusiasm from becoming a substitute for accountable decision-making. A named business owner should oversee the use case connected to the deployment decision, with support from technology and relevant risk stakeholders. As such, the product evaluation should combine evidence from the proposed workflow with an understanding of the provider relationship.