How do we know whether an AI tool still works after we deploy it?

Only by measuring it, which means deciding before go-live what you will measure.

A workable monitoring plan names the metrics, who reviews them and how often, the threshold that triggers investigation, the accountable owner, and the procedure for withdrawing the tool if it is no longer performing. Without the last item the plan is a reporting exercise rather than a control.

The reason this matters more for AI than for most purchased software is that the product can change underneath you. A vendor model update can alter behavior without any change on your side, which is why update notification terms belong in the contract.