Forrester conversational AI research is most useful to a mid-sized buyer as a source of questions, not answers. The evaluation criteria analysts use are a good starting checklist. The vendor scores are calibrated for large organisations. This article shows a company of 10 to 250 people how to turn analyst-style criteria into a weighted requirements list, then run a scripted acceptance test on the levelpartner.ai platform so the decision rests on your own conversations.
What analyst evaluations are built for
Analyst firms evaluate conversational AI vendors against a published set of criteria, weighted for a typical buyer, and place them on a chart. The work is thorough. We do not quote any specific report or score, because editions change and the content is licensed.
The typical buyer in those evaluations is a large enterprise. The weights reflect that: scalability, developer tooling, language coverage and global hosting carry real weight. For a 60-person firm, those criteria may be close to irrelevant, while criteria such as "works with my calendar out of the box" barely feature.
Turning criteria into your own requirements
Take the criteria headings from any evaluation you can access, or from vendor comparison guides. For each one, ask two questions: does this matter to us, and how much? Then add the criteria that matter to you but rarely appear in analyst work: time to first live conversation, who on your team can maintain it, and whether it books into your existing calendars.
Weight the list so the total is out of ten. Most mid-sized buyers end up with three or four criteria carrying almost all the weight.
Suppose the firm keeps four criteria. Books into adviser calendars without integration work: weight 3. Records every conversation on the client record: weight 3. Maintainable by the operations manager: weight 2. Handles SMS and WhatsApp as well as chat: weight 2. Scalability and language coverage, prominent in enterprise evaluations, get zero. A tool meeting every criterion would score 10; one meeting only the two weight-3 criteria would score 6.
Before you start the acceptance test
- Your weighted requirements list.
- Thirty real customer messages from your inbox, anonymised, covering your common cases and some awkward ones.
- A note of the correct outcome for each message: answer, book, hand over or refuse.
- A test account with a calendar and your approved FAQs.
How to set it up
- Go to AI Agents > Conversation AI and open Agent List.
- Select the bot you are testing, or create one.
- Open the Bot Training tab.
- Choose the knowledge base holding your approved material.
- Select Exact URL and enter each service page, with https://.
- Click Get Data, then select only the relevant pages.
- Click Train Bot and wait until each URL shows Trained.
- Click + Add Q & A for each approved FAQ.
- Save each Q&A and test variations of the question.
- Type each scoring-sheet message into Test your Bot.
What analyst-led buying gets wrong
The common error is treating a strong position in an analyst chart as evidence the tool will work for you. A position reflects performance against the analyst's weights. If your weights are different, so is the answer. A scripted test with your own messages is a more reliable predictor, and it takes a day.
The second error is reading only the summary. If you do have access to a full report, the vendor profiles describe typical customers. Check whether those customers look like you.
Buyers often run acceptance tests with questions written by the vendor or by the project team. Use messages your customers actually sent, including the badly spelled ones.
The trade-off of skipping enterprise criteria
By setting scalability and developer tooling to zero, you accept that you may outgrow the choice if the business changes shape dramatically. For a company aiming to restart growth without adding staff, the far bigger risk is a tool nobody on the team can run. That is the trade most mid-sized firms should make.
For a view of how a different analyst chart is structured, read conversational AI magic quadrant, which also covers the narrower chatbot category and how to read it. For larger deployments, enterprise conversational AI covers governance.
Time cost of the process
Building the weighted list takes a morning with the people who will run the tool. The acceptance test takes a day per product. After you choose, expect an hour a week of review for the first month.
Troubleshooting
Every product passes the easy messages. Your test set is too clean. Add messages with two questions in one, messages in a hurry and messages that should be refused.
Scores are close between options. Your weights are spread too evenly. Push more weight onto the criteria you would regret most.
The test agent fails because content is missing. That is a content gap, not a product failure. Add the content and rerun, but count the time it took.
Frequently asked questions
Do I need access to the Forrester conversational AI report to use this method?
No. The method works with criteria from any source. A report helps you avoid missing a criterion, but your weights and your test decide the outcome.
How many test messages are enough?
Thirty covers most of a mid-sized firm's common cases. Fewer than twenty tends to miss the awkward ones.
Should the test include voice calls?
Only if calls are part of the job. If they are, script ten calls and test them separately. The AI phone answering guide covers voice set-up.