First, break down the Proof of Concept into a manageable business task.
The POC (Proof of Concept) should focus on tasks that have relatively clear rules, predictable call volumes, results that can be verified in the backend, and can be corrected if errors occur. For example, creating a work order after collecting repair data is more suitable for verification than "handling all customer service issues." Before starting, it's important to clearly define the call entry points, required fields, executable actions, prohibited actions, handover conditions, and which entity provides the company API and testing environment. If the scope cannot be clearly defined, the verification process will simply become a subjective assessment.
First, establish a baseline for the existing manual processes, and then discuss how AI can improve them.
Without a baseline, it's impossible to determine whether the POC (Proof of Concept) has improved. At a minimum, record who currently handles the task, how success is defined, common errors, peak wait times, and any data that needs to be re-entered. The baseline doesn't necessarily need to be a polished KPI; a small number of verified actual cases are preferable to assumptions made by the vendor. When comparing, use the same tasks, similar incoming conditions, and consistent definitions of success.
The gold test cases should include scenarios for normal, ambiguous, and failed paths.
Initially, the business, customer service, and system personnel would jointly create the input, expected follow-up questions, necessary fields, allowed actions, and final states. This would then be submitted to the system for repeated testing. The test cases should not rely solely on verbatim readings; they should include variations in user speech, missing data, homophones, background noise, silence, interruptions, API timeouts, and requests for human interaction. When dealing with names, addresses, amounts, or identification information, it is also important to test whether the AI will rephrase and confirm, rather than simply comparing the output to the original text.
Minimum Test Matrix for AI Voice Customer Service Proof of Concept (POC)| Context | What to observe | Verifiable results |
|---|
| Completed successfully | Required fields, question order, and tool calls | CRM/Work Order/Assignment Status must match the call record. |
| The information is unclear or inaccurate. | Did you re-confirm and overwrite the old value? | Only keep the final confirmed data, and do not create duplicate orders. |
| Misunderstanding of speech | Low self-confidence, repetitive questioning, and reliance on manual processes | The error was not directly written into the official system. |
| API timeout or rejection | Reply, Retry, Pending, and Idempotency | Do not announce success prematurely; the final state should be trackable afterward. |
| Requests for human interaction or high-risk activities | The routing protocol establishes a connection with the appropriate network based on the context. | Fields marked as "Confirmed" and unresolved issues have been manually reviewed. |
The metrics should be aligned with the tasks, and not just focused on accuracy rates.
Accurate speech recognition does not guarantee task completion. Errors in the transcript do not necessarily affect the outcome. Acceptance should consider task completion, the accuracy of required fields, successful tool calls, misunderstandings or fallback scenarios, planned handoffs, abnormal upgrades, user abandonment, response delays, and the amount of manual correction. Each factor should have a clear denominator and data source, for example, using "successful test calls within the defined range" as the denominator, and verifying this through phone records, model events, API logs, and the final state of the enterprise system.
Focusing on the acceptance criteria for both conversation quality and system results.| Indicator | Definition | Avoiding Misinterpretations |
|---|
| Task completion rate | The percentage of calls that fall within the defined range and have the correct final status. | A conversation alone does not constitute the completion of a work order. |
| Accuracy of required fields | The degree to which the verified fields match the actual answers. | The average value cannot obscure key fields such as address and amount. |
| Tool executed successfully. | The API call was successful and produced no duplicate or erroneous side effects. | Online transactions cannot be directly classified as either successful or unsuccessful due to the possibility of delays. |
| Misunderstandings and fallback mechanisms | The system is unable to understand or process the frequency of repeated questions. | Distinguish between reasonable questioning and unproductive repetition. |
| Manual handover | Planned transfers, unexpected upgrades, and user-initiated requests | Not all instances of outsourcing are failures; they need to be categorized based on the reasons for the failure. |
The threshold for launching a project must be set according to the estimated cost.
There is no single passing line that applies to all AI phone systems. The costs associated with incorrect operating hours and modifying payment details are entirely different; even a single typo in an address can be more problematic than unnatural phrasing. The approach is to first categorize the errors into three categories: those that can be automatically retried, those that require manual review, and those that cannot be automatically executed. Then, set different thresholds and assign responsibility for each category. If the sample size is insufficient, we can only say that the POC has not yet identified specific issues, and we cannot conclude that the system will definitely meet the standards in a production environment.
The acceptance testing process should also include verifying the system's ability to function correctly after a failure.
Formal services will inevitably encounter issues such as network outages, timeouts, API errors, full call center lines, and vendor maintenance. During the POC, it is crucial to verify how each type of issue is handled, where the data is placed during the outage, who receives notifications, and whether it is safe to retry after the issue is resolved. The front-end and operational back-end must clearly display understandable error messages, without swallowing exceptions and leading customer service to believe that the issue has been resolved.
GoGoCha can serve as evidence in a structured manner, rather than being a general acceptance data.
GoGoCha has publicly demonstrated Falcon's capabilities in AI-powered phone entry, shared order backend, queuing, real-time notifications, and integration with websites, LINE, apps, and back-end systems. However, the case study did not disclose specific metrics such as recognition rate, average latency, connection rate, or cost savings, nor did it provide formal service level agreements (SLAs). Therefore, these figures cannot be used to set benchmarks for other companies. The new proof of concept still requires re-validation using the target company's phone environment, customer language, and task data.
What deliverables should be produced before a Proof of Concept (POC) is ready for release to a full version?
At a minimum, the following should be documented: fixed test cases, detailed results, unresolved risks, system architecture, data flow, permissions, monitoring, manual intervention and recovery procedures. The final version should also include peak load testing, real PBX/SIP testing, redundancy testing, recording policies, and operational permissions. The POC should only indicate that further development is worthwhile, not that the system can be directly deployed and operated without further testing or adjustments.
