Handling Unintended Queries with AI Builder in Copilot Studio

Intro:

The use case was to build a Copilot chatbot that operates on a knowledge base managed in SharePoint. This knowledge base included a combination of Standard Operating Procedures (SoPs), best practices, and how-to guides, all subject to periodic review and managed by a specific team.

With the latest version of Copilot Studio, the aim was to utilize its advanced search capabilities, including custom search queries and semantic indexing. The legacy version of the bot relied on classic orchestration, where the agent responded to user queries by triggering predefined topics based on matching trigger phrases. This structured and rule-based system ensured predictable interactions but resulted in a suboptimal user experience due to its limited adaptability to dynamic and complex queries. With this in mind, the idea was to explore the Generative orchestration.

Learnings from Switching to Generative Orchestration:

Transitioning from classic to generative orchestration in Copilot Studio brought several valuable insights:

Generative orchestration proved to be incredibly powerful, enabling the agent to respond appropriately based on instructions. However, ensuring accurate responses required distinguishing between correct usage, unintended use, and out-of-syllabus prompts. This process involved using prompt tuning to customize responses and effectively manage queries that fell outside the knowledge base.

Happy Path:

When the answer is in the knowledge base, the orchestration engine performs a search and retrieves the content, providing citations. Using instruction as a prompt tuner helps tailor the response.

Unhappy Path:

Incorrect Usage: If the user’s intent does not align with the use case, capturing this is crucial. For example, in career coaching and goal setting, if a user asks, "How can I communicate that Mark has not been a good performer in the team?", the agent should respond with, "Does not answer individual or specific situation-related questions, please consult the appropriate personnel." This is guided by clear instructions to avoid answering questions related to specific individuals.

Out-of-Syllabus Queries: Distinguishing between unintended use and out-of-syllabus prompts is essential. If a query falls outside the knowledge source, it should be routed appropriately, ensuring the agent handles it effectively.

The goal was to differentiate between various happy paths and respond to the user appropriately, ensuring a seamless and accurate interaction.

Empowering Orchestration:

For both unhappy paths, the AI orchestration engine will search the knowledge base and then route to the system topic "Conversational boosting" when it encounters an unknown intent. This is where leveraging AI Builder's "Bring Your Prompt" feature becomes particularly useful.

The AI prompt built on GPT 40-mini is an instruction explaining the person as "AI communication specialist" to handle user inquiries while prioritizing privacy and confidentiality. The input is the user prompt on the agent and the AI model would simply classify the outcome as personal question or not .

This approach helps maintain respectful and informative interactions, especially when dealing with sensitive topics.

The idea was to use this classifier on the conversational booster topic and leverage the model to determine appropriate response to the user query.

Combining the AI Prompt Builder with generative orchestration provides a powerful solution for managing agent responses. While good instructions offer general control over interactions, the AI Prompt Builder enriches responses, especially in edge cases. This approach ensures that the agent can handle a wide range of queries effectively, maintaining accuracy and relevance even in complex or unexpected situations.

By leveraging these advanced tools, you can achieve a more adaptable and responsive chatbot, enhancing the overall user experience.