Skip to main content

AiEvaluationDefinition

Bill Appleton

Metadata Type: AiEvaluationDefinition

AiEvaluationDefinition is a metadata type in Salesforce that represents the structure and criteria for testing Agentforce agents. This metadata type allows organizations to define an evaluation's purpose, specify the agent being tested, and outline the exact test cases and expected outcomes to ensure their AI implementations behave predictably and accurately.

Overview

The AiEvaluationDefinition metadata type was introduced in API version 63.0 and is a critical component of Salesforce's Testing API for Agentforce. It enables administrators and developers to create comprehensive test definitions in XML format, specifying inputs like natural-language utterances and expected results. This functionality is essential for organizations that need to validate AI agent responses, ensure the correct subagents or actions are triggered, and maintain high-quality AI interactions.

Key Features

  • Structured test cases for evaluating Agentforce agent responses
  • Support for complex inputs including context variables and conversation history
  • Customizable expectation criteria (e.g., expected actions, subagents, and string comparisons)
  • Seamless integration with the Connect REST API and Salesforce CLI for test execution

Structure and Components

An AiEvaluationDefinition typically consists of the following components:

  • Subject: The specific Agentforce agent being tested and its version
  • Test Cases: The individual scenarios being evaluated, with a limit of 1,000 cases per definition
  • Inputs: The data provided to the agent, including the primary utterance and optional conversation history
  • Expectations: The criteria used to grade the agent, such as expected subagents, actions, or specific output strings

Deployment Considerations

When deploying AiEvaluationDefinition metadata, administrators should be aware of several potential issues:

1. Feature Enablement

Ensure that Agentforce is enabled in the target org. If the Agentforce feature is turned off, the AiEvaluationDefinition metadata type will not be available, and deployments will fail.

2. Component Dependencies

Verify that all subagents (GenAiPlugin) and actions (GenAiFunction) referenced in the test case expectations exist in the target org. Missing dependencies will cause the tests to fail during execution or deployment.

3. File Structure

Confirm that your files are properly formatted. AiEvaluationDefinition components must have the .aiEvaluationDefinition suffix and be stored in the designated aiEvaluationDefinitions folder in your DX project.

4. Size Limits

Be cautious of the maximum limits. A single AiEvaluationDefinition component can contain a maximum of 1,000 test cases. Exceeding this limit will result in deployment errors.

5. Compatibility

Ensure your expected value data types match the expectation operator (e.g., using numerical operators like greater_than only for numerical values). Invalid comparison logic can lead to runtime test failures.

Best Practices for Salesforce Administrators

To effectively use and manage AiEvaluationDefinition metadata, Salesforce administrators should follow these best practices:

1. Document Your Test Cases

Maintain clear documentation of all test cases, including the business scenario they validate, expected outcomes, and necessary context variables. This documentation will be invaluable for troubleshooting and refining AI logic.

2. Use Descriptive Names

Choose clear and descriptive names for your AiEvaluationDefinition instances and individual test cases. This makes it easier to understand test results at a glance and simplifies management as your AI footprint grows.

3. Test Multi-Turn Conversations

Instead of only testing single-shot utterance-response pairs, leverage the conversation history input to test utterances within the context of an ongoing conversation. This ensures your agent is robust and can handle complex, multi-turn interactions.

4. Leverage Sandboxes

Always build and run new AiEvaluationDefinition configurations in a sandbox environment before deploying to production. This allows you to identify and resolve any AI hallucination or logic routing issues without exposing end-users to untested agents.

5. Utilize Agentforce DX

Take advantage of Salesforce CLI (sf agent test commands) to generate test specs, create agent tests interactively, and run tests locally. This streamlines the development process and integrates easily into continuous integration (CI) pipelines.

6. Version Control

Use a version control system to track changes to your AiEvaluationDefinition metadata. Because AI models and business logic evolve, maintaining a history of test definitions allows you to track regression and roll back if agent performance degrades.

7. Include Context Variables

Use context variables to simulate different user states or system contexts. This allows you to verify that the agent behaves correctly and adheres to conditional logic based on specific data inputs.

8. Regular Audits

Conduct periodic audits of your test definitions to ensure they remain aligned with current business processes. Remove or update obsolete test cases to keep test execution times efficient and results relevant.

9. Define Precise Expectations

Use specific evaluation criteria, such as verifying the exact subagent or action used, rather than relying solely on outcome text. This ensures the agent is following the correct internal routing path, even if the natural language response varies slightly.

10. Training and Documentation

Provide comprehensive training for developers and administrators who will be building or maintaining Agentforce tests. Ensuring consistent understanding of test specs, YAML configurations, and XML metadata is crucial for a unified AI strategy.

Conclusion

The AiEvaluationDefinition metadata type is a powerful tool for Salesforce administrators and developers to ensure their Agentforce agents deliver accurate and reliable experiences. By understanding its structure, potential deployment issues, and following best practices, teams can effectively leverage this feature to build robust testing frameworks for their AI implementations. As with any AI configuration, continuous testing, careful planning, and ongoing management are key to maximizing the value of Agentforce within your organization.

Powered by Zendesk