root.skills =========== .. py:module:: root.skills Attributes ---------- .. autoapisummary:: root.skills.ModelName Classes ------- .. autoapisummary:: root.skills.ACalibrateBatchParameters root.skills.ACalibrateBatchResult root.skills.AEvaluator root.skills.APresetEvaluatorRunner root.skills.CalibrateBatchParameters root.skills.CalibrateBatchResult root.skills.Evaluator root.skills.EvaluatorDemonstration root.skills.Evaluators root.skills.InputVariable root.skills.ModelParams root.skills.PresetEvaluatorRunner root.skills.ReferenceVariable root.skills.Versions Module Contents --------------- .. py:class:: ACalibrateBatchParameters(name: str, prompt: str, model: ModelName, pii_filter: bool = False, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_aclient.models.reference_variable_request.ReferenceVariableRequest]]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_aclient.models.input_variable_request.InputVariableRequest]]] = None) .. py:attribute:: input_variables :value: None .. py:attribute:: model .. py:attribute:: name .. py:attribute:: pii_filter :value: False .. py:attribute:: prompt .. py:attribute:: reference_variables :value: None .. py:class:: ACalibrateBatchResult Bases: :py:obj:`pydantic.BaseModel` .. py:attribute:: mae_errors_model :type: Dict[str, float] .. py:attribute:: mae_errors_prompt :type: Dict[str, float] .. py:attribute:: results :type: List[root.generated.openapi_aclient.models.evaluator_calibration_output.EvaluatorCalibrationOutput] .. py:attribute:: rms_errors_model :type: Dict[str, float] .. py:attribute:: rms_errors_prompt :type: Dict[str, float] .. py:class:: AEvaluator Bases: :py:obj:`root.generated.openapi_aclient.models.evaluator.Evaluator` Wrapper for a single Evaluator. For available attributes, please check the (automatically generated) superclass documentation. .. py:method:: arun(response: Optional[str] = None, request: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_aclient.models.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, variables: Optional[dict[str, str]] = None, tags: Optional[List[str]] = None, *, _client: root.generated.openapi_aclient.ApiClient, _request_timeout: Optional[int] = None) -> root.generated.openapi_aclient.models.EvaluatorExecutionResult :async: Asynchronously run the evaluator. :param response: LLM output. :param request: The prompt sent to the LLM. :param contexts: Optional documents passed to RAG evaluators :param functions: Optional function definitions to LLM tool call validation :param expected_output: Optional expected output for the evaluator. :param variables: Optional additional variable mappings for the evaluator. For example, if the evaluator predicate is "evaluate the output based on {subject}: {output}", then variables={"subject": "clarity"}. :param tags: Optional tags to add to the evaluator execution .. py:attribute:: client_context :type: root.utils.ClientContextCallable .. py:class:: APresetEvaluatorRunner(client_context: root.utils.ClientContextCallable, evaluator_id: str, eval_name: str, evaluator_version_id: Optional[str] = None) .. py:attribute:: client_context :type: root.utils.ClientContextCallable .. py:attribute:: evaluator_id .. py:attribute:: evaluator_version_id :value: None .. py:class:: CalibrateBatchParameters(name: str, prompt: str, model: ModelName, pii_filter: bool = False, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_client.models.reference_variable_request.ReferenceVariableRequest]]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_client.models.input_variable_request.InputVariableRequest]]] = None) .. py:attribute:: input_variables :value: None .. py:attribute:: model .. py:attribute:: name .. py:attribute:: pii_filter :value: False .. py:attribute:: prompt .. py:attribute:: reference_variables :value: None .. py:class:: CalibrateBatchResult Bases: :py:obj:`pydantic.BaseModel` .. py:attribute:: mae_errors_model :type: Dict[str, float] .. py:attribute:: mae_errors_prompt :type: Dict[str, float] .. py:attribute:: results :type: List[root.generated.openapi_client.models.evaluator_calibration_output.EvaluatorCalibrationOutput] .. py:attribute:: rms_errors_model :type: Dict[str, float] .. py:attribute:: rms_errors_prompt :type: Dict[str, float] .. py:class:: Evaluator Bases: :py:obj:`root.generated.openapi_aclient.models.evaluator.Evaluator` Wrapper for a single Evaluator. For available attributes, please check the (automatically generated) superclass documentation. .. py:method:: run(response: Optional[str] = None, request: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, variables: Optional[dict[str, str]] = None, tags: Optional[List[str]] = None, *, _client: root.generated.openapi_client.ApiClient, _request_timeout: Optional[int] = None) -> root.generated.openapi_client.models.evaluator_execution_result.EvaluatorExecutionResult Run the evaluator. :param response: LLM output. :param request: The prompt sent to the LLM. :param contexts: Optional documents passed to RAG evaluators :param functions: Optional function definitions to LLM tool call validation :param expected_output: Optional expected output for the evaluator. :param variables: Optional additional variable mappings for the evaluator. For example, if the evaluator predicate is "evaluate the output based on {subject}: {output}", then variables={"subject": "clarity"}. :param tags: Optional tags to add to the evaluator execution .. py:attribute:: client_context :type: root.utils.ClientContextCallable .. py:class:: EvaluatorDemonstration Bases: :py:obj:`pydantic.BaseModel` Evaluator demonstration Demonstrations are used to train an evaluator to adjust its behavior. .. py:attribute:: justification :type: Optional[str] :value: None .. py:attribute:: request :type: Optional[str] :value: None .. py:attribute:: response :type: str .. py:attribute:: score :type: float .. py:class:: Evaluators(client_context: root.utils.ClientContextCallable) Evaluators (sub) API .. note:: The construction of the API instance should be handled by accesing an attribute of a :class:`root.client.RootSignals` instance. .. py:class:: Eval(*args, **kwds) Bases: :py:obj:`enum.Enum` Create a collection of name/value pairs. Example enumeration: >>> class Color(Enum): ... RED = 1 ... BLUE = 2 ... GREEN = 3 Access them by: - attribute access:: >>> Color.RED - value lookup: >>> Color(1) - name lookup: >>> Color['RED'] Enumerations can be iterated over, and know how many members they have: >>> len(Color) 3 >>> list(Color) [, , ] Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. .. py:attribute:: Answer_Correctness :value: 'd4487568-4243-4da8-9c76-adbaf762dbe0' .. py:attribute:: Answer_Relevance :value: '0907d422-e94f-4c9c-a63d-ec0eefd8a903' .. py:attribute:: Answer_Semantic_Similarity :value: 'ff350bce-4b07-4af7-9640-803c9d3c2ff9' .. py:attribute:: Clarity :value: '9976d9f3-7265-4732-b518-d61c2642b14e' .. py:attribute:: Coherence :value: 'e599886c-c338-458f-91b3-5d7eba452618' .. py:attribute:: Completeness :value: 'f0832c32-6beb-4383-a1ea-cdeb883d9044' .. py:attribute:: Compliance_Preview :value: '4613f248-b60e-403a-bcdc-157d1c44194a' .. py:attribute:: Conciseness :value: 'be828d33-158a-4e92-a2eb-f4d96c13f956' .. py:attribute:: Confidentiality :value: '2eaa0a02-47a9-48f7-9b47-66ad257f93eb' .. py:attribute:: Context_Precision :value: '9d1e9a25-7e76-4771-b1e3-40825d7918c5' .. py:attribute:: Context_Recall :value: '8bb60975-5062-4367-9fc6-a920044cba56' .. py:attribute:: Engagingness :value: '64729487-d4a8-42d8-bd9e-72fd8390c134' .. py:attribute:: Faithfulness :value: '901794f9-634c-4852-9e41-7c558f1ff1ab' .. py:attribute:: Faithfulness_Swift :value: 'a3a5e97b-7fcb-441e-92f2-6e59aa473b89' .. py:attribute:: Formality :value: '8ab6cf1a-42b5-4a23-a15c-21372816483d' .. py:attribute:: Harmlessness :value: '379fee0a-4fd1-4942-833b-7d78d78b334d' .. py:attribute:: Helpfulness :value: '88bc92d5-bebf-45e4-9cd1-dfa33309c320' .. py:attribute:: JSON_Content_Accuracy :value: 'b6a9aeff-c888-46d7-9e9c-7cf8cb461762' .. py:attribute:: JSON_Empty_Values_Ratio :value: '03829088-1799-438e-ae30-1db60832e52d' .. py:attribute:: JSON_Property_Completeness :value: 'e5de37f7-d20c-420f-8072-f41dce96ecfc' .. py:attribute:: JSON_Property_Name_Accuracy :value: '740923aa-8ffd-49cc-a95d-14f831243b25' .. py:attribute:: JSON_Property_Type_Accuracy :value: 'eabc6924-1fec-4e96-82ce-c03bf415c885' .. py:attribute:: Non_toxicity :value: 'e296e374-7539-4eb2-a74a-47847dd26fb8' .. py:attribute:: Originality :value: 'e72cb54f-548a-44f9-a6ca-4e14e5ade7f7' .. py:attribute:: Persuasiveness :value: '85bb6a74-f5dd-4130-8dcc-cffdf72327cc' .. py:attribute:: Politeness :value: '2856903a-e48c-4548-b3fe-520fd88c4f25' .. py:attribute:: Precision :value: '767bdd49-5f8c-48ca-8324-dfd6be7f8a79' .. py:attribute:: Quality_of_Writing_Creative :value: '060abfb6-57c9-43b5-9a6d-8a1a9bb853b8' .. py:attribute:: Quality_of_Writing_Professional :value: '059affa9-2d1c-48de-8e97-f81dd3fc3cbe' .. py:attribute:: Relevance :value: 'bd789257-f458-4e9e-8ce9-fa6e86dc3fb9' .. py:attribute:: Safety_for_Children :value: '39a8b5ba-de77-4726-a6b0-621d40b3cdf5' .. py:attribute:: Sentiment_recognition :value: 'e3782c1e-eaf4-4b2d-8d26-53db2160f1fd' .. py:attribute:: Truthfulness :value: '053df10f-b0c7-400b-892e-46ce3aa1e430' .. py:attribute:: Truthfulness_Swift :value: 'c8c65e61-2dc8-4f29-865a-a5e59127d208' .. py:method:: acalibrate(*, name: str, test_dataset_id: Optional[str] = None, test_data: Optional[List[List[str]]] = None, prompt: str, model: ModelName, pii_filter: bool = False, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_aclient.models.reference_variable_request.ReferenceVariableRequest]]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_aclient.models.input_variable_request.InputVariableRequest]]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> List[root.generated.openapi_aclient.models.evaluator_calibration_output.EvaluatorCalibrationOutput] :async: Asynchronously run calibration set for an evaluator definition. See the create evaluator method for more details on the parameters. .. py:method:: acalibrate_batch(*, evaluator_definitions: List[ACalibrateBatchParameters], test_dataset_id: Optional[str] = None, test_data: Optional[List[List[str]]] = None, parallel_requests: int = 1, _request_timeout: Optional[int] = None) -> ACalibrateBatchResult :async: Asynchronously run calibration for a set of prompts and models :param evaluator_definitions: List of evaluator definitions. :param test_dataset_id: ID of the dataset to be used to test the evaluator. :param test_data: Snapshot of data to be used to test the evaluator. :param parallel_requests: Number of parallel requests. Returns a model with the results and errors for each model and prompt. .. py:method:: acalibrate_existing(evaluator_id: str, *, test_dataset_id: Optional[str] = None, test_data: Optional[List[List[str]]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> List[root.generated.openapi_aclient.models.evaluator_calibration_output.EvaluatorCalibrationOutput] :async: Asynchronously run calibration set on an existing evaluator. .. py:method:: acreate(predicate: str = '', *, name: Optional[str] = None, intent: Optional[str] = None, model: Optional[ModelName] = None, fallback_models: Optional[List[ModelName]] = None, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_aclient.models.reference_variable_request.ReferenceVariableRequest]]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_aclient.models.input_variable_request.InputVariableRequest]]] = None, model_params: Optional[Union[ModelParams, root.generated.openapi_aclient.models.ModelParamsRequest]] = None, evaluator_demonstrations: Optional[List[EvaluatorDemonstration]] = None, objective_id: Optional[str] = None, overwrite: bool = False, _client: root.generated.openapi_client.ApiClient, _request_timeout: Optional[int] = None) -> AEvaluator :async: Asynchronously create a new evaluator and return the result :param predicate: The question / predicate that is provided to the semantic quantification layer to :param transform it into a final prompt before being passed to the model: :param name: Name of the evaluator (defaulting to ) :param objective_id: Optional pre-existing objective id to assign to the evaluator. :param intent: The intent of the evaluator (defaulting to name); not available if objective_id is set. :param model: The model to use (defaults to 'root', which means Root Signals default at the time of evaluator creation) :param fallback_models: The fallback models to use in case the primary model fails. :param reference_variables: An optional list of reference variables for the evaluator. :param input_variables: An optional list of input variables for the evaluator. :param model_params: An optional set of additional parameters to the model (e.g., temperature). :param evaluator_demonstrations: An optional list of evaluator demonstrations to guide the evaluator's behavior. :param overwrite: Whether to overwrite an evaluator with the same name if it exists. .. py:method:: adelete(evaluator_id: str, *, _client: root.generated.openapi_aclient.ApiClient) -> None :async: Delete the evaluator. .. py:method:: aget(evaluator_id: str, *, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> AEvaluator :async: Asynchronously get a Evaluator instance by ID. .. py:method:: aget_by_name(name: str, *, _client: root.generated.openapi_client.ApiClient) -> AEvaluator :async: Asynchronously get an evaluator instance by name. Args: name: The evaluator to be fetched. Note this only works for uniquely named evaluators. .. py:method:: alist(search_term: Optional[str] = None, *, limit: int = 100, name: Optional[str] = None, only_root_evaluators: bool = False) -> AsyncIterator[root.generated.openapi_aclient.models.evaluator_list_output.EvaluatorListOutput] :async: Asynchronously iterate through the evaluators. :param search_term: Can be used to limit returned evaluators. :param limit: Number of entries to iterate through at most. :param name: Specific name the returned evaluators must match. :param only_root_evaluators: Returns only Root Signals defined evaluators. .. py:method:: arun(evaluator_id: str, *, request: Optional[str] = None, response: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_aclient.models.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, evaluator_version_id: Optional[str] = None, variables: Optional[dict[str, str]] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> root.generated.openapi_aclient.models.EvaluatorExecutionResult :async: Asynchronously run the evaluator. :param evaluator_id: The ID of the evaluator to run. :param request: The prompt sent to the LLM. :param response: LLM output. :param contexts: Optional documents passed to RAG evaluators. :param functions: Optional function definitions to LLM tool call validation. :param expected_output: Optional expected output for the evaluator. :param evaluator_version_id: Version ID of the evaluator to run. If omitted, the latest version is used. :param variables: Optional additional variable mappings for the evaluator. For example, if the evaluator predicate is "evaluate the output based on {subject}: {output}", then variables={"subject": "clarity"}. :param tags: Optional tags to add to the evaluator execution :param _request_timeout: Optional timeout for the request. .. py:method:: arun_by_name(name: str, *, request: Optional[str] = None, response: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_aclient.models.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, evaluator_version_id: Optional[str] = None, variables: Optional[dict[str, str]] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> root.generated.openapi_aclient.models.EvaluatorExecutionResult :async: Asynchronously run an evaluator by name. :param name: The name of the evaluator to run. :param request: The prompt sent to the LLM. :param response: LLM output. :param contexts: Optional documents passed to RAG evaluators. :param functions: Optional function definitions to LLM tool call validation. :param expected_output: Optional expected output for the evaluator. :param evaluator_version_id: Version ID of the evaluator to run. If omitted, the latest version is used. :param variables: Optional additional variable mappings for the evaluator. For example, if the evaluator predicate is "evaluate the output based on {subject}: {output}", then variables={"subject": "clarity"}. :param tags: Optional tags to add to the evaluator execution :param _request_timeout: Optional timeout for the request. .. py:method:: aupdate(evaluator_id: str, *, change_note: Optional[str] = None, fallback_models: Optional[List[ModelName]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_aclient.models.input_variable_request.InputVariableRequest]]] = None, model: Optional[ModelName] = None, name: Optional[str] = None, predicate: Optional[str] = None, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_aclient.models.reference_variable_request.ReferenceVariableRequest]]] = None, model_params: Optional[Union[ModelParams, root.generated.openapi_aclient.models.ModelParamsRequest]] = None, evaluator_demonstrations: Optional[List[EvaluatorDemonstration]] = None, objective_id: Optional[str] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> AEvaluator :async: Asynchronously update an evaluator and return the result See the create method for more information on the arguments. .. py:method:: calibrate(*, name: str, test_dataset_id: Optional[str] = None, test_data: Optional[List[List[str]]] = None, prompt: str, model: ModelName, pii_filter: bool = False, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_client.models.reference_variable_request.ReferenceVariableRequest]]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_client.models.input_variable_request.InputVariableRequest]]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> List[root.generated.openapi_client.models.evaluator_calibration_output.EvaluatorCalibrationOutput] Run calibration set for an evaluator definition. See the create evaluator method for more details on the parameters. .. py:method:: calibrate_batch(*, evaluator_definitions: List[CalibrateBatchParameters], test_dataset_id: Optional[str] = None, test_data: Optional[List[List[str]]] = None, parallel_requests: int = 1, _request_timeout: Optional[int] = None) -> CalibrateBatchResult Run calibration for a set of prompts and models :param evaluator_definitions: List of evaluator definitions. :param test_dataset_id: ID of the dataset to be used to test the evaluator. :param test_data: Snapshot of data to be used to test the evaluator. :param parallel_requests: Number of parallel requests. Returns a model with the results and errors for each model and prompt. .. py:method:: calibrate_existing(evaluator_id: str, *, test_dataset_id: Optional[str] = None, test_data: Optional[List[List[str]]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> List[root.generated.openapi_client.models.evaluator_calibration_output.EvaluatorCalibrationOutput] Run calibration set on an existing evaluator. .. py:method:: create(predicate: str = '', *, name: Optional[str] = None, intent: Optional[str] = None, model: Optional[ModelName] = None, fallback_models: Optional[List[ModelName]] = None, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_client.models.reference_variable_request.ReferenceVariableRequest]]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_client.models.input_variable_request.InputVariableRequest]]] = None, model_params: Optional[Union[ModelParams, root.generated.openapi_client.models.model_params_request.ModelParamsRequest]] = None, evaluator_demonstrations: Optional[List[EvaluatorDemonstration]] = None, objective_id: Optional[str] = None, overwrite: bool = False, _client: root.generated.openapi_client.ApiClient, _request_timeout: Optional[int] = None) -> Evaluator Create a new evaluator and return the result :param predicate: The question / predicate that is provided to the semantic quantification layer to :param transform it into a final prompt before being passed to the model: :param name: Name of the evaluator (defaulting to ) :param objective_id: Optional pre-existing objective id to assign to the evaluator. :param intent: The intent of the evaluator (defaulting to name); not available if objective_id is set. :param model: The model to use (defaults to 'root', which means Root Signals default at the time of evaluator creation) :param fallback_models: The fallback models to use in case the primary model fails. :param reference_variables: An optional list of reference variables for the evaluator. :param input_variables: An optional list of input variables for the evaluator. :param model_params: An optional set of additional parameters to the model (e.g., temperature). :param evaluator_demonstrations: An optional list of evaluator demonstrations to guide the evaluator's behavior. :param overwrite: Whether to overwrite an evaluator with the same name if it exists. .. py:method:: delete(evaluator_id: str, *, _client: root.generated.openapi_client.ApiClient) -> None Delete the evaluator. .. py:method:: get(evaluator_id: str, *, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> Evaluator Get a Evaluator instance by ID. .. py:method:: get_by_name(name: str, *, _client: root.generated.openapi_client.ApiClient) -> Evaluator Get an evaluator instance by name. Args: name: The evaluator to be fetched. Note this only works for uniquely named evaluators. .. py:method:: list(search_term: Optional[str] = None, *, limit: int = 100, name: Optional[str] = None, only_root_evaluators: bool = False, _client: root.generated.openapi_client.ApiClient) -> Iterator[root.generated.openapi_client.models.evaluator_list_output.EvaluatorListOutput] Iterate through the evaluators. :param search_term: Can be used to limit returned evaluators. :param limit: Number of entries to iterate through at most. :param name: Specific name the returned evaluators must match. :param only_root_evaluators: Returns only Root Signals defined evaluators. .. py:method:: run(evaluator_id: str, *, request: Optional[str] = None, response: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, evaluator_version_id: Optional[str] = None, variables: Optional[dict[str, str]] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> root.generated.openapi_client.models.evaluator_execution_result.EvaluatorExecutionResult Run the evaluator. :param evaluator_id: The ID of the evaluator to run. :param request: The prompt sent to the LLM. :param response: LLM output. :param contexts: Optional documents passed to RAG evaluators. :param functions: Optional function definitions to LLM tool call validation. :param expected_output: Optional expected output for the evaluator. :param evaluator_version_id: Version ID of the evaluator to run. If omitted, the latest version is used. :param variables: Optional additional variable mappings for the evaluator. For example, if the evaluator predicate is "evaluate the output based on {subject}: {output}", then variables={"subject": "clarity"}. :param tags: Optional tags to add to the evaluator execution :param _request_timeout: Optional timeout for the request. .. py:method:: run_by_name(name: str, *, request: Optional[str] = None, response: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, evaluator_version_id: Optional[str] = None, variables: Optional[dict[str, str]] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> root.generated.openapi_client.models.evaluator_execution_result.EvaluatorExecutionResult Run an evaluator by name. :param name: The name of the evaluator to run. :param request: The prompt sent to the LLM. :param response: LLM output. :param contexts: Optional documents passed to RAG evaluators. :param functions: Optional function definitions to LLM tool call validation. :param expected_output: Optional expected output for the evaluator. :param evaluator_version_id: Version ID of the evaluator to run. If omitted, the latest version is used. :param variables: Optional additional variable mappings for the evaluator. For example, if the evaluator predicate is "evaluate the output based on {subject}: {output}", then variables={"subject": "clarity"}. :param tags: Optional tags to add to the evaluator execution :param _request_timeout: Optional timeout for the request. .. py:method:: update(evaluator_id: str, *, change_note: Optional[str] = None, fallback_models: Optional[List[ModelName]] = None, input_variables: Optional[Union[List[InputVariable], List[root.generated.openapi_client.models.input_variable_request.InputVariableRequest]]] = None, model: Optional[ModelName] = None, name: Optional[str] = None, predicate: Optional[str] = None, reference_variables: Optional[Union[List[ReferenceVariable], List[root.generated.openapi_client.models.reference_variable_request.ReferenceVariableRequest]]] = None, model_params: Optional[Union[ModelParams, root.generated.openapi_client.models.model_params_request.ModelParamsRequest]] = None, evaluator_demonstrations: Optional[List[EvaluatorDemonstration]] = None, objective_id: Optional[str] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> Evaluator Update an evaluator and return the result See the create method for more information on the arguments. .. py:attribute:: EvaluatorName .. py:attribute:: client_context .. py:attribute:: versions .. py:class:: InputVariable Bases: :py:obj:`pydantic.BaseModel` Input variable definition. `name` within prompt gets populated with the provided variable. .. py:attribute:: name :type: str .. py:class:: ModelParams Bases: :py:obj:`pydantic.BaseModel` Additional model parameters. All fields are made optional in practice. .. py:attribute:: temperature :type: Optional[float] :value: None .. py:class:: PresetEvaluatorRunner(client_context: root.utils.ClientContextCallable, evaluator_id: str, eval_name: str, evaluator_version_id: Optional[str] = None) .. py:attribute:: client_context :type: root.utils.ClientContextCallable .. py:attribute:: evaluator_id .. py:attribute:: evaluator_version_id :value: None .. py:class:: ReferenceVariable Bases: :py:obj:`pydantic.BaseModel` Reference variable definition. `name` within prompt gets populated with content from `dataset_id`. .. py:attribute:: dataset_id :type: str .. py:attribute:: name :type: str .. py:class:: Versions(client_context: root.utils.ClientContextCallable) Version listing (sub)API Note that this should not be directly instantiated. .. py:method:: alist(evaluator_id: str) -> root.generated.openapi_aclient.models.paginated_evaluator_list.PaginatedEvaluatorList :async: Asynchronously list all versions of a evaluator. .. py:method:: list(evaluator_id: str, *, _client: root.generated.openapi_client.ApiClient) -> root.generated.openapi_client.models.paginated_evaluator_list.PaginatedEvaluatorList List all versions of a evaluator. .. py:attribute:: client_context .. py:data:: ModelName