root.judges =========== .. py:module:: root.judges Classes ------- .. autoapisummary:: root.judges.AJudge root.judges.Judge root.judges.Judges Module Contents --------------- .. py:class:: AJudge Bases: :py:obj:`root.generated.openapi_aclient.models.judge.Judge` Async wrapper for a single Judge. For available attributes, please check the (automatically generated) superclass documentation. .. py:method:: arun(*, response: str, request: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_aclient.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> root.generated.openapi_aclient.models.judge_execution_response.JudgeExecutionResponse :async: Asynchronously run the judge. :param response: LLM output to evaluate :param request: The prompt sent to the LLM. Optional. :param contexts: Optional documents passed to RAG evaluators :param functions: Optional functions to execute :param expected_output: Optional expected output :param tags: Optional tags to add to the judge execution :param _request_timeout: Optional timeout for the request .. py:attribute:: client_context :type: root.utils.ClientContextCallable .. py:class:: Judge Bases: :py:obj:`root.generated.openapi_client.models.judge.Judge` Wrapper for a single Judge. For available attributes, please check the (automatically generated) superclass documentation. .. py:method:: run(*, response: str, request: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> root.generated.openapi_client.models.judge_execution_response.JudgeExecutionResponse Run the judge. :param response: LLM output to evaluate :param request: The prompt sent to the LLM. Optional. :param contexts: Optional documents passed to RAG evaluators :param functions: Optional functions to execute :param expected_output: Optional expected output :param tags: Optional tags to add to the judge execution :param _request_timeout: Optional timeout for the request .. py:attribute:: client_context :type: root.utils.ClientContextCallable .. py:class:: Judges(client_context: root.utils.ClientContextCallable) Judges API .. note:: The construction of the API instance should be handled by accessing an attribute of a :class:`root.client.RootSignals` instance. .. py:method:: acreate(*, name: str, intent: str, evaluator_references: Optional[List[root.generated.openapi_aclient.models.evaluator_reference_request.EvaluatorReferenceRequest]] = None, stage: Optional[str] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> AJudge :async: Asynchronously create a new judge with a name, intent, and list of evaluators. :param name: Name for the judge :param intent: Intent for the judge :param evaluator_references: List of evaluator references to include in the judge :param stage: Stage for the judge :param _request_timeout: Optional timeout for the request .. py:method:: adelete(judge_id: str, *, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> None :async: Asynchronously delete the judge. :param judge_id: The judge to be deleted. .. py:method:: agenerate(*, intent: str, visibility: Literal['public', 'unlisted'] = 'unlisted', stage: Optional[str] = None, extra_contexts: Optional[Dict[str, str | None]] = None, strict: bool = False, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> root.generated.openapi_aclient.models.judge_generator_response.JudgeGeneratorResponse :async: Asynchronously generate a judge. :param intent: Describe what you want the judge to build for. Example: I am building a chatbot for ecommerce and I would like to measure the quality of the responses. :param visibility: Whether the judge should be visible to everyone or only to your organization. :param stage: If the intent is ambiguous, you can specify the stage of the judge. Example: For a chatbot judge, we can specify the stage to be "response generation". :param extra_contexts: Extra contexts to be passed to the judge. Example: {"domain": "Ecommerce selling clothing"}, {"audience": "Women aged 25-35"} :param strict: Whether to fail generation if the intent is ambiguous. :param _request_timeout: Optional timeout for the request :returns: Wrapper for the judge id and optionally an error code if the generation failed. .. py:method:: aget(judge_id: str, *, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> AJudge :async: Asynchronously get a judge by ID. :param judge_id: The judge to be fetched. .. py:method:: alist(*, limit: int = 100) -> AsyncIterator[AJudge] :async: Asynchronously iterate through the judges. :param limit: Number of entries to iterate through at most. .. py:method:: arun(judge_id: str, *, response: str, request: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_aclient.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> root.generated.openapi_aclient.models.judge_execution_response.JudgeExecutionResponse :async: Asynchronously run a judge directly by ID. :param judge_id: ID of the judge to run :param response: LLM output to evaluate :param request: The prompt sent to the LLM. Optional. :param contexts: Optional documents passed to RAG evaluators :param functions: Optional functions to execute :param expected_output: Optional expected output :param tags: Optional tags to add to the judge execution :param _request_timeout: Optional timeout for the request .. py:method:: aupdate(judge_id: str, *, name: Optional[str] = None, evaluator_references: Optional[List[root.generated.openapi_aclient.models.evaluator_reference_request.EvaluatorReferenceRequest]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_aclient.ApiClient) -> AJudge :async: Asynchronously update an existing judge. :param judge_id: The judge to be updated. :param name: New name for the judge :param evaluator_references: New list of evaluator references .. py:method:: create(*, name: str, intent: str, evaluator_references: Optional[List[root.generated.openapi_client.models.evaluator_reference_request.EvaluatorReferenceRequest]] = None, stage: Optional[str] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> Judge Create a new judge with a name, intent, and list of evaluators. :param name: Name for the judge :param intent: Intent for the judge :param evaluator_references: List of evaluator references to include in the judge :param stage: Stage for the judge :param _request_timeout: Optional timeout for the request .. py:method:: delete(judge_id: str, *, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> None Delete the judge. :param judge_id: The judge to be deleted. .. py:method:: generate(*, intent: str, visibility: Literal['public', 'unlisted'] = 'unlisted', stage: Optional[str] = None, extra_contexts: Optional[Dict[str, str | None]] = None, strict: bool = False, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> root.generated.openapi_client.models.judge_generator_response.JudgeGeneratorResponse Generate a judge. :param intent: Describe what you want the judge to build for. Example: I am building a chatbot for ecommerce and I would like to measure the quality of the responses. :param visibility: Whether the judge should be visible to everyone or only to your organization. :param stage: If the intent is ambiguous, you can specify the stage of the judge. Example: For a chatbot judge, we can specify the stage to be "response generation". :param extra_contexts: Extra contexts to be passed to the judge. Example: {"domain": "Ecommerce selling clothing"}, {"audience": "Women aged 25-35"} :param strict: Whether to fail generation if the intent is ambiguous. :param _request_timeout: Optional timeout for the request :returns: Wrapper for the judge id and optionally an error code if the generation failed. .. py:method:: get(judge_id: str, *, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> Judge Get a judge by ID. :param judge_id: The judge to be fetched. .. py:method:: list(*, limit: int = 100, _client: root.generated.openapi_client.ApiClient) -> Iterator[Judge] Iterate through the judges. :param limit: Number of entries to iterate through at most. .. py:method:: run(judge_id: str, *, response: str, request: Optional[str] = None, contexts: Optional[List[str]] = None, functions: Optional[List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]] = None, expected_output: Optional[str] = None, tags: Optional[List[str]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> root.generated.openapi_client.models.judge_execution_response.JudgeExecutionResponse Run a judge directly by ID. :param judge_id: ID of the judge to run :param response: LLM output to evaluate :param request: The prompt sent to the LLM. Optional. :param contexts: Optional documents passed to RAG evaluators :param functions: Optional functions to execute :param expected_output: Optional expected output :param tags: Optional tags to add to the judge execution :param _request_timeout: Optional timeout for the request .. py:method:: update(judge_id: str, *, name: Optional[str] = None, evaluator_references: Optional[List[root.generated.openapi_client.models.evaluator_reference_request.EvaluatorReferenceRequest]] = None, _request_timeout: Optional[int] = None, _client: root.generated.openapi_client.ApiClient) -> Judge Update an existing judge. :param judge_id: The judge to be updated. :param name: New name for the judge :param evaluator_references: New list of evaluator references .. py:attribute:: client_context