root.judges

Classes

AJudge

Async wrapper for a single Judge.

Judge

Wrapper for a single Judge.

Judges

Judges API

Module Contents

class root.judges.AJudge

Bases: root.generated.openapi_aclient.models.judge.Judge

Async wrapper for a single Judge.

For available attributes, please check the (automatically generated) superclass documentation.

async arun(*, response: str, request: str | None = None, contexts: List[str] | None = None, functions: List[root.generated.openapi_aclient.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest] | None = None, expected_output: str | None = None, tags: List[str] | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_aclient.ApiClient) → root.generated.openapi_aclient.models.judge_execution_response.JudgeExecutionResponse

Asynchronously run the judge.

Parameters:

  • response (str) – LLM output to evaluate

  • request (Optional[str]) – The prompt sent to the LLM. Optional.

  • contexts (Optional[List[str]]) – Optional documents passed to RAG evaluators

  • functions (Optional[List[root.generated.openapi_aclient.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]]) – Optional functions to execute

  • expected_output (Optional[str]) – Optional expected output

  • tags (Optional[List[str]]) – Optional tags to add to the judge execution

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_aclient.ApiClient)

Return type:

root.generated.openapi_aclient.models.judge_execution_response.JudgeExecutionResponse

client_context: root.utils.ClientContextCallable
class root.judges.Judge

Bases: root.generated.openapi_client.models.judge.Judge

Wrapper for a single Judge.

For available attributes, please check the (automatically generated) superclass documentation.

run(*, response: str, request: str | None = None, contexts: List[str] | None = None, functions: List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest] | None = None, expected_output: str | None = None, tags: List[str] | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_client.ApiClient) → root.generated.openapi_client.models.judge_execution_response.JudgeExecutionResponse

Run the judge.

Parameters:

  • response (str) – LLM output to evaluate

  • request (Optional[str]) – The prompt sent to the LLM. Optional.

  • contexts (Optional[List[str]]) – Optional documents passed to RAG evaluators

  • functions (Optional[List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]]) – Optional functions to execute

  • expected_output (Optional[str]) – Optional expected output

  • tags (Optional[List[str]]) – Optional tags to add to the judge execution

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_client.ApiClient)

Return type:

root.generated.openapi_client.models.judge_execution_response.JudgeExecutionResponse

client_context: root.utils.ClientContextCallable
class root.judges.Judges(client_context: root.utils.ClientContextCallable)

Judges API

Note

The construction of the API instance should be handled by accessing an attribute of a root.client.RootSignals instance.

Parameters:

client_context (root.utils.ClientContextCallable)

async acreate(*, name: str, intent: str, evaluator_references: List[root.generated.openapi_aclient.models.evaluator_reference_request.EvaluatorReferenceRequest] | None = None, stage: str | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_aclient.ApiClient) → AJudge

Asynchronously create a new judge with a name, intent, and list of evaluators.

Parameters:

  • name (str) – Name for the judge

  • intent (str) – Intent for the judge

  • evaluator_references (Optional[List[root.generated.openapi_aclient.models.evaluator_reference_request.EvaluatorReferenceRequest]]) – List of evaluator references to include in the judge

  • stage (Optional[str]) – Stage for the judge

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_aclient.ApiClient)

Return type:

AJudge

async adelete(judge_id: str, *, _request_timeout: int | None = None, _client: root.generated.openapi_aclient.ApiClient) → None

Asynchronously delete the judge.

Parameters:

  • judge_id (str) – The judge to be deleted.

  • _request_timeout (Optional[int])

  • _client (root.generated.openapi_aclient.ApiClient)

Return type:

None

async agenerate(*, intent: str, visibility: Literal['public', 'unlisted'] = 'unlisted', stage: str | None = None, extra_contexts: Dict[str, str | None] | None = None, strict: bool = False, _request_timeout: int | None = None, _client: root.generated.openapi_aclient.ApiClient) → root.generated.openapi_aclient.models.judge_generator_response.JudgeGeneratorResponse

Asynchronously generate a judge.

Parameters:

  • intent (str) – Describe what you want the judge to build for. Example: I am building a chatbot for ecommerce and I would like to measure the quality of the responses.

  • visibility (Literal['public', 'unlisted']) – Whether the judge should be visible to everyone or only to your organization.

  • stage (Optional[str]) – If the intent is ambiguous, you can specify the stage of the judge. Example: For a chatbot judge, we can specify the stage to be “response generation”.

  • extra_contexts (Optional[Dict[str, str | None]]) – Extra contexts to be passed to the judge. Example: {“domain”: “Ecommerce selling clothing”}, {“audience”: “Women aged 25-35”}

  • strict (bool) – Whether to fail generation if the intent is ambiguous.

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_aclient.ApiClient)

Returns:

Wrapper for the judge id and optionally an error code if the generation failed.

Return type:

root.generated.openapi_aclient.models.judge_generator_response.JudgeGeneratorResponse

async aget(judge_id: str, *, _request_timeout: int | None = None, _client: root.generated.openapi_aclient.ApiClient) → AJudge

Asynchronously get a judge by ID.

Parameters:

  • judge_id (str) – The judge to be fetched.

  • _request_timeout (Optional[int])

  • _client (root.generated.openapi_aclient.ApiClient)

Return type:

AJudge

async alist(*, limit: int = 100) → AsyncIterator[AJudge]

Asynchronously iterate through the judges.

Parameters:

limit (int) – Number of entries to iterate through at most.

Return type:

AsyncIterator[AJudge]

async arun(judge_id: str, *, response: str, request: str | None = None, contexts: List[str] | None = None, functions: List[root.generated.openapi_aclient.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest] | None = None, expected_output: str | None = None, tags: List[str] | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_aclient.ApiClient) → root.generated.openapi_aclient.models.judge_execution_response.JudgeExecutionResponse

Asynchronously run a judge directly by ID.

Parameters:

  • judge_id (str) – ID of the judge to run

  • response (str) – LLM output to evaluate

  • request (Optional[str]) – The prompt sent to the LLM. Optional.

  • contexts (Optional[List[str]]) – Optional documents passed to RAG evaluators

  • functions (Optional[List[root.generated.openapi_aclient.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]]) – Optional functions to execute

  • expected_output (Optional[str]) – Optional expected output

  • tags (Optional[List[str]]) – Optional tags to add to the judge execution

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_aclient.ApiClient)

Return type:

root.generated.openapi_aclient.models.judge_execution_response.JudgeExecutionResponse

async aupdate(judge_id: str, *, name: str | None = None, evaluator_references: List[root.generated.openapi_aclient.models.evaluator_reference_request.EvaluatorReferenceRequest] | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_aclient.ApiClient) → AJudge

Asynchronously update an existing judge.

Parameters:

  • judge_id (str) – The judge to be updated.

  • name (Optional[str]) – New name for the judge

  • evaluator_references (Optional[List[root.generated.openapi_aclient.models.evaluator_reference_request.EvaluatorReferenceRequest]]) – New list of evaluator references

  • _request_timeout (Optional[int])

  • _client (root.generated.openapi_aclient.ApiClient)

Return type:

AJudge

create(*, name: str, intent: str, evaluator_references: List[root.generated.openapi_client.models.evaluator_reference_request.EvaluatorReferenceRequest] | None = None, stage: str | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_client.ApiClient) → Judge

Create a new judge with a name, intent, and list of evaluators.

Parameters:

  • name (str) – Name for the judge

  • intent (str) – Intent for the judge

  • evaluator_references (Optional[List[root.generated.openapi_client.models.evaluator_reference_request.EvaluatorReferenceRequest]]) – List of evaluator references to include in the judge

  • stage (Optional[str]) – Stage for the judge

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_client.ApiClient)

Return type:

Judge

delete(judge_id: str, *, _request_timeout: int | None = None, _client: root.generated.openapi_client.ApiClient) → None

Delete the judge.

Parameters:

  • judge_id (str) – The judge to be deleted.

  • _request_timeout (Optional[int])

  • _client (root.generated.openapi_client.ApiClient)

Return type:

None

generate(*, intent: str, visibility: Literal['public', 'unlisted'] = 'unlisted', stage: str | None = None, extra_contexts: Dict[str, str | None] | None = None, strict: bool = False, _request_timeout: int | None = None, _client: root.generated.openapi_client.ApiClient) → root.generated.openapi_client.models.judge_generator_response.JudgeGeneratorResponse

Generate a judge.

Parameters:

  • intent (str) – Describe what you want the judge to build for. Example: I am building a chatbot for ecommerce and I would like to measure the quality of the responses.

  • visibility (Literal['public', 'unlisted']) – Whether the judge should be visible to everyone or only to your organization.

  • stage (Optional[str]) – If the intent is ambiguous, you can specify the stage of the judge. Example: For a chatbot judge, we can specify the stage to be “response generation”.

  • extra_contexts (Optional[Dict[str, str | None]]) – Extra contexts to be passed to the judge. Example: {“domain”: “Ecommerce selling clothing”}, {“audience”: “Women aged 25-35”}

  • strict (bool) – Whether to fail generation if the intent is ambiguous.

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_client.ApiClient)

Returns:

Wrapper for the judge id and optionally an error code if the generation failed.

Return type:

root.generated.openapi_client.models.judge_generator_response.JudgeGeneratorResponse

get(judge_id: str, *, _request_timeout: int | None = None, _client: root.generated.openapi_client.ApiClient) → Judge

Get a judge by ID.

Parameters:

  • judge_id (str) – The judge to be fetched.

  • _request_timeout (Optional[int])

  • _client (root.generated.openapi_client.ApiClient)

Return type:

Judge

list(*, limit: int = 100, _client: root.generated.openapi_client.ApiClient) → Iterator[Judge]

Iterate through the judges.

Parameters:

  • limit (int) – Number of entries to iterate through at most.

  • _client (root.generated.openapi_client.ApiClient)

Return type:

Iterator[Judge]

run(judge_id: str, *, response: str, request: str | None = None, contexts: List[str] | None = None, functions: List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest] | None = None, expected_output: str | None = None, tags: List[str] | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_client.ApiClient) → root.generated.openapi_client.models.judge_execution_response.JudgeExecutionResponse

Run a judge directly by ID.

Parameters:

  • judge_id (str) – ID of the judge to run

  • response (str) – LLM output to evaluate

  • request (Optional[str]) – The prompt sent to the LLM. Optional.

  • contexts (Optional[List[str]]) – Optional documents passed to RAG evaluators

  • functions (Optional[List[root.generated.openapi_client.models.evaluator_execution_functions_request.EvaluatorExecutionFunctionsRequest]]) – Optional functions to execute

  • expected_output (Optional[str]) – Optional expected output

  • tags (Optional[List[str]]) – Optional tags to add to the judge execution

  • _request_timeout (Optional[int]) – Optional timeout for the request

  • _client (root.generated.openapi_client.ApiClient)

Return type:

root.generated.openapi_client.models.judge_execution_response.JudgeExecutionResponse

update(judge_id: str, *, name: str | None = None, evaluator_references: List[root.generated.openapi_client.models.evaluator_reference_request.EvaluatorReferenceRequest] | None = None, _request_timeout: int | None = None, _client: root.generated.openapi_client.ApiClient) → Judge

Update an existing judge.

Parameters:

  • judge_id (str) – The judge to be updated.

  • name (Optional[str]) – New name for the judge

  • evaluator_references (Optional[List[root.generated.openapi_client.models.evaluator_reference_request.EvaluatorReferenceRequest]]) – New list of evaluator references

  • _request_timeout (Optional[int])

  • _client (root.generated.openapi_client.ApiClient)

Return type:

Judge

client_context