o `+ iYH@sdZddlmZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZdd lmZdd lmZeeZGd ddee ZGdddeZGdddZGdddeeZ GdddeeZ!GdddeeZ"dS)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Sequence)Enum)AnyOptionalUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)Chainc@seZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZdS) EvaluatorTypezThe types of the evaluators.ZqaZcot_qaZ context_qaZpairwise_stringZ score_stringZlabeled_pairwise_stringZlabeled_score_stringZ trajectorycriteriaZlabeled_criteriaZstring_distanceZ exact_matchZ regex_matchZpairwise_string_distanceZembedding_distanceZpairwise_embedding_distanceZ json_validityZ json_equalityZjson_edit_distanceZjson_schema_validationN)__name__ __module__ __qualname____doc__ZQAZCOT_QAZ CONTEXT_QAZPAIRWISE_STRINGZ SCORE_STRINGZLABELED_PAIRWISE_STRINGZLABELED_SCORE_STRINGZAGENT_TRAJECTORYZCRITERIAZLABELED_CRITERIAZSTRING_DISTANCEZ EXACT_MATCHZ REGEX_MATCHZPAIRWISE_STRING_DISTANCEZEMBEDDING_DISTANCEZPAIRWISE_EMBEDDING_DISTANCEZ JSON_VALIDITYZ JSON_EQUALITYZJSON_EDIT_DISTANCEZJSON_SCHEMA_VALIDATIONrrg/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/langchain/evaluation/schema.pyrsTrc@s"eZdZdZeed ddZd S) LLMEvalChainz,A base class for evaluators that use an LLM.llmr kwargsrreturncKdS)z#Create a new evaluator from an LLM.Nr)clsrrrrrfrom_llmNzLLMEvalChain.from_llmN)rr rrrr)rrrr classmethodrrrrrrrKs rc@sXeZdZdZedddZedddZedd d Zedd d Z ddddZ d S)_EvalArgsMixinz(Mixin for checking evaluation arguments.rboolcCrz2Whether this evaluator requires a reference label.Frselfrrrrequires_referenceWz!_EvalArgsMixin.requires_referencecCr)0Whether this evaluator requires an input string.Frr#rrrrequires_input\r&z_EvalArgsMixin.requires_inputstrcCd|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__rr#rrr_skip_input_warningasz"_EvalArgsMixin._skip_input_warningcCr*)z*Warning to show when reference is ignored.zIgnoring reference in r+r,r#rrr_skip_reference_warningfsz&_EvalArgsMixin._skip_reference_warningN reference Optional[str]input_NonecCs|jr|dur|jjd}t||dur |js t|jdd|jr2|dur2|jjd}t||durB|jsDt|jdddSdSdS)aCheck if the evaluation arguments are valid. Args: reference (Optional[str], optional): The reference label. input_ (Optional[str], optional): The input string. Raises: ValueError: If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided. Nz requires an input string.) stacklevelz requires a reference string.)r(r-r ValueErrorr r.r%r/)r$r0r2msgrrr_check_evaluation_argsmsz%_EvalArgsMixin._check_evaluation_argsrr!rr))NN)r0r1r2r1rr3) rrrrpropertyr%r(r.r/r8rrrrr Ts    r c@sxeZdZdZedddZedddZed d d dddZd d d dddZ d d d dddZ d d d dddZ d S) StringEvaluatorzcGrade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.rr)cCs|jjS)zThe name of the evaluation.r,r#rrrevaluation_nameszStringEvaluator.evaluation_namer!cCrr"rr#rrrr%r&z"StringEvaluator.requires_referenceNr0input predictionUnion[str, Any]r0Optional[Union[str, Any]]r?rrdictcKr)a:Evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrr$r@r0r?rrrr_evaluate_stringsrz!StringEvaluator._evaluate_stringscs$td|jf|||d|IdHS)aIAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr@r0r?)r rErDrrr_aevaluate_stringss z"StringEvaluator._aevaluate_stringsr1cKs&|j||d|jd|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r0r2rFNr)r8rErDrrrevaluate_stringssz StringEvaluator.evaluate_stringscs.|j||d|jd|||d|IdHS)a Asynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rHrFNr)r8rGrDrrraevaluate_stringss z!StringEvaluator.aevaluate_stringsr:r9) r@rAr0rBr?rBrrrrC) r@r)r0r1r?r1rrrrC) rrrrr;r=r%rrErGrIrJrrrrr<s&  #r<c@s\eZdZdZeddddddZddddddZddddddZddddddZdS)PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).Nr>r@r) prediction_br0r1r?rrrrCcKr)1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nrr$r@rLr0r?rrrr_evaluate_string_pairsrz.PairwiseStringEvaluator._evaluate_string_pairscs&td|jf||||d|IdHS)@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr@rLr0r?)r rOrNrrr_aevaluate_string_pairss z/PairwiseStringEvaluator._aevaluate_string_pairscKs(|j||d|jd||||d|S)rMrHrQNr)r8rOrNrrrevaluate_string_pairs8sz-PairwiseStringEvaluator.evaluate_string_pairscs0|j||d|jd||||d|IdHS)rPrHrQNr)r8rRrNrrraevaluate_string_pairsUs z.PairwiseStringEvaluator.aevaluate_string_pairs) r@r)rLr)r0r1r?r1rrrrC) rrrrrrOrRrSrTrrrrrKs#"rKc@sbeZdZdZedddZedddddZdddddZdddddZ dddddZ dS)AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.rr!cCr)r'Trr#rrrr(vr&z'AgentTrajectoryEvaluator.requires_inputN)r0r@r)agent_trajectory!Sequence[tuple[AgentAction, str]]r?r0r1rrrCcKr)Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nrr$r@rVr?r0rrrr_evaluate_agent_trajectory{rz3AgentTrajectoryEvaluator._evaluate_agent_trajectorycs&td|jf||||d|IdHS)Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)r@rVr0r?)r rZrYrrr_aevaluate_agent_trajectorys z4AgentTrajectoryEvaluator._aevaluate_agent_trajectorycKs(|j||d|jd||||d|S)rXrHr@r?rVr0Nr)r8rZrYrrrevaluate_agent_trajectorysz2AgentTrajectoryEvaluator.evaluate_agent_trajectorycs0|j||d|jd||||d|IdHS)r[rHr]Nr)r8r\rYrrraevaluate_agent_trajectorys z3AgentTrajectoryEvaluator.aevaluate_agent_trajectoryr9) r@r)rVrWr?r)r0r1rrrrC) rrrrr;r(rrZr\r^r_rrrrrUss %$rU)#r __future__rloggingabcrrcollections.abcrenumrtypingrrr warningsr Zlangchain_core.agentsr Zlangchain_core.language_modelsr Zlangchain_core.runnables.configr Zlangchain.chains.baser getLoggerrloggerr)rrr r<rKrUrrrrs&         6 3zr