o `+ i4@sTdZddlmZddlmZmZmZmZddlm Z ddl m Z ddl m Z ddlmZddlmZmZdd lmZmZdd lmZmZmZdd lmZdd lmZdd lm Z m!Z!ddl"m#Z$eeeegeeee%ffZ&eeeeeegeeee%ffZ'GdddeZ(Gddde(Z)ee&ee!fZ*ee e+e(fZ,GdddeZ-dS)z!Configuration for run evaluators.)Sequence)AnyCallableOptionalUnion) Embeddings)BaseLanguageModel)BasePromptTemplate) RunEvaluator)EvaluationResultEvaluationResults)ExampleRun) BaseModel ConfigDictField) CRITERIA_TYPE)EmbeddingDistance) EvaluatorTypeStringEvaluator)StringDistancec@s0eZdZUdZeed<deeeffddZ dS) EvalConfigaConfiguration for a given run evaluator. Parameters ---------- evaluator_type : EvaluatorType The type of evaluator to use. Methods ------- get_kwargs() Get the keyword arguments for the evaluator configuration. evaluator_typereturncCs0i}|D]\}}|dks|durq|||<q|S)zGet the keyword arguments for the load_evaluator call. Returns ------- Dict[str, Any] The keyword arguments for the load_evaluator call. rN)selfkwargsfieldvalrrm/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/langchain/smith/evaluation/config.py get_kwargs2s   zEvalConfig.get_kwargsN) __name__ __module__ __qualname____doc__r__annotations__dictstrrr rrrrr!s rcsfeZdZUdZdZeeed< dZeeed< dZ eeed< de ee fffdd Z Z S) SingleKeyEvalConfigzBConfiguration for a run evaluator that only requires a single key.N reference_keyprediction_key input_keyrcs$t}dD]}||dq|S)N)r)r*r+)superr pop)rrkey __class__rrr Qs zSingleKeyEvalConfig.get_kwargs)r!r"r#r$r)rr'r%r*r+r&rr __classcell__rrr/rr(Cs "r(c@seZdZUdZeedZeeee fe d< dZ e ee e d< dZ e eee d< dZe ee d< dZe ee d< dZe ee d < dZe ee d < ed d ZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGddde Z!Gdd d eZ"Gd!d"d"eZ#Gd#d$d$eZ$Gd%d&d&e$Z%dS)' RunEvalConfiga0Configuration for a run evaluation. Parameters ---------- evaluators : List[Union[EvaluatorType, EvalConfig, RunEvaluator, Callable]] Configurations for which evaluators to apply to the dataset run. Each can be the string of an :class:`EvaluatorType `, such as EvaluatorType.QA, the evaluator type string ("qa"), or a configuration for a given evaluator (e.g., :class:`RunEvalConfig.QA `). custom_evaluators : Optional[List[Union[RunEvaluator, StringEvaluator]]] Custom evaluators to apply to the dataset run. reference_key : Optional[str] The key in the dataset run to use as the reference string. If not provided, it will be inferred automatically. prediction_key : Optional[str] The key from the traced run's outputs dictionary to use to represent the prediction. If not provided, it will be inferred automatically. input_key : Optional[str] The key from the traced run's inputs dictionary to use to represent the input. If not provided, it will be inferred automatically. eval_llm : Optional[BaseLanguageModel] The language model to pass to any evaluators that use a language model. )default_factory evaluatorsNcustom_evaluatorsbatch_evaluatorsr)r*r+eval_llmTZarbitrary_types_allowedcfeZdZUdZdZeeed<dZee ed<e j Z e ed< d deede ddffdd ZZS) zRunEvalConfig.Criteriaa"Configuration for a reference-free criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Ncriteriallmrrrc tjdd|i|dSNr:rr,__init__rr:rr/rrr?zRunEvalConfig.Criteria.__init__N)r!r"r#r$r:rrr%r;rrZCRITERIArrr?r1rrr/rCriterias  rCcr9) zRunEvalConfig.LabeledCriteriaa,Configuration for a labeled (with references) criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Nr:r;rrrc r<r=r>r@r/rrr?rAz&RunEvalConfig.LabeledCriteria.__init__rB)r!r"r#r$r:rrr%r;rrZLABELED_CRITERIArrr?r1rrr/rLabeledCriterias  rDc@sJeZdZUdZejZeed<dZe e ed<dZ e e ed<e ddZdS)zRunEvalConfig.EmbeddingDistanceaIConfiguration for an embedding distance evaluator. Parameters ---------- embeddings : Optional[Embeddings] The embeddings to use for computing the distance. distance_metric : Optional[EmbeddingDistanceEnum] The distance metric to use for computing the distance. rN embeddingsdistance_metricTr8)r!r"r#r$rZEMBEDDING_DISTANCErr%rErrrFEmbeddingDistanceEnumr model_configrrrrrs   rc@s>eZdZUdZejZeed<dZe e ed< dZ e ed<dS)zRunEvalConfig.StringDistancezConfiguration for a string distance evaluator. Parameters ---------- distance : Optional[StringDistanceEnum] The string distance metric to use. rNdistanceTnormalize_score) r!r"r#r$rZSTRING_DISTANCErr%rIrStringDistanceEnumrJboolrrrrrs   rc@@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.QAa-Configuration for a QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr;prompt) r!r"r#r$rQArr%r;rrrNr rrrrrOs  rOc@rM)zRunEvalConfig.ContextQA<Configuration for a context-based QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr;rN r!r"r#r$rZ CONTEXT_QArr%r;rrrNr rrrr ContextQA   rRc@rM)zRunEvalConfig.CoTQArPrNr;rNrQrrrrCoTQArSrTc@ eZdZUdZejZeed<dS)zRunEvalConfig.JsonValidityz\Configuration for a json validity evaluator. Parameters ---------- rN)r!r"r#r$rZ JSON_VALIDITYrr%rrrr JsonValidity* rVc@rU)z#RunEvalConfig.JsonEqualityEvaluatorz\Configuration for a json equality evaluator. Parameters ---------- rN)r!r"r#r$rZ JSON_EQUALITYrr%rrrrJsonEqualityEvaluator3rWrXc@sDeZdZUdZejZeed<dZe ed<dZ e ed<dZ e ed<dS)zRunEvalConfig.ExactMatchaxConfiguration for an exact match string evaluator. Parameters ---------- ignore_case : bool Whether to ignore case when comparing strings. ignore_punctuation : bool Whether to ignore punctuation when comparing strings. ignore_numbers : bool Whether to ignore numbers when comparing strings. rF ignore_caseignore_punctuationignore_numbersN) r!r"r#r$rZ EXACT_MATCHrr%rYrLrZr[rrrr ExactMatch<s   r\c@s,eZdZUdZejZeed<dZe ed<dS)zRunEvalConfig.RegexMatchzConfiguration for a regex match string evaluator. Parameters ---------- flags : int The flags to pass to the regex. Example: re.IGNORECASE. rrflagsN) r!r"r#r$rZ REGEX_MATCHrr%r]intrrrr RegexMatchNs r_c seZdZUdZejZeed<dZe e ed<dZ e e ed<dZ e eed<dZe eed<  d de e de eded dffd d ZZS) zRunEvalConfig.ScoreStringaConfiguration for a score string evaluator. This is like the criteria evaluator but it is configured by default to return a score on the scale from 1-10. It is recommended to normalize these scores by setting `normalize_by` to 10. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. normalize_by: Optional[int] = None If you want to normalize the score, the denominator to use. If not provided, the score will be between 1 and 10 (by default). prompt : Optional[BasePromptTemplate] rNr:r; normalize_byrNrrc stjd||d|dS)N)r:r`rr>)rr:r`rr/rrr?usz"RunEvalConfig.ScoreString.__init__)NN)r!r"r#r$rZ SCORE_STRINGrr%r:rrr;rr`floatrNr rr?r1rrr/r ScoreStringZs$ rbc@seZdZUejZeed<dS)z RunEvalConfig.LabeledScoreStringrN)r!r"r#rZLABELED_SCORE_STRINGrr%rrrrLabeledScoreString}s rc)&r!r"r#r$rlistr4rSINGLE_EVAL_CONFIG_TYPECUSTOM_EVALUATOR_TYPEr%r5rr6BATCH_EVALUATOR_LIKEr)r'r*r+r7rrrHr(rCrDrrrOrRrTrVrrXr\r_rbrcrrrrr2]sN #   #r2N).r$collections.abcrtypingrrrrZlangchain_core.embeddingsrZlangchain_core.language_modelsrZlangchain_core.promptsr Z langsmithr Zlangsmith.evaluation.evaluatorr r Zlangsmith.schemasr rZpydanticrrrZ(langchain.evaluation.criteria.eval_chainrZ,langchain.evaluation.embedding_distance.baserrGZlangchain.evaluation.schemarrZ)langchain.evaluation.string_distance.baserrKr&ZRUN_EVALUATOR_LIKErgrr(rfr'rer2rrrrs:           "