o `+ i@sVUdZddlmZddlmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZmZdd lmZmZdd lmZdd lmZmZdd lmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,ddl-m.Z.m/Z/de0de1e2fddZ3ie'j4e#e'j5e"e'j6e!e'j7ee'j8e,e'j9ee'j:e+e'j;e e'je/e'j?e.e'j@ee'jAee'jBee'jCee'jDee'jEee'jFe%e'jGeiZHe2e'eeIe(eIe eIe)ffeJd<ddde'deededee e)ffddZKddd d!ee'deed"ee2dede1ee e)ff d#d$ZLdS)%z Loading datasets and evaluators.)Sequence)AnyOptionalUnion)BaseLanguageModel)Chain)TrajectoryEvalChain)PairwiseStringEvalChain)LabeledPairwiseStringEvalChain)CriteriaEvalChainLabeledCriteriaEvalChain)EmbeddingDistanceEvalChain"PairwiseEmbeddingDistanceEvalChain)ExactMatchStringEvaluator)JsonEqualityEvaluatorJsonValidityEvaluator)JsonEditDistanceEvaluator)JsonSchemaEvaluator)ContextQAEvalChainCotQAEvalChain QAEvalChain)RegexMatchStringEvaluator) EvaluatorType LLMEvalChainStringEvaluator)LabeledScoreStringEvalChainScoreStringEvalChain)PairwiseStringDistanceEvalChainStringDistanceEvalChainurireturnc CsRzddlm}Wnty}zd}t||d}~ww|d|}t|dS)aLoad a dataset from the `LangChainDatasets on HuggingFace `_. Args: uri: The uri of the dataset to load. Returns: A list of dictionaries, each representing a row in the dataset. **Prerequisites** .. code-block:: shell pip install datasets Examples -------- .. code-block:: python from langchain.evaluation import load_dataset ds = load_dataset("llm-math") r) load_datasetzXload_dataset requires the `datasets` package. Please install with `pip install datasets`NzLangChainDatasets/train)Zdatasetsr! ImportErrorlist)rr!emsgZdatasetr'h/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/langchain/evaluation/loading.pyr!(s  r!_EVALUATOR_MAPN)llm evaluatorr*kwargsc Ks|tvrd|dtt}t|t|}t|trxz9zddlm}Wn%tyKzddl m}WntyH}zd}t||d}~wwYnw|pT|dddd }Wnt ym}z d |d }t||d}~ww|j d d |i|S|d i|S)a<Load the requested evaluation chain specified by a string. Parameters ---------- evaluator : EvaluatorType The type of evaluator to load. llm : BaseLanguageModel, optional The language model to use for evaluation, by default None **kwargs : Any Additional keyword arguments to pass to the evaluator. Returns ------- Chain The loaded evaluation chain. Examples -------- >>> from langchain.evaluation import load_evaluator, EvaluatorType >>> evaluator = load_evaluator(EvaluatorType.QA) zUnknown evaluator type: z Valid types are: r) ChatOpenAIzCould not import langchain_openai or fallback onto langchain_community. Please install langchain_openai or specify a language model explicitly. It's recommended to install langchain_openai AND specify a language model explicitly.Nzgpt-4*)modelseedZ temperaturezEvaluation with the z requires a language model to function. Failed to create the default 'gpt-4' model. Please manually provide an evaluation LLM or check your openai credentials.r*r') r)r$keys ValueError issubclassrZlangchain_openair-r#Z&langchain_community.chat_models.openai ExceptionZfrom_llm)r+r*r,r&Z evaluator_clsr-r%r'r'r(load_evaluatorfs>       r5)r*config evaluatorsr6cKsHg}|D]}|r||ini}|t|fd|ii||q|S)aeLoad evaluators specified by a list of evaluator types. Parameters ---------- evaluators : Sequence[EvaluatorType] The list of evaluator types to load. llm : BaseLanguageModel, optional The language model to use for evaluation, if none is provided, a default ChatOpenAI gpt-4 model will be used. config : dict, optional A dictionary mapping evaluator types to additional keyword arguments, by default None **kwargs : Any Additional keyword arguments to pass to all evaluators. Returns ------- List[Chain] The loaded evaluators. Examples -------- >>> from langchain.evaluation import load_evaluators, EvaluatorType >>> evaluators = [EvaluatorType.QA, EvaluatorType.CRITERIA] >>> loaded_evaluators = load_evaluators(evaluators, criteria="helpfulness") r*)getappendr5)r7r*r6r,Zloadedr+_kwargsr'r'r(load_evaluatorss !$r;)M__doc__collections.abcrtypingrrrZlangchain_core.language_modelsrZlangchain.chains.baserZ1langchain.evaluation.agents.trajectory_eval_chainrZlangchain.evaluation.comparisonr Z*langchain.evaluation.comparison.eval_chainr Z(langchain.evaluation.criteria.eval_chainr r Z,langchain.evaluation.embedding_distance.baser rZ%langchain.evaluation.exact_match.baserZ!langchain.evaluation.parsing.baserrZ*langchain.evaluation.parsing.json_distancerZ(langchain.evaluation.parsing.json_schemarZlangchain.evaluation.qarrrZ%langchain.evaluation.regex_match.baserZlangchain.evaluation.schemarrrZ'langchain.evaluation.scoring.eval_chainrrZ)langchain.evaluation.string_distance.baserrstrr$dictr!ZQAZCOT_QAZ CONTEXT_QAZPAIRWISE_STRINGZ SCORE_STRINGZLABELED_PAIRWISE_STRINGZLABELED_SCORE_STRINGZAGENT_TRAJECTORYZCRITERIAZLABELED_CRITERIAZSTRING_DISTANCEZPAIRWISE_STRING_DISTANCEZEMBEDDING_DISTANCEZPAIRWISE_EMBEDDING_DISTANCEZ JSON_VALIDITYZ JSON_EQUALITYZJSON_EDIT_DISTANCEZJSON_SCHEMA_VALIDATIONZ REGEX_MATCHZ EXACT_MATCHr)type__annotations__r5r;r'r'r'r(s          &       F