o {q i*@sdZddlmZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddlmZdddZdddZ GdddeeeZ!GdddeeeZ"Gddde"Z#dS) z-LLM Chains for evaluating question answering.) annotationsN)Sequence)AnyOptional) Callbacks)BaseLanguageModel)PromptTemplate) ConfigDict)override)LLMChain)CONTEXT_PROMPT COT_PROMPTPROMPT) LLMEvalChainStringEvaluator)RUN_KEYtextstrreturnOptional[tuple[str, int]]cCstd|tj}|r"|ddkrdS|ddkr"dSzI|dt ddt j }|dkr=WdS|dkrFWdS|d t ddt j }|dkr`WdS|dkriWdSWdSt yuYdSw) Nzgrade:\s*(correct|incorrect)CORRECT)rr INCORRECT)rrr) researchstrip IGNORECASEgroupuppersplit translater maketransstring punctuation IndexError)rmatchZ first_word last_wordr)i/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/langchain/evaluation/qa/eval_chain.py _get_scores8      r+dictcCs6|}t|}|durd\}}n|\}}|||dS)zParse the output text. Args: text (str): The output text to parse. Returns: Any: The parsed output. N)NN) reasoningvaluescore)rr+)rr-Z parsed_scoresr.r/r)r)r*_parse_string_eval_output3s  r0c@seZdZUdZdZded<eddZed7d d Z e d8d d Z e d7d dZ e d7ddZ e d9d:ddZ   d;dddd3d4Zedddd,d-d>d5d6ZdS)? QAEvalChainz,LLM Chain for evaluating question answering.resultsr output_keyignoreextrarboolcCdSNFr)clsr)r)r*is_lc_serializableRzQAEvalChain.is_lc_serializablecCr8)NZ correctnessr)selfr)r)r*evaluation_nameVr=zQAEvalChain.evaluation_namecCr8NTr)r>r)r)r*requires_referenceZr=zQAEvalChain.requires_referencecCr8rAr)r>r)r)r*requires_input^r=zQAEvalChain.requires_inputNllmrpromptOptional[PromptTemplate]kwargsrcKsL|pt}hd}|t|jkrd|d|j}t||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'input', 'answer' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: QAEvalChain: the loaded QA eval chain. >resultqueryanswerInput variables should be , but got rDrENr))rsetinput_variables ValueError)r;rDrErGexpected_input_varsmsgr)r)r*from_llmbszQAEvalChain.from_llmrIrJrH callbacksexamplesSequence[dict] predictions question_key answer_keyprediction_keyrUr list[dict]c*fddt|D}|j||dS)5Evaluate question answering examples and predictions.c,g|]\}}|||dqS)rIrJrHr).0iZexamplerZr[rXrYr)r*  z(QAEvalChain.evaluate..rT enumerateapply)r?rVrXrYrZr[rUinputsr)rdr*evaluate  zQAEvalChain.evaluater,cC&t||j}t|vr|t|t<|SNr0r3rr?rH parsed_resultr)r)r*_prepare_output zQAEvalChain._prepare_outputF referenceinputrUinclude_run_info predictionru Optional[str]rvrwcK ||||d||d}||S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): the LLM or chain prediction to evaluate. reference (Optional[str], optional): the reference label to evaluate against. input (Optional[str], optional): the input to consider during evaluation callbacks (Callbacks, optional): the callbacks to use for tracing. include_run_info (bool, optional): whether to include run info in the returned results. **kwargs: additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r`rUrwrrr?rxrurvrUrwrGrHr)r)r*_evaluate_stringss zQAEvalChain._evaluate_stringsc*|j|||d||dIdH}||S)Nr`rjrUrwZacallrrr}r)r)r*_aevaluate_strings  zQAEvalChain._aevaluate_stringsrr7rrrn)rDrrErFrGrrr1r`)rVrWrXrWrYrrZrr[rrUrrr\rHr,rr,rxrruryrvryrUrrwr7rGrrr,)__name__ __module__ __qualname____doc__r3__annotations__r model_config classmethodr<propertyr@rBrCrSrkrrr r~rr)r)r)r*r1IsH      # #r1c@seZdZdZed8ddZed8ddZed8dd Ze d d Z ed9ddZ ed:ddZ e d;dd(d)Zd?d+d,Zedddd-d.d@d4d5Zedddd-d.d@d6d7ZdS)AContextQAEvalChainz3LLM Chain for evaluating QA w/o GT based on contextrr7cCr8r9r)r:r)r)r*r<r=z%ContextQAEvalChain.is_lc_serializablecCr8)z.Whether the chain requires a reference string.Tr)r>r)r)r*rBz%ContextQAEvalChain.requires_referencecCr8)z+Whether the chain requires an input string.Tr)r>r)r)r*rCrz!ContextQAEvalChain.requires_inputr4r5rErNonecCs4hd}|t|jkrd|d|j}t|dS)N>contextrHrIrKrL)rNrOrP)r;rErQrRr)r)r*_validate_input_varssz'ContextQAEvalChain._validate_input_varsrcCr8)NzContextual Accuracyr)r>r)r)r*r@r=z"ContextQAEvalChain.evaluation_nameNrDrrFrGrcK&|pt}|||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'query', 'context' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: ContextQAEvalChain: the loaded QA eval chain. rMNr))r rr;rDrErGr)r)r*rSs zContextQAEvalChain.from_llmrIrrHrTrVr\rXrY context_keyr[rUrcr])r^cr_)rIrrHr)rarr[rXrYr)r*rerfz/ContextQAEvalChain.evaluate..rTrg)r?rVrXrYrr[rUrjr)rr*rkrlzContextQAEvalChain.evaluater,cCrmrnrorpr)r)r*rr(rsz"ContextQAEvalChain._prepare_outputFrtrxruryrvrwcKrz)Nrr{r|r}r)r)r*r~.s  z$ContextQAEvalChain._evaluate_stringscr)Nrrrr}r)r)r*rDrz%ContextQAEvalChain._aevaluate_stringsr)rErrrrrn)rDrrErFrGrrrr)rVr\rXr\rYrrrr[rrUrrr\rr)rrrrrr<rrBrCr rrr@rSrkrrr r~rr)r)r)r*rsJ      rc@s>eZdZdZedddZedddZe ddddZd S)CotQAEvalChainz=LLM Chain for evaluating QA using chain of thought reasoning.rr7cCr8r9r)r:r)r)r*r<Zr=z!CotQAEvalChain.is_lc_serializablercCr8)NzCOT Contextual Accuracyr)r>r)r)r*r@^r=zCotQAEvalChain.evaluation_nameNrDrrErFrGrcKr)zLoad QA Eval Chain from LLM.rMNr))r rrr)r)r*rSbs zCotQAEvalChain.from_llmrrrn)rDrrErFrGrrr) rrrrrr<rr@rSr)r)r)r*rWs  r)rrrr)rrrr,)$r __future__rrr$collections.abcrtypingrrZlangchain_core.callbacksrZlangchain_core.language_modelsrZlangchain_core.promptsrZpydanticr Ztyping_extensionsr Zlangchain.chains.llmr Z#langchain.evaluation.qa.eval_promptr r rZlangchain.evaluation.schemarrZlangchain.schemarr+r0r1rrr)r)r)r*s,