o )i@sUdZddlZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl mZddlmZddlmZmZmZmZddlZdd lmZdd lmZdd lmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)zddl*m+Z+Wne,ye)dZ*e*-dZ+Ynwzddl.Z/Wn e,ye)dZ/Ynwzddl0Z0Wn e,ye)dZ0Ynwzddl(m1Z1Wne,yddl2m3Z1Ynwe4e5Z6e GdddZ7GdddeZ8    d[d!e9d"e9d#e9d$e9d%e9d&e:d'e:fd(d)Z;ed*ee?e9e&fe@d-<d.ed'e ed>e8ZHGd?d@d@e8ZIGdAdBdBe8ZJGdCdDdDeJZKGdEdFdFeJZLGdGdHdHeJZMGdIdJdJeJZNGdKdLdLeJZOdMZP Nd\dOe?dPe)a% Abstract method to generate sample requests from the dataset. Subclasses must override this method to implement dataset-specific logic for generating a list of SampleRequest objects. Args: tokenizer (PreTrainedTokenizerBase): The tokenizer to be used for processing the dataset's text. num_requests (int): The number of sample requests to generate. Returns: list[SampleRequest]: A list of sample requests generated from the dataset. z)sample must be implemented in subclasses.r?)r4rCrLr,r,r-sampleszBenchmarkDataset.samplerequestscCsLt||kr$t|jtj||t|d}||td|dSdS)a Oversamples the list of requests if its size is less than the desired number. Args: requests (List[SampleRequest]): The current list of sampled requests. num_requests (int): The target number of requests. kz/Oversampled requests to reach %d total samples.N)lenrGseedr0choicesextendloggerinfo)r4rNrL additionalr,r,r-maybe_oversample_requestss   z*BenchmarkDataset.maybe_oversample_requestsNr1NNN)r#r$r%r2 IS_MULTIMODALr r'r)r5rr+r*r=rBr tuplerrrKrrrMrXr,r,r,r-r.VsV   , r.Fr output_lenmin_lenmax_prompt_len max_total_lenskip_min_output_len_checkr1c Cs<||k}| o ||k}||k}|||k} |p|p|p| S)a Validate a sequence based on prompt and output lengths. Default pruning criteria are copied from the original `sample_hf_requests` and `sample_sharegpt_requests` functions in benchmark_serving.py, as well as from `sample_requests` in benchmark_throughput.py. r,) rrarbrcrdreZprompt_too_shortZoutput_too_shortZprompt_too_longZcombined_too_longr,r,r-is_valid_sequences  rfrEcCst|SrYr)rEr,r,r-rIsrIrJimagecCst|trd|vrtt|d}t|tjrLt|d}t}|j|ddt |  d}Wdn1s=wYddd |id St|t rd| d rX|nd |}dd|id Std |d)a Process a single image input and return a multimedia content dictionary. Supports three input types: 1. Dictionary with raw image bytes: - Expects a dict with a 'bytes' key containing raw image data. - Loads the bytes as a PIL.Image.Image. 2. PIL.Image.Image input: - Converts the image to RGB. - Saves the image as a JPEG in memory. - Encodes the JPEG data as a base64 string. - Returns a dictionary with the image as a base64 data URL. 3. String input: - Treats the string as a URL or local file path. - Prepends "file://" if the string doesn't start with "http://" or "file://". - Returns a dictionary with the image URL. Raises: ValueError: If the input is not a supported type. bytesRGBZJPEG)formatutf-8N image_urlurlzdata:image/jpeg;base64,)r8rl)zhttp://file://rnzInvalid image input zF. Must be a PIL.Image.Image or str or dictionary with raw image bytes.) isinstancer*r openrriosavebase64 b64encodegetvaluedecoder' startswith ValueError)rgZ image_dataZ image_base64rlr,r,r- process_image s0     rycsfeZdZdZdZdZdZ  dfdd Zeeeefd ed e d e d e d e de de e fddZ ZS) RandomDatasetrr_r1Nc 0tjdi|t|jtj|jdSNr,superr5rGrRr0npr4kwargs __class__r,r-r5A zRandomDataset.__init__rCrL prefix_len range_ratio input_lenrac Ksv|dksJd|j}|} || } |dkr"tjjd||dng} t| d|} t| d|} t|d|}t|d|}td| | ||tjj| | d|d}tjj||d|d}tjjd||d}g}t |D]F}|||t |||}| |}| |}|t||}|j |ddd|}| |}t |}|t||t||d qr|S) Ng?zArandom_range_ratio must be < 1.0 to ensure a valid sampling rangersizerFz=Sampling input_len from [%s, %s] and output_len from [%s, %s]FZadd_special_tokensrrr ) vocab_sizeZnum_special_tokens_to_addrrGrHtolistr)rUrVrangeZarangervencoderQr<r)r4rCrLrrrrarrZnum_special_tokensZreal_input_lenZprefix_token_idsZ input_lowZ input_highZ output_lowZ output_highZ input_lensZ output_lensoffsetsrNiZ inner_seqZtoken_sequencerZtotal_input_lenZre_encoded_sequencer,r,r-rMIsl      zRandomDataset.samplerZ)r#r$r%DEFAULT_PREFIX_LENZDEFAULT_RANGE_RATIODEFAULT_INPUT_LENDEFAULT_OUTPUT_LENr5r r)floatr+rrM __classcell__r,r,rr-rz:s6  rzcsheZdZdZdfdd ZdddZ    dd ed ed ee d eed eede de fddZ Z S)ShareGPTDatasetz Implements the ShareGPT dataset. Loads data from a JSON file and generates sample requests based on conversation turns. r1Nc tjdi||dSr~rr5rBrrr,r-r5 zShareGPTDataset.__init__cCsz|jdur tdt|jdd}t||_Wdn1s!wYdd|jD|_t|jt |jdS)N/dataset_path must be provided for loading data.rkencodingcSs(g|]}d|vrt|ddkr|qS) conversationsrQ).0entryr,r,r- s z-ShareGPTDataset.load_data..) r/rxrpjsonloadr3rGrRr0shuffler4fr,r,r-rBs  zShareGPTDataset.load_dataFrCrLrErDraenable_multimodal_chatc Ksg}|jD]i} t||krn`| ddd| ddd} } |j|||d\} }|| j} || j}t| }|dur@t|n|}t|||dudsLq| d}rXt|}nd}|rb|| |} |t | ||| |dq| |||S) NrrvaluerFrCrDrE)rerg)rrr r"r!) r3rQrK input_idsrfgetryr=r<rrX)r4rCrLrErDrarrsamplesrr completionr" prompt_idscompletion_idsrZnew_output_lenZ image_pathr6r,r,r-rMsT      zShareGPTDataset.samplerZ)NNNFr#r$r%r&r5rBr r)r r'boolr+rMrr,r,rr-rs.  rparsercCs|jdtdd|jdtddd|jdtd gd d d |jd ddd|jdtddd|d}|jdtddd|jdddd|d}|jdtddd|jdtddd|jd td!d"d|d#}|jd$tdd%d|d&}|jd'td(d)d|jd*td+d,d|jd-td.d/d|jd0tdd1d|d2}|jd3tdd4d|jd5tdd6d|jd7tdd8d|d9}|jd:tdd;d|jdtd?d@d|jdAtd+dBddS)CNz--seedr)r8defaultz --num-promptsizNumber of prompts to process.)r8rhelpz--dataset-namerG)sharegptburstgptsonnetrGhfcustomprefix_repetitionz$Name of the dataset to benchmark on.)r8rrSrz --no-stream store_truez*Do not load the dataset in streaming mode.)actionrz--dataset-pathzWPath to the sharegpt/sonnet dataset. Or the huggingface dataset ID if using HF dataset.zcustom dataset optionsz--custom-output-lenzBNumber of output tokens per request, used only for custom dataset.z--custom-skip-chat-templatezDSkip applying chat template to prompt, used only for custom dataset.zsonnet dataset optionsz--sonnet-input-len&zANumber of input tokens per request, used only for sonnet dataset.z--sonnet-output-lenzBNumber of output tokens per request, used only for sonnet dataset.z--sonnet-prefix-lenzBNumber of prefix tokens per request, used only for sonnet dataset.zsharegpt dataset optionsz--sharegpt-output-lenzVOutput length for each request. Overrides the output length from the ShareGPT dataset.zrandom dataset optionsz--random-input-lenr_zBNumber of input tokens per request, used only for random sampling.z--random-output-lenr|zCNumber of output tokens per request, used only for random sampling.z--random-range-ratior{zRange ratio for sampling input/output length, used only for random sampling. Must be in the range [0, 1) to define a symmetric sampling range[length * (1 - range_ratio), length * (1 + range_ratio)].z--random-prefix-lenzNumber of fixed prefix tokens before the random context in a request. The total input length is the sum of `random-prefix-len` and a random context length sampled from [input_len * (1 - range_ratio), input_len * (1 + range_ratio)].zhf dataset optionsz --hf-subsetzSubset of the HF dataset.z --hf-splitzSplit of the HF dataset.z--hf-output-lenzYOutput length for each request. Overrides the output lengths from the sampled HF dataset.z!prefix repetition dataset optionsz--prefix-repetition-prefix-lenzMNumber of prefix tokens per request, used only for prefix repetition dataset.z--prefix-repetition-suffix-lenz|Number of suffix tokens per request, used only for prefix repetition dataset. Total input length is prefix_len + suffix_len.z --prefix-repetition-num-prefixes z|Number of prefixes to generate, used only for prefix repetition dataset. Prompts per prefix is num_requests // num_prefixes.z--prefix-repetition-output-lenzMNumber of output tokens per request, used only for prefix repetition dataset.) add_argumentr)r'add_argument_groupr)rZ custom_groupZ sonnet_groupZsharegpt_groupZ random_groupZhf_groupZprefix_repetition_groupr,r,r-add_dataset_parsers        rc s`jdkrtjd}|jjjjd}|SjdkrUtjd}jdkr:|jjj j j dd}|Sj sDj sDJd|jjj j j d d}|Sjd krjtjvrit}d _d_nhjtjvrut}d _n\jtjvrt}d _nPjtjvrt}nGjtjvrt}d _n;jtjvrt}d _n/jtjvrt}d _n#jtjvrt}d _ntd d tD}tdjd|d|jr݈jdvrtd|jjjjjdjjj d}|Sfddfddfddfddd}z |j}W|St!y/}z tdj|d}~ww)Nr)r/)rLrCraskip_chat_templater openai-chatF)rLrrarrCreturn_prompt_formattedz;Tokenizer/model must have chat template for sonnet dataset.TrtraincSsg|] }|jD]}|qqSr,)SUPPORTED_DATASET_PATHS)rcls dataset_namer,r,r-rszget_samples..Unsupported dataset path: zH. Huggingface dataset only supports dataset_path from one of following: z_. Please consider contributing if you would like to add support for additional dataset formats.)rz openai-audiozRMulti-modal content is only supported on 'openai-chat' and 'openai-audio' backend.)r/dataset_subset dataset_splitr0 no_stream)rLrCracs tjjdjjjdS)Nr0r/)rCrLra)rrRr/rM num_promptsZsharegpt_output_lenr,argsrCr,r-szget_samples..cstjjdjjdS)Nr)rCrL)BurstGPTDatasetrRr/rMrr,rr,r-rs c,tjjdjjjjjjdS)Nr)rCrLrrrar) rzrRr/rMrZrandom_prefix_lenZrandom_input_lenZrandom_output_lenZrandom_range_ratior,rr,r-rscr)Nr)rCrLr suffix_len num_prefixesra) PrefixRepetitionRandomDatasetrRr/rMrZprefix_repetition_prefix_lenZprefix_repetition_suffix_lenZprefix_repetition_num_prefixesZprefix_repetition_output_lenr,rr,r-rs)rrrGrzUnknown dataset: )"r CustomDatasetr/rMrZcustom_output_lenZcustom_skip_chat_template SonnetDatasetZ endpoint_typeZsonnet_input_lenZsonnet_output_lenZsonnet_prefix_lenZ chat_templateZdefault_chat_templateVisionArenaDatasetrZhf_splitZ hf_subsetInstructCoderDatasetMTBenchDatasetConversationDataset AIMODatasetNextEditPredictionDataset ASRDataset MLPerfDatasetsetHuggingFaceDataset__subclasses__rxr\rRrZ hf_output_lenKeyError)rrCZdatasetZinput_requestsZ dataset_classZsupported_datasetsZdataset_mappingerrr,rr- get_sampless    yn         6   #rcsneZdZdZdfdd ZdddZ     dd ed ed ee d eed eede de de fddZ Z S)ra* Implements the Custom dataset. Loads data from a JSONL file and generates sample requests based on conversation turns. E.g., ``` {"prompt": "What is the capital of India?"} {"prompt": "What is the capital of Iran?"} {"prompt": "What is the capital of China?"} ``` r1Nc rr~rrrr,r-r5rzCustomDataset.__init__cCs|jdur tdg|_|jdr5tj|jdd}d|jvr#td|D] \}}|j| q'nt dt |j t |jdS)Nrz.jsonlT)Z path_or_buflinesrz*JSONL file must contain a 'prompt' column.z1Only JSONL format is supported for CustomDataset.)r/rxr3endswithpd read_jsoncolumnsZiterrowsr<to_dictr@rGrRr0r)r4Z jsonl_data_rowr,r,r-rB s"    zCustomDataset.load_dataFrCrLrErDrarrc Ksvg} |jD]-} t| |krn$| d} |s!|jd| dgddd} t|| j} | t| | |dq|| || S)Nrr9r:TFZadd_generation_prompttokenizer)r3rQapply_chat_templaterr<rrX) r4rCrLrErDrarrrsampled_requestsitemrrr,r,r-rMAs.    zCustomDataset.samplerZ)NNNFFrr,r,rr-rs4 % rzDSonnetDataset is deprecated and will be removed in a future version.c sheZdZdZdZdZdZ  dfdd Zdd d Zeeed fd e d e de de de de f ddZ Z S)rz Simplified implementation of the Sonnet dataset. Loads poem lines from a text file and generates sample requests. Default values here copied from `benchmark_serving.py` for the sonnet dataset. rrrr1Nc rr~rrrr,r-r5{s zSonnetDataset.__init__cCsL|jstdt|jdd}||_WddS1swYdS)Nzdataset_path must be provided.rkr)r/rxrp readlinesr3rr,r,r-rBs  "zSonnetDataset.load_dataFrLrrrarc s@fdd|jD}tdd|Dt|} d} d| dg} j| dd d } t| j} || kr:td | d t|| | }tt|| | d }|jd|}g}t||krtj |j||d}| d ||}d|dg}j|dd d }t|j}||kr| t |r|n|||dt||ks\|S)Ncsg|]}|jqSr,)r)rlinerCr,r-rsz(SonnetDataset.sample..css|]}t|VqdSrYr)rtokensr,r,r- sz'SonnetDataset.sample..z5Pick as many lines as you can from these poem lines: r9r:TFrz8'input_len' must be higher than the base prompt length (z).rrOr) r3sumrQrrrxroundmaxrGrSjoinr<r)r4rCrLrrrarrZtokenized_linesZavg_lenZ base_promptZbase_msgZbase_fmtZ base_offsetZnum_input_linesZnum_prefix_linesZ prefix_linesr extra_linesrmsgprompt_formattedrr,rr-rMsZ     zSonnetDataset.samplerZ)r#r$r%r&rrrr5rBr)rr+rMrr,r,rr-rms4   rc sleZdZdZdfdd ZddZdedefd d Z  dd e ded e ed e e dee f ddZ ZS)rz Implements the BurstGPT dataset. Loads data from a CSV file and generates sample requests based on synthetic prompt generation. Only rows with Model "GPT-4" and positive response tokens are used. r1Nc rr~rrrr,r-r5rzBurstGPTDataset.__init__cCsH|jdur tdt|j}||ddk}||ddk}||_dS)NrZModelzGPT-4zResponse tokensr)r/rxrZread_csvr3)r4dfZgpt4_dfr,r,r-rBs   zBurstGPTDataset.load_datarLcCs@|t|jkr|jj||jd}n |jj||jdd}|jS)N)n random_stateT)r r replace)rQr3rMr0valuesr)r4rLr3r,r,r-_sample_loaded_datas z#BurstGPTDataset._sample_loaded_datarCrDrEc sg}|j|d}t|D];t|d}t|d} |j|||d\} }|jfddt|D} || } |t| || | dq |S)N)rLrrcsg|]}|qSr,r,)rjrrr,r-rsz*BurstGPTDataset.sample..)rrr r")r rr)rKrrvr<r) r4rCrLrDrErrr3rraZlora_reqZ token_idsrr,rr-rMs(    zBurstGPTDataset.samplerZr[)r#r$r%r&r5rBr)r+r r r r'rrMrr,r,rr-rs$ rc sneZdZUdZeZeeeeee ffe d<  ddedede de ed df fd d Z dd d ZZS)rz.Base class for datasets hosted on HuggingFace.rFNr/rrrr1c s6tjdd|i|||_||_| |_|dS)Nr/r,)rr5rr load_streamrB)r4r/rrrrrr,r-r5 s  zHuggingFaceDataset.__init__cCs0t|j|j|j|jd|_|jj|jd|_dS)z$Load data from HuggingFace datasets.)namesplitZ streaming)rRN)rr/rrrr3rr0rAr,r,r-rBszHuggingFaceDataset.load_data)FNrZ)r#r$r%r&rrr r'r*r r(rr r5rBrr,r,rr-rs" "rc @sDeZdZdZddhZdZ  ddeded eed e d e f d d Z dS)rz6Dataset for conversation data with multimodal support.zlmms-lab/LLaVA-OneVision-Dataz Aeala/ShareGPT_Vicuna_unfilteredTNFrCrLrarr1c Ks|jdd}g}|du}|D]e} t||krn\| d} | dd| dd} } || j} || j}t| }t|}|rA|n|}t|trL|dksNJ|rVt||sVqd| vr`t| dnd}|rj|| |} | t | |||dq| |||S) NcSst|ddkS)Nrrrxr,r,r-r9sz,ConversationDataset.sample..rrrrFrgrrr r!) r3filterrQrror)rfryr=r<rrX)r4rCrLrarrZ filtered_datardynamic_outputrconvrrrrrcompletion_lenr6r,r,r-rM1sP     zConversationDataset.sampleNF) r#r$r%r&rr\r r)r rr+rMr,r,r,r-r*s$rc @sReZdZdZdZdddddZdZ  dd ed ed e ed e de f ddZ dS)rz Vision Arena Dataset. r|cC|ddddS)NZ conversationrr;r,rr,r,r-rlzVisionArenaDataset.cCr)Nturnsrr;r,rr,r,r-rnr)zlmarena-ai/VisionArena-Chatz"lmarena-ai/vision-arena-bench-v0.1TNFrCrLrarr1c Ks|dur|n|j}g}|jD]C}t||krn:|j|j}|dur+td|j||} t|dd} t|| j} |rF| | | } | t | | || dq| |||S)NrZimagesrr) rr3rQrrr/rxryrr=r<rrX) r4rCrLrarrrrZ parser_fnrr6rr,r,r-rMrs<     zVisionArenaDataset.sampler) r#r$r%r&rrr\r r)r rr+rMr,r,r,r-rds(rc @BeZdZdZdZdhZ  d dededeed e d e f d d Z dS)ra InstructCoder Dataset. https://huggingface.co/datasets/likaixin/InstructCoder InstructCoder is the dataset designed for general code editing. It consists of 114,239 instruction-input-output triplets, and covers multiple distinct code editing scenario. rzlikaixin/InstructCoderNFrCrLrarr1c Ks|dur|n|j}g}|jD]3}t||krn*|dd|dd}|jd|dgddd }t||j} |t|| |d q||||S) Ninputz Z instructionzB Just output the code, do not include any explanation.r9r:TFrrrr3rQrrr<rrX r4rCrLrarrrrrrr,r,r-rMs4    zInstructCoderDataset.sampler r#r$r%r&rrr r)r rr+rMr,r,r,r-rs$ rc @r)ra' MT-Bench Dataset. https://huggingface.co/datasets/philschmid/mt-bench We create a single turn dataset for MT-Bench. This is similar to Spec decoding benchmark setup in vLLM https://github.com/vllm-project/vllm/blob/9d98ab5ec/examples/offline_inference/eagle.py#L14-L18 rzphilschmid/mt-benchNFrCrLrarr1c Ks|dur|n|j}g}|jD]-}t||krn$|dd}|jd|dgddd}t||j} |t|| |dq||||S) Nrrr9r:TFrrr!r"r,r,r-rMs4      zMTBenchDataset.samplerr#r,r,r,r-rs$ rc @s:eZdZdZhdZ d dededeedefdd Z dS) rzO Dataset class for processing a AIMO dataset with reasoning questions. >zAI-MO/aimo-validation-aimezAI-MO/NuminaMath-CoTzAI-MO/NuminaMath-1.5NrCrLrar1c Ksg}|du}|jD]L}t||krnC|d|d}} ||j} || j} t| } t| } |r2| n|}t|tr=|dks?J|rJt| | dddsJq |t|| |ddq ||||S)NproblemZsolutionrr`i})rcrdr) r3rQrror)rfr<rrX)r4rCrLrarrrrrrrrrrr,r,r-rMs8      zAIMODataset.samplerY) r#r$r%r&rr r)r r+rMr,r,r,r-rsra*### Instruction: You are a code completion assistant and your task is to analyze user edits and then rewrite an excerpt that the user provides, suggesting the appropriate edits within the excerpt, taking into account the cursor location. ### User Edits: {} ### User Excerpt: {} ### Response: <|editable_region_start|>rMoriginal_start_markerc CsH|d}|d}|d}t||}||}||d}|}||dS)a4Format the zeta prompt for the Next Edit Prediction (NEP) dataset. This function formats examples from the NEP dataset into prompts and expected outputs. It could be further extended to support more NEP datasets. Args: sample: The dataset sample containing events, inputs, and outputs. original_start_marker: The marker indicating the start of the editable region. Defaults to "<|editable_region_start|>". Returns: A dictionary with the formatted prompts and expected outputs. eventsr outputN)rexpected_output) zeta_promptrjfind) rMr&r'r r(rZoutput_start_indexZoutput_focused_regionr)r,r,r-_format_zeta_promptPs    r,c@s0eZdZdZdhZdeiZdedefddZ dS)rzF Dataset class for processing a Next Edit Prediction dataset. zzed-industries/zetarCrLc Ks|j|j}|durtd|jg}|jD](}||}|t|dt||djt||djdt||kr@nq| |||S)Nrrr)r) MAPPING_PROMPT_FUNCSrr/rxr3r<rrQrrX)r4rCrLrZformatting_prompt_funcrrMr,r,r-rM}s,    z NextEditPredictionDataset.sampleN) r#r$r%r&rr,r-r r)rMr,r,r,r-rqsrc @sTeZdZUdZhdZdZdZdZdZe e d< dde d e d e e d efd d ZdS)ra Dataset class for processing a ASR dataset for transcription. Tested on the following set: +----------------+----------------------------------------+--------------------------+-----------------------------+ | Dataset | Domain | Speaking Style | hf-subset | +----------------+----------------------------------------+--------------------------+-----------------------------+ | TED-LIUM | TED talks | Oratory | release1, release2, release3| | | | | release3-speaker-adaptation | | VoxPopuli | European Parliament | Oratory | en, de, it, fr, ... | | LibriSpeech | Audiobook | Narrated | "LIUM/tedlium" | | GigaSpeech | Audiobook, podcast, YouTube | Narrated, spontaneous | xs, s, m, l, xl, dev, test | | SPGISpeech | Financial meetings | Oratory, spontaneous | S, M, L, dev, test | | AMI | Meetings | Spontaneous | ihm, sdm | +----------------+----------------------------------------+--------------------------+-----------------------------+ >zfacebook/voxpopulizopenslr/librispeech_asrz LIUM/tedliumzspeechcolab/gigaspeechzkensho/spgispeechzedinburghcstr/amir|Tz9<|startoftranscript|><|en|><|transcribe|><|notimestamps|>skip_long_audiosNrCrLrar1c Ks|dur|n|j}tj}t||j}g}d}|jD];} t||kr$n2| d} | d| d} } tj| | d} |jrD| dkrD|d7}qd| | fi}| t ||||dq|r^t d || |||S) NraudioarrayZ sampling_rate)ysrrFrz_%d samples discarded from dataset due to their length being greater than what Whisper supports.)rrTRANSCRIPTION_PREAMBLErQrr3rZ get_durationr.r<rrUwarningrX)r4rCrLrarrrrZskippedrr/r1r2Z duration_sr6r,r,r-rMsB    zASRDataset.samplerY)r#r$r%r&rrr\r4r.rr(r r)r r+rMr,r,r,r-rs$   rc @s>eZdZdZddhZ d dededeedee fd d Z dS) ra2 MLPerf Inference Dataset. Dataset on HF: https://huggingface.co/datasets/mgoin/mlperf-inference-llama2-data https://huggingface.co/datasets/mgoin/mlperf-inference-llama3.1-data Each record contains: - "system_prompt": system role instruction. - "question": user question. - "output": reference answer. We combine the system prompt and question into a chat-formatted prompt (using the tokenizer's chat template) and set the expected output length to the tokenized length of the provided reference answer. z"mgoin/mlperf-inference-llama2-dataz$mgoin/mlperf-inference-llama3.1-dataNrCrLrar1cKs|du}g}|jD]N}t||krnE|d}|d} |d} d|dd| dg} |j| ddd } t|| j} t|| dd j}|rE|n|}t| |sMq |t| | |d q ||||S) N system_promptquestionr(systemr:r9TFrrr)r3rQrrrfr<rrX)r4rCrLrarrrrr6r7Zreference_answermessagesrrZ ref_out_lenr r,r,r-rMs<      zMLPerfDataset.samplerY) r#r$r%r&rr r)r r+rrMr,r,r,r-rs rcsfeZdZdZdZdZdZ  dfdd Zeeeefded e d e d e d e d e de e fddZ Z S)rrrr|r1Nc r}r~rrrr,r-r5Crz&PrefixRepetitionRandomDataset.__init__rCrLrrrrac sj||}|dkrtd|d|ddtdttffdd g} t|D](} |} t|D]} |} | | } | }t| }| t|||d q5q+t | | S) Nrznum_requests (z1) must be greater than or equal to num_prefixes () target_lengthr1csrtjjd|d}|}j|dd}t||kr|St||kr3|t|}|}||S|d|S)zOGenerate tokens that decode and re-encode to exactly target_length.rrFrN)rrGrHrrvrrQ)r;rr7Z re_encodedneededZ extra_tokens_generate_exact_length_tokensrCrr,r-r>]s     zKPrefixRepetitionRandomDataset.sample.._generate_exact_length_tokensr) rrxr)r+rrvrQr<rrGr)r4rCrLrrrrarZprompts_per_prefixrNrZ prefix_tokensZ suffix_tokensZcombined_tokensrrr,r=r-rMKs6     z$PrefixRepetitionRandomDataset.samplerZ)r#r$r%rZDEFAULT_SUFFIX_LENZDEFAULT_NUM_PREFIXESrr5r r)r+rrMrr,r,rr-r;s6  r)r^r_r`F)r%)Vr&rsrqrloggingrGabcrrcollections.abcr dataclassesr functoolsrrtypingrr r r numpyrZPILr Z transformersr Ztyping_extensionsrZvllm.lora.requestrZvllm.lora.utilsrZvllm.multimodalrZvllm.multimodal.imagerZ!vllm.transformers_utils.tokenizerrrZ vllm.utilsrrr ImportErrorZplaceholder_attrrrrrargparser getLoggerr#rUrr.r)rrfr'rIrJr*r(ryrzrrr+rrrrrrrrrrr*r,rrrrr,r,r,r-s                       1YK%\QE$:98:1 !'TO