o )iY@sUddlmZddlZddlmZddlmZddlmZm Z ddl m Z ddl m Z ddlmZeeZdZejed <dZejed <d ZGd d d ejZdejdeededejdeejdejdejde dejfddZdejdejde dejfddZ d5dejdejdeded edejf d!d"Zdedeed#eeej fd$ej!dejf d%d&Z"dedeedejdejdeejdejde d$ej!dejfd'd(Z#e j$dgd)d*d+Z%e j$dgd)d,ejfd-d.Z&e j$dd gd)d/ejfd0d1Z'e j$d2ejd,ejfd3d4Z(dS)6)OptionalN) init_logger)tltriton)SamplingMetadata)apply_top_k_top_p)SpecDecodeMetadataPLACEHOLDER_TOKEN_IDGREEDY_TEMPERATURE c @sbeZdZdZdedeejdejdejdedejf dd Z e d ejd e de e e fd d Z dS)RejectionSamplerau The implementation strictly follows the algorithm described in https://arxiv.org/abs/2211.17192. However, we want to clarify the terminology used in the implementation: accepted tokens: tokens that are accepted based on the relationship between the "raw" draft and target probabilities. recovered tokens: tokens that are sampled based on the adjusted probability distribution, which is derived from both the draft and target probabilities. bonus tokens: If all proposed tokens are accepted, the bonus token is added to the end of the sequence. The bonus token is only sampled from the target probabilities. We pass in the bonus tokens instead of sampling them in the rejection sampler to allow for more flexibility in the sampling process. For example, we can use top_p, top_k sampling for bonus tokens, while spec decode does not support these sampling strategies. output tokens: Tokens are finally generated with the rejection sampler. output tokens = accepted tokens + recovered tokens + bonus tokens metadata draft_probs target_logitsbonus_token_idssampling_metadatareturnc Cs>|jtksJt||j|}t|j|j|j|j||||}|S)a* Args: metadata: Metadata for spec decoding. draft_probs (Optional[torch.Tensor]): Probability distribution for the draft tokens. Shape is [num_tokens, vocab_size]. Can be None if probabilities are not provided, which is the case for ngram spec decode. target_logits (torch.Tensor): Target model's logits probability distribution. Shape is [num_tokens, vocab_size]. Here, probabilities from different requests are flattened into a single tensor because this is the shape of the output logits. NOTE: `target_logits` can be updated in place to save memory. bonus_token_ids_tensor (torch.Tensor): A tensor containing bonus tokens. Shape is [batch_size, 1]. Bonus tokens are added to the end of the sequence if all proposed tokens are accepted. We generate the bonus tokens outside of the rejection sampler with the default sampling strategy. It allows for more flexibility in the sampling process such as top_p, top_k sampling. sampling_metadata (vllm.v1.sample.metadata.SamplingMetadata): Additional metadata needed for sampling, such as temperature, top-k/top-p parameters, or other relevant information. Returns: output_token_ids (torch.Tensor): A tensor containing the final output token IDs. ) max_spec_len MAX_SPEC_LEN compute_probscu_num_draft_tokensrejection_sampledraft_token_idsnum_draft_tokens)selfrrrrr target_probsoutput_token_idsrl/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/v1/sample/rejection_sampler.pyforward.s"' zRejectionSampler.forwardr vocab_sizecs6|}|tk||k@fddt|D}|S)aParse the output of the rejection sampler. Args: output_token_ids: The sampled token IDs in shape [batch_size, max_spec_len + 1]. The rejected tokens are replaced with `PLACEHOLDER_TOKEN_ID` by the rejection sampler and will be filtered out in this function. vocab_size: The size of the vocabulary. Returns: A list of lists of token IDs. cs g|] \}}||qSr)tolist).0irowZ valid_maskrr sz1RejectionSampler.parse_output..)cpunumpyr enumerate)rr!Zoutput_token_ids_npZoutputsrr&r parse_outputks  zRejectionSampler.parse_outputN)__name__ __module__ __qualname____doc__rrtorchTensorrr staticmethodintlistr+rrrrr s.  = r rrrrrrrrrcCsv|jdksJ|dus|jdksJ|jdksJ|jdks Jt|}|jd} |jd} |j} |s7J|dusA|sAJ|sGJ|sMJ|j| | fksVJtj||dftj| d} | t |j rnd} n|j t k} |j s|jdd}t|f| ||||| |dd|j r| St| ||j| }t|||||||| }t|f| |||||||| || |dudd | S) Nrr dtypedevice)dim) num_warps)NO_DRAFT_PROBSr;)ndimlenshaper9Z is_contiguousr0emptyZint32Zfill_r all_greedy temperaturer Z all_randomargmaxrejection_greedy_sample_kernelgenerate_uniform_probs generatorssample_recovered_tokensrejection_random_sample_kernel)rrrrrrrr batch_size num_tokensr!r9r is_greedyZ target_argmax uniform_probsrecovered_token_idsrrrrs            rlogitscCs|jdksJ|jdksJ|jr|S|jd}t|j||tdd}||dd}|jdur8t|j||}d}|j durFt|j ||}t |||}|j dt j d}|S)aCompute probability distribution from logits based on sampling metadata. This function applies temperature scaling to the logits and converts them to probabilities using softmax. For greedy decoding, it returns the original logits. Args: logits: Input logits tensor to be converted to probabilities. cu_num_draft_tokens: Cumulative number of draft tokens. sampling_metadata: Metadata containing sampling parameters such as temperature and whether greedy sampling is used. Returns: torch.Tensor: Probability distribution (softmax of scaled logits) if non-greedy sampling is used, otherwise returns the original logits. r6r5r) replace_from replace_tor N)r:r8)r=rAr?expand_batch_to_tokensrBr Zdiv_Z unsqueezetop_ktop_prZsoftmaxr0float32)rNrrrJrBrRrSZ output_probrrrrs<    rx cu_num_tokensrJrOrPc CsF|jd}|jd|ksJ||}t|f|||||tdd|S)aExpand [batch_size] tensor to [num_tokens] tensor based on the number of tokens per batch in cu_num_tokens. For example, if x = [a, b, c] and cu_num_tokens = [2, 5, 6], then num_tokens = 6, and expanded_x = [a, a, b, b, b, c]. Args: x: [batch_size] tensor to expand. cu_num_tokens: [batch_size] tensor containing the cumulative number of tokens per batch. Each element represents the total number of tokens up to and including that batch. num_tokens: Total number of tokens. replace_from: int = 0 Value to be replaced if it is found in x. replace_to: int = 0 Value to replace with when replace_from is found. Returns: expanded_x: [num_tokens] tensor. rr5)MAX_NUM_TOKENSr;)r?Z new_empty expand_kernelr)rUrVrJrOrPrIZ expanded_xrrrrQ(s   rQrFr9c Csjtj|ftj|d}d}t|D]"\}}|dkrq||}||} | dur0|||j| d|}q|S)a Generates a batch of uniform random samples, with optional seeding if available. This method creates a tensor of shape `(num_tokens, )` filled with uniform random values in the range [0, 1). If `generators` is provided, the requests with their own seeds will use the provided `torch.Generator` for reproducibility. The samples for the other requests will be generated without a seed. Args: num_tokens : int Total number of tokens. num_draft_tokens : List[List[int]] Number of draft tokens per request. generators : Optional[Dict[int, torch.Generator]] A dictionary mapping indices in the batch to `torch.Generator` objects. device : torch.device The device on which to allocate the tensor. Returns: uniform_rand : torch.Tensor A tensor of shape `(num_tokens, )` containing uniform random values in the range [0, 1). r7rN generator)r0ZrandrTr*getZuniform_) rJrrFr9rL start_idxreq_idxnend_idxrZrrrrEQs rEc Cst|}|jd} tj|| ftj|d} | |jD]\} } || dkr/| | j| dqt|} t ||f| ||||| | t | |dud | S)Nr r7rrY)r<) r>r?r0r@rTZ exponential_rFitemsZ empty_likesample_recovered_tokens_kernelrZnext_power_of_2)rrrrrrrr9rIr!qr$rZrMrrrrGs2     rG)Zdo_not_specializecCstd}|dur d}nt||}|sdS|dkrd} n t||d} t||} | | } d} t| D]*} | sbt|| | }t|| | }t|||d| |||krbd} q8| s|t||}t|||d| |dSdS)NrTr5Fr program_idloadrangestore)output_token_ids_ptrcu_num_draft_tokens_ptrdraft_token_ids_ptrZtarget_argmax_ptrbonus_token_ids_ptr is_greedy_ptrrr]rKr\r_rrejectedposdraft_token_idZtarget_argmax_idbonus_token_idrrrrDs>  rDr<c CsPtd} t|| } | rdS| dkrd}n t|| d}t|| }||}d}t|D][}|st|||}| rCd}n t|||| |}t|||| |}t|||}|dkrs|||krs|}n d}t|||}t|| | d||q1|st|| }t|| | d||dSdS)Nrr5FTrc)rhrirjdraft_probs_ptrtarget_probs_ptrrkZrecovered_token_ids_ptrZuniform_probs_ptrrlrr!r<r]rKr\r_rrmrnro draft_prob target_probZ uniform_probZtoken_idrprrrrHsV    rHrWc Cstd}|dkr d}n t||d}t||}||} t||} t| |k|| } td|} tj||| | | | kddS)Nrr5)mask)rrdrewherearangerg) Z output_ptrZ input_ptrZcu_num_tokens_ptrrOrPrWr]r\r_rJZsrc_valoffsetrrrrXs   rXPADDED_VOCAB_SIZEc Cstd} | dkr d} n t|| d} t|| } | | } td} | | kr+dStd|}|rjt|| | }t|| | ||}t|| | ||dtj|| | ||||kdd}n,tj|| | ||||kdd}tj|| | ||||kdd}t||d}tj|| ||||ktdd}tj||dd}t|| | ||rt|| | |||dSdS)Nrr5)ruotherz-infr )Zaxis)rrdrerwrgmaximumfloatrC)rhrirjrqrrZq_ptrr!ryr<r]r\r_rrnZ vocab_offsetroZ orig_probZprobrsrtrbZ recovered_idrrrra8sh    ra)rr))typingrr0Ztorch.nnnnZ vllm.loggerrZvllm.triton_utilsrrZvllm.v1.sample.metadatarZ$vllm.v1.sample.ops.topk_topp_samplerrZvllm.v1.spec_decode.metadatarr,loggerr Z constexpr__annotations__r rModuler r1r4r3rrrQdict Generatorr9rErGZjitrDrHrXrarrrrs     p    d A )  2    -  / >