o )i$@sZdZddlZddlmZddlmZmZddlmZddl m Z dZ Gdddej Z dS) z4Sampler layer implementing TPU supported operations.N)LogprobsTensors SamplerOutput)TopKTopPSampler)TPUSupportedSamplingMetadatagh㈵>cseZdZfddZdejdedefddZdejdejdejfd d Z dejdejfd d Z dejdedejfd dZ dejdejfddZ dejde dejdefddZdejdejdejfddZZS)Samplercstt|_dSN)super__init__rtopk_topp_sampler)self __class__f/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/v1/sample/tpu/sampler.pyr s  zSampler.__init__logitssampling_metadatareturncCs.|tj}|||}t|ddd}|S)N)Zsampled_token_idsZlogprobs_tensors)totorchfloat32sampler unsqueeze)r rrsampledZsampler_outputrrrforwards  zSampler.forwardtempcCs||jddS)Ndim)Zdiv_r)r rrrrrapply_temperature)szSampler.apply_temperaturecCs|jdddS)Nrr)Zargmaxviewr rrrr greedy_sample0szSampler.greedy_samplecCsl||}|jdus J|||j}|jdur|||j}|||j|j|j}t |jt k||}|Sr) r"Z temperaturermin_p apply_min_pr generatorsZtop_kZtop_prwhere _SAMPLING_EPS)r rrZgreedy_sampledZrandom_sampledrrrrr3s   zSampler.samplecCs|jdtjdS)Nr)rZdtype)Z log_softmaxrrr!rrrcompute_logprobsOszSampler.compute_logprobslogprobs num_logprobs token_idsc Csttj||dd\}}|d}|d|}||kd}tj||fdd}tj||fdd}|tj}t|||S)a Gather logprobs for topk and sampled/prompt token. Args: logits: (num tokens) x (vocab) tensor num_logprobs: minimum number of logprobs to retain per token token_ids: prompt tokens (if prompt logprobs) or sampled tokens (if sampled logprobs); 1D token ID tensor with (num tokens) elements Returns: Top-k int indices tensor, (num tokens) x (num_logprobs + 1) Top-k float logprobs tensor, (num tokens) x (num_logprobs + 1) Sampled token rank tensor, (num tokens) rrr) rZtopkrgathersumcatrZint32r) r r)r*r+Z topk_logprobsZ topk_indicesZtoken_logprobsZ token_ranksindicesrrrgather_logprobsRs    zSampler.gather_logprobsr#cCsPtjjj|dd}tj|ddd}|d|}||k}||td |S)zI Filters logits using adaptive probability thresholding. rrT)rZkeepdimrinf)rnnZ functionalZsoftmaxZamaxrZ masked_fill_float)r rr#Zprobability_valuesZmax_probabilitiesZadjusted_min_pZvalid_token_maskrrrr$~s zSampler.apply_min_p)__name__ __module__ __qualname__r rZTensorrrrrr"rr(intrr0r$ __classcell__rrr rrsR     ,r)__doc__rZtorch.nnr2Zvllm.v1.outputsrrZ$vllm.v1.sample.ops.topk_topp_samplerrZvllm.v1.sample.tpu.metadatarr'Modulerrrrrs