o �J�hq��@s�dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl mZmZm Z mZmZmZmZddlZddlmZmZmZmZmZmZmZmZmZmZmZm Z e�r_ddl!Z!dd�Z"dUd e#d e$fdd�Z%dUd e#d e$fdd�Z&dd�Z'Gdd�d�Z(Gdd�de�Z)Gdd�de�Z*Gdd�de�Z+dZ,e�-de,d�Z.dd�Z/Gdd �d e�Z0Gd!d"�d"e�Z1Gd#d$�d$e�Z2Gd%d&�d&e�Z3Gd'd(�d(e�Z4d)ee5e6fd*e6fd+d,�Z7d*ee5e6ffd-d.�Z8d*ee5e6ffd/d0�Z9d1d2�Z:d*ee5e6ffd3d4�Z;Gd5d6�d6e�Z<d7d8�Z=d9d:�Z>dVd;d<�Z?Gd=d>�d>e�Z@Gd?d@�d@�ZAdAdB�ZBdCdD�ZCdEdF�ZD dWdHeEdIe#dJe$fdKdL�ZFGdMdN�dNe�ZGGdOdP�dP�ZHdUdQe5dRe$fdSdT�ZIdS)Xz6 PyTorch-independent utilities for the Trainer class. �N)�Any�Dict�List� NamedTuple�Optional�Tuple�Union�)�ExplicitEnum�is_psutil_available�is_tf_available�is_torch_available�is_torch_cuda_available�is_torch_mlu_available�is_torch_mps_available�is_torch_musa_available�is_torch_npu_available�is_torch_xla_available�is_torch_xpu_available�requires_backendscCst��d}t|�dS)zN Helper function to set worker seed during Dataloader initialization. lN)�torch�initial_seed�set_seed)�_Zworker_seed�r�XC:\pinokio\api\whisper-webui.git\app\env\lib\site-packages\transformers\trainer_utils.py�seed_worker4srF�seed� warn_onlycCs�t|�t�r1dtjd<dtjd<dtjd<dtjd<dtjd<tjd|d �dtjj_d tjj_ t �r@ddl}|jj ��dSdS) a Helper function for reproducible behavior during distributed training. See - https://pytorch.org/docs/stable/notes/randomness.html for pytorch - https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism for tensorflow �1ZCUDA_LAUNCH_BLOCKINGz:16:8�CUBLAS_WORKSPACE_CONFIGZASCEND_LAUNCH_BLOCKINGZHCCL_DETERMINISTICZFLASH_ATTENTION_DETERMINISTICT)rFrN)rr �os�environr�use_deterministic_algorithms�backends�cudnn� deterministic� benchmarkr� tensorflow�config�experimental�enable_op_determinism)rr�tfrrr�enable_full_determinism<s �r-r&cCs�t�|�tj�|�t�r t�|�tj�|�|r t�d�t �r)tj �|�t�r2tj�|�t �r;tj�|�t�rDtj�|�t�r[ddl}|j�|�|r]|jj��dSdSdS)aY Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch` and/or `tf` (if installed). Args: seed (`int`): The seed to set. deterministic (`bool`, *optional*, defaults to `False`): Whether to use deterministic algorithms where available. Can slow down training. TrN)�randomr�npr r�manual_seed�cuda�manual_seed_allr#r�mlur�musar�npur�xpurr(rr)r*r+)rr&r,rrrr\s, �rcCsL|jr$t�|�d�|�d��}|jt�|�}|t�|��||�}|S)aL Implements the NEFTune forward pass for the model using forward hooks. Note this works only for torch.nn.Embedding layers. This method is slightly adapted from the original source code that can be found here: https://github.com/neelsjain/NEFTune Simply add it to your model as follows: ```python model = ... model.embed_tokens.neftune_noise_alpha = 0.1 model.embed_tokens.register_forward_hook(neftune_post_forward_hook) ``` Args: module (`torch.nn.Module`): The embedding module where the hook is attached. Note that you need to set `module.neftune_noise_alpha` to the desired noise alpha value. input (`torch.Tensor`): The input tensor to the model. output (`torch.Tensor`): The output tensor of the model (i.e. the embeddings). r �)�trainingr�tensor�size�neftune_noise_alpha�sqrt� zeros_like�uniform_)�module�input�output�dimsZmag_normrrr�neftune_post_forward_hook~s rCc @s�eZdZdZ d deejeejfdeejeejfdeeejeejfdeeejeejffdd�Z d d �Z dd�ZdS)�EvalPredictionaz Evaluation output (always contains labels), to be used to compute metrics. Parameters: predictions (`np.ndarray`): Predictions of the model. label_ids (`np.ndarray`): Targets to be matched. inputs (`np.ndarray`, *optional*): Input data passed to the model. losses (`np.ndarray`, *optional*): Loss values computed during evaluation. N�predictions� label_ids�inputs�lossescCsf||_||_||_||_|j|jf|_|jdur!|j|jf7_|jdur1|j|jf7_dSdS�N)rErFrGrH�elements)�selfrErFrGrHrrr�__init__�s �zEvalPrediction.__init__cCs t|j�SrI)�iterrJ�rKrrr�__iter__�s zEvalPrediction.__iter__cCs(|dks|t|j�krtd��|j|S)Nrztuple index out of range)�lenrJ� IndexError)rK�idxrrr�__getitem__�s zEvalPrediction.__getitem__)NN)�__name__� __module__�__qualname__�__doc__rr/�ndarrayrrrLrOrSrrrrrD�s�� rDc@sbeZdZUeejeejfed<eeejeejfed<ee e efed<eeed<dS)�EvalLoopOutputrErF�metrics�num_samplesN) rTrUrVrr/rXr�__annotations__rr�str�float�intrrrrrY�s rYc@sVeZdZUeejeejfed<eeejeejfed<ee e efed<dS)�PredictionOutputrErFrZN)rTrUrVrr/rXrr\rrr]r^rrrrr`�s r`c@s.eZdZUeed<eed<eeefed<dS)�TrainOutput�global_step� training_lossrZN)rTrUrVr_r\r^rr]rrrrra�s ra� checkpoint�^z\-(\d+)$csFt��}�fdd�|D�}t|�dkrdStj��t|dd�d��S)Ncs4g|]}t�|�durtj�tj��|��r|�qSrI)�_re_checkpoint�searchr!�path�isdir�join)�.0rh��folderrr� <listcomp>�s $�z'get_last_checkpoint.<locals>.<listcomp>rcSstt�|��d�S)Nr)r_rfrg�groups)�xrrr�<lambda>��z%get_last_checkpoint.<locals>.<lambda>��key)r!�listdirrPrhrj�max)rm�content�checkpointsrrlr�get_last_checkpoint�s �ryc@�eZdZdZdZdZdS)�IntervalStrategy�no�steps�epochN�rTrUrV�NO�STEPS�EPOCHrrrrr{��r{c@�eZdZdZdZdZdZdS)�SaveStrategyr|r}r~�bestN)rTrUrVr�r�r��BESTrrrrr�� r�c@rz)�EvaluationStrategyr|r}r~Nrrrrrr��r�r�c@r�)�HubStrategy�end� every_saverdZall_checkpointsN)rTrUrV�ENDZ EVERY_SAVE� CHECKPOINTZALL_CHECKPOINTSrrrrr��r�r�c@sNeZdZUdZeed<eeeefed<e ee fed<dZee ed<dS)�BestRunac The best run found by a hyperparameter search (see [`~Trainer.hyperparameter_search`]). Parameters: run_id (`str`): The id of the best run (if models were saved, the corresponding checkpoint will be in the folder ending with run-{run_id}). objective (`float`): The objective that was obtained for this run. hyperparameters (`Dict[str, Any]`): The hyperparameters picked to get this run. run_summary (`Optional[Any]`): A summary of tuning experiments. `ray.tune.ExperimentAnalysis` object for Ray backend. �run_id� objective�hyperparametersN�run_summary) rTrUrVrWr]r\rr^rrrr�rrrrrr��s r�rZ�returncCsft�|�}|�dd�}|�dd�}dd�|��D�}|D]}|�|d�}qt|�dkr-|St|��S)aj The default objective to maximize/minimize when doing an hyperparameter search. It is the evaluation loss if no metrics are provided to the [`Trainer`], the sum of all metrics otherwise. Args: metrics (`Dict[str, float]`): The metrics returned by the evaluate method. Return: `float`: The objective to minimize or maximize � eval_lossNr~cSs.g|]}|�d�s|�d�s|�d�r|�qS)�_runtimeZ_per_secondZ_compilation_time)�endswith�rk�mrrrrns��z-default_compute_objective.<locals>.<listcomp>r)�copy�deepcopy�pop�keysrP�sum�values)rZ�lossr� speed_metrics�smrrr�default_compute_objectives �r�cCsVddlm}|�s Jd��|jddddd�|�d dd �|�ddd�|�d gd��d�S)Nr )�is_optuna_availablez:This function needs Optuna installed: `pip install optuna`� learning_rate��ư>�-C��6?T)�log�num_train_epochs�r�(�per_device_train_batch_size�� @�r�r�rr�)�integrationsr�� suggest_float�suggest_int�suggest_categorical)�trialr�rrr�default_hp_space_optuna)s�r�cCs^ddlm}|�s Jd��ddlm}|�dd�|�ttdd��|�dd �|�gd ��d�S)Nr )�is_ray_tune_availablez:This function needs ray installed: `pip install ray[tune]`r)�tuner�r��r�r�r�) r�r��rayr�� loguniform�choice�list�range�uniform)r�r�r�rrr�default_hp_space_ray5s �r�cCsDddd�dddd�dd d�d dd�dd d�ddd�gd�ddd�gS)Nr�r�)�minrvr��doubler�)�bounds�name�typeZtransformamtionr r�r�r_)r�r�r�r�r)�4�8�16�32�64r��categorical)Zcategorical_valuesr�r�r)r�rrr�default_hp_space_sigoptCs��r�cCsVddlm}|�s td��dddd�dd d d�ddd d�dddd�dgd�id�d�S)Nr )�is_wandb_availablez8This function needs wandb installed: `pip install wandb`r.r��minimize)r�Zgoalr�r�r�)�distributionr�rvZint_uniformr�r�r�r�r�)�method�metric� parameters)r�r��ImportError)r�r�rrr�default_hp_space_wandbPs ��r�c@r�)�HPSearchBackend�optunar��sigopt�wandbN)rTrUrVZOPTUNAZRAYZSIGOPTZWANDBrrrrr�br�r�cCs,t�rddlmm}|��dkS|dvS)z� Whether or not the current process is the local process, based on `xm.get_ordinal()` (for TPUs) first, then on `local_rank`. rN)��r)r�torch_xla.core.xla_model�core� xla_model�get_ordinal)� local_rank�xmrrr�is_main_processisr�cCsDt�rddlmm}|��S|dkr t�r ddl}|j��SdS)zg Return the number of processes launched in parallel. Works with `torch.distributed` and TPUs. rNr�r ) rr�r�r��xrt_world_sizer r�distributed�get_world_size)r�r�rrrr�total_processes_numberus r�c Cs�t��|}|�d�t|d�i}|dkr|S|dur(||}t|d�||�d�<|dur:||}t|d�||�d�<|durL||} t| d�||�d�<|S) a� Measure and return speed performance metrics. This function requires a time snapshot `start_time` before the operation to be measured starts and this function should be run immediately after the operation to be measured has completed. Args: - split: name to prefix metric (like train, eval, test...) - start_time: operation start time - num_samples: number of samples processed - num_steps: number of steps processed - num_tokens: number of tokens processed r�r�rN�Z_samples_per_secondZ_steps_per_secondZ_tokens_per_second)�time�round) �split� start_timer[� num_steps� num_tokens�runtime�result�samples_per_second�steps_per_second�tokens_per_secondrrrr��sr�c@s8eZdZdZdZdZdZdZdZdZ dZ d Zd ZdZ dS) � SchedulerTypeaL Scheduler names for the parameter `lr_scheduler_type` in [`TrainingArguments`]. By default, it uses "linear". Internally, this retrieves `get_linear_schedule_with_warmup` scheduler from [`Trainer`]. Scheduler types: - "linear" = get_linear_schedule_with_warmup - "cosine" = get_cosine_schedule_with_warmup - "cosine_with_restarts" = get_cosine_with_hard_restarts_schedule_with_warmup - "polynomial" = get_polynomial_decay_schedule_with_warmup - "constant" = get_constant_schedule - "constant_with_warmup" = get_constant_schedule_with_warmup - "inverse_sqrt" = get_inverse_sqrt_schedule - "reduce_lr_on_plateau" = get_reduce_on_plateau_schedule - "cosine_with_min_lr" = get_cosine_with_min_lr_schedule_with_warmup - "warmup_stable_decay" = get_wsd_schedule �linear�cosineZcosine_with_restarts� polynomial�constantZconstant_with_warmupZinverse_sqrtZreduce_lr_on_plateauZcosine_with_min_lrZwarmup_stable_decayN)rTrUrVrWZLINEARZCOSINEZCOSINE_WITH_RESTARTSZ POLYNOMIAL�CONSTANTZCONSTANT_WITH_WARMUPZINVERSE_SQRT�REDUCE_ON_PLATEAUZCOSINE_WITH_MIN_LRZWARMUP_STABLE_DECAYrrrrr��sr�c@sdeZdZdZdddddd�Zddd �Zd d�Zdd �Zdd�Zdd�Z dd�Z dd�Zddd�ZdS)�TrainerMemoryTrackera� A helper class that tracks cpu and gpu memory. This class will silently skip unless `psutil` is available. Install with `pip install psutil`. When a stage completes, it can pass metrics dict to update with the memory metrics gathered during this stage. Example : ```python self._memory_tracker = TrainerMemoryTracker(self.args.skip_memory_metrics) self._memory_tracker.start() # code ... metrics = {"train_runtime": 10.5} self._memory_tracker.stop_and_update_metrics(metrics) ``` At the moment GPU tracking is only for `pytorch`, but can be extended to support `tensorflow`. To understand this class' intricacies please read the documentation of [`~Trainer.log_metrics`]. �init�train�eval�test)rLr�Z_inner_training_loop�evaluate�predictFcCs�||_t�s d|_|jrdSddl}t�st�st�r&ddl}||_i|_n-t�r4ddl}||_i|_nt �rBddl}||_i|_nt �rPddl}||_i|_nd|_|��|_d|_ i|_d|_dS)NTrF)�skip_memory_metricsr�psutilrrrr�gpurrr�Process�process� cur_stage�cpu� init_reported)rKrrrrrrrL�s6 zTrainerMemoryTracker.__init__cCs>t��jjjj}||jvr|j|Std|�d|j��)z+derives the stage/caller name automaticallyzwas called from z+, but only expect to be called from one of )�inspect�currentframe�f_back�f_code�co_name�stages� ValueErrorr�)rK�callerrrr�derive_stages �z!TrainerMemoryTracker.derive_stagecCs|j��jS)z4get resident set size memory for the current process)r�memory_info�rssrNrrr�cpu_mem_usedsz!TrainerMemoryTracker.cpu_mem_usedcCs&d|_ t|��|j�|_|jsdSq)Nr�)�cpu_mem_used_peakrvr�peak_monitoringrNrrr�peak_monitor_funcs�z&TrainerMemoryTracker.peak_monitor_funccCs�|jrdS|��}|jdur|j|krdS||_t��|jdur|tj��r3|jj��|jj� �nIt �rC|jj��|jj� �n9t�rS|jj ��|jj � �n)t�rc|jj��|jj� �nt�rs|jj��|jj� �n t�r||jj� �|jdur�tj��r�|jj��|_n6t �r�|jj��|_n+t�r�|jj ��|_n t�r�|jj��|_nt�r�|jj��|_n t�r�|jj��|_|��|_d|_tj|jd�}d|_|��dS)z%start tracking for the caller's stageNT)�target)rrr�gc�collectrr1�is_available�reset_peak_memory_stats�empty_cacherr3rr4rr6rr5r�mps�memory_allocated�gpu_mem_used_at_start�current_allocated_memoryr�cpu_mem_used_at_startr� threading�Threadr�daemon�start)rK�stageZpeak_monitor_threadrrrr'"sV zTrainerMemoryTracker.startcCs.|jdur|j|krdSd|_t��|jdurUtj��r$|jj��n1t�r.|jj ��n't �r8|jj��nt�rB|jj ��nt�rL|jj��n t�rU|jj��|jdur�tj��rn|jj��|_|jj��|_nZt�r�|jj ��|_|jj ��|_nHt �r�|jj��|_|jj��|_n6t�r�|jj ��|_|jj ��|_n$t�r�|jj��|_|jj��|_nt�r�|jj��|_d|_ntd��|j|j|j|jd�|j|j<|jdur�td|j|j�|j|jd<nd|j|jd<|��|_|j|j|j|jtd|j|j�d�|j|j<d|_dS) z"stop tracking for the passed stageNFzNo available GPU device found!)�beginr��allocr�peakedz Not available)r)r�r*r+) rrrrrr1rrrr3rr4rr6rr5rrr Zgpu_mem_used_now�max_memory_allocatedZgpu_mem_used_peakr"rr!rrvrZcpu_mem_used_nowr#rr)rKr(rrr�stopZsf � � zTrainerMemoryTracker.stopcCs|jrdS|jdur|j|krdS|g}|js |�dd�d|_|D]B}dD]=}||jvrC||j|vrC|j||||�d|�d�<|jdurc||jvrc||j|vrc|j||||�d|�d�<q&q"|ddkr�|jdd |d <|jdur�|jdd |d<dSdSdS)zupdates the metricsNrr�T)r*r+Z _mem_cpu_�_deltaZ _mem_gpu_r)Zbefore_init_mem_cpuZbefore_init_mem_gpu)rrr �insertrrr)rKr(rZr�trrr�update_metrics�s,"�� z#TrainerMemoryTracker.update_metricsNcCs8|jrdS|��}|�|�|dur|�||�dSdS)z<combine stop and metrics update in one call for simpler codeN)rrr-r1)rKrZr(rrr�stop_and_update_metrics�s �z,TrainerMemoryTracker.stop_and_update_metrics�FrI) rTrUrVrWrrLrrrr'r-r1r2rrrrr��s � ) 8O)r�cCs$zt|�duWStyYdSw)zR Checks if the dataset implements __len__() and it doesn't raise an error NF)rP� TypeError)�datasetrrr� has_length�s �r6cCs�t|ttf�rt|�dd�|D��St|t�r$t|�dd�|��D��St|tj�r.|��St �rAt|t j�rA|��dkrA|��S|S)zM Recursively calls `.item()` on the element of the dictionary passed css�|]}t|�VqdSrI��denumpify_detensorizer�rrr� <genexpr>��z(denumpify_detensorize.<locals>.<genexpr>cSsi|] \}}|t|��qSrr7�rk�k�vrrr� <dictcomp>�sz)denumpify_detensorize.<locals>.<dictcomp>r ) � isinstancer��tupler��dict�itemsr/�generic�itemr r�Tensor�numel)rZrrrr8�s r8cCsFt|tj�rtt�|j�j�}|t|j�t|j �Stt�|�j�S)za Return the number of arguments of the passed function, even if it's a partial function. ) r?� functools�partialrPr � signature�funcr��args�keywords)rJ� total_argsrrr�number_of_arguments�srN��function�starting_batch_size�auto_find_batch_sizecCsL|durtjt||d�S|rttd�ddlm}|||d�Stj||d�S)a% Args: A basic decorator that will try to execute `function`. If it fails from exceptions related to out-of-memory or CUDNN, the batch size is cut in half and passed to `function`. `function` must take in a `batch_size` parameter as its first argument. function (`callable`, *optional*) A function to wrap starting_batch_size (`int`, *optional*) The batch size to try and fit into memory auto_find_batch_size (`bool`, *optional*) If False, will just execute `function` N)rQrR� accelerater)�find_executable_batch_size)rPrQ)� batch_size)rGrHrTr�accelerate.utils)rPrQrRZ%accelerate_find_executable_batch_sizerrrrTs� rTc@s(eZdZdZdZdZdZdZdZdZ dS) � FSDPOptionZ full_shardZ shard_grad_opZno_shardZhybrid_shardZhybrid_shard_zero2�offloadZ auto_wrapN) rTrUrV� FULL_SHARD� SHARD_GRAD_OP�NO_SHARD�HYBRID_SHARDZHYBRID_SHARD_ZERO2�OFFLOAD� AUTO_WRAPrrrrrW"srWc@sVeZdZdZ ddeedeefdd�Zdedefd d �Zde efdd �Z dS)�RemoveColumnsCollatorzWWrap the data collator to remove unused columns before they are passed to the collator.N� model_name�descriptioncCs(||_||_||_||_||_d|_dS)NF)� data_collator�signature_columns�loggerrar`�message_logged)rKrbrcrdr`rarrrrL/s zRemoveColumnsCollator.__init__�featurer�c s�t|t�s|S�jsR�jrR�jrRtt|��t�j��}t |�dkrR�j dur*dnd�j �d�}�j�d|�d�j�dd�|��d d�|��d �j�d��d�_�fd d�|� �D�S)Nr�zin the z setzThe following columns z) don't have a corresponding argument in `z!.forward` and have been ignored: z, z. If z are not expected by `z/.forward`, you can safely ignore this message.Tcs i|]\}}|�jvr||�qSr)rcr;rNrrr>Ls z9RemoveColumnsCollator._remove_columns.<locals>.<dictcomp>)r?rArerdr`r��setr�rcrPra�inforjrB)rKrfZignored_columnsZdset_descriptionrrNr�_remove_columns>s& ��z%RemoveColumnsCollator._remove_columns�featurescs�fdd�|D�}��|�S)Ncsg|]}��|��qSr)rj)rkrfrNrrrnOrrz2RemoveColumnsCollator.__call__.<locals>.<listcomp>)rb)rKrkrrNr�__call__Ns zRemoveColumnsCollator.__call__�NNN)rTrUrVrWrr]rLrArjrrlrrrrr_,s�� r_rt�return_is_regexcs�d}d}t|t�rtt�|��}|�ksdnd}n$�|vr!d}nt�fdd�|D��r/d}nt�fdd�|D��r>d}d}|rD||fS|S)aKA helper method to check if the passed module's key name matches any of the target modules in the optim_target_modules. Args: optim_target_modules (`Union[str, List[str]]`): A list of strings to try to match. Can be also a full string. key (`str`): A key to search any matches in optim_target_modules return_is_regex (`bool`): If set to `True`, the method will return whether the passed `optim_target_modules` is a regex or not. Returns: `bool` : True of match object if key matches any target modules from config, False or None if no match found `bool` : If the matched target module is a regex to silence out the warnings in Trainer for extra modules being found (only if `target_module_found=True` for an array of regex). FTc3s�|]}|�vVqdSrIr)rkZ target_keyrsrrr9nr:z-check_target_module_exists.<locals>.<genexpr>c3s �|]}tt�|��VqdSrI)�bool�re� fullmatch)rkZoptim_target_modulersrrr9ps�)r?r]rorprq�any)�optim_target_modulesrtrnZtarget_module_found�is_regexrrsr�check_target_module_existsSs rur3rm)NrOF)JrWr�rGrr r!r.rpr$r��typingrrrrrrr�numpyr/�utilsr rrr rrrrrrrrrrr_ror-rrCrDrYr`raZPREFIX_CHECKPOINT_DIR�compilerfryr{r�r�r�r�r]r^r�r�r�r�r�r�r�r�r�r�r�r6r8rN�callablerTrWr_rurrrr�<module>st$8 "% "�� '