o
�J�h�- � @ s� zd dl ZW n
ey d dlZY nw d dlZd dlZd dlmZ d dlmZ d dl m
Z
d dl m Z d dl
mZmZmZmZmZmZ d dlmZ dd lmZmZmZmZmZm Z e
e d
�d e d e fd
d��Z e
e d
�d e d e fdd��Z e
e d
�d e d ee fdd��Z!e
e d
�d e d e fdd��Z"e
e d
�d e d e fdd��Z#e
e d
�d e d e fdd��Z$e
e d
�d e d e fdd��Z%e
e d
�d e d e fdd ��Z&e
e d
�d e d e fd d ��Z'e
e d
�d e d e fd d ��Z(d e d e fd!d"�Z)e
e d
�d e d e fd#d$��Z*e
e d
�d e d e fd%d&��Z+e
e d
�d e d e fd'd(��Z,e
e d
�d e d e fd)d*��Z-e
e d
�d e d e fd+d,��Z.e
e/e�d
�d-e d e fd.d/��Z0e
e d
�d e d e fd0d1��Z1d_d3e2d4e3d ee fd5d6�Z4e
d7d
�d8e d e fd9d:��Z5d3e2d eee e2f fd;d<�Z6d=e d e fd>d?�Z7d`dAe dBe d e fdCdD�Z8dEe d ee fdFdG�Z9dHe dIe d e:fdJdK�Z;dHe dIe d e fdLdM�Z<dNej=dOfd8e dPe3dQe d dfdRdS�Z> dadTe2dUe dVe?dWe3dXe dYe dZe2d[e d\ee d ee ddf fd]d^�Z@dS )b� N)�IncrementalDecoder)�aliases)� lru_cache)�findall)� Generator�List�Optional�Set�Tuple�Union)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION)�maxsize� character�returnc C sT zt �| �}W n
ty Y dS w d|v p)d|v p)d|v p)d|v p)d|v p)d|v S )NFz
WITH GRAVEz
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
WITH TILDE�� unicodedata�name�
ValueError�r � description� r �VC:\pinokio\api\whisper-webui.git\app\env\lib\site-packages\charset_normalizer\utils.py�is_accentuated s ������r c C s. t �| �}|s | S |�d�}tt|d d��S )N� r � )r �
decomposition�split�chr�int)r Z
decomposed�codesr r r �
remove_accent, s
r' c C s. t | �}t�� D ] \}}||v r| S qdS )zK
Retrieve the Unicode range official name from a single character.
N)�ordr �items)r Z
character_ord�
range_nameZ ord_ranger r r �
unicode_range7 s �r+ c C �* z
t �| �}W d|v S ty Y dS w )NFZLATINr r r r r �is_latinE s ��r- c C s& z| � d� W dS ty Y dS w )N�asciiFT)�encode�UnicodeEncodeError�r r r r �is_asciiN s ��r2 c C s2 t �| �}d|v r dS t| �}|d u rdS d|v S )N�PTF� Punctuation�r �categoryr+ �r �character_category�character_ranger r r �is_punctuationW s
r: c C s: t �| �}d|v s
d|v rdS t| �}|d u rdS d|v S )N�S�NTF�Formsr5 r7 r r r � is_symbolf s
r> c C s t | �}|d u r
dS d|v S )NF� Emoticons)r+ )r r9 r r r � is_emoticonu s r@ c C s& | � � s| dv r
dS t�| �}d|v S )N> u ｜�<�>�+�,�;T�Z)�isspacer r6 �r r8 r r r � is_separator s
rI c C s | � � | �� kS �N)�islower�isupperr1 r r r �is_case_variable� s rM c C s t �| �}|dkS )NZCo)r r6 rH r r r �is_private_use_only� s
rN c C r, )NF�CJKr �r Zcharacter_namer r r �is_cjk� � ��rQ c C r, )NFZHIRAGANAr rP r r r � is_hiragana� rR rS c C r, )NFZKATAKANAr rP r r r � is_katakana� rR rT c C r, )NFZHANGULr rP r r r � is_hangul� rR rU c C r, )NFZTHAIr rP r r r �is_thai� rR rV r* c s t � fdd�tD ��S )Nc 3 s � | ]}|� v V qd S rJ r )�.0�keyword�r* r r � <genexpr>� s � z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr rY r rY r �is_unicode_range_secondary� s r\ c C s( | � � du o| �� du o| dko| dkS )NF�u )rG � isprintabler1 r r r �is_unprintable� s
���r_ � �sequence� search_zonec C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkr$dS |D ]'}|�� � dd�}t
� � D ]\}}||krB| S ||krL| S q4q&dS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
Nr. �ignore��errorsr �-�_) �
isinstance�bytes� TypeError�lenr r �min�decode�lower�replacer r) )ra rb Zseq_len�results�specified_encoding�encoding_alias�
encoding_ianar r r �any_specified_encoding� s&
� ��rt � r c C s | dv pt t�d�| ��jt�S )zQ
Verify is a specific encoding is a multi byte one based on it IANA name
> � utf_16_be�utf_7�utf_32� utf_32_le�utf_16� utf_16_le� utf_32_be� utf_8_sig�utf_8� encodings.{})�
issubclass� importlib�
import_module�formatr r )r r r r �is_multi_byte_encoding� s
��r� c C sJ t D ] }t | }t|t�r|g}|D ]}| �|�r!||f S qqdS )z9
Identify and extract SIG/BOM in given sequence.
)N� )r rh ri �
startswith)ra �
iana_encodingZmarksZmarkr r r �identify_sig_or_bom s
��r� r� c C s | dvS )N> rz rx r )r� r r r �should_strip_sig_or_bom s r� T�cp_name�strictc C sL | � � �dd�} t�� D ]\}}| ||fv r| S q |r$td�| ���| S )Nrf rg z Unable to retrieve IANA for '{}')rn ro r r) r r� )r� r� rr rs r r r � iana_name! s �r� �decoded_sequencec C s4 t � }| D ]}t|�}|d u rq|�|� qt|�S rJ )�setr+ �add�list)r� �rangesr r9 r r r �
range_scan1 s r� � iana_name_a� iana_name_bc C s� t | �st |�r
dS t�d�| ��j}t�d�|��j}|dd�}|dd�}d}td�D ]}t|g�}|�|�|�|�krA|d7 }q,|d S ) Ng r rc rd r �� r
�� )r� r� r� r� r �rangeri rm ) r� r� Z decoder_aZ decoder_bZid_aZid_b�character_match_count�iZ
to_be_decodedr r r �
cp_similarity? s* ����
�r� c C s | t v o |t | v S )z�
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )r� r� r r r �
is_cp_similarX s
�r� �charset_normalizerz)%(asctime)s | %(levelname)s | %(message)s�level�
format_stringc C s: t �| �}|�|� t �� }|�t �|�� |�|� d S rJ )�logging� getLogger�setLevel�
StreamHandler� setFormatter� Formatter�
addHandler)r r� r� �logger�handlerr r r �set_logging_handlerc s
r� � sequencesrs �offsets�
chunk_size�bom_or_sig_available�strip_sig_or_bom� sig_payload�is_multi_byte_decoder�decoded_payloadc c s2 � |r |du r |D ]} || | | � }
|
s d S |
V q d S |D ]v} | | } | t | �d kr/q | | | | � } |rA|du rA|| } | j||rHdndd�}
|r�| dkr�| | dkr�t|d�}
|r�|
d |
� |vr�t| | d d
�D ]#}| || � } |r�|du r�|| } | j|dd�}
|
d |
� |v r� nqo|
V q d S ) NF� rc r� rd r ru r! � �����)rk rm rl r� )r� rs r� r� r� r� r� r� r� r� �chunkZ chunk_endZ cut_sequenceZchunk_partial_size_chk�jr r r �cut_sequence_chunksq sD � �
�
� ��r� )r` )TrJ )AZ unicodedata2r � ImportErrorr� r� �codecsr Zencodings.aliasesr � functoolsr �rer �typingr r r r r
r Z_multibytecodecr �constantr r r r r r �str�boolr r' r+ r- r2 r: r> r@ rI rM rN rQ rS rT rU rV rk r\ r_ ri r% rt r� r� r� r� r� �floatr� r� �INFOr� r� r� r r r r �<module> s� �
����
���������� �
�