o
�J�hoP �
@ s� d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
m Z m Z m
Z
ddlmZmZmZmZ ddlmZ ddlmZmZmZ dd lmZ dd
lmZ dd l m Z m Z e� e �Z!d e"fd
d�Z#dd� Z$dd� Z%d e"de&fdd�Z'de(de
e e f fdd�Z)de"de"fdd�Z*d e"de"fd d �Z+d e"de"fd d!�Z,d e"de"fd"d#�Z-d e"de"fd$d%�Z.d!e"d&e"de"fd'd(�Z/d!e"d)e"de"fd*d+�Z0d,ede"fd-d.�Z1d/e"de"fd0d1�Z2d2e"de
e"e e" f fd3d2�Z3d/e"de fd4d5�Z4d/e"de fd6d7�Z5d/e"de"fd8d9�Z6d:e
d;e
d<e"ddfd=d>�Z7d:e
d;e
de"d?e"ddf
d@dA�Z8dBe
de e e
fdCdD�Z9d e"de(fdEdF�Z:d e"de"fdGdH�Z;de e fdIdJ�Z<dS )KzBThis module contains all non-cipher related data extraction logic.� N)� OrderedDict)�datetime)�Any�Dict�List�Optional�Tuple)�parse_qs�quote� urlencode�urlparse)�Cipher)�HTMLParseError�LiveStreamError�RegexMatchError�� regex_search)�YouTubeMetadata)�parse_for_object�parse_for_all_objects�
watch_htmlc C s> zt �d| �}|rt�|�d��W S W dS ty Y dS w )z�Extract publish date and return it as a datetime object
:param str watch_html:
The html contents of the watch page.
:rtype: datetime
:returns:
Publish date of the video as a datetime object with timezone.
z\(?<=itemprop=\"datePublished\" content=\")\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:\d{2}r N)�re�searchr �
fromisoformat�group�AttributeError)r �result� r �OC:\pinokio\api\whisper-webui.git\app\env\lib\site-packages\pytubefix\extract.py� publish_date s �� �r c C s" dg}|D ] }|| v r dS qdS )z�Check if live stream recording is available.
:param str watch_html:
The html contents of the watch page.
:rtype: bool
:returns:
Whether or not the content is private.
z,This live stream recording is not available.FTr )r Zunavailable_strings�stringr r r �recording_available& s
��r! c C s$ g d�}|D ] }|| v r dS qdS )z�Check if content is private.
:param str watch_html:
The html contents of the watch page.
:rtype: bool
:returns:
Whether or not the content is private.
)zFThis is a private video. Please sign in to verify that you may see it.z "simpleText":"Private video"zThis video is private.TFr )r Zprivate_stringsr r r r �
is_private8 s �r" �returnc C s* z
t d| dd� W dS ty Y dS w )z�Check if content is age restricted.
:param str watch_html:
The html contents of the watch page.
:rtype: bool
:returns:
Whether or not the content is age restricted.
zog:restrictions:ager �r FT)r r )r r r r �is_age_restrictedL s ��r% �player_responsec C sh | � di �}d| v rd| d v rdS d|v r/d|v r#|d |d gfS d|v r/|d |d fS ddgfS ) a� Return the playability status and status explanation of a video.
For example, a video may have a status of LOGIN_REQUIRED, and an explanation
of "This is a private video. Please sign in to verify that you may see it."
This explanation is what gets incorporated into the media player overlay.
:param str player_response:
Content of the player's response.
:rtype: bool
:returns:
Playability status and reason of the video.
�playabilityStatusZ videoDetailsZisLive)Z LIVE_STREAMzVideo is a live stream.�status�reason�messagesN)�get)r& Z status_dictr r r �playability_status\ s
r, �jsc C � t d| dd�S )NzsignatureTimestamp:(\d*)� r$ r )r- r r r �signature_timestampz � r0 �response_contextc C r. )Nz>visitor_data[',\"\s]+value['\"]:\s?['\"]([a-zA-Z0-9_%-]+)['\"]r/ r$ r )r2 r r r � visitor_data~ r1 r3 �urlc C r. )ar Extract the ``video_id`` from a YouTube url.
This function supports the following patterns:
- :samp:`https://youtube.com/watch?v={video_id}`
- :samp:`https://youtube.com/embed/{video_id}`
- :samp:`https://youtu.be/{video_id}`
:param str url:
A YouTube url containing a video id.
:rtype: str
:returns:
YouTube video id.
z (?:v=|\/)([0-9A-Za-z_-]{11}).*r/ r$ r )r4 r r r �video_id� s r5 c C s t j�| �}t|j�d d S )ao Extract the ``playlist_id`` from a YouTube url.
This function supports the following patterns:
- :samp:`https://youtube.com/playlist?list={playlist_id}`
- :samp:`https://youtube.com/watch?v={video_id}&list={playlist_id}`
:param str url:
A YouTube url containing a playlist id.
:rtype: str
:returns:
YouTube playlist id.
�listr )�urllib�parser r �query)r4 �parsedr r r � playlist_id� s r; c C s� g d�}|D ]5}t �|�}|�| �}|r;t�d|� |�d�}|�d�}|dkr2d|� d|� � S d|� |� � S qtddd ��)
a Extract the ``channel_name`` or ``channel_id`` from a YouTube url.
This function supports the following patterns:
- :samp:`https://youtube.com/c/{channel_name}/*`
- :samp:`https://youtube.com/channel/{channel_id}/*
- :samp:`https://youtube.com/u/{channel_name}/*`
- :samp:`https://youtube.com/user/{channel_id}/*
- :samp:`https://youtube.com/@{channel_id}/*
:param str url:
A YouTube url containing a channel name.
:rtype: str
:returns:
YouTube channel name.
)z (?:\/(c)\/([%\d\w_\-]+)(\/.*)?)z%(?:\/(channel)\/([%\w\d_\-]+)(\/.*)?)z (?:\/(u)\/([%\d\w_\-]+)(\/.*)?)z"(?:\/(user)\/([%\w\d_\-]+)(\/.*)?)z (?:\/(\@)([%\d\w_\-\.]+)(\/.*)?)�"finished regex search, matched: %sr/ � �@�/� channel_name�patterns��caller�pattern�r �compiler �logger�debugr r )r4 rA rD �regex�function_matchZ uri_styleZuri_identifierr r r r@ � s
.��r@ � watch_urlc C s* t d| fddt|�fddddg�}t|�S )a Construct the video_info url.
:param str video_id:
A YouTube video identifier.
:param str watch_url:
A YouTube watch url.
:rtype: str
:returns:
:samp:`https://youtube.com/get_video_info` with necessary GET
parameters.
r5 )�ps�default�eurl)�hl�en_US��html5�1��cZTVHTML5�Zcverz
7.20201028)r r
�_video_info_url)r5 rK �paramsr r r �video_info_url� s
�� rY �
embed_htmlc C s\ z t d|dd�}W n ty d}Y nw d| � �}td| fd|fd|fd d
d g�}t|�S ) a< Construct the video_info url.
:param str video_id:
A YouTube video identifier.
:param str embed_html:
The html contents of the embed page (for age restricted videos).
:rtype: str
:returns:
:samp:`https://youtube.com/get_video_info` with necessary GET
parameters.
z"sts"\s*:\s*(\d+)r/ r$ � z!https://youtube.googleapis.com/v/r5 rN �stsrQ rT rV )r r r rW )r5 rZ r\ rN rX r r r � video_info_url_age_restricted� s �
��
r] rX c C s dt | �� �S )Nz'https://www.youtube.com/get_video_info?)r )rX r r r rW r1 rW �htmlc C s> z
t | �d d }W n ttfy t| �}Y nw d|� �S )z�Get the base JavaScript url.
Construct the base JavaScript url, which contains the decipher
"transforms".
:param str html:
The html contents of the watch page.
�assetsr- zhttps://youtube.com)�get_ytplayer_config�KeyErrorr �get_ytplayer_js)r^ Zbase_jsr r r �js_url s �
rc �mime_type_codecc C sL d}t �|�}|�| �}|std|d��|�� \}}|dd� |�d�D �fS )a� Parse the type data.
Breaks up the data in the ``type`` key of the manifest, which contains the
mime type and codecs serialized together, and splits them into separate
elements.
**Example**:
mime_type_codec('audio/webm; codecs="opus"') -> ('audio/webm', ['opus'])
:param str mime_type_codec:
String containing mime type and codecs.
:rtype: tuple
:returns:
The mime type and a list of codecs.
z,(\w+\/\w+)\;\scodecs=\"([a-zA-Z-0-9.,\s]*)\"rd rB c S s g | ]}|� � �qS r )�strip)�.0rU r r r �
<listcomp>1 � z#mime_type_codec.<locals>.<listcomp>�,)r rF r r �groups�split)rd rD rI �results� mime_type�codecsr r r rd s
c C s` dg}|D ]$}t �|�}|�| �}|r)t�d|� |�d�}t�d| � | S qtddd��)z�Get the YouTube player base JavaScript path.
:param str html
The html contents of the watch page.
:rtype: str
:returns:
Path to YouTube's base.js file.
z'(/s/player/[\w\d]+/[\w\d_/.]+/base\.js)r< r/ z player JS: rb �js_url_patternsrB rE )r^ ro rD rI rJ Z yt_player_jsr r r rb 4 s
�
��rb c C s� t �d� ddg}|D ])}zt| |�W S ty4 } zt �d|� �� t �|� W Y d}~q d}~ww dg}|D ]}zt| |�W S tyM Y q:w tddd ��)
a� Get the YouTube player configuration data from the watch html.
Extract the ``ytplayer_config``, which is json data embedded within the
watch html and serves as the primary source of obtaining the stream
manifest data.
:param str html:
The html contents of the watch page.
:rtype: str
:returns:
Substring of the html containing the encoded manifest data.
z finding initial function namezytplayer\.config\s*=\s*� ytInitialPlayerResponse\s*=\s*zPattern failed: Nz,yt\.setConfig\(.*['\"]PLAYER_CONFIG['\"]:\s*r` z#config_patterns, setconfig_patternsrB )rG rH r r r )r^ Zconfig_patternsrD �eZsetconfig_patternsr r r r` N s0
�
��
� ��r` c C s^ i }ddg}|D ] }zt | |�}|D ]}|�|� qW q ty$ Y qw |r)|S tddd��)a; Get the entirety of the ytcfg object.
This is built over multiple pieces, so we have to find all matches and
combine the dicts together.
:param str html:
The html contents of the watch page.
:rtype: str
:returns:
Substring of the html containing the encoded manifest data.
z
ytcfg\s=\sz ytcfg\.set\(� get_ytcfgZytcfg_pattenrsrB )r �updater r )r^ ZytcfgZytcfg_patternsrD Z
found_objects�objr r r rr | s$ �
� ��rr �stream_manifest�vid_info�po_tokenc C s� t �d� t| �D ]R\}}z|d }W n ty+ |�di ��d�}|r)td��Y nw t|�}tt|�j�}dd� |� � D �}||d< |j
� d |j � |j � d
t
|�� �}|| | d< q d S ) z�Apply the proof of origin token to the stream manifest
:param dict stream_manifest:
Details of the media streams available.
:param str po_token:
Proof of Origin Token.
zApplying poTokenr4 r' �liveStreamability�UNKNOWNc S � i | ] \}}||d �qS �r r �rf �k�vr r r �
<dictcomp>� � �z"apply_po_token.<locals>.<dictcomp>�pot�://�?N)rG rH � enumeratera r+ r r r r9 �items�scheme�netloc�pathr ) ru rv rw �i�streamr4 � live_stream�
parsed_url� query_paramsr r r �apply_po_token� s*
����"�r� �url_jsc C sv t ||d�}t� }t| �D ]�\}}z|d }W n ty/ |�di ��d�} | r-td��Y nw t|�}
tt|�j�} dd� | � � D �} d|v sTd |vrZd
|v sTd |v rZt
� d � n|j |d d
�} t
� d|d � | | d< d| �
� v r�| d }
t
� d|
� �� |
|vr�|�|
�||
< nt
� d� ||
}|| d< t
� d|� �� |
j� d|
j� |
j� dt| �� �}|| | d< q
dS )a Apply the decrypted signature to the stream manifest.
:param dict stream_manifest:
Details of the media streams available.
:param str js:
The contents of the base.js asset file.
:param str url_js:
Full base.js url
)r- rc r4 r' rx ry c S rz r{ r r| r r r r � r� z#apply_signature.<locals>.<dictcomp>� signature�sz&sig=z&lsig=z signature found, skip decipher)Zciphered_signaturez+finished descrambling signature for itag=%s�itag�sig�nzParameter n is: z%Parameter n found skipping decryptionzParameter n deciphered: r� r� N)r
�dictr� ra r+ r r r r9 r� rG rH �
get_signature�keysZget_throttlingr� r� r� r )ru rv r- r� �cipherZ discovered_nr� r� r4 r� r� r� r� Z initial_n�new_nr r r �apply_signature� sL
���� �
"�r� � stream_datac C s� d| v rdS g }d| � � v r|�| d � d| � � v r"|�| d � |D ]@}d|vrId|v rIt|d �}|d d |d< |d d |d< d|d < nd|vr[d|vr[| d
|d< d |d < |�d �d
k|d< q$t�d� |S )a- Apply various in-place transforms to YouTube's media stream data.
Creates a ``list`` of dictionaries by string splitting on commas, then
taking each list item, parsing it as a query string, converting it to a
``dict`` and unquoting the value.
:param dict stream_data:
Dictionary containing query string encoded values.
**Example**:
>>> d = {'foo': 'bar=1&var=test,em=5&t=url%20encoded'}
>>> apply_descrambler(d, 'foo')
>>> print(d)
{'foo': [{'bar': '1', 'var': 'test'}, {'em': '5', 't': 'url encoded'}]}
r4 N�formatsZadaptiveFormatsZsignatureCipherr r� F�is_sabrZserverAbrStreamingUrlT�typeZFORMAT_STREAM_TYPE_OTF�is_otfzapplying descrambler)r� �extendr r+ rG rH )r� r� �dataZ
cipher_urlr r r �apply_descrambler s&
r� c C �@ ddg}|D ]}zt | |�W S ty Y qw tddd��)z�Extract the ytInitialData json from the watch_html page.
This mostly contains metadata necessary for rendering the page on-load,
such as video information, copyright notices, etc.
@param watch_html: Html of the watch page
@return:
z'window\[['\"]ytInitialData['\"]]\s*=\s*zytInitialData\s*=\s*� initial_dataZinitial_data_patternrB �r r r �r rA rD r r r r� 5 s
� � r� c C r� )a Extract the ytInitialPlayerResponse json from the watch_html page.
This mostly contains metadata necessary for rendering the page on-load,
such as video information, copyright notices, etc.
@param watch_html: Html of the watch page
@return:
z1window\[['\"]ytInitialPlayerResponse['\"]]\s*=\s*rp �initial_player_responseZ initial_player_response_patternrB r� r� r r r r� K s
� ��r� c C st z| d d d d d d d d d d }W n t tfy' tg � Y S w td d
� |�}d d � |D �}t|�S )
u< Get the informational metadata for the video.
e.g.:
[
{
'Song': '강남스타일(Gangnam Style)',
'Artist': 'PSY',
'Album': 'PSY SIX RULES Pt.1',
'Licensed to YouTube by': 'YG Entertainment Inc. [...]'
}
]
:rtype: YouTubeMetadata
�contentsZtwoColumnWatchNextResultsrl r/ ZvideoSecondaryInfoRendererZmetadataRowContainerZ metadataRowContainerRenderer�rowsc S s d| � � v S )N�metadataRowRenderer)r� )�xr r r �<lambda>~ s zmetadata.<locals>.<lambda>c S s g | ]}|d �qS )r� r )rf r� r r r rg � rh z metadata.<locals>.<listcomp>)ra �
IndexErrorr �filter)r� Z
metadata_rowsr r r �metadatad s6
�������� ��r� )=�__doc__�logging� urllib.parser7 r � collectionsr r �typingr r r r r r r
r r Zpytubefix.cipherr
Zpytubefix.exceptionsr r r �pytubefix.helpersr Zpytubefix.metadatar Zpytubefix.parserr r � getLogger�__name__rG �strr r! r" �boolr% r� r, r0 r3 r5 r; r@ rY r] rW rc rd rb r` rr r� r� r� r� r� r� r r r r �<module> sL
% ." #H,