Your IP : 3.129.63.214
o
6��f�$ � @ sZ d dl Z d dlZd dlZdgZe �dd�ZG dd� d�ZG dd� d�ZG dd � d �Z dS )
� N�RobotFileParser�RequestRatezrequests secondsc @ sp e Zd Z ddd�Zdd� Zdd� Zdd � Zd
d� Zdd
� Zdd� Z dd� Z
dd� Zdd� Zdd� Z
dd� ZdS )r � c C s2 g | _ g | _d | _d| _d| _| �|� d| _d S )NFr )�entries�sitemaps�
default_entry�disallow_all� allow_all�set_url�last_checked��self�url� r �9/opt/alt/python310/lib64/python3.10/urllib/robotparser.py�__init__ s
zRobotFileParser.__init__c C s | j S �N)r �r
r r r �mtime% s zRobotFileParser.mtimec C s dd l }|� � | _d S )Nr )�timer )r
r r r r �modified. s zRobotFileParser.modifiedc C s( || _ tj�|�dd� \| _| _d S )N� � )r �urllib�parse�urlparseZhost�pathr r r r r
6 s zRobotFileParser.set_urlc
C s� z t j�| j�}W n@ t jjyJ } z2|jdv rd| _n|jdkr1|jdk r?d| _W Y d }~d S W Y d }~d S W Y d }~d S W Y d }~d S d }~ww |� � }| �
|�d��� � d S )N)i� i� Ti� i� zutf-8)
r ZrequestZurlopenr �errorZ HTTPError�coder r �readr �decode�
splitlines)r
�f�err�rawr r r r ; s
����zRobotFileParser.readc C s2 d|j v r| jd u r|| _d S d S | j�|� d S �N�*)�
useragentsr r �append)r
�entryr r r �
_add_entryH s
�zRobotFileParser._add_entryc C sL d}t � }| �� |D �]
}|s)|dkrt � }d}n|dkr)| �|� t � }d}|�d�}|dkr8|d |� }|�� }|s?q|�dd�}t|�dk�r|d �� �� |d<