Your IP : 3.15.186.27
o
6��f�$ � @ s\ d Z ddlZddlZddlZdgZe�dd�ZG dd� d�ZG dd� d�Z G d d
� d
�Z
dS ) a% robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://www.robotstxt.org/norobots-rfc.txt
� N�RobotFileParser� RequestRatezrequests secondsc @ sr e Zd ZdZd dd�Zdd� Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� ZdS ) r zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
� c C s2 g | _ g | _d | _d| _d| _| �|� d| _d S )NFr )�entries�sitemaps�
default_entry� disallow_all� allow_all�set_url� last_checked��self�url� r �9/opt/alt/python310/lib64/python3.10/urllib/robotparser.py�__init__ s
zRobotFileParser.__init__c C s | j S )z�Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r �r
r r r �mtime% s zRobotFileParser.mtimec C s ddl }|� � | _dS )zYSets the time the robots.txt file was last fetched to the
current time.
r N)�timer )r
r r r r �modified. s zRobotFileParser.modifiedc C s& || _ tj�|�dd� \| _| _dS )z,Sets the URL referring to a robots.txt file.� � N)r �urllib�parse�urlparseZhost�pathr r r r r
6 s zRobotFileParser.set_urlc
C s� z t j�| j�}W n@ t jjyI } z2|jdv r d| _n|jdkr0|jdk r>d| _W Y d}~dS W Y d}~dS W Y d}~dS W Y d}~dS d}~ww |� � }| �
|� d�� � � dS )z4Reads the robots.txt URL and feeds it to the parser.)i� i� Ti� i� Nzutf-8)
r ZrequestZurlopenr �errorZ HTTPError�coder r �readr �decode�
splitlines)r
�f�err�rawr r r r ; s
����zRobotFileParser.readc C s2 d|j v r| jd u r|| _d S d S | j�|� d S �N�*)�
useragentsr r �append)r
�entryr r r �
_add_entryH s
�zRobotFileParser._add_entryc C sJ d}t � }| �� |D �]
}|s(|dkrt � }d}n|dkr(| �|� t � }d}|�d�}|dkr7|d|� }|�� }|s>q |�dd�}t|�dk�r|d �� �� |d< tj �
|d �� �|d< |d dkr~|dkrs| �|� t � }|j � |d � d}q |d dkr�|dkr�|j
� t|d d �� d}q |d d
kr�|dkr�|j
� t|d d �� d}q |d d kr�|dkr�|d �� �� r�t|d �|_d}q |d d
k�r|dk�r|d �d�}t|�dk�r|d �� �� �r|d �� �� �rtt|d �t|d ��|_d}q |d dk�r| j� |d � q |dk�r#| �|� dS dS )z�Parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r � �#N�:z
user-agentZdisallowFZallowTz crawl-delayz request-rate�/Zsitemap)�Entryr r) �find�strip�split�len�lowerr r �unquoter& r' � rulelines�RuleLine�isdigit�int�delayr �req_rater )r
�lines�stater( �line�iZnumbersr r r r Q sv
� � �
� ��
�zRobotFileParser.parsec C s� | j rdS | jr
dS | jsdS tj�tj�|��}tj�dd|j|j |j
|j f�}tj� |�}|s3d}| j
D ]}|�|�rD|�|� S q6| jrN| j�|�S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr r- )r r r r r r r4 �
urlunparser �paramsZqueryZfragment�quoter �
applies_to� allowancer )r
� useragentr Z
parsed_urlr( r r r � can_fetch� s( �
� zRobotFileParser.can_fetchc C �> | � � sd S | jD ] }|�|�r|j S q | jr | jjS d S �N)r r rB r9 r �r
rD r( r r r � crawl_delay� �
�zRobotFileParser.crawl_delayc C rF rG )r r rB r: r rH r r r � request_rate� rJ z RobotFileParser.request_ratec C s | j sd S | j S rG )r r r r r � site_maps� s zRobotFileParser.site_mapsc C s, | j }| jd ur|| jg }d�tt|��S )Nz
)r r �join�map�str)r
r r r r �__str__� s
zRobotFileParser.__str__N)r )�__name__�
__module__� __qualname__�__doc__r r r r
r r) r rE rI rK rL rP r r r r r s
I
c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) r6 zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s< |dkr|sd}t j�t j�|��}t j�|�| _|| _d S )Nr T)r r r? r rA r rC )r
r rC r r r r � s
zRuleLine.__init__c C s | j dkp
|�| j �S r$ )r �
startswith)r
�filenamer r r rB � s zRuleLine.applies_toc C s | j rdndd | j S )NZAllowZDisallowz: )rC r r r r r rP � s zRuleLine.__str__N)rQ rR rS rT r rB rP r r r r r6 � s
r6 c @ s0 e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
S ) r. z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d | _d | _d S rG )r& r5 r9 r: r r r r r � s
zEntry.__init__c C s� g }| j D ]
}|�d|� �� q| jd ur |�d| j� �� | jd ur3| j}|�d|j� d|j� �� |�tt| j �� d�
|�S )Nz User-agent: z
Crawl-delay: zRequest-rate: r- �
) r& r' r9 r: ZrequestsZseconds�extendrN rO r5 rM )r
Zret�agentZrater r r rP � s
z
Entry.__str__c C sF |� d�d �� }| jD ]}|dkr dS |�� }||v r dS q dS )z2check if this entry applies to the specified agentr- r r% TF)r1 r3 r& )r
rD rY r r r rB � s
�zEntry.applies_toc C s$ | j D ] }|�|�r|j S qdS )zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)r5 rB rC )r
rV r= r r r rC
s
�zEntry.allowanceN)rQ rR rS rT r rP rB rC r r r r r. � s
r. ) rT � collectionsZ urllib.parser Zurllib.request�__all__�
namedtupler r r6 r. r r r r �<module> s B