3

�]9Y��@s0dd
lZdd
l
Z
ddlmZ
Gdd�de�Zd
S)�N�
)
�ProbingStatec@sneZ
dZd
Zddd�
Zdd�Zedd��
Zd	d
�Zedd��
Z	d
d�Z
edd��
Zedd��
Z
edd��
ZdS)�
CharSetProbergffffff�?NcCsd|_|
|_
tjt�
|_dS)
N)�_state�lang_filter�loggingZ	getLogger�__name__Zlogger)�selfr�r
�#/usr/lib/python3.6/charsetprober.py�__init__'s


zCharSetProber.__init__c

Cstj
|_dS)
N)rZ	DETECTINGr)
r	r
r
r�reset,s
zCharSetProber.resetc


CsdS)
Nr
)
r	r
r
r�charset_name/szCharSetProber.charset_namec
CsdS)
Nr
)r	�bufr
r
r�feed3s
zCharSetProber.feedc


Cs|jS)
N)
r)
r	r
r
r�state6szCharSetProber.statec


Csd
S)Ngr
)
r	r
r
r�get_confidence:s
zCharSetProber.get_confidencec

Cstj
d
d|�}|S)Ns([-])+�
 )�re�sub)
rr
r
r�filter_high_byte_only=s
z#CharSetProber.filter_high_byte_onlyc
Csbt�}
t
jd
|�}xJ|D]B}|
j|dd��

|dd�}|j�rP|dkrPd}|
j|�

qW|
S)u9
        We define three types of bytes:
        alphabet: english alphabets [a-zA-Z]
        international: international characters [€-ÿ]
        marker: everything else [^a-zA-Z€-ÿ]

        The input buffer can be thought to contain a series of words delimited
        by markers. This function works to filter all words that contain at
        least one international character. All contiguous sequences of markers
        are replaced by a single space ascii character.

        This filter applies to all scripts which do not use English characters.
        s%[a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?Nr�
�r���r)�	bytearrayr�findall�extend�isalpha)r�filteredZwordsZwordZ	last_charr
r
r�filter_international_wordsBs





z(CharSetProber.filter_international_wordsc
Cs�t�}
d
}d}x�t
t|�
�
D]r}|||d�}|dkr>d
}n|dkrJd}|dkr|j�r||kr�|r�|
j|||��

|
jd�

|d}qW|s�|
j||d	��

|
S)
a�

        Returns a copy of ``buf`` that retains only the sequences of English
        alphabet and high byte characters that are not between <> characters.
        Also retains English alphabet and high byte characters immediately
        before occurrences of >.

        This filter can be applied to all scripts which contain both English
        characters and extended ASCII characters, but is currently only used by
        ``Latin1Prober``.
        Fr
r�
>�
<TrrN)r�range�lenrr)rrZin_tag�prevZcurrZbuf_charr
r
r�filter_with_english_lettersgs"






z)CharSetProber.filter_with_english_letters)
N)r�
__module__�__qualname__ZSHORTCUT_THRESHOLDrr
�propertyrrrr�staticmethodrrr$r
r
r
rr#s
%r)rrZenumsr�objectrr
r
r
r�<module>s