Your IP : 3.137.177.204
B
<_9Y�0 � @ s� d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
m Z ddl m
Z
ddlmZ dd lmZ G d
d � d e�ZdS ) a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.
:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
� N� )�CharSetGroupProber)�
InputState�LanguageFilter� ProbingState)�EscCharSetProber)� Latin1Prober)�MBCSGroupProber)�SBCSGroupProberc @ sn e Zd ZdZdZe�d�Ze�d�Ze�d�Z dddd d
d d d
d�Z
e j fdd�Z
dd� Zdd� Zdd� ZdS )�UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function
and coordinates all of the different charset probers.
To get a ``dict`` containing an encoding and its confidence, you can simply
run:
.. code::
u = UniversalDetector()
u.feed(some_bytes)
u.close()
detected = u.result
g�������?s [�-�]s (|~{)s [�-�]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8z
iso-8859-9z iso-8859-13c C sN d | _ g | _d | _d | _d | _d | _d | _|| _t� t
�| _ d | _ | �
� d S )N)�_esc_charset_prober�_charset_probers�result�done� _got_data� _input_state�
_last_char� lang_filter�logging� getLogger�__name__�logger�_has_win_bytes�reset)�selfr � r �J/opt/alt/python37/lib/python3.7/site-packages/chardet/universaldetector.py�__init__Q s zUniversalDetector.__init__c C sZ dddd�| _ d| _d| _d| _tj| _d| _| jr>| j� � x| j
D ] }|� � qFW dS )z�
Reset the UniversalDetector and all of its probers back to their
initial states. This is called by ``__init__``, so you only need to
call this directly in between analyses of different documents.
Ng )�encoding�
confidence�languageF� ) r r r r r �
PURE_ASCIIr r r r r
)r �proberr r r r ^ s
zUniversalDetector.resetc C s> | j r
dS t|�sdS t|t�s(t|�}| js�|�tj�rJdddd�| _nv|�tj tj
f�rldddd�| _nT|�d�r�dddd�| _n:|�d �r�d
ddd�| _n |�tj tj f�r�d ddd�| _d | _| jd
dk r�d | _ dS | j
tjk�r.| j�|��rtj| _
n*| j
tjk�r.| j�| j| ��r.tj| _
|dd� | _| j
tjk�r�| j�s^t| j�| _| j�|�tjk�r:| jj | j� � | jj d�| _d | _ n�| j
tjk�r:| j �s�t | j�g| _ | jt!j"@ �r�| j �#t$� � | j �#t%� � x@| j D ]6}|�|�tjk�r�|j |� � |j d�| _d | _ P �q�W | j&�|��r:d | _'dS )a�
Takes a chunk of a document and feeds it through all of the relevant
charset probers.
After calling ``feed``, you can check the value of the ``done``
attribute to see if you need to continue feeding the
``UniversalDetector`` more data, or if it has made a prediction
(in the ``result`` attribute).
.. note::
You should always call ``close`` when you're done feeding in your
document if ``done`` is not already ``True``.
Nz UTF-8-SIGg �?� )r r r zUTF-32s �� zX-ISO-10646-UCS-4-3412s ��zX-ISO-10646-UCS-4-2143zUTF-16Tr ���)(r �len�
isinstance� bytearrayr �
startswith�codecs�BOM_UTF8r � BOM_UTF32_LE� BOM_UTF32_BE�BOM_LE�BOM_BEr r r" �HIGH_BYTE_DETECTOR�search� HIGH_BYTE� ESC_DETECTORr Z ESC_ASCIIr r r �feedr ZFOUND_IT� charset_name�get_confidencer r
r r ZNON_CJK�appendr
r �WIN_BYTE_DETECTORr )r Zbyte_strr# r r r r4 o s|
zUniversalDetector.feedc C s� | j r | jS d| _ | js&| j�d� n�| jtjkrBdddd�| _n�| jtjkr�d}d}d}x,| j D ]"}|slqb|�
� }||krb|}|}qbW |r�|| j kr�|j }|j �
� }|�
� }|�d �r�| jr�| j�||�}|||jd�| _| j�� tjk�rz| jd
dk�rz| j�d � xn| j D ]d}|�s �qt|t��rZxF|jD ] }| j�d |j |j|�
� � �q4W n| j�d |j |j|�
� � �qW | jS )
z�
Stop analyzing the current document and come up with a final
prediction.
:returns: The ``result`` attribute, a ``dict`` with the keys
`encoding`, `confidence`, and `language`.
Tzno data received!�asciig �?r$ )r r r Ng ziso-8859r z no probers hit minimum thresholdz%s %s confidence = %s)r r r r �debugr r r" r2 r
r6 �MINIMUM_THRESHOLDr5 �lowerr) r � ISO_WIN_MAP�getr �getEffectiveLevelr �DEBUGr' r Zprobers) r Zprober_confidenceZmax_prober_confidenceZ
max_proberr# r5 Zlower_charset_namer Z group_proberr r r �close� s`
zUniversalDetector.closeN)r �
__module__� __qualname__�__doc__r; �re�compiler0 r3 r8 r= r ZALLr r r4 rA r r r r r 3 s"
mr )rD r* r rE Zcharsetgroupproberr Zenumsr r r Z escproberr Z latin1proberr Zmbcsgroupproberr Zsbcsgroupproberr
�objectr r r r r �<module>$ s