§ <ˆj:ãóЗdZddlZddlZddlZddlmZmZmZddlm Z ddl m Z ddl m Z mZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZGd„d¦«ZdS)a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco éN)ÚListÚOptionalÚUnioné)ÚCharSetGroupProber)Ú CharSetProber)Ú InputStateÚLanguageFilterÚ ProbingState)ÚEscCharSetProber)Ú Latin1Prober)ÚMacRomanProber)ÚMBCSGroupProber)Ú ResultDict)ÚSBCSGroupProber)Ú UTF1632Proberc óX—eZdZdZdZejd¦«Zejd¦«Zejd¦«Z dddd d d d d dœZ dddd ddddœZ e j dfde deddfd„Zedefd„¦«Zedefd„¦«Zedeefd„¦«Zd!d„Zdeeefddfd„Zdefd „ZdS)"ÚUniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result gš™™™™™É?s[€-ÿ]s(|~{)s[€-Ÿ]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)ú iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8ú iso-8859-9z iso-8859-13z ISO-8859-11ÚGB18030ÚCP949úUTF-16)Úasciirztis-620rÚgb2312zeuc-krzutf-16leFÚ lang_filterÚshould_rename_legacyÚreturnNcó —d|_d|_g|_ddddœ|_d|_d|_t j|_d|_ ||_ tj t¦«|_d|_||_| ¦«dS)Nç©ÚencodingÚ confidenceÚlanguageFó)Ú_esc_charset_proberÚ_utf1632_proberÚ_charset_probersÚresultÚdoneÚ _got_datar Ú PURE_ASCIIÚ _input_stateÚ _last_charrÚloggingÚ getLoggerÚ__name__ÚloggerÚ_has_win_bytesrÚreset)Úselfrrs úP/opt/cloudlinux/venv/lib64/python3.11/site-packages/chardet/universaldetector.pyÚ__init__zUniversalDetector.__init__ds’€ð @DˆÔ Ø8<ˆÔØ57ˆÔàØØð# ð# ˆŒ 𠈌 ؈ŒÝ&Ô1ˆÔ؈ŒØ&ˆÔÝÔ'­Ñ1Ô1ˆŒ Ø#ˆÔØ$8ˆÔ!Ø � Š ‰ Œ ˆ ˆ ˆ r%có—|jS©N)r-©r5s r6Ú input_statezUniversalDetector.input_state{s €àÔ Ð r%có—|jSr9)r3r:s r6Ú has_win_byteszUniversalDetector.has_win_bytess €àÔ"Ð"r%có—|jSr9)r(r:s r6Úcharset_probersz!UniversalDetector.charset_probersƒs €àÔ$Ð$r%có2—ddddœ|_d|_d|_d|_tj|_d|_|jr|j  ¦«|j r|j   ¦«|j D]}|  ¦«ŒdS)zæ Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. Nr r!Fr%) r)r*r+r3r r,r-r.r&r4r'r()r5Úprobers r6r4zUniversalDetector.reset‡s¨€ð $(°sÈÐMÐMˆŒ ؈Œ ؈ŒØ#ˆÔÝ&Ô1ˆÔ؈ŒØ Ô #ð -Ø Ô $× *Ò *Ñ ,Ô ,Ð ,Ø Ô ð )Ø Ô × &Ò &Ñ (Ô (Ð (ØÔ+ð ð ˆFØ �LŠL‰NŒNˆNˆNð ð r%Úbyte_strcóp—|jrdS|sdSt|t¦«st|¦«}|js÷| t j¦«r ddddœ|_n¯| t jt j f¦«r ddddœ|_nx| d¦«r ddddœ|_nW| d ¦«r d dddœ|_n6| t j t j f¦«r d dddœ|_d |_|jd � d |_dS|j tjkrt|j |¦«rtj|_ nH|j tjkr3|j |j|z¦«rtj|_ |dd…|_|jst-¦«|_|jjt0jkr]|j |¦«t0jkr5|jj|j ¦«ddœ|_d |_dS|j tjkr‰|jst?|j ¦«|_|j |¦«t0jkr?|jj|j ¦«|jj!dœ|_d |_dSdS|j tjk�r'|j"s tG|j ¦«g|_"|j tHj%zr&|j" &tO¦«¦«|j" &tQ¦«¦«|j" &tS¦«¦«|j"D]U}| |¦«t0jkr0|j| ¦«|j!dœ|_d |_nŒV|j* |¦«r d |_+dSdSdS)aý Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIGçð?Úr!zUTF-32sþÿzX-ISO-10646-UCS-4-3412sÿþzX-ISO-10646-UCS-4-2143rTr"éÿÿÿÿ),r*Ú isinstanceÚ bytearrayr+Ú startswithÚcodecsÚBOM_UTF8r)Ú BOM_UTF32_LEÚ BOM_UTF32_BEÚBOM_LEÚBOM_BEr-r r,ÚHIGH_BYTE_DETECTORÚsearchÚ HIGH_BYTEÚ ESC_DETECTORr.Ú ESC_ASCIIr'rÚstater Ú DETECTINGÚfeedÚFOUND_ITÚ charset_nameÚget_confidencer&r rr$r(rr ÚNON_CJKÚappendrr rÚWIN_BYTE_DETECTORr3)r5rBrAs r6rWzUniversalDetector.feedšs]€ð Œ9ð Ø ˆFàð Ø ˆFå˜(¥IÑ.Ô.ð +Ý  Ñ*Ô*ˆHðŒ~ð% à×"Ò"¥6¤?Ñ3Ô3ð Xð!,Ø"%Ø "ðð�” � ð ×$Ò$¥fÔ&9½6Ô;NÐ%OÑPÔPð Xð,4À3ÐTVÐWÐW�” � Ø×$Ò$Ð%8Ñ9Ô9ð Xð!9Ø"%Ø "ð ð�” � ð ×$Ò$Ð%8Ñ9Ô9ð Xð!9Ø"%Ø "ð ð�” � ð ×$Ò$¥f¤mµV´]Ð%CÑDÔDð Xð,4À3ÐTVÐWÐW�” à!ˆDŒNØŒ{˜:Ô&Ð2Ø �” Ø�ð Ô ¥ Ô 5Ò 5Ð 5ØÔ&×-Ò-¨hÑ7Ô7ð 9Ý$.Ô$8�Ô!Ð!àÔ!¥ZÔ%:Ò:Ð:ØÔ%×,Ò,¨T¬_¸xÑ-GÑHÔHð;õ%/Ô$8�Ô!à" 2 3 3œ-ˆŒðÔ#ð 3Ý#0¡?¤?ˆDÔ à Ô Ô %­Ô)?Ò ?Ð ?ØÔ#×(Ò(¨Ñ2Ô2µlÔ6KÒKÐKà $Ô 4Ô AØ"&Ô"6×"EÒ"EÑ"GÔ"GØ "ðð�” ð !�” Ø�ð Ô ¥ Ô 4Ò 4Ð 4ØÔ+ð NÝ+;¸DÔÔ>Ð>ØÔ/ð ð �Ø—;’;˜xÑ(Ô(­LÔ,AÒAÐAà$*Ô$7Ø&,×&;Ò&;Ñ&=Ô&=Ø$*¤Oð#ð#�D”Kð !%�D”IØ�EðBðÔ%×,Ò,¨XÑ6Ô6ð +Ø&*�Ô#Ð#Ð#ð%7Ð 6ð" +ð +r%c ó¤—|jr|jSd|_|js|j d¦«�n%|jt jkr ddddœ|_�n|jt jkrîd}d}d}|j D]#}|sŒ|  ¦«}||kr|}|}Œ$|r»||j kr°|j }|€J‚|  ¦«}|  ¦«}| d ¦«r"|jr|j ||¦«}|jr/|j |pd  ¦«|¦«}|||jdœ|_|j ¦«t,jkrÆ|jd €¹|j d ¦«|j D]—}|sŒt1|t2¦«rD|jD];}|j d |j |j|  ¦«¦«Œ<Œ^|j d |j |j|  ¦«¦«Œ˜|jS) zæ Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!rrDrEr!Nr ziso-8859r"z no probers hit minimum thresholdz%s %s confidence = %s)r*r)r+r2Údebugr-r r,rRr(rZÚMINIMUM_THRESHOLDrYÚlowerrIr3Ú ISO_WIN_MAPÚgetrÚ LEGACY_MAPr$ÚgetEffectiveLevelr/ÚDEBUGrGrÚprobers) r5Úprober_confidenceÚmax_prober_confidenceÚ max_proberrArYÚlower_charset_namer#Ú group_probers r6ÚclosezUniversalDetector.closes€ð Œ9ð Ø”;Р؈Œ àŒ~ð( Ø ŒK× Ò Ð1Ñ 2Ô 2Ð 2Ñ 2ðÔ ¥*Ô"7Ò 7Ð 7Ø'.¸cÈrÐRÐRˆDŒK‰KðÔ ¥*Ô"6Ò 6Ð 6Ø $Ð Ø$'Ð !؈JØÔ/ð (ð (�ØðØØ$*×$9Ò$9Ñ$;Ô$;Ð!Ø$Ð'<Ò<Ð<Ø,=Ð)Ø!'�JøØð Ð4°tÔ7MÒMÐMØ)Ô6� Ø#Ð/Ð/Ð/Ø%1×%7Ò%7Ñ%9Ô%9Ð"Ø'×6Ò6Ñ8Ô8� ð&×0Ò0°Ñ<Ô<ðØÔ*ðØ'+Ô'7×';Ò';Ø.° ñ(ô(˜ ðÔ,ðØ#'¤?×#6Ò#6Ø%Ð+¨×2Ò2Ñ4Ô4°lñ$ô$�Lð!-Ø",Ø *Ô 3ðð�” ð Œ;× (Ò (Ñ *Ô *­g¬mÒ ;Ð ;ØŒ{˜:Ô&Ð.Ø” ×!Ò!Ð"DÑEÔEÐEØ$(Ô$9ðð�LØ'ð!Ø Ý! ,Õ0BÑCÔCðØ&2Ô&:ðð˜FØ œK×-Ò-Ø 7Ø &Ô 3Ø &¤Ø &× 5Ò 5Ñ 7Ô 7ñ ôððððœ ×)Ò)Ø3Ø(Ô5Ø(Ô1Ø(×7Ò7Ñ9Ô9ñ ôððð Œ{Ðr%)rN)r1Ú __module__Ú __qualname__Ú__doc__r`ÚreÚcompilerPrSr]rbrdr ÚALLÚboolr7ÚpropertyÚintr;r=rrr?r4rÚbytesrHrWrrm©r%r6rr8sÅ€€€€€ððð ÐØ#˜œ NÑ3Ô3ÐØ�2”:˜lÑ+Ô+€LØ"˜œ  >Ñ2Ô2Ðà$Ø$Ø$Ø$Ø$Ø$Ø$Ø%ð ð €Kð Ø$Ø Ø$ØØØðð€Jð'5Ô&8Ø%*ððà#ðð#ðð ð ðððð.ð!˜Sð!ð!ð!ñ„Xð!ðð#˜tð#ð#ð#ñ„Xð#ðð%  mÔ!4ð%ð%ð%ñ„Xð%ððððð&A+˜U 5¨)Ð#3Ô4ðA+¸ðA+ðA+ðA+ðA+ðFM�zðMðMðMðMðMðMr%r)rprJr/rqÚtypingrrrÚcharsetgroupproberrÚ charsetproberrÚenumsr r r Ú escproberr Ú latin1proberr ÚmacromanproberrÚmbcsgroupproberrÚ resultdictrÚsbcsgroupproberrÚ utf1632proberrrrxr%r6úr„sFðð8ðð€ € € Ø€€€Ø € € € Ø(Ð(Ð(Ð(Ð(Ð(Ð(Ð(Ð(Ð(à2Ð2Ð2Ð2Ð2Ð2Ø(Ð(Ð(Ð(Ð(Ð(Ø;Ð;Ð;Ð;Ð;Ð;Ð;Ð;Ð;Ð;Ø'Ð'Ð'Ð'Ð'Ð'Ø&Ð&Ð&Ð&Ð&Ð&Ø*Ð*Ð*Ð*Ð*Ð*Ø,Ð,Ð,Ð,Ð,Ð,Ø"Ð"Ð"Ð"Ð"Ð"Ø,Ð,Ð,Ð,Ð,Ð,Ø(Ð(Ð(Ð(Ð(Ð(ðrðrðrðrðrñrôrðrðrðrr%