§ <ˆj÷Jãóì—ddlZddlZddlmZddlmZmZddlmZm Z m Z mZmZddl mZmZmZmZddlmZmZmZmZddlmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"m#Z#ej$d¦«Z%ej&¦«Z'e' (ej)d¦«¦« d"de*de+de+de,dee e-dee e-de.de.defd„Z/ d"de de+de+de,dee e-dee e-de.de.defd„Z0 d"ddde+de+de,dee e-dee e-de.de.defd „Z1 d#ddde+de+de,dee e-dee e-de.defd!„Z2dS)$éN)ÚPathLike)ÚbasenameÚsplitext)ÚAnyÚBinaryIOÚListÚOptionalÚSeté)Úcoherence_ratioÚencoding_languagesÚmb_encoding_languagesÚmerge_coherence_ratios)ÚIANA_SUPPORTEDÚTOO_BIG_SEQUENCEÚTOO_SMALL_SEQUENCEÚTRACE)Ú mess_ratio)ÚCharsetMatchÚCharsetMatches)Úany_specified_encodingÚcut_sequence_chunksÚ iana_nameÚidentify_sig_or_bomÚ is_cp_similarÚis_multi_byte_encodingÚshould_strip_sig_or_bomÚcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)sééçš™™™™™É?TFÚ sequencesÚstepsÚ chunk_sizeÚ thresholdÚcp_isolationÚcp_exclusionÚpreemptive_behaviourÚexplainÚreturncó—t|ttf¦«s/td t|¦«¦«¦«‚|rJtj}t t¦«t t¦«t|¦«} | dkrƒt d¦«|rEt t¦«t |ptj¦«t#t%|dddgd¦«g¦«S|At td d |¦«¦«d„|D¦«}ng}|At tdd |¦«¦«d „|D¦«}ng}| ||zkr't td||| ¦«d}| }|dkr| |z|krt+| |z¦«}t|¦«t,k} t|¦«t.k}| r4t td | ¦«¦«n5|r3t td | ¦«¦«g}|rt1|¦«nd} | 6| | ¦«t td| ¦«t5¦«}g}g}d}d}d}t#¦«}t7|¦«\}}|D| |¦«t tdt|¦«|¦«| d¦«d|vr| d¦«|t8zD]Ì}|r||vrŒ |r||vrŒ||vrŒ| |¦«d}||k}|ot=|¦«}|dvr$|s"t td|¦«Œl t?|¦«}n8#t@tBf$r$t td|¦«YŒ±wxYw |rS|durOtE|dur|dt+d¦«…n#|t|¦«t+d¦«…|¬¦«n,tE|dur|n|t|¦«d…|¬¦«}nx#tFtHf$rd}t|tH¦«s/t td|tE|¦«¦«| |¦«Yd}~Œ¨d}~wwxYwd}|D]}tK||¦«rd}nŒ|r$t td||¦«ŒñtM|sdnt|¦«| t+| |z¦«¦«}|o|duot|¦«| k} | r!t td|¦«t+t|¦«dz¦«}!tO|!d¦«}!d}"d}#g}$g}% tQ|||||||||¦ « D]Y}&|$ |&¦«|% tS|&|¦«¦«|%d |kr|"dz }"|"|!ks|r|durnŒZnJ#tF$r=}t td!|tE|¦«¦«|!}"d}#Yd}~nd}~wwxYw|#sŽ|rŒ|sŠ |t+d"¦«d… *|d#¬$¦«n\#tF$rO}t td%|tE|¦«¦«| |¦«Yd}~ŒÛd}~wwxYw|%rtW|%¦«t|%¦«znd}'|'|ks|"|!kr}| |¦«t td&||"tY|'d'zd(¬)¦«¦«|dd| fvr*|#s(t%|||dg|¦«}(|| kr|(}n|dkr|(}n|(}Œt td*|tY|'d'zd(¬)¦«¦«|st[|¦«})nt]|¦«})|)rAt td+ |tE|)¦«¦«¦«g}*|dkrB|$D]?}&t_|&d,|)rd- |)¦«nd¦«}+|* |+¦«Œ@ta|*¦«},|,r4t td. |,|¦«¦«| t%|||'||,|¦«¦«|| ddfvrt|'d,krnt d/|¦«|r9t t¦«t |¦«t#||g¦«cS||krnt d0|¦«|r9t t¦«t |¦«t#||g¦«cSŒÎt|¦«dkrÙ|s|s|r t td1¦«|r6t d2|j1¦«| |¦«n{|r||r|r|j2|j2ks|0t d3¦«| |¦«n1|r/t d4¦«| |¦«|rDt d5| 3¦«j1t|¦«dz ¦«nt d6¦«|r9t t¦«t |¦«|S)7ae Given a raw bytes sequence, return the best possibles charset usable to render str objects. If there is no results, it is a strong indicator that the source is binary/not text. By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page but never take it for granted. Can improve the performance. You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that purpose. This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. Custom logging format and handler can be set manually. z4Expected object of type bytes or bytearray, got: {0}rzzfrom_bytes..[ó"€ÐDÐDÐD° " eÑ,Ô,ÐDÐDÐDózacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.có.—g|]}t|d¦«‘ŒSr/r0r1s r4r5zfrom_bytes..fr6r7z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.rz>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.Úascii>Úutf_16Úutf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecoderg€„A)Úencodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.éééÿÿÿÿzaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sgjè@Ústrict)Úerrorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.édé)Úndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}gš™™™™™¹?ú,z We detected language {} using {}z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4Ú isinstanceÚ bytearrayÚbytesÚ TypeErrorÚformatÚtypeÚloggerÚlevelÚ addHandlerÚexplain_handlerÚsetLevelrÚlenÚdebugÚ removeHandlerÚloggingÚWARNINGrrÚlogÚjoinÚintrrrÚappendÚsetrrÚaddrrÚModuleNotFoundErrorÚImportErrorÚstrÚUnicodeDecodeErrorÚLookupErrorrÚrangeÚmaxrrÚdecodeÚsumÚroundr rrrr<ÚfingerprintÚbest)-r"r#r$r%r&r'r(r)Úprevious_logger_levelÚlengthÚis_too_small_sequenceÚis_too_large_sequenceÚprioritized_encodingsÚspecified_encodingÚtestedÚtested_but_hard_failureÚtested_but_soft_failureÚfallback_asciiÚfallback_u8Úfallback_specifiedÚresultsÚsig_encodingÚsig_payloadÚ encoding_ianaÚdecoded_payloadÚbom_or_sig_availableÚstrip_sig_or_bomÚis_multi_byte_decoderÚeÚsimilar_soft_failure_testÚencoding_soft_failedÚr_Úmulti_byte_bonusÚmax_chunk_gave_upÚearly_stop_countÚlazy_str_hard_failureÚ md_chunksÚ md_ratiosÚchunkÚmean_mess_ratioÚfallback_entryÚtarget_languagesÚ cd_ratiosÚchunk_languagesÚcd_ratios_mergeds- r4Ú from_bytesr#s<€õ8i¥)UÐ!3Ñ4Ô4ð ÝØB×IÒIÝY‘”ñ ô ñ ô ð ððÝ%+¤\ÐÝ×Ò/Ñ*Ô*Ð*ÝŠÑÔÐåi‘.”.€Fà ‚{€{ÝŠÐSÑTÔTÐTØð FÝ× Ò ¥Ñ1Ô1Ð1ÝOŠOÐ1ÐDµW´_ÑEÔEÐEÝ|¨I°wÀÀUÈBÐPRÑSÔSÐTÑUÔUÐUàÐÝ Š Ýð 5àIŠIlÑ#Ô#ñ ô ð ðEÐD°|ÐDÑDÔDˆˆàˆàÐÝ Š Ýð 6àIŠIlÑ#Ô#ñ ô ð ðEÐD°|ÐDÑDÔDˆˆàˆà *˜uÑ$Ò%Ð%Ý Š ÝØlØØØñ ô ð ðˆØˆ àˆq‚y€yV˜e‘^ jÒ0Ð0Ý˜ %™Ñ(Ô(ˆ å"% i¡.¤.Õ3EÒ"EÐÝ"% i¡.¤.Õ4DÒ"DÐàð Ý Š ÝØL×SÒSØñ ô ñ ô ð ð ð ð Ý Š ÝØW×^Ò^Øñ ô ñ ô ð ð(*Ðð.BÐKÕ˜yÑ)Ô)Ð)ÀtððÐ%Ø×$Ò$Ð%7Ñ8Ô8Ð8Ý Š ÝØNØñ ô ð õ‘u”u€FØ)+ÐØ)+Ðà-1€NØ*.€KØ15Ðå,Ñ.Ô.€Gå 3°IÑ >Ô >Ñ€L+àÐØ×$Ò$ \Ñ2Ô2Ð2Ý Š ÝØWÝÑÔØñ ô ð ð× Ò Ñ)Ô)Ð)àÐ+Ð+Ð+Ø×$Ò$ WÑ-Ô-Ð-à.µÑ?ðE<ñE<ˆ àð ˜M°Ð=Ð=Øàð ˜M¨\Ð9Ð9Øà˜FÐ"Ð"Øà Š =Ñ!Ô!Ð!à)-ˆØ%1°]Ò%BÐØ!5ð" Õ:QØñ; ô; ÐðÐ0Ð0Ð0Ð9MÐ0ÝJŠJÝØmØñ ô ð ð ð Ý*@ÀÑ*OÔ*OÐ!Ð!øÝ#¥[Ð1ð ð ð ÝJŠJÝØDØñ ô ð ð ˆHð øøøð Ø$ð Ð)>À%Ð)GÐ)GÝà'¨5Ð0Ð0ð˜k¥ D¡ ¤ ˜kÔ*Ð*à"¥3 {Ñ#3Ô#3µc¸$±i´iÐ#?Ô@Ø*ð ñôððõ#&à'¨5Ð0Ð0ðIà"¥3 {Ñ#3Ô#3Ð#5Ð#5Ô6Ø*ð #ñ#ô#øøõ#¥KÐ0ð ð ð Ý˜a¥Ñ-Ô-ð Ý— ’ ÝØOØ!Ý˜‘F”Fñ ôðð $×*Ò*¨=Ñ9Ô9Ð9ØˆHˆHˆH‰Høøøøð øøøð+0Ð!à$;ð ð Ð Ý˜]Ð,@ÑAÔAð Ø,0Ð)Øð ð%ð ÝJŠJÝØiØØ$ñ ô ð ñ å Ø)Ð?ˆAˆAs°;Ñ/?Ô/?ØÝ˜‘ÑÔñ ô ˆð "ð .Ø tÐ+ð .åOÑ$Ô$ vÒ-ð ðð ÝJŠJÝð-àñ ô ð õ"%¥S¨¡W¤W¨q¡[Ñ!1Ô!1ÐåÐ 1°1Ñ5Ô5ÐØ !ÐØ %Ðà!ˆ Øˆ ð )Ý,ØØØØØ$Ø ØØ%Øñ ô ð ð ð× Ò Ñ'Ô'Ð'à× Ò ¥¨E°9Ñ!=Ô!=Ñ>Ô>Ð>à˜R”= IÒ-Ð-Ø$¨Ñ)Ð$à$Ð(9Ò9Ð9Ø(ð:Ø-=ÀÐ-FÐ-FàEøøøÝ!ð )ð )ð )ÝJŠJÝØsØÝA‘”ñ ô ð ð 1ÐØ$(Ð!Ð!Ð!Ð!Ð!Ð!øøøøð )øøøð&ð à%ð ð*ð ð Ø#˜d™)œ)˜+˜+Ô&×-Ò-¨mÀHÐ-ÑMÔMÐMÐMøÝ%ð ð ð Ý— ’ ÝØtØ!Ý˜‘F”Fñ ôðð(×.Ò.¨}Ñ=Ô=Ð=Ø‘øøøøð øøøðENÐ!V¥ Y¡¤µ#°i±.´.Ñ!@Ð!@ÐSVˆØ˜iÒ'Ð'Ð+;Ð?PÒ+PÐ+PØ#×*Ò*¨=Ñ9Ô9Ð9ÝJŠJÝð0àØ Ýo¨Ñ+°QÐ7Ñ7Ô7ñ ô ð ð '¨7Ð4FÐ!GÐGÐGØ-ðHõ".Ø˜}¨i¸ÀÀOñ"ô"ð!Ð$6Ò6Ð6Ø)7Ð&Ð&Ø" gÒ-Ð-Ø%3NNà"0KÙå Š ÝØKØÝ/ CÑ'°Ð3Ñ3Ô3ñ ô ð ð%ð DÝ*<¸]Ñ*KÔ*KÐÐå4°]ÑCÔCÐàð ÝJŠJÝØ8×?Ò?Ø!¥3Ð'7Ñ#8Ô#8ñôñ ô ð ðˆ ð˜GÒ#Ð#Ø"ð 2ð 2Ý"1Ø˜3Ð>NÐ X §¢Ð)9Ñ :Ô :Ð :ÐTXñ#ô#ð× Ò Ñ1Ô1Ð1Ð1å1°)Ñ<Ô<Ðàð ÝJŠJÝØ2×9Ò9Ø$ mñôñ ô ð ð ŠÝØØØØ$Ø Øñ ô ñ ô ð ð Ð0°'¸7ÐCÐCÐCØ #Ò%Ð%åLŠLØ@À-ñ ô ð ðð 7Ý×$Ò$¥_Ñ5Ô5Ð5Ý—’Ð 5Ñ6Ô6Ð6Ý! 7¨=Ô#9Ð":Ñ;Ô;Ð;Ð;Ð;à˜LÒ(Ð(ÝLŠLð1àñ ô ð ð ð 7Ý×$Ò$¥_Ñ5Ô5Ð5Ý—’Ð 5Ñ6Ô6Ð6Ý! 7¨=Ô#9Ð":Ñ;Ô;Ð;Ð;Ð;ñ)õˆ7|„|qÒÐØð ˜.ð Ð,>ð ÝJŠJÝØañ ô ð ð ð +ÝLŠLØIØ"Ô+ñ ô ð ð NŠNÐ-Ñ.Ô.Ð.Ð.à ð +Ø+Ð3àð4ð#ð4ð Ô+¨~Ô/IÒIÐIàÐ'åLŠLÐUÑVÔVÐVØNŠN˜;Ñ'Ô'Ð'Ð'Ø ð +ÝLŠLÐUÑVÔVÐVØNŠN˜>Ñ*Ô*Ð*àðVÝŠØkØLŠL‰NŒNÔ#Ý‰LŒL˜1Ññ ô ð ð õ ŠÐTÑUÔUÐUàð/Ý×Ò_Ñ-Ô-Ð-ÝŠÐ-Ñ.Ô.Ð.à€Ns^ÐP&Ð&2QÑQÑBS!Ó!UÓ2AUÕUÙA1Z2Ú2 [9Ú<3[4Û4[9Ü,\0Ü0 ^ Ü:A^Þ^ Úfpc óR—t| ¦«|||||||¦«S)z† Same thing than the function from_bytes but using a file pointer that is already ready. Will not close the file pointer. )rÚread)rŽr#r$r%r&r'r(r)s r4Úfrom_fpr‘ås6€õØ Š‰ Œ Ø ØØØØØØñ ô ð r7Úpathz PathLike[Any]có‚—t|d¦«5}t||||||||¦«cddd¦«S#1swxYwYdS)z• Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. Can raise IOError. ÚrbN)Úopenr‘) r’r#r$r%r&r'r(r)rŽs r4Ú from_pathr–ÿsš€õ ˆdDÑ Ô ð ˜RÝØØØØØØØ Øñ ô ð ð ð ð ñ ô ð ð ð ð ð ð øøøð ð ð ð ð ð s‘4´8»8c ó¨—tjdt¦«t|||||||¦«}t |¦«}tt |¦«¦«} t|¦«dkr"td |¦«¦«‚| ¦«} | dxxd| jzz cc<td t|¦« |d | ¦«¦«¦«d¦«5}| | ¦«¦«ddd¦«n#1swxYwY| S) zi Take a (text-based) file path and try to create another file next to it, this time using UTF-8. z2normalize is deprecated and will be removed in 3.0rz;Unable to normalize "{}", no encoding charset seems to fit.ú-z{}r-ÚwbN)ÚwarningsÚwarnÚDeprecationWarningr–rÚlistrrQÚIOErrorrJrgr<r•r^ÚreplacerWÚwriteÚoutput)r’r#r$r%r&r'r(rtÚfilenameÚtarget_extensionsÚresultrŽs r4Ú normalizer¥sŠ€õ „MØ<Ýñôðõ ØØ ØØØØØñô€Gõ˜‰~Œ~€HÝX hÑ/Ô/Ñ0Ô0Ðå ˆ7|„|qÒÐÝØI×PÒPØñ ô ñ ô ð ð\Š\‰^Œ^€FàaÐÐÔ˜C &¤/Ñ1Ñ1ÐÐÑå ØŠ•C˜‘I”I×%Ò% h°·²Ð8IÑ0JÔ0JÑKÔKÑLÔLÈdñ ô ð"à Ø Š—’‘”Ñ!Ô!Ð!ð"ð"ð"ñ"ô"ð"ð"ð"ð"ð"ð"øøøð"ð"ð"ð"ð €MsÄ(EÅEÅE)rr r!NNTF)rr r!NNT)3rTršÚosrÚos.pathrrÚtypingrrrr r Úcdrr rrÚconstantrrrrÚmdrÚmodelsrrÚutilsrrrrrrrÚ getLoggerrLÚ StreamHandlerrOÚsetFormatterÚ FormatterrHrXÚfloatr^Úboolrr‘r–r¥©r7r4úrµsðØ€€€Ø€€€ØÐÐÐÐÐØ&Ð&Ð&Ð&Ð&Ð&Ð&Ð&Ø5Ð5Ð5Ð5Ð5Ð5Ð5Ð5Ð5Ð5Ð5Ð5Ð5Ð5ðððððððððððððRÐQÐQÐQÐQÐQÐQÐQÐQÐQÐQÐQØÐÐÐÐÐØ0Ð0Ð0Ð0Ð0Ð0Ð0Ð0ððððððððððððððððððð ˆÔ Ð/Ñ 0Ô 0€Ø''Ô'Ñ)Ô)€Ø×ÒØ€GÔÐAÑBÔBñôððØØØ(,Ø(,Ø!%ØððØðàððððð ð ˜4 œ9Ô%ðð˜4 œ9Ô%ð ððððððððððHØØØ(,Ø(,Ø!%ØððØðàððððð ð ˜4 œ9Ô%ðð˜4 œ9Ô%ð ðððððððððð8ØØØ(,Ø(,Ø!%Øð ð Ø ð àð ðð ðð ð ˜4 œ9Ô%ð ð˜4 œ9Ô%ð ðð ðð ðð ð ð ð ð:ØØØ(,Ø(,Ø!%ð.ð.Ø ð.àð.ðð.ðð .ð ˜4 œ9Ô%ð.ð˜4 œ9Ô%ð .ðð.ðð.ð.ð.ð.ð.ð.r7