ūjp` d Z dZddlZddlZddlZddlZddlmZ ddlm Z m
Z
m Z ddl m Z m
Z
ddlmZ ddlmZmZmZmZmZmZ dd lmZ ddlZdd
lm Z dd l m Z dd l m Z dd
l!m"Z"m#Z# ddl$m%Z%m&Z&m'Z' ddl(m)Z) ddl*m+Z+ ddl,m-Z-m.Z. ddl/m0Z0 dZ1 ddl2m3Z4 n# e5$ rZ6e6Z1dZ4Y dZ6[6ndZ6[6ww xY we G d d Z7e G d d Z8e G d d Z9 G d de Z:de;d e:fd Z< G d d e Z=d e;fd Z>d!e;fd"Z?e@d#k rH e0 ejA e? e> ejB $ e= ZC eC dS dS )%z'
This module contains Autotracer class
z ssa.modules N) defaultdict) dataclassfieldasdict)datetime timedelta) fnmatchcase)DictIteratorOptional
NamedTupleListIterable)urlparse)setup_database)SSAError)Common)iter_domains_dataiter_domain_url_data)statusdisabled_usersmisconfiguration_checks)
load_tunables)autotracing_tunables_schema)autotracing_logautotracing_stats_file) sentry_initc Z e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S )
URLzURL data containeruri avg_duration max_durationtotal_slow_reqs
total_reqs
belongs_tobelongs_to_userN)__name__
__module__ __qualname____doc__str__annotations__int M/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/autotracer.pyr r - sc HHHOOOOOOr/ r c e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ
eed< dZ eed< dZ eed < dZ
eed
< dZeed < dZeed < dZeed
< dS ) RulesRejectszRejects counters containerr non_wp throttleddensityslowness max_slownessdisablednginx frequency server_limit domain_limit no_domainN)r' r( r) r* r3 r- r, r4 r5 r6 r7 r8 r9 r: r; r< r= r. r/ r0 r2 r2 : s $$FCOOOIsGSHcL#HcE3NNNIsL#L#Isr/ r2 c \ e Zd ZU dZeed< dZeed< dZeed< e e
Z e
ed< dS ) Statsz AutoTracer statistics container
rules_versionr urls_processed
urls_selected)default_factoryrejectsN) r' r( r) r* r+ r, rA r- rB r r2 rD r. r/ r0 r? r? K sa ))NCM3 !E, ? ? ?G\?????r/ r? c ( e Zd ZU dZeed< eed< dS )URLSz"
Representation of an URL
domain_nameuri_pathN)r' r( r) r* r+ r, r. r/ r0 rF rF U s1 MMMMMr/ rF urlreturnc t | }|j r
d|j nd}|j r |j | nd}|j dd }| d d }t
|| S )z
Split URL into domain_name and uripath including query string
:param url: URL of format protocol://domain/path;parameters?query#fragment
:return: namedtuple URL(domain_name, uripath)
? /zwww.:r )r querypathnetlocreplacesplitrF )rI fragmentsqsr _no_www_netloc_no_port_netlocs r0 url_splitrY ^ s
I"+/ 9 Y_ rB%.^
Says if nginx is enabled for a specific user
NF)
skip_nginxro NginxUserCache
is_enabled)ri r s r0 nginx_is_enabledzAutoTracer.nginx_is_enabled s3 ? A u*955@ @ur/ c t d | j D }| j d| | j |z
S )z>Recalculate limit per server taking into account running tasksc B g | ] }|d dk |d dk | S )r runninguserz
*autotracing*r. ).0tasks r0
z. s8 q q qdh 91L1LQUV\Q]apQpQpTQpQpQpr/ z&Number of running autotracing tasks %s)lenrf rb ru per_server_limit)ri
running_counts r0 per_server_limit_recalculatedz(AutoTracer.per_server_limit_recalculated sM q qdo q q q
BMRRR$}44r/ full_listc |.t j }| j d| t j t
| j z
fd|D S )zQ
Excludes tasks older than N days from the general list of tasks
NzTask list loaded %s)daysc g | ]S}| d | d ndt k Q|TS )
createtimeNr )getr- timestamp)r r
n_days_agos r0 r z1AutoTracer.excludes_old_tasks.. sg
*.((<*@*@*L &&&RSWZ[e[o[o[q[qWrWrrr
rrrr/ )ro rf rb ru r nowr min_retracing_interval)ri r r s @r0 excludes_old_tasksz AutoTracer.excludes_old_tasks s{ +--I K 3Y ? ? ? \^^iT5P&Q&Q&QQ
!
r/ current_urlc \ t | }| j D ]}t |d }|j |j k o|j |j k }t |j |j ot |j |j }|s|r%| j d||d dS dS )z
Excludes url from the list if it completely matches the current url
or if domain names match and "*" follows the domain name in the list
rI z2Skipped: URL %s was traced recently. Matched by %sTF)rY rf rG rH r rb ru )ri r c task_datat direct_matchwildcard_matchs r0 exclude_thesame_urlsz AutoTracer.exclude_thesame_urls s
k " " I)E*++A =AM9VajAJ>VL (
F F n;WXWacdcmKnKnN
~
!!H e$
tt
ur/ url_total_reqsdomain_total_reqsc | j rMt j t j || }| j d| || j k S dS )z-Check that URL density passes given thresholdzCalculated density %sT)r5 npamincorrcoefrb ru density_threshold)ri r r url_densitys r0 pass_by_densityzAutoTracer.pass_by_density sT < 8 '"+n>O"P"PQQK K 5{ C C C !77 7tr/
url_durationsc d| j z
}t j || }| j d|| || j k S )z
The measure of "slowness" for URL is:
at least N% of its requests take more than X seconds.
N% -- self.slow_duration_percentage
X -- self.slow_duration_threshold
d z.Calculated %sth percentile %s for min duration)slow_duration_percentager
percentilerb ru slow_duration_threshold)ri r reversed_percentilereversed_percentile_values r0 pass_by_slowness_percentilez&AutoTracer.pass_by_slowness_percentile sX "D$A A$&M-AT$U$U! < %
)D,HHHr/ c t j || j }| j d| j | || j k S )a
The opposite to pass_by_slowness_percentile method.
The measure of "much slowness" for URL is:
at least N% of its requests must take less than X seconds.
N% -- self.max_slow_duration_percentage
X -- self.max_slow_duration_threshold
z.Calculated %sth percentile %s for max duration)r r max_slow_duration_percentagerb ru max_slow_duration_threshold)ri r percentile_values r0 pass_by_max_slowness_percentilez*AutoTracer.pass_by_max_slowness_percentile sO = 8YZZ < -
4#CCCr/ url_throttled_reqsc |dS t | t | z dz }| j d| || j k S )zY
Check that percent of throttled requests per URL passes given threshold
NFr z Calculated throttled percent %s)sumrb ru allowed_throttling_percentage)ri r r throttled_percents r0 pass_by_allowed_throttlingz%AutoTracer.pass_by_allowed_throttling sX %5 !3 4 4s>7J7J Jc Q ;=NOOO D$FFFr/ wp_statusc 2 | j rt | S dS )zx
Check that URLs of a particular domain should be analyzed.
For now we skip non-wordpress sites
T)only_wpbool)ri r s r0 pass_by_enginezAutoTracer.pass_by_engine s
< # ?? "tr/ c # K t | j D ]}d}| |j sQ| j d |j }| j j xj |z
c_ | j xj
|z
c_
pt | j |j D ][\ }}}|| j
v r| j d| | j xj
dz
c_
| | d |d s5| j d | j j xj dz
c_ | |d |j s5| j d | j j xj dz
c_ | | s6| j d | j j xj dz
c_ F| | s6| j d
| j j xj dz
c_ |Y| |j }|6| j d | j j xj dz
c_ |j }t7 |t9 t; j | t? |d t? |d tA | |j |
V ]dS )a
Select all URLs suitable for auto tracing by very basic rules:
- WP site
- suitable throttling
- suitable density
- measure of "slow" URL
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Nz Skipped by engine: non-wordpresszProcessing URL %srl r r z Skipped by throttled percentzSkipped by densityz Skipped by slowness percentilez"Skipped by max slowness percentilez"Skipped by unavailable domain info
url_slow_reqs)r! r# r$ r" r% r& )!r rh r is_a_wordpress_domainrb ru urls_numberrg rD r3 rA r rG non_url_fieldsr r r4 r r r5 r r6 r r7 r r= r r r- r meanr max)ri domain_data domain_owner
skipped_countrI durationsdatar s r0 urls_computationally_filteredz(AutoTracer.urls_computationally_filtered# s. -T[99 J J K L&&{'HII
!!"DEEE !, 7
"))]:))
))]:)) )=T[+Ja(b(b 8
8
$Y$--- !!"5s;;;
))Q.))66txx@T7U7UW[\lWmnn K%%&DEEEJ&00A500 ++D1A,BKDabb K%%&:;;;J&..!3.. 77 BB K%%&FGGGJ&//14// ;;IFF K%%&JKKKJ&33q833 '"&"6"6{7N"O"OK"* ))*NOOO
*44944 '2'7 !$RWY%7%7!8!8$'_(=$>$> "4(8#9 : :!$Y * 6$0 a8
%J J r/ stats_collectedc # K | | D ]}|j | j v r;| j d|j | j j xj dz
c_ K| |j r;| j d|j | j j xj
dz
c_
|V dS )z
From selected by computed thresholds URLs take those for which:
- autotracing enabled
- nginx disabled
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
z'Skipped: autotracing is disabled for %srl z Skipped: nginx is enabled for %sN) fill_in_disabled_usersr r& r rb ru rg rD r8 r r9 )ri r rI s r0 rB zAutoTracer.urls_selectedx s
##%%%5577
C"d&999 !!"KSM`aaa
"++q0++$$S%899
!!"DcFYZZZ
"((A-((IIII
r/ c # K d}t t }| | _ | }| }| j d| |D ]}| |j r| j
j xj dz
c_ 7||k rv||j
| j k r |dz
}||j
xx dz
cc< |V r| j d|j | j | j
j xj dz
c_ | j d|j | | j
j xj dz
c_ dS )a
Schedule autotracing by sorted list taking into account the limits:
- no same task for 10 days
- limit per server
- limit per domain
Rule ordering still affects statistics counters, but the counting for
rules #8 (frequency) and #9 (domain_limit) now happens during candidate
selection in urls_sorted() (which bounds the candidate list to the best
per_domain_limit URLs per domain). Only server_limit (#10) is counted
here. Keep this attribution in sync with urls_sorted() and
docs/design/autotracer-and-decision-maker.md.
Return resulting list of URLs scheduled for auto tracing
r zSorted scheduled list %srl z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r- r rf r urls_sortedrb ru r r rg rD r: r% per_domain_limitr< r; )ri general_tasks_countertasks_counter_per_domainper_server_smart_limit sorted_urlsrI s r0 rw zAutoTracer.urls_scheduled s !" #.s#3#3 1133!%!C!C!E!E &&(( 4kBBB 5 5C((11
",,1,,$'===+CN;d>SSS)Q.),S^<<<A<<< IIII K%%= -
J&33q8333 !!9G*
"//14///9 5 5r/ c | j }t t }d}| D ]}| |j r| j j xj dz
c_ 7|dz
}|j |j
| f|f}||j }t | |k rt j || |dk rH|d |d d k r0t j || | j j xj dz
c_ | j j xj dz
c_ d | D }| d d d |D S )u1
Select candidate URLs for scheduling, sorted by total number of
requests first and by average duration second.
Only the best ``per_domain_limit`` candidates per domain are retained:
a domain can never schedule more than that (see urls_scheduled), so
lower-ranked same-domain candidates are unschedulable. Frequency
excluded URLs (min_retracing_interval) are dropped up front as well,
since they never consume a schedule slot. This bounds peak memory to
O(domains * per_domain_limit) instead of materializing and fully
sorting every candidate — an attacker can inflate the candidate count
with distinct cache-busting query strings, but they all collapse onto
their own domain's small per-domain cap (CLPRO-3310 F-09).
r rl c g | ] }|D ]}|
S r. r. )r heapentrys r0 r z*AutoTracer.urls_sorted.. s% SSSdSSUeSSSSr/ c | d S )Nr r. )r s r0 z(AutoTracer.urls_sorted.. s
%( r/ T)keyreversec g | ]\ }}|S r. r. )r _rI s r0 r z*AutoTracer.urls_sorted.. s ---3---r/ )r r re rB r r rg rD r: r$ r! r% r heapqheappush heapreplacer< valuessort)ri r per_domain_bestorderrI r r
candidatess r0 r zAutoTracer.urls_sorted s 0 ,7t+<+<%%'' 5 5C((11
",,1,, QJEnc&6?EE"3>2D4yy+++tU++++!A%%%(T!WQZ*?*? !$...
"//14///
"//14///SS(>(>(@(@SSS
2 2DAAA--*----r/ c , t | _ dS )z6
Fill internal list of disabled users
N)r )ri s r0 r z!AutoTracer.fill_in_disabled_users s - . .r/ c D | j t t d 5 }t j t | j | ddd dS # 1 swxY w Y dS # t $ r3}| j dt | Y d}~dS d}~ww xY wdS )zI
Save collected statistics for current iteration to file
Nwz*Unable to save iteration stats to file: %s)
rg openr jsondumpr OSErrorrb warningr+ )ri
stats_filer} s r0 rz z AutoTracer.save_iteration_stats s : !
Z0#66 >*IfTZ 0 0*===> > > > > > > > > > > > > > > > > >
Z
Z
Z ##$PRUVWRXRXYYYYYYYYY
Z " !s: A (A A AA AA
B *(BB c x t t 5 }t j | }t |d |d |d t di |d }ddd n # 1 swxY w Y n6# t t j t f$ r t | j }Y nw xY wt | S )z@
Load statistics for latest iteration from file
r@ rA rB rD Nr. ) r r r loadr? r2 r JSONDecodeErrorKeyErrorr@ r )ri r _data stat_datas r0 load_iteration_statsz AutoTracer.load_iteration_stats s
2,--
*-- !/**+/* 445#344
-x8 2 2 2 d 011III 2i s5 A7 A A+ A7 +A//A7 2A/3A7 70B*)B*c | }| t t d t t |S )rM r )r disabled_users_quantity)r updatedictr r r )ri stats_loadeds r0 get_statszAutoTracer.get_stats sS 0022 D SQ_QaQaMbMb c c cdddr/ )N)rJ N)&r' r( r) r* r_ r~ staticmethodr r r+ r objectr r r r- r re r r r r r r r r r r r rB rw r r r rz r r
__classcell__)rj s @r0 r[ r[ l s = = = = = =#$ #$ #$ #$ #$J Nt N N N \N 1S 1Xf-= 1 1 1 \1# $ 5s 5 5 5 5
HTN
d
*d t PT I# I4 I I I I DXc] Dt D D D D GXd^ G]a Gfj G G G G 4 Sx} S S S Sj Xd^ xPS} 265
65 65 65 65p-.T#Y -. -. -. -.^/ / / / Z Z Z Z!d ! ! ! !" 4 r/ r[ c t j t j t d S )a= Absolute path of the standalone auto-tracer log.
Always resolves inside the fixed, root-owned clos_ssa log directory
(derived from ``autotracing_log``) rather than against the caller's cwd,
so the destination cannot be redirected by the directory the dev entry
point happens to be launched from.
zauto_tracer_standalone.log)rp rQ joindirnamer r. r/ r0 standalone_log_pathr " s) 7<<88:V W WWr/ log_pathc t j | t j t j z t j z t j z d }t j |d S )a Open ``log_path`` for appending and return the file object.
Uses ``O_NOFOLLOW`` so a symlink pre-planted at the target is refused
rather than followed: the open raises instead of appending through the
link to an arbitrary root-owned file.
i a)rp r O_WRONLYO_CREATO_APPEND
O_NOFOLLOWfdopen)r log_fds r0 open_standalone_logr - sA WXr{RZ 7"+ E
UW\
]
]F
9VS ! !!r/ __main__)streamlevel)Dr* rd r r r` rp collectionsr dataclassesr r r r r fnmatchr typingr
r r r
r r urllib.parser numpyr ssa.dbr ssa.internal.exceptionsr ssa.modules.commonr ssa.modules.storager r ssa.autotracingr r r ssa.configurationr ssa.configuration.schemesr ssa.internal.constantsr r ssa.internal.utilsr rs xray.shared_libraryshared_libraryro ImportErrorr} r r2 r? rF r+ rY r[ r r r' basicConfigDEBUGr r. r/ r0 r1 s # # # # # # 0 0 0 0 0 0 0 0 0 0 ( ( ( ( ( ( ( ( G G G G G G G G G G G G G G G G ! ! ! ! ! ! ! ! ! ! ! ! , , , , , , % % % % % % G G G G G G G G K K K K K K K K K K + + + + + + A A A A A A J J J J J J J J * * * * * * ******* HHHHHH
@ @ @ @ @ @ @ @ : &3 &4 & & & & s s s s s s s s l
XS X X X X"# " " " " zKMMM G 2 23F3F3H3H I IQXQ^____
AACCCCC s B B(B##B(