ūj' d Z ddlZddlZddlmZ ddlmZmZmZm Z ddl
Z
ddl
m Z m Z m
Z
ddlmZmZ dZdee fd Ze G d
d Zdee fd Zd
Zde eee f fdZd ZdS )zF
Represents storage where ssa data is collected, stored and extracted
N) dataclass)ListIteratorTupleDict)funccastdistinct)
session_scope
RequestResulti@B returnc . g }dx}}d}t | D ]g\ }\ }}|dz }|||k r|}|||k r|}|t k r| | Bt j d| }|t k r|||< h|t k r
||d< ||d< |S )ai
Collect a per-path durations list bounded to ``MAX_URL_DURATION_SAMPLES``.
Below the cap every duration is kept (order preserved). Above it, reservoir
sampling (Algorithm R) keeps a uniform random sample so percentile/mean
estimates stay statistically representative without letting one hot path
materialise an arbitrarily large list.
Nr ) enumerateMAX_URL_DURATION_SAMPLESappendrandomrandint) dur_group reservoirtrue_mintrue_maxtotalseen_durationidxs J/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/storage.py_sample_durationsr s I Hx
E ( 3 3 * *mq(q x( 2 2 H x( 2 2 H * * * X & & & &.D))C---!) #
''' ! "
c D e Zd ZU eed< ee ed< eed< eed< dS )
DomainData domain_namedomain_total_reqsis_a_wordpress_domain urls_numberN)__name__
__module__ __qualname__str__annotations__r intbool r! r r# r# ? sF Cy r! r# c # z K t | 5 }| t j t j dt j t j t j t j t j
t j t t j
t j t j dt j t j t j dt j }t! j |d }|D ]^\ }}t% | }d}dgdz }|D ]*\ } }
} } }
| |t' |
< t ||
}+t) ||| | V _ ddd dS # 1 swxY w Y dS )z7
Iterates data from database domain-by-domain.
%Hc | d S Nr r/ items r z#iter_domains_data..X s
VZ[\V] r! keyr )r$ r% r& r' N)r queryr domainr strftime
created_atCountidmax wordpresscountr
pathgroup_byorder_by itertoolsgroupbytupler- r# )enginedbresults_by_hourresults_by_hour_groupedr$ groupdomain_results_by_hourr' requests_number_by_hourr hour requests_num is_wordpressurlss r iter_domains_datarT G s
v " HH $
dM$<==
=+,, 011
8M$67788
Xm*DM$
@X,Y,Y
Z
Z
Xm*DM$
@X,Y,Y
Z
Z #,"3OI]I]"^"^"^ "9 K%*5\\ "K ()cBh #=S
5
594 |T5A'D 2 !+t 4 4 '"9&2 '
s FF00F47F4c
# K t | 5 }| t j t j dt j t j t t j t j d t j
t j d t j t t j t j d t j |k t j | t j t j dt j t j t j dt j }d}dgdz dgdz dgdz }}}|D ]q\ } }
} } }
|r/|| k r)|t+ |||| fV dgdz dgdz dgdz }}}| |t- |
< | |t- |
< |
|t- |
< | }r| t+ | ||| fV ddd dS # 1 swxY w Y dS ) z6
Iterates urls data from database url-by-url.
r1 url_throttled_reqsurl_total_reqs
url_slow_reqsNr r9 rC rV rW rX )r r: r rC r r< r= Sumr hitting_limits
sqlalchemyIntegerlabelr> r? is_slow_requestfilterr; in_rD rE dictr- )rI r$ all_pathsrJ urls_data
previous_pathrV rW rX rC rP url_throttled_req
url_total_req url_slow_reqs r iter_urls_datari m s
v -
" HH "
dM$<==m :JV*W*W
X
X
Xm($-m>V*W*W
X
X
=>C"HqcBhQRPSVXPXMNJS ! ! FD$)=,
a $!6!6! *+='5&3 FGC"HqcTVhYZX[^`X`MN",= s4yy )(5N3t99 %'3M#d)) $ MM
#5 - +
K-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
s I$JJ
Jc # K t | 5 }| t j t j t j |k t j }t j |d }|D ]\ }}|d |D fV ddd dS # 1 swxY w Y dS )zA
Get information about durations of requests url-by-url.
c | d S r3 r/ r4 s r r6 z#get_url_durations.. s
d1g r! c g | ]\ }}|S r/ r/ ).0r r s r
z%get_url_durations.. s :::[Q:::r! N)
r r: r rC r r` r; rE rF rG )rI r$ rJ rd durations_by_pathr8 rM s r get_url_durationsrp s
v
;" HH]' )? @ @
VM(K7
8
8
Xm(
)
) & -i9M9M N N+ ; ;JC::E:::: : : : : ;
;
;
;
;
;
;
;
;
;
;
;
;
;
;
;
;
;
;s BB77B;>B;c
# K t | 5 }| t j t j dt j t j t t j t j d t j
t j d t j t t j t j d t j |k t j t j dt j t j t j dt j }| t j t j t j |k t j }t+ j |d }t+ j |d }t/ |d }t/ |d }|||\ } }
|\ } } | | k rd gd
z }
d gd
z }d gd
z }|
D ]>\ }}}}}||
t1 | < ||t1 | < ||t1 | < ?t3 | }| |t5 | |
|| fV t/ |d }t/ |d }n'| | k rt/ |d }nt/ |d }||ddd dS # 1 swxY w Y dS ) u
Stream per-URL data for a single domain.
Yields ``(path, durations, url_data)`` per path, where:
- ``path`` is the URL path (str);
- ``durations`` is a list of raw request durations for that path;
- ``url_data`` is a dict with per-hour aggregates
(``url_throttled_reqs``, ``url_total_reqs``, ``url_slow_reqs``
as 24-element lists) plus ``path``.
Merges two ``ORDER BY path`` streams in lock-step instead of
materialising the per-domain ``{path: [durations...]}`` dict up
front, which caused OOM on large databases (CLPRO-3077). A second
benefit: this drops the redundant ``WHERE path IN (...)`` clause —
both queries already filter by ``domain``, so the path list adds
no selectivity, only memory pressure on the SQL parser.
r1 rV rW rX c | d S r3 r/ rs r r6 z&iter_domain_url_data..
1 r! r7 c | d S r3 r/ rs s r r6 z&iter_domain_url_data.. ru r! Nr r9 rY )r r: r rC r r< r= rZ r r[ r\ r] r^ r> r? r_ r` r; rD rE r rF rG nextr- r rb )rI r$ rJ agg_query dur_query agg_by_path dur_by_pathagg_itemdur_itemagg_path agg_groupdur_pathr rV rW rX r rP throttledr slow durationss r iter_domain_url_datar sD $
v =3" HH "
dM$<==m :JV*W*W
X
X
Xm($-m>V*W*W
X
X
HH]' )? @ @
VM(K7
8
8
Xm(
)
) ' ~~FFF ' ~~FFF T** T**"x';"* Hi"* Hi8##&'S2X""#r!"b
7@ 4 43AtYt4=&s4yy105N3t99-/3M#d)),, -i 8 8 %+='5&3 T22 T22H$$
T22 T22G "x';5=3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3 =3s L%MM M)__doc__rF r dataclassesr typingr r r r r\ r r r
ssa.dbr r r r- r r# rT ri r+ rp r r/ r! r r sq
! ! ! ! ! ! . . . . . . . . . . . . + + + + + + + + + + / / / / / / / / % DI F #*!5 # # # #L1
1
1
h;d3c
?.C ; ; ; ;"O3 O3 O3 O3 O3r!