EHi d Z dZddlZddlZddlZddlmZ ddlmZ ddl m
Z
m Z ddl m
Z
ddlmZ dd lmZmZ dd
lmZ dd lmZ dd lmZ d
dlmZ d
dlm Z ddl m Z ddl m Z ddl!m"Z"m#Z# ddl$m%Z%m&Z&m'Z'm(Z( G d de Z)e*dk rc e& ej+ dej, e) - dS # e $ rZ. e/e. e0d
dZ.[.ww xY wdS )zd
This module contains SSA classes for sending e-mails with report
and sending reports to ClickHouse
z ssa.modules N) EmailMessage)partial)OptionalAny) gethostname)get_admin_email)SessionResponse) HTTPAdapter)RequestException)Retry )Common)
DecisionMaker ) stat_server)SSAError)Mailerrender_report_table) read_sys_id sentry_init
duration_cast format_datec e Zd ZdZde dfdef fd
Zddeddfd Zd
ede e
fd Z de e fd Z de e fd
Z
d
edefdZededefd ZdedefdZdedefdZ xZS )StatisticsSenderz4
Send report to ClickHouse and over e-mails
https://z
/api/clos-ssa ch_endpointc : t t j d | _ | j dt t | _ || _ t | _ t dt dg t g d d }t | }t | _ | j d| t% | j j d
| j _ d S ) N stat_senderz StatisticsSender enabled: %s POST)i i i )totalallowed_methodsstatus_forcelistbackoff_factor) max_retriesr
)timeout)super__init__logging getLoggerloggerinfo __package__r sys_idr r mail_senderr
frozensetr r sessionmountr request)selfr
retry_confadapter __class__s N/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/stat_sender.pyr* zStatisticsSender.__init__* s
'
66 7EEE!mm &!88+4fX+>+>,5ooo,F,F*+- - -
*555 yy :w/// &t|';R H H H Nreportreturnc | t }| j r| | | | dS )zi
Send given report over e-mail
If no report given, get it from DecisionMaker API
N)r get_json_report summary_notification_enabled email_reportclickhouse_report)r6 r< s r: sendzStatisticsSender.send; sW
>"__4466F , & f % % % v&&&&&r; report_viewc | }|r| d rt | \ }}| j |dt |d t
| }| t j | dd|d d | |d d|d d
| j
| |S d S d S ) z4
Create and send e-mail with report
domains
ssa_reportdate) recipienttemplaterH hostnamehtmljsonreport_z.json)subtypefilenamerL z.htmlN) get_mail_recipientgetr r1 _messager r add_attachmentrM dumps_send)r6 rD mail_to report_tablemailmsgs r: rA z StatisticsSender.email_reportG s/ ))++ {y11 !4[!A!A L$"++!% V!455$ , C
tz+ 6 6'-(L+f2E(L(L(L
N
N
N
|'-(L+f2E(L(L(L
N
N
N
" "3 ' ' 'J% r; c | p
t S # t $ r3}| j dt | Y d}~dS d}~ww xY w)z
Retrieve a recipient's e-mail:
1. get address from a wmt-api utility
2. if command failed or address is empty, get address of server admin
z get_admin_email failed with: %sN)wmt_api_report_emailr Exceptionr- errorstr)r6 es r: rQ z#StatisticsSender.get_mail_recipienta sy I,,..C/2C2C C I I I K ?Q H H H H H H H H H Is !$
A!(A A!c p d} t j d| dgddd j }nX# t j t
t t f$ r3}| j dt | Y d}~dS d}~ww xY w t j |
d
d S # t j $ r3}| j d
t | Y d}~dS d}~wt
$ r | j d | Y dS w xY w) zD
Retrieve a recipient's e-mail address from WMT API
zwmt-apiz&/usr/share/web-monitoring-tool/wmtbin/z --config-getT)checktextcapture_outputzwmt-api utility failed: %sNconfig report_emailz!wmt-api returned invalid json: %sz(wmt-api returned unexpected response: %s)
subprocessrunstdoutstripCalledProcessErrorAttributeErrorOSError
ValueErrorr- r^ r_ rM loadsrR JSONDecodeError)r6 _util api_responser` s r: r\ z%StatisticsSender.wmt_api_report_emailm s %>A%AA ! # % % % &,EEGG
L
-~w K :CFF C C C FFFFF
,:l++//99==nMM M# & & & K A !!ff
&
&
&
&
&
&
&
&
& , , , K H *
,
,
,
,
,
, ,s3 49 "B(B B9C D5(D (D54D5c | d r| j d| j | j | j | | | }nV# t $ rI}| j d|d| j i t dt | |d}~ww xY w| | S | j d d
S ) z+
Send report to ClickHouse
rF zSending POST request to %s)rM zPOST failed with %sendpointextraz'Failed to POST data to SSA API server: Nz*Report is empty, not sending to ClickHouseF)
rR r- r. r r3 post_ch_packclickhouse_formatr r^ r r_ _process_response)r6 rD respr` s r: rB z"StatisticsSender.clickhouse_report s5 ??9 % % K 9 ! -
/
/
/
O|(()9
**;77A9 A9( : :#
O
O
O !!"7)3T5E(F " H H H Fc!ffFFH HMNO
O
))$// / K I J J J5s AA? ?
C AC
Cvaluec " t | S )z<
Pack given value into data field of a dict
)data)dict)r| s r: rx zStatisticsSender._ch_pack s
r; original_reportc f t }| d D ]}| | j | d | d | d | d d | d D d |S ) zd
Format local report for sending to ClickHouse
(required structures differ)
rF name slow_urls slow_reqs
total_reqsc g | ]p}| d | d t | d t | dd dqS )r reqs_numaverage_duration correlationr )urlcount_requests avg_durationr )rR r float).0us r:
z6StatisticsSender.clickhouse_format.. s !uuV}}*+%%
*;*;(5 EE"4 5 5)7 )7',QUU=!-D-D'E'E r; urls) system_iddomaincount_slow_urlscount_slow_requeststotal_requestsdetails)listrR appendr0 )r6 r ch_reportr s r: ry z"StatisticsSender.clickhouse_format s
FF %)))44 F ![ **V,,#)::k#:#:'-zz+'>'>"(**\":": &zz& 1 1
r; responsec |j s1| j d|j |j d|j i dS | j d|j |j |j | }|d dk r'| j d|d d |i dS | j d
d S ) z
Check received response
:param response: a requests.Response object
:return: True in case of success, False otherwise
z&Unable to connect to server with %s:%s resp_textru Fz [%s:%s] Response received %sstatusokz Received response with status %sr z Sent to ClickHouse successfullyT) r r- r^ status_codereasonrc r. r rM )r6 r results r: rz z"StatisticsSender._process_response s { + K F & 2HO%0(-$@
B
B
B 5 K ; % 18? %\
+
+
+ ( t # # K @ $X .z66J
L
L
L5 :;;;tr; )N)__name__
__module__ __qualname____doc__r r_ r* r rC r r rA rQ r\ boolrB staticmethodr rx r ry r
rz
__classcell__)r9 s @r: r r % s +P[*O*O*O I IC I I I I I I"
'
'4
'4
'
'
'
' ,1G 4
IHSM
I
I
I
I,hsm , , , ,2T d ( \ $ 2( t r; r __main__zstat_sender_standalone.log)rP level)1r r/ rM r+ rg
email.messager functoolsr typingr r socketr clcommon.cpapir requestsr r
requests.adaptersr requests.exceptionsr $requests.packages.urllib3.util.retryr
commonr decision_makerr internal.constantsr internal.exceptionsr internal.mailerr r internal.utilsr r r r r r basicConfigINFOrC excprint
SystemExit r; r: r sU & & & & & & * * * * * * & & & & & & & & ) ) ) ) ) ) 0 0 0 0 0 0 6 6 6 6 6 6 ) ) ) ) ) ) , , , , , , * * * * * * 9 9 9 9 9 9 9 9 l l l l lv l l l^ zKMMMG!= %l, , , , !!!!!
c
jmm
s - C C+C&&C+