Rlj& U d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlZd dlZd dl m Z d dl
m Z m Z m
Z
d dlmZ d dlZd dlmZ dZdZdZej G d d
Z G d d ej j Zd
efdZ de
de d
ede fdZ ej G d d Z i a e!edz e f e"d< efdedz d
ede dej# fdZ$defdZ%dS ) N)datetime)EventProcessQueue)
SentryHandler) init_lve_utils_sentry_client
i, z4%(asctime)s - %(name)s - %(levelname)s - %(message)sc e Zd ZU dZeed< edz ed< edz ed< edz ed< edz ed< edz ed< ed e j
fd
Z d Z dS ) ProcessDescriptiona
Process description object
pid: int - process id
ppid: int | None - parent process id
user: str | None - process owner
status: str | None - process status
start_time: datetime | None - process start time
cmdline: str | None - process command line
pidNppiduserstatus
start_timecmdlineprocessc | }n# t $ r d}Y nw xY w d | }n# t $ r d}Y nw xY w t j | }n# t $ r d}Y nw xY w | }n# t $ r d}Y nw xY w | }n# t $ r d}Y nw xY w | |j ||||| S )z>DD DDD hhw 0 011GG GGG !/0C0C0E0EFFJJ JJJ ##%%DD DDD ^^%%FF FFF s7;dFJHHHsS &&'A A! A!%&B BB B4 4 CCC C+*C+c f d| j d| j d| j d| j d| j d| j S )NzPID: z, PPID: z, User: z
, Status: z
Start time: z
Command: )r r
r r r r selfs r to_strzProcessDescription.to_strM sr Etx E E E EDI E EQUQ\ E E ?E E6:lE E E )
__name__
__module__ __qualname____doc__int__annotations__strr classmethodpsutilr r r r! r r r s
HHH
*
* $J4
4ZI&. I I I [I0E E E E Er! r c t e Zd Zeeddedef fdZd Z fdZd Z d Z
d Z d
Z e
d Zd Z xZS )
QueueListenerWithHistory)timeoutrecords_history_sizer. r/ c t j |g|R ddi t j g | | _ || _ d| _ dS )a Queue listener with log records history
:param multiprocessing.Queue queue: log records queue
:param int timeout: log records flow timeout, defaults to DEFAULT_LOG_EVENTS_TIMEOUT
In case timeput is reached, last log records and process ancestors are logged
:param int records_history_size: the max number of records to log in case of timeout
respect_handler_levelT)iterablemaxlenFN)super__init__ collectionsdeque_records_history_queue_timeout_timout_registred)r queuer. r/ handlers __class__s r r5 z!QueueListenerWithHistory.__init__S sf FFFFFFFLWL]'M
M
M
#
!& r! c | j | j }| j | n*# t j $ r | Y nw xY wf|S )zDequeue log eventTr. )r; getr9 r8 appendEmpty_handle_queue_timeout)r _blockrecords r dequeuez QueueListenerWithHistory.dequeuei s~ -
-
>>+226:::;
-
-
-**,,,,,
- -
s :> #A$#A$c | t | dS )z Main loop to process log eventsN)_handle_monitor_startedr4 _monitor_handle_monitor_finished)r r= s r rI z!QueueListenerWithHistory._monitort sC $$&&&
%%'''''r! c 6 t j | _ dS )z+Start monitoring log events processing timeN)time _start_tsr s r rH z0QueueListenerWithHistory._handle_monitor_startedz s r! c | j du rc| t j dt j | j z
t
j }| | dS dS )z0Log that monitor finished processing log recordsTz/Processing log queue took %.4f seconds. Pid: %sN) r: _create_recordloggingERRORrL rM osgetpidhandle)r rs r rJ z1QueueListenerWithHistory._handle_monitor_finished~ sc !T ) )##
A dn, A
KKNNNNN * )r! c v g } t j }t | g}| x}dk r[t j | }| t | | x}dk [|S # t $ r |cY S w xY w)z%Collect process ancestors up to pid 1r )r* r r r r
rA r )r ancestorsr r
s r _get_ancestorsz'QueueListenerWithHistory._get_ancestors s .0 n&&G+77@@AI"<<>>)4a// ... !3!?!?!H!HIII #<<>>)4a// s B$B) ) B87B8c B t j t |dd||d S )zCreate log record N)namelevelpathnamelinenomsgargsexc_info)rP LogRecordr" )r r] r` ra s r rO z'QueueListenerWithHistory._create_record s1
r! c t j | j }| j t j k r| j }ndt | j d}d| j d| d| S )a6
Format one buffered record for the timeout report. DEBUG (sub-INFO)
message bodies are redacted so secrets logged only at DEBUG are not
promoted into the durable on-disk / Sentry timeout diagnostic; the
level, timestamp and body length are retained for hang diagnosis.
z[z] - z: ) r r createdlevelnorP INFOmessagelen levelname)rE timestampbodys r _format_history_recordz/QueueListenerWithHistory._format_history_record si *6>:: >W\ ) )>DDIC,?,?IIID<6#<<< K | ]}| V d S N)r ).0
proc_descrs r zAQueueListenerWithHistory._handle_queue_timeout.. s? ' '(2
!!##' ' ' ' ' 'r! Nc 3 B K | ]} | V d S rr )rn )rs rE r s r ru zAQueueListenerWithHistory._handle_queue_timeout.. sE $ $ ++F33$ $ $ $ $ $r! z8Log queue timeout. Last events:
%s
Process ancestors:
%sT)r: r rX r8 rO rP rQ rT )r proc_ancestors last_events error_msg
error_argsrU s ` r rC z.QueueListenerWithHistory._handle_queue_timeout s !U * * !YY ' '6:6I6I6K6KDDbD6Q' ' ' N )) $ $ $ $ " 9$ $ $ K VI%~6J##GM9JzJJJA KKNNN%)D " " " + *r! )r" r# r$ DEFAULT_LOG_EVENTS_TIMEOUT DEFAULT_RECORDS_HISTORY_SIZEr& r5 rF rI rH rJ rX rO staticmethodrn rC
__classcell__)r= s @r r- r- R s
6(D ' ' ' '
#& ' ' ' ' ' ', ( ( ( ( ( % % %
= = \ = * * * * * * *r! r- file_namec t j | d r t j n t j S )z*Get log level based on debug flag presencez .debug.flag)pathlibPathexistsrP DEBUGrh )r s r
get_log_levelr s6 #LI)B)B)B C C J J L L ^7==RYR^^r! q
stop_eventr. c t t d t j }t j | }| t j t | t | t | ||| }| | |
dS )z
Main function to process log events in subprocess
:param Queue q: log records queue
:param Event stop_event: stop event
:param str file_name: log file name
lvectl)r] r? N)r r rP rQ FileHandler setFormatter FormatterDEFAULT_LOG_FORMATsetLevelr r- startwaitstop)r r r r. sentry_handlerfh q_handlers r run_subpprocess_loggerr s ##?#I#IQXQ^___N Y ' 'BOOG%&899:::KK
i(()))(BPPPI
OOOO
NNr! c $ e Zd ZU eed< eed< dS )SubprocessLoggerlogger_processr N)r" r# r$ r r' r r+ r! r r r s* r! r subprocess_loggers logger_namereturnc h t | }|t j | S t j | }| t j t
}t j | }| t j | | t }t t ||||f }|
t j |j t# || t | < |S )a
Get asynchronous logger running in subprocess
Log records asynchronously in subprocess (>=info by default, >= debug when debug flag is present).
Log records from debug and higher in subprocess in case of timeout.
WARNING: multiprocessing.Queue is used to pass log records. It works asynchornously
(passing to subprocess in handled by separate thread). This may lead to log records not
being passed to the subprocess in case main process got stuck e.g. in kmodlve kernel space.
:param str | None logger_name: logger name
:param str file_name: log file
:param int timeout: log events processing timeout
N)targetra )r r@ rP getLoggerr r r r< QueueHandler
addHandlerr r r r atexitregistersetr ) r r r. subprocess_loggerloggersubprocess_logger_queue
queue_handlerr r s r init_subprocess_loggerr s ( +..{;;$ ---
{
+
+F OOGM"""%*WW$112IJJM7=)))
m$$$J $:#:J SZ"[] ] ]N
OJN###&6~z&R&R{# Mr! r\ c t | }|dS |j |j t | = dS )z
Stop subprocess logger
N)r r@ r r r r )r\ r s r stop_subprocess_loggerr s[
+..t44 $$&&&$))+++4 r! )&r r6 dataclassesrP logging.handlersrR r r; rL r multiprocessingr r r raven.handlers.loggingr r* lve_utils.sentryr r} r| r dataclassr r<
QueueListenerr- r( r r& r r r dictr' Loggerr r r+ r! r r sg
1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0
9 9 9 9 9 9 " K
.E .E .E .E .E .E .E .Ebq* q* q* q* q*w / = q* q* q*h_S _ _ _ _
e 3 QT (
:< Dt%556 ; ; ; .) )t)) )
^ ) ) ) )X ! ! ! ! ! ! !r!