@jQ d Z ddlZddlZddlZddlZddlZddlZddlZddlm Z m
Z
m Z m Z m
Z
ddlZddlmZ ddlmZ ddlmZmZmZmZ ddlmZ dZd Zd
Z ej e d Z ej e d Z d
Z!dZ"dZ#dZ$dZ%dZ& e'd Z( ej) d Z*de e+ fdZ,de+de+de-fdZ.de+de
e+e f de e+ de/fd Z0d e de
e+d f fd Z1de
e+e f fd Z2d!e
e+e f ddfd"Z3d#e+d$e e+ de
e e+ e e+ f fd%Z4de
e+e+f fd&Z5d=d(e+d)e+d*e-ddfd+Z6d(e+de/fd,Z7de+de/fd-Z8de+ddfd.Z9d/e e+ ddfd0Z:d1e
e+e f de e+ de
e-e-f fd2Z;de-fd3Zd5e e+ de+fd6Z>d7e e
e+e f ddfd8Z?d?d:e/d#e e+ de-fd;Z@de e
e+e f fd<ZAdS )@az Fetch the published settings and materialise this server's feature flags.
The whole module is written to fail quiet and change nothing. A feature flag
file that flips because the repository was briefly unreachable would be worse
than one that is briefly stale: the flags gate real behaviour on customer
servers, and the settings file is the only thing entitled to change them.
N)AnyDictOptionalSetTuple)get_rhn_systemid_valuelist_flags_info)DEFAULT_CONFIG_STORAGE_PATHDEFAULT_FLAGS_STORAGE_PATH DEFAULT_OPT_OUT_STORAGE_PATH DEFAULT_REMOTE_OFF_STORAGE_PATH) FLAG_NAME_REz6https://repo.cloudlinux.com/static/cl-settings-v1.jsonCL_FLAGS_SETTINGS_URLz/var/lve/cl-flags-syncz
settings.jsonz
state.json i i i, iQ < )z
cl-statisticszcl-website-collector clflagssyncreturnc t d } n-# t $ r t dd Y dS w xY w| S )zThe identifier this server is bucketed by, or None when it has none.
The same `system_id` the telemetry and the existing rollout use. A server
that is not registered has none; see resolve() for what that means.
system_idzcould not read system_idT)exc_infoN)r Exceptionloggerwarning)r s py/clflagssync/sync.py server_idr K sW #9+#F#F 1DAAAtt s &<<identitykeyc t j | d| d }t |dd d dz S )a A stable 0-99 for this (server, feature-or-cohort) pair.
Deterministic, so a server never moves between polls, and evenly spread,
so `bucket < percent` selects about that share of the fleet. Because the
number never changes, raising a percentage only ever adds servers.
The key includes the feature name (or its cohort) so that the 5% for one
feature and the 5% for another are different servers - otherwise the same
unlucky servers would be first for everything.
:utf-8N d )hashlibsha256encode hexdigestint)r r digests r bucketr+ Y sX ^x / /# / / 6 6w ? ?
@
@
J
J
L
LF vcrc{B # %% featureentryc N t |t s*t d| t d |L|t | d v rdS |t | d v rdS | d du rdS | d }|dS t |t r$t |t sd
|cxk rd k s n t d | | dS |d k rdS |d
k rdS | t d
| | dS | d }|3t |t s t d| | d}t ||p| |k S )zAWhether `feature` should be on for this server. First match wins.z,%s: entry is not an object, leaving it alonezmalformed entryNz servers-offFz
servers-onTenabledpercentr r$ z7%s: percent %r is not an integer 0-100, treating as offz3%s: no system_id, so not included in a %d%% rolloutcohortz*%s: cohort %r is not a string, ignoring it)
isinstancedictr r
ValueError_as_listgetr) boolinfostrr+ )r- r. r r1 r2 s r resolver; l s eT " " ,EwOOO*+++ x - 8 899 9 95 x , 7 788 8 84 yyu$$uii ""Gt gs # # z'4'@'@ WH[H[H[H[X[H[H[H[H[PRY[bcccu#~~t!||u
I7T[\\\u
YYx
F
*VS"9"9CWfUUU (F -g . . 88r, value.c b t | t sdS t d | D S )N c 3 D K | ]}t |t |V d S N)r3 r: ).0items r z_as_list.. s1 AA$:dC+@+@AAAAAAAr, )r3 listtuple)r< s r r6 r6 s6 eT " " r AA%AAA A AAr, c t t d 5 } t j | }d d d n # 1 swxY w Y t |t
r|ni S # t t f$ r i cY S w xY wNr! )encoding)open META_FILEjsonloadr3 r4 OSErrorr5 )handlestates r _load_staterP s
)g
.
.
. &&If%%E & & & & & & & & & & & & & & &"5$//7uuR7 Z s. A 9 A =A =A A32A3rO c X t t t j | d d S )Ni mode)
_write_atomicrJ rK dumps)rO s r _save_staterV s' )TZ . .U;;;;;;r, urletagc |rd|ini }t j | |t d 5 }|j dk rd|fcddd S | |j d }|F| r2t | t k rt d| dt t }| d D ]7}||z
}t | t k rt d
t d 8|
d |j d
fcddd S # 1 swxY w Y dS )a (body, etag) from the settings URL.
(None, etag) means the server said nothing changed - the common case, and
the reason polling costs requests rather than bandwidth.
Raises on anything that leaves us without an answer; the caller treats that
as "change nothing".
z
If-None-MatchT)headerstimeoutstreami0 NzContent-Lengthzsettings claims z bytes, more than i zsettings is larger than z bytesr! ETag)requestsr7 HTTP_TIMEOUT status_coderaise_for_statusrZ isdigitr) MAX_SETTINGS_BYTESr5 bytearray iter_contentlendecode)rW rX rZ responsedeclaredbodychunks r fetchrl s *.5%%2G c7L N N N BRZ 3 & &:B B B B B B B B !!####''(899 H$4$4$6$6 3x==K];];] ` ` `L^ ` `aa a {{ **400 X XE EMD4yy--- !V )namepath)rA flags r
z"declared_flags.. s ? ? ?TDIty ? ? ?r, r r> r, r declared_flagsrs s @ ?_->-> ? ? ??r, rp contentrS c t j | }t j |d t j dd|dd } |5 | | d d d n # 1 swxY w Y t j |j | t j |j | d S # t $ r t |j w xY w)NT)exist_okwr! z.tmp-F)rS rH dirprefixdelete) osrp dirnamemakedirstempfileNamedTemporaryFilewritechmodro renamer _unlink)rp ru rS directoryrN s r rT rT s %%IK D))))
(cG[bkp
q
q
qF
" " LL ! ! ! " " " " " " " " " " " " " " "
d###
&+t$$$$$
s0 B5 A4( B5 4A88B5 ;A8<7B5 5 Cc t j | dS # t $ r!}|j t j k r Y d}~dS d}~ww xY w)zGRemove `path`; True when it was there. Absent is success, not an error.TNF)r| unlinkrM errnoENOENT)rp errs r r r sW
$t 9 $ $ uuuuus
A>Ac t j t j t | dz S )z8Whether an admin has opted this server out of `feature`..flag)r| rp isfilejoinr
)r- s r
_opted_outr s,
7>>"',,'CWwEVWW X XXr, c Z t j t | dz }t j | rdS t |d n4# t $ r'}t d| | Y d}~dS d}~ww xY wt d| dS )zRecord in remote-off.d that the settings switch `feature` off here.
Best effort: the caller removes the feature's switch file whatever happens
here, and removing a file works on a full disk where writing one may not.
r N z %s: could not mark it off: %sz%s: marked off)
r| rp r r r rT rM r errorr9 )r- rp r s r _mark_offr s 7<<779J K KD w~~d dB 4gsCCC KK '*****s A
B
& BB
keepc t j t }n# t $ r Y dS w xY wd |D }|D ]}|| v st j | s t
t j t |dz rt
d| m# t $ r&}t d|| Y d}~d}~ww xY wdS )z?Remove every off marker in remote-off.d except those of `keep`.Nc h g | ]/}| d |dt d 0S r Nendswithrf rA r. s r
z _unmark_off.. : WWW5u~~g?V?VWoW
o&WWWr, r z%s: no longer marked offz'%s: could not remove its off marker: %s) r| listdirr rM r matchr rp r r r9 r )r entriesfeaturesr- r s r _unmark_offr s * <== WWGWWWH R R d??,"4W"="=? Rrw||$CWwEVWWXX
A 6@@@ R R R LLBGS Q Q Q Q Q Q Q Q R
R Rs%
**AB((
C2 CCsettingsc 0 i }t }| D ]\ }}| d s |t v r$t j | s t d| T t ||| ||< j# t $ r |
| Y w xY wd | D }t | D ]}t | t }dx}} | D ]\ }}
||vr t d| &t j t$ |dz } t j || | k r#t d||| |
r,t) | r t d| d }
t j t, |d
z } |
r| | d }
t1 |
t2 r'|
r%t5 | t7 j |
d
nt; | t j | s+t5 | d t d| |dz
}t; | rt d| t; | | dz
} t? ||z || fS )a Bring the flag files in line with `settings`. Returns (enabled, disabled).
Every feature the settings switch off here gets an off marker in
remote-off.d, whether or not it is installed: clflags.is_enabled() answers
"off" while the marker is there, so a package that ships the feature on
installs it off. Switch files are touched only for features named in the
settings AND declared on this server; everything else keeps whatever state
it shipped with. A feature an admin opted out of is never switched on.
_z)%r is not a usable flag name, skipping itc h | ] \ }}||
S r> r> )rA r- wanteds r z!apply_settings..3 s! SSSFSGSSSr, r z.%s is not declared on this server, skipping itr z=%s declares its own path %s; remote settings do not manage itz&%s: opted out on this server, kept offF.jsonconfigT) sort_keysr z %s: enabled z %s: disabled) setitems
startswithNOT_OURSr r r r r; r5 addsortedr rs debugr| rp r r normpathr r9 r r7 r3 r4 rT rK rU r r r )r r decisions malformedr- r. switched_offri
enabled_countdisabled_countr flag_path config_pathr s r apply_settingsr s% "$I %%I"..** # # c " " g&9&9 !'** NNF P P P #!(%!B!BIg # # # MM' " " " " " #
TS9??3D3DSSSL,'' ' H%&&MN$??,, $ $ ( " " LLI7 S S S GLL!;Ww=NOO
7 HW - . .) ; ;
NNZ\cemnuev w w w j))
KK@' J J JFgll#>'@QRR g&**844F&$''
%F
% k4:f+M+M+MNNNN $$$7>>),,
4 i,,, M7333 Q MMy!!
5 NG444 K a NN y())) . ((s :B B10B1c t j t } n# t $ r Y dS w xY wd | D }d |D }|sdS t }d}|D ]}t j t |dz }||vs)t j || |k rX t | r t d| |dz
}t t j t |dz # t $ r&}t
d|| Y d }~d }~ww xY w|S )
a Switch off every feature an admin opted out of. Returns how many were on.
Called at the start of every run, whether or not a fetch is due and whether
or not the settings can be reached: a package that ships a feature on puts
its marker back on every update, and an opt-out has to outlast that. Like
the rest of the bridge it touches only flags declared at the default marker
path - a flag with a path of its own is a local switch of its component.
r c h g | ]/}| d |dt d 0S r r r s r r z$enforce_opt_outs..u r r, c : g | ]}t j | |S r> )r r )rA r- s r r z$enforce_opt_outs..v s( OOOG<3Eg3N3NOOOOr, r z*%s: opted out on this server, switched offr r z.%s: opted out, but could not switch it off: %sN)r| r r
rM rs rp r r r r r r9 r r )r r ri r r- r r s r enforce_opt_outsr h s * 9:: qqWWGWWWHOOxOOOH q HL
Y
YGLL!;Ww=NOO ( " "bg&6&6x7H&I&IY&V&V Yy!!
" H'RRR ! BGLL!OPP Q Q Q Q Y Y Y LLI7TW X X X X X X X X Y s%
**4A$D
E # EE c \ t | d t r*| d pi d nd}t |t rt |t rt
S t
t t t | S )zHow long to wait between fetches, as the settings file asks.
Lets the fleet be slowed down centrally without shipping a package, which
is the one idea worth borrowing from how Imunify schedules the same job.
_bridgez
poll-intervalN)
r3 r7 r4 r) r8 DEFAULT_POLL_INTERVALmaxMIN_POLL_INTERVALminMAX_POLL_INTERVAL)r raws r _poll_intervalr s CMX\\ZcMdMdfjBkBk
u8<< " " (b
-
-o
>
>
>quC c3 %:c4#8#8 %$$ #&7"="= > >>r, overridec | p#t j t }|r t d| |S t S )aJ Where to fetch the settings from.
Production has exactly one answer; the override exists so the mechanism can
be exercised against a file we control before it is released, without a
round trip through the real publishing path. Logged at warning level so an
override left behind is visible rather than silent.
z using overridden settings URL %s)r| environr7 SETTINGS_URL_ENVr r SETTINGS_URL)r chosens r settings_urlr sE
9 (899F
96BBB
r, cachedc | dS t | t dS # t $ r&}t d| Y d}~dS d}~ww xY w)a Re-apply the last settings fetched, on a run that got nothing new.
A package that ships a feature's marker reinstalls it on every update, and a
hand edit can change one at any time. Holding the flags to the last
published state on every run - not only when a fetch is due and succeeds -
keeps a switched-off feature off within one run of the timer, outage or not.
Nothing is invented: without a cached copy there is nothing to apply.
N#could not update the flag files: %s)r r rM r r )r r s r _apply_lastr sx ~Avy{{+++++ A A A :C@@@@@@@@@As $
AAAFforcec t n*# t $ r t d Y nw xY wt | }t }t
}| st | dd t |pi z t z
}t j |k r9t dt | t | dS t j |d< t | n2# t $ r%}t d| Y d}~nd}~ww xY w t% ||| d nd \ }}nC# t $ r6}t d|| t | Y d}~dS d}~ww xY w| t' j | } t+ | t, st/ d nC# t. $ r6}t d
|| t | Y d}~dS d}~ww xY w t3 t4 |d nB# t $ r5}t d
| t | Y d}~dS d}~ww xY w| }n | t d dS ||d< t j |d< t | n2# t $ r%}t d| Y d}~nd}~ww xY w t7 |t9 \ }
} n3# t $ r&}t d| Y d}~dS d}~ww xY wt d|
| dS )zOne pass. Returns 0 always: a timer unit that goes red on every network
blip trains people to ignore it, and the journal already carries the why.z"could not apply the local opt-outs attempted_atr z not due yet, next fetch at %dz could not record the attempt: %sNrX z8could not fetch %s (%s); the last settings stay in forcez settings is not a JSON objectz6%s is not usable (%s); the last settings stay in forcert rR zBcould not cache the settings (%s); the last settings stay in forcezKserver reports no change but nothing is cached; leaving every flag as it is
fetched_atz could not record fetch state: %sr z)flags up to date: %d enabled, %d disabled) r r r exceptionr rP _read_cachefloatr7 r
POLL_SLACKtimer r) r rV rM r rl rK loadsr3 r4 r5 r rT
CACHE_FILEr r r9 ) r rW sourcerO r duer rj rX parsedr0 disableds r runr s ? ? ? ?=>>>>>? #
F MME
]]F EIIna0011N6 r, r r s
2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 # # # # # # ( ' ' ' ' 'G + $
W\\)_
5
5
GLLL 1 1
9> ? ? = ) ) 8C= &S &s &s & & & &&'9S '9c3h '98C= '9T '9 '9 '9 '9TBC BE#s(O B B B BT#s(^