3
nNj @ sF d dl mZ d dlmZ d dlZd dlZd dlZd dlZd dlZd dl Z d dl
m Z d dl
m
Z ddlmZ ddlmZmZmZmZmZmZmZmZ dd lmZ m Z dd
l m Z dd l!m"Z"m#Z# dd l$m%Z%m&Z& ej'e(Z)G d
d de*Z+G dd de*Z,G dd de*Z-G dd de*Z.dd Z/G dd dZ0G dd dej1Z2dS ) )absolute_import)contextmanagerN)timeout)error )HTTPHeaderDict)BodyNotHttplibCompatible
ProtocolError DecodeErrorReadTimeoutErrorResponseNotChunkedIncompleteRead
InvalidHeaderDependencyWarning) string_typesPY3) http_client)
HTTPException BaseSSLError) is_fp_closedis_response_to_headc @ s2 e Zd Zdd Zdd Zd ddZedd Zd
S )
DeflateDecoderc C s d| _ d| _d| _tj | _d S )NT )
_first_try_first_try_data _unfed_datazlib
decompressobj_obj)self r /usr/lib/python3.6/response.py__init__ s zDeflateDecoder.__init__c C s t | j|S )N)getattrr )r namer r r! __getattr__! s zDeflateDecoder.__getattr__r c C s | j | }d| _ | r$| jj r$|S |}|dk r6d}n|dkrH|| _ dS | jsf| jj| jj| |dS | j|7 _y$| jj||d}|rd| _d| _|S tjk
r d| _tjtj | _z| j| j|dS d| _X Y nX d S )Nr r )
max_lengthF)
r r unconsumed_tailr
decompressr r r r MAX_WBITS)r datar& Zoriginal_max_lengthZ decompressedr r r! r( $ s6
zDeflateDecoder.decompressc C s t | jp t | jjo | j S )N)boolr r r' r )r r r r! has_unconsumed_tailN s
z"DeflateDecoder.has_unconsumed_tailN)r- )__name__
__module__ __qualname__r" r% r( propertyr, r r r r! r s
*r c @ s e Zd ZdZdZdZdS )GzipDecoderStater r N)r. r/ r0 FIRST_MEMBER
OTHER_MEMBERS SWALLOW_DATAr r r r! r2 U s r2 c @ s2 e Zd Zdd Zdd Zd ddZedd Zd
S )
GzipDecoderc C s$ t jdt j | _tj| _d| _d S )N r )r r r) r r2 r4 _state_unconsumed_tail)r r r r! r" ^ s zGzipDecoder.__init__c C s t | j|S )N)r# r )r r$ r r r! r% c s zGzipDecoder.__getattr__r c
C s" t }| jtjkrt|S |dkr4| j|7 _dS | j| }| rT| jjrTt|S xy&|| jj|t |t
| dd7 }W n> t j k
r | j}tj| _d| _|tj
krt|S Y nX | jjp| jj | _}|dkrt
||krP |st|S | jjrVtj
| _t jdt j | _qVW t|S )Nr r )r& r8 ) bytearrayr9 r2 r6 bytesr: r eofr( maxlenr r r5 r' Z unused_datar r) )r r* r& retZprevious_stater r r! r( f s:
zGzipDecoder.decompressc C s
t | jS )N)r+ r: )r r r r! r, s z GzipDecoder.has_unconsumed_tailNr- )r- )r. r/ r0 r" r% r( r1 r, r r r r! r7 \ s
2r7 c @ s: e Zd ZdZdZdd Zdd Zddd Zed
d Z d S ) MultiDecodera
From RFC7231:
If one or more encodings have been applied to a representation, the
sender that applied the encodings MUST generate a Content-Encoding
header field that lists the content codings in the order in which
they were applied.
c C sP dd |j dD }t|| jkr s z)MultiDecoder.__init__..,z)Too many content encodings in the chain: z > c S s g | ] }t |qS r ) _get_decoder)rD er r r! rF s )splitr? max_decode_linksr
_decoders)r Zmodes encodingsr r r! r" s
zMultiDecoder.__init__c C s | j d j S )Nr )rL flush)r r r r! rN s zMultiDecoder.flushr c C s |dkr*x t | jD ]}|j|}qW |S t }x`d}x0t | jD ]"}|j||t| d}|rBd}qBW ||7 }| st||krt|S d}q2W d S )Nr F)r& Tr )reversedrL r( r; r? r< )r r* r& dr@ Zany_datar r r! r( s zMultiDecoder.decompressc C s t dd | jD S )Nc s s | ] }|j V qd S )N)r, )rD rP r r r! s z3MultiDecoder.has_unconsumed_tail..)anyrL )r r r r! r, s z MultiDecoder.has_unconsumed_tailNr- )r- )
r. r/ r0 __doc__rK r" rN r( r1 r, r r r r! rA s
rA c C s$ d| krt | S | dkr t S t S )NrG gzip)rA r7 r )moder r r! rH s
rH c @ s0 e Zd ZdZdd Zdd Zdd Zdd Zd
S ) BytesQueueBuffera Memory-efficient bytes buffer
To return decoded data in read() and still follow the BufferedIOBase API, we need a
buffer to always return the correct amount of bytes.
This buffer should be filled using calls to put()
Our maximum memory usage is determined by the sum of the size of:
* self.buffer, which contains the full data
* the largest chunk that we will copy in get()
c C s t j | _d| _d S )Nr ) collectionsdequebuffer_size)r r r r! r" s
zBytesQueueBuffer.__init__c C s | j S )N)rZ )r r r r! __len__ s zBytesQueueBuffer.__len__c C s" | j j| | jt|7 _d S )N)rY appendrZ r? )r r* r r r! put s zBytesQueueBuffer.putc C s |dkr dS | j s tdn|dk r,tdt| j d |krft| j d trf| j|8 _| j j S d}tj }x||k r || }| j j }t|}||k r|d | ||d }}|j
| | j j | | j|8 _P n|j
| | j|8 _||7 }| j stP qtW |j S )Nr r zbuffer is emptyzn should be > 0)
rY RuntimeError
ValueErrorr?
isinstancer< rZ popleftioBytesIOwrite
appendleftgetvalue) r nZfetchedr@ Z remainingchunkZ chunk_lengthZ
left_chunkZ right_chunkr r r! get s6
"
zBytesQueueBuffer.getN)r. r/ r0 rS r" r[ r] ri r r r r! rV s
rV c @ s8 e Zd ZdZddgZdddddgZdLddZdd Zdd Ze dd Z
e dd Z dd Z dd Z
d d Zd d ZdMd d!Zd"d# Zed$d% ZdNd&d'ZdOd(d)ZdQd,d-Zed.d/ Zd0d1 ZdRd2d3Zd4d5 Zd6d7 Z e d8d9 Z d:d; Z dd? Z d@dA Z!dBdC Z"dDdE Z#dFdG Z$dSdHdIZ%dJdK Z&d
S )T HTTPResponsea
HTTP Response container.
Backwards-compatible to httplib's HTTPResponse but the response ``body`` is
loaded and decoded on-demand when the ``data`` property is accessed. This
class is also compatible with the Python standard library's :mod:`io`
module, and can hence be treated as a readable object in the context of that
framework.
Extra parameters for behaviour not present in httplib.HTTPResponse:
:param preload_content:
If True, the response's body will be preloaded during construction.
:param decode_content:
If True, will attempt to decode the body based on the
'content-encoding' header.
:param original_response:
When this HTTPResponse wrapper is generated from an httplib.HTTPResponse
object, it's convenient to include the original for debug purposes. It's
otherwise unused.
:param retries:
The retries contains the last :class:`~urllib3.util.retry.Retry` that
was used during the request.
:param enforce_content_length:
Enforce content length checking. Body returned by server must match
value of Content-Length header, if present. Otherwise, raise error.
rT Zdeflatei- i. i/ i3 i4 Nr TFc C s t |tr|| _n
t|| _|| _|| _|| _|| _|| _d| _|
| _ || _
d | _ d | _ d | _
| | _d| _| | _|| _|rt |ttfr|| _ |
| _| | _t|dr|| _
d| _d | _| jjddj }dd |jdD }d |krd
| _| j || _ t | _ |r | j r | j |d | _ d S ) NFr readztransfer-encodingrk c s s | ]}|j V qd S )N)rC )rD encr r r! rQ h s z(HTTPResponse.__init__..rG chunkedT)decode_content)!r` r headersstatusversionreasonstrictro _has_decoded_contentretriesenforce_content_length_decoder_body_fp_original_response_fp_bytes_readmsg _request_url
basestringr< _pool _connectionhasattrrn
chunk_leftri lowerrJ _init_lengthlength_remainingrV _decoded_bufferrl )r bodyrp rq rr rs rt Zpreload_contentro original_responseZpool
connectionr} rv rw request_methodZ request_urlZtr_encrM r r r! r" ? sD
zHTTPResponse.__init__c C s | j | jkr| jjdS dS )a
Should we redirect and where to?
:returns: Truthy redirect location string if we got a redirect status
code and valid location. ``None`` if redirect status and no
location. ``False`` if not a redirect status code.
locationF)rq REDIRECT_STATUSESrp ri )r r r r! get_redirect_locationv s z"HTTPResponse.get_redirect_locationc C s, | j s| j rd S | j j| j d | _d S )N)r r Z _put_conn)r r r r! release_conn s zHTTPResponse.release_connc C s" | j r | j S | jr | jddS d S )NT)
cache_content)ry rz rl )r r r r! r* s zHTTPResponse.datac C s | j S )N)r )r r r r! r s zHTTPResponse.connectionc C s
t | jS )N)r rz )r r r r! isclosed s zHTTPResponse.isclosedc C s | j S )z
Obtain the number of bytes pulled over the wire so far. May differ from
the amount of content returned by :meth:``HTTPResponse.read`` if bytes
are encoded on the wire (e.g, compressed).
)r| )r r r r! tell s zHTTPResponse.tellc C s | j jd}|dk r| jr(tjd dS y.rG r z8Content-Length contained multiple unmatching values (%s)r 0 d ZHEAD)r r )
rp ri rn logZwarningsetrJ r? r popr_ r rq )r r ZlengthZlengthsrq r r r! r s,
(zHTTPResponse._init_lengthc sh j jddj } jdkrd| jkr2t| _n2d|krd fdd|jdD }t|rdt| _dS )z=
Set-up the _decoder attribute if necessary.
zcontent-encodingrk NrG c s" g | ]}|j jkr|j qS r )rC CONTENT_DECODERS)rD rI )r r r! rF s z.HTTPResponse._init_decoder..)rp ri r rx r rH rJ r? )r content_encodingrM r )r r!
_init_decoder s
zHTTPResponse._init_decoderc C s | r| j rtd|dks |r$d }y$|rF| jrF| jj||d}d| _ W nH ttjfk
r } z&| jjddj }t
d| |W Y dd}~X nX |r|r|| j 7 }|S )
zN
Decode the data passed in and potentially flush the decoder.
z_Calling read(decode_content=False) is not supported after read(decode_content=True) was called.Nr )r& Tzcontent-encodingrk zEReceived response with content-encoding: %s, but failed to decode it.r- ) ru r^ rx r( IOErrorr r rp ri r r
_flush_decoder)r r* ro
flush_decoderr& rI r r r r! _decode s"
zHTTPResponse._decodec C s$ | j r | j jd}|| j j S dS )zk
Flushes the decoder. Should only be called if the decoder is actually
being used.
r )rx r( rN )r Zbufr r r! r s zHTTPResponse._flush_decoderc c s d}zy
dV W n t k
r2 t| jddY np tk
rn } z"dt|krP t| jddW Y dd}~X n4 ttfk
r } ztd| |W Y dd}~X nX d}W d|s| jr| jj | j
r| j
j | jr| jj r| j X dS )z
Catch low-level python exceptions, instead re-raising urllib3
variants, so that low-level exceptions are not leaked in the
high-level api.
On exit, release the connection back to the pool.
FNzRead timed out.zread operation timed outzConnection broken: %rT)
SocketTimeoutr r r strr SocketErrorr r{ closer r r )r Z
clean_exitrI r r r! _error_catcher s(
zHTTPResponse._error_catcherc C s | j dkrdS t| j dd}| j ~ |dkrB|s<| j j nd}n|sR| j j|nd}|dk r|dkr| r| j j | jr| jdk r| jdkrt| j| jW dQ R X |r| jt |7 _| jdk r| jt |8 _|S )z7
Reads `amt` of bytes from the socket.
NclosedFr r )
rz r# r rl r rw r r
r| r? )r amtZ fp_closedr* r r r! _raw_readA s$
zHTTPResponse._raw_readc C s | j |dkr| j}|dk rrd}| jrX| jjrX| jd|d|t| j d}| jj| t| j|krr| jj|S | j |}d}|dkrd}n|dkr| rd}| rt| jdkr| jo| jj r|S |dkr| j|||}|r|| _
n|s| j r t d|S | j||||t| j d}| jj| xNt| j|k rz|rz| j |}| j||||t| j d}| jj| q.W | jj|}|S ) aP
Similar to :meth:`httplib.HTTPResponse.read`, but with two additional
parameters: ``decode_content`` and ``cache_content``.
:param amt:
How much of the content to read. If specified, caching is skipped
because it doesn't make sense to cache partial content as the full
response.
:param decode_content:
If True, will attempt to decode the body based on the
'content-encoding' header.
:param cache_content:
If True, will save the returned data such that the same result is
returned despite of the state of the underlying file object. This
is useful if you want the ``.data`` property to continue working
after having ``.read()`` the file object. (Overridden if ``amt`` is
set.)
NFr )r r& Tr z_Calling read(decode_content=False) is not supported after read(decode_content=True) was called.)r& )
r ro rx r, r r? r r] ri r ry ru r^ )r r ro r Z decoded_datar* r r r r! rl l sb
zHTTPResponse.readr3 r8 c c sx | j r.| j r.xd| j||dD ]
}|V q W nFxDt| j sXt| jdksX| jrr| jjrr| j ||d}|r0|V q0W dS )a_
A generator wrapper for the read() method. A call will block until
``amt`` bytes have been read from the connection or until the
connection is closed.
:param amt:
How much of the content to read. The generator will return up to
much data per iteration, but may return less. This is particularly
likely when using compressed data. However, the empty string will
never be returned.
:param decode_content:
If True, will attempt to decode the body based on the
'content-encoding' header.
)ro r )r ro N)
rn supports_chunked_reads read_chunkedr rz r? r rx r, rl )r r ro liner* r r r! stream s zHTTPResponse.streamc
K s` |j }t|ts,tr"t|j }n
tj|}t|dd}| f |||j|j|j ||d|}|S )a
Given an :class:`httplib.HTTPResponse` instance ``r``, return a
corresponding :class:`urllib3.response.HTTPResponse` object.
Remaining parameters are passed to the HTTPResponse constructor, along
with ``original_response=r``.
rt r )r rp rq rr rs rt r )
r} r` r r items from_httplibr# rq rr rs )Z ResponseClsrZ response_kwrp rt Zrespr r r! r s
zHTTPResponse.from_httplibc C s | j S )N)rp )r r r r!
getheaders s zHTTPResponse.getheadersc C s | j j||S )N)rp ri )r r$ defaultr r r! getheader s zHTTPResponse.getheaderc C s | j S )N)rp )r r r r! info s zHTTPResponse.infoc C s$ | j s| jj | jr | jj d S )N)r rz r r )r r r r! r s
zHTTPResponse.closec C s@ | j d krdS t| j dr$| j j S t| j dr8| j jS dS d S )NTr r )rz r r r )r r r r! r s
zHTTPResponse.closedc C s6 | j d krtdn t| j dr*| j j S tdd S )Nz-HTTPResponse has no file to get a fileno fromfilenozOThe file-like object this HTTPResponse is wrapped around has no file descriptor)rz r r r )r r r r! r ! s
zHTTPResponse.filenoc C s$ | j d k r t| j dr | j j S d S )NrN )rz r rN )r r r r! rN * s zHTTPResponse.flushc C s dS )NTr )r r r r! readable. s zHTTPResponse.readablec C s: | j t|}t|dkr dS ||d t|< t|S d S )Nr )rl r? )r bZtempr r r! readinto2 s
zHTTPResponse.readintoc C s t | jdS )z
Checks if the underlying file-like object looks like a
httplib.HTTPResponse object. We do this by testing for the fp
attribute. If it is present we assume it returns raw chunks as
processed by read_chunked().
fp)r rz )r r r r! r ; s z#HTTPResponse.supports_chunked_readsc C sf | j d k rd S | jjj }|jddd }yt|d| _ W n& tk
r` | j tj |Y nX d S )N ;r r r8 )
r rz r readlinerJ r r_ r httplibr
)r r r r r! _update_chunk_lengthD s
z!HTTPResponse._update_chunk_lengthc C s d }|d kr2| j j| j}|}| j jd d | _nv|| jk rZ| j j|}| j| | _|}nN|| jkr| j j|}| j jd d | _|}n | j j| j}| j jd d | _|S )Nr3 )rz Z
_safe_readr )r r Zreturned_chunkrh valuer r r!
_handle_chunkR s&
zHTTPResponse._handle_chunkc
c s | j | jstd| j s&td| j | jrNt| jrN| jj dS | j j
dkr^dS x>| j | j dkrtP | j
|}| j||d|d}|r`|V q`W |r| j }|r|V x | j j
j }|sP |dkrP qW | jr| jj W dQ R X dS )a
Similar to :meth:`HTTPResponse.read`, but with an additional
parameter: ``decode_content``.
:param amt:
How much of the content to read. If specified, caching is skipped
because it doesn't make sense to cache partial content as the full
response.
:param decode_content:
If True, will attempt to decode the body based on the
'content-encoding' header.
zHResponse is not chunked. Header 'transfer-encoding: chunked' is missing.zgBody should be httplib.HTTPResponse like. It should have have an fp attribute which returns raw chunks.Nr F)ro r r& s
)r rn r r r r r{ r r rz r r r r r r r )r r ro rh Zdecodedr r r r! r h sD
zHTTPResponse.read_chunkedc C s. | j dk r$t| j jr$| j jd jS | jS dS )z
Returns the URL that was the source of this response.
If the request that generated this response redirected, this method
will return the final redirect location.
Nr r- )rv r? historyZredirect_locationr~ )r r r r! geturl s zHTTPResponse.geturl)rk Nr r Nr TTNNNNNFNN)N)N)NNF )r N)N)NN)'r. r/ r0 rS r r r" r r r1 r* r r r r r r r r r r rl r classmethodr r r r r r r rN r r r r r r r r r r r! rj sJ
3
0
9
+
\
Crj )3Z
__future__r
contextlibr rW r rb ZloggingsyswarningsZsocketr r r r _collectionsr
exceptionsr r r
r r r
r r Z packages.sixr r r Zpackages.six.movesr r r r r Z
util.responser r Z getLoggerr. r objectr r2 r7 rA rH rV IOBaserj r r r r! s. (
<A8
<