>> # Create a DataSource, use os.curdir (default) for local storage.
>>> from numpy import DataSource
>>> ds = DataSource()
>>>
>>> # Open a remote file.
>>> # DataSource downloads the file, stores it locally in:
>>> # './www.google.com/index.html'
>>> # opens the file and returns a file object.
>>> fp = ds.open('http://www.google.com/') # doctest: +SKIP
>>>
>>> # Use the file as you normally would
>>> fp.read() # doctest: +SKIP
>>> fp.close() # doctest: +SKIP
N )
set_modulec d| v rd| v rt d| dS |t d |t d dS )zCheck mode and that encoding and newline are compatible.
Parameters
----------
mode : str
File open mode.
encoding : str
File encoding.
newline : str
Newline for text files.
tbzInvalid mode: Nz0Argument 'encoding' not supported in binary modez/Argument 'newline' not supported in binary mode)
ValueErrormodeencodingnewlines L/opt/cloudlinux/venv/lib64/python3.11/site-packages/numpy/lib/_datasource.py _check_moder . sf d{{ $;; *44 9:: : ; OPP P NOO O c * e Zd ZdZd Zd Zd Zd ZdS ) _FileOpenersa
Container for different methods to open (un-)compressed files.
`_FileOpeners` contains a dictionary that holds one method for each
supported file format. Attribute lookup is implemented in such a way
that an instance of `_FileOpeners` itself can be indexed with the keys
of that dictionary. Currently uncompressed files as well as files
compressed with ``gzip``, ``bz2`` or ``xz`` compression are supported.
Notes
-----
`_file_openers`, an instance of `_FileOpeners`, is made available for
use in the `_datasource` module.
Examples
--------
>>> import gzip
>>> np.lib._datasource._file_openers.keys()
[None, '.bz2', '.gz', '.xz', '.lzma']
>>> np.lib._datasource._file_openers['.gz'] is gzip.open
True
c : d| _ d t j i| _ d S )NF)_loadedioopen
_file_openersselfs r
__init__z_FileOpeners.__init__c s "BG_r c 0 | j rd S dd l}|j | j d< n# t $ r Y nw xY w dd l}|j | j d< n# t $ r Y nw xY w dd l}|j | j d< |j | j d< n# t t f$ r Y nw xY wd| _ d S )Nr z.bz2z.gzz.xzz.lzmaT)r bz2r r ImportErrorgziplzmaAttributeError)r r r r s r
_loadz_FileOpeners._loadg s < F JJJ),D v & & D KKK(, D u % % D KKK(, D u %*.)D w ' '^,
D
s/
,,A
AA"A8 8B B c v | t | j S )a[
Return the keys of currently supported file openers.
Parameters
----------
None
Returns
-------
keys : list
The keys are None for uncompressed files and the file extension
strings (i.e. ``'.gz'``, ``'.xz'``) for supported compression
methods.
)r listr keysr s r
r# z_FileOpeners.keys s.
D&++--...r c D | | j | S N)r r )r keys r
__getitem__z_FileOpeners.__getitem__ s
!#&&r N)__name__
__module__ __qualname____doc__r r r# r' r r
r r J sZ 0- - - 6/ / /&' ' ' ' 'r r rc R t | }| | ||| S )a
Open `path` with `mode` and return the file object.
If ``path`` is an URL, it will be downloaded, stored in the
`DataSource` `destpath` directory and opened from there.
Parameters
----------
path : str
Local file path or URL to open.
mode : str, optional
Mode to open `path`. Mode 'r' for reading, 'w' for writing, 'a' to
append. Available modes depend on the type of object specified by
path. Default is 'r'.
destpath : str, optional
Path to the directory where the source file gets downloaded to for
use. If `destpath` is None, a temporary directory will be created.
The default path is the current directory.
encoding : {None, str}, optional
Open text file with given encoding. The default encoding will be
what `io.open` uses.
newline : {None, str}, optional
Newline to use when reading text file.
Returns
-------
out : file object
The opened file.
Notes
-----
This is a convenience function that instantiates a `DataSource` and
returns the file object from ``DataSource.open(path)``.
r r )
DataSourcer )pathr
destpathr r dss r
r r s, J
H B
774'7 B BBr numpyc p e Zd ZdZej fdZd Zd Zd Z d Z
d Z d Z d Z
d
Zd Zd Zd
ZddZdS )r0 a
DataSource(destpath='.')
A generic data source file (file, http, ftp, ...).
DataSources can be local files or remote files/URLs. The files may
also be compressed or uncompressed. DataSource hides some of the
low-level details of downloading the file, allowing you to simply pass
in a valid file path (or URL) and obtain a file object.
Parameters
----------
destpath : str or None, optional
Path to the directory where the source file gets downloaded to for
use. If `destpath` is None, a temporary directory will be created.
The default path is the current directory.
Notes
-----
URLs require a scheme string (``http://``) to be used, without it they
will fail::
>>> repos = np.DataSource()
>>> repos.exists('www.google.com/index.html')
False
>>> repos.exists('http://www.google.com/index.html')
True
Temporary directories are deleted when the DataSource is deleted.
Examples
--------
::
>>> ds = np.DataSource('/home/guido')
>>> urlname = 'http://www.google.com/'
>>> gfile = ds.open('http://www.google.com/')
>>> ds.abspath(urlname)
'/home/guido/www.google.com/index.html'
>>> ds = np.DataSource(None) # use with temporary file
>>> ds.open('/home/guido/foobar.txt')
>>> ds.abspath('/home/guido/foobar.txt')
'/tmp/.../home/guido/foobar.txt'
c |r-t j | | _ d| _ dS ddl}| | _ d| _ dS )z2Create a DataSource with a local path at destpath.Fr NT)osr1 abspath _destpath
_istmpdesttempfilemkdtemp)r r2 r; s r
r zDataSource.__init__ sQ # W__X 6 6DN #DOOO OOO % - - / /DN "DOOOr c x t | d r'| j r"dd l}| | j d S d S d S )Nr: r )hasattrr: shutilrmtreer9 )r r? s r
__del__zDataSource.__del__ sT 4 & & *4? * MMM MM$. ) ) ) ) ) * * * *r c | t j | \ }}|t v S )zNTest if the filename is a zip file by looking at the file extension.
)r7 r1 splitextr r# )r filenamefnameexts r
_iszipzDataSource._iszip s4 W%%h//
sm((****r c " d}|D ] }||v r dS
dS )z4Test if the given mode will open a file for writing.)w+TFr, )r r
_writemodescs r
_iswritemodezDataSource._iswritemode s4 ! AK tt ur c r | | r t j | S |dfS )zSplit zip extension from filename and return filename.
Returns
-------
base, zip_ext : {tuple}
N)rG r7 r1 rC )r rD s r
_splitzipextzDataSource._splitzipext s8 ;;x "7##H-- -T> !r c |g}| | s6t D ] }|r| ||z |S )z9Return a tuple containing compressed filename variations.)rG r r# append)r rD nameszipexts r
_possible_nameszDataSource._possible_names& s\
{{8$$ 2',,..
2
2 2LL& 111 r c T ddl m} || \ }}}}}}t |o| S )z=Test if path is a net location. Tests the scheme and netloc.r urlparse) urllib.parserW bool) r r1 rW schemenetlocupathuparamsuqueryufrags r
_isurlzDataSource._isurl/ sF *))))) 9A5wF%v&&&r c 8 ddl }ddlm} | | }t j t j | s1t j t j | | | rb || 5 }t |d 5 }| || ddd n # 1 swxY w Y ddd n # 1 swxY w Y n| || |S )zhCache the file specified by path.
Creates a copy of the file in the datasource cache.
r Nurlopenwb)
r? urllib.requestrc r8 r7 r1 existsdirnamemakedirsr` _open copyfileobjcopyfile)r r1 r? rc r\ openedurlfs r
_cachezDataSource._cache> s~
****** T"" w~~bgooe 4 455 0 K.. / / / ;;t )
5)5$'' 51&&y!4445 5 5 5 5 5 5 5 5 5 5 5 5 5 5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
OOD% ( ( ( s6 -C8>C! C8!C% %C8(C% )C88C<?C<c | | sA| | }|| | | z
}n@| | | }|| | z }|D ]E}| | r.| | r| | }|c S FdS )ay Searches for ``path`` and returns full path if found.
If path is an URL, _findfile will cache a local copy and return the
path to the cached file. If path is a local file, _findfile will
return a path to that local file.
The search will include possible compressed versions of the file
and return the first occurrence found.
N)r` rT r8 rf rn )r r1 filelistnames r
_findfilezDataSource._findfileX s {{4 =++D11H ,,T\\$-?-?@@ @HH ++DLL,>,>??H $"6"6t"<"< ddl m} | | j d }t | dk r|d } || \ }}}}}} | | }| | }t j | j || S )aF
Return absolute path of file in the DataSource directory.
If `path` is an URL, then `abspath` will return either the location
the file exists locally or the location it would exist when opened
using the `open` method.
Parameters
----------
path : str
Can be a local file or a remote URL.
Returns
-------
out : str
Complete path, including the `DataSource` destination directory.
Notes
-----
The functionality is based on `os.path.abspath`.
r rV r ) rX rW splitr9 len_sanitize_relative_pathr7 r1 join)
r r1 rW splitpathrZ r[ r\ r] r^ r_ s
r
r8 zDataSource.abspathw s 0 *))))) JJt~q11 y>>A Q>$ 4 +*****)))))) T""
7>>% 4 ;;t
!'$--
t
uu
us : B B%$B%r- Nc ` | | r$| | rt d | | }|rM| | \ }}|dk r| dd t
| |||| S t | d )aD
Open and return file-like object.
If `path` is an URL, it will be downloaded, stored in the
`DataSource` directory and opened from there.
Parameters
----------
path : str
Local file path or URL to open.
mode : {'r', 'w', 'a'}, optional
Mode to open `path`. Mode 'r' for reading, 'w' for writing,
'a' to append. Available modes depend on the type of object
specified by `path`. Default is 'r'.
encoding : {None, str}, optional
Open text file with given encoding. The default encoding will be
what `io.open` uses.
newline : {None, str}, optional
Newline to use when reading text file.
Returns
-------
out : file object
File object.
zURLs are not writeabler rJ r z not found.)r` rM r rr rO replacer FileNotFoundError)r r1 r
r r found_fnamerF s r
r zDataSource.open s D ;;t 7!2!24!8!8 7 566 6 t$$ : ++E22KFCe|| S"%%% %e$/7J J J
J $t$8$8$899 9r r- NN)r( r) r* r+ r7 curdirr rA rG rM rO rT r` rn rr r8 rw rf r r, r r
r0 r0 s . .` !# # # # #* * *+ + + " " "
'
'
' 4 >(; (; (;T 6 6 6p.: .: .: .: .: .:r r0 c R e Zd ZdZej fdZd Zd Zd Z d Z
d Z d d
Z d Z
d S )
Repositorya
Repository(baseurl, destpath='.')
A data repository where multiple DataSource's share a base
URL/directory.
`Repository` extends `DataSource` by prepending a base URL (or
directory) to all the files it handles. Use `Repository` when you will
be working with multiple files from one base URL. Initialize
`Repository` with the base URL, then refer to each file by its filename
only.
Parameters
----------
baseurl : str
Path to the local directory or remote location that contains the
data files.
destpath : str or None, optional
Path to the directory where the source file gets downloaded to for
use. If `destpath` is None, a temporary directory will be created.
The default path is the current directory.
Examples
--------
To analyze all files in the repository, do something like this
(note: this is not self-contained code)::
>>> repos = np.lib._datasource.Repository('/home/user/data/dir/')
>>> for filename in filelist:
... fp = repos.open(filename)
... fp.analyze()
... fp.close()
Similarly you could use a URL for a repository::
>>> repos = np.lib._datasource.Repository('http://www.xyz.edu/data')
c L t | | || _ dS )z>Create a Repository with a shared url or directory of baseurl.)r2 N)r0 r _baseurl)r baseurlr2 s r
r zRepository.__init__@ s& D8444
r c : t | d S r% )r0 rA r s r
rA zRepository.__del__E s 4 r c | | j d }t | dk r&t j | j | }n|}|S )z>Return complete path for path. Prepends baseurl if necessary.r rt )ru r rv r7 r1 rx )r r1 ry results r
_fullpathzRepository._fullpathH sK JJt}a00 y>>Q W\\$-66FFF
r c ^ t | | | S )z8Extend DataSource method to prepend baseurl to ``path``.)r0 rr r r r1 s r
rr zRepository._findfileQ s$ ##D$..*>*>???r c ^ t | | | S )ak
Return absolute path of file in the Repository directory.
If `path` is an URL, then `abspath` will return either the location
the file exists locally or the location it would exist when opened
using the `open` method.
Parameters
----------
path : str
Can be a local file or a remote URL. This may, but does not
have to, include the `baseurl` with which the `Repository` was
initialized.
Returns
-------
out : str
Complete path, including the `DataSource` destination directory.
)r0 r8 r r s r
r8 zRepository.abspathU s& * !!$t(<(<===r c ^ t | | | S )a
Test if path exists prepending Repository base URL to path.
Test if `path` exists as (and in this order):
- a local file.
- a remote URL that has been downloaded and stored locally in the
`DataSource` directory.
- a remote URL that has not been downloaded, but is valid and
accessible.
Parameters
----------
path : str
Can be a local file or a remote URL. This may, but does not
have to, include the `baseurl` with which the `Repository` was
initialized.
Returns
-------
out : bool
True if `path` exists.
Notes
-----
When `path` is an URL, `exists` will return True if it's either
stored locally in the `DataSource` directory, or is a valid remote
URL. `DataSource` does not discriminate between the two, the file
is accessible if it exists in either location.
)r0 rf r r s r
rf zRepository.existsl s' @ t~~d';';<< > >. = = =D C C C C@- - - - -r r )
r+ r7 r _utilsr r ri r r r r r0 r r, r r
r s7 # #H
P P P8M' M' M' M' M' M' M' M'^
")dD &C &C &C &CR GP: P: P: P: P: P: P: P:f
h- h- h- h- h-* h- h- h- h- h-r