easycore.common.path¶

class easycore.common.path.PathManager[source]¶

Bases: object

A general path manager for URI.

static register(handler: easycore.common.path.path_handler.PathHandler, override: bool = False) → None[source]¶

Register a path handler.

Parameters:	handler (PathHandler) – override (bool) – allow overriding existing handler for prefix.

static open(path: str, mode: str = 'r', **kwargs)[source]¶

Open a stream to a URI, similar to the built-in open.

Parameters:	path (str) – A URI supported by registered PathHandler. mode (str) – Specifies the mode in which the file is opened. It defaults to ‘r’. arguments of built-in open such as encoding are also available. (other) –
Returns:	IO – a file-like object.

static copy(src_path: str, dst_path: str, overwrite: bool = False) → bool[source]¶

Copy a source path to a destination path.

Parameters:	src_path (str) – A URI or local path supported by registered PathHandler. dst_path (str) – A URI or local path supported by registered PathHandler. overwrite (bool) – forcing overwrite of existing URI.
Returns:	bool – True on success.

static copy_from_local(local_path: str, dst_path: str, overwrite: bool = False) → bool[source]¶

Copy a resource form local path to destination path.

Note

This interface is for custom PathHandler, it is prefered to use copy() instead.

Parameters:	local_path (str) – a local path. dst_path (str) – A URI supported by registered PathHandler. overwrite (bool) – forcing overwrite of existing URI.
Returns:	bool – True on success.

static get_local_path(path: str) → str[source]¶

Get a file path which is compatible with native Python I/O such as open and os.path.

Note

If URI points to a remote resource, this function may download and cache the resource to local disk.

Parameters:	path (str) – A URI supported by registered PathHandler.
Returns:	str – a file path which exists on the local file system.

static exists(path: str) → bool[source]¶

Checks if there is a resource at the given URI.

Parameters:	path (str) – A URI supported by registered PathHandler.
Returns:	bool – True if the path exists.

static isfile(path: str) → bool[source]¶

Checks if there the resource at the given URI is a file.

Parameters:	path (str) – A URI supported by registered PathHandler.
Returns:	bool – True if the path is a file.

static isdir(path: str) → bool[source]¶

Checks if the resource at the given URI is a directory.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a directory.

static listdir(path: str) → List[str][source]¶

List the contents of the directory at the given URI.

Parameters:	path (str) – A URI supported by registered PathHandler.
Returns:	List[str] – list of contents in the given path.

static makedirs(path: str) → None[source]¶

Recursive directory creation function. Similar to os.makedirs().

Parameters:	path (str) – A URI supported by registered PathHandler.

static remove(path: str) → None[source]¶

Remove the file (not directory) at the given URI.

Parameters:	path (str) – A URI supported by registered PathHandler.

static removedirs(path: str) → None[source]¶

Remove directories recursively.

Parameters:	path (str) – A URI supported by registered PathHandler.

class easycore.common.path.PathHandler[source]¶

Bases: object

Base Path handler class for a URI. It routes I/O for a generic URI which may look like “protocol://path/to/file”.

get_cache_dir(protocol: Optional[str] = None) → str[source]¶

Return a cache directory like <base-cache-dir>/protocol.

The <base-cache-dir> is

$EASYCORE_CACHE, if set
otherwise ~/.easycore/cache

Parameters:	protocol (str or None) – protocol such as ‘http’, ‘https’. If None, returns the base cache dir.

get_supported_prefixes() → List[str][source]¶

Returns:	List[str] – the list of URI prefixes the PathHandler can support.

get_local_path(path: str) → str[source]¶

Get a file path which is compatible with native Python I/O such as open and os.path.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	local_path (str) – a file path which exists on the local file system.

open(path: str, mode: str = 'r', **kwargs)[source]¶

Open a stream to a URI, similar to the built-in open.

Parameters:	path (str) – A URI supported by this PathHandler. mode (str) – Specifies the mode in which the file is opened. It defaults to ‘r’.
Returns:	IO – a file-like object.

exists(path: str) → bool[source]¶

Checks if there is a resource at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path exists.

isfile(path: str) → bool[source]¶

Checks if the resource at the given URI is a file.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

isdir(path: str) → bool[source]¶

Checks if the resource at the given URI is a directory.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

listdir(path: str) → bool[source]¶

List the contents of the directory at the given URI.

Parameters:	path (str) – A URI supported by the PathHandler.
Returns:	List[str] – list of contents in given path.

makedirs(path: str) → None[source]¶

Recursive directory creation function. Similar to os.makedirs

Parameters:	path (str) – A URI supported by this PathHandler.

remove(path: str) → None[source]¶

Remove the file (not directory) at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.

removedirs(path: str) → None[source]¶

Remove directories recursively.

Parameters:	path (str) – A URI supported by this PathHandler.

copy_from_local(local_path: str, dst_path: str, overwrite: bool = False) → None[source]¶

Copy a local file to the given URI.

Parameters:	local_path (str) – a local file path dst_path (str) – A URI supported by this PathHandler. overwrite (bool) – forcing overwirte of existing URI.
Returns:	bool – True on success.

class easycore.common.path.NativePathHandler[source]¶

Bases: easycore.common.path.path_handler.PathHandler

PathHandler for local path.

get_local_path(path: str) → str[source]¶

Get a file path which is compatible with native Python I/O such as open and os.path.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	local_path (str) – a file path which exists on the local file system.

open(path: str, mode: str = 'r', **kwargs)[source]¶

Open a stream to a URI, similar to the built-in open.

Parameters:	path (str) – A URI supported by this PathHandler. mode (str) – Specifies the mode in which the file is opened. It defaults to ‘r’.
Returns:	IO – a file-like object.

exists(path: str) → bool[source]¶

Checks if there is a resource at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path exists.

isfile(path: str) → bool[source]¶

Checks if the resource at the given URI is a file.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

isdir(path: str) → bool[source]¶

Checks if the resource at the given URI is a directory.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

listdir(path: str) → bool[source]¶

List the contents of the directory at the given URI.

Parameters:	path (str) – A URI supported by the PathHandler.
Returns:	List[str] – list of contents in given path.

makedirs(path: str) → None[source]¶

Recursive directory creation function. Similar to os.makedirs

Parameters:	path (str) – A URI supported by this PathHandler.

remove(path: str) → None[source]¶

Remove the file (not directory) at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.

removedirs(path: str) → None[source]¶

Remove directories recursively.

Parameters:	path (str) – A URI supported by this PathHandler.

copy_from_local(local_path: str, dst_path: str, overwrite: bool = False) → None[source]¶

Copy a local file to the given URI.

Parameters:	local_path (str) – a local file path dst_path (str) – A URI supported by this PathHandler. overwrite (bool) – forcing overwirte of existing URI.
Returns:	bool – True on success.

get_cache_dir(protocol: Optional[str] = None) → str¶

Return a cache directory like <base-cache-dir>/protocol.

The <base-cache-dir> is

$EASYCORE_CACHE, if set
otherwise ~/.easycore/cache

Parameters:	protocol (str or None) – protocol such as ‘http’, ‘https’. If None, returns the base cache dir.

get_supported_prefixes() → List[str]¶

Returns:	List[str] – the list of URI prefixes the PathHandler can support.

class easycore.common.path.HTTPURLHandler[source]¶

Bases: easycore.common.path.path_handler.PathHandler

Download URLs and cache them to disk.

get_supported_prefixes() → List[str][source]¶

Returns:	List[str] – the list of URI prefixes the PathHandler can support.

get_local_path(path: str) → str[source]¶

Get a file path which is compatible with native Python I/O such as open and os.path.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	local_path (str) – a file path which exists on the local file system.

open(path: str, mode: str = 'r', **kwargs)[source]¶

Open a stream to a URI, similar to the built-in open.

Parameters:	path (str) – A URI supported by this PathHandler. mode (str) – Specifies the mode in which the file is opened. It defaults to ‘r’.
Returns:	IO – a file-like object.

copy_from_local(local_path: str, dst_path: str, overwrite: bool = False) → None¶

Copy a local file to the given URI.

Parameters:	local_path (str) – a local file path dst_path (str) – A URI supported by this PathHandler. overwrite (bool) – forcing overwirte of existing URI.
Returns:	bool – True on success.

exists(path: str) → bool¶

Checks if there is a resource at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path exists.

get_cache_dir(protocol: Optional[str] = None) → str¶

Return a cache directory like <base-cache-dir>/protocol.

The <base-cache-dir> is

$EASYCORE_CACHE, if set
otherwise ~/.easycore/cache

Parameters:	protocol (str or None) – protocol such as ‘http’, ‘https’. If None, returns the base cache dir.

isdir(path: str) → bool¶

Checks if the resource at the given URI is a directory.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

isfile(path: str) → bool¶

Checks if the resource at the given URI is a file.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

listdir(path: str) → bool¶

List the contents of the directory at the given URI.

Parameters:	path (str) – A URI supported by the PathHandler.
Returns:	List[str] – list of contents in given path.

makedirs(path: str) → None¶

Recursive directory creation function. Similar to os.makedirs

Parameters:	path (str) – A URI supported by this PathHandler.

remove(path: str) → None¶

Remove the file (not directory) at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.

removedirs(path: str) → None¶

Remove directories recursively.

Parameters:	path (str) – A URI supported by this PathHandler.

class easycore.common.path.RedirectPathHandler(new_prefix: str, old_prefix: str)[source]¶

Bases: easycore.common.path.path_handler.PathHandler

Redirect a new prefix to existed prefix.

Example

PathManager.register(RedirectPathHandler("easycore://", "http://xxx.com/download/"))

get_supported_prefixes() → List[str][source]¶

Returns:	List[str] – the list of URI prefixes the PathHandler can support.

redirect(path: str) → str[source]¶

Redirect path from new_prefix to old_prefix path.

Parameters:	path (str) – path of new_prefix.
Returns:	str – path of old_prefix.

get_local_path(path: str) → str[source]¶

Get a file path which is compatible with native Python I/O such as open and os.path.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	local_path (str) – a file path which exists on the local file system.

open(path: str, mode: str = 'r', **kwargs)[source]¶

Open a stream to a URI, similar to the built-in open.

Parameters:	path (str) – A URI supported by this PathHandler. mode (str) – Specifies the mode in which the file is opened. It defaults to ‘r’.
Returns:	IO – a file-like object.

exists(path: str) → bool[source]¶

Checks if there is a resource at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path exists.

isfile(path: str) → bool[source]¶

Checks if the resource at the given URI is a file.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

isdir(path: str) → bool[source]¶

Checks if the resource at the given URI is a directory.

Parameters:	path (str) – A URI supported by this PathHandler.
Returns:	bool – True if the path is a file.

listdir(path: str) → bool[source]¶

List the contents of the directory at the given URI.

Parameters:	path (str) – A URI supported by the PathHandler.
Returns:	List[str] – list of contents in given path.

makedirs(path: str) → None[source]¶

Recursive directory creation function. Similar to os.makedirs

Parameters:	path (str) – A URI supported by this PathHandler.

remove(path: str) → None[source]¶

Remove the file (not directory) at the given URI.

Parameters:	path (str) – A URI supported by this PathHandler.

removedirs(path: str) → None[source]¶

Remove directories recursively.

Parameters:	path (str) – A URI supported by this PathHandler.

copy_from_local(local_path: str, dst_path: str, overwrite: bool = False) → None[source]¶

Copy a local file to the given URI.

Parameters:	local_path (str) – a local file path dst_path (str) – A URI supported by this PathHandler. overwrite (bool) – forcing overwirte of existing URI.
Returns:	bool – True on success.

get_cache_dir(protocol: Optional[str] = None) → str¶

Return a cache directory like <base-cache-dir>/protocol.

The <base-cache-dir> is

$EASYCORE_CACHE, if set
otherwise ~/.easycore/cache

Parameters:	protocol (str or None) – protocol such as ‘http’, ‘https’. If None, returns the base cache dir.

easycore.common.path.file_lock(path: str)[source]¶

A file lock. Once entered, it is guaranteed that no one else holds the same lock. Others trying to enter the lock will block for 30 minutes and raise an exception.

This is useful to make sure workers don’t cache files to the same location.

Parameters:	path (str) – a path to be locked. This function will create a lock named path + “.lock”.

Examples

>>> filename = "/path/to/file"
>>> with file_lock(filename):
>>>    if not os.path.isfile(filename):
>>>        do_create_file()