easycore.common.parallel

class easycore.common.parallel.BaseRunner(devices, cfg={}, queue_scale=3.0)

Bases: object

A Multi-process runner whose consumer receive data in unorder. The runner will start multi-processes for producers and 1 thread for consumer.

__init__(devices, cfg={}, queue_scale=3.0)

Parameters:

• devices (int or Iterable) – If the devices is int, it will use devices cpu to do the work. If the devices is an iterable object, such as list, it will use the devices specified by the iterable object, such as [“cpu”, “cuda:0”, “cuda:1”].
• cfg (easycore.common.config.CfgNode) – user custom data.
• queue_scale (float) – scale the queues for communication between processes.

is_activate

whether the runner is alive.

static producer_init(device, cfg)

function for producer initialization.

Parameters:

• device (str) – device for the this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to transfer data to producer_work and producer_end function.

static producer_work(device, cfg, data)

function specify how the producer processes the data.

Parameters:

• device (str) – device for this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from producer_init function and transfer data to the next producer_work and producer_end function.
• data (Any) – data get from input of __call__ method.

Returns:

Any – processed data

static producer_end(device, cfg)

function after finishing all of its task and before close the process.

Parameters:

• device (str) – device for this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from producer_init and producer_work function.

static consumer_init(cfg)

function for consumer initialization.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it to transfer data to consumer_work and consumer_end function.

static consumer_work(cfg, data)

function specify how the consumer processses the data from producers.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from consumer_init function and transfer data to the next consumer_work and consumer_end function.

static consumer_end(cfg)

function after receiving all data from producers.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it get data from consumer_work function. Returns:Any – processed data

__call__(data_iter)

Parameters:data_iter (Iterable) – iterator of data Returns:Any – result

close()

Shutdown all processes if this runner is alive.

activate()

Restart all processes if this runner is closed.

class easycore.common.parallel.UnorderedRunner(devices, cfg={}, queue_scale=3.0)

Bases: easycore.common.parallel.engine.BaseRunner

A Multi-process runner whose consumer receive data in unorder. The runner will start multi-processes for producers and 1 thread for consumer.

__init__(devices, cfg={}, queue_scale=3.0)

Parameters:

• devices (int or Iterable) – If the devices is int, it will use devices cpu to do the work. If the devices is an iterable object, such as list, it will use the devices specified by the iterable object, such as [“cpu”, “cuda:0”, “cuda:1”].
• cfg (easycore.common.config.CfgNode) – user custom data.
• queue_scale (float) – scale the queues for communication between processes.

__call__(data_iter)

Parameters:data_iter (Iterable) – iterator of data Returns:Any – result

activate()

Restart all processes if this runner is closed.

close()

Shutdown all processes if this runner is alive.

static consumer_end(cfg)

function after receiving all data from producers.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it get data from consumer_work function. Returns:Any – processed data

static consumer_init(cfg)

function for consumer initialization.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it to transfer data to consumer_work and consumer_end function.

static consumer_work(cfg, data)

function specify how the consumer processses the data from producers.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from consumer_init function and transfer data to the next consumer_work and consumer_end function.

is_activate

whether the runner is alive.

static producer_end(device, cfg)

function after finishing all of its task and before close the process.

Parameters:

• device (str) – device for this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from producer_init and producer_work function.

static producer_init(device, cfg)

function for producer initialization.

Parameters:

• device (str) – device for the this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to transfer data to producer_work and producer_end function.

static producer_work(device, cfg, data)

function specify how the producer processes the data.

Parameters:

• device (str) – device for this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from producer_init function and transfer data to the next producer_work and producer_end function.
• data (Any) – data get from input of __call__ method.

Returns:

Any – processed data

class easycore.common.parallel.OrderedRunner(devices, cfg={}, queue_scale=3.0)

Bases: easycore.common.parallel.engine.BaseRunner

A Multi-process runner whose consumer receive data in order. The runner will start multi-processes for producers and 1 thread for consumer.

__init__(devices, cfg={}, queue_scale=3.0)

Parameters:

• devices (int or Iterable) – If the devices is int, it will use devices cpu to do the work. If the devices is an iterable object, such as list, it will use the devices specified by the iterable object, such as [“cpu”, “cuda:0”, “cuda:1”].
• cfg (easycore.common.config.CfgNode) – user custom data.
• queue_scale (float) – scale the queues for communication between processes.

close()

Shutdown all processes if this runner is alive.

activate()

Restart all processes if this runner is closed.

__call__(data_iter)

Parameters:data_iter (Iterable) – iterator of data Returns:Any – result

static consumer_end(cfg)

function after receiving all data from producers.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it get data from consumer_work function. Returns:Any – processed data

static consumer_init(cfg)

function for consumer initialization.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it to transfer data to consumer_work and consumer_end function.

static consumer_work(cfg, data)

function specify how the consumer processses the data from producers.

Parameters:cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from consumer_init function and transfer data to the next consumer_work and consumer_end function.

is_activate

whether the runner is alive.

static producer_end(device, cfg)

function after finishing all of its task and before close the process.

Parameters:

• device (str) – device for this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from producer_init and producer_work function.

static producer_init(device, cfg)

function for producer initialization.

Parameters:

• device (str) – device for the this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to transfer data to producer_work and producer_end function.

static producer_work(device, cfg, data)

function specify how the producer processes the data.

Parameters:

• device (str) – device for this process.
• cfg (easycore.common.config.CfgNode) – config of this process, you can use it to get data from producer_init function and transfer data to the next producer_work and producer_end function.
• data (Any) – data get from input of __call__ method.

Returns:

Any – processed data