noc.core.clickhouse.shard

Module Contents

class noc.core.clickhouse.shard.BaseSharder(table=None, chunk=None)

Bases: object

TOPIC = chwriter
feed(self, records)
iter_msg(self)

Yields topic, chunk pairs :return:

pub(self)
class noc.core.clickhouse.shard.ReplicatedSharder(table=None, chunk=None)

Bases: noc.core.clickhouse.shard.BaseSharder

TOPIC = chwriter-1-%s
feed(self, records)
class noc.core.clickhouse.shard.ShardingFunction(topology=None)

Bases: object

DEFAULT_SHARDING_KEY = managed_object
SHARDING_KEYS
__call__(self, table, record)

Get list of shards for record

Parameters:
  • table – Table name
  • record – Record as dict
Returns:

List of strings

_get_sharding_function(self, topology=None)

Returns expression to be evaluated for sharding Build expression like [1, 2] if k < 2 else [3, 4] [1, 2] if k < 2 else [3, 4] if k < 3 else [5, 6] [1, 2] if k < 2 else [3, 4] if k < 3 else [5, 6] if k < 4 else [7, 8] :return:

class noc.core.clickhouse.shard.ShardingSharder(table=None, chunk=None)

Bases: noc.core.clickhouse.shard.BaseSharder

TOPIC = chwriter-%s-%s
get_shard
DEFAULT_SHARDING_KEY = managed_object
SHARDING_KEYS
feed(self, records)

Shard and replicate records

Parameters:records
Returns:
noc.core.clickhouse.shard.topo
noc.core.clickhouse.shard.Sharder