noc.core.datastream.base

Module Contents

class noc.core.datastream.base.DataStream

Bases: object

Datastream stored in collection named ds_<name>. Fields: _id: Object id changeid: Change ID data: stream data (serialized JSON) hash: Object hash

name
F_ID = _id
F_CHANGEID = change_id
F_HASH = hash
F_DATA = data
F_META = meta
HASH_LEN = 16
DEFAULT_LIMIT = 1000
rx_ts
classmethod get_collection_name(cls)
classmethod get_collection(cls)

Get pymongo Collection object :return:

classmethod ensure_collection(cls)

Ensure collection is created and properly indexed :return:

classmethod get_object(cls, id)

Generate datastream object for given id. Raise KeyError if object is not found Must be overriden :param id: Object id :return: dict containing object data

classmethod get_meta(cls, data)

Extract additional metadata from .get_object() result for additional indexing :param data: .get_object() result :return: dict or None

classmethod get_deleted_object(cls, id)

Generate item for deleted object :param id: :return:

static get_hash(data)
classmethod update_object(cls, id, delete=False)

Generate and update object in stream :param id: Object id :param delete: Object must be marked as deleted :return: True if obbject has been updated

classmethod delete_object(cls, id)

Mark object as deleted :param id: :return:

classmethod get_total(cls)

Return total amount of items in datastream :return:

classmethod clean_change_id(cls, change_id)

Convert change_id to ObjectId. Following formats are possible: * ObjectId * string containing ObjectId * ISO 8601 timestamp either in form

  • YYYY-DD-MM
  • YYYY-DD-MMThh:mm:ss
Parameters:change_id – Cleaned change_id
Returns:
classmethod iter_data(cls, change_id=None, limit=None, filters=None)

Iterate over data items beginning from change id

Raises ValueError if filters has incorrect input parameters :param change_id: Staring change id :param limit: Records limit :param filters: List of strings with filter expression :return: (id, change_id, data)

classmethod on_change(cls, data)

Called when datastream changed. May alter data :param data: :return: True, if data is altered and hash must be recalculated

classmethod clean_id(cls, id)

Convert arbitrary string to id data type Raise ValueError if invalid type given :param id: :return:

classmethod clean_id_int(cls, id)

Convert arbitrary string id to int :param id: :return:

classmethod clean_id_bson(cls, id)

Convert arbitrary string id to bson int :param id: :return:

classmethod wait(cls)

Block until datastream receives changes :return:

static qs(s)

Encode string to utf-8 :param s: :return:

classmethod _parse_filter(cls, expr)

Convert single filter expression to a S-expression :param expr: filter expression in form name(arg1, .., argN) :return: (name, arg1, argN)

classmethod compile_filters(cls, exprs)

Compile list of filter expressions to MongoDB query :param exprs: List of strings with expressions :return: dict with query

classmethod filter_id(cls, id1, *args)

Filter by id. Usage:

id(id1, .., idN) :param id1: :param args: :return:

classmethod filter_shard(cls, instance, n_instances)

Sharding by id :param instance: :param n_instances: :return: