noc.core.etl.loader.base

Module Contents

noc.core.etl.loader.base.logger
class noc.core.etl.loader.base.BaseLoader(chain)

Bases: object

Import directory structure: var/

import/
<system name>/
<loader name>/

import.csv[.gz] – state to load, can have .gz extension mappings.csv – ID mappings archive/

import-YYYY-MM-DD-HH-MM-SS.csv.gz – imported state

Import file format: CSV, unix end of lines, UTF-8, comma-separated First column - record id in the terms of connected system, other columns must be defined in fields variable.

File must be sorted by first field either as string or as numbers, sort order must not be changed.

mappings.csv - CSV, unix end of lines, UTF-8 comma separated mappings of ID between NOC and remote system. Populated by loader automatically.

Parameters:fields – List of either field names or tuple of (field name, related loader name)
name
model
mapped_fields
fields = []
tags = []
PREFIX
rx_archive
discard_deferred = False
ignore_unique
REPORT_INTERVAL = 1000
is_document

Returns True if model is Document, False - if Model

load_mappings(self)

Load mappings file

get_new_state(self)

Returns file object of new state, or None when not present

get_current_state(self)

Returns file object of current state or None

diff(self, old, new)

Compare old and new CSV files and yield pair of matches * old, new – when changed * old, None – when removed * None, new – when added

load(self)

Import new data

find_object(self, v)

Find object by remote system/remote id :param v: :return:

create_object(self, v)

Create object with attributes. Override to save complex data structures

change_object(self, object_id, v)

Change object with attributes

on_add(self, row)

Create new record

on_change(self, o, n)

Create change record

on_delete(self, row)

Delete record

purge(self)

Perform pending deletes

save_state(self)

Save current state

clean(self, row)

Cleanup row and return a dict of field name -> value

clean_str(self, value)
clean_map_str(self, mappings, value)
clean_bool(self, value)
clean_reference(self, mappings, r_model, value)
clean_int_reference(self, mappings, r_model, value)
set_mappings(self, rv, lv)
update_document_clean_map(self)
update_model_clean_map(self)
check(self, chain)
check_diff(self)
check_diff_summary(self)