NOC 19.3

In accordance to our Release Policy we’re proudly present release 19.3.

19.3 release contains 318 bugfixes, optimisations and improvements.

Highlights

ConfDB Object and Interface Validation Policies

NOC 19.3 brings ability to validate object and interface settings using ConfDB Queries. This feature will became primary way of validation and will replace old CLIPS-based implementation in NOC 19.5.

It is good time to begin legacy validation policy migrations.

ConfDB Syntax Expansion

ConfDB got additional syntax for:

  • NTP support

  • Video, Audio settings and media streaming

ConfDB ‘raw’ Section

Raw tokenized input can be applied to ConfDB under the raw node. So platform-depended syntax can be processed via ConfDB Queries and validators natively, even if no platform-independent syntax exists. raw section may be enabled on Managed Object Profile level.

ConfDB ‘meta’ Section

NOC 19.3 adds meta section, containing valuable NOC database information directly exposed to ConfDB queries. Lots of additional information included:

  • Vendor

  • Platform

  • Software Version

  • Profile

  • Administrative Domain

  • Network Segment

  • Managed Object Tags

  • Managed Object Profile

  • Interface Profile

  • Interface Network Neighbors (i.e. Links)

ConfDB Normalizers

NOC 19.3 introduces 4 new profiles:

Topology RCA Optimization

Topology-based Root-Cause analysis may be resource consumption. NOC 19.3 introduces new experimental accelerated mode called RCA Neighbor Cache. Smarter data precalculation and caching in combination of database query optimization and bulk updates allows to achieve 2-3 times speedup on real-world installations.

To enable the feature perform following steps:

  • Run fix:

    $ noc fix apply fix_object_uplinks
    
  • Stop correlator processes

  • Enable config-fm-enable_rca_neighbor_cache configuration variable

  • Start correlator processes

Warning

Alarm processing will be postponed when correlator process is stopped, so alarm creation and clearing will be delayed until the correlator process will be started again.

Prometheus Histograms

Prometheus histograms and quantiles may be exported via /metrics endpoint. Additional metrics may be enabled in config. See metrics section for details.

ObjectModel Tags

Inventory models got additional marking, which may be useful in various cases. See ObjectModel Tags for details. Model’s tags are also exposed into managedobject DataStream.

Django upgrade

Previous releases of NOC relied on venerable Django 1.4 dated back to 2012. Django’s team worked hard on improving their product according to their vision. Unfortunately they tend to introduce a lot of incompatibilities and upgrading to each next Django's major release is the real pain. Django 1.4 fits our needs well but is not maintained and is incompatible with Python 3. So it is the time to to collect the pains.

We’d migrated from 1.4 to 1.5, then from 1.5 to 1.6, then followed by upgrades to 1.7, 1.8, 1.9, 1.10 and stopped at 1.11. During our stroll we’d became very disappointed by Django's API stability and the high maintenance costs for the complex applications and applied some countermeasures.

NOC 19.3 brings following changes:

  • Django 1.11.22

  • Django's auth contrib package has been replaced with AAA module.

  • South migrations has been replaced with our own Migration Engine.

  • All legacy Django admin applications (ModelApplication) has been replaced with ExtJS implementations.

  • Django will never create or modify database structure on its own (so-called syncdb).

  • Django static media repacked as django-media<https://code.getnoc.com/npkg/django-media> npkg package.

AAA module

User and Groups use NOC’s own implementation instead of Django's ones. Besides the native ExtJS UI it allows future independent development according our needs. User Profile became the part of User model.

Migration Engine

South database migration engine stopped in development and users are urged to move to Django’s 1.7 built-in migration engine. During our investigations we’d found that we need to completely rewrite 500+ of existing migrations, migrations code will be bloated by the unnecessary abstractions and we need to invite the way to preserve old migration history.

So we’d developed migration engine, simple but powerful. Key benefits are:

  • Small, clean API.

  • Semi-automatical translation of existing migration.

  • Seamless migration history conversion.

  • Skipped migrations with from other development branch, may be applied later.

Development Process Changes

Code Formatting

NOC adopts black - the python code formatter. CI pipeline checks code formatting of changed python files. Any misformatting considered the error and CI pipeline fails at the lint stage. We recommend to add black formatting to git’s pre-commit hook or to the IDE’s on-save hook.

We’d already reformatted all ours codebase and NOC is now fully PEP8-compatible. Docker container is also available. Use:

docker run --rm \
    -w /src \
    -v $PWD:/src \
    registry.getnoc.com/infrastructure/black:master \
    /usr/local/bin/black <file name>

to format file

Towards Python 3 compatibility

Python 3 compatibility became one of our priorities. With 19.3 we’d fixed lots of incompatibilities, upgraded same dependencies and becoming to get rid of unsupported ones. Though a lots of work and testing still required we’re expecting to reach full Python 3 compatibility in one of future releases.

MR Labels

We’re developed the policy for Merge Request’s (MR) labels. CI pipeline checks the labels and fails at the lint stage in case of errors. Label policy helps to organize testing and code reviewing process and quickly explains the goals of MR and subsystems affected.

Breaking Changes

Explicit MongoDB Connections

Prior to 19.3 NOC relied that importing of noc.lib.nosql automatically creates MongoDB connection. This kind of auto-magic used to work but requires to access all mongo-related stuff via noc.lib.nosql. Starting from 19.3 we’re beginning to cleanup API and the code and demand, that MongoDB connection is to be initialized implicitly.

For custom commands and python scripts

from noc.core.mongo.connection import connect

...
connect()

For custom services set service’s use_mongo property to True

Other Changes

  • ManagedObjectSelector.resolve_expression() renamed to ManagedObjectSelector.get_objects_from_expression()

New features

MR

Title

1682

ClickHouse LowCardinality support

2091

New migrations framework

2098

Migration loader, planner and tests

2179

Prometheus histograms

2181

ObjectModels tags field

2190

RCA neighbor cache and accelerated topology correlation

2220

Merge UserProfile into User model

2228

Platform tags

2245

./noc test –idea-bookmarks option

2285

Uplink Policy

2372

Add MySQL Extractor

2400

ConfDB: ntp support

2418

#1077 ConfDB raw policy

2419

Add new Profile Eltex.WOPLR

2420

ConfDB: media section

2426

ConfDB: Object Validation

2433

ConfDB: meta section

2438

ConfDB: Interface validation

Improvements

MR

Title

1888

Django 1.5

1965

Add input_vlan_map and output_vlan_map fields to ConfDB syntax.

2006

noc/noc#1032

2020

Add autowidth column option to ReportLinkDetail.

2021

Add autowidth column to ReportIfacesStatus.

2023

Add autowidth column option to ReportAlarmDetail.

2039

Add frozen first row in Detail Report.

2041

Add subscribers profile filter to AlarmDetail Report.

2048

Add ability to get vlans from bridge on MikroTik.RouterOS

2055

Add ignoring snmp to profile checker.

2065

ensure-indexes: Create index on fm.Uptime

2078

add get_mac

2084

Python3 dk improve

2087

Django 1.6

2088

Add Content-Transfer-Encoding header to mailsender.

2093

noc/noc#1008

2100

Add search form from Maintenance

2102

Allow - in git tags

2105

User, Group ExtJS version

2108

Add QSW-3470-28T-AC platform to Qtech. Switch get_version prefer to SNMP.

2110

Django 1.7.11

2112

noc/noc#914 Return first find profile that loader.

2115

Add initial support for Extreme.Summit200 profile

2117

Update apply-pools

2118

Add depends on set_bi_id migration.

2122

copy tags to clipboard

2125

Cleanup Qtech.QSW2800.get_chassis_id for matcher use.

2132

Update Angtel.Topaz profile

2137

Add ingnore_errors param to http_get activator method.

2138

Add config policy for IBM NOS

2139

Add managed param to clickhouse model meta.

2142

new profile - Polygon

2143

Get serial number for Cisco ASR1000

2144

Add noc user to docker container

2150

Remove index field from clickhouse model.

2152

HP Comware: platform matching, added getting serial number

2155

Check column type when execute ch-migrate.

2157

Added check of empty lines in the Object and Segment fields

2158

Add get_inventory support for Eltex.MES profile

2161

Move report metric to Report Detail format.

2162

Docker with memcache

2168

Set noc dir permissions

2178

added OID for HP A3600-48-PoE

2180

Merge noc-docs to main repo

2186

aaa module

2187

mongoengine 0.18

2188

export filename as template appId_YYYYMMDDHHMMSS.csv

2189

cachetools 3.1.1

2192

tagfield add trigger copy to clipboard

2193

define environmets

2194

new/cm-notify

2195

managedobject layout fix

2196

ref book admin ExtJS version

2197

docs: API autodocumentation

2198

redirect after success login

2204

Add administrative domain field to Report Latest Changes.

2206

Add new Radio Metrics Row for rssi/cinr and rx/tx power metrics

2207

Migrate to ComboTree

2213

Add image options to script command.

2214

Add config-violatile to Raisecom.ROS.

2217

Add remote:deleted tag when managedobject removed from etl.

2218

Update Ericsson SEOS Profile

2219

Django 1.8

2222

Check MR labels

2223

documentation last releases description added

2224

Mongo test hc

2225

Reorganized vendor and profile documentation

2226

Add Eltex.MES5448.get_inventory script

2227

Add Eltex.DSLAM.get_inventory script

2229

Add Eltex.LTP.get_inventory script

2233

Add support for unpriveleged prompt for Eltex.MES5448

2235

Django 1.9

2240

Update Eltex.ESR profile

2244

Add Eltex.LTE.get_inventory script

2246

Service.use_mongo options to auto-connect to mongo database

2249

Add Eltex.MA4000.get_inventory script

2250

documentation historical releases description added

2252

Build docs when merging to master

2253

Replace yapf with black

2254

Check changed fields when calculate datastream.

2256

Docs config refactor

2257

Django 1.10

2258

Move tests to tmpfs

2260

Add mirror options to gridvcs command.

2261

Explicit MongoDB database connection

2262

Django 1.11

2272

Lower artifacts time

2273

flake8: Disable E203 check

2280

bandit analyzer check

2282

Move custom mongoengine fields from noc.lib.nosql to noc.core.mongo.fields

2284

caniusepython3 and pylint3k checks

2287

flake8: black-friendly settings

2288

Set discovery fatal error of profile do not detect profile on Generic devices.

2290

Fix Eltex.MES5448.get_config script

2293

Fix title and additional column to LinkDetailReport.

2297

Django 1.11.22

2309

docs: Tools documentation

2322

black: Ignore deleted files

2323

SA CLI/SNMP tests

2327

Add Address column to Latest Changes report.

2336

Fix column name in ReportLinkDetail.

2349

Update DLink.DxS_Smart.__init__.py add DES-1210-52 v4

2351

Update Huawei.VRP.get_version.py add CE platform.

2355

Add DLink.DxS_Smart.get_capabilities for detecting of enabled LLDP protocol

2365

Add hw_version and bootpromversion to ManagedObject BI models.

2376

Add confdb support to Hikvision.DSKV8 profile.

2381

Add confdb support to Beward.BD profile.

2395

Add Eltex.WOP profile.

2404

docs: ConfDB query language

2405

Fix doc typo

2412

MR: Check confdb label

2413

Update NAG.SNR.get_arp.py

2416

ConfDB: Fix Query doc

2421

noc confdb syntax path parameter

2423

docs: GA integration

2429

Add confdb normalizer to Dahua.DH profile.

2434

check status http query

2436

CI: Disable test html report

2440

install-packages: -v flag

2441

Add connect() to some commands.

2442

Extract collections to build docs

2449

Update EdgeCore.ES profile

2450

Add ConfDB normalizer to Cisco.IOS profile.

2455

config: web.max_upload_size configuration parameter

2459

login: register_last_login option

2460

Update RU translation for Web services.

2461

./noc confdb: tokenizer and normalizer helpers

2472

Add confdb to managedobject card backend.

2475

Use django-media package

Bugfixes

MR

Title

1847

Fix DLink.DVG.get_chassis_id script

1952

Fix SKS.SKS.get_interfaces script

2007

Fix SKS.SKS.get_spanning_tree script

2008

Fix Alstec.24xx.get_interfaces script

2009

Fix detect Catalyst 4k platform

2010

Fix path for release Dockerfiles

2017

Fix Huawei.VRF.get_interfaces untagged from pvid.

2037

Fix Generic.get_capabilities script when SNMP false.

2052

Fix get_config scripts.

2056

Fix TFortis.PSW get_interfaces.

2058

Fix bulk update IPAM address usage cache.

2066

ConfDB: NotMatch doesn’t yield context if unresolved unbound variables left

2068

Fix get_displayed_type method for clickhouse field.

2090

Fix RouterOS parser

2097

fix not ascii in description

2104

Fix fix-pip

2116

fix_metric_qtech_vendors

2119

fix get_version script for old H3C devices

2127

Fix upstream_connected_graph_template.

2128

Fix Eltex.MES profile

2140

Fix export inv.objectmodel to JSON

2151

Fix Huawei.MA5600T pattern more.

2159

Fix activator http_get params typo.

2167

Fix add user in docker release image

2170

Fix managed_object logger.

2173

Fix ipv4 address validator

2175

Fix inetrace speed attribute on ch dictionary

2177

fixed H3C get_version for old devices like H3C S3100-8T-SI

2183

Fix datasource interfaceattributes dictionary.

2191

Fix Cisco.IOSXR.convert_interface_name

2199

Fix caches

2203

Fix telnet SB .. SE parsing

2205

Fix command_disable_pager for NAG.SNR.

2208

Fix reportmetrics field order.

2211

Fix trace when convert UUID field on extdocapplication.

2216

Fix LRUcache missing on etl chain

2230

Fix typo in extdocapplication backend.

2237

Fix trace when check type for new column

2247

Profile: Compile syntax/operation error patterns as multiline

2255

Fix do_pending_operations in selectorcache.

2259

noc/noc#1047 Delete unused code from prefix delete_recursive.

2267

Fix Huawei.MA5300.get_interfaces script

2268

Fix Tag search query.

2269

Replace XML namespace parameter on profile Hikvision.DSKV8.

2270

Skip send_on_syntax_error when beef cli_error.

2274

Fix prompt in Eltex.DSLAM profile

2275

Fix backport label check

2276

Fix Qtech.QOS.get_version script

2277

Fix MikroTik.RouterOS.get_fqdn script

2278

Fix MikroTik.RouterOS.get_cdp_neighbors script

2279

Add operation_error in Cisco.IOS profile

2286

Fix DCN.DCWL.get_interfaces profile for WL8200-TL1

2298

Update Eltex.DSLAM profile

2302

Fix Beward.BD more than 1 value trace.

2305

Fix Caps model, Update caps when sync collections

2308

Add yandex apikey configuration

2317

Fix software_image option on script command.

2326

Fix DCN.DCWL.get_interface_status_ex for WL8200.

2340

Fix regex for duplicates packets in DCN.DCWL.ping script.

2341

Fix refresh cfgping datastream when timepattern change

2343

Fix DialPlan and NetworkSegment links in Project Card

2347

Update NAG.SNR.get_interfaces.py add QSFP+

2348

Update Huawei.VRP.get_portchannel.py add dynamic

2350

Update DLink.DxS_Smart.get_portchannel.py - fix “type” output

2353

simple report fixed

2356

Update DLink.DxS_Smart.get_lldp_neighbors.py

2357

reportmetrics fixed

2360

Fix docker push command

2362

Fix container column on ReportObjectDetail.

2364

Return User.get_full_name() method, uses in ActiveAlarm backend.

2366

Fix ReportLinkDetail when empty tags.

2367

Add convert_interface_name to Iskratel.ESCOM.

2373

fix_version_regex_eltex_mes

2375

fix_get_config_MXA24

2377

Fix Alcatel.TIMOS profile. Trace when iface has empty MAC.

2378

Fix NSN.TIMOS.get_lldp_neighbors. Fix Multiline Remote PortsID and RemotePortDescription.

2379

fix_get_config_Eltex_RG

2380

Add empty values to Reports choices fields.

2386

Add display omit to Juniper show configuration command

2387

Fix Eltex.MES.get_version script on non stack devices

2388

Fix deleted missing parameter on LRUCache.

2389

Fix model cannot be resolved on managedobjectselector field.

2390

Add set_unusable_password method to User model.

2393

noc/noc#1042

2396

#1068 Fix SQL and QTags broken by django upgrade

2397

#1064 Migrate Handlers

2398

#1066 Refactored CachedForeignKeyField

2399

Fix Qtech.QSW8200.get_version regex.

2402

Fix DCN.DCWL.get_interface_status_ex trace if not bss return.

2403

Fix Alcatel.AOS.get_switchport untagged vlan list to int.

2407

Fix AlarmEscalation wait_tt processing

2414

ConfDB: Fix syntax glitches

2427

Fix ./noc newapp

2432

filter horizontal scroll fixed

2443

Move _archive_db attribute from archive bi extractor to method.

2447

Add configvalidation field on report_discovery result.

2448

Fix Stream Audio config section on Hikvision.DSKV8.

2451

ConfDB: Fix bound variables handling in NotMatch

2453

users remove is_staff from model

2457

Fix User’s preferred language

2473

Fix typo in Beward.BD normalizer.

2474

Fix Huawei.VRP.get_version on S5600-HI platform

Code Cleanup

MR

Title

1885

Migrate KB to ExtModelApplication

1977

Model __str__ methods

2029

Remove copy/paste mistake from Eltex.MES.get_config script

2034

noc-model-str-cm-facts2

2035

noc-model-str-core-ip

2036

noc-model-str-services-lib

2049

Test model’s __str__

2050

py3 dict .iteritems(), .itervalues(), .iterkeys()

2051

2to3 except fix

2054

2to3 exec fix

2060

Remove urllib usage in profiles

2063

test_base_parametrize. ver1

2064

Wrap urllib/urllib2 usage

2067

wrap StringIO

2074

2to3 import fix

2075

2to3 has_key fix

2076

2to3 fix urlparse

2080

wrap maketrans call

2081

2to3 print fix

2082

2to3 fix map and filter

2083

2to3 imports fix

2085

2to3 itertools and itertools_import fixes

2086

2to3 long fix

2089

2to3 types fix

2094

test_ecma48

2095

test_ber

2101

Remove south usage from BaseMigration class

2106

Migration Runner, Bye-bye South

2107

Cleanup models’ __init__.py

2120

fix_DeprecationWarning_main_0049_update_tags

2121

fix_test_ip

2123

test_crypto

2124

test_mac

2133

remove_total_seconds

2135

test_matcher

2136

test_validators

2221

docs format mr tables

2236

Fix docs path

2242

Fix mongoengine imports

2243

Fix mongoengine imports

2251

Clean Huawei.MA5600T profile

2263

2to3: zip fix

2264

Remove unused cm templates

2265

2to3: next fix

2266

Fix ObjectId import

2271

2to3: dict fix

2283

2to3: xrange fix

2291

black: aaa, bi, cm

2306

black: crm, dev, dns, fixes, fm

2307

black: gis, inv, ip, kb

2310

fix_import_lib_nosql_part1

2311

fix_import_lib_nosql_part2

2314

black: migrations

2315

fix_import_lib_nosql_part3

2316

fix_import_lib_nosql_part4

2318

black: sa

2319

black: lib

2320

black: core

2321

black: Rest of stuff

2330

Bump version

2354

Add mongo connect to commands.

2358

Speedup docker release images build with targeted builds

2406

Drop unused fields

2411

ConfDB syntax refactoring

2422

Move profiles to profile.py

2424

Fix get_version Infinet.Wanflex

2428

Clean up noc.lib.nosql imports

2439

Fix documentation glitches

2452

Update Dynamic Dashboards.