NOC 19.3

In accordance to our Release Policy we’re proudly present release 19.3.

19.3 release contains 318 bugfixes, optimisations and improvements.

Highlights

ConfDB Object and Interface Validation Policies

NOC 19.3 brings ability to validate object and interface settings using ConfDB Queries. This feature will became primary way of validation and will replace old CLIPS-based implementation in NOC 19.5.

It is good time to begin legacy validation policy migrations.

ConfDB Syntax Expansion

ConfDB got additional syntax for:

  • NTP support
  • Video, Audio settings and media streaming

ConfDB ‘raw’ Section

Raw tokenized input can be applied to ConfDB under the raw node. So platform-depended syntax can be processed via ConfDB Queries and validators natively, even if no platform-independent syntax exists. raw section may be enabled on Managed Object Profile level.

ConfDB ‘meta’ Section

NOC 19.3 adds meta section, containing valuable NOC database information directly exposed to ConfDB queries. Lots of additional information included:

  • Vendor
  • Platform
  • Software Version
  • Profile
  • Administrative Domain
  • Network Segment
  • Managed Object Tags
  • Managed Object Profile
  • Interface Profile
  • Interface Network Neighbors (i.e. Links)

ConfDB Normalizers

NOC 19.3 introduces 4 new profiles:

Topology RCA Optimization

Topology-based Root-Cause analysis may be resource consumption. NOC 19.3 introduces new experimental accelerated mode called RCA Neighbor Cache. Smarter data precalculation and caching in combination of database query optimization and bulk updates allows to achieve 2-3 times speedup on real-world installations.

To enable the feature perform following steps:

Warning

Alarm processing will be postponed when correlator process is stopped, so alarm creation and clearing will be delayed until the correlator process will be started again.

Prometheus Histograms

Prometheus histograms and quantiles may be exported via /metrics endpoint. Additional metrics may be enabled in config. See metrics section for details.

ObjectModel Tags

Inventory models got additional marking, which may be useful in various cases. See ObjectModel Tags for details. Model’s tags are also exposed into managedobject DataStream.

Django upgrade

Previous releases of NOC relied on venerable Django 1.4 dated back to 2012. Django’s team worked hard on improving their product according to their vision. Unfortunately they tend to introduce a lot of incompatibilities and upgrading to each next Django’s major release is the real pain. Django 1.4 fits our needs well but is not maintained and is incompatible with Python 3. So it is the time to to collect the pains.

We’d migrated from 1.4 to 1.5, then from 1.5 to 1.6, then followed by upgrades to 1.7, 1.8, 1.9, 1.10 and stopped at 1.11. During our stroll we’d became very disappointed by Django’s API stability and the high maintenance costs for the complex applications and applied some countermeasures.

NOC 19.3 brings following changes:

  • Django 1.11.22
  • Django’s auth contrib package has been replaced with AAA module.
  • South migrations has been replaced with our own Migration Engine.
  • All legacy Django admin applications (ModelApplication) has been replaced with ExtJS implementations.
  • Django will never create or modify database structure on its own (so-called syncdb).
  • Django static media repacked as django-media<https://code.getnoc.com/npkg/django-media> npkg package.

AAA module

User and Groups use NOC’s own implementation instead of Django’s ones. Besides the native ExtJS UI it allows future independent development according our needs. User Profile became the part of User model.

Migration Engine

South database migration engine stopped in development and users are urged to move to Django’s 1.7 built-in migration engine. During our investigations we’d found that we need to completely rewrite 500+ of existing migrations, migrations code will be bloated by the unnecessary abstractions and we need to invite the way to preserve old migration history.

So we’d developed migration engine, simple but powerful. Key benefits are:

  • Small, clean API.
  • Semi-automatical translation of existing migration.
  • Seamless migration history conversion.
  • Skipped migrations with from other development branch, may be applied later.

Development Process Changes

Code Formatting

NOC adopts black - the python code formatter. CI pipeline checks code formatting of changed python files. Any misformatting considered the error and CI pipeline fails at the lint stage. We recommend to add black formatting to git’s pre-commit hook or to the IDE’s on-save hook.

We’d already reformatted all ours codebase and NOC is now fully PEP8-compatible. Docker container is also available. Use:

docker run --rm \
    -w /src \
    -v $PWD:/src \
    registry.getnoc.com/infrastructure/black:master \
    /usr/local/bin/black <file name>

to format file

Towards Python 3 compatibility

Python 3 compatibility became one of our priorities. With 19.3 we’d fixed lots of incompatibilities, upgraded same dependencies and becoming to get rid of unsupported ones. Though a lots of work and testing still required we’re expecting to reach full Python 3 compatibility in one of future releases.

MR Labels

We’re developed the policy for Merge Request’s (MR) labels. CI pipeline checks the labels and fails at the lint stage in case of errors. Label policy helps to organize testing and code reviewing process and quickly explains the goals of MR and subsystems affected.

Breaking Changes

Explicit MongoDB Connections

Prior to 19.3 NOC relied that importing of noc.lib.nosql automatically creates MongoDB connection. This kind of auto-magic used to work but requires to access all mongo-related stuff via noc.lib.nosql. Starting from 19.3 we’re beginning to cleanup API and the code and demand, that MongoDB connection is to be initialized implicitly.

For custom commands and python scripts

from noc.core.mongo.connection import connect

...
connect()

For custom services set service’s use_mongo property to True

Other Changes

  • ManagedObjectSelector.resolve_expression() renamed to ManagedObjectSelector.get_objects_from_expression()

New features

MR Title
1682 ClickHouse LowCardinality support
2091 New migrations framework
2098 Migration loader, planner and tests
2179 Prometheus histograms
2181 ObjectModels tags field
2190 RCA neighbor cache and accelerated topology correlation
2220 Merge UserProfile into User model
2228 Platform tags
2245 ./noc test –idea-bookmarks option
2285 Uplink Policy
2372 Add MySQL Extractor
2400 ConfDB: ntp support
2418 #1077 ConfDB raw policy
2419 Add new Profile Eltex.WOPLR
2420 ConfDB: media section
2426 ConfDB: Object Validation
2433 ConfDB: meta section
2438 ConfDB: Interface validation

Improvements

MR Title
1888 Django 1.5
1965 Add input_vlan_map and output_vlan_map fields to ConfDB syntax.
2006 noc/noc#1032
2020 Add autowidth column option to ReportLinkDetail.
2021 Add autowidth column to ReportIfacesStatus.
2023 Add autowidth column option to ReportAlarmDetail.
2039 Add frozen first row in Detail Report.
2041 Add subscribers profile filter to AlarmDetail Report.
2048 Add ability to get vlans from bridge on MikroTik.RouterOS
2055 Add ignoring snmp to profile checker.
2065 ensure-indexes: Create index on fm.Uptime
2078 add get_mac
2084 Python3 dk improve
2087 Django 1.6
2088 Add Content-Transfer-Encoding header to mailsender.
2093 noc/noc#1008
2100 Add search form from Maintenance
2102 Allow - in git tags
2105 User, Group ExtJS version
2108 Add QSW-3470-28T-AC platform to Qtech. Switch get_version prefer to SNMP.
2110 Django 1.7.11
2112 noc/noc#914 Return first find profile that loader.
2115 Add initial support for Extreme.Summit200 profile
2117 Update apply-pools
2118 Add depends on set_bi_id migration.
2122 copy tags to clipboard
2125 Cleanup Qtech.QSW2800.get_chassis_id for matcher use.
2132 Update Angtel.Topaz profile
2137 Add ingnore_errors param to http_get activator method.
2138 Add config policy for IBM NOS
2139 Add managed param to clickhouse model meta.
2142 new profile - Polygon
2143 Get serial number for Cisco ASR1000
2144 Add noc user to docker container
2150 Remove index field from clickhouse model.
2152 HP Comware: platform matching, added getting serial number
2155 Check column type when execute ch-migrate.
2157 Added check of empty lines in the Object and Segment fields
2158 Add get_inventory support for Eltex.MES profile
2161 Move report metric to Report Detail format.
2162 Docker with memcache
2168 Set noc dir permissions
2178 added OID for HP A3600-48-PoE
2180 Merge noc-docs to main repo
2186 aaa module
2187 mongoengine 0.18
2188 export filename as template appId_YYYYMMDDHHMMSS.csv
2189 cachetools 3.1.1
2192 tagfield add trigger copy to clipboard
2193 define environmets
2194 new/cm-notify
2195 managedobject layout fix
2196 ref book admin ExtJS version
2197 docs: API autodocumentation
2198 redirect after success login
2204 Add administrative domain field to Report Latest Changes.
2206 Add new Radio Metrics Row for rssi/cinr and rx/tx power metrics
2207 Migrate to ComboTree
2213 Add image options to script command.
2214 Add config-violatile to Raisecom.ROS.
2217 Add remote:deleted tag when managedobject removed from etl.
2218 Update Ericsson SEOS Profile
2219 Django 1.8
2222 Check MR labels
2223 documentation last releases description added
2224 Mongo test hc
2225 Reorganized vendor and profile documentation
2226 Add Eltex.MES5448.get_inventory script
2227 Add Eltex.DSLAM.get_inventory script
2229 Add Eltex.LTP.get_inventory script
2233 Add support for unpriveleged prompt for Eltex.MES5448
2235 Django 1.9
2240 Update Eltex.ESR profile
2244 Add Eltex.LTE.get_inventory script
2246 Service.use_mongo options to auto-connect to mongo database
2249 Add Eltex.MA4000.get_inventory script
2250 documentation historical releases description added
2252 Build docs when merging to master
2253 Replace yapf with black
2254 Check changed fields when calculate datastream.
2256 Docs config refactor
2257 Django 1.10
2258 Move tests to tmpfs
2260 Add mirror options to gridvcs command.
2261 Explicit MongoDB database connection
2262 Django 1.11
2272 Lower artifacts time
2273 flake8: Disable E203 check
2280 bandit analyzer check
2282 Move custom mongoengine fields from noc.lib.nosql to noc.core.mongo.fields
2284 caniusepython3 and pylint3k checks
2287 flake8: black-friendly settings
2288 Set discovery fatal error of profile do not detect profile on Generic devices.
2290 Fix Eltex.MES5448.get_config script
2293 Fix title and additional column to LinkDetailReport.
2297 Django 1.11.22
2309 docs: Tools documentation
2322 black: Ignore deleted files
2323 SA CLI/SNMP tests
2327 Add Address column to Latest Changes report.
2336 Fix column name in ReportLinkDetail.
2349 Update DLink.DxS_Smart.__init__.py add DES-1210-52 v4
2351 Update Huawei.VRP.get_version.py add CE platform.
2355 Add DLink.DxS_Smart.get_capabilities for detecting of enabled LLDP protocol
2365 Add hw_version and bootpromversion to ManagedObject BI models.
2376 Add confdb support to Hikvision.DSKV8 profile.
2381 Add confdb support to Beward.BD profile.
2395 Add Eltex.WOP profile.
2404 docs: ConfDB query language
2405 Fix doc typo
2412 MR: Check confdb label
2413 Update NAG.SNR.get_arp.py
2416 ConfDB: Fix Query doc
2421 noc confdb syntax path parameter
2423 docs: GA integration
2429 Add confdb normalizer to Dahua.DH profile.
2434 check status http query
2436 CI: Disable test html report
2440 install-packages: -v flag
2441 Add connect() to some commands.
2442 Extract collections to build docs
2449 Update EdgeCore.ES profile
2450 Add ConfDB normalizer to Cisco.IOS profile.
2455 config: web.max_upload_size configuration parameter
2459 login: register_last_login option
2460 Update RU translation for Web services.
2461 ./noc confdb: tokenizer and normalizer helpers
2472 Add confdb to managedobject card backend.
2475 Use django-media package

Bugfixes

MR Title
1847 Fix DLink.DVG.get_chassis_id script
1952 Fix SKS.SKS.get_interfaces script
2007 Fix SKS.SKS.get_spanning_tree script
2008 Fix Alstec.24xx.get_interfaces script
2009 Fix detect Catalyst 4k platform
2010 Fix path for release Dockerfiles
2017 Fix Huawei.VRF.get_interfaces untagged from pvid.
2037 Fix Generic.get_capabilities script when SNMP false.
2052 Fix get_config scripts.
2056 Fix TFortis.PSW get_interfaces.
2058 Fix bulk update IPAM address usage cache.
2066 ConfDB: NotMatch doesn’t yield context if unresolved unbound variables left
2068 Fix get_displayed_type method for clickhouse field.
2090 Fix RouterOS parser
2097 fix not ascii in description
2104 Fix fix-pip
2116 fix_metric_qtech_vendors
2119 fix get_version script for old H3C devices
2127 Fix upstream_connected_graph_template.
2128 Fix Eltex.MES profile
2140 Fix export inv.objectmodel to JSON
2151 Fix Huawei.MA5600T pattern more.
2159 Fix activator http_get params typo.
2167 Fix add user in docker release image
2170 Fix managed_object logger.
2173 Fix ipv4 address validator
2175 Fix inetrace speed attribute on ch dictionary
2177 fixed H3C get_version for old devices like H3C S3100-8T-SI
2183 Fix datasource interfaceattributes dictionary.
2191 Fix Cisco.IOSXR.convert_interface_name
2199 Fix caches
2203 Fix telnet SB .. SE parsing
2205 Fix command_disable_pager for NAG.SNR.
2208 Fix reportmetrics field order.
2211 Fix trace when convert UUID field on extdocapplication.
2216 Fix LRUcache missing on etl chain
2230 Fix typo in extdocapplication backend.
2237 Fix trace when check type for new column
2247 Profile: Compile syntax/operation error patterns as multiline
2255 Fix do_pending_operations in selectorcache.
2259 noc/noc#1047 Delete unused code from prefix delete_recursive.
2267 Fix Huawei.MA5300.get_interfaces script
2268 Fix Tag search query.
2269 Replace XML namespace parameter on profile Hikvision.DSKV8.
2270 Skip send_on_syntax_error when beef cli_error.
2274 Fix prompt in Eltex.DSLAM profile
2275 Fix backport label check
2276 Fix Qtech.QOS.get_version script
2277 Fix MikroTik.RouterOS.get_fqdn script
2278 Fix MikroTik.RouterOS.get_cdp_neighbors script
2279 Add operation_error in Cisco.IOS profile
2286 Fix DCN.DCWL.get_interfaces profile for WL8200-TL1
2298 Update Eltex.DSLAM profile
2302 Fix Beward.BD more than 1 value trace.
2305 Fix Caps model, Update caps when sync collections
2308 Add yandex apikey configuration
2317 Fix software_image option on script command.
2326 Fix DCN.DCWL.get_interface_status_ex for WL8200.
2340 Fix regex for duplicates packets in DCN.DCWL.ping script.
2341 Fix refresh cfgping datastream when timepattern change
2343 Fix DialPlan and NetworkSegment links in Project Card
2347 Update NAG.SNR.get_interfaces.py add QSFP+
2348 Update Huawei.VRP.get_portchannel.py add dynamic
2350 Update DLink.DxS_Smart.get_portchannel.py - fix “type” output
2353 simple report fixed
2356 Update DLink.DxS_Smart.get_lldp_neighbors.py
2357 reportmetrics fixed
2360 Fix docker push command
2362 Fix container column on ReportObjectDetail.
2364 Return User.get_full_name() method, uses in ActiveAlarm backend.
2366 Fix ReportLinkDetail when empty tags.
2367 Add convert_interface_name to Iskratel.ESCOM.
2373 fix_version_regex_eltex_mes
2375 fix_get_config_MXA24
2377 Fix Alcatel.TIMOS profile. Trace when iface has empty MAC.
2378 Fix NSN.TIMOS.get_lldp_neighbors. Fix Multiline Remote PortsID and RemotePortDescription.
2379 fix_get_config_Eltex_RG
2380 Add empty values to Reports choices fields.
2386 Add display omit to Juniper show configuration command
2387 Fix Eltex.MES.get_version script on non stack devices
2388 Fix deleted missing parameter on LRUCache.
2389 Fix model cannot be resolved on managedobjectselector field.
2390 Add set_unusable_password method to User model.
2393 noc/noc#1042
2396 #1068 Fix SQL and QTags broken by django upgrade
2397 #1064 Migrate Handlers
2398 #1066 Refactored CachedForeignKeyField
2399 Fix Qtech.QSW8200.get_version regex.
2402 Fix DCN.DCWL.get_interface_status_ex trace if not bss return.
2403 Fix Alcatel.AOS.get_switchport untagged vlan list to int.
2407 Fix AlarmEscalation wait_tt processing
2414 ConfDB: Fix syntax glitches
2427 Fix ./noc newapp
2432 filter horizontal scroll fixed
2443 Move _archive_db attribute from archive bi extractor to method.
2447 Add configvalidation field on report_discovery result.
2448 Fix Stream Audio config section on Hikvision.DSKV8.
2451 ConfDB: Fix bound variables handling in NotMatch
2453 users remove is_staff from model
2457 Fix User’s preferred language
2473 Fix typo in Beward.BD normalizer.
2474 Fix Huawei.VRP.get_version on S5600-HI platform

Code Cleanup

MR Title
1885 Migrate KB to ExtModelApplication
1977 Model __str__ methods
2029 Remove copy/paste mistake from Eltex.MES.get_config script
2034 noc-model-str-cm-facts2
2035 noc-model-str-core-ip
2036 noc-model-str-services-lib
2049 Test model’s __str__
2050 py3 dict .iteritems(), .itervalues(), .iterkeys()
2051 2to3 except fix
2054 2to3 exec fix
2060 Remove urllib usage in profiles
2063 test_base_parametrize. ver1
2064 Wrap urllib/urllib2 usage
2067 wrap StringIO
2074 2to3 import fix
2075 2to3 has_key fix
2076 2to3 fix urlparse
2080 wrap maketrans call
2081 2to3 print fix
2082 2to3 fix map and filter
2083 2to3 imports fix
2085 2to3 itertools and itertools_import fixes
2086 2to3 long fix
2089 2to3 types fix
2094 test_ecma48
2095 test_ber
2101 Remove south usage from BaseMigration class
2106 Migration Runner, Bye-bye South
2107 Cleanup models’ __init__.py
2120 fix_DeprecationWarning_main_0049_update_tags
2121 fix_test_ip
2123 test_crypto
2124 test_mac
2133 remove_total_seconds
2135 test_matcher
2136 test_validators
2221 docs format mr tables
2236 Fix docs path
2242 Fix mongoengine imports
2243 Fix mongoengine imports
2251 Clean Huawei.MA5600T profile
2263 2to3: zip fix
2264 Remove unused cm templates
2265 2to3: next fix
2266 Fix ObjectId import
2271 2to3: dict fix
2283 2to3: xrange fix
2291 black: aaa, bi, cm
2306 black: crm, dev, dns, fixes, fm
2307 black: gis, inv, ip, kb
2310 fix_import_lib_nosql_part1
2311 fix_import_lib_nosql_part2
2314 black: migrations
2315 fix_import_lib_nosql_part3
2316 fix_import_lib_nosql_part4
2318 black: sa
2319 black: lib
2320 black: core
2321 black: Rest of stuff
2330 Bump version
2354 Add mongo connect to commands.
2358 Speedup docker release images build with targeted builds
2406 Drop unused fields
2411 ConfDB syntax refactoring
2422 Move profiles to profile.py
2424 Fix get_version Infinet.Wanflex
2428 Clean up noc.lib.nosql imports
2439 Fix documentation glitches
2452 Update Dynamic Dashboards.