Skip to content

NOC 19.3

In accordance to our Release Policy <releases-policy> we're proudly present release 19.3.

19.3 release contains 318 bugfixes, optimisations and improvements.

Highlights

New Profiles

  • Extreme.Summit200<profile-Extreme.Summit200>
  • Polygon.IOS<profile-Polygon.IOS>
  • Eltex.WOP<profile-Eltex.WOP>

ConfDB Object and Interface Validation Policies

NOC 19.3 brings ability to validate object and interface settings using ConfDB Queries<dev-confdb-query>. This feature will became primary way of validation and will replace old CLIPS-based implementation in NOC 19.5.

It is good time to begin legacy validation policy migrations.

ConfDB Syntax Expansion

ConfDB got additional syntax for:

  • NTP support
  • Video, Audio settings and media streaming

ConfDB 'raw' Section

Raw tokenized input can be applied to ConfDB under the raw node. So platform-depended syntax can be processed via ConfDB Queries and validators natively, even if no platform-independentent syntax exists. raw section may be enabled on Managed Object Profile level.

ConfDB 'meta' Section

NOC 19.3 adds meta section, containing valuable NOC database information directly exposed to ConfDB queries. Lots of additional information included:

  • Vendor
  • Platform
  • Software Version
  • Profile
  • Administrative Domain
  • Network Segment
  • Managed Object Tags
  • Managed Object Profile
  • Interface Profile
  • Interface Network Neighbors (i.e. Links)

ConfDB Normalizers

NOC 19.3 introduces 4 new profiles:

  • Beward.BD<profile-Beward.BD>
  • Cisco.IOS<profile-Cisco.IOS>
  • Dahua.DH<profile-Dahua.DH>
  • Hikvision.DSKV8<profile-Hikvision.DSKV8>

Uplink detection algorithm for RCA became configurable now. Following policies may be used:

  • Segment Hierarchy (previous algorithm)
  • Object Level
  • All Segment Objects
  • Lesser Management Address
  • Greater Management Address

Uplink policies may be configured at Network Segment Profile level. Multiple policies may be used with falling back to next policy until uplinks are detected.

Uplink policies are greatly improve the quality of topology-based RCA. See Uplink Policy<reference-network-segment-profile-uplink-policy> for details.

Topology RCA Optimization

Topology-based Root-Cause analysis may be resource consumption. NOC 19.3 introduces new experimental accelerated mode called RCA Neighbor Cache. Smarter data precalculation and caching in combination of database query optimization and bulk updates allows to achieve 2-3 times speedup on real-world installations.

To enable the feature perform following steps:

  • Run fix:

    $ noc fix apply fix_object_uplinks

  • Stop services-correlator processes

  • Enable config-fm-enable_rca_neighbor_cache configuration variable

  • Start services-correlator processes

Warning
Alarm processing will be postponed when correlator process is stopped, so alarm creation and clearing will be delayed until the correlator process will be started again.

Prometheus Histograms

Prometheus histograms and quantiles may be exported via /metrics endpoint. Additional metrics may be enabled in config. See metrics section<config-metrics> for details.

ObjectModel Tags

Inventory models got additional marking, which may be useful in various cases. See ObjectModel Tags<dev-objectmodel-tags> for details. Model's tags are also exposed into managedobject DataStream<api-datastream-managedobject>.

Django upgrade

Previous releases of NOC relied on venerable Django 1.4 dated back to

  1. Django's team worked hard on improving their product according to their vision. Unfortunately they tend to introduce a lot of incompatibilities and upgrading to each next Django's major release is the real pain. Django 1.4 fits our needs well but is not maintained and is incompatible with Python 3. So it is the time to to collect the pains.

We'd migrated from 1.4 to 1.5, then from 1.5 to 1.6, then followed by upgrades to 1.7, 1.8, 1.9, 1.10 and stopped at 1.11. During our stroll we'd became very disappointed by Django's API stability and the high maintenance costs for the complex applications and applied some countermeasures.

NOC 19.3 brings following changes:

  • Django 1.11.22
  • Django's auth contrib package has been replaced with AAA module<release-19.3-aaa>.
  • South migrations has been replaced with our own Migration Engine<release-19.3-migrations>.
  • All legacy Django admin applications (ModelApplication) has been replaced with ExtJS implementations.
  • Django will never create or modify database structure on its own (so-called syncdb).
  • Django static media repacked as django-media&lt;https://code.getnoc.com/npkg/django-media&gt; npkg package.

AAA module

User and Groups use NOC's own implementation instead of Django's ones. Besides the native ExtJS UI it allows future independentent development according our needs. User Profile became the part of User model.

Migration Engine

South database migration engine stopped in development and users are urged to move to Django's 1.7 built-in migration engine. During our investigations we'd found that we need to completely rewrite 500+ of existing migrations, migrations code will be bloated by the unnecessary abstractions and we need to invite the way to preserve old migration history.

So we'd developed migration engine, simple but powerful. Key benefits are:

  • Small, clean API.
  • Semi-automatic translation of existing migration.
  • Seamless migration history conversion.
  • Skipped migrations with from other development branch, may be applied later.

Development Process Changes

Code Formatting

NOC adopts black -the python code formatter. CI pipeline checks code formatting of changed python files. Any misformatting considered the error and CI pipeline fails at the lint stage. We recommend to add black formatting to git's pre-commit hook or to the IDE's on-save hook.

We'd already reformatted all ours codebase and NOC is now fully PEP8-compatible. Docker container is also available. Use:

docker run --rm \
    -w /src \
    -v $PWD:/src \
    registry.getnoc.com/infrastructure/black:master \
    /usr/local/bin/black <file name>

to format file

Towards Python 3 compatibility

Python 3 compatibility became one of our priorities. With 19.3 we'd fixed lots of incompatibilities, upgraded same dependencies and becoming to get rid of unsupported ones. Though a lots of work and testing still required we're expecting to reach full Python 3 compatibility in one of future releases.

MR Labels

We're developed the policy<dev-mr-labels> for Merge Request's (MR) labels. CI pipeline checks the labels and fails at the lint stage in case of errors. Label policy helps to organize testing and code reviewing process and quickly explains the goals of MR and subsystems affected.

Breaking Changes

Explicit MongoDB Connections

Prior to 19.3 NOC relied that importing of noc.lib.nosql automatically creates MongoDB connection. This kind of auto-magic used to work but requires to access all mongo-related stuff via noc.lib.nosql. Starting from 19.3 we're beginning to cleanup API and the code and demand, that MongoDB connection is to be initialized implicitly.

For custom commands and python scripts

from noc.core.mongo.connection import connect

...
connect()

For custom services set service's use_mongo property to True

Other Changes

  • ManagedObjectSelector.resolve_expression() renamed to ManagedObjectSelector.get_objects_from_expression()

New features

MR Title
MR1682 ClickHouse LowCardinality support
MR2091 New migrations framework
MR2098 Migration loader, planner and tests
MR2179 Prometheus histograms
MR2181 ObjectModels tags field
MR2190 RCA neighbor cache and accelerated topology correlation
MR2220 Merge UserProfile into User model
MR2228 Platform tags
MR2245 ./noc test --idea-bookmarks option
MR2285 Uplink Policy
MR2372 Add MySQL Extractor
MR2400 ConfDB: ntp support
MR2418 #1077 ConfDB raw policy
MR2419 Add new Profile Eltex.WOPLR
MR2420 ConfDB: media section
MR2426 ConfDB: Object Validation
MR2433 ConfDB: meta section
MR2438 ConfDB: Interface validation

Improvements

MR Title
MR1888 Django 1.5
MR1965 Add input_vlan_map and output_vlan_map fields to ConfDB syntax.
MR2006 noc/noc#1032
MR2020 Add autowidth column option to ReportLinkDetail.
MR2021 Add autowidth column to ReportIfacesStatus.
MR2023 Add autowidth column option to ReportAlarmDetail.
MR2039 Add frozen first row in Detail Report.
MR2041 Add subscribers profile filter to AlarmDetail Report.
MR2048 Add ability to get vlans from bridge on MikroTik.RouterOS
MR2055 Add ignoring snmp to profile checker.
MR2065 ensure-indexes: Create index on fm.Uptime
MR2078 add get_mac
MR2084 Python3 dk improve
MR2087 Django 1.6
MR2088 Add Content-Transfer-Encoding header to mailsender.
MR2093 noc/noc#1008
MR2100 Add search form from Maintenance
MR2102 Allow - in git tags
MR2105 User, Group ExtJS version
MR2108 Add QSW-3470-28T-AC platform to Qtech. Switch get_version prefer to SNMP.
MR2110 Django 1.7.11
MR2112 noc/noc#914 Return first find profile that loader.
MR2115 Add initial support for Extreme.Summit200 profile
MR2117 Update apply-pools
MR2118 Add depends on set_bi_id migration.
MR2122 copy tags to clipboard
MR2125 Cleanup Qtech.QSW2800.get_chassis_id for matcher use.
MR2132 Update Angtel.Topaz profile
MR2137 Add ingnore_errors param to http_get activator method.
MR2138 Add config policy for IBM NOS
MR2139 Add managed param to clickhouse model meta.
MR2142 new profile - Polygon
MR2143 Get serial number for Cisco ASR1000
MR2144 Add noc user to docker container
MR2150 Remove index field from clickhouse model.
MR2152 HP Comware: platform matching, added getting serial number
MR2155 Check column type when execute ch-migrate.
MR2157 Added check of empty lines in the Object and Segment fields
MR2158 Add get_inventory support for Eltex.MES profile
MR2161 Move report metric to Report Detail format.
MR2162 Docker with memcache
MR2168 Set noc dir permissions
MR2178 added OID for HP A3600-48-PoE
MR2180 Merge noc-docs to main repo
MR2186 aaa module
MR2187 mongoengine 0.18
MR2188 export filename as template appId_YYYYMMDDHHMMSS.csv
MR2189 cachetools 3.1.1
MR2192 tagfield add trigger copy to clipboard
MR2193 define environmets
MR2194 new/cm-notify
MR2195 managedobject layout fix
MR2196 ref book admin ExtJS version
MR2197 docs: API autodocumentation
MR2198 redirect after success login
MR2204 Add administrative domain field to Report Latest Changes.
MR2206 Add new Radio Metrics Row for rssi/cinr and rx/tx power metrics
MR2207 Migrate to ComboTree
MR2213 Add image options to script command.
MR2214 Add config-violatile to Raisecom.ROS.
MR2217 Add remote:deleted tag when managedobject removed from etl.
MR2218 Update Ericsson SEOS Profile
MR2219 Django 1.8
MR2222 Check MR labels
MR2223 documentation last releases description added
MR2224 Mongo test hc
MR2225 Reorganized vendor and profile documentation
MR2226 Add Eltex.MES5448.get_inventory script
MR2227 Add Eltex.DSLAM.get_inventory script
MR2229 Add Eltex.LTP.get_inventory script
MR2233 Add support for unpriveleged prompt for Eltex.MES5448
MR2235 Django 1.9
MR2240 Update Eltex.ESR profile
MR2244 Add Eltex.LTE.get_inventory script
MR2246 Service.use_mongo options to auto-connect to mongo database
MR2249 Add Eltex.MA4000.get_inventory script
MR2250 documentation historical releases description added
MR2252 Build docs when merging to master
MR2253 Replace yapf with black
MR2254 Check changed fields when calculate datastream.
MR2256 Docs config refactor
MR2257 Django 1.10
MR2258 Move tests to tmpfs
MR2260 Add mirror options to gridvcs command.
MR2261 Explicit MongoDB database connection
MR2262 Django 1.11
MR2272 Lower artifacts time
MR2273 flake8: Disable E203 check
MR2280 bandit analyzer check
MR2282 Move custom mongoengine fields from noc.lib.nosql to noc.core.mongo.fields
MR2284 caniusepython3 and pylint3k checks
MR2287 flake8: black-friendly settings
MR2288 Set discovery fatal error of profile do not detect profile on Generic devices.
MR2290 Fix Eltex.MES5448.get_config script
MR2293 Fix title and additional column to LinkDetailReport.
MR2297 Django 1.11.22
MR2309 docs: Tools documentation
MR2322 black: Ignore deleted files
MR2323 SA CLI/SNMP tests
MR2327 Add Address column to Latest Changes report.
MR2336 Fix column name in ReportLinkDetail.
MR2349 Update DLink.DxS_Smart.init.py add DES-1210-52 v4
MR2351 Update Huawei.VRP.get_version.py add CE platform.
MR2355 Add DLink.DxS_Smart.get_capabilities for detecting of enabled LLDP protocol
MR2365 Add hw_version and bootpromversion to ManagedObject BI models.
MR2376 Add confdb support to Hikvision.DSKV8 profile.
MR2381 Add confdb support to Beward.BD profile.
MR2395 Add Eltex.WOP profile.
MR2404 docs: ConfDB query language
MR2405 Fix doc typo
MR2412 MR: Check confdb label
MR2413 Update NAG.SNR.get_arp.py
MR2416 ConfDB: Fix Query doc
MR2421 noc confdb syntax path parameter
MR2423 docs: GA integration
MR2429 Add confdb normalizer to Dahua.DH profile.
MR2434 check status http query
MR2436 CI: Disable test html report
MR2440 install-packages: -v flag
MR2441 Add connect() to some commands.
MR2442 Extract collections to build docs
MR2449 Update EdgeCore.ES profile
MR2450 Add ConfDB normalizer to Cisco.IOS profile.
MR2455 config: web.max_upload_size configuration parameter
MR2459 login: register_last_login option
MR2460 Update RU translation for Web services.
MR2461 ./noc confdb: tokenizer and normalizer helpers
MR2472 Add confdb to managedobject card backend.
MR2475 Use django-media package

Bugfixes

MR Title
MR1847 Fix DLink.DVG.get_chassis_id script
MR1952 Fix SKS.SKS.get_interfaces script
MR2007 Fix SKS.SKS.get_spanning_tree script
MR2008 Fix Alstec.24xx.get_interfaces script
MR2009 Fix detect Catalyst 4k platform
MR2010 Fix path for release Dockerfiles
MR2017 Fix Huawei.VRF.get_interfaces untagged from pvid.
MR2037 Fix Generic.get_capabilities script when SNMP false.
MR2052 Fix get_config scripts.
MR2056 Fix TFortis.PSW get_interfaces.
MR2058 Fix bulk update IPAM address usage cache.
MR2066 ConfDB: NotMatch doesn't yield context if unresolved unbound variables left
MR2068 Fix get_displayed_type method for clickhouse field.
MR2090 Fix RouterOS parser
MR2097 fix not ascii in description
MR2104 Fix fix-pip
MR2116 fix_metric_qtech_vendors
MR2119 fix get_version script for old H3C devices
MR2127 Fix upstream_connected_graph_template.
MR2128 Fix Eltex.MES profile
MR2140 Fix export inv.objectmodel to JSON
MR2151 Fix Huawei.MA5600T pattern more.
MR2159 Fix activator http_get params typo.
MR2167 Fix add user in docker release image
MR2170 Fix managed_object logger.
MR2173 Fix ipv4 address validator
MR2175 Fix inetrace speed attribute on ch dictionary
MR2177 fixed H3C get_version for old devices like H3C S3100-8T-SI
MR2183 Fix datasource interfaceattributes dictionary.
MR2191 Fix Cisco.IOSXR.convert_interface_name
MR2199 Fix caches
MR2203 Fix telnet SB .. SE parsing
MR2205 Fix command_disable_pager for NAG.SNR.
MR2208 Fix reportmetrics field order.
MR2211 Fix trace when convert UUID field on extdocapplication.
MR2216 Fix LRUcache missing on etl chain
MR2230 Fix typo in extdocapplication backend.
MR2237 Fix trace when check type for new column
MR2247 Profile: Compile syntax/operation error patterns as multiline
MR2255 Fix do_pending_operations in selectorcache.
MR2259 noc/noc#1047 Delete unused code from prefix delete_recursive.
MR2267 Fix Huawei.MA5300.get_interfaces script
MR2268 Fix Tag search query.
MR2269 Replace XML namespace parameter on profile Hikvision.DSKV8.
MR2270 Skip send_on_syntax_error when beef cli_error.
MR2274 Fix prompt in Eltex.DSLAM profile
MR2275 Fix backport label check
MR2276 Fix Qtech.QOS.get_version script
MR2277 Fix MikroTik.RouterOS.get_fqdn script
MR2278 Fix MikroTik.RouterOS.get_cdp_neighbors script
MR2279 Add operation_error in Cisco.IOS profile
MR2286 Fix DCN.DCWL.get_interfaces profile for WL8200-TL1
MR2298 Update Eltex.DSLAM profile
MR2302 Fix Beward.BD more than 1 value trace.
MR2305 Fix Caps model, Update caps when sync collections
MR2308 Add yandex apikey configuration
MR2317 Fix software_image option on script command.
MR2326 Fix DCN.DCWL.get_interface_status_ex for WL8200.
MR2340 Fix regex for duplicates packets in DCN.DCWL.ping script.
MR2341 Fix refresh cfgping datastream when timepattern change
MR2343 Fix DialPlan and NetworkSegment links in Project Card
MR2347 Update NAG.SNR.get_interfaces.py add QSFP+
MR2348 Update Huawei.VRP.get_portchannel.py add dynamic
MR2350 Update DLink.DxS_Smart.get_portchannel.py - fix "type" output
MR2353 simple report fixed
MR2356 Update DLink.DxS_Smart.get_lldp_neighbors.py
MR2357 reportmetrics fixed
MR2360 Fix docker push command
MR2362 Fix container column on ReportObjectDetail.
MR2364 Return User.get_full_name() method, uses in ActiveAlarm backend.
MR2366 Fix ReportLinkDetail when empty tags.
MR2367 Add convert_interface_name to Iskratel.ESCOM.
MR2373 fix_version_regex_eltex_mes
MR2375 fix_get_config_MXA24
MR2377 Fix Alcatel.TIMOS profile. Trace when iface has empty MAC.
MR2378 Fix NSN.TIMOS.get_lldp_neighbors. Fix Multiline Remote PortsID and RemotePortDescription.
MR2379 fix_get_config_Eltex_RG
MR2380 Add empty values to Reports choices fields.
MR2386 Add display omit to Juniper show configuration command
MR2387 Fix Eltex.MES.get_version script on non stack devices
MR2388 Fix deleted missing parameter on LRUCache.
MR2389 Fix model cannot be resolved on managedobjectselector field.
MR2390 Add set_unusable_password method to User model.
MR2393 noc/noc#1042
MR2396 #1068 Fix SQL and QTags broken by django upgrade
MR2397 #1064 Migrate Handlers
MR2398 #1066 Refactored CachedForeignKeyField
MR2399 Fix Qtech.QSW8200.get_version regex.
MR2402 Fix DCN.DCWL.get_interface_status_ex trace if not bss return.
MR2403 Fix Alcatel.AOS.get_switchport untagged vlan list to int.
MR2407 Fix AlarmEscalation wait_tt processing
MR2414 ConfDB: Fix syntax glitches
MR2427 Fix ./noc newapp
MR2432 filter horizontal scroll fixed
MR2443 Move _archive_db attribute from archive bi extractor to method.
MR2447 Add configvalidation field on report_discovery result.
MR2448 Fix Stream Audio config section on Hikvision.DSKV8.
MR2451 ConfDB: Fix bound variables handling in NotMatch
MR2453 users remove is_staff from model
MR2457 Fix User's preferred language
MR2473 Fix typo in Beward.BD normalizer.
MR2474 Fix Huawei.VRP.get_version on S5600-HI platform

Code Cleanup

MR Title
MR1885 Migrate KB to ExtModelApplication
MR1977 Model __str__ methods
MR2029 Remove copy/paste mistake from Eltex.MES.get_config script
MR2034 noc-model-str-cm-facts2
MR2035 noc-model-str-core-ip
MR2036 noc-model-str-services-lib
MR2049 Test model's __str__
MR2050 py3 dict .iteritems(), .itervalues(), .iterkeys()
MR2051 2to3 except fix
MR2054 2to3 exec fix
MR2060 Remove urllib usage in profiles
MR2063 test_base_parametrize. ver1
MR2064 Wrap urllib/urllib2 usage
MR2067 wrap StringIO
MR2074 2to3 import fix
MR2075 2to3 has_key fix
MR2076 2to3 fix urlparse
MR2080 wrap maketrans call
MR2081 2to3 print fix
MR2082 2to3 fix map and filter
MR2083 2to3 imports fix
MR2085 2to3 itertools and itertools_import fixes
MR2086 2to3 long fix
MR2089 2to3 types fix
MR2094 test_ecma48
MR2095 test_ber
MR2101 Remove south usage from BaseMigration class
MR2106 Migration Runner, Bye-bye South
MR2107 Cleanup models' init.py
MR2120 fix_DeprecationWarning_main_0049_update_tags
MR2121 fix_test_ip
MR2123 test_crypto
MR2124 test_mac
MR2133 remove_total_seconds
MR2135 test_matcher
MR2136 test_validators
MR2221 docs format mr tables
MR2236 Fix docs path
MR2242 Fix mongoengine imports
MR2243 Fix mongoengine imports
MR2251 Clean Huawei.MA5600T profile
MR2263 2to3: zip fix
MR2264 Remove unused cm templates
MR2265 2to3: next fix
MR2266 Fix ObjectId import
MR2271 2to3: dict fix
MR2283 2to3: xrange fix
MR2291 black: aaa, bi, cm
MR2306 black: crm, dev, dns, fixes, fm
MR2307 black: gis, inv, ip, kb
MR2310 fix_import_lib_nosql_part1
MR2311 fix_import_lib_nosql_part2
MR2314 black: migrations
MR2315 fix_import_lib_nosql_part3
MR2316 fix_import_lib_nosql_part4
MR2318 black: sa
MR2319 black: lib
MR2320 black: core
MR2321 black: Rest of stuff
MR2330 Bump version
MR2354 Add mongo connect to commands.
MR2358 Speedup docker release images build with targeted builds
MR2406 Drop unused fields
MR2411 ConfDB syntax refactoring
MR2422 Move profiles to profile.py
MR2424 Fix get_version Infinet.Wanflex
MR2428 Clean up noc.lib.nosql imports
MR2439 Fix documentation glitches
MR2452 Update Dynamic Dashboards.